trapexit

For anyone who wishes to see exactly what mergerfs is doing here... https://github.com/trapexit/mergerfs/blob/latest-release/src/policy_epff.cpp#L108 You don't need to be a developer to understand this is extremely trivial. @Rysz Glad it is useful. And thank you for building and maintaining this plugin.

mergerfs does not care about the underlying filesystem type. All policies work exactly as described in the docs in relation to selection algorithm, filtering, response to readonly, etc. If `/boot` is first and doesn't get filtered out for whatever reason as described in the docs... it would be selected by `ff`.

It does. How are you testing what is opened? As I say in my support docs... I need you to explain very very explicitly what you are doing for me to help. The ff policy is extremely simple and been around for a decade. It works.

Sorry, I thought the question was "use" as in mechanically... safe to use together. Not legally. Yes. mergerfs is ISC licensed. Do whatever you wish within the license requirements. All that I ask is that if you are building tooling / services on top of mergerfs that you tell me about it so I can include it in the docs. https://trapexit.github.io/mergerfs/related_projects/

I'm not a user of UnRAID or all that familiar with the setup being discussed so I really can't speak to what is appropriate or not.

Sorry but I'm not sure what you are asking. It would be more helpful if you could explain what is unclear in the documentation. https://trapexit.github.io/mergerfs/usage_patterns/#tiered-cache

No problem. What you show is `category.create=ff` which means on *create* it will choose the first branch found. You asked about overlapping existing files. That is controlled by the `open` and other function policies. The category wide setting being `category.search=ff`.

Policies decide how a file is selected. You asked how mergerfs selects a file, the "priority". Policies. The first found policy picks the first found. rand chooses random. Etc.

That is all decided by the different policies and such: https://trapexit.github.io/mergerfs/config/functions_categories_and_policies/

My scripts linked to in the docs will move files by time or by percentage full. It sounds to me you should use the latter.

I'm not aware of any software that does that generically. You have to script something yourself. I have some simple shell scripts that do size and time based movement linked from the mergerfs docs. I've been meaning to write something more involved but just haven't gotten around to it. You could look at https://gist.github.com/trapexit/14ef971951b6c1096a7acec203754faa as a template. Move files instead of remove them. I have mine run ever hour.

What's the reason for the cache here? If you have files constantly being written to the cache then you will simply be constantly emptying the cache. Or at least, assuming you can empty the ssd faster than what is written, will do it in regular batches. I use mergerfs to store camera recordings and for the above reason I just let it do its thing directly. A cache is really only useful for bursty loads. I do have a simple app I wrote to clear out old files but that is not mergerfs specific.

Just download the package and install them wherever. https://github.com/trapexit/mergerfs/releases/download/2.36.0/mergerfs-static-linux_amd64.tar.gz It's just a standard tarball with statically compiled binaries. I build them every release.

Because you aren't pulling the latest version of the build image. Docker doesn't just pull updates on demand. It pulls if not already cached or when explicitly told. docker pull trapexit/mergerfs-static-build

1) 2.37.0 isn't released yet so technically you're on your own. 2) I provide static builds now so you could just use those 3) I updated the build image yesterday just for this reason. You have to pull the new version of the image if you want it to work. Apparently MUSL 4 years ago didn't support features I'm using. https://github.com/trapexit/mergerfs/issues/1246

re mergerfs.balance: The original idea was to leave things like that to tools outside the main program so the community could do whatever they want and the it would minimize the risk of having such behavior inside the app where it wasn't really necessary to live. But it really didn't work out that way. Those tools were really just examples which is why I never properly packaged them. For mergerfs v3 (which is still a ways off) I'm considering incorporating some sort of active balance behavior. For me I just spin up another pool and rsync to it where I've removed the underlying drive in question. That way I don't need to recreate any of the policy behavior. I might still keep "balance" and "draining" or whatnot as external to the main app but reuse the code.

I'm not really familiar with unraid's filesystem so I can't speak to what this "hardlink off" thing is or does. No such feature in mergerfs. Unless there is some faking of hardlinks I'm not sure I see how it'd matter here. FUSE's "remember" or "noforget" is about the internal node value. The 2x64bit node + generation value used between the kernel and the FUSE server to keep track of entries in the filesystem. Normally the kernel will tell the server to forget an entry/node when it no longer references it. It might ask about it again in the future in which case you use a new node + generation values. However, as I understand it, NFS may store that info and ask for it even if the kernel told the fuse server to forget about it. So those options cause libfuse to keep the values around for a certain amount of time or forever *just in case* someone asks about them. If NFS asks and that entry is no longer known... stale file handle error. Theoretically it's possible to store those values to disk so as to limit memory consumption but that's not a feature of the library. There would also be other tradeoffs there. If a user facing inode value changes out of band that too can cause the stale file error as from the perspective of NFS shit just changed out of band and it can't trust the situation. This happens in mergerfs when someone has a pool and the move a file from say... local disk to rclone mount. From your perspective it's the same file but the change was out of band. Neither mergerfs or NFS know it's safe to treat the same. The inode changing means it's a different file even if other values like size and timestamps are the same. mergerfs just doesn't care. It reports what it sees. One idea is to optionally cache inode values. Whatever value is seen the first time we see the value we always use it going forward. There is an increased memory usage of course to keep track of that as well as a slight compute cost but would probably help in these out of band move situations. Adding it to mergerfs of course wouldn't help unraid but could be something they may look to add (assuming that's the/a cause here.)

There simply isn't a practical way to manage the situation which might be why there isn't much of a response (though there perhaps should be.) UnRAID is in the same boat as any other filesystem. If you change things out of band from SMB or NFS they get unhappy. union filesystems make this more likely but it is a difference of degrees not kind. Statelessness of older protocols was given up in favor of stateful so as to increase performance. Could SMB and NFS have been designed to hide some of the issues from out of band changes when detected? Perhaps. But that's not the world we live in and to attempt to present to SMB / NFS a consistent world view when the union filesystem itself doesn't know is very non-trivial. mergerfs uses some optional tricks to help with the situation but they aren't perfect. NFS and the kernel can notice certain situations where an inode should have changed and didn't or shouldn't have and did.

The complexity of one's setup isn't generally speaking relevant to this issue. It's due to out of band changes to the underlying storage and NFS or Samba losing track of the files. I don't know enough about unraid's union filesystem but from what I gather some people's issues are likely directly related to that. Others might just be casual apps doing out of band changes.

Stale File Handle errors occur due to NFS or Samba losing track of a file it was referencing. Discounting bugs that means out of band changes. The cache feature of unraid (with the moving of that data) is exactly that. I'm not expert on the SMB/CIFS protocol but I'm guessing v1.0 is more stateless and therefore not as prone to the problem. I created a work around in mergerfs for this but it does have downsides. When dealing with out of band changes it's not that easy to manage this situation.

The other thing that can slow down data transfers when using file caching is extended attributes. Did anyone try disabling them? xattr=nosys The kernel unfortunately doesn't cache positive or negative responses for getxattr or listxattr so mergerfs can get hammered with security.capability requests *every single write*. It completely destroys performance on most systems. https://github.com/trapexit/mergerfs#xattr Perhaps I can go into more detail in the docs as to why this might be a problem but it is mentioned in the performance testing section.

Caching can lead to weird issues on some systems depending on the workload which is why I suggest turning it off normally. I don't know what kernel you're running but there have been some funny stuff lately. I can remove that example. These tests are not trivial. Especially on an in use system. There are a lot of variables involved.

minfreespace is a global value so sure that'd setup work. Though it means that when both fill to the point of having only 100GB free it will return it's out of space. I could be more clear there but the idea is that you have 2 pools and regularly are syncing files over to the slower drives. Setting minfreespace on the "slow" drive pool to the largest cache drive means that worse case you should be able to fit a full drive worth of content onto the slow pool. I'll update the docs to make it more clear.

Why are you setting async_read=false? You appear to be using local filesystems.

Outside what is described in the performance section of the docs I really don't have anything else to add without additional information. https://github.com/trapexit/mergerfs#performance Clearly something has changed in your setup because mergerfs really hasn't. You'll need to narrow that down. Has the UnRAID kernel changed in this time? Hardware? There are a lot of things that effect performance.