trapexit

Just download the package and install them wherever. https://github.com/trapexit/mergerfs/releases/download/2.36.0/mergerfs-static-linux_amd64.tar.gz It's just a standard tarball with statically compiled binaries. I build them every release.

Because you aren't pulling the latest version of the build image. Docker doesn't just pull updates on demand. It pulls if not already cached or when explicitly told. docker pull trapexit/mergerfs-static-build

1) 2.37.0 isn't released yet so technically you're on your own. 2) I provide static builds now so you could just use those 3) I updated the build image yesterday just for this reason. You have to pull the new version of the image if you want it to work. Apparently MUSL 4 years ago didn't support features I'm using. https://github.com/trapexit/mergerfs/issues/1246

re mergerfs.balance: The original idea was to leave things like that to tools outside the main program so the community could do whatever they want and the it would minimize the risk of having such behavior inside the app where it wasn't really necessary to live. But it really didn't work out that way. Those tools were really just examples which is why I never properly packaged them. For mergerfs v3 (which is still a ways off) I'm considering incorporating some sort of active balance behavior. For me I just spin up another pool and rsync to it where I've removed the underlying drive in question. That way I don't need to recreate any of the policy behavior. I might still keep "balance" and "draining" or whatnot as external to the main app but reuse the code.

I'm not really familiar with unraid's filesystem so I can't speak to what this "hardlink off" thing is or does. No such feature in mergerfs. Unless there is some faking of hardlinks I'm not sure I see how it'd matter here. FUSE's "remember" or "noforget" is about the internal node value. The 2x64bit node + generation value used between the kernel and the FUSE server to keep track of entries in the filesystem. Normally the kernel will tell the server to forget an entry/node when it no longer references it. It might ask about it again in the future in which case you use a new node + generation values. However, as I understand it, NFS may store that info and ask for it even if the kernel told the fuse server to forget about it. So those options cause libfuse to keep the values around for a certain amount of time or forever *just in case* someone asks about them. If NFS asks and that entry is no longer known... stale file handle error. Theoretically it's possible to store those values to disk so as to limit memory consumption but that's not a feature of the library. There would also be other tradeoffs there. If a user facing inode value changes out of band that too can cause the stale file error as from the perspective of NFS shit just changed out of band and it can't trust the situation. This happens in mergerfs when someone has a pool and the move a file from say... local disk to rclone mount. From your perspective it's the same file but the change was out of band. Neither mergerfs or NFS know it's safe to treat the same. The inode changing means it's a different file even if other values like size and timestamps are the same. mergerfs just doesn't care. It reports what it sees. One idea is to optionally cache inode values. Whatever value is seen the first time we see the value we always use it going forward. There is an increased memory usage of course to keep track of that as well as a slight compute cost but would probably help in these out of band move situations. Adding it to mergerfs of course wouldn't help unraid but could be something they may look to add (assuming that's the/a cause here.)

There simply isn't a practical way to manage the situation which might be why there isn't much of a response (though there perhaps should be.) UnRAID is in the same boat as any other filesystem. If you change things out of band from SMB or NFS they get unhappy. union filesystems make this more likely but it is a difference of degrees not kind. Statelessness of older protocols was given up in favor of stateful so as to increase performance. Could SMB and NFS have been designed to hide some of the issues from out of band changes when detected? Perhaps. But that's not the world we live in and to attempt to present to SMB / NFS a consistent world view when the union filesystem itself doesn't know is very non-trivial. mergerfs uses some optional tricks to help with the situation but they aren't perfect. NFS and the kernel can notice certain situations where an inode should have changed and didn't or shouldn't have and did.

The complexity of one's setup isn't generally speaking relevant to this issue. It's due to out of band changes to the underlying storage and NFS or Samba losing track of the files. I don't know enough about unraid's union filesystem but from what I gather some people's issues are likely directly related to that. Others might just be casual apps doing out of band changes.

Stale File Handle errors occur due to NFS or Samba losing track of a file it was referencing. Discounting bugs that means out of band changes. The cache feature of unraid (with the moving of that data) is exactly that. I'm not expert on the SMB/CIFS protocol but I'm guessing v1.0 is more stateless and therefore not as prone to the problem. I created a work around in mergerfs for this but it does have downsides. When dealing with out of band changes it's not that easy to manage this situation.

The other thing that can slow down data transfers when using file caching is extended attributes. Did anyone try disabling them? xattr=nosys The kernel unfortunately doesn't cache positive or negative responses for getxattr or listxattr so mergerfs can get hammered with security.capability requests *every single write*. It completely destroys performance on most systems. https://github.com/trapexit/mergerfs#xattr Perhaps I can go into more detail in the docs as to why this might be a problem but it is mentioned in the performance testing section.

Caching can lead to weird issues on some systems depending on the workload which is why I suggest turning it off normally. I don't know what kernel you're running but there have been some funny stuff lately. I can remove that example. These tests are not trivial. Especially on an in use system. There are a lot of variables involved.

minfreespace is a global value so sure that'd setup work. Though it means that when both fill to the point of having only 100GB free it will return it's out of space. I could be more clear there but the idea is that you have 2 pools and regularly are syncing files over to the slower drives. Setting minfreespace on the "slow" drive pool to the largest cache drive means that worse case you should be able to fit a full drive worth of content onto the slow pool. I'll update the docs to make it more clear.

Why are you setting async_read=false? You appear to be using local filesystems.

Outside what is described in the performance section of the docs I really don't have anything else to add without additional information. https://github.com/trapexit/mergerfs#performance Clearly something has changed in your setup because mergerfs really hasn't. You'll need to narrow that down. Has the UnRAID kernel changed in this time? Hardware? There are a lot of things that effect performance.

mergerfs hasn't changed in any significant way in 5+ months. In fact I only just released a new version a 2 weeks ago and had no default behavior changes from the release prior.

Well, this is where I'm a bit confused. The "broken build" was literally a broken build. It didn't spit out a binary. So it couldn't have lead to any runtime mergerfs issues but there seems to be claims of that.