[PLUGIN] mergerFS for UNRAID (Support Topic)

[Rysz]

mergerFS for UNRAID (6.10+)

A plugin that installs mergerFS, a featureful union filesystem, onto UNRAID systems.

 

mergerfs is a union filesystem geared towards simplifying storage and management of files across numerous commodity storage devices.

It is similar to mhddfs, unionfs, and aufs.

 

How to install?

Installable via Community Applications

 

How does it work?

!!! PLEASE READ THE DOCUMENTATION BEFORE RUNNING ANY COMMANDS !!!

mergerFS is a non-supported filesystem and you should know what you are doing.

If you don't know what you are doing, you can easily wreak havoc on your system!

 

https://github.com/trapexit/mergerfs#readme

 

After installation the mergerFS binaries are available on your UNRAID system and will persist reboots:

/usr/bin/mergerfs (i.e for mounting)

/usr/bin/mergerfs-fusermount (i.e. for unmounting)

 

You can now make use of them via shell scripting - see "mergerFS Settings" for array-status based scripts surviving reboots.

If you do not want to use the inbuilt event hooks, you can also use the binaries through i.e. the "User Scripts" plugin! 

 

How to contribute?

https://github.com/desertwitch/mergerFS-unRAID

... and of most importantly - please do test and report back here!

 

Special Credits?

@trapexit - creator of mergerFS 🏆

 

Edited June 18 by Rysz

[milouz]

May I ask you what is the prupose of using mergerfs on unraid?

[Rysz]

17 minutes ago, milouz said:

May I ask you what is the prupose of using mergerfs on unraid?

 

One example would be merging cloud drives (i.e. rclone mounts) with local files for use with Plex etc.

 

Edited September 12, 2023 by Rysz

[Admin9705]

1 hour ago, Rysz said:

 

One example would be merging cloud drives (i.e. rclone mounts) with local files for use with Plex etc.

 

MergerFS is super useful. I wrote plexguide in utilizing unionfs/mergerfs. There is one warning I do have and maybe you're well ahead of this one. There are rare times mergerfs can fail and I mean super rare. Also, other locations if they go offline, it can cause problems.

 

I had to write a script in the program to shutdown the docker containers when mergerfs would go offline. Plex assumed everything just went missing. When that happens, plex will remove all of its own content because it assumes that you deleted it. To fix the problem, I wrote a background script check a dummy file every 1 minute. If that dummy file went missing, it would check it 2 more times. If it was still missing, then the containers would all shutdown... and then the script would attempt to bring mergerfs back and when the dummy file was found again. When found, the script would restart all the containers.

 

Anyways, good luck and yes adding cloud and other locations can be super useful.

 

Edited September 12, 2023 by Admin9705

[drpudding]

On 9/12/2023 at 7:25 AM, Rysz said:

 

One example would be merging cloud drives (i.e. rclone mounts) with local files for use with Plex etc.

 

 

basically what i did. wrote out a whole script for it once i saw this plugin was on the CA page so the binaries are always there.

more or less here.

[Rysz]

8 minutes ago, drpudding said:

 

basically what i did. wrote out a whole script for it once i saw this plugin was on the CA page so the binaries are always there.

more or less here.

 

Nice work and thanks for posting it here - nice to see this working and incorporated into another script. 🙂

[drpudding]

13 hours ago, Rysz said:

 

Nice work and thanks for posting it here - nice to see this working and incorporated into another script. 🙂

np, i was wondering if it could just be made into a plugin tbh.
but i couldnt find any documentation on how plugins are created for unraid.

that way the rclone config and paths can be configured via the webui and maybe have some sort of control in terms of remount in the event of network loss or fuse just crashing

 

[Pauven]

A year ago I created an easy, affordable backup solution for my Unraid server.  Essentially just a stack of external USB drives that I mounted with Unassigned Devices and joined together in a BTRFS JBOD style pool.  With 5x 16TB drives, this gave me a single 80TB storage volume.  At the time, this solution seemed perfect.  I had a backup script that used RSYNC to copy my files to the single mount point, and I thought that BTRFS filled up each drive one-by-one.  Since my Unraid data is basically already a backup of my physical data, having this portable backup volume that could be stored offsite was more than I needed, even without any built-in redundancy.

 

This week, while adding a new 20TB drive to expand this pool up to 100TB, I learned I made several mistakes in my backup solution.

 

First, when adding the new drive I made a few mistakes and ended up corrupting the BTRFS pool.  And since my pool had no redundancy, BTRFS prohibits mounting it in RW mode to fix it, so the only option was to start over, recreate the entire pool, and re-backup the original 80TB of data. 

 

That was painful enough.  But in redoing all this, I discovered that BTRFS is automatically balancing, writing to the drive with the most free space for each file.  With the nature of the data I'm storing, losing a single drive would now make the entire backup worthless as I need each directory to remain whole on a single drive, and can't lose any files inside each directory.  While my BTRFS backup pool is better than nothing, this is way too fragile for me to continue using it.

 

While researching solutions, I came across MergeFS and eventually this thread.  This sounds like the right type of solution.

 

My core requirements are to plug in my USB drives, mount them as a single filesystem, and run a backup script to copy any new/altered data to my backup pool, with data filling up each drive, one-by-one, before moving on to the next drive.  That way, if I lose a drive, I only lose the data backed up to that one drive, plus any directories that happened to be spanning the transition between drives.

 

Sorry for the long lead-in.  Now to my questions:

 

Is the plugin on CA yet?  I searched and can't find it, so I'm assuming I have to install it via URL.

 

Can someone help me with the configuration?  I read through the MergerFS github page, and there's tons of options and the examples don't seem to apply to my use case.  I'm a bit overwhelmed.  I need commands for configuring, mounting, unmounting, and expanding the pool.

 

Thanks!

-Paul

 

 

Edited November 16, 2023 by Pauven

[Rysz]

6 hours ago, Pauven said:

A year ago I created an easy, affordable backup solution for my Unraid server.  Essentially just a stack of external USB drives that I mounted with Unassigned Devices and joined together in a BTRFS JBOD style pool.  With 5x 16TB drives, this gave me a single 80TB storage volume.  At the time, this solution seemed perfect.  I had a backup script that used RSYNC to copy my files to the single mount point, and I thought that BTRFS filled up each drive one-by-one.  Since my Unraid data is basically already a backup of my physical data, having this portable backup volume that could be stored offsite was more than I needed, even without any built-in redundancy.

 

This week, while adding a new 20TB drive to expand this pool up to 100TB, I learned I made several mistakes in my backup solution.

 

First, when adding the new drive I made a few mistakes and ended up corrupting the BTRFS pool.  And since my pool had no redundancy, BTRFS prohibits mounting it in RW mode to fix it, so the only option was to start over, recreate the entire pool, and re-backup the original 80TB of data. 

 

That was painful enough.  But in redoing all this, I discovered that BTRFS is automatically balancing, writing to the drive with the most free space for each file.  With the nature of the data I'm storing, losing a single drive would now make the entire backup worthless as I need each directory to remain whole on a single drive, and can't lose any files inside each directory.  While my BTRFS backup pool is better than nothing, this is way too fragile for me to continue using it.

 

While researching solutions, I came across MergeFS and eventually this thread.  This sounds like the right type of solution.

 

My core requirements are to plug in my USB drives, mount them as a single filesystem, and run a backup script to copy any new/altered data to my backup pool, with data filling up each drive, one-by-one, before moving on to the next drive.  That way, if I lose a drive, I only lose the data backed up to that one drive, plus any directories that happened to be spanning the transition between drives.

 

Sorry for the long lead-in.  Now to my questions:

 

Is the plugin on CA yet?  I searched and can't find it, so I'm assuming I have to install it via URL.

 

Can someone help me with the configuration?  I read through the MergerFS github page, and there's tons of options and the examples don't seem to apply to my use case.  I'm a bit overwhelmed.  I need commands for configuring, mounting, unmounting, and expanding the pool.

 

Thanks!

-Paul

 

 

 

It's on CA, but it's for UNRAID versions 6.10+, so your 6.8.3 (according to your signature) is too outdated. 😕

 

Edited November 16, 2023 by Rysz

[Pauven]

Thanks Rysz.  Actually, it's my signature that's really outdated, hah!  But I was still on 6.9.2, and I had to upgrade to 6.10+ even to use the URL method.

 

I'm on the latest 6.12 now, and I was able to install from URL.  I assume it's the same MergerFS release as the CA version.

 

I like MergerFS, it's working as I hoped.  But it's not perfect.  The "Create" option is static for files vs. directories, and I was finding that it would create a directory, write some files to it, drop below the minimum free space, and then create a new directory on a different branch.  Considering that I'm backing up uncompressed blu-rays, typically around 45 GB in size, I need the min free space for creating a directory to be at least 45 GB higher than the min free space for creating files.

 

To solve this, I customized the mirror.sh script someone else wrote (which is used to create each directory right before files are written to it, rather than creating all empty directories first and then copying files).  I changed it to have it create directories based upon 100 GB min free space, and to evaluate my MergerFS branches in a particular sequence.  I then was able to configure MergerFS with a much lower 4 GB min free space, which only applies to files since my script creates the directories. 

 

When used with MergerFS's "ep" Existing Path option, I now have MergerFS writing the backup files to where my backup script creates the directories.  This allows me to keep my blu-ray disc directories whole on a single drive, and all my MergerFS branches fill up one-by-one.  I'm in backup nirvana!!!

[Rysz]

1 minute ago, Pauven said:

Thanks Rysz.  Actually, it's my signature that's really outdated, hah!  But I was still on 6.9.2, and I had to upgrade to 6.10+ even to use the URL method.

 

I'm on the latest 6.12 now, and I was able to install from URL.  I assume it's the same MergerFS release as the CA version.

 

I like MergerFS, it's working as I hoped.  But it's not perfect.  The "Create" option is static for files vs. directories, and I was finding that it would create a directory, write some files to it, drop below the minimum free space, and then create a new directory on a different branch.  Considering that I'm backing up uncompressed blu-rays, typically around 45 GB in size, I need the min free space for creating a directory to be at least 45 GB higher than the min free space for creating files.

 

To solve this, I customized the mirror.sh script someone else wrote (which is used to create each directory right before files are written to it, rather than creating all empty directories first and then copying files).  I changed it to have it create directories based upon 100 GB min free space, and to evaluate my MergerFS branches in a particular sequence.  I then was able to configure MergerFS with a much lower 4 GB min free space, which only applies to files since my script creates the directories. 

 

When used with MergerFS's "ep" Existing Path option, I now have MergerFS writing the backup files to where my backup script creates the directories.  This allows me to keep my blu-ray disc directories whole on a single drive, and all my MergerFS branches fill up one-by-one.  I'm in backup nirvana!!!

 

Glad to hear it - thanks for reporting back! 🙂

[sjtuross]

Thanks for this plugin. Do you think if the tools from https://github.com/trapexit/mergerfs-tools can be packaged together?

Edited December 17, 2023 by sjtuross

[Rysz]

19 hours ago, sjtuross said:

Thanks for this plugin. Do you think if the tools from https://github.com/trapexit/mergerfs-tools can be packaged together?

 

I've just pushed an update including mergerFS-Tools, beware you need to install Python 3.x to use the mergerFS-Tools.

Python 3.x. can be installed - for example - via NerdTools, if you don't already have it installed on your system for something else. 🙂

[sjtuross]

I put some script in /etc/mergerfsp/array_start.sh to wait a container ready before running mergerfs. It blocks array start forever because docker won't start until array is started. Is it possible to run this array_start.sh asynchronously?

[Rysz]

28 minutes ago, sjtuross said:

I put some script in /etc/mergerfsp/array_start.sh to wait a container ready before running mergerfs. It blocks array start forever because docker won't start until array is started. Is it possible to run this array_start.sh asynchronously?

 

In this case it's probably better to use the "User Scripts" plugin and hook into a different array event where both the array and Docker are started already. array_start.sh is just a simple helper script that hooks into the specific event where the disks are mounted (but nothing else is started yet). The reason is because in most cases users probably want their mergerFS filesystem to be up before the other services are starting (and depending on it being there) and not the other way around. So in your case: "User Scripts" plugin would be a better choice. 🙂

 

Edited January 26 by Rysz

[sjtuross]

Thanks for the quick response. Yes, in my case, it's the other way around because I want to merge some cloud drives which are mounted by a container. Do you happen to know if the "At Startup of Array" event in the "User Scripts" plugin would block array start?

[Rysz]

I've just pushed the update including more event hooks, check out "array_start_complete.sh".

Please test and let me know if that works for your use case, this should run after everything else is done. 🙂

 

Make sure to delete your previous code from the other script ("array_start.sh") so your array startup doesn't hang as before.

 

Edited January 26 by Rysz

[sjtuross]

On 1/26/2024 at 6:47 PM, Rysz said:

I've just pushed the update including more event hooks, check out "array_start_complete.sh".

Please test and let me know if that works for your use case, this should run after everything else is done. 🙂

 

Make sure to delete your previous code from the other script ("array_start.sh") so your array startup doesn't hang as before.

 

array_start_complete works as expected. Thank you!

Edited February 17 by sjtuross

[spike32]

Hi,

 

I have just started using Unraid and I setup mergerfs and it's been working perfectly under /mnt/user/mini-pc. However when I look at the logs, I get this message over and over.

 

 

Not sure how to fix this, I have included diags.

 

Thanks!

mini-pc-diagnostics-20240302-1710.zip

[Rysz]

On 3/3/2024 at 6:45 AM, spike32 said:

Hi,

 

I have just started using Unraid and I setup mergerfs and it's been working perfectly under /mnt/user/mini-pc. However when I look at the logs, I get this message over and over.

 

 

Not sure how to fix this, I have included diags.

 

Thanks!

mini-pc-diagnostics-20240302-1710.zip 139.57 kB · 0 downloads

 

This is because you put the mergerFS mount inside /mnt/user which interferes with the regular Unraid shares. You'll have to put it either in /mnt/addons/ and set up a custom Samba share directive OR create a regular Unraid share e.g. "media" through the GUI and put your mergerFS mount inside that (/mnt/user/media/mini-pc/). Let me know if that worked for you 🙂 

 

Edited April 23 by Rysz

[spike32]

5 hours ago, Rysz said:

 

This is because you put the mergerFS mount inside /mnt/user which interferes with the regular Unraid shares. You'll have to put it either in /mnt and set up a custom Samba share directive OR create a regular Unraid share e.g. "media" through the GUI and put your mergerFS mount inside that (/mnt/user/media/mini-pc/). Let me know if that worked for you 🙂

 

Thanks for the quick and helpful response! I did the regular Unraid share and so far no errors!

[AgentXXL]

As I'm dealing with very erratic power pricing and have limited disability income, I'm getting desperate to combine my two unRAID systems into one. I was hoping that 6.13/7.0 would have the ability to run multiple pools with the unRAID parity scheme, but the 6.13 beta test is just underway so it'll be a while yet before we see a RC. It's now confirmed that 6.13 will not implement the change to an 'all pool' model and hence the idea of using multiple pools with the unRAID parity scheme is moot until a future release implements it. As I need to tackle my power consumption ASAP, I'm looking at 2 options:

 

1. Create an unRAID VM on my main unRAID system. I have a HBA and USB ports to passthrough so that shouldn't be an issue. It will be for storage only - no other containers or VMs would be run on the unRAID VM. Only the absolutely necessary plugins like the UD series will be installed.

 

2. Use the mergerfs plugin to create a pool of different sized devices. Alas I don't see a way to do this and also implement something like SnapRAID so that there's some fault tolerance. I see multiple SnapRAID containers on Dockerhub but none are available in the unRAID App store so I'll have to try and build a container template for it. Looks like the official releases are from https://github.com/amadvance/snapraid

 

Suggestions or other ideas?

Edited March 17 by AgentXXL
Added that all-pool model is delayed to a future unRAID release

[AgentXXL]

Hi Rysz. I'm giving your mergerfs plugin a try and have a couple of quick questions. For my test I'm using my test box. I've added 10 drives from my offline backups ranging in size from 2TB to 8TB. All are XFS formatted with a single partition and all use the same root disk name of OfflineBU, eg. OfflineBU00, OfflineBU01, etc.

 

If I try to use mergerfs via commandline with the following command, it fails:

root@AnimTest:~# mergerfs -o cache.files=partial,dropcacheonclose=true,category.create=mfs /mnt/disks/OfflineBU* /mnt/addons/BUPool

fuse: invalid argument `/mnt/disks/OfflineBU02'

 

It appears to work if I don't use a wildcard but list each drive separately like this:

mergerfs -o cache.files=partial,dropcacheonclose=true,category.create=mfs /mnt/disks/OfflineBU00:/mnt/disks/OfflineBU01:/mnt/disks/OfflineBU02:/mnt/disks/OfflineBU03:/mnt/disks/OfflineBU04:/mnt/disks/OfflineBU05:/mnt/disks/OfflineBU06:/mnt/disks/OfflineBU07:/mnt/disks/OfflineBU08:/mnt/disks/OfflineBU09 /mnt/addons/BUPool

 

It appears all disks in the mergerfs pool will still show up under Unassigned Devices. I suspect there's no way to hide these drives from showing under UD?

 

Any thoughts on why I can't use a wildcard with all drives mounted using the same root disk name? If I have to use all drives on the command line, it's going to get quite long - the pool I want to mount on my production server will have 30 disks. Not sure if unRAID has a limit to the length of the command? Appearing under UD isn't a big issue, other than each visit to the Main tab takes longer to refresh while it waits for the list from UD to populate.

 

[Rysz]

5 hours ago, AgentXXL said:

Hi Rysz. I'm giving your mergerfs plugin a try and have a couple of quick questions. For my test I'm using my test box. I've added 10 drives from my offline backups ranging in size from 2TB to 8TB. All are XFS formatted with a single partition and all use the same root disk name of OfflineBU, eg. OfflineBU00, OfflineBU01, etc.

 

If I try to use mergerfs via commandline with the following command, it fails:

root@AnimTest:~# mergerfs -o cache.files=partial,dropcacheonclose=true,category.create=mfs /mnt/disks/OfflineBU* /mnt/addons/BUPool

fuse: invalid argument `/mnt/disks/OfflineBU02'

 

It appears to work if I don't use a wildcard but list each drive separately like this:

mergerfs -o cache.files=partial,dropcacheonclose=true,category.create=mfs /mnt/disks/OfflineBU00:/mnt/disks/OfflineBU01:/mnt/disks/OfflineBU02:/mnt/disks/OfflineBU03:/mnt/disks/OfflineBU04:/mnt/disks/OfflineBU05:/mnt/disks/OfflineBU06:/mnt/disks/OfflineBU07:/mnt/disks/OfflineBU08:/mnt/disks/OfflineBU09 /mnt/addons/BUPool

 

It appears all disks in the mergerfs pool will still show up under Unassigned Devices. I suspect there's no way to hide these drives from showing under UD?

 

Any thoughts on why I can't use a wildcard with all drives mounted using the same root disk name? If I have to use all drives on the command line, it's going to get quite long - the pool I want to mount on my production server will have 30 disks. Not sure if unRAID has a limit to the length of the command? Appearing under UD isn't a big issue, other than each visit to the Main tab takes longer to refresh while it waits for the list from UD to populate.

 

 

Regarding UD, I'm not sure if you can hide drives from there - I don't know how that plugin selects the drives to display or not.

Looks like there is no way at the moment:

 

Regarding Globbing, you need to escape the globbing characters as follows:

mergerfs -o cache.files=partial,dropcacheonclose=true,category.create=mfs /mnt/disks/OfflineBU\* /mnt/addons/BUPool

See also here for more details: https://github.com/trapexit/mergerfs#globbing

 

Edited April 19 by Rysz

[DiscoDuck]

Has anyone tried to utilize the preload.so?

https://github.com/trapexit/mergerfs/blob/2.39.0/README.md#preloadso

To my understanding it should be possible to obtain native disk speeds in qbit/rtorrent if it's passed to the docker container(s)