September 13, 20241 yr 7 hours ago, Rysz said: Get this dependency package (the commands downloads it to /boot/extra and installs it, so it persists reboots): curl -o /boot/extra/libffi-3.3-x86_64-3.txz https://slackware.uk/slackware/slackware64-15.0/slackware64/l/libffi-3.3-x86_64-3.txz installpkg /boot/extra/libffi-3.3-x86_64-3.txz Please let me know if that worked for you! Worked perfectly! I do realize that balance might not be what I'm looking for exactly, I'm using my merge mount as storage for my Frigate camera recordings, wanted mergerfs.balance to move older recordings from the SSD to the spinning disk within my merge (1TB /mnt/frigate_cache to 6TB /mnt/frigate_archive) hence the 25% balance, but I realize now that balance probably doesn't pay attention to the age of the files. Is there another tool or perhaps a script that you know of that would accomplish what I'm looking for?
September 13, 20241 yr Author 7 minutes ago, MTerracciano said: Worked perfectly! I do realize that balance might not be what I'm looking for exactly, I'm using my merge mount as storage for my Frigate camera recordings, wanted mergerfs.balance to move older recordings from the SSD to the spinning disk within my merge (1TB /mnt/frigate_cache to 6TB /mnt/frigate_archive) hence the 25% balance, but I realize now that balance probably doesn't pay attention to the age of the files. Is there another tool or perhaps a script that you know of that would accomplish what I'm looking for? This might be what you are looking for: https://github.com/trapexit/mergerfs#tiered-caching But have no experience with it personally...
September 13, 20241 yr What's the reason for the cache here? If you have files constantly being written to the cache then you will simply be constantly emptying the cache. Or at least, assuming you can empty the ssd faster than what is written, will do it in regular batches. I use mergerfs to store camera recordings and for the above reason I just let it do its thing directly. A cache is really only useful for bursty loads. I do have a simple app I wrote to clear out old files but that is not mergerfs specific.
September 13, 20241 yr Author Just now, trapexit said: What's the reason for the cache here? If you have files constantly being written to the cache then you will simply be constantly emptying the cache. Or at least, assuming you can empty the ssd faster than what is written, will do it in regular batches. I use mergerfs to store camera recordings and for the above reason I just let it do its thing directly. A cache is really only useful for bursty loads. I do have a simple app I wrote to clear out old files but that is not mergerfs specific. Y'all heard the boss! 😄 ...thanks for chiming in @trapexit ... 🙂
September 14, 20241 yr 19 hours ago, trapexit said: What's the reason for the cache here? If you have files constantly being written to the cache then you will simply be constantly emptying the cache. Or at least, assuming you can empty the ssd faster than what is written, will do it in regular batches. I use mergerfs to store camera recordings and for the above reason I just let it do its thing directly. A cache is really only useful for bursty loads. I do have a simple app I wrote to clear out old files but that is not mergerfs specific. Specifically for reviewing recently recorded events/timeline scrubbing, everything loads much faster when stored on the SSD and I'm far more likely to be reviewing the last few days of recording than I am to go back 30days where I'm fine with it loading a bit slower due to the footage being moved to/living on the hard drive. Ideally I'd schedule the move every day or two to move files older than a week to the hard drive, gives me the best of both worlds, long term storage and fast reviewing of recent events. Thanks again for your help!
September 14, 20241 yr I'm not aware of any software that does that generically. You have to script something yourself. I have some simple shell scripts that do size and time based movement linked from the mergerfs docs. I've been meaning to write something more involved but just haven't gotten around to it. You could look at https://gist.github.com/trapexit/14ef971951b6c1096a7acec203754faa as a template. Move files instead of remove them. I have mine run ever hour.
September 15, 20241 yr 17 hours ago, trapexit said: I'm not aware of any software that does that generically. You have to script something yourself. I have some simple shell scripts that do size and time based movement linked from the mergerfs docs. I've been meaning to write something more involved but just haven't gotten around to it. You could look at https://gist.github.com/trapexit/14ef971951b6c1096a7acec203754faa as a template. Move files instead of remove them. I have mine run ever hour. Oof modifying that script is unfortunately beyond my abilities, is it safer to run an rsync command script like this: CACHE="/mnt/frigate_cache" ARCHIVE="/mnt/frigate_archive" N=7 #days find "${CACHE}" -type f -atime +${N} -printf '%P\n' | \ rsync --files-from=- -axqHAXWES --preallocate --remove-source-files "${CACHE}/" "${ARCHIVE}/" Again, really appreciate your help!
September 17, 20241 yr My scripts linked to in the docs will move files by time or by percentage full. It sounds to me you should use the latter.
January 13, 20251 yr may a general question, i readed through the docs of mergefs (honestly a fast read through ) is there a conflict if equal files (and same path) resides on 2 disks ? and if not, is there way to prio a disk on mount ? sample like Unraid implementation, cache > array setup, cache is always prio on read (also same file on same path) / write ? thanks ahead for a note if known.
January 13, 20251 yr 13 minutes ago, alturismo said: may a general question, i readed through the docs of mergefs (honestly a fast read through ) is there a conflict if equal files (and same path) resides on 2 disks ? and if not, is there way to prio a disk on mount ? sample like Unraid implementation, cache > array setup, cache is always prio on read (also same file on same path) / write ? thanks ahead for a note if known. That is all decided by the different policies and such: https://trapexit.github.io/mergerfs/config/functions_categories_and_policies/
January 13, 20251 yr 26 minutes ago, trapexit said: That is all decided by the different policies and such: https://trapexit.github.io/mergerfs/config/functions_categories_and_policies/ i ve been through there starting from git and from there to this part ... but look slike im not finding my answer but thanks for the link anyway looks like i have to make a test install to see what happens when ...
January 13, 20251 yr Policies decide how a file is selected. You asked how mergerfs selects a file, the "priority". Policies. The first found policy picks the first found. rand chooses random. Etc. Edited January 13, 20251 yr by trapexit
January 13, 20251 yr 21 minutes ago, trapexit said: Policies decide how a file is selected. You asked how mergerfs selects a file, the "priority". Policies. The first found policy picks the first found. rand chooses random. Etc. thanks, in my understanding its simple as the order are the disks are in line on mount command line, ff will then always pick "first found" ordered by ... mergerfs -o cache.files=off,dropcacheonclose=false,category.create=ff /mnt/hdd0:/mnt/hdd1 /mnt/mergefs/media so like this simple sample (as read only used mount), hdd0 would always be first if a duplicate would be on hdd0 && hdd1 while accessind /mnt/mergefs/media thanks @trapexit for the clarification, i readed to fast through the headline what is a branch in terminology of mergefs too fast (or lazy) here
January 13, 20251 yr No problem. What you show is `category.create=ff` which means on *create* it will choose the first branch found. You asked about overlapping existing files. That is controlled by the `open` and other function policies. The category wide setting being `category.search=ff`.
January 14, 20251 yr may asking if its ok to advice using mergerfs as Plex, Emby, Jelly ... Mediasource mount in terms when using my plugin cache mover in combination with "external" pools as cache drive ... really nice and easy to handle i have to admit and solves alot of trouble with cache drives outside an Unraid Share and so on ... so i would like to refer to mergerfs to build mounts in order to make use of the cache drives. if not wanted, just let me know please
January 14, 20251 yr Sorry but I'm not sure what you are asking. It would be more helpful if you could explain what is unclear in the documentation. https://trapexit.github.io/mergerfs/usage_patterns/#tiered-cache
January 14, 20251 yr Author 1 minute ago, trapexit said: Sorry but I'm not sure what you are asking. It would be more helpful if you could explain what is unclear in the documentation. https://trapexit.github.io/mergerfs/usage_patterns/#tiered-cache I think he just wants to make use of it as part of his plugins and asks it that's okay (to use the software).
January 14, 20251 yr I'm not a user of UnRAID or all that familiar with the setup being discussed so I really can't speak to what is appropriate or not.
January 14, 20251 yr 57 minutes ago, Rysz said: I think he just wants to make use of it as part of his plugins and asks it that's okay (to use the software). not even using it, just recommend it to create useable Mediaserver pools, thats all ... sample, my cache mover (Media file cache reader) is "limited" to an "Standard" Unraid cache > array Array setup, now there where questions if it would be possible to use "external" pool, uad, ... ssd disks therefore ... from my side (copy move files around) its easy, but the end result not ... as the Mediaserver needs to access them "easy", so i would recommend those ppl to use mergerfs and build a mount like /mnt/disks/ssd1:/mnt/cache:mnt/user0 /mnt/addond/whatever and use /mnt/addon/whatever as read point for the Mediaserver, so whenever i change files "underneath" and the file descriptor its still /mnt/addons/whatever and there for the Mediaserver ... otherwise i would have to do alot of rescanning as sample to get in Plex the right order to look for a file ... cause 1st in is the Top and so on, nevermind about that now, just wanted to ask if its ok to recommend this plugin to make use of mergerfs
January 14, 20251 yr Author 1 hour ago, alturismo said: not even using it, just recommend it to create useable Mediaserver pools, thats all ... sample, my cache mover (Media file cache reader) is "limited" to an "Standard" Unraid cache > array Array setup, now there where questions if it would be possible to use "external" pool, uad, ... ssd disks therefore ... from my side (copy move files around) its easy, but the end result not ... as the Mediaserver needs to access them "easy", so i would recommend those ppl to use mergerfs and build a mount like /mnt/disks/ssd1:/mnt/cache:mnt/user0 /mnt/addond/whatever and use /mnt/addon/whatever as read point for the Mediaserver, so whenever i change files "underneath" and the file descriptor its still /mnt/addons/whatever and there for the Mediaserver ... otherwise i would have to do alot of rescanning as sample to get in Plex the right order to look for a file ... cause 1st in is the Top and so on, nevermind about that now, just wanted to ask if its ok to recommend this plugin to make use of mergerfs I think it should be fine to recommend the software or even use it in your plugins, the license is very permissive in nature: https://github.com/trapexit/mergerfs/blob/latest-release/LICENSE 🙂
January 14, 20251 yr Sorry, I thought the question was "use" as in mechanically... safe to use together. Not legally. Yes. mergerfs is ISC licensed. Do whatever you wish within the license requirements. All that I ask is that if you are building tooling / services on top of mergerfs that you tell me about it so I can include it in the docs. https://trapexit.github.io/mergerfs/related_projects/
January 15, 20251 yr On 1/13/2025 at 8:38 PM, trapexit said: The category wide setting being `category.search=ff`. just as 1st impressions here, testing simple like (samsung USB disk and local nvme disk) mergerfs -o cache.files=off,dropcacheonclose=false,category.search=ff /mnt/cache/Media:/mnt/disks/samsung950pro/Media /mnt/addons/mergerfs mergerfs -o cache.files=off,dropcacheonclose=false,category.search=ff /mnt/disks/samsung950pro/Media:/mnt/cache/Media /mnt/addons/mergerfs now browsing /mnt/addons/mergerfs/... and open a file which resides in same path on both disks will always open the /mnt/cache/Media/... version, so category.search=ff has no real function in my terminology ... i thought the order of /mnt/disks/samsung950pro/Media:/mnt/cache/Media will decide then which will be opened ?
January 15, 20251 yr It does. How are you testing what is opened? As I say in my support docs... I need you to explain very very explicitly what you are doing for me to help. The ff policy is extremely simple and been around for a decade. It works.
January 15, 20251 yr 4 hours ago, trapexit said: It does. How are you testing what is opened? by opening the file via smb as sample so, my expectation would be that with this setup the file on /mnt/disks/Samsung... would be the "first found" ... when i now play "Mayfair Witches - S02E01 - Lasher WEBDL-1080p.mkv" root@AlsServerII:/boot/Download/Media# ls -la /mnt/disks/Samsung_SSD_950_PRO/Media/TVRIPS/Mayfair\ Witches/Season\ 02/ total 686468 drwxrwxrwx 1 root root 98 Jan 15 08:00 ./ drwxrwxrwx 1 root root 18 Jan 15 08:00 ../ -rw-rw-rw- 1 root root 702939305 Jan 6 23:29 Mayfair\ Witches\ -\ S02E01\ -\ Lasher\ WEBDL-1080p.mkv root@AlsServerII:/boot/Download/Media# ls -la /mnt/cache/Media/TVRIPS/Mayfair\ Witches/Season\ 02/ total 1328460 drwxrwxrwx 2 nobody users 127 Jan 13 23:30 ./ drwxrwxrwx 3 nobody users 23 Jan 6 04:31 ../ -rwxrwxrwx 1 nobody users 702939305 Jan 6 23:29 Mayfair\ Witches\ -\ S02E01\ -\ Lasher\ WEBDL-1080p.mkv* -rwxrwxrwx 1 nobody users 657399777 Jan 13 23:30 Mayfair\ Witches\ -\ S02E02\ -\ Ten\ of\ Swords\ WEBDL-1080p.mkv* root@AlsServerII:/boot/Download/Media# mergerfs -o cache.files=off,dropcacheonclose=false,category.search=ff /mnt/disks/Samsung_SSD_950_PRO/Media:/mnt/cache/Media /mnt/addons/mergerfs root@AlsServerII:/boot/Download/Media# ls -la /mnt/addons/mergerfs/TVRIPS/Mayfair\ Witches/Season\ 02/ total 1328460 drwxrwxrwx 1 root root 98 Jan 15 08:00 ./ drwxrwxrwx 1 root root 18 Jan 15 08:00 ../ -rw-rw-rw- 1 root root 702939305 Jan 6 23:29 Mayfair\ Witches\ -\ S02E01\ -\ Lasher\ WEBDL-1080p.mkv -rwxrwxrwx 1 nobody users 657399777 Jan 13 23:30 Mayfair\ Witches\ -\ S02E02\ -\ Ten\ of\ Swords\ WEBDL-1080p.mkv* as the chronologie is 1st /mnt/disks/Samsung... 2nd /mnt/cache/... with enabled search "ff" ... and i have to say, now it works like expected using a different disk ... ### i tested in the morning fast with the /boot drive ... and there it doesnt work, prolly as /boot is "special" ... so this wont work actually mergerfs -o cache.files=off,dropcacheonclose=false,category.search=ff /boot/Download/Media:/mnt/cache/Media /mnt/addons/mergerfs when i now open the file vie /mnt/addons/mergerfs/... , effectivly /mnt/cache/ is used and not /boot/Download/... ls -alrt /proc/[0-9]*/fd/* 2> /dev/null | grep -i ".mkv" lr-x------ 1 root root 64 Jan 15 18:10 /proc/1147747/fd/33 -> /mnt/addons/mergerfs/TVRIPS/Mayfair\ Witches/Season\ 02/Mayfair\ Witches\ -\ S02E01\ -\ Lasher\ WEBDL-1080p.mkv lr-x------ 1 root root 64 Jan 15 18:10 /proc/1146252/fd/11 -> /mnt/cache/Media/TVRIPS/Mayfair\ Witches/Season\ 02/Mayfair\ Witches\ -\ S02E01\ -\ Lasher\ WEBDL-1080p.mkv* so, nevermind, my fault as /boot is special, just made a quick & dirty test run today morning
January 15, 20251 yr Author 10 minutes ago, alturismo said: by opening the file via smb as sample so, my expectation would be that with this setup the file on /mnt/disks/Samsung... would be the "first found" ... when i now play "Mayfair Witches - S02E01 - Lasher WEBDL-1080p.mkv" root@AlsServerII:/boot/Download/Media# ls -la /mnt/disks/Samsung_SSD_950_PRO/Media/TVRIPS/Mayfair\ Witches/Season\ 02/ total 686468 drwxrwxrwx 1 root root 98 Jan 15 08:00 ./ drwxrwxrwx 1 root root 18 Jan 15 08:00 ../ -rw-rw-rw- 1 root root 702939305 Jan 6 23:29 Mayfair\ Witches\ -\ S02E01\ -\ Lasher\ WEBDL-1080p.mkv root@AlsServerII:/boot/Download/Media# ls -la /mnt/cache/Media/TVRIPS/Mayfair\ Witches/Season\ 02/ total 1328460 drwxrwxrwx 2 nobody users 127 Jan 13 23:30 ./ drwxrwxrwx 3 nobody users 23 Jan 6 04:31 ../ -rwxrwxrwx 1 nobody users 702939305 Jan 6 23:29 Mayfair\ Witches\ -\ S02E01\ -\ Lasher\ WEBDL-1080p.mkv* -rwxrwxrwx 1 nobody users 657399777 Jan 13 23:30 Mayfair\ Witches\ -\ S02E02\ -\ Ten\ of\ Swords\ WEBDL-1080p.mkv* root@AlsServerII:/boot/Download/Media# mergerfs -o cache.files=off,dropcacheonclose=false,category.search=ff /mnt/disks/Samsung_SSD_950_PRO/Media:/mnt/cache/Media /mnt/addons/mergerfs root@AlsServerII:/boot/Download/Media# ls -la /mnt/addons/mergerfs/TVRIPS/Mayfair\ Witches/Season\ 02/ total 1328460 drwxrwxrwx 1 root root 98 Jan 15 08:00 ./ drwxrwxrwx 1 root root 18 Jan 15 08:00 ../ -rw-rw-rw- 1 root root 702939305 Jan 6 23:29 Mayfair\ Witches\ -\ S02E01\ -\ Lasher\ WEBDL-1080p.mkv -rwxrwxrwx 1 nobody users 657399777 Jan 13 23:30 Mayfair\ Witches\ -\ S02E02\ -\ Ten\ of\ Swords\ WEBDL-1080p.mkv* as the chronologie is 1st /mnt/disks/Samsung... 2nd /mnt/cache/... with enabled search "ff" ... and i have to say, now it works like expected using a different disk ... ### i tested in the morning fast with the /boot drive ... and there it doesnt work, prolly as /boot is "special" ... so this wont work actually mergerfs -o cache.files=off,dropcacheonclose=false,category.search=ff /boot/Download/Media:/mnt/cache/Media /mnt/addons/mergerfs when i now open the file vie /mnt/addons/mergerfs/... , effectivly /mnt/cache/ is used and not /boot/Download/... ls -alrt /proc/[0-9]*/fd/* 2> /dev/null | grep -i ".mkv" lr-x------ 1 root root 64 Jan 15 18:10 /proc/1147747/fd/33 -> /mnt/addons/mergerfs/TVRIPS/Mayfair\ Witches/Season\ 02/Mayfair\ Witches\ -\ S02E01\ -\ Lasher\ WEBDL-1080p.mkv lr-x------ 1 root root 64 Jan 15 18:10 /proc/1146252/fd/11 -> /mnt/cache/Media/TVRIPS/Mayfair\ Witches/Season\ 02/Mayfair\ Witches\ -\ S02E01\ -\ Lasher\ WEBDL-1080p.mkv* so, nevermind, my fault as /boot is special, just made a quick & dirty test run today morning Just for sake of completeness (for @trapexit who - I think - doesn't use Unraid). /boot being "special" (although I'm not sure that mergerFS cares for this) would usually mean the following mountpoint: /dev/sda1 /boot vfat rw,noatime,nodiratime,fmask=0177,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,flush,errors=remount-ro 0 0 But, why do you even want to pool /boot in the first place? It seems odd to use the USB flash drive, which Unraid loads the OS into RAM from, for media consumption? If it's for intermediate, fast-access (cached) files why not a ramdisk or otherwise better suited storage like SSD/NVMe?
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.