DZMM

Logs from mount script and script options please

Sorry I don't really understand python and I think I fluked completing this step as I didn't really understand what I was doing! Hopefully someone else can help. Or, have you tried asking the autorclone author?

More control over how files are uploaded and when

Multiple mounts, one upload and one tidy-up script. @watchmeexplode5 did some testing and performance gets worse as you get closer to the 400k mark, so you'll need to do something like below soon: 1. My folder structure looks something like this: mount_mergerfs/tdrive_vfs/movies mount_mergerfs/tdrive_vfs/music mount_mergerfs/tdrive_vfs/uhd mount_mergerfs/tdrive_vfs/tv_adults mount_mergerfs/tdrive_vfs/tv_kids 2. I created separate tdrives / rclone mounts for some of the bigger folders e.g. mount_rclone/tdrive_vfs/movies mount_rclone/tdrive_vfs/music mount_rclone/tdrive_vfs/uhd mount_rclone/tdrive_vfs/adults_tv for each of those I created a mount script instance where I do NOT create a mergerfs mount 3. I mount each in turn and for the final main mount add the extra tdrive rclone mounts as extra mergerfs folders: ############################################################### ###################### mount tdrive ######################### ############################################################### # REQUIRED SETTINGS RcloneRemoteName="tdrive_vfs" RcloneMountShare="/mnt/user/mount_rclone" LocalFilesShare="/mnt/user/local" MergerfsMountShare="/mnt/user/mount_mergerfs" # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="/mnt/user/mount_rclone/music" LocalFilesShare3="/mnt/user/mount_rclone/uhd" LocalFilesShare4="/mnt/user/mount_rclone/adults_tv" 4. Run the single upload script - everything initially gets moved from /mnt/user/local/tdrive_vfs to the tdrive_vfs teamdrive 5. Overnight I run another script to move files from the folders that are in tdrive_vfs: to the correct teamdrive. You have to work out the encrypted folder names for this to work. Because rclone is moving the files, the mergerfs mount gets updated i.e. it looks to plex etc like they haven't moved #!/bin/bash rclone move tdrive:crypt/music_tdrive_encrypted_folder_name gdrive:crypt/music_tdrive_encrypted_folder_name \ --user-agent="transfer" \ -vv \ --buffer-size 512M \ --drive-chunk-size 512M \ --tpslimit 8 \ --checkers 8 \ --transfers 4 \ --order-by modtime,ascending \ --exclude *fuse_hidden* \ --exclude *_HIDDEN \ --exclude .recycle** \ --exclude .Recycle.Bin/** \ --exclude *.backup~* \ --exclude *.partial~* \ --drive-stop-on-upload-limit \ --delete-empty-src-dirs rclone move tdrive:crypt/tv_tdrive_encrypted_folder_name tdrive_t_adults:crypt/tv_tdrive_encrypted_folder_name \ --user-agent="transfer" \ -vv \ --buffer-size 512M \ --drive-chunk-size 512M \ --tpslimit 8 \ --checkers 8 \ --transfers 4 \ --order-by modtime,ascending \ --exclude *fuse_hidden* \ --exclude *_HIDDEN \ --exclude .recycle** \ --exclude .Recycle.Bin/** \ --exclude *.backup~* \ --exclude *.partial~* \ --drive-stop-on-upload-limit \ --delete-empty-src-dirs rclone move tdrive:crypt/uhd_tdrive_encrypted_folder_name tdrive_uhd:crypt/uhd_tdrive_encrypted_folder_name \ --user-agent="transfer" \ -vv \ --buffer-size 512M \ --drive-chunk-size 512M \ --tpslimit 8 \ --checkers 8 \ --transfers 4 \ --order-by modtime,ascending \ --exclude *fuse_hidden* \ --exclude *_HIDDEN \ --exclude .recycle** \ --exclude .Recycle.Bin/** \ --exclude *.backup~* \ --exclude *.partial~* \ --drive-stop-on-upload-limit \ --delete-empty-src-dirs exit

MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv"\} # comma separated list of folders to create within the mount I guess it fails if you only add one folder. just create folder manually in your mergerfs mount.

Probably not up to date. There are comments on the new scripts which make moving easy - just: 1. make sure you haven't got any rclone activity going on - stop old scripts, uploads and any dockers using the mount 2. setup new paths - but /mount_unionfs etc as your mergerfs mount paths etc if that's what you have setup now. Be careful to put your existing paths in 3. choose other script options 4. run scripts and if all ok, launch dockers

that's what's supposed to happen!

What folders are in /mnt/disks/Plex ? I've got mergerfs mounts that include UD and they work fine.

Set /mnt/disks/Plex as your local location in the mount and upload scripts

The new mergerfs based scripts do this and much more...

My current 360/180 ISP sent me an email saying my upload was going at 100% for a few months and they wondered if I'd been hacked. I thanked them for their concern and said I was ok and knew what the traffic was. I use bwlimits so my upload runs now at about 80Mbps average over course of a day as my big upload days are over. My previous 1000/1000 ISP didn't say anything despite my upload running at about 60-70% for over a year. I keep a copy of the most recent vm and appdata backups locally. If there was ....an accidental deletion I probably would just write off all the content as 1) I can't build a 0.5PB array and 2) it'd probably be easier to ....replace the content I want than spend weeks/months downloading it from gdrive. I did look into backing up my tdrives on another user's setup (he currently backs his up to mine), but I stopped as the actual process of downloading off his tdrive would face the same problems.

It's the logical next step. I've ditched my parity drive (I backup to gdrive using duplicati), sold all but 2 of my HDDs that store seeds, pending uploads and my work/personal documents. I don't really use the mass storage functionality anymore other than pooling the 2 HDDs - kinda impossible and would be mega expensive to store 0.5PB+ of content..... My unRAID server main purpose is to power VMs (3xW10 VMs for me and the kids + pfsense VM) and Dockers (plex server with remote storage, Home Assistant, unifi, minecraft server, nextcloud, radarr etc).

If you want some partitioning, you could do /mergerfs --> /mnt/user/gdrive_mergerfs and then within your dockers use the following paths: /mergerfs/downloads for /mnt/user/gdrive_mergerfs/downloads/ /mergerfs/media/tv for /mnt/user/gdrive_mergerfs/tv/ The trick is your torrent, radarr, sonarr etc dockers have to be moving files around the mergerfs mount i.e. /mergerfs. If you map: /mergerfs --> /mnt/user/gdrive_mergerfs /downloads --> /mnt/user/gdrive_mergerfs/downloads /downloads_local (adding for another example) --> /mnt/user/gdrive_local/downloads when you ask the docker to hardlink a file from /downloads or /downloads_local to /mergerfs it won't work. It has to be from /mergerfs/downloads to /mergerfs/media/tv - within /mergerfs. To be clear, when I say I do /user --> /mnt/user it's because it just makes my life easier when I'm setting up all dockers to talk to each other (I'm lazy) - within my media dockers I still only use paths within /user/mount_mergerfs e.g. /user/mount_mergerfs/downloads and /user/mount_mergerfs/tv_shows

You're messing up your mappings somewhere as hardlinks work

If want full hardlink support map all docker paths to /user --> /mnt/user then within the docker set all locations to a sub path of /user/mount-mergerfs. Then behind the scenes unraid and rclone will behave as normal and manage where the files really reside

I just spotted that Document server 0.1.7 is out - has anyone successfully upgraded from 0.1.5?

I may be misunderstanding what you are trying to do, but wouldn't it be simplier to just use the mover or the custom mover script? i.e. set your download/mergerfs local share to be prefer cache and then mover will move files to the array when the cache fills up? If you need torrents to move off your cache faster than your mover settings or on a different schedule, you could do something like what I do. I use diskmv to move certain shares and folders off my cache when the cache gets to a certain capacity - that way I can keep say work files on the cache longer (almost forever) and 'archive' files where I don't need the fast access. Below you'll see I move .../downloads/complete (torrents that have completed but haven't been imported i.e. seeding) and ..../downloads/seeds (seeds that have been imported) to the array when my cache gets to a certain utilisation. ######################################## ####### Move Cache to Array ########## ######################################## # check if mover running if [ -f /var/run/mover.pid ]; then if ps h `cat /var/run/mover.pid` | grep mover ; then echo "$(date "+%d.%m.%Y %T") INFO: mover already running. Not moving files." else # move files ReqSpace=150000000 AvailSpace=$(df /mnt/cache | awk 'NR==2 { print $4 }') if [[ "$AvailSpace" -ge "$ReqSpace" ]];then echo "$(date "+%d.%m.%Y %T") INFO: Space ok - exiting" else echo "$(date "+%d.%m.%Y %T") INFO: Cache space low. Moving Files." # /usr/local/sbin/mdcmd set md_write_method 1 # echo "Turbo write mode now enabled" echo "$(date "+%d.%m.%Y %T") INFO: moving backup." bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/backup" cache disk2 echo "$(date "+%d.%m.%Y %T") INFO: moving local." bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/local/tdrive_vfs/downloads/complete" cache disk2 bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/local/tdrive_vfs/downloads/seeds" cache disk2 echo "$(date "+%d.%m.%Y %T") INFO: moving media." bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/media/other_media/books" cache disk2 bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/media/other_media/calibre" cache disk2 bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/media/other_media/magazines" cache disk2 bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/media/other_media/photos" cache disk2 bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/media/other_media/videos" cache disk2 echo "$(date "+%d.%m.%Y %T") INFO: moving software." bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/software" cache disk2 # /usr/local/sbin/mdcmd set md_write_method 0 # echo "Turbo write mode now disabled" fi fi fi Edit: I disable turbo write as I don't have a parity drive anymore

Yes, but if you upload a file from the local folder the mount won't register the change. and a one-stop solution.

you have to install it at every boot. Once you've got python installed via nerdpark just add this to a script that runs at array start or similar: pip install plexapi

I had time to play at the weekend and there's unfortunately a few showstoppers - 1) it doesn't poll the local folder for changes and 2) it doesn't poll the remotes either. 2) might be because 1) isn't working, so if 1) can be fixed 2) might be fixed. @ncw from rclone is having a look, so we might have a fix/workaround. I hope he finds one as playback is definitely better than mergerfs I guess from removing the middleman.

@watchmeexplode5Are you having any joy with union? I just had another go and I've encountered 2 show-stopping issues: 1. Changes to the local folder aren't picked up in a timely manner 2. Changes to remotes, EVEN if via rclone e.g. via the mount aren't picked up Have you got a link for this please?

I'm curious how many people are now running this setup? unionfs or mergergs. The threads got quite big over nearly 2 years. It's a shame I can't add a poll to the first post.

I have a similar problem. Whilst trying to do the same upgrade I'm now stuck on the Maintenance page and can't access nextcloud:

make sure you are running the latest version of the unmount script at array start or manually now to fix your problem. There was an error in an earlier version.

Setup Service Accounts - covered on github