DZMM

I just spotted that Document server 0.1.7 is out - has anyone successfully upgraded from 0.1.5?

I may be misunderstanding what you are trying to do, but wouldn't it be simplier to just use the mover or the custom mover script? i.e. set your download/mergerfs local share to be prefer cache and then mover will move files to the array when the cache fills up? If you need torrents to move off your cache faster than your mover settings or on a different schedule, you could do something like what I do. I use diskmv to move certain shares and folders off my cache when the cache gets to a certain capacity - that way I can keep say work files on the cache longer (almost forever) and 'archive' files where I don't need the fast access. Below you'll see I move .../downloads/complete (torrents that have completed but haven't been imported i.e. seeding) and ..../downloads/seeds (seeds that have been imported) to the array when my cache gets to a certain utilisation. ######################################## ####### Move Cache to Array ########## ######################################## # check if mover running if [ -f /var/run/mover.pid ]; then if ps h `cat /var/run/mover.pid` | grep mover ; then echo "$(date "+%d.%m.%Y %T") INFO: mover already running. Not moving files." else # move files ReqSpace=150000000 AvailSpace=$(df /mnt/cache | awk 'NR==2 { print $4 }') if [[ "$AvailSpace" -ge "$ReqSpace" ]];then echo "$(date "+%d.%m.%Y %T") INFO: Space ok - exiting" else echo "$(date "+%d.%m.%Y %T") INFO: Cache space low. Moving Files." # /usr/local/sbin/mdcmd set md_write_method 1 # echo "Turbo write mode now enabled" echo "$(date "+%d.%m.%Y %T") INFO: moving backup." bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/backup" cache disk2 echo "$(date "+%d.%m.%Y %T") INFO: moving local." bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/local/tdrive_vfs/downloads/complete" cache disk2 bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/local/tdrive_vfs/downloads/seeds" cache disk2 echo "$(date "+%d.%m.%Y %T") INFO: moving media." bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/media/other_media/books" cache disk2 bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/media/other_media/calibre" cache disk2 bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/media/other_media/magazines" cache disk2 bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/media/other_media/photos" cache disk2 bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/media/other_media/videos" cache disk2 echo "$(date "+%d.%m.%Y %T") INFO: moving software." bash /boot/config/plugins/user.scripts/scripts/unraid-diskmv/script -f -v "/mnt/user/software" cache disk2 # /usr/local/sbin/mdcmd set md_write_method 0 # echo "Turbo write mode now disabled" fi fi fi Edit: I disable turbo write as I don't have a parity drive anymore

Yes, but if you upload a file from the local folder the mount won't register the change. and a one-stop solution.

you have to install it at every boot. Once you've got python installed via nerdpark just add this to a script that runs at array start or similar: pip install plexapi

I had time to play at the weekend and there's unfortunately a few showstoppers - 1) it doesn't poll the local folder for changes and 2) it doesn't poll the remotes either. 2) might be because 1) isn't working, so if 1) can be fixed 2) might be fixed. @ncw from rclone is having a look, so we might have a fix/workaround. I hope he finds one as playback is definitely better than mergerfs I guess from removing the middleman.

@watchmeexplode5Are you having any joy with union? I just had another go and I've encountered 2 show-stopping issues: 1. Changes to the local folder aren't picked up in a timely manner 2. Changes to remotes, EVEN if via rclone e.g. via the mount aren't picked up Have you got a link for this please?

I'm curious how many people are now running this setup? unionfs or mergergs. The threads got quite big over nearly 2 years. It's a shame I can't add a poll to the first post.

I have a similar problem. Whilst trying to do the same upgrade I'm now stuck on the Maintenance page and can't access nextcloud:

make sure you are running the latest version of the unmount script at array start or manually now to fix your problem. There was an error in an earlier version.

Setup Service Accounts - covered on github

Could be - we're not sure. At the moment my system has been shutting down ok. Don't know - but if /mnt/user/local is working I'd just run with it! #1 I have no idea what's going on there as I see negliable performance difference and I think @watchmeexplode5 has said the same #2 I got my first API ban in over a year a few weeks ago, but I was doing some big scans in Plex as well as in Radarr, Sonarr etc all at the same time sounds like something is wrong with your radarr mappings not rclone which just uploads anything it sees in the local share - radarr controls where files are moved.

@TecneoI haven't had a chance to play with union this week - have you had any progress before I start?

ncw just said they should work - let me know how you get on. I'm going to focus on fixing the local polling tonight.

It doesn't appear in your /dockers page - it's a bit of an odd case. It has to be re-installed everytime unRAID starts as part of the script. It's why I want to remove it if possible - because unRAID can't support it natively e.g. via a plugin or a 'normal' docker, it's a bit confusing for unRAID users. Not mergerfs' problem, but just makes it a bit clunky. Having everything in rclone will be a lot cleaner.

I've just read the full post you linked to and saw Nick's comment - I'm going to try after work removing the -dir-cache-time and having poll set to maybe 1s. Need to read up a bit first/have a refresher on what both are doing

nope but it defaults to 1m so it wouldn't help - I think rclone looks for new changes based on --dir-cache-time - when I had this set to 720h as usual changes weren't getting picked up - I waited about 5 mins. With --dir-cache-time 1m they got picked up pretty quickly.

@watchmeexplode5 @Kaizac @testdasi and everyone else - I need help testing rclone union please which landed yesterday as part of rclone 1.5.2. https://forum.rclone.org/t/rclone-1-52-release/16718 I've created a test union ok and playback seems good - better than mergerfs, although only tried a few files. It'd be great if we can get this running as I think it'll be easier to support than mergerfs which has been brilliant but must be installed via a docker. We'll also be just using one app. I've encountered one problem so far in that -dir-cache-time applies to local folder changes as well, so a small number is needed to spot any changes made to /mnt/user/local. I've asked if there's a way to have a long cache for just the remotes. I've asked for advice here: https://forum.rclone.org/t/rclone-union-cache-time/16728/1 My settings so far: [tdrive_union] type = union upstreams = /mnt/user/local/tdrive_vfs tdrive_vfs: tdrive_uhd_vfs: tdrive_t_adults_vfs: gdrive_media_vfs: action_policy = all create_policy = ff search_policy = ff rclone mount --allow-other --buffer-size 256M --dir-cache-time 1m --drive-chunk-size 512M --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off --vfs-cache-mode writes tdrive_union: /mnt/user/mount_mergerfs/union_test

@axeman #1 if you delete locally a file that rclone has synced,then rather than delete it on the remote it is moved to the backupdir for your chosen number of days #2 it should show your files. Are you using krusader? If so, restart it as it has problems. Or try ssh or Windows explorer

Or look in your mergerfs location. Or look in /bin

yes the upload script doesn't run if the mount is not running.

and the files you are upload aren't appearing in the rclone mount? no remote files should be going there - it's only used for the script checker files, so I've no idea how you managed that. Maybe when you were trying to setup you did something odd?? My advice is to unmount everything, delete /appdata/other/rclone, delete any residual files and folders out of /mnt/user/mount_rclone and /mnt/user/mount_mergerfs and start with a fresh run. Lots of people have problems initially when they don't set it up right. I think you know what you are doing now, so a clean start is probably best.

what settings do you have for your mount and upload scripts?

That's weird as the version 0.9.2 on my machine had it but github had a weird 0.9.1 version. Anyway thanks for spotting and I've fixed - other people with this problem please update unmount script to 0.9.2 #!/bin/bash ####################### ### Cleanup Script #### ####################### #### Version 0.9.2 #### ####################### echo "$(date "+%d.%m.%Y %T") INFO: *** Starting rclone_cleanup script ***" ####### Cleanup Tracking Files ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Removing Tracking Files ***" find /mnt/user/appdata/other/rclone/remotes -name dockers_started* -delete find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete find /mnt/user/appdata/other/rclone/remotes -name upload_running* -delete echo "$(date "+%d.%m.%Y %T") INFO: ***Finished Cleanup! ***" exit

You haven't setup correctly the cleanup script properly to delete this or something else went wrong while you were trying to fix everything. Look in /mnt/user/appdata/other/rclone/remotes/whatever_your_upload_remote_is_called and delete whatever file is in there. After you've deleted, run the upload script again and it should recreate the file once you've started and delete it when it's finished. Double-check to make sure. If you end up rebooting or something while an upload is in progress, setting the cleanup/unmount script at array start will make sure all the control files are deleted.

Yes. You haven't setup the unmount script properly to remove all the verification files. Best way to run imo is at array start. Delete that file manually and the upload will start.