blizz Posted May 14, 2020 Share Posted May 14, 2020 (edited) 20 minutes ago, DZMM said: Yes it does if done correctly. If you're making changes to the mount outside of rclone that will be the source of your problem. Or, your settings aren't right - best advice I can give is use my scripts. Sorry I think I may not have explained my situation properly. I've not been using mergerfs. And without mergerfs, the partial scan doesn't work correct? And now I'm intending to use mergerfs to solve this problem. However, I don't have write access to the drive. It's a shared drive and I only have read access. If the owner makes changes directly to the gdrive, like adding folders etc, will the changes be reflected in the mergerfs folder? So basically I'm not going to be having a local folder that uploads to the drive. I just want to use mergerfs for the updates if that makes sense. I wish to use your scripts except for the local folder part where it uploads to the drive. Will it still work or have I understood this wrongly? Edited May 14, 2020 by blizz Quote Link to comment
Bjur Posted May 14, 2020 Share Posted May 14, 2020 (edited) On my latest reboot I get this from the second mountscript: And the mergerfs directory is empty. Script Starting May 14, 2020 11:30.01 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount_sh/log.txt 14.05.2020 11:30:01 INFO: Creating local folders. 14.05.2020 11:30:01 INFO: *** Starting mount of remote googleSh_crypt 14.05.2020 11:30:01 INFO: Checking if this script is already running. 14.05.2020 11:30:01 INFO: Script not running - proceeding. 14.05.2020 11:30:01 INFO: Mount not running. Will now mount googleSh_crypt remote. 14.05.2020 11:30:01 INFO: Recreating mountcheck file for googleSh_crypt remote. 2020/05/14 11:30:01 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "googleSh_crypt:" "-vv" "--no-traverse"] 2020/05/14 11:30:01 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf" 2020/05/14 11:30:03 DEBUG : mountcheck: Modification times differ by -15m0.133405247s: 2020-05-14 11:30:01.832405247 +0200 CEST, 2020-05-14 09:15:01.699 +0000 UTC 2020/05/14 11:30:05 INFO : mountcheck: Copied (replaced existing) 2020/05/14 11:30:05 INFO : Transferred: 32 / 32 Bytes, 100%, 19 Bytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 1.6s 2020/05/14 11:30:05 DEBUG : 11 go routines active 2020/05/14 11:30:05 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "googleSh_crypt:" "-vv" "--no-traverse"] 14.05.2020 11:30:05 INFO: *** Creating mount for remote googleSh_crypt 14.05.2020 11:30:05 INFO: sleeping for 5 seconds 14.05.2020 11:30:10 INFO: continuing... 14.05.2020 11:30:10 INFO: Successful mount of googleSh_crypt mount. 14.05.2020 11:30:10 INFO: Mergerfs already installed, proceeding to create mergerfs mount 14.05.2020 11:30:10 INFO: Creating googleSh_crypt mergerfs mount. 14.05.2020 11:30:10 INFO: Checking if googleSh_crypt mergerfs mount created. 14.05.2020 11:30:10 INFO: Check successful, googleSh_crypt mergerfs mount created. 14.05.2020 11:30:10 INFO: Starting dockers. sonarr 14.05.2020 11:30:10 INFO: Script complete Script Finished May 14, 2020 11:30.10 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount_sh/log.txt 14.05.2020 11:34:57 INFO: Creating local folders. 14.05.2020 11:34:57 INFO: *** Starting mount of remote googleSh_crypt 14.05.2020 11:34:57 INFO: Checking if this script is already running. 14.05.2020 11:34:57 INFO: Script not running - proceeding. 14.05.2020 11:34:57 INFO: Mount not running. Will now mount googleSh_crypt remote. 14.05.2020 11:34:57 INFO: Recreating mountcheck file for googleSh_crypt remote. 2020/05/14 11:34:57 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "googleSh_crypt:" "-vv" "--no-traverse"] 2020/05/14 11:34:57 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf" 2020/05/14 11:34:58 DEBUG : mountcheck: Modification times differ by -4m55.337592306s: 2020-05-14 11:34:57.169592306 +0200 CEST, 2020-05-14 09:30:01.832 +0000 UTC 2020/05/14 11:35:00 INFO : mountcheck: Copied (replaced existing) 2020/05/14 11:35:00 INFO : Transferred: 32 / 32 Bytes, 100%, 20 Bytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 1.5s 2020/05/14 11:35:00 DEBUG : 11 go routines active 2020/05/14 11:35:00 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "googleSh_crypt:" "-vv" "--no-traverse"] 14.05.2020 11:35:00 INFO: *** Creating mount for remote googleSh_crypt 14.05.2020 11:35:00 INFO: sleeping for 5 seconds 2020/05/14 11:35:01 Fatal error: Directory is not empty: /mnt/user/mount_rclone/googleSh_crypt If you want to mount it anyway use: --allow-non-empty option 14.05.2020 11:35:05 INFO: continuing... 14.05.2020 11:35:05 INFO: Successful mount of googleSh_crypt mount. 14.05.2020 11:35:05 INFO: Check successful, googleSh_crypt mergerfs mount in place. 14.05.2020 11:35:05 INFO: dockers already started. 14.05.2020 11:35:05 INFO: Script complete Script Finished May 14, 2020 11:35.05 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount_sh/log.txt Edited May 14, 2020 by Bjur Quote Link to comment
Bjur Posted May 14, 2020 Share Posted May 14, 2020 it places a mountcheck file in /mnt/user/rclone_mount/googleSh_crypt and if I delete it and run the script again the same error occurs. Quote Link to comment
DZMM Posted May 14, 2020 Author Share Posted May 14, 2020 4 hours ago, blizz said: Sorry I think I may not have explained my situation properly. I've not been using mergerfs. And without mergerfs, the partial scan doesn't work correct? And now I'm intending to use mergerfs to solve this problem. However, I don't have write access to the drive. It's a shared drive and I only have read access. If the owner makes changes directly to the gdrive, like adding folders etc, will the changes be reflected in the mergerfs folder? So basically I'm not going to be having a local folder that uploads to the drive. I just want to use mergerfs for the updates if that makes sense. I wish to use your scripts except for the local folder part where it uploads to the drive. Will it still work or have I understood this wrongly? Mergerfs just shows what rclone knows about. If changes are made direct to gdrive, rclone won't know about these until it polls the remote to see if there are new files - that's why your scans are 'failing'. If you're mounting a remote where changes are made outside of the rclone i.e. your mate is loading files and not rclone, then you need your mate to install plex autoscan or something similar, so that it tells plex/rclone to re-scan the changed directory. Quote Link to comment
DZMM Posted May 14, 2020 Author Share Posted May 14, 2020 3 hours ago, Bjur said: 2020/05/14 11:35:01 Fatal error: Directory is not empty: /mnt/user/mount_rclone/googleSh_crypt If you want to mount it anyway use: --allow-non-empty option there's something in /mnt/user/mount_rclone/googleSh_crypt prior to the mount that you need to check and delete if appropriate. Quote Link to comment
blizz Posted May 14, 2020 Share Posted May 14, 2020 6 minutes ago, DZMM said: Mergerfs just shows what rclone knows about. If changes are made direct to gdrive, rclone won't know about these until it polls the remote to see if there are new files - that's why your scans are 'failing'. If you're mounting a remote where changes are made outside of the rclone i.e. your mate is loading files and not rclone, then you need your mate to install plex autoscan or something similar, so that it tells plex/rclone to re-scan the changed directory. Hey mate! Thanks for the reply again. Rclone does work properly ; my poll interval is set to 5m so the changes show up in the rclone folder. It was just that Plex was unable to see the changes unless I did a manual scan. And I just set up mergerfs and it works! The changes get reflected in the mergerfs folder, even when it is added directly to the remote by the owner. The problem now is still Plex is not seeing it with partial scan, even though it is in mergerfs. I gather from your last reply that I might need to install the plex autoscan plugin to solve this. My only doubt is why even though the file appears in mergerfs, which is "local", it is not appearing in Plex when it should. By right, for local files you don't need plex autoscan, correct? Quote Link to comment
Bjur Posted May 14, 2020 Share Posted May 14, 2020 16 minutes ago, DZMM said: there's something in /mnt/user/mount_rclone/googleSh_crypt prior to the mount that you need to check and delete if appropriate. Thanks. If using Krusader nothing is there even when enabling show hidden files. When browsing in shell it showed mountcheck file. Even when deleting it wouldn't mount correctly and show folders/mountcheck in mergerfs. I think I got it working now, but will report back if not. Thanks. Quote Link to comment
Bjur Posted May 14, 2020 Share Posted May 14, 2020 @watchmeexplode5 Would it make more sense to me to make a Download folder to /local/download instead of /local/REMOTE/downloads. Limit the local folder share to only 1 disk and the completed folders as: /local/googleFi_crypt/movies or /local/googleSh_crypt/series It should then be easy to move the completed files quickly and still have a separate download folder before the remotes. Does that makes sense? Quote Link to comment
watchmeexplode5 Posted May 14, 2020 Share Posted May 14, 2020 (edited) @Bjur Shouldn't be any issue to have a single download and that location is fine. In your case with the two mounts, I think it makes sense to have a single download like you listed. The scripts (if you leave it default) will auto create /local/REMOTE/downloads folder but feel free to ignore it or edit it to something like this: googleFi_crypt: MountFolders=\{"movies"\} # comma separated list of folders to create within the mount googleSh_crypt: MountFolders=\{"series"\} # comma separated list of folders to create within the mount Edited May 14, 2020 by watchmeexplode5 Quote Link to comment
DZMM Posted May 14, 2020 Author Share Posted May 14, 2020 4 hours ago, blizz said: My only doubt is why even though the file appears in mergerfs, which is "local", it is not appearing in Plex when it should. By right, for local files you don't need plex autoscan, correct? Not sure how Plex scans for changes in a directory. My setup has sonarr/radarr etc telling plex something has changed which works perfectly. Quote Link to comment
DZMM Posted May 14, 2020 Author Share Posted May 14, 2020 3 hours ago, Bjur said: Thanks. If using Krusader nothing is there even when enabling show hidden files. When browsing in shell it showed mountcheck file. Even when deleting it wouldn't mount correctly and show folders/mountcheck in mergerfs. I think I got it working now, but will report back if not. Thanks. Krusader has caused problems for other people - it needs restarting after a new mount. I get confused with your filepaths, but if the mount isn't working then the mountcheck file shouldn't be there and needs deleting...the mountcheck file should be on the remote and it's presence is used to check the mount has completed successfully. Quote Link to comment
1activegeek Posted May 15, 2020 Share Posted May 15, 2020 (edited) hey @DZMM finally got around to testing out your script, but I seem to be hitting a wall. I've gotten the script to run successfully, and it seems at first glance to be working. Problem is, as soon as I try to touch any of the data - aka mount anything, move anything, copy anything, etc - the mouths disconnect and are going away. Any tips on where I can start to look to debug? I know not a lot of detail, hoping to start with debug so I can see what I can find first before bogging you down with lots of info to troubleshoot. I'll add an edit - I thought it might be relative to the whole mapping scenario you highlighted, but another issue I've noticed before I could even get to that - after a short while the mounts actually disappear. So whether I do anything or not, they seem to disappear after a short time as well. I'm really scratching my head. Edited May 15, 2020 by 1activegeek Quote Link to comment
DZMM Posted May 15, 2020 Author Share Posted May 15, 2020 @1activegeek post your rclone config and your mount script please. When you say the mount goes missing, where do you mean? In dockers, file explorer, shell etc? Quote Link to comment
Bjur Posted May 15, 2020 Share Posted May 15, 2020 @watchmeexplode5: Thanks. So I think I will create a manual download folder in /mnt/local/Downloads /mnt/local/Downloads/temp /mnt/local/Downloads/finished and then let Filebot move them to: /mnt/local/googleFi_crypt /mnt/local/googleSh_crypt What will happen when you move files to local folder while an upload is running. Will that be fine or will it mess the upload up? Quote Link to comment
watchmeexplode5 Posted May 15, 2020 Share Posted May 15, 2020 @Bjur Files will only be upload once the min age time (set in upload script) has been hit. Default is 15 minutes. So no issue moving files into the upload location while upload is running. Quote Link to comment
Bjur Posted May 15, 2020 Share Posted May 15, 2020 (edited) Okay have you tried it. Just read earlier today on Rclone it was not a good idea, so I just want to make sure before I try it out. I don't won't to mess with the upload I'm running since it will take a long time to finish. Thanks for the help. Edited May 15, 2020 by Bjur Quote Link to comment
DZMM Posted May 15, 2020 Author Share Posted May 15, 2020 (edited) 15 minutes ago, Bjur said: Okay have you tried it. Just read earlier today on Rclone it was not a good idea, so I just want to make sure before I try it out. I don't won't to mess with the upload I'm running since it will take a long time to finish. Thanks for the help. I think you're confusing uploading files directly via the mount (which we don't) which is bad if you don't know what you are doing , and via rclone upload (which we do) which is 100% safe. If it didn't work, this whole 69 page thread would be a waste of time! Edited May 15, 2020 by DZMM Quote Link to comment
Bjur Posted May 15, 2020 Share Posted May 15, 2020 No I have my files in /mnt/local/remote. I'm running the upload script. While it is running I want to add new files to my /mnt/local/remote while it is running Quote Link to comment
DZMM Posted May 15, 2020 Author Share Posted May 15, 2020 44 minutes ago, Bjur said: Will that work. Yes, otherwise this whole thread would be pointless. Quote Link to comment
1activegeek Posted May 15, 2020 Share Posted May 15, 2020 16 hours ago, DZMM said: @1activegeek post your rclone config and your mount script please. When you say the mount goes missing, where do you mean? In dockers, file explorer, shell etc? [1a_mediaz] type = drive scope = drive team_drive = ~~REDACTED~~ service_account_file = /boot/config/plugins/rclone-beta/sa/1.json server_side_across_configs = TRUE Sorry for the lack of details. So I mean as in after I mount it, I can LS and see the contents. A few moments will pass, then if I try and LS on the directory - the folders are gone and hence the "disappear". rclone_mount.sh Quote Link to comment
DZMM Posted May 15, 2020 Author Share Posted May 15, 2020 40 minutes ago, 1activegeek said: [1a_mediaz] type = drive scope = drive team_drive = ~~REDACTED~~ service_account_file = /boot/config/plugins/rclone-beta/sa/1.json server_side_across_configs = TRUE Sorry for the lack of details. So I mean as in after I mount it, I can LS and see the contents. A few moments will pass, then if I try and LS on the directory - the folders are gone and hence the "disappear". rclone_mount.sh 9.52 kB · 1 download 1a_mediaz is where your files are, unencrypted? Quote Link to comment
1activegeek Posted May 16, 2020 Share Posted May 16, 2020 1 hour ago, DZMM said: 1a_mediaz is where your files are, unencrypted? Yes, correct. That's just one mount, I have some others I'll work on and/or work on adjusting to point at crypt volume - but for now yes that's the example I'm working with to make sure its working as expected first. Did I miss something about this being reliant on crypt mount? I figured it should work either way as long as the destination is my intended destination. Quote Link to comment
1activegeek Posted May 16, 2020 Share Posted May 16, 2020 Ok, so after some consolidated and focused testing - I think I'm ok. I think I had a mixup of signals, way too much chaos at work, and extreme late nights trying to play with tech and not remembering what I did. The short of my issue I believe - and to help in case others do similar - is that I was switching between rclone variants, and testing the new shortcuts function, and shifting between running the script in the UR UI vs running in background. The initial tests I think had an issue because of the rclone vs rclone-beta because I was testing the new shortcuts. Then I tried shifting to the beta, and tried simply running the script - a few times I left the window open, but after time I eventually closed the window. This is what I believe led to the "miraculous disappearance" scenario I was left with. I had originally been doing it with the background, but when I hit an issue initially, it was annoying finding the process to kill so I just ran it via the GUI And left it there. That switch is where I believe I messed it up. Right now, I've got it up and running using rclone-beta with shortcuts enabled (that 1a_mediaz is a pool from multiple TD locations under one mount). Next I'll venture into the territory with the merger_fs stuff, but at the moment that's not a requirement for me. I just want to be sure I've got a solid mount/unmount for my core rclone location. I'll adjust the script for my need to add other mounts beyond the one. This is awesome, thank you for the work and effort. Sorry for the firedrill over nothing!! Quote Link to comment
bugster Posted May 16, 2020 Share Posted May 16, 2020 Question: Every time the scripts is run I get the following error on multiple Movies/TV Quote Duplicate directory found in destination - ignoring I'm not even trying to upload the one listed but somehow they show on the log. Why is it doing this? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.