Kaizac Posted February 14, 2019 Share Posted February 14, 2019 34 minutes ago, mestep said: I don't mean to sound stupid or anything. If I put stuff in the union folder instead of the upload folder, it will still get uploaded to the cloud, right? Right now I have sabnzbd set to download to the upload folder, can/should I have it set to download to union? No it will not. In your union you define which folder is Read Only (RO) and which one is Read Write (RW). So if you followed the tutorial you will have your local folder as your RW folder and your remote is RO. When you then move files to the union folder it will be written to your local folder. So all your dockers like Sab you can point to your union folder as through the union it knows it has to move the files to the rclone_upload folder. In this setup files only get moved through using a script. If you want to direct write to your mount, you will have to do that through the mount_rclone folder. Through which you are directly accessing your cloud folders. Makes sense? Quote Link to comment
mestep Posted February 14, 2019 Share Posted February 14, 2019 (edited) Alright, I think I finally understand everything now. Thank you for all your help! It just took me too long to understand the simple concept of: mount_rclone = cloud files rclone_upload = local files mount_unionfs = combines mount_rclone and rclone_upload One final thing. I saw someone post this earlier, but never got a definitive answer. The cleanup script isn't working: rm: cannot remove '/mnt/user/mount_rclone/google_vfs/4k Movies/FOLDER': Is a directory rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/.unionfs/4k_ Movies/FOLDER_HIDDEN~': Is a directory Not sure what is going wrong with it. It deletes the files but leaves the directories. Edited February 14, 2019 by mestep Quote Link to comment
DZMM Posted February 15, 2019 Author Share Posted February 15, 2019 On 2/14/2019 at 3:13 PM, mestep said: Alright, I think I finally understand everything now. Thank you for all your help! It just took me too long to understand the simple concept of: mount_rclone = cloud files rclone_upload = local files mount_unionfs = combines mount_rclone and rclone_upload One final thing. I saw someone post this earlier, but never got a definitive answer. The cleanup script isn't working: rm: cannot remove '/mnt/user/mount_rclone/google_vfs/4k Movies/FOLDER': Is a directory rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/.unionfs/4k_ Movies/FOLDER_HIDDEN~': Is a directory Not sure what is going wrong with it. It deletes the files but leaves the directories. Not sure what's going wrong. Does your cleanup script look like this? (i use a teamdrive) ################### Clean-up UnionFS Folder ######################### echo "$(date "+%d.%m.%Y %T") INFO: starting unionfs cleanup." find /mnt/user/mount_unionfs/google_vfs/.unionfs -name '*_HIDDEN~' | while read line; do oldPath=${line#/mnt/user/mount_unionfs/google_vfs/.unionfs} newPath=/mnt/user/mount_rclone/tdrive_vfs${oldPath%_HIDDEN~} rm "$newPath" rm "$line" done find "/mnt/user/mount_unionfs/google_vfs/.unionfs" -mindepth 1 -type d -empty -delete rm /mnt/user/appdata/other/rclone/rclone_mount_cleanup exit Quote Link to comment
NewDisplayName Posted February 15, 2019 Share Posted February 15, 2019 (edited) I have also a question. Cleanups cript is also not working for me, u gave me a changed script, no change. But i wonder if i really need it. All mountpoints where i write data to are RW mounted, so a DEL should delete it on all points (local and remote), correct? Quote unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/rclone_upload/google_vfs=RO:/mnt/user/Archiv=RW:/mnt/user/mount_rclone/google_vfs=RO /mnt/user/mount_unionfs/google_vfs I only add data to /mnt/user/Archiv and upload it later via upload script. Edited February 15, 2019 by nuhll Quote Link to comment
DZMM Posted February 15, 2019 Author Share Posted February 15, 2019 25 minutes ago, nuhll said: I have also a question. Cleanups cript is also not working for me, u gave me a changed script, no change. But i wonder if i really need it. All mountpoints where i write data to are RW mounted, so a DEL should delete it on all points (local and remote), correct? I only add data to /mnt/user/Archiv and upload it later via upload script. depends what you are uploading. If you've uploaded a movie and then replace it with a better copy, this will be hidden and 'deleted' from the unionfs folder, but not from the cloud folder. So, if you ever mount the cloud folder elsewhere or differently, suddenly all the files you thought you'd deleted would reappear. Quote Link to comment
mestep Posted February 15, 2019 Share Posted February 15, 2019 (edited) 4 hours ago, DZMM said: Not sure what's going wrong. Does your cleanup script look like this? (i use a teamdrive) Mine is basically the same. I am not using teamdrive: #!/bin/bash touch /mnt/user/appdata/other/rclone/rclone_mount_cleanup ################### Clean-up UnionFS Folder ######################### echo "$(date "+%d.%m.%Y %T") INFO: starting unionfs cleanup." find /mnt/user/mount_unionfs/google_vfs/.unionfs -name '*_HIDDEN~' | while read line; do oldPath=${line#/mnt/user/mount_unionfs/google_vfs/.unionfs} newPath=/mnt/user/mount_rclone/google_vfs${oldPath%_HIDDEN~} rm "$newPath" rm "$line" done find "/mnt/user/mount_unionfs/google_vfs/.unionfs" -mindepth 1 -type d -empty -delete rm /mnt/user/appdata/other/rclone/rclone_mount_cleanup exit I delete the "test" folder in the root of mount_unionfs/google_vfs/ and run the script, this is the log: 15.02.2019 15:37:19 INFO: starting unionfs cleanup. rm: cannot remove '/mnt/user/mount_rclone/google_vfs/test': Is a directory rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/.unionfs/test_HIDDEN~': Is a directory Script Finished Fri, 15 Feb 2019 15:37:19 -0500 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_cleanup/log.txt I think it is a permissions issue, but I don't know enough about linux to figure it out. Edited February 15, 2019 by mestep Quote Link to comment
NewDisplayName Posted February 16, 2019 Share Posted February 16, 2019 (edited) 22 hours ago, DZMM said: depends what you are uploading. If you've uploaded a movie and then replace it with a better copy, this will be hidden and 'deleted' from the unionfs folder, but not from the cloud folder. So, if you ever mount the cloud folder elsewhere or differently, suddenly all the files you thought you'd deleted would reappear. Okay, so this would be a problem, or lets say inconviniece then. Which folder i need to check if it gets deleted correct? (like where they are hidden?`) I get this error on running (but it might be an old file which was not uploaded by upload script) 16.02.2019 10:38:43 INFO: starting unionfs cleanup. rm: cannot remove '/mnt/user/mount_rclone/google_vfs/Filme/some/thing.mkv': No such file or directory rm: cannot remove '/mnt/user/rclone_upload/google_vfs/Filme/some/thing.mkv': No such file or directory rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/.unionfs/Filme/some/thing_HIDDEN~': Permission denied But i cant manually RM it because it says no file... Edited February 16, 2019 by nuhll Quote Link to comment
DZMM Posted February 16, 2019 Author Share Posted February 16, 2019 37 minutes ago, nuhll said: Okay, so this would be a problem, or lets say inconviniece then. Which folder i need to check if it gets deleted correct? (like where they are hidden?`) I get this error on running (but it might be an old file which was not uploaded by upload script) 16.02.2019 10:38:43 INFO: starting unionfs cleanup. rm: cannot remove '/mnt/user/mount_rclone/google_vfs/Filme/1922 (2017)/1922 (2017).mkv': No such file or directory rm: cannot remove '/mnt/user/rclone_upload/google_vfs/Filme/1922 (2017)/1922 (2017).mkv': No such file or directory rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/.unionfs/Filme/1922 (2017)/1922 (2017).mkv_HIDDEN~': Permission denied But i cant manually RM it because it says no file... I'm not sure why people are getting this error - try running the fix permissions tool and check your radarr permissions settings - mine's at 777. To manually check if working browse the mount mount_rclone folder directly - test by moving a file from the unionfs folder to a non-mount folder, run the script, check the rclone mount to see if deleted and then move the file back to rclone_upload folder so it gets uploaded again Quote Link to comment
NewDisplayName Posted February 16, 2019 Share Posted February 16, 2019 (edited) That might be my fault. I think this file is left from my first tryin to get the mount working and testing if it works. SO i might have uploaded it to wrong dir or seomthing, but how to get rid? I think if the script itself would not work, it would be more then just one movie (bc im dln like crazy) Edited February 16, 2019 by nuhll Quote Link to comment
DZMM Posted February 16, 2019 Author Share Posted February 16, 2019 5 hours ago, nuhll said: SO i might have uploaded it to wrong dir or seomthing, but how to get rid? use a file manager like krusader or mc and do manually Quote Link to comment
shimlapinks Posted February 16, 2019 Share Posted February 16, 2019 (edited) I unfortunately had a powercut when the script was uploading to gdrive, now I keep getting this in the logs: 2019/02/16 23:22:28 NOTICE: Serving remote control on http://127.0.0.1:5572/ 2019/02/16 23:22:29 NOTICE: pacer: Invalid error in fs.Pacer: %!t() 2019/02/16 23:22:29 NOTICE: pacer: Invalid error in fs.Pacer: %!t() 2019/02/16 23:22:29 NOTICE: pacer: Invalid error in fs.Pacer: %!t() So I ran all the unmount/cleanup etc scripts, uninstalled the rclone beta (including the config files in the boot folder) and redid my gdrive and crypt remotes but it keeps on spamming this message still, I cannot find a single entry on google about it. I get this message even after starting again - ie deleting the remotes and recreating them, then upon entering the 2nd command with your setup routine (ie "rclone copy mountcheck gdrive_media_vfs: -vv --no-traverse", i get the same error message - so it must be an rclone issue I did notice that it is showing port 5572 for the remote control above (at top of my post) and when I recreated a gdrive remote and did the token refresh it shows a different port ie 53682 http://127.0.0.1:53682/auth but not sure if that is relevant to the error or not.. Edited February 16, 2019 by shimlapinks Quote Link to comment
NewDisplayName Posted February 17, 2019 Share Posted February 17, 2019 I really cant help u with this, sorry, but a UPC only costs <60€ and will prevent such things in feature. I would really get that asap. Quote Link to comment
scloutier Posted March 13, 2019 Share Posted March 13, 2019 (edited) ok, so I got everything configured .. however when I run the upload script I get the following: Script Starting Tue, 12 Mar 2019 22:35:24 -0400 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_gdrive_upload/log.txt /tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 4: $'\357\273\277': command not found /tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 26: $'\357\273\277': command not found 12.03.2019 22:35:24 INFO: rclone not installed - will try again later. Script Finished Tue, 12 Mar 2019 22:35:24 -0400 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_gdrive_upload/log.txt I also get this when running the cleanup script: Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_gdrive_cleanup/log.txt 12.03.2019 22:44:55 INFO: starting unionfs cleanup. find: `/mnt/user/mount_unionfs/google_vfs/.unionfs': No such file or directory find: `/mnt/user/mount_unionfs/google_vfs/.unionfs': No such file or directory Script Finished Tue, 12 Mar 2019 22:44:55 -0400 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_gdrive_cleanup/log.txt both the mount and unmount script seems to run fine. any advice ? thanks edit ------------------------------ so if I just change the entries to remove one of the bracket set, (if [ -f "/mnt/user/appdata/other/rclone/rclone_upload" ]; then) script seems to run. Edited March 13, 2019 by scloutier issue Quote Link to comment
NewDisplayName Posted March 13, 2019 Share Posted March 13, 2019 (edited) If u remove that bracket the script will run over and over again starting unlimited concurrent uploads. (bad idea) /tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 4: $'\357\273\277': command not found /tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 26: $'\357\273\277': command not found Its due to wrong text encoding. Be sure to copy it from github and place it in a e.g. vi shell, then it should work. (like "vi" your script file and replace it with the one from github) OR use config editor plugin. 12.03.2019 22:35:24 INFO: rclone not installed - will try again later. also this means that the mount didnt work correct. Edited March 13, 2019 by nuhll Quote Link to comment
scloutier Posted March 13, 2019 Share Posted March 13, 2019 (edited) I tried to copy both from github in raw directly into putty, and also tried to paste it in the unraid config editor. results are the same. will see if I can download it directly from the shell then. I'm also confused as the if man page only shows single bracket (https://ss64.com/bash/syntax-file-operators.html) their example: Check if a file exists: if [ -f ~/some-file ]; then Echo the file exists fi which also has no " " around the file path" thanks Edited March 13, 2019 by scloutier Quote Link to comment
NewDisplayName Posted March 13, 2019 Share Posted March 13, 2019 I had th eexact same problem. There is some sort of encoding error while copy pasting, i think i got it via terminal or config editor - not sure. Quote Link to comment
scloutier Posted March 13, 2019 Share Posted March 13, 2019 (edited) alright .. I used VI to paste the script. same issue, running out of idea .. will try to wget the file directly from the shell and rename see if that works, however running out of ideas. so what's weird is that part of it works .. /tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 4: $'\357\273\277': command not found 13.03.2019 15:00:01 INFO: rclone installed successfully - proceeding with upload. the first if file check for the rclone_upload fails .. the second one that looks for mountcheck seems to work Edited March 13, 2019 by scloutier Quote Link to comment
trurl Posted March 13, 2019 Share Posted March 13, 2019 As nuhll said 1 hour ago, scloutier said: /tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 4: $'\357\273\277': command not found Those escaped octals means you are getting invalid characters when you paste. Quote Link to comment
scloutier Posted March 13, 2019 Share Posted March 13, 2019 2 minutes ago, trurl said: As nuhll said Those escaped octals means you are getting invalid characters when you paste. right .. but even with downloading the file from the raw github page directly in my unraid terminal and renaming it does the same. no copy/paste involved .. Quote Link to comment
scloutier Posted March 13, 2019 Share Posted March 13, 2019 so clearly this has something to do with utf-8 encoding. digging to find a solution. Quote Link to comment
scloutier Posted March 13, 2019 Share Posted March 13, 2019 (edited) ok, so using cat -v filename actually shows where the issues are .. in the upload script, issue was inside two blank lines. will go through the other script. really weird issue. edit seems like going through and removing any broken portion and retyping worked. Edited March 13, 2019 by scloutier 1 Quote Link to comment
NewDisplayName Posted March 13, 2019 Share Posted March 13, 2019 fine you got it! Quote Link to comment
slimshizn Posted March 15, 2019 Share Posted March 15, 2019 Locally I've been using unionfs and rclone mount as stated in this guide. My cloud dedicated server is running plexdrive V4 and unionfs. So far this has been a great combination but I think i'd like to move onto the mergefs and rclone_vfs and off of plexdrive/unionfs due to some of the benefits. Has anyone been running this setup? Quote Link to comment
NewDisplayName Posted March 15, 2019 Share Posted March 15, 2019 (edited) I have currently nearly 10TB up and i couldnt think of any better way, what should be the benefit? Only thing left for me is better internet. But i get 1gbits until end of year... Edited March 15, 2019 by nuhll Quote Link to comment
Spladge Posted March 15, 2019 Share Posted March 15, 2019 Server side renames and stability at scale. Otherwise whatever works.... I use rclone VFS and mergerfs combo on my other servers but not unraid, that is currently as per the guide in this thread. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.