34 minutes ago, mestep said:

 

I don't mean to sound stupid or anything. If I put stuff in the union folder instead of the upload folder, it will still get uploaded to the cloud, right? Right now I have sabnzbd set to download to the upload folder, can/should I have it set to download to union?

No it will not. In your union you define which folder is Read Only (RO) and which one is Read Write (RW). So if you followed the tutorial you will have your local folder as your RW folder and your remote is RO. When you then move files to the union folder it will be written to your local folder. So all your dockers like Sab you can point to your union folder as through the union it knows it has to move the files to the rclone_upload folder.

 

In this setup files only get moved through using a script. If you want to direct write to your mount, you will have to do that through the mount_rclone folder. Through which you are directly accessing your cloud folders.

 

Makes sense?

Alright, I think I finally understand everything now. Thank you for all your help!

 

It just took me too long to understand the simple concept of:

 

mount_rclone = cloud files

rclone_upload = local files

mount_unionfs = combines mount_rclone and rclone_upload

 

One final thing. I saw someone post this earlier, but never got a definitive answer. The cleanup script isn't working:

 

rm: cannot remove '/mnt/user/mount_rclone/google_vfs/4k Movies/FOLDER': Is a directory
rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/.unionfs/4k_ Movies/FOLDER_HIDDEN~': Is a directory

 

Not sure what is going wrong with it.  It deletes the files but leaves the directories.

 

Edited February 14, 20197 yr by mestep

On 2/14/2019 at 3:13 PM, mestep said:

Alright, I think I finally understand everything now. Thank you for all your help!

 

It just took me too long to understand the simple concept of:

 

mount_rclone = cloud files

rclone_upload = local files

mount_unionfs = combines mount_rclone and rclone_upload

 

One final thing. I saw someone post this earlier, but never got a definitive answer. The cleanup script isn't working:

 

rm: cannot remove '/mnt/user/mount_rclone/google_vfs/4k Movies/FOLDER': Is a directory
rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/.unionfs/4k_ Movies/FOLDER_HIDDEN~': Is a directory

 

Not sure what is going wrong with it.  It deletes the files but leaves the directories.

 

Not sure what's going wrong.  Does your cleanup script look like this?  (i use a teamdrive)

 

################### Clean-up UnionFS Folder  #########################

echo "$(date "+%d.%m.%Y %T") INFO: starting unionfs cleanup."

find /mnt/user/mount_unionfs/google_vfs/.unionfs -name '*_HIDDEN~' | while read line; do
oldPath=${line#/mnt/user/mount_unionfs/google_vfs/.unionfs}
newPath=/mnt/user/mount_rclone/tdrive_vfs${oldPath%_HIDDEN~}
rm "$newPath"
rm "$line"
done

find "/mnt/user/mount_unionfs/google_vfs/.unionfs" -mindepth 1 -type d -empty -delete

rm /mnt/user/appdata/other/rclone/rclone_mount_cleanup

exit

 

I have also a question. Cleanups cript is also not working for me, u gave me a changed script, no change.

 

But i wonder if i really need it.

 

All mountpoints where i write data to are RW mounted, so a DEL should delete it on all points (local and remote), correct?
 

Quote

 

unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/rclone_upload/google_vfs=RO:/mnt/user/Archiv=RW:/mnt/user/mount_rclone/google_vfs=RO /mnt/user/mount_unionfs/google_vfs

 

 

I only add data to /mnt/user/Archiv

 

and upload it later via upload script.

Edited February 15, 20197 yr by nuhll

25 minutes ago, nuhll said:

I have also a question. Cleanups cript is also not working for me, u gave me a changed script, no change.

 

But i wonder if i really need it.

 

All mountpoints where i write data to are RW mounted, so a DEL should delete it on all points (local and remote), correct?
 

 

I only add data to /mnt/user/Archiv

 

and upload it later via upload script.

depends what you are uploading.  If you've uploaded a movie and then replace it with a better copy, this will be hidden and 'deleted' from the unionfs folder, but not from the cloud folder.  So, if you ever mount the cloud folder elsewhere or differently, suddenly all the files you thought you'd deleted would reappear.

4 hours ago, DZMM said:

Not sure what's going wrong.  Does your cleanup script look like this?  (i use a teamdrive)

 

Mine is basically the same. I am not using teamdrive:

 

 

#!/bin/bash
touch /mnt/user/appdata/other/rclone/rclone_mount_cleanup
################### Clean-up UnionFS Folder  #########################
echo "$(date "+%d.%m.%Y %T") INFO: starting unionfs cleanup."
find /mnt/user/mount_unionfs/google_vfs/.unionfs -name '*_HIDDEN~' | while read line; do
oldPath=${line#/mnt/user/mount_unionfs/google_vfs/.unionfs}
newPath=/mnt/user/mount_rclone/google_vfs${oldPath%_HIDDEN~}
rm "$newPath"
rm "$line"
done
find "/mnt/user/mount_unionfs/google_vfs/.unionfs" -mindepth 1 -type d -empty -delete
rm /mnt/user/appdata/other/rclone/rclone_mount_cleanup
exit

I delete the "test" folder in the root of mount_unionfs/google_vfs/ and run the script, this is the log:

15.02.2019 15:37:19 INFO: starting unionfs cleanup.
rm: cannot remove '/mnt/user/mount_rclone/google_vfs/test': Is a directory
rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/.unionfs/test_HIDDEN~': Is a directory
Script Finished Fri, 15 Feb 2019 15:37:19 -0500

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_cleanup/log.txt

 

I think it is a permissions issue, but I don't know enough about linux to figure it out.

Edited February 15, 20197 yr by mestep

22 hours ago, DZMM said:

depends what you are uploading.  If you've uploaded a movie and then replace it with a better copy, this will be hidden and 'deleted' from the unionfs folder, but not from the cloud folder.  So, if you ever mount the cloud folder elsewhere or differently, suddenly all the files you thought you'd deleted would reappear.

Okay, so this would be a problem, or lets say inconviniece then.

 

Which folder i need to check if it gets deleted correct? (like where they are hidden?`)

 

I get this error on running (but it might be an old file which was not uploaded by upload script)

16.02.2019 10:38:43 INFO: starting unionfs cleanup.
rm: cannot remove '/mnt/user/mount_rclone/google_vfs/Filme/some/thing.mkv': No such file or directory
rm: cannot remove '/mnt/user/rclone_upload/google_vfs/Filme/some/thing.mkv': No such file or directory
rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/.unionfs/Filme/some/thing_HIDDEN~': Permission denied

 

But i cant manually RM it because it says no file...

 

Edited February 16, 20197 yr by nuhll

37 minutes ago, nuhll said:

Okay, so this would be a problem, or lets say inconviniece then.

 

Which folder i need to check if it gets deleted correct? (like where they are hidden?`)

 

I get this error on running (but it might be an old file which was not uploaded by upload script)

16.02.2019 10:38:43 INFO: starting unionfs cleanup.
rm: cannot remove '/mnt/user/mount_rclone/google_vfs/Filme/1922 (2017)/1922 (2017).mkv': No such file or directory
rm: cannot remove '/mnt/user/rclone_upload/google_vfs/Filme/1922 (2017)/1922 (2017).mkv': No such file or directory
rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/.unionfs/Filme/1922 (2017)/1922 (2017).mkv_HIDDEN~': Permission denied

 

But i cant manually RM it because it says no file...

 

I'm not sure why people are getting this error - try running the fix permissions tool and check your radarr permissions settings - mine's at 777. 

To manually check if working browse the mount mount_rclone folder directly - test by moving a file from the unionfs folder to a non-mount folder, run the script, check the rclone mount to see if deleted and then move the file back to rclone_upload folder so it gets uploaded again

That might be my fault. I think this file is left from my first tryin to get the mount working and testing if it works.

 

SO i might have uploaded it to wrong dir or seomthing, but how to get rid?

 

I think if the script itself would not work, it would be more then just one movie (bc im dln like crazy)

Edited February 16, 20197 yr by nuhll

5 hours ago, nuhll said:

SO i might have uploaded it to wrong dir or seomthing, but how to get rid?

use a file manager like krusader or mc and do manually

I unfortunately had a powercut when the script was uploading to gdrive, now I keep getting this in the logs:

2019/02/16 23:22:28 NOTICE: Serving remote control on http://127.0.0.1:5572/
2019/02/16 23:22:29 NOTICE: pacer: Invalid error in fs.Pacer: %!t()
2019/02/16 23:22:29 NOTICE: pacer: Invalid error in fs.Pacer: %!t()

2019/02/16 23:22:29 NOTICE: pacer: Invalid error in fs.Pacer: %!t()

 

So I ran all the unmount/cleanup etc scripts, uninstalled the rclone beta (including the config files in the boot folder) and redid my gdrive and crypt remotes but it keeps on spamming this message still, I cannot find a single entry on google about it.

I get this message even after starting again - ie deleting the remotes and recreating them, then upon entering the 2nd command with your setup routine (ie "rclone copy mountcheck gdrive_media_vfs: -vv --no-traverse", i get the same error message - so it must be an rclone issue 

 

I did notice that it is showing port 5572 for the remote control above (at top of my post) and when I recreated a gdrive remote and did the token refresh it shows a different port ie 53682   http://127.0.0.1:53682/auth but not sure if that is relevant to the error or not..

 

 

 

Edited February 16, 20197 yr by shimlapinks

I really cant help u with this, sorry, but a UPC only costs <60€ and will prevent such things in feature. I would really get that asap.

ok, so I got everything configured .. however when I run the upload script I get the following: 

 

Script Starting Tue, 12 Mar 2019 22:35:24 -0400

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_gdrive_upload/log.txt

/tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 4: $'\357\273\277': command not found
/tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 26: $'\357\273\277': command not found
12.03.2019 22:35:24 INFO: rclone not installed - will try again later.
Script Finished Tue, 12 Mar 2019 22:35:24 -0400

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_gdrive_upload/log.txt

 

I also get this when running the cleanup script: 

 

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_gdrive_cleanup/log.txt

12.03.2019 22:44:55 INFO: starting unionfs cleanup.
find: `/mnt/user/mount_unionfs/google_vfs/.unionfs': No such file or directory
find: `/mnt/user/mount_unionfs/google_vfs/.unionfs': No such file or directory
Script Finished Tue, 12 Mar 2019 22:44:55 -0400

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_gdrive_cleanup/log.txt

 

both the mount and unmount script seems to run fine.

 

any advice ? 

 

thanks

 

edit ------------------------------

 

so if I just change the entries to remove one of the bracket set, (if [ -f "/mnt/user/appdata/other/rclone/rclone_upload" ]; then) script seems to run.

Edited March 13, 20197 yr by scloutier
issue

If u remove that bracket the script will run over and over again starting unlimited concurrent uploads. (bad idea)

 

/tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 4: $'\357\273\277': command not found
/tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 26: $'\357\273\277': command not found

 

Its due to wrong text encoding.

 

Be sure to copy it from github and place it in a e.g. vi shell, then it should work. (like "vi" your script file and replace it with the one from github)

OR use config editor plugin.

 

 

12.03.2019 22:35:24 INFO: rclone not installed - will try again later.
 

also this means that the mount didnt work correct.

Edited March 13, 20197 yr by nuhll

I tried to copy both from github in raw directly into putty, and also tried to paste it in the unraid config editor.  results are the same.  will see if I can download it directly from the shell then.

 

I'm also confused as the if man page only shows single bracket (https://ss64.com/bash/syntax-file-operators.html) 

 

their example: 

 

Check if a file exists:

if [ -f ~/some-file ]; then
 Echo the file exists 
fi

which also has no " " around the file path"

 

thanks

Edited March 13, 20197 yr by scloutier

I had th eexact same problem.

 

There is some sort of encoding error while copy pasting, i think i got it via terminal or config editor - not sure.

 

 

alright .. I used VI to paste the script.  same issue, running out of idea .. will try to wget the file directly from the shell and rename see if that works, however running out of ideas.

 

so what's weird is that part of it works .. 

 

/tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 4: $'\357\273\277': command not found
13.03.2019 15:00:01 INFO: rclone installed successfully - proceeding with upload.

 

the first if file check for the rclone_upload fails .. the second one that looks for mountcheck seems to work 

 

Edited March 13, 20197 yr by scloutier

As nuhll said

1 hour ago, scloutier said:

/tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 4: $'\357\273\277': command not found

Those escaped octals means you are getting invalid characters when you paste.

2 minutes ago, trurl said:

As nuhll said

Those escaped octals means you are getting invalid characters when you paste.

right .. but even with downloading the file from the raw github page directly in my unraid terminal and renaming it does the same.  no copy/paste involved .. 

so clearly this has something to do with utf-8 encoding.  digging to find a solution.

ok, so using cat -v filename actually shows where the issues are .. in the upload script, issue was inside two blank lines.

 

will go through the other script.  really weird issue.

 

 

edit

 

seems like going through and removing any broken portion and retyping worked.

Edited March 13, 20197 yr by scloutier

fine you got it!

Locally I've been using unionfs and rclone mount as stated in this guide. My cloud dedicated server is running plexdrive V4 and unionfs. So far this has been a great combination but I think i'd like to move onto the mergefs and rclone_vfs and off of plexdrive/unionfs due to some of the benefits.

Has anyone been running this setup?

I have currently nearly 10TB up and i couldnt think of any better way, what should be the benefit?

 

Only thing left for me is better internet. But i get 1gbits until end of year...  

Edited March 15, 20197 yr by nuhll

 

 

Server side renames and stability at scale. Otherwise whatever works.... I use rclone VFS and mergerfs combo on my other servers but not unraid, that is currently as per the guide in this thread.