[Plugin] rclone

[Waseh]

23 minutes ago, unraidyn said:

If you want to know something really weird, if I modify the plugin to delete the plugin directory as the first action in the installation script:

 

if [ -d /boot/config/plugins/rclone ]; then
  rm -rf /boot/config/plugins/rclone
fi;

 

It works fine. There is something about that specific '/boot/config/plugins/rclone/install/rclone' file that my install really doesn't like.

In the logs you posted it already starts by throwing errors trying to copy the existing rclone binary to the sbin directory.
Then you get errors trying to just download the untouched zip file from the rclone webpage to your flash drive and then errors trying to create a config file.
If anything it's the rclone folder it doesn't like. I suggest reading up on IO errors and see if you can find a solution that way.

[karldonteljames]

Hello!
I've managed to install rRlone and finally figured out how to connect it to my OneDrive, GoogleDrive as well as GooglePhotos, but now I'm stuck and would very much appreciate some assistance.

 

I would like to upload a local folder, including subfolders, and files to my GooglePhotos
I would also like to sync a local folder to both my onedrive and googledrive remote folders, but i have literally no idea where to start with this.

I don't need file encryption etc. just not sure on the commands or where i put the commands to the achieve the above. Thanks in advance.

[jluerken]

Hey, 

 

I have used rClone WebUI Container to access pCloud via Webdav. 

This is working and I can see the online files.

Now I want to map this into a local unraid folder to obtain a sync between the local share and my pCloud drive.

When I go under "Mounts" and try to create a new mount I am always getting an error 500.

 

Any help appreciated.

[Mamaun]

On 2/17/2020 at 12:58 AM, Stupifier said:

Example command....specifically, pay attention to the uid, gid, and default permissions flags

rclone mount \
   --allow-other \
   --dir-cache-time 96h \
   --fast-list \
   --drive-chunk-size 512M \
   --buffer-size 256M \
   --vfs-read-chunk-size 128M \
   --vfs-read-chunk-size-limit off \
   --uid 99 --gid 100 \
   --default-permissions \
   --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36" \
   rclone_remote: /mnt/disks/rclone_remote

 

 

Hi,
I'm new to unraid (and with little Linux experience)

 

I'm using this exact script and all my folders gets owned by root with drwxr-xr-x permissions. I can delete files with crusader but i can't delete folders.

I want to mount my google drive using rclone (works)
Sync a specific folder to a share(works but permissions issue)
Copy all new files from this folder to another folder used by qbittorrent (works but permissions issue) 

 

I use this scripts:

MOUNT:

mkdir -p /mnt/addons/rclone_google_drive

rclone mount \
   --allow-other \
   --dir-cache-time 96h \
   --fast-list \
   --drive-chunk-size 512M \
   --buffer-size 256M \
   --vfs-read-chunk-size 128M \
   --vfs-read-chunk-size-limit off \
   --uid 99 --gid 100 \
   --default-permissions \
   --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36" \
   RcloneGoogleDrive: /mnt/addons/rclone_google_drive

COPY TO SHARE:

rclone copy RcloneGoogleDrive:unRAIDSync /mnt/user/rclone_unraidsync_drive -vv 

 

COPY TO MONITORED FOLDER:

watch_dir=/mnt/user/rclone_unraidsync_drive/AutoDownloadTorrent/
copy_dir=/mnt/user/qbittorrent_monitored_folder

 

inotifywait -m -r -e create,move,modify,delete $watch_dir |
    while read path action file; do
        echo "El archivo '$file' ha sido $action en la carpeta '$path'"
        rsync -avz -r $watch_dir $copy_dir
    done

 

 

I've tried with umask 000 too. Same results.

 

Edited March 17, 2023 by Mamaun

[HHUBS]

I use rclone with Google Drive to sync my files but I transferred to OneDrive for Business (my account for work). It's the same config and settings but when it start to sync (on schedule), my CPU usage is 100% and RAM is about 94-96%. I've been monitoring this for the past days and I turned the sync schedule off and the server is stable for 4 straight days. Why is it eating my RAM and CPU?

 

The sync script. It run's every 01:00.

#!/bin/bash
#Google Drive
#rclone sync /mnt/user/Media/Videos /mnt/disks/"Google Drive"/Videos --exclude *.ini
#rclone sync /mnt/user/Pictures/Album /mnt/disks/"Google Drive"/Pictures --exclude *.ini
#rclone sync /mnt/user/Backup /mnt/disks/"Google Drive"/Backup --exclude *.ini
#rclone sync /mnt/user/Documents /mnt/disks/"Google Drive"/Documents --exclude *.ini

#OneDrive
rclone sync /mnt/user/Media/Videos /mnt/disks/OneDrive/Backup/Videos --exclude *.ini
rclone sync /mnt/user/Pictures/Album /mnt/disks/OneDrive/Backup/Pictures --exclude *.ini
rclone sync /mnt/user/Backup /mnt/disks/OneDrive/Backup/Backup --exclude *.ini
rclone sync /mnt/user/Documents /mnt/disks/OneDrive/Backup/Documents --exclude *.ini

 

I will turn on the sync again today and update you if my CPU and RAM will get 100% later at 01:00

 

 

UPDATE: @1:30

 

 

Now I can't access the dashboard anymore. How can I prevent this?

 

Edited March 17, 2023 by HHUBS
UPDATE

[bally12345]

Hi guys, probably a really dumb question but I have never used rclone before. Followed spaceinvaders video on youtube from 6 years ago and setup onedrive, mounted and can view using SMB. 

However if I understand this correctly, this is just direct mapping the drive. Similar to the onedrive app... What I am actually after is onedrive back up to my array. Is this possible...? Assuming I would need to create another separate script to copy /mnt/disks/onedrive to /mnt/user/shares/onedrive weekly?

Edited March 27, 2023 by bally12345

[mackid1993]

Is there a way to force the plugin to run an older version of rclone. My hosting provider is running an ancient version of ProFTPd and supposedly it will be upgraded next month. It's causing issues with the latest version of rclone and I was hoping to find a way to revert to v1.61.1. Is there any way I can downgrade the plugin? Thanks.

 

Edit: just incase anyone is wondering I figured this out myself. It's only a temporary solution until my hosting provider updates ProFTPD.

 

I went to /usr/local/emhttp/plugins/rclone and made a copy of updaterclone.sh.

 

I then edited my copy and changed rcloneurl on lines 4 and 6 to the download for 1.61.1. I then ran the script and it downgraded and my FTPS connection works again. Once my provider updates their servers I'll reinstall the plugin to clean this up but for now I'm back online!

Edited March 31, 2023 by mackid1993

[photonix]

On 1/25/2023 at 1:13 PM, Waseh said:

In the logs you posted it already starts by throwing errors trying to copy the existing rclone binary to the sbin directory.
Then you get errors trying to just download the untouched zip file from the rclone webpage to your flash drive and then errors trying to create a config file.
If anything it's the rclone folder it doesn't like. I suggest reading up on IO errors and see if you can find a solution that way.

Just want to add my findings regarding IO issues during boot that may help someone. I am seeing the same issues on my new hardware (Intel 12600K running on MSI PRO Z690-A DDR4). I ruled out the USB stick right away as it continues to work fine on other hardware.

I've narrowed down the IO issues occurring during the copy of rclone -> /sbin/rcloneorig . The file is 80% copied and the USB drive is reset. This appears to be a timing issue with something going on in the background at the time the copy takes place in my setup. I added a 10 second wait just before the copy and that resolved the boot issue:
 

if [ -f /boot/config/plugins/rclone/install/rclone ]; then 
  sleep 10
  cp /boot/config/plugins/rclone/install/rclone /usr/sbin/rcloneorig.new
  mv /usr/sbin/rcloneorig.new /usr/sbin/rcloneorig
  chown root:root /usr/sbin/rcloneorig
  chmod 755 /usr/sbin/rcloneorig
  version=`rcloneorig --version | head -n 1`
fi;

 

Tough to diagnose this edge case since it could be a factor of hardware, number of plugins loading, etc. Hope the helps others that might run into this. 

[Waseh]

35 minutes ago, photonix said:

Just want to add my findings regarding IO issues during boot that may help someone. I am seeing the same issues on my new hardware (Intel 12600K running on MSI PRO Z690-A DDR4). I ruled out the USB stick right away as it continues to work fine on other hardware.

I've narrowed down the IO issues occurring during the copy of rclone -> /sbin/rcloneorig . The file is 80% copied and the USB drive is reset. This appears to be a timing issue with something going on in the background at the time the copy takes place in my setup. I added a 10 second wait just before the copy and that resolved the boot issue:
 

if [ -f /boot/config/plugins/rclone/install/rclone ]; then 
  sleep 10
  cp /boot/config/plugins/rclone/install/rclone /usr/sbin/rcloneorig.new
  mv /usr/sbin/rcloneorig.new /usr/sbin/rcloneorig
  chown root:root /usr/sbin/rcloneorig
  chmod 755 /usr/sbin/rcloneorig
  version=`rcloneorig --version | head -n 1`
fi;

 

Tough to diagnose this edge case since it could be a factor of hardware, number of plugins loading, etc. Hope the helps others that might run into this. 

If anybody else who has this problem can confirm it as a fix I'm happy to implement it for everybody 

[olco]

I would like my rclone-remote to be mounted when the server starts. Only after mounting the remote I want EmbyServer to start Docker.

I have tried to do this via the User Scripts plugin.
However, there are problems with this. First, the directory "/mnt/gsuite/media" is deleted on every startup, so I added "mkdir /mnt/gsuite" and "mkdir /mnt/gsuite/media" to the User Script.

As trigger for the script I defined "@reboot". Nevertheless, nothing happens when the server is started. At the first manual start of the script the rclone-remote is mounted. Docker is not started until the second manual execution of the script.

Does anyone know what could be the reason for this?

[mackid1993]

I was suddenly having my rclone FTP mount chew through memory and crash my server. I use rclone combined with Sonarr to copy content from my seedbox to my server. I noticed that all of the sudden my seedbox speeds have increased (before I would get 200-300 mbps but now I'm getting closer to 500-700 mbps) but at the same time the increased bandwidth may have broken something in my config or mount script because my server would suddenly run out of memory after a few minutes of Sonarr importing. What fixed it for me was to add 

 --cache-chunk-no-memory 

to my mount script. I wanted to post this as I saw some others were having issues where rclone was using all of their memory and although the use cases may be different this option helped me and didn't impact transfer speeds in my scenario. Based on my limited understanding it seemed that rclone was buffering data into memory faster than it could release it.

[mackid1993]

Looking into this a little more I created a folder in /mnt/cache/appdata called rclonecache and set the parameter in my mount script

--cache-dir=/mnt/cache/appdata/rclonecache

I found that when using --vfs-cache-mode writes which is necessary in some cases particularly with cloud storage like Dropbox my server was running out of memory. This was happening with FTP as well. I think the problem was that because Unraid runs in memory and that this is a plugin, the default cache location is essentially on ramdisk. Directing rclone's cache to my cache drive rather than system memory fixed my lockups. During a large transfer I saw the cache folder I created in appdata grow to 15 GB in size and then later on reduce to about 50 MB. A 15 GB cache is enough to use up the free RAM in my server (32 GB total) and cause the WebUI to lockup.

Edited May 12, 2023 by mackid1993

[leawhig]

Hi everyone,

I use rclone to mount a Google Drive on the host machine and pass it through to a VM using. 

Can anyone please guide me on how to achieve this? thx

[xander87003]

Hello,

I am having permission issues since upgrading from 6.9 to 6.10. When syncing from my SFTP remote, the files created by rclone copy command end up with r-x and I am unable to move them until I run docker safe permissions after every sync. When checking ls -l, the files show as root root. I tried making a new unraid usb with only rclone installed with unraid version 6.12 and the issue still persists that the newly synced files from my remote are being created as root and not as usernobody. I have tried searching for days to solve the issue but everything I find is about using rclone and mounting remote drives which I am not doing, I am using it simply as a one way sync from a remote to my local cache drive --> array. I am a novice with linux so I am hitting a wall with what I can do to fix the issue other than just letting my sync run and running docker safe permissions everyday, which is tiring. Can anyone please help me fix why rclone is syncing new files on my array as root and just have them be regular unraid permissions?

 

Edit: looks like this bug is caused by using rclone with userscripts and is a known issue from userscript author. Recommended fix is to add permission change at the end of your script to fix the issue. 

Edited June 28, 2023 by xander87003

[qy2009]

@Waseh I Have a strange issue, and I don't know why. 

I installed the plugin, setup everything, and it can mount my cloud drives using user scripts during the array boot up. 

 

but when I actually run another script to get rclone sync jobs, it said "command not found"

 

I SSH into my server, first time I just run "rclone", it seems fine. but when I actually run the full command again to do sync jobs, it said"command not found" again. 

 

why this happened? 

/tmp/user.scripts/tmpScripts/rclone_sync/script: line 22: rclone: command not found
/tmp/user.scripts/tmpScripts/rclone_sync/script: line 29: chmod: command not found

 

interesting that, chmond command also not found? 

 

thanks so much!

[Alvin Leong]

i have problem putting the whole content of my rclone config in the edit config text box. 2 google drive and 2 onedrive. then all these 4 remotes has crypt remotes. so basically 8 total remotes. i would like to know is it the plugin limitation or is there a way for my to copy my rclone config file from windows to this plugin

[gswarriorfan]

On 7/10/2023 at 9:18 AM, Alvin Leong said:

i have problem putting the whole content of my rclone config in the edit config text box. 2 google drive and 2 onedrive. then all these 4 remotes has crypt remotes. so basically 8 total remotes. i would like to know is it the plugin limitation or is there a way for my to copy my rclone config file from windows to this plugin

open your rclone config in a text editor, copy the entireity, and then paste into the rclone config in the UNRAID settings section.

[Alvin Leong]

10 hours ago, gswarriorfan said:

open your rclone config in a text editor, copy the entireity, and then paste into the rclone config in the UNRAID settings section.

correct. this is exactly what i did. even so, when the whole content from text file in my windows is copied. few remote drives settings all been cut off. it consistently only able to take up to certain total line in the UNRAID rclone config field

[Qwent]

I recently updated my Unraid machine, and found it stuck when booting - no access to web UI (connection refused), has access to default SSH port (not the custom port), nginx not running, etc. After some force-shutdown and rebooting to investigate, I found out it's this plugin. And i think there's some potential optimization here.

 

We all know that Unraid OS sits on the USB drive, it contains system files, plugins, configuration files and more. When the OS boots up from the USB drive, Unraid load the filesystem into the RAM, during this process it re-installs all the plugins - from USB drive to the RAM filesystem. And when re-installing the Rclone plugin, it will trigger an update check back to rclone.org, if there's a new release then it will try to download the latest ZIP file and replace the rclone binary executable.

 

Well, the thing is, in some region of the world, you cannot access rclone.org without using a proxy (or at least at a usable speed). So the curl command just keep downloading at a speed of around 100B/s, preventing the OS from continue to boot up. Once I figured out what's going on, I manually downloaded the latest ZIP file and put it into the USB drive (via SSH), and used `kill -SIGINT` to interrupt the curl process, so the rclone plugin updated the binary executable (from the ZIP file I provided) and the OS continued to boot up fully operational.

 

So how did I install the rclone in the first place, if I cannot access rclone.org without a network proxy? I installed ProxyEditor plugin, and set a system-wide proxy for my Unraid machine. So after a reboot (after the ProxyEditor plugin was re-installed and loaded its configuration file during OS boot), all system components is using the network proxy I configured (by injecting HTTP_PROXY environment variables, I guess). At this time I installed Rclone plugin via CA, so there's no problem.

 

When the OS is booting up, however, it would be doing all the plugin re-installation without HTTP_PROXY and other environment variables (since the ProxyEditor plugin has not yet been re-installed), so the network request to rclone.org is really slow, effectively blocking the OS from further booting up.

 

The workaround is pretty easy - just like what I did - when it's downloading the file, manually uploading the latest ZIP file to the USB drive and interrupt the curl process.

Edited July 20, 2023 by Qwent
Add link to the ProxyEditor plugin post, since it's a common name

[steve1977]

I downloaded the plugin (not docker container). It shows installed as plugin.

 

I have an rclone command that I would like to run once per week. How to best do? Just set up a userscript with the rclone command? Or do I need to do anything with the rclone script directly?

[Botafoguense1965]

Hello all...another newbie here...so...I installed Rclone and Rclone browser...in Rclone browser I set up a Dropbox connection, went fine, I can browse my Dropbox through it...but when I go to Unraid terminal and type Rclone config it says "no remotes found"...and it appears in the Rclone Browser...what´s wrong here? And plus, when I try to download a file from my Dropbox through Rclone Browser, I set the destination file to a mapped folder in one of my shares, click ok but nothing happens...lost here!

[Maginos]

On 7/29/2023 at 5:03 AM, steve1977 said:

I downloaded the plugin (not docker container). It shows installed as plugin.

 

I have an rclone command that I would like to run once per week. How to best do? Just set up a userscript with the rclone command? Or do I need to do anything with the rclone script directly?

Yes, you can use the user scripts plugin for that. Just paste your rclone command into a new script, set the suitable cronjob for the execution of the script and your good to go.

I highly recommend you to control the result of the rclone job.

The second option would be to paste your rclone command in the crontab of the root user of unraid itself.

 

 

19 hours ago, Botafoguense1965 said:

Hello all...another newbie here...so...I installed Rclone and Rclone browser...in Rclone browser I set up a Dropbox connection, went fine, I can browse my Dropbox through it...but when I go to Unraid terminal and type Rclone config it says "no remotes found"...and it appears in the Rclone Browser...what´s wrong here? And plus, when I try to download a file from my Dropbox through Rclone Browser, I set the destination file to a mapped folder in one of my shares, click ok but nothing happens...lost here!

 

I personally would use rclone only via the cli, since the terminology of rclone is not very complicated. So I personally would not use the rclone-browser.

Edited July 31, 2023 by Maginos

[Howboys]

I don't have diagnostics because the server was hung (couldn't do anything), but any idea why this might be happening?

 

Would retrying downloading rclone help with transient network glitches?

[5footbanana]

Any chance of an update to this plugin?  Really want to utilise Proton Drive.  I set this plugin up years ago with Googledrive/onedrive and userscripts. 

 

Not sure the best way to get protondrive working Thanks

[Waseh]

1 hour ago, 5footbanana said:

Any chance of an update to this plugin?  Really want to utilise Proton Drive.  I set this plugin up years ago with Googledrive/onedrive and userscripts. 

 

Not sure the best way to get protondrive working Thanks

Beta support seems to have been added to the rclone beta branch a couple of months ago, so just go to the settings page for the plugin and change to the beta branch? No need for an update.