[Plugin] rclone

[Dimtar]

Thanks for your help bobbintb and Waseh. I hope they support rsync etc. in future but for now rclone copy is working and the mount remains working.

[bobbintb]

Thanks for your help bobbintb and Waseh. I hope they support rsync etc. in future but for now rclone copy is working and the mount remains working.

 

I doubt they will. Rclone is "rsync for the cloud". It is meant to satisfy a role that rsync does not. It doesn't really make sense to be using them together like that.

[bobbintb]

I have tried with a setting of 50Mb but no luck.

 

As another test I also mounted rclone under my encfs (replacing acd_cli) and I still get similar speeds starting playback, so it doesn't appear to be the encryption layer. Trying to use verbose to debug, but I might need to post on the rclone logs for help with that.

 

Thanks,

Wob

 

So you are getting similar speeds using rclone to mount and encfs to encrypt as you are with using rclone for both mounting and encrypting? Is anyone else getting similar results? Hopefully I'll be able to help test soon. Just too busy lately. That would be very telling because if it is the encryption layer, I don't know if much can be done to improve the performance as doing so would likely have serious repercussions on existing setups. Maybe I'm wrong about that. If it is mounting, I think there is room for improvement. Personally, even if acd_cli/encfs is better now, I still might bank on the fact that rclone is able to improve performance.  acd_cli and encfs being two separate projects, I don't expect much to change in relation to each other.

 

Edit: One of posts I mentioned earlier, I forget which thread I mention it in, seems to indicate it is indeed crypt causing the issue.

 

https://forum.rclone.org/t/plex-server-with-amazon-drive-rclone-crypt-vs-encfs-speed/106/17

 

The author of rclone is looking into it. Just based on the fact that the performance is decent enough and the fact that there is active and significant work being done on rclone, I think I'll stick with that. I'm fairly confident the performance issues are just due to how new this is and will be resolved eventually.

[Wob76]

So you are getting similar speeds using rclone to mount and encfs to encrypt as you are with using rclone for both mounting and encrypting? Is anyone else getting similar results? Hopefully I'll be able to help test soon. Just too busy lately. That would be very telling because if it is the encryption layer, I don't know if much can be done to improve the performance as doing so would likely have serious repercussions on existing setups. Maybe I'm wrong about that. If it is mounting, I think there is room for improvement. Personally, even if acd_cli/encfs is better now, I still might bank on the fact that rclone is able to improve performance.  acd_cli and encfs being two separate projects, I don't expect much to change in relation to each other.

 

Edit: One of posts I mentioned earlier, I forget which thread I mention it in, seems to indicate it is indeed crypt causing the issue.

 

https://forum.rclone.org/t/plex-server-with-amazon-drive-rclone-crypt-vs-encfs-speed/106/17

 

The author of rclone is looking into it. Just based on the fact that the performance is decent enough and the fact that there is active and significant work being done on rclone, I think I'll stick with that. I'm fairly confident the performance issues are just due to how new this is and will be resolved eventually.

 

Hi,

 

Yep, for me at lest using encfs on top of rclone or using crypt in rclone results in the same speeds, both slightly slower than acd_cli/encfs. So at least for my config, and with my testing process the encryption layer is not having any effect, it seems to be rclone. The post you linked seems to think the fix was introduced in a 1.33 beta, so I assume it's included in the 1.34 beta I am using. I have posted on the rclone forum, so hopefully I can assist in some improvement.

 

I tend to agree with your statement, I will most likely lean with rclone as the development seems much faster, and once I choose an encryption layer swapping will be a pain (re-uploading will be needed).

 

I think I'll just continue with some stability testing before I choose an option, I have both mounted at the moment, so will see if the mounts remain stable and playback smooth from both sources.

 

Wob

[bobbintb]

I can't seem to find any sort of changelog for rclone beta. Not the plugin but the program itself. I can find the regular changelog but nothing for the beta version. Can someone direct me?

 

Beta releases are generated from each commit to master. Note these are named like

 

Source: http://rclone.org/downloads/

 

Still not quite seeing it. I see that there is a gitlog but that really just shows commits. I can download the beta and there is a changelog in the zip file but it only shows the stable release. Is there any place that shows what has changed between beta versions? I'm still not finding one.

[DZMM]

WARNING - do not run your rclone scripts at the start of the array.  I did this and it froze my server - the jobs worked, but my server wasn't accessible:

 

 

 

 

[/size]My first guess was right and the problem was my rclone script which was doing my backup job at the start of the array.  Previously I was using overnight cron jobs, so having the script run at the start of the array was allowing the cron job run but was freezing the machine.

[bobbintb]

WARNING - do not run your rclone scripts at the start of the array.  I did this and it froze my server - the jobs worked, but my server wasn't accessible:

 

 

 

 

[/size]My first guess was right and the problem was my rclone script which was doing my backup job at the start of the array.  Previously I was using overnight cron jobs, so having the script run at the start of the array was allowing the cron job run but was freezing the machine.

 

What script did you run? Sync cron? I did mount that way with no problem. There is a sync script that runs when changes are made instead on cron.

[DZMM]

My rclone sync jobs.  Tested and they would run, but the rest of the array wouldn't get running.  Set them to run overnight now.

[ninthwalker]

I've decided to stick with rclone + crypt as its a nice all in one package.

While it's still a little slower than acd_cli/encfs I don't think the little extra start/load times will impact my older archived movies/tv shows that I have planned for ACD streaming.

 

And I figure rclone can only get better. Began my uploads today.

[Waseh]

Hey guys!

 

I've pushed an update to the beta branch that updates the rclone version to the latest beta.

However in other and bigger news the new beta includes a very (very) basic webgui that allows you to edit the rclone.config as well as a few template scripts that can be edited and moved to the User Scripts Plugin all from the unraid webgui. I'm sure you might find some bugs but it's a start, and i've been using it in my own setup the last couple of days with no hickups

Any suggestions for improvement are very welcome

[Wob76]

Great work,

 

I haven't tested the scripts yet, I already have some working once, but it is a good step forward for new users.

 

Only issue I did spot is that your icon seems to be a different size to the others.

 

[Waseh]

Try clearing the browser cache - I have downsized the icon, so either its not getting overwritten with the new 48px icon or your browser cached the old one

[Wob76]

Try clearing the browser cache - I have downsized the icon, so either its not getting overwritten with the new 48px icon or your browser cached the old one

 

I just reopened the web interface it and its fine now.

 

Thanks

[Wob76]

Hi,

 

A Couple of questions.

 

Just looking at your scripts, with your mount I note you are calling rclone as normal.

I had issues with this, it would time out, usually after my terminal ended. Is this not an issue when calling from user scripts?

My solution was to call rclone via screen.

 

as such.

screen -S rclone -d -m rclone mount --allow-other --dir-cache-time 60m --max-read-ahead 200M --uid 99 --gid 100 secret:media /mnt/disks/cloud/rclone

 

Also on the point of using /mnt/disks/ to help with dockers, this doesn't seem to work as well with rclone as it was for acd_cli. If I remount rclone for some reason I then get and error "Transport endpoint is not connected" when I try and access the mount \ volume mapping inside my docker, a restart fixes it. I have the volume set as Slave:RW as I did with acd_cli. Anyone else seeing that?

 

Thanks,

Wob

[bobbintb]

Love the additions so far. I'd like to request that I be able to go to the GUI page by clicking on the icon on the plugins page. I think most of the plugins work that way in addition to going through the settings page. It would be nice for consistency and ease of use.

[bobbintb]

I have made a daemon for user scripts. It requires inotify, which can be found in nerd tools.

 

#DECLARE VARIABLES
backup_dir=""
server_backup_dir=""
log_dir="/boot/config/plugins/rclone-beta/logs/backuperrors.txt"

trap 'kill -HUP 0' EXIT

#FUNCTION TO USE RYSNC TO BACKUP DIRECTORIES
function backup () {

if rclone sync $backup_dir $server_backup_dir --transfers 1  2>&1 >>$log_dir
then
echo "backup succeeded $backup_dir"
else
echo "rclone failed on $backup_dir"
return 1
fi
}

#CHECK IF INOTIFY-TOOLS IS INSTALLED
type -P inotifywait &>/dev/null || { echo "inotifywait command not found."; exit 1; }

#INFINITE WHILE LOOP
while true
do

#START RCLONE AND ENSURE DIR ARE UPTO DATA
backup  || exit 0

#START RCLONE AND TRIGGER BACKUP ON CHANGE
inotifywait -r -e modify,attrib,close_write,move,create,delete  --format '%T %:e %f' --timefmt '%c' $backup_dir  2>&1 >>$log_dir && backup

done

 

Just fill in the two variables. I have partially testing it with user scripts. It works if manually run but I have not set it to run at array start so let me know if there are issues. I suspect there might be if this script somehow gets ran before the mount script. If so, a simple sleep at the beginning of this script should fix that. Waseh, feel free to include this in the plugin if you'd like.

[Waseh]

Hi,

 

A Couple of questions.

 

Just looking at your scripts, with your mount I note you are calling rclone as normal.

I had issues with this, it would time out, usually after my terminal ended. Is this not an issue when calling from user scripts?

My solution was to call rclone via screen.

 

as such.

screen -S rclone -d -m rclone mount --allow-other --dir-cache-time 60m --max-read-ahead 200M --uid 99 --gid 100 secret:media /mnt/disks/cloud/rclone

 

Also on the point of using /mnt/disks/ to help with dockers, this doesn't seem to work as well with rclone as it was for acd_cli. If I remount rclone for some reason I then get and error "Transport endpoint is not connected" when I try and access the mount \ volume mapping inside my docker, a restart fixes it. I have the volume set as Slave:RW as I did with acd_cli. Anyone else seeing that?

 

Thanks,

Wob

 

Indeed - When using User Scripts you can just run the script in the background

 

I was actually able to recreate the "Transport endpoint is not connected" - But i can only recreate it when i close down the mount script incorrectly for example in your case closing screen before unmounting. Also i could fix it by unmounting correctly and then remounting, without restarting.

This also made me discover that i can still browse the mount from my dockers even when it's unmounted and cant be browsed from neither cmd line or my samba share. I can even playback media. It seems the mount somehow survives inside the dockers even when unmounted from the system.

[Waseh]

Love the additions so far. I'd like to request that I be able to go to the GUI page by clicking on the icon on the plugins page. I think most of the plugins work that way in addition to going through the settings page. It would be nice for consistency and ease of use.

 

Yea that's definitely missing! i'll push an update that lets you go to settings from the plugin page!

In regards to your script i think i might link to it from the OP instead. I was thinking that the included scripts would be the very basics and any more advanced scripts should probably be added manually. One could always use the custom (and empty) script for this

 

[Wob76]

 

Indeed - When using User Scripts you can just run the script in the background

 

I was actually able to recreate the "Transport endpoint is not connected" - But i can only recreate it when i close down the mount script incorrectly for example in your case closing screen before unmounting. Also i could fix it by unmounting correctly and then remounting, without restarting.

This also made me discover that i can still browse the mount from my dockers even when it's unmounted and cant be browsed from neither cmd line or my samba share. I can even playback media. It seems the mount somehow survives inside the dockers even when unmounted from the system.

 

Thanks for the info, at the moment I think I will leave mine running under screen as it is working nicely

 

I have moved my rclone mount to under a union-fs mount, it is now the lower layer, with RO access and my local data as the upper layer. This seems to have resolved my issue with dockers for now.

 

Interesting, what happens when you attempt to access some data from the mount?

[Waseh]

Inside dockers data loads as if mounted - It's very peculiar.

[meyerjoe83]

Hey,

 

I don't understand your directions on how to mount: I created a folder called mount and tried running this command:

rclone mount --allow-other secret: /mnt/user/mount/ &

 

I get this error:  Fatal error: failed to mount FUSE fs: mountpoint does not exist: /mnt/user/mount

 

What am I missing here?

 

For this part of the instructions: Set the container/host volume with a mode of Read Write,Slave, else the files will not show up inside the container.

 

That is when I am creating the container for plex?

[bobbintb]

Hey,

 

I don't understand your directions on how to mount: I created a folder called mount and tried running this command:

rclone mount --allow-other secret: /mnt/user/mount/ &

 

I get this error:  Fatal error: failed to mount FUSE fs: mountpoint does not exist: /mnt/user/mount

 

What am I missing here?

 

It's just a hunch but I'd say the mointpoint does not exist. 

Also, update your plugin if you don't have the latest. The OP needs to be updated to reflect this but the latest version supplies scripts for mounting and unmounting (for the user scripts plugin). All you have to do is specify the source and destination so you don't have to deal with things like forgetting to create the mountpoint first. If you do have the latest version just go into the settings page for the plugin to activate the script.

 

For this part of the instructions: Set the container/host volume with a mode of Read Write,Slave, else the files will not show up inside the container.

 

That is when I am creating the container for plex?

 

Yes, or editing an existing container.

[meyerjoe83]

Hey,

 

I don't understand your directions on how to mount: I created a folder called mount and tried running this command:

rclone mount --allow-other secret: /mnt/user/mount/ &

 

I get this error:  Fatal error: failed to mount FUSE fs: mountpoint does not exist: /mnt/user/mount

 

What am I missing here?

 

It's just a hunch but I'd say the mointpoint does not exist. 

Also, update your plugin if you don't have the latest. The OP needs to be updated to reflect this but the latest version supplies scripts for mounting and unmounting (for the user scripts plugin). All you have to do is specify the source and destination so you don't have to deal with things like forgetting to create the mountpoint first. If you do have the latest version just go into the settings page for the plugin to activate the script.

 

For this part of the instructions: Set the container/host volume with a mode of Read Write,Slave, else the files will not show up inside the container.

 

That is when I am creating the container for plex?

 

Yes, or editing an existing container.

 

Thanks for the help. I was able to get the mount run without errors and added it to plex, but don't see any contents. 

 

This is my Mount Script

mkdir -p $/mnt/disk3/Mount/

rclone mount --max-read-ahead 1024k --allow-other $secret: $/mnt/disk3/Mount/ &

 

This seems to work because I am not getting any errors in the logs, but I may be incorrect

 

I have my container for plex setup like this:

/mnt/disk3/Mount/

Access Mode: RW/Slave

 

Any thoughts on why content isn't showing up?

 

Appreciate the help, I'm definitely a noob when it comes to this

[bobbintb]

You're making things much more difficult by reinventing the wheel. Just use the included scripts. They were created specifically to avoid this type of situation.

[Waseh]

I agree with bobbintb!

Just be aware that only the beta branch has included scripts and gui. I will update the stable branch (and the OP) one of the comming days