Jump to content
stignz

Guide: Cloud Storage Backup via rclone (DropBox, GoogleDrive, OneDrive...)

35 posts in this topic Last Reply

Recommended Posts

This is a guide of how I use 'rclone' to backup/sync folders from my unRAID 5 Tower to my cloud storage.

For various reasons CrashPlan was no longer suiting my needs, so I went looking for an alternative solution.

I use it with AmazonCloudServices but it will work for others:

At the time of writing: DropBox; OneDrive; GoogleDrive; Amazon S3; Backblaze B2; and many more.

Refer to http://rclone.org for further details.

 

I'm not intimate with Linux and Slackware, but I have done everything below from the unRAID command line for clarity.

I have broken it down into Installation, Configure, Usage, and making sure it re-installs itself on system reboot.

 

Installation:

# make supporting directory structure on flash drive
mkdir -p /boot/rclone
mkdir -p /boot/rclone/install
mkdir -p /boot/rclone/scripts
mkdir -p /boot/rclone/logs

#download dependencies to /boot/rclone/install
wget http://mirrors.slackware.com/slackware/slackware-13.1/slackware/ap/man-1.6f-i486-2.txz -O //boot/rclone/install/man-1.6f-i486-2.txz
wget http://mirrors.slackware.com/slackware/slackware-13.1/slackware/a/infozip-6.0-i486-1.txz -O //boot/rclone/install/infozip-6.0-i486-1.txz
curl -o /boot/rclone/install/ca-certificates.crt https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt

#Install dependencies (skip infozip if you already have it installed)
installpkg /boot/rclone/install/man-1.6f-i486-2.txz
installpkg /boot/rclone/install/infozip-6.0-i486-1.txz

#Download rclone and unzip
wget http://downloads.rclone.org/rclone-current-linux-386.zip -O //boot/rclone/install/rclone-current-linux-386.zip
unzip /boot/rclone/install/rclone-current-linux-386.zip

#copy binary file
cp /boot/rclone/install/rclone-v*/rclone /usr/sbin/
chown root:root /usr/sbin/rclone
chmod 755 /usr/sbin/rclone
#install manpage (makewhatis is also man-db on some Linux distributions)
mkdir -p /usr/local/share/man/man1
cp /boot/rclone/install/rclone-v*/rclone.1 /usr/local/share/man/man1/
makewhatis
#copy ca-certificates
mkdir -p /etc/ssl/certs/
cp /boot/rclone/install/ca-certificates.crt /etc/ssl/certs/

 

Okay, rclone is now installed.

 

Configuring:

Configuring rclone to connect to your cloud provider is covered in the documentation at http://rclone.org/docs

You are going to create a "Remote" which contains all the connection details and credentials to connect to your cloud service.

You can now run "rclone -config" to set it up from the command line.

This will generate a ".rclone.conf" file in the /root/ directory.

you will want to make a copy of this to the flash so you can re-install on reboot.

#cp /root/.rclone.conf /boot/rclone/install/

 

You could also install rclone on another Windows/Linux/OSx machine which has a gui, generate this file, and then copy it to the "/boot/rclone/install/" directory.

Then run 'cp /boot/rclone/install/.rclone.conf /root/' to copy it to the rclone config default directory.

Note: Some cloud providers require authentication via a web browser, so this is the only way to generate the config file.

 

Useage:

To test that the config is loaded properly

Run "rclone -config" again and you should see your remote.

Run "rclone size <RemoteName>:<FolderPath>" and it should spit out the number of files and size.

 

rclone has two modes, copy, and sync. (both uni directional)

'Copy' will just copy anything from the source directory to the remote directory, if it doesn't exist in the remote directory.

'Sync' will sync the two directories, i.e. if it finds something in the remote directory that is not in the source directory, it will delete it.

 

#to start a copy from a folder in an unraid share:
rclone copy /mnt/user/<ShareName>/<FolderName> <RemoteName>:<RemoteFolderName>
#likewise for a sync:
rclone sync /mnt/user/<ShareName>/<FolderName> <RemoteName>:<RemoteFolderName>

 

(If you are doing a long running test from putty then use "nohup <command> &" to prevent a hangup on the running command when you exit the client.)

 

Running it manually may be of little value to you, so now we make a script and add it to crontab.

 

I made a script called "rcloneCopy.sh", in the /boot/rclone/scripts folder, with the following inside:

(The logfiles are optional but would be useful for trouble shooting and checking up on it etc.)

LOGFILE=/boot/rclone/logs/rclone-$(date "+%Y%m%d").log
echo rclone log $(date) $'\r'$'\r' >> $LOGFILE 2>&1
echo "Starting rclone copy" $'\r'>> $LOGFILE 2>&1
rclone copy /mnt/user/<ShareName>/<FolderName> <RemoteName>:<RemoteFolderName> >> $LOGFILE 2>&1

#To make it executable:
chmod u+x /boot/rclone/scripts/rcloneCopy.sh
#To add it to crontab to be run at midnight, everynight:
crontab -l | { cat; echo "0 0 * * * /boot/rclone/scripts/rcloneCopy.sh > /dev/null 2>&1"; } | crontab -

 

If you are doing an initial bulk upload and don't want it using bandwidth during the day then the following script inside "rcloneStop.sh" would stop it:

LOGFILE=/boot/rclone/logs/rclone-$(date "+%Y%m%d").log
PROCESSID=$(pgrep rclone)
if [ -n $PROCESSID ]; then
echo "Attempting to terminate rclone" $'\r'>> $LOGFILE 2>&1
echo "processid of rclone: $PROCESSID" $'\r'  >> $LOGFILE 2>&1
kill -s 15 $(pgrep rclone)
echo "rclone job termination requested by cron at "$(date "+%T") $'\r'>> $LOGFILE 2>&1
fi

#again make it executable, and add to crontab to be run at say 8am:
chmod u+x /boot/rclone/scripts/rcloneStop.sh
crontab -l | { cat; echo "0 8 * * * /boot/rclone/scripts/rcloneStop.sh > /dev/null 2>&1"; } | crontab -

 

Auto Re-install on system restart

Lastly, we don't want to do this again when unRAID is shutdown and restarted.

The following code inside the script "rcloneStartup.sh" would auto install and set everything running again.

sleep 5m &&

#Install dependancies
installpkg /boot/rclone/install/man-1.6f-i486-2.txz
installpkg /boot/rclone/install/infozip-6.0-i486-1.txz

#Download fresh version of rclone
wget http://downloads.rclone.org/rclone-current-linux-386.zip -O //boot/rclone/rclone-current-linux-386.zip

#check if file downloaded
if [ -f "/boot/rclone/rclone-current-linux-386.zip" ]
then
    #remove existing rclone zip
rm /boot/rclone/install/rclone-current-linux-386.zip
rm -rf /boot/rclone/install/rclone-v*
#move new rclone zip to install directory
mv /boot/rclone/rclone-current-linux-386.zip /boot/rclone/install
#unzip the rclone download into the install directory
unzip /boot/rclone/install/rclone-current-linux-386.zip -d /boot/rclone/install/
fi

#Install rclone
#copy binary file
cp /boot/rclone/install/rclone-v*/rclone /usr/sbin/
chown root:root /usr/sbin/rclone
chmod 755 /usr/sbin/rclone
#install manpage
mkdir -p /usr/local/share/man/man1
cp /boot/rclone/install/rclone-v*/rclone.1 /usr/local/share/man/man1/
makewhatis

# copy config file to default location
cp /boot/rclone/install/.rclone.conf /root/

# set ssl certs
mkdir -p /etc/ssl/certs/
cp /boot/rclone/install/ca-certificates.crt /etc/ssl/certs/t

#make any scripts executable
chmod u+x /boot/rclone/scripts/rcloneCopy.sh

# set cron job, this will run at 1 min past midnight every night
crontab -l | { cat; echo "1 0 * * * /boot/rclone/scripts/rcloneCopy.sh > /dev/null 2>&1"; } | crontab -

 

Then add the following line to the end of your "go" file:

chmod u+x /boot/rclone/scripts/rcloneStartup.sh | /boot/rclone/scripts/rcloneStartup.sh

 

I have attached the scripts.

 

Hopefully my version of how to execute this makes enough sense.

Any suggestions to improve on the above would be welcome as well. :)

 

I'd like to add functionality to email monthly stats at a later date.

 

Lastly, a very big thank you to the writer of rclone.

It has solved a massive problem for me.

rcloneCopy.sh.txt

rcloneStartup.sh.txt

  • Like 1

Share this post


Link to post

This looks great!

 

I have been looking for something to use my Amazon Cloud Drive with.  Might take a few days, but I plan on following the guide.  Very detailed!!!!

Share this post


Link to post

Thanks for this, can setup my Google Drive without having the OS version installed AND it gives me a brilliant idea on how to approach setting up my sshfs mounts

Share this post


Link to post

You make me happy!

 

For a 64-bit CPU I used these parameters:

 

wget http://omegageek.noip.me/mirrors/slackware/slackware64-13.1/slackware64/ap/man-1.6f-x86_64-2.txz -O //boot/rclone/install/man-1.6f-i486-2.txz
wget http://slackware.bokxing-it.nl/mirror/slackware64-13.1/slackware64/a/infozip-6.0-x86_64-1.txz -O //boot/rclone/install/infozip-6.0-i486-1.txz
wget http://downloads.rclone.org/rclone-v1.28-linux-amd64.zip -O //boot/rclone/install/rclone-v1.28-linux-amd64.zip

 

rcloneStartup.sh.txt

Share this post


Link to post

New Problem: now my problem is it creates a duplicate folder in the amazon cloud drive, and uploads everything to that new folder.

if i have an RClone folder in the cloud, and then rclone copy /mnt/user/RClone remote:RClone, it will create another RClone folder

 

Previous Solved: When I try to install it via a windows terminal to my unraid server, I get to this point

"cp /boot/rclone/install/rclone-v*/rclone /usr/sbin/"

and get the following message "cp: cannot stat C /boot/rclone/install/rclone-v*/rclone C : No such file or directory"

 

any idea on how to fix it? (not using " " on actual command)

 

Edit - Got it running, had to use amd64 version though, and also had to specify the directory when using the unzip command.  havent tested reinstall script yet though.

 

Question: When it runs using cron, is there a way to check its status, to see how much or what its uploading? or perhaps have an ETA?

I havent gotten logs to work, had to delete that part from the .sh to get it to run.

got everything working right after i made sure each .sh and the go file were using the UNIX language, as shown on the bottom right side of Notepad++

 

now my problem is it creates a duplicate folder in the amazon cloud drive, and uploads everything to that new folder.

 

Share this post


Link to post

I also can't get the logs to work with rcloneCopy.sh.

 

Code:

 

LOGFILE=/boot/rclone/logs/rclone-$(date "+%Y%m%d").log

echo rclone log $(date) $'\r'$'\r' >> $LOGFILE 2>&1

echo "Starting rclone copy" $'\r'>> $LOGFILE 2>&1

rclone copy /mnt/user/Pictures remote:Backup_Pictures >> $LOGFILE 2>&1

 

Error:

 

: Invalid argumentts/rcloneCopy.sh: line 2: /boot/rclone/logs/rclone-20160607.log

: Invalid argumentts/rcloneCopy.sh: line 3: /boot/rclone/logs/rclone-20160607.log

: Invalid argumentts/rcloneCopy.sh: line 4: /boot/rclone/logs/rclone-20160607.log

 

 

I can get these commands to work manually copying and pasting them in a telnet session.

 

I tried adding delays, but that didn't help.  Not sure why this code isn't working from a script.

 

Share this post


Link to post

Sorry, I haven't visited in a while.

 

Ayefly:

Glad you worked though most of the problems.

I'm not sure why you are getting the duplicate folders.

If you want to see progress in the telnet session then just run the copy command from your rcloneCopy script, and add the option for verbose.

Verbose which will give you much more detail and may give more clues to the folder duplication

i.e. "nohup rclone copy -v /mnt/user/Photos AmazonCloudDrive:Photos &"

If it a long runtime is expected, you can run it with NoHangUp so that you can close the session and the command will keep running, and you can fetch the nohup.out file later.

 

nblain:

The first line creates the logfile, so all other references to it on the following lines are invalid.

 

If you used Notepad++ to edit the script, make sure that the format is 'UNIX' (right click, near bottom right corner)

As Ayefly found out, if it is Macintosh or Windows then the line endings will be invalid in bash.

 

Else, has the script has been made executable?

e.g. "chmod u+x /boot/rclone/scripts/rcloneCopy.sh"

 

If the above is all correct, can you let me know which version of unRAID you are running?

I'm thinking we may need to specify the shell in the first line of the script.

 

Edit: It looks like rclone now has logging built into it. I'll have a play and update.

 

Share this post


Link to post

What update are you looking for? I installed this last week with the latest rclone. Works great. It'd be awesome if someone built a plugin/docker with a web interface for it.

Share this post


Link to post

Hi folks!

  Working this same project to get b2 working--Thanks stignz!  I've got the command line rclone and script working, but cron is making me scratch my head.

I followed your input command with

  crontab -l | { cat; echo "0 0 * * SUN,WED /boot/rclone/scripts/rcloneStop.sh > /dev/null 2>&1"; } | crontab -

but then realized I wanted it to run at 2 am instead...

so I dumped the crontab with

crontab -l  >> tmp.cron

and edited it to

# If you don't want the output of a cron job mailed to you, you have to direct
# any output to /dev/null.  We'll do this here since these jobs should run
# properly on a newly installed system.  If a script fails, run-parts will
# mail a notice to root.
#
# Run the hourly, daily, weekly, and monthly cron jobs.
# Jobs that need different timing may be entered into the crontab as before,
# but most really don't need greater granularity than this.  If the exact
# times of the hourly, daily, weekly, and monthly cron jobs do not suit your
# needs, feel free to adjust them.
#
# Run hourly cron jobs at 47 minutes after the hour:
47 * * * * /usr/bin/run-parts /etc/cron.hourly 1> /dev/null
#
# Run daily cron jobs at 4:40 every day:
40 4 * * * /usr/bin/run-parts /etc/cron.daily 1> /dev/null
#
# Run weekly cron jobs at 4:30 on the first day of the week:
30 4 * * 0 /usr/bin/run-parts /etc/cron.weekly 1> /dev/null
#
# Run monthly cron jobs at 4:20 on the first day of the month:
20 4 1 * * /usr/bin/run-parts /etc/cron.monthly 1> /dev/null
#
# Runs twice a week at 2:00 to execute Backblaze b2 rclone job:
55 17 * * SAT,WED /boot/rclone/scripts/rcloneCopy.sh > /dev/null 2>&1

and then ran crontab on that file (I put the 17:55 Saturday time in there to check to see if it ran.)

Unfortunately, I got this ( a couple times) in the syslog with no output in the rclone logs

unServer crond[2440]: failed parsing crontab for user root: SAT,WED /boot/rclone/scripts/rcloneCopy.sh > /dev/null 2>&1

Any idea where I boinked this thing? Thanks guys!!

 

Josh

Share this post


Link to post

Have you tried specifying the days as numerals instead? Specifying only one day?

Share this post


Link to post

Hmm.

  So not all crontabs are created equally. Good point. I tried this and it worked. I guess I'll just need two entries to do the job. Thanks Squid!!

 

  For shitzengiggles, what's the difference between 'committing' a change via "crontab [file]" and editing /etc/cron.d/root or cron.[interval]? Is there a better/more secure way of doing this?

 

Josh

Share this post


Link to post

trurl,

  Thanks for that link. I guess my answer was more the academic one of best practices--got some further info from this link http://unix.stackexchange.com/questions/196009/location-of-the-crontab-file , if any other newbies are curious.

 

  In case anyone other than me is confused, it seems (again, grains of salt here, friends) that the unRAID folks have a plugin cron "injector" running that injects entries from /boot/config/plugins/dynamix (with .cron endings) into /etc/ cron.d/root, cron.hourly, cron.daily, cron.weekly, and cron.monthly. I'm guessing this happens at boot time. These injected cron jobs (specific to unRAID operations) can be controlled from the GUI: settings->scheduler and flexibility of this is expanded with the schedules and SSD trim plugins (and those are only the ones I'm using).

  When I commit a change to the crontab using the crontab command, it is changing the file at /var/spool/cron/crontabs/root, which already has hooks to the periodic cronjobs in /etc/ in it, which I'm guessing get put there by the "injector" code whenever I reboot or make changes.

 

It's pretty academic at this point, but I'm guessing that makes rclone scheduling best done via the recurring crontab command/scripts at the top, so I guess this was a long circular road. At least I learned something!

 

Thanks again,

Josh

 

 

 

Share this post


Link to post

Suggestion:

 

Rather than using crontab directly, just create a .cron script anywhere in /boot/config, and Unraid v6+ will automatically add it to  the cronjob list.  I have my scripts in /boot/config/custom_cron/.  (Credit goes to this post: https://lime-technology.com/forum/index.php?topic=44172.msg421807#msg421807)

 

Upon system reboot ALL .cron files are automatically loaded in the cronjob.

 

If you make changes to the cron files and you can reload changes by executing update_cron.

 

I find this approach much easier than having scripts inject stuff into the schedule using crontab.

Share this post


Link to post

I made a simple plugin that installs rclone, adds a helper script to avoid typing the path to the config file ("config/plugins/rclone/.rclone.conf") all the time and also adds a .cron file template to the plugin folder.

 

To setup the cloud storage you need to follow the setup wizard:

myrclone config

 

I use myrclone with Blackblaze B2 like that:

myrclone sync /mnt/user/Public/test b2:mytest

 

The myrclone script has a --log option that stores the logs to "config/plugins/rclone/logs"

 

The cron template file "config/plugins/rclone/~daily_backup.cron" can be edited/renamed/duplicated to suit your needs (you also need to remove the comment in line 2):

# daily backup of test folder to blackblaze b2 mytest bucket (with log option)
#0 0 * * * /usr/sbin/myrclone --log sync /mnt/user/Public/test b2:mytest &> /dev/null

 

Logs and config file are kept on uninstall.

 

Andreas

rclone.plg

Share this post


Link to post

I made a simple plugin that installs rclone, adds a helper script to avoid typing the path to the config file ("config/plugins/rclone/.rclone.conf") all the time and also adds a .cron file template to the plugin folder.

 

To setup the cloud storage you need to follow the setup wizard:

myrclone config

 

I use myrclone with Blackblaze B2 like that:

myrclone sync /mnt/user/Public/test b2:mytest

 

The myrclone script has a --log option that stores the logs to "config/plugins/rclone/logs"

 

The cron template file "config/plugins/rclone/~daily_backup.cron" can be edited/renamed/duplicated to suit your needs (you also need to remove the comment in line 2):

# daily backup of test folder to blackblaze b2 mytest bucket (with log option)
#0 0 * * * /usr/sbin/myrclone --log sync /mnt/user/Public/test b2:mytest &> /dev/null

 

Logs and config file are kept on uninstall.

 

Andreas

 

I'd appreciate it if you could get this up on the community apps. I am testing the rclone docker right now but being a CLI tool, I think this kind of app might be better implemented as a plugin. I'm also not sure if I would be able to make a remote share available to the entire system if done in a docker.

Share this post


Link to post

I made a simple plugin that installs rclone, adds a helper script to avoid typing the path to the config file ("config/plugins/rclone/.rclone.conf") all the time and also adds a .cron file template to the plugin folder.

 

To setup the cloud storage you need to follow the setup wizard:

myrclone config

 

I use myrclone with Blackblaze B2 like that:

myrclone sync /mnt/user/Public/test b2:mytest

 

The myrclone script has a --log option that stores the logs to "config/plugins/rclone/logs"

 

The cron template file "config/plugins/rclone/~daily_backup.cron" can be edited/renamed/duplicated to suit your needs (you also need to remove the comment in line 2):

# daily backup of test folder to blackblaze b2 mytest bucket (with log option)
#0 0 * * * /usr/sbin/myrclone --log sync /mnt/user/Public/test b2:mytest &> /dev/null

 

Logs and config file are kept on uninstall.

 

Andreas

 

I get an error:

/usr/sbin/myrclone: line 22: rclone: command not found

Share this post


Link to post

I'm was on the cusp of purchasing a Syncovery license to synch with ACD but rclone on my server directly might be more straightforward but crypto is a 100% must for both content and filenames. Rclone appears to support this, is anyone using crypto on their UnRAID backups? How are folks throttling bandwidth usage? Lastly, what rclone files exactly need to be backed up in order to restore access in the event you need to do a restore on fresh hardware? My goal in doing this is to have a secure backup in the event of a fire, flood, or theft so an offsite backup of critical files to decrypt my backup is kind of important :)

 

Some of the best information I've found on rclone has been on datahoarders and this thread below has been helpful. Folks might also be interested in acd_cli for mounting ACD drives remotely although it looks like rclone may be able to do that now.

 

 

http://rclone.org/crypt/ this is the rclone documentation for rclone crypto.

Share this post


Link to post

No I hadn't, thank you! I've been tinkering with Syncovery's beta and a manual install of rclone on a Linux box I've got, hopefully this will make life easier - i'll try it now :)

 

Share this post


Link to post

Hello,

 

When I try this step of the installation:

 

#Download rclone and unzip

wget http://downloads.rclone.org/rclone-current-linux-386.zip -O //boot/rclone/install/rclone-current-linux-386.zip

unzip /boot/rclone/install/rclone-current-linux-386.zip

 

I receive the message:

 

-bash: /usr/bin/unzip: cannot execute binary file

 

Any ideas as to the problem?

 

I have followed all the preceding steps with successful results.

Share this post


Link to post

Hello,

 

When I try this step of the installation:

 

#Download rclone and unzip

wget http://downloads.rclone.org/rclone-current-linux-386.zip -O //boot/rclone/install/rclone-current-linux-386.zip

unzip /boot/rclone/install/rclone-current-linux-386.zip

 

I receive the message:

 

-bash: /usr/bin/unzip: cannot execute binary file

 

Any ideas as to the problem?

 

I have followed all the preceding steps with successful results.

 

There's a plugin. No need to do it manually.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now