Process to copy "appdata" docker folder form cache to array for backup

[mostlydave]

The title says it all, I'm looking for a way to schedule an automatic nightly backup of my appdata folder on my cache drive to a backup share in order to have it saved to the array in the event of or drive failure.

 

I was looking at this thread as a starting point http://lime-technology.com/forum/index.php?topic=31246.0

 

it's in the 5.x thread and looks like its the right direction, I guess it would just need to be modified to stop and start the docker service to work for me. I have zero scripting skill so if anyone can let us know how you're backing up your appdata directory that would be great!

[johnsanc]

I am also looking to do this. Any help from the scripting gurus is greatly appreciated!

[johnsanc]

Nevermind, I think I figured it out. For anyone who is interested I used this script and set it to run daily in my Go file.

 

You will need to modify your directories accordingly. I have my appdata on a disk mounted with SNAP outside of the array at /mnt/disk/appdisk/appdata/. My docker img file is under /mnt/disk/appdisk/docker/

 

Also, rsync does not like Plex and will give you a ton of "failed to set times" errors if you backup to a user share. However, it works fine if you backup directly to a disk share, which is why the script below references disk1 instead of my Backup user share.

 

Script:

 

#!/bin/bash

#Stop Docker Apps
docker stop CouchPotato
docker stop CrashPlan
docker stop Deluge
docker stop MariaDB
docker stop NzbDrone
docker stop NZBGet
docker stop PlexMediaServer
docker stop Syncthing

#Backup Docker App data via rsync
date >/var/log/docker_app_backup.log
date >/var/log/docker_img_backup.log
/usr/bin/rsync -avrtH --delete /mnt/disk/appdisk/appdata/ /mnt/disk1/Backup/appdata >>/var/log/docker_app_backup.log
/usr/bin/rsync -avrtH --delete /mnt/disk/appdisk/docker/ /mnt/disk1/Backup/docker >>/var/log/docker_img_backup.log

#Start Docker Apps
docker start CouchPotato
docker start CrashPlan
docker start Deluge
docker start MariaDB
docker start NzbDrone
docker start NZBGet
docker start PlexMediaServer
docker start Syncthing

 

Go file:

 

# Set up daily Docker App backup
cp /boot/custom/backup.sh /etc/cron.daily

 

This was pretty simple, but I hope someone else finds this useful.

 

[gundamguy]

Nevermind, I think I figured it out. For anyone who is interested I used this script and set it to run daily in my Go file.

 

You will need to modify your directories accordingly. I have my appdata on a disk mounted with SNAP outside of the array at /mnt/disk/appdisk/appdata/. My docker img file is under /mnt/disk/appdisk/docker/

 

Also, rsync does not like Plex and will give you a ton of "failed to set times" errors if you backup to a user share. However, it works fine if you backup directly to a disk share, which is why the script below references disk1 instead of my Backup user share.

 

Script:

 

#!/bin/bash

#Stop Docker Apps
docker stop CouchPotato
docker stop CrashPlan
docker stop Deluge
docker stop MariaDB
docker stop NzbDrone
docker stop NZBGet
docker stop PlexMediaServer
docker stop Syncthing

#Backup Docker App data via rsync
date >/var/log/docker_app_backup.log
date >/var/log/docker_img_backup.log
/usr/bin/rsync -avrtH --delete /mnt/disk/appdisk/appdata/ /mnt/disk1/Backup/appdata >>/var/log/docker_app_backup.log
/usr/bin/rsync -avrtH --delete /mnt/disk/appdisk/docker/ /mnt/disk1/Backup/docker >>/var/log/docker_img_backup.log

#Start Docker Apps
docker start CouchPotato
docker start CrashPlan
docker start Deluge
docker start MariaDB
docker start NzbDrone
docker start NZBGet
docker start PlexMediaServer
docker start Syncthing

 

Go file:

 

# Set up daily Docker App backup
cp /boot/custom/backup.sh /etc/cron.daily

 

This was pretty simple, but I hope someone else finds this useful.

 

Couple questions about rsync your code. 

rsync -avrtH --delete 

 

First: I am confused as to why you've specified -a (archive mode) and also -r (recursive) and -t (preserve modification times) when archive mode already includes -r and -t, -a is exactly the same as individually to specifying the following options -rlptgoD (recursive, copy symlinks as symlinks, preserve permissions, preserve mod time, preserve group, preserve owner, preserve device files, and preserve special files).

 

Second: Does it really need to be verbose (-v) since you are running it daily and in the background? Is this for logging purposes? Wouldn't it be better have rsync make a log file (--log-file=FILENAME) in that case?

 

Third: Are Hard-links (-H) really important? Does it break things if you don't preserve Hard-links? I legitimately want to know more about this and what should or should not be specified to make a good Rsync backup.

 

Question about using Rsync this way.... won't running Rsync Daily with --delete prevent you from being able to roll back more than 24 hours? In the case of a disk dying that should be ok, but in the case of file system corruption or non disk failure events that cause errors this means you have to catch it within 24 hours or your backup will not be able to roll back... right?

 

Can Rsync be used to create a “Time-Machine” like backup where you can roll back to more than one earlier state? If yes what should / should not be specified.

 

[johnsanc]

Good points. Frankly I am by no means an rsync expert so I hope someone else can chime in. I just modified a script from someone else on the 5.x support forum so it works with docker. I didn't tinker with the arguments.

[CHBMB]

Nevermind, I think I figured it out. For anyone who is interested I used this script and set it to run daily in my Go file.

 

You will need to modify your directories accordingly. I have my appdata on a disk mounted with SNAP outside of the array at /mnt/disk/appdisk/appdata/. My docker img file is under /mnt/disk/appdisk/docker/

 

Also, rsync does not like Plex and will give you a ton of "failed to set times" errors if you backup to a user share. However, it works fine if you backup directly to a disk share, which is why the script below references disk1 instead of my Backup user share.

 

Script:

 

#!/bin/bash

#Stop Docker Apps
docker stop CouchPotato
docker stop CrashPlan
docker stop Deluge
docker stop MariaDB
docker stop NzbDrone
docker stop NZBGet
docker stop PlexMediaServer
docker stop Syncthing

#Backup Docker App data via rsync
date >/var/log/docker_app_backup.log
date >/var/log/docker_img_backup.log
/usr/bin/rsync -avrtH --delete /mnt/disk/appdisk/appdata/ /mnt/disk1/Backup/appdata >>/var/log/docker_app_backup.log
/usr/bin/rsync -avrtH --delete /mnt/disk/appdisk/docker/ /mnt/disk1/Backup/docker >>/var/log/docker_img_backup.log

#Start Docker Apps
docker start CouchPotato
docker start CrashPlan
docker start Deluge
docker start MariaDB
docker start NzbDrone
docker start NZBGet
docker start PlexMediaServer
docker start Syncthing

 

Go file:

 

# Set up daily Docker App backup
cp /boot/custom/backup.sh /etc/cron.daily

 

This was pretty simple, but I hope someone else finds this useful.

 

Brilliant, will be using this, one quick question, is there anyway to specify when the job runs.  I'd like it to run at about 7.00am for example.

[mostlydave]

OK, I'm having some trouble with this,

 

I created the following script:

 

#!/bin/bash

#Stop Docker Apps
docker stop CouchPotato
docker stop HTPC-Manager
docker stop NzbDrone
docker stop PlexMediaServer
docker stop plexWatch
docker stop SABnzbd

#Backup Docker App data via rsync
date >/var/log/docker_app_backup.log
#date >/var/log/docker_img_backup.log
/usr/bin/rsync -avrtH --delete /mnt/user/appdata/ /mnt/user/backup/appdatabackup >>/var/log/docker_app_backup.log
#/usr/bin/rsync -avrtH --delete /mnt/cache/docker.img /mnt/user/backup/docker >>/var/log/docker_img_backup.log

#Start Docker Apps
docker start CouchPotato
docker start HTPC-Manager
docker start NzbDrone
docker start PlexMediaServer
docker start plexWatch
docker start SABnzbd

 

Saved as backup.sh in my custom folder, I'm using notepad Windows 8.1 save as backup.sh, save as type all files, encoding ANSI

 

I then run chmod 775 backup.sh

 

running ./backup.sh to test or just backup.sh gives me the following  "/bin/bash^M: bad interpreter: No such file or directory"

 

I'm totally new to Linux and I'm not sure what I'm doing wrong

[itimpi]

It sounds as if the file has windows style end-of-line sequences rather than Unix (Linux) style ones.    You need to create the file in an editor that understands Linux end-of-line sequences.  Alternatively you can put the file through the Linux dos2unix utility to convert the file to Linux style end-of-lines.

[trurl]

...Saved as backup.sh in my custom folder, I'm using notepad Windows 8.1 save as backup.sh, save as type all files, encoding ANSI

 

I then run chmod 775 backup.sh

 

running ./backup.sh to test or just backup.sh gives me the following  "/bin/bash^M: bad interpreter: No such file or directory"

 

I'm totally new to Linux and I'm not sure what I'm doing wrong

Never use notepad with files you want unRAID to understand. The ^M in your error message is typical of this problem.

 

You must use a linux compatible text editor. Search for Notepad++

 

[mostlydave]

I have recreated the .sh file with notepad++ but I'm still getting the same error, anyone have any ideas?

[trurl]

I have recreated the .sh file with notepad++ but I'm still getting the same error, anyone have any ideas?

[mostlydave]

thanks, that worked and my script ran, one more question, the script is saving log file to /var, where is that in unraid? I'm not seeing in on the root of my flash, do I nee to create this directory?

[itimpi]

thanks, that worked and my script ran, one more question, the script is saving log file to /var, where is that in unraid? I'm not seeing in on the root of my flash, do I nee to create this directory?

The /var directory is in RAM.

 

If you want the log file to survive a reboot, then you need to take action to have it copied to the flash drive when the system shuts down.

[mostlydave]

How do I browse to the var directory and view my logs?

[trurl]

How do I browse to the var directory and view my logs?

You can't get there over the network, at least not simply. You can "browse" from telnet or console with mc, or just use linux commands to get there.

[NAS]

This script can be simplified for just appdata:

 

#!/bin/bash

#Stop Docker Containers
docker stop $(docker ps -a -q)

#Backup Docker App data via rsync
date >>/var/log/docker_appdate_backup.log
/usr/bin/rsync -avrtH --delete /mnt/disk/appdisk/appdata/ /mnt/disk1/Backup/appdata >>/var/log/docker_appdate_backup.log

#Start Docker Containers
/etc/rc.d/rc.docker start

 

there is still quite a bit of room for improvement and the logging is kinda fudly but its a starter for 10

[kaiguy]

For my own curiosity and knowledge, is there a reason its preferable to use

docker stop $(docker ps -a -q)

over

/etc/rc.d/rc.docker stop

??

 

Thanks!

[BRiT]

For my own curiosity and knowledge, is there a reason its preferable to use

docker stop $(docker ps -a -q)

over

/etc/rc.d/rc.docker stop

??

 

Thanks!

 

The first only stops the docker containers, the second stops the containers and docker service.

[smdion]

When does cron.daily run on unRAID?

[gundamguy]

When does cron.daily run on unRAID?

 

I think it's set at 4:40 AM, but this can also be changed.

 

This script can be simplified for just appdata:

 

#!/bin/bash

#Stop Docker Containers
docker stop $(docker ps -a -q)

#Backup Docker App data via rsync
date >>/var/log/docker_appdate_backup.log
/usr/bin/rsync -avrtH --delete /mnt/disk/appdisk/appdata/ /mnt/disk1/Backup/appdata >>/var/log/docker_appdate_backup.log

#Start Docker Containers
/etc/rc.d/rc.docker start

 

there is still quite a bit of room for improvement and the logging is kinda fudly but its a starter for 10

 

Question about this, I see

docker stop $(docker ps -a -q)

is used to stop docker containers without stopping the docker service, but then

/etc/rc.d/rc.docker start

is used to start the docker service (and I assume containers) again, is there a command to just start docker containers without having to issue the start command to the docker service?

[smdion]

When does cron.daily run on unRAID?

 

I think it's set at 4:40 AM, but this can also be changed.

 

This script can be simplified for just appdata:

 

#!/bin/bash

#Stop Docker Containers
docker stop $(docker ps -a -q)

#Backup Docker App data via rsync
date >>/var/log/docker_appdate_backup.log
/usr/bin/rsync -avrtH --delete /mnt/disk/appdisk/appdata/ /mnt/disk1/Backup/appdata >>/var/log/docker_appdate_backup.log

#Start Docker Containers
/etc/rc.d/rc.docker start

 

there is still quite a bit of room for improvement and the logging is kinda fudly but its a starter for 10

 

Question about this, I see

docker stop $(docker ps -a -q)

is used to stop docker containers without stopping the docker service, but then

/etc/rc.d/rc.docker start

is used to start the docker service (and I assume containers) again, is there a command to just start docker containers without having to issue the start command to the docker service?

 

I think you can use the below code to stop/start the individual containers without starting the service.  If auto-start is turned on the containers will restart with the rc.docker start command

 

Nevermind, I think I figured it out. For anyone who is interested I used this script and set it to run daily in my Go file.

 

You will need to modify your directories accordingly. I have my appdata on a disk mounted with SNAP outside of the array at /mnt/disk/appdisk/appdata/. My docker img file is under /mnt/disk/appdisk/docker/

 

Also, rsync does not like Plex and will give you a ton of "failed to set times" errors if you backup to a user share. However, it works fine if you backup directly to a disk share, which is why the script below references disk1 instead of my Backup user share.

 

Script:

 

#!/bin/bash

#Stop Docker Apps
docker stop CouchPotato
docker stop CrashPlan
docker stop Deluge
docker stop MariaDB
docker stop NzbDrone
docker stop NZBGet
docker stop PlexMediaServer
docker stop Syncthing

#Backup Docker App data via rsync
date >/var/log/docker_app_backup.log
date >/var/log/docker_img_backup.log
/usr/bin/rsync -avrtH --delete /mnt/disk/appdisk/appdata/ /mnt/disk1/Backup/appdata >>/var/log/docker_app_backup.log
/usr/bin/rsync -avrtH --delete /mnt/disk/appdisk/docker/ /mnt/disk1/Backup/docker >>/var/log/docker_img_backup.log

#Start Docker Apps
docker start CouchPotato
docker start CrashPlan
docker start Deluge
docker start MariaDB
docker start NzbDrone
docker start NZBGet
docker start PlexMediaServer
docker start Syncthing

 

Go file:

 

# Set up daily Docker App backup
cp /boot/custom/backup.sh /etc/cron.daily

 

This was pretty simple, but I hope someone else finds this useful.

[gundamguy]

 

I think you can use the below code to stop/start the individual containers without starting the service.  If auto-start is turned on the containers will restart with the rc.docker start command

 

 

Thanks that is good info. It shows how to stop / start individual containers which might be relevant if there is a docker you don't want to stop while the back up is created.

 

My question is mostly this though, if our goal is to stop all the containers without stopping the docker service, why then restart the service isntead of just starting the containers.

 

I did a bit more research myself and found this list of Docker Command lines and I think I've got a few more ideas now.

 

1) Should we be using docker start $(docker ps -a -q) to restart ALL containers instead of /etc/rc.d/rc.docker start which does start all the containers that are set to auto-start but also restarts the docker service as well. Is this done because docker start $(docker ps -a -q) would start all containers even those that aren't set to auto-start? I guess I might need to understand the auto-start system better to figure out what this really should be...

 

2) Should we really be using docker pause $(docker ps -a -q) and docker unpause $(docker ps -a -q) to suspend the containers processes but not actually stop / restart them. Is this better in anyway, or are there drawbacks that I am not thinking of here?

 

Just trying to learn and therefor asking questions.

[damonq]

This should be a plugin. Just saying.

[ironicbadger]

This should be a plugin. Just saying.

 

Or a Docker...

[wickedathletes]

This has been extremely helpful on my end, and for anyone who didn't look up all the options rsync has one that is great, especially if you backup plex is --exclude-from. This lets you setup an exclude file, so for example, mine looks like:

 

#!/bin/bash

#Stop Docker Containers
docker stop $(docker ps -a -q)

#Backup Docker App data via rsync
date >>/mnt/user/Backup/HADES/docker_app_backup.log
/usr/bin/rsync -avrtH --exclude-from '/boot/custom/exclude.txt' --delete /mnt/cache/apps/ /mnt/user/Backup/HADES/cache/apps >>/mnt/user/Backup/HADES/docker_app_backup.log

#Start Docker Containers
/etc/rc.d/rc.docker start

#Backup Flash Drive data via rsync
/usr/bin/rsync -avrtH --delete /boot/ /mnt/user/Backup/HADES/flash >>/mnt/user/Backup/HADES/docker_app_backup.log

 

in my exclude.txt I have this:

 

Plex/Library/Application Support/Plex Media Server/Cache/Transcode

 

Because I really don't care about backing up my synced content, and its also like 60GB of data, who cares if you have to re-sync, as long as your DB is backed up etc etc.

 

*note, you don't need a file, you can just use --exclude but I figured I might have some more exclusions down the road.