Process to copy "appdata" docker folder form cache to array for backup


Recommended Posts

  • Replies 72
  • Created
  • Last Reply

Top Posters In This Topic

Lot of posts regarding developing this into a plugin. Is it available yet? Google isn't showing anything much besides this thread.

 

I've not made any progress on this, becuase life... hope someone else is working on this.

 

It's a thing I'd really like to see.

I would too. Guess for now I'll have to see if I can just get a basic backup for appdata working. Is there a command I can use to have logs written/overwritten to a txt file on the flash instead of ram? Would make it a lot easier to see if the backup is running properly.

Link to comment

Lot of posts regarding developing this into a plugin. Is it available yet? Google isn't showing anything much besides this thread.

 

I've not made any progress on this, becuase life... hope someone else is working on this.

 

It's a thing I'd really like to see.

I would too. Guess for now I'll have to see if I can just get a basic backup for appdata working. Is there a command I can use to have logs written/overwritten to a txt file on the flash instead of ram? Would make it a lot easier to see if the backup is running properly.

Just edit the rsync line in the script to use a different path for the log. Using the script NAS posted in this thread, for example:

 

Change

/usr/bin/rsync -avrtH --delete /mnt/disk/appdisk/appdata/ /mnt/disk1/Backup/appdata >>/var/log/docker_appdata_backup.log

to

/usr/bin/rsync -avrtH --delete /mnt/disk/appdisk/appdata/ /mnt/disk1/Backup/appdata >>/boot/log/docker_appdata_backup.log

would append the log to the file /boot/log/docker_appdata_backup.log

 

And

/usr/bin/rsync -avrtH --delete /mnt/disk/appdisk/appdata/ /mnt/disk1/Backup/appdata >/boot/log/docker_appdata_backup.log

would overwrite that file instead.

Link to comment

Here's another vote to hopefully have someone develop a plugin for this feature.  In addition to dockers (both app data folder AND docker.img file)  I would also like to see the plugin have the ability to support VMs as well.  They have essentially the same issue where the VM needs to be stopped in order to back up the .img file and then automatically restarted.

 

I know virtually nothing about linux and need this kind of service to be part of the unRAID GUI.  I've seen posts as well as youtube videos state that the direction lime is moving in is to simplify the usability of unRAID and make it more of a mainstream product by putting more and more functionality into the GUI... this would be a great place to start!

Link to comment
  • 2 weeks later...

Just to throw my method in for this, I use rsnapshot to do nightly backups. I like this method because it'll keep version information in case I want to revert to an old version of some configuration.

 

I'm really interested in rsnapshot on unraid. Can you point me in the right direction to install this?

 

Sorry for my delayed response. You'll need the rsnapshot and the perl packages. I got them from here:

http://www.slackware.com/~alien/slackbuilds/rsnapshot/pkg/13.37/rsnapshot-1.3.1-i486-1alien.tgz

http://slackware.cs.utah.edu/pub/slackware/slackware64-current/slackware64/d/perl-5.18.1-x86_64-1.txz

 

I saved them to /boot/packages and added these lines to my go script:

installpkg /boot/packages/perl-5.18.1-x86_64-1.txz
installpkg /boot/packages/rsnapshot-1.3.1-i486-1alien.tgz

 

I then edited my rsnapshot.conf file (you can find many tutorials online for setting up rsnapshot). I saved the rsnapshot.conf file to my flash drive and added in a line to my go script to copy this file to where rsnapshot wants it to be (/etc/rsnapshot.conf).

cp /boot/rsnapshot/rsnapshot.conf /etc/rsnapshot.conf

 

I also created a crontab file and saved it to my flash drive. There's another line in my go script to copy its contents to the root crontab file.

cat /boot/scripts/crontab >> /var/spool/cron/crontabs/root

 

I hope this helps you out. I'm happy to help if you run into snags.

Link to comment

Just to throw my method in for this, I use rsnapshot to do nightly backups. I like this method because it'll keep version information in case I want to revert to an old version of some configuration.

 

I'm really interested in rsnapshot on unraid. Can you point me in the right direction to install this?

 

Sorry for my delayed response. You'll need the rsnapshot and the perl packages. I got them from here:

http://www.slackware.com/~alien/slackbuilds/rsnapshot/pkg/13.37/rsnapshot-1.3.1-i486-1alien.tgz

http://slackware.cs.utah.edu/pub/slackware/slackware64-current/slackware64/d/perl-5.18.1-x86_64-1.txz

 

I saved them to /boot/packages and added these lines to my go script:

installpkg /boot/packages/perl-5.18.1-x86_64-1.txz
installpkg /boot/packages/rsnapshot-1.3.1-i486-1alien.tgz

 

I then edited my rsnapshot.conf file (you can find many tutorials online for setting up rsnapshot). I saved the rsnapshot.conf file to my flash drive and added in a line to my go script to copy this file to where rsnapshot wants it to be (/etc/rsnapshot.conf).

cp /boot/rsnapshot/rsnapshot.conf /etc/rsnapshot.conf

 

I also created a crontab file and saved it to my flash drive. There's another line in my go script to copy its contents to the root crontab file.

cat /boot/scripts/crontab >> /var/spool/cron/crontabs/root

 

I hope this helps you out. I'm happy to help if you run into snags.

Shouldn't that be the 64-bit rsnapshot?

http://www.slackware.com/~alien/slackbuilds/rsnapshot/pkg64/13.37/rsnapshot-1.3.1-x86_64-1alien.tgz

Link to comment
  • 2 weeks later...

I suppose that would work too. If I understand it correctly, rsnapshot is really just a script and not a binary, which would explain why it works fine on my 64-bit machine.

 

After seeing this topic i try to backup my cache drive with rsnapshot. But i m not sure it is incremental or not.

 

rsnapshot du says that all backup folders are 914MB, i run df -k before and after the backup free disk space also decreases 914MB after backup. From windows i also get the same results.

root@Tower:/mnt/disk2/Depo/Backups/rsnapshots# rsnapshot du
914M    /mnt/user/Depo/Backups/rsnapshots/weekly.0/
914M    /mnt/user/Depo/Backups/rsnapshots/weekly.1/
914M    /mnt/user/Depo/Backups/rsnapshots/weekly.2/
914M    /mnt/user/Depo/Backups/rsnapshots/weekly.3/
914M    /mnt/user/Depo/Backups/rsnapshots/monthly.2/
4.5G    total

 

if i use rsnapshot diff it looks like it is incremental if i dont get it wrong.

root@Tower:/mnt/disk2/Depo/Backups/rsnapshots# rsnapshot diff
Comparing /mnt/user/Depo/Backups/rsnapshots/weekly.1 to /mnt/user/Depo/Backups/rsnapshots/weekly.0
Between /mnt/user/Depo/Backups/rsnapshots/weekly.1 and /mnt/user/Depo/Backups/rsnapshots/weekly.0:
  6051 were added, taking 943975850 bytes;
  5956 were removed, saving 943522993 bytes;

 

does it eating up space or not?

i google it for some time, seems like hardlinks cause this misrepoting. What happens when i run out of disk space? will unraid throw an error or can i continue to write on that disk?

or am i missing smtg?

Link to comment

There's a couple of things to keep in mind when setting up rsnapshot. Make sure that you specify that it stores the data on a specific disk, not on the array. This is because rsnapshot uses hard links, which does not work on the /mnt/user fuse file system.

 

Also, since rsnapshot uses hard links, the data appears to make copies of the original data, but it's actually linking to the same place on the disk. Go into where your rsnapshot backups are saved. Run this command to find the total amount of disk usage:

 

du -sh /mnt/user/Depo/Backups/rsnapshots/

 

That'll sum up all of your rsnapshot backups and you should be able to easily tell if it gives you a reasonable number.

Link to comment

I take that back. rsnapshot du should give you the actual disk usage. If you're seeing 914M on each, something isn't working correctly. This is what mine looks like:

 

25G     /mnt/disk3/rsnapshot/daily.0/
813M    /mnt/disk3/rsnapshot/daily.1/
1.2G    /mnt/disk3/rsnapshot/daily.2/
613M    /mnt/disk3/rsnapshot/daily.3/
1.9G    /mnt/disk3/rsnapshot/daily.4/
29G     total

 

I'm guessing your problem is with the hardlinks and the fuse file system. Try switching your config to point to a specific disk. It can be a cache disk or an array disk.

Link to comment

I'm guessing your problem is with the hardlinks and the fuse file system. Try switching your config to point to a specific disk. It can be a cache disk or an array disk.

 

Thanks Bungy you rock. Changing it to disk share definately solves the issue.

I will share my files soon to help others.

Link to comment

I'm guessing your problem is with the hardlinks and the fuse file system. Try switching your config to point to a specific disk. It can be a cache disk or an array disk.

 

Thanks Bungy you rock. Changing it to disk share definately solves the issue.

I will share my files soon to help others.

Awesome! I can knock that off my list. I was going to out together a docker for rsnapshot and maybe a gui to make configuration easier. Documentation works would be the first step with a docker being useful, but maybe overkill.

Link to comment

Awesome! I can knock that off my list. I was going to out together a docker for rsnapshot and maybe a gui to make configuration easier. Documentation works would be the first step with a docker being useful, but maybe overkill.

 

I think a docker and some documentation is a great idea. My knowledge level for command line stuff is pretty limited and I can use all the help I can get. Am I understanding correctly that rsnapshot can do versioned backups so that prior backups are saved? If that's true I would definately use the docker.

Link to comment

Thanks for Bungy and others for everything about rsnapshot. I finally get it work properly. I do some steps sightly different and want to share. (i definately need to work more on sending stats to pushover)

 

Download needed packages:

 

I saved them to /boot/extra so you dont need to add any line to go file for these, they will be installed automaticly on startup.

to manually install:

installpkg /boot/extra/perl-5.18.1-x86_64-1.txz
installpkg /boot/extra/rsnapshot-1.3.1-i486-1alien.tgz

 

/boot/custom/rsnapshot/rsnapshot.conf file:

You may need to change snapshot_root, backup and interval lines. (uncomment hourly and daily if you need)

!!!Dont forget snapshot_root must be a disk share not user share!!!

!!!Dont use space in this file use tab!!!

config_version	1.2
snapshot_root	/mnt/disk2/Depo/Backups/rsnapshots/
cmd_cp		/usr/bin/cp
cmd_rm		/usr/bin/rm
cmd_rsync	/usr/bin/rsync
cmd_logger	/usr/bin/logger
cmd_du		/usr/bin/du
cmd_preexec	/boot/custom/rsnapshot/preexec_script.sh
cmd_postexec	/boot/custom/rsnapshot/postexec_script.sh
#interval	hourly	6
#interval	daily	5
interval	weekly	4
interval	monthly	3
verbose		2
loglevel	3
lockfile	/var/run/rsnapshot.pid
rsync_long_args	--stats
du_args	-csh
exclude_file	/boot/custom/rsnapshot/exclude.txt
link_dest	1
backup	/mnt/cache/No_move/docker/	docker/

 

/boot/custom/rsnapshot/preexec_script.sh file:

Dont forget to write your pushover token & key

#!/bin/bash
#send pushover notification
START_TIME=$(date +"%H:%M:%S")

curl -k https://api.pushover.net/1/messages.json -F token="YOUR APP TOKEN" -F user="YOUR USER KEY" -F title="Backup Started" -F message="Backup started at $START_TIME"
echo "Pushover notification send." | logger -t rsnapshot

#Stop Docker Containers
docker stop $(docker ps -q)

 

/boot/custom/rsnapshot/postexec_script.sh file:

#!/bin/bash
#Start Docker Containers
/etc/rc.d/rc.docker start

 

/boot/custom/rsnapshot/exclude.txt file:

Transmission/downloads

 

Instead of caling rsnapshot directly i wrote a script to:

  • copy conf file before running snapshot incase of any changes after reboot

  • get the stdout & stderr to a file to send that with pushover (as i said needs more work to format the message properly)

Dont forget to write your pushover token & key

/boot/custom/rsnapshot/rsnap.sh file:

#!/bin/bash
INTERVAL=$1
NOW=$(date +"%Y%m%d_%H")

#copy conf file
cp /boot/custom/rsnapshot/rsnapshot.conf /etc/rsnapshot.conf

#take snapshot
rsnapshot $INTERVAL &> /boot/custom/rsnapshot/logs/$NOW.txt

#send pushover notification
FINISH_TIME=$(date +"%H:%M:%S")
MESSAGE=$(cat /boot/custom/rsnapshot/logs/$NOW.txt)
curl -k https://api.pushover.net/1/messages.json -F token="YOUR APP TOKEN" -F user="YOUR USER KEY" -F title="Backup Complete" -F message="$INTERVAL backup completed at $FINISH_TIME
$MESSAGE"

 

/boot/custom/rsnapshot/rsnap_cron file:

(uncomment hourly and daily if you need)

#
# rsnapshot hourly
#0 */4 * * * /boot/custom/rsnapshot/rsnap.sh hourly
#
# rsnapshot daily
#20 3 * * * /boot/custom/rsnapshot/rsnap.sh daily
#
# rsnapshot weekly
0 3 * * 0 /boot/custom/rsnapshot/rsnap.sh weekly
#
# rsnapshot monthly
40 2 1 * * /boot/custom/rsnapshot/rsnap.sh monthly

 

add these lines to go file:

# rsnapshot
cp /boot/custom/rsnapshot/rsnapshot.conf /etc/rsnapshot.conf
cat /boot/custom/rsnapshot/rsnap_cron >> /var/spool/cron/crontabs/root

 

At the end you will have these 6 files on your /boot/custom/rsnapshot folder

root@Tower:/boot/custom/rsnapshot# ls
exclude.txt*  postexec_script.sh*  rsnap.sh*
preexec_script.sh*   rsnap_cron*  rsnapshot.conf*

 

Now you can take backups manually with commands below or wait for the cron time.

/boot/custom/rsnapshot/rsnap.sh hourly
/boot/custom/rsnapshot/rsnap.sh daily
/boot/custom/rsnapshot/rsnap.sh weekly
/boot/custom/rsnapshot/rsnap.sh monthly

 

some extra commands that you may need to look at:

rsnapshot configtest
rsnapshot -t [interval]
rsnapshot du

 

some links you may need to look at:

https://github.com/rsnapshot/rsnapshot

http://rsnapshot.org/faq.html

https://raw.githubusercontent.com/rsnapshot/rsnapshot/master/docs/HOWTOs/rsnapshot-HOWTO.en.html (right click, save target as)

Link to comment

FYI: I found the WebSync docker in sparklyballs' beta repository and am using it for my backups. Was easier to deal with for me than setting up cron jobs with rsync manually. I have one backup scheduled weekly and one monthly for some redundancy.

 

I'm only backing up my app data so I'm not stopping docker. My understanding is that files that change after rsync has started will not be synced, which I'm ok with. Can get them the next time.

 

EDIT: After more reading it looks like this may not be a good method for backing up app data. Based on what I've read, if a file is being written while rsync is running then rsync will simply copy what is there at that instant. So if an application is in the middle of a write then there's a chance the copy will be corrupt. This may be a very rare situation, but given that this is meant to be a backup it needs to be reliable.

Link to comment

Well going to look into Rsnapshot tonight, but if all it's doing is installing a script and not no binaries then I'm not really in favor of installing it like a binary package. I might take some time to look at how the script is working and maybe reverse engineer it into a plugin. This would likely work as a docker, but it's the kind of functionality that would make sense to be a plugin as well since it's not installing binaries.

 

Link to comment
  • 6 months later...
  • 4 weeks later...

Now that I've moved everything to the cache drive (for speed) I want to backup everything on the array too. This approach looks solid if you want to spend the time to get it working, an alternative but probably more process intense is just to use CrashPlan.

Link to comment
  • 4 weeks later...

  • I made some changes to run two different backup conf files with different interval settings.
  • made a config file for user variables. I think i came the half way for a plugin but i feel too lazy for that :)
  • and i tidy up the notifications.

I honestly feel this would be a great plugin to have. I recently completely lost my cache drive and the idea of resetting everything up just makes my stomach hurt.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.