[Plugin] CA Appdata Backup / Restore v2

[GHunter]

 

@unRaide  I use this plugin and there is the ability to include running a "Custom Stop Script" which I use to do a few backups of additional things. It works out really well for me.

[Squid]

9 hours ago, unRaide said:

This might be a dumb question but I couldn’t find out how to search for it correctly in this thread 😁

 

Is anyone using this plugin to backup random non-appdata folders on their cache? Seems like it could be as easy as changing the source to mnt/cache and then using the "Excluded Folders" to deselect those I don't want to backup? Basically trying to backup the appdata folder and a few others that are only on the cache.

 

Everywhere in the plugin it refers specifically to the appdata folder so I wanted to make sure there isn’t anything specific to just that use case before trying to set this up.

Nothing says that you can't do that

[unRaide]

1 hour ago, GHunter said:

 

@unRaide  I use this plugin and there is the ability to include running a "Custom Stop Script" which I use to do a few backups of additional things. It works out really well for me.

That sounds interesting, any chance you could share the script you're using for that.  

[unRaide]

15 hours ago, Squid said:

Nothing says that you can't do that

Hi Squid, I’m trying to set this up but I’m getting the error msg below. Am I doing something wrong or is it not possible to backup all folders on my cache drive minus a few excluded ones?

 

 

 

[Squid]

2 hours ago, unRaide said:

Hi Squid, I’m trying to set this up but I’m getting the error msg below. Am I doing something wrong or is it not possible to backup all folders on my cache drive minus a few excluded ones?

 

 

 

Ok.  Looks like I lied.  Must've had a reason that I set the plug-in like that, but can't think of why right now. 🤔

[bonienl]

Don't you need "appdata" in the source path?

[trurl]

2 hours ago, bonienl said:

Don't you need "appdata" in the source path?

He was trying to adapt it to backup the whole cache

3 hours ago, Squid said:

Must've had a reason that I set the plug-in like that, but can't think of why right now.

Maybe to prevent overwriting the source

[Squid]

31 minutes ago, trurl said:

Maybe to prevent overwriting the source

I *think* I have that test in there due to what happens on a restore.  A fail-safe because when restoring, any excess files from the source are deleted.  (and also to prevent a source and destination overlap)

Edited January 30, 2019 by Squid

[GHunter]

@unRaide My script is highly customized for my use case (it does file deletes, renaming, ignores some folders, backup rotation and, well, poorly written for general use LOL) so I'd rather not share it, however I can edit it and add some additional commenting so it would be easier to digest and you can run with it from there. I'll let you make the call though.

 

Do you want me to post it as is or edit it a bit to just do some example backups?

 

Gary

[unRaide]

2 hours ago, GHunter said:

@unRaide My script is highly customized for my use case (it does file deletes, renaming, ignores some folders, backup rotation and, well, poorly written for general use LOL) so I'd rather not share it, however I can edit it and add some additional commenting so it would be easier to digest and you can run with it from there. I'll let you make the call though.

 

Do you want me to post it as is or edit it a bit to just do some example backups?

 

Gary

Hey @GHunter, I'm quite the noob so any editing you could do would much appreciated by me and thothethat follow I'm sure.

 

That said im grateful for any assistance 😁 

[unRaide]

20 hours ago, Squid said:

Ok.  Looks like I lied.  Must've had a reason that I set the plug-in like that, but can't think of why right now. 🤔

Ah, ok. Would be cool to include such functionality into the plugin as a one stop onsite backup shop. 

[GHunter]

Posting my custom backup script for @unRaide and anyone else that wants to use it. I've been using it for months now and it works great.

 

Note: I only use the Appdata backup function of this plugin and have set backup rotation to off as it is handled by this script.

 

What it does: This script will backup your unRaid USB flash drive, your VM XML files, and OVMF files. Backup rotation is set to 3 days in the script and can be changed. Any dated backup folder will not be deleted if text is appended to the folder name. Example: this folder will not be deleted. 2019-01-22@02.30 Last backup of v6.6.6 before upgrade

 

CA_Backup_Custom_Stop.sh file is attached at the bottom of this post.

 

READ THE COMMENTS AND USE AT YOUR OWN RISK!! Comments are welcome. I'm still learning so be gentle! LOL There's always room for improvement.

#!/bin/bash
# CA_Backup_Custom_Stop.sh

# Change the locations below to match your backup locations
flashbackuplocation="/mnt/user/Backups-Gary/unRAID/Flash/"
nvrambackuplocation="/mnt/user/Backups-Gary/unRAID/NVRAM/"
vmxmlbackuplocation="/mnt/user/Backups-Gary/unRAID/VM_XML/"
appdatabackuplocation="/mnt/user/Backups-Gary/unRAID/Appdata/"

daysofbackupstokeep=3
# 1440 minutes in 1 day 7 days is 10080
(( daystominutes = daysofbackupstokeep * 1440 - 1 ))

# dont change anything below here
echo "..."
echo "Running CA Backup Custom Stop script"
echo "..."
datestamp=""`date '+%F'`"@"`date '+%H'`"."`date '+%M'`
dir="$flashbackuplocation"/"$datestamp"

# make a subfolder to store backup files
if [ ! -d $dir ] ; then
 
echo "Making a subfolder for todays date $datestamp"

# make the directory as it doesnt exist
mkdir -vp $dir
else
echo "As $dir exists continuing."
fi

# save the files using rsync and mod permissions
echo "Saving USB Flash drive files"
rsync -a --no-o /boot/* $dir/
chmod -R 777 $dir
echo "Finished backup of USB Flash drive files"

# deleting backup folders older than X number of days as declared at the top
echo "Deleting USB Flash Drive backups older than $daysofbackupstokeep days except those the folders that have"
echo "text appended to it"

# don't delete folders that have text appened to it. Useful to save older backups. Example: this folder will not be deleted 2019-01-22@02.30 Last backup of v6.6.6 before upgrade
find $flashbackuplocation -type d -maxdepth 1 -mindepth 1 -mmin +$daystominutes -not -name '?????????????????*' | xargs rm -rf
echo "..."
sleep 2

dir="$nvrambackuplocation"/"$datestamp"

# make a subfolder to store backup files
if [ ! -d $dir ] ; then
 
echo "Making a subfolder for todays date $datestamp"

# make the directory as it doesnt exist
mkdir -vp $dir
else
echo "As $dir exists continuing."
fi

# save the files using rsync and mod permissions
echo "Saving OVMF NVRAM files"
rsync -a --no-o /etc/libvirt/qemu/nvram/* $dir/
chmod -R 777 $dir
echo "Finished backup of OVMF NVRAM files"

# deleting backup folders older than X number of days as declared at the top
echo "Deleting NVRAM backups older than $daysofbackupstokeep days except those the folders that have text"
echo "appended to it"

# don't delete folders that have text appened to it. Useful to save older backups. Example: this folder will not be deleted 2019-01-22@02.30 Last backup of v6.6.6 before upgrade
find $nvrambackuplocation -type d -maxdepth 1 -mindepth 1 -mmin +$daystominutes -not -name '?????????????????*' | xargs rm -rf
echo "..."

dir="$vmxmlbackuplocation"/"$datestamp"

# make a subfolder to store backup files
if [ ! -d $dir ] ; then
 
echo "Making a subfolder for todays date $datestamp"

# make the directory as it doesnt exist
mkdir -vp $dir
else
echo "As $dir exists continuing."
fi

# save the files using rsync and mod permissions
echo "Saving VM XML files"
rsync -a --no-o /etc/libvirt/qemu/*xml $dir/
chmod -R 777 $dir
echo "Finished backup of VM XML files"

# rename xml files to *.txt as it's easier to open with a text editor
# this fuction is commented out but if you want to use it then uncomment the following 4 lines
# echo "Renaming all the XML files to TXT"
# cd $dir/
# rename .xml .txt *.xml
# echo "Finished renaming the XML files"

# deleting backup folders older than X number of days as declared at the top
echo "Deleting VM XML backups older than $daysofbackupstokeep days except those the folders that have"
echo "text appended to it"

# don't delete folders that have text appened to it. Useful to save older backups. Example: this folder will not be deleted 2019-01-22@02.30 Last backup of v6.6.6 before upgrade
find $vmxmlbackuplocation -type d -maxdepth 1 -mindepth 1 -mmin +$daystominutes -not -name '?????????????????*' | xargs rm -rf
echo "..."

# lets check and delete older backups from Appdata too for consistency that was performed by the plugin. Note: I disabled backup rotation of appdata folder in the plugin as it is handled by this script
echo "Deleting AppData backups older than $daysofbackupstokeep days except those the folders that have"
echo "text appended to it"

# don't delete folders that have text appened to it. Useful to save older backups. Example: this folder will not be deleted 2019-01-22@02.30 Last backup of v6.6.6 before upgrade
find $appdatabackuplocation -type d -maxdepth 1 -mindepth 1 -mmin +$daystominutes -not -name '?????????????????*' | xargs rm -rf
echo "..."

echo "Finished running CA Backup Custom Stop script"
echo "..."
exit

 

Enjoy, Gary

CA_Backup_Custom_Stop.sh

Edited January 31, 2019 by GHunter
attached file

[Squid]

7 hours ago, unRaide said:

Ah, ok. Would be cool to include such functionality into the plugin as a one stop onsite backup shop. 

Thought about a generic backup plugin, and ultimately decided that something like Duplicati is better for that.

[unRaide]

21 hours ago, GHunter said:

Posting my custom backup script for @unRaide and anyone else that wants to use it. I've been using it for months now and it works great.

 

Note: I only use the Appdata backup function of this plugin and have set backup rotation to off as it is handled by this script.

 

What it does: This script will backup your unRaid USB flash drive, your VM XML files, and OVMF files. Backup rotation is set to 3 days in the script and can be changed. Any dated backup folder will not be deleted if text is appended to the folder name. Example: this folder will not be deleted. 2019-01-22@02.30 Last backup of v6.6.6 before upgrade

 

CA_Backup_Custom_Stop.sh file is attached at the bottom of this post.

 

READ THE COMMENTS AND USE AT YOUR OWN RISK!! Comments are welcome. I'm still learning so be gentle! LOL There's always room for improvement.

#!/bin/bash
# CA_Backup_Custom_Stop.sh

# Change the locations below to match your backup locations
flashbackuplocation="/mnt/user/Backups-Gary/unRAID/Flash/"
nvrambackuplocation="/mnt/user/Backups-Gary/unRAID/NVRAM/"
vmxmlbackuplocation="/mnt/user/Backups-Gary/unRAID/VM_XML/"
appdatabackuplocation="/mnt/user/Backups-Gary/unRAID/Appdata/"

daysofbackupstokeep=3
# 1440 minutes in 1 day 7 days is 10080
(( daystominutes = daysofbackupstokeep * 1440 - 1 ))

# dont change anything below here
echo "..."
echo "Running CA Backup Custom Stop script"
echo "..."
datestamp=""`date '+%F'`"@"`date '+%H'`"."`date '+%M'`
dir="$flashbackuplocation"/"$datestamp"

# make a subfolder to store backup files
if [ ! -d $dir ] ; then
 
echo "Making a subfolder for todays date $datestamp"

# make the directory as it doesnt exist
mkdir -vp $dir
else
echo "As $dir exists continuing."
fi

# save the files using rsync and mod permissions
echo "Saving USB Flash drive files"
rsync -a --no-o /boot/* $dir/
chmod -R 777 $dir
echo "Finished backup of USB Flash drive files"

# deleting backup folders older than X number of days as declared at the top
echo "Deleting USB Flash Drive backups older than $daysofbackupstokeep days except those the folders that have"
echo "text appended to it"

# don't delete folders that have text appened to it. Useful to save older backups. Example: this folder will not be deleted 2019-01-22@02.30 Last backup of v6.6.6 before upgrade
find $flashbackuplocation -type d -maxdepth 1 -mindepth 1 -mmin +$daystominutes -not -name '?????????????????*' | xargs rm -rf
echo "..."
sleep 2

dir="$nvrambackuplocation"/"$datestamp"

# make a subfolder to store backup files
if [ ! -d $dir ] ; then
 
echo "Making a subfolder for todays date $datestamp"

# make the directory as it doesnt exist
mkdir -vp $dir
else
echo "As $dir exists continuing."
fi

# save the files using rsync and mod permissions
echo "Saving OVMF NVRAM files"
rsync -a --no-o /etc/libvirt/qemu/nvram/* $dir/
chmod -R 777 $dir
echo "Finished backup of OVMF NVRAM files"

# deleting backup folders older than X number of days as declared at the top
echo "Deleting NVRAM backups older than $daysofbackupstokeep days except those the folders that have text"
echo "appended to it"

# don't delete folders that have text appened to it. Useful to save older backups. Example: this folder will not be deleted 2019-01-22@02.30 Last backup of v6.6.6 before upgrade
find $nvrambackuplocation -type d -maxdepth 1 -mindepth 1 -mmin +$daystominutes -not -name '?????????????????*' | xargs rm -rf
echo "..."

dir="$vmxmlbackuplocation"/"$datestamp"

# make a subfolder to store backup files
if [ ! -d $dir ] ; then
 
echo "Making a subfolder for todays date $datestamp"

# make the directory as it doesnt exist
mkdir -vp $dir
else
echo "As $dir exists continuing."
fi

# save the files using rsync and mod permissions
echo "Saving VM XML files"
rsync -a --no-o /etc/libvirt/qemu/*xml $dir/
chmod -R 777 $dir
echo "Finished backup of VM XML files"

# rename xml files to *.txt as it's easier to open with a text editor
# this fuction is commented out but if you want to use it then uncomment the following 4 lines
# echo "Renaming all the XML files to TXT"
# cd $dir/
# rename .xml .txt *.xml
# echo "Finished renaming the XML files"

# deleting backup folders older than X number of days as declared at the top
echo "Deleting VM XML backups older than $daysofbackupstokeep days except those the folders that have"
echo "text appended to it"

# don't delete folders that have text appened to it. Useful to save older backups. Example: this folder will not be deleted 2019-01-22@02.30 Last backup of v6.6.6 before upgrade
find $vmxmlbackuplocation -type d -maxdepth 1 -mindepth 1 -mmin +$daystominutes -not -name '?????????????????*' | xargs rm -rf
echo "..."

# lets check and delete older backups from Appdata too for consistency that was performed by the plugin. Note: I disabled backup rotation of appdata folder in the plugin as it is handled by this script
echo "Deleting AppData backups older than $daysofbackupstokeep days except those the folders that have"
echo "text appended to it"

# don't delete folders that have text appened to it. Useful to save older backups. Example: this folder will not be deleted 2019-01-22@02.30 Last backup of v6.6.6 before upgrade
find $appdatabackuplocation -type d -maxdepth 1 -mindepth 1 -mmin +$daystominutes -not -name '?????????????????*' | xargs rm -rf
echo "..."

echo "Finished running CA Backup Custom Stop script"
echo "..."
exit

 

Enjoy, Gary

CA_Backup_Custom_Stop.sh

This is great Gary, thanks!!,

 

Really well commented and laid out. I’ll start playing around with it this weekend. 

[unRaide]

15 hours ago, Squid said:

Thought about a generic backup plugin, and ultimately decided that something like Duplicati is better for that.

Fair enough. I looked at duplicati briefly but seemed like a mixed bag from what I read.

 

I really liked the idea of using a more straight fwd plugin integrated into Unraid with just enough of a UI. 😁

[Stupifier]

Posting to promote/upvote a feature-request:

Would be nice to have the Appdata Backup separate the tar into individual tars for each app or primary folder instead of one gigantic tar.

[Diggewuff]

1 hour ago, Stupifier said:

Posting to promote/upvote a feature-request:

Would be nice to have the Appdata Backup separate the tar into individual tars for each app or primary folder instead of one gigantic tar.

I would like to upvote that too. A feature to backup the folders sequentially would also be nice. In that way small services can be started again and not all of them have to be down just to wait for the large Plex database do finish backing up.

[nuhll]

Yeah, i would like that as a option too.

 

But more important i also would like to have a set of (e.g.) 10 last backups from dockers and flash. Not only from plugins.

 

Like

plugins 14.02.19.tar

plugins 15.02.19.tar

plugins 16.02.19.tar

 

dockers 14.02.19.tar

dockers 15.02.19.tar

dockers 16.02.19.tar

 

flash 14.02.19.tar

flash 15.02.19.tar

flash 16.02.19.tar

Edited February 16, 2019 by nuhll

[ieronymous]

   Ok after a long read of more pro questions a noobicon one coming up but its essential to me and however i write it I can t seem to find a straight answer. At least maybe it will be useful for someone having the same (understanding probably) problem like me.

   So, I m in CA Backup / Restore Appdata Configuration settings and need to browse to the path of appdata share. So question follows...

is mnt/user/appdata stored in flash drive or the array? because I have 2 options (having the same outcome?) either select /mnt/user/appdata or mnt/disk1/appdata (why i find same folder different locations?) I dont use a cache drive and inside flash only files i found about appdata are the .cfg ones (for configuration)

 

   Thank you (please dont point me to a book in which I ll find an answer...i need to set it up at last,,,,been several days of endless searching for every single option of why and how  to do it)

Edited February 21, 2019 by ieronymous

[JonathanM]

11 minutes ago, ieronymous said:

because I have 2 options (having the same outcome?) either select /mnt/user/appdata or mnt/disk1/appdata (why i find same folder different locations?)

It IS the same folder. User shares are a combined view of all the root folders in all the disks. /mnt/user/appdata shows the contents of /mnt/disk1/appdata , /mnt/disk2/appdata , etc. That way you can have a user share that spans multiple physical disks. I recommend ignoring /mnt/diskX and /mnt/cache locations until you get a little more experience, only work with /mnt/user.

[ieronymous]

11 minutes ago, jonathanm said:

It IS the same folder. User shares are a combined view of all the root folders in all the disks. /mnt/user/appdata shows the contents of /mnt/disk1/appdata , /mnt/disk2/appdata , etc. That way you can have a user share that spans multiple physical disks. I recommend ignoring /mnt/diskX and /mnt/cache locations until you get a little more experience, only work with /mnt/user.

....which is exactly what I did but wanted to make sure. i got confused by spaceinvader's video because  when he had his mnt/cache/appdata path typed he had choices to exclude folders in the  Excluded Folders: setting (folders showing app) but in my /mnt/user/appdata was empty (which seemed to me pretty weird - probably because I have no dockers installed yet only plug ins)

 

Thank you very much for the clarification !!!!!

[fc0712]

I have now backed up my appdata and USB to an external usb hard drive using unassigned devices. 

 

Would i then be able to just copy the back from the external usb hard drive to another device for extra safety or should be done through the plugin?

[ebnerjoh]

Hi,

 

I am using this script now for long time to backup my Appdata as well as my docker-image and flash-drive.

 

On Saturday I had an issue with my Home Assistant installation. By mistake I have removed it from my Dockers so I though, no problem - I have a backup.

 

I went to the Backup/Restore tool, took the backup from the last day (doing every week a backup) and restored it. BIG MISTAKE. 

 

I know it was for sure my mistake, but I am not sure, if the backup/restore couldnt be better.

 

So what happened:

 

1) I am using a Nextcloud Docker for syncing files

2) The restore didnt allow me to select only e.g. home-assistant, I had to restore everything

3) I am 95% sure I took the backup from the day before, but the restore was done on the oldest available backup (end of Jan). But yes, here I could have made an mistake and select the wrong file

 

After restoring the backup file from more then one month I had big issues with my Nextcloud Installation. So I disconnected Nextcloud and tried to recover the Nextcloud-Server. I had no luck and at the end I had to reinstall the nextcloud server.

 

What I didnt expected was, that Nextcloud removed the newer files from my clients after I restored the backup. So I lost all files for one month.

I had a second time no luck, because normaly I am doing also an offline Backup of my files on a monthly basis, but this was also one month ago...

[trurl]

1 hour ago, ebnerjoh said:

I am using this script now for long time to backup ... my docker-image...

I don't see an option to backup docker image, wouldn't expect there to be, and there is no good reason to do so, since your dockers can just be reinstalled exactly as before using the Previous Apps feature on the Apps page. In fact, we often recommend deleting docker image and reinstalling them when people have certain issues with their dockers.

[trurl]

I can only assume the rest of your post had to do with restoring an older version of your appdata, which was probably a mistake since that certainly wouldn't be required to reinstall that docker you removed.