[Plugin] CA Appdata Backup / Restore v2


Squid

Recommended Posts

securityIcon-WithShadow.png

 

CA Appdata Backup / Restore (v2)

 

Due to some fundamental problems with XFS / BTRFS, the original version of Appdata Backup / Restore became unworkable and caused lockups for many users.  Development has ceased on the original version, and is now switched over to this replacement.

 

Fundamentally they are both the same and accomplish the same goals (namely, backing up your Appdata share and USB / libvirt), but this version is significantly faster at the job. 

 

This version uses tar instead of rsync (and offers optional compression of the archive - roughly 50% if not including any downloads in the archive - which you really shouldn't be anyways). 

 

Because of using tar, there is no longer any incremental backups.  Rather, every backup goes into its own separate dated subfolder.  Old backups can optionally be deleted after a successful backup.  Even without incremental backups, the speed increase afforded by tar means that there should be no real difference in the end.  (ie:  A full backup using my appdata on the old plugin takes ~1.5 hours.  This plugin can do the same thing uncompressed in about 10 minutes, and compressed in 20 minutes.  The optional verification of the archive takes a similar amount of time.  An incremental backup on the old plugin using my appdata averaged around 35 minutes).

 

The options for separate destination for USB / VM libvirt backups is changed so that if there is no destination set for those backup's then they will not be backed up.

 

Additionally, unlike the original plugin, no cache drive is necessary, and the appdata source can be stored on any device in your system (ie: unassigned devices).  The destination as usual can go to any mount point within your system.  Unfortunately because of no more incremental backups, this version may no longer be suitable for ultimately backing up offsite to a cloud service (ie: via rclone)

 

You can find this within the Apps tab (Search for Appdata Backup).

 

The original v1 plugin should be uninstalled if migrating to this version.

 

 

Edited by Squid
  • Like 10
  • Thanks 2
  • Upvote 1
Link to comment
18 minutes ago, shooga said:

Anything we should know about switching to the new version? It is as simple as:

  1. Uninstall the old plugin
  2. Install the new plugin
  3. Point the new plugin at the same destinations

For me it was. I took a screenshot of the old settings but didn’t really need it. Settings for v2 haven’t changed too much.

Link to comment

The settings haven't really changed much.  Its IMHO a bit more simplified.

 

It always is set to be dated backups.  Default setting for # of days to keep is 1 so that on the next run it will delete the previous backups (but set to whatever you want)

 

Flash / libvirt backups you always have to choose where to put them.  If left blank, then no backup is done.

 

The biggest thing (if its a problem for you) is deleting the old backups if they were causing your system problems before.  I found the midnight commander was the best way to do it (one app's config at a time), as it winds up deleting each file individually instead of all in one shot.

  • Upvote 1
Link to comment

Thanks for this great plugin. Vill try it some more and buy you a beer later on :-)

 

I have a request :-)

 

  • Set up a possibility to exclude a specific sub folder within the specific dockers appdata folder. For example, I would like to exclude the "CACHE" folder from Plex docker, but ensure I have everything else backed up.

Thank You!

  • Haha 1
  • Upvote 1
Link to comment
2 minutes ago, hobbis said:

Thanks for this great plugin. Vill try it some more and buy you a beer later on :-)

 

I have a request :-)

 

  • Set up a possibility to exclude a specific sub folder within the specific dockers appdata folder. For example, I would like to exclude the "CACHE" folder from Plex docker, but ensure I have everything else backed up.

Thank You!

Type it in manually like plex/cache  comma separated 

  • Upvote 1
Link to comment

First let me say that I'm not complaining, I appreciate all the great work you do for the Unraid community, this is just an FYI. For my system, this version is much slower than the prior. V1 used to take about 3 minutes to run a nightly incremental backup whereas V2 now takes about 30 minutes to do a full, uncompressed, backup of the same data. My data doesn't change very much from day-to-day so an incremental was ideal.

 

Having just seen the post above on how to exclude the Plex cache folder, I'll try that and see how much it helps.

 

 

Edited by Lignumaqua
  • Like 1
Link to comment
6 hours ago, Lignumaqua said:

First let me say that I'm not complaining, I appreciate all the great work you do for the Unraid community, this is just an FYI. For my system, this version is much slower than the prior. V1 used to take about 3 minutes to run a nightly incremental backup whereas V2 now takes about 30 minutes to do a full, uncompressed, backup of the same data. My data doesn't change very much from day-to-day so an incremental was ideal.

 

Having just seen the post above on how to exclude the Plex cache folder, I'll try that and see how much it helps.

 

 

If you have no problems with v1, then continue to use it.  Once I managed to replicate the problems people were having though, I had to drop support for the rsync system, and rather than force everyone to switch to tars I elected to do a new plugin for it.

 

Speed:  So long as you are stopping all of the applications, and not keeping anything running via the advanced settings, you should be safe to turn off verification.  

 

During testing, I experimented with updating changed files in place, and deleting old files no longer present, but there was more or less no change in speed due to the fact that you have to run through the archive twice to accomplish this.  So for simplicity sake it was easier to always create a new archive.

Link to comment

I’ve had it running fine for a couple of days now.  A bit of tweaking and updating the excluded paths and I’m down to a 3gb tar which Is very quick to complete. 

 

One that doesn’t seem to be working is the flash backup.  I have the path in there but no Tar to speak of. Any idea where I can look?

Link to comment

I've been very happy with this update. Much faster than before, no weird system hangs if it runs when other stuff is happening - but it also helps that I took the time to exclude a bunch of cache and thumbnail files from my Plex backup. Still need to test restoring from it, just to be sure I didn't exclude *too much*.

Link to comment

Thank you so much for this!

 

I'm a bit on the edge though as when doing a test backup as my whole server froze again when V2 tried to delete the old V1 backup folders BUT it's actually deleting them.

At least it deleted the first folder so far. I'll update this post if/when the other folder gets removed.

 

Hooray! It did it! :D

Edited by SelfSD
It completed!
Link to comment

Why does all data disks in array spin up, when backup is done? For now I have backup location set to unassigned disk and when backup is running, all data disks in array and disk with backup location (on unassigned device) spin up. Parity disk and other unassigned disks remain inactive.

Link to comment
Why does all data disks in array spin up, when backup is done? For now I have backup location set to unassigned disk and when backup is running, all data disks in array and disk with backup location (on unassigned device) spin up. Parity disk and other unassigned disks remain inactive.
IDK maybe because all the containers are being stopped. Something with your setup
Link to comment

I suspect EmbyServer docker. Since my media shares spread across all 3 data disk, it probably does media scan/update when it is turned on again and spins up all data disks.

 

Request: can you add option to choose number of backups that are kept in backup location before they are deleted? Option "Delete backups if they are this many days old" is a little "dangerous", because it can delete all backups if server was off for a long time. Not long ago my server was off for almost 7 months, but luckily I remembered to turn off option for deleting old backups or all backups would be deleted.

Link to comment
On 11/7/2017 at 11:29 PM, Lignumaqua said:

First let me say that I'm not complaining, I appreciate all the great work you do for the Unraid community, this is just an FYI. For my system, this version is much slower than the prior. V1 used to take about 3 minutes to run a nightly incremental backup whereas V2 now takes about 30 minutes to do a full, uncompressed, backup of the same data. My data doesn't change very much from day-to-day so an incremental was ideal.

 

I second the request for incrementals. My cache drive is 676GB in total, with the bulk of that being:

 

476 GB in "/mnt/user/.appdata/Plex/Library/Application Support/Plex Media Server/Media/"

106 GB in "/mnt/user/.appdata/Plex/Library/Application Support/Plex Media Server/Metadata/"

 

Sure, I could exclude those during my regularly scheduled backup, dropping my backup to only 94GB but, since I have the space, I'd rather keep them backed up so that if I do lose the cache drive, Plex won't have to download/transcode/whatever all of that again. That's over half my monthly data cap on data that's pretty consistent, hence the v1 plug-in approach of renaming the oldest and then just updating the diff worked really well. Still took several hours to run, though, making me nervous about how long this new method is going to take.

 

I did indeed have cache drive failure earlier this year and your v1 plug-n saved my bacon! Unfortunately, though, it's now been throwing errors so each backup has "-error" at the end and I can't figure out the reason. I saw this thread about v2, figured I was one of the ones affected and went ahead and upgraded. First backup scheduled to run tomorrow night, so fingers crossed!

 

Overall, though, to be clear - *thank you* for providing this plug-in! It's a life saver!

 

 

Edited by Flick
Link to comment

For me, this was started at 4am today and now just verifying. I thought it should only take 10-20min? What am I doing wrong? This is my first back up.. @Squid

 

On 11/6/2017 at 3:41 PM, Squid said:

Type it in manually like plex/cache  comma separated 

That can add up though when you have more than 10 dockers running. Specially when each docker config folder has more than one sub folder. Would be great to have the option within the UI 

Edited by jrdnlc
Link to comment
On 11/12/2017 at 6:50 PM, Flick said:

First backup scheduled to run tomorrow night, so fingers crossed!

 

Backup ran for 7 hours and 11 minutes. Verify and compression both turned off. TAR is 617GB in size. So, for doing a full backup, I am indeed impressed. I don't mind not having compression turned on but not having verify makes me a wee bit nervous. That would double the time, though, so I guess it's a trade-off.

 

Now if I could just figure out why it insists on creating a new folder called "Backup" on my disk10 when it completes the backup job. Sigh. (My backups go to disk1\Backups).

 

Thank you again for all your work on this.

Link to comment
  • Squid locked this topic
Guest
This topic is now closed to further replies.