[Plugin] CA Appdata Backup / Restore v2

[KnifeFed]

On 6/22/2021 at 1:43 AM, xrqp said:

If I use this app, CA Appdata Backup, then is there any reason to backup Sonarr (using the backup feature inside Sonarr) ?  I run Sonarr with a Unraid container.

 

Not really, unless you want to be able to restore a backup zip from the Sonarr GUI.

[lotekjunky]

On 6/16/2021 at 2:11 PM, dingd0ng said:

Is there any way to check how much space each docker is taking up in the completed backup?

After excluding Plex from the backup my backups went from 50 GB down to 5 GB, but now they have ballooned to over 30 GB, and I can't figure out why. I haven't installed any new dockers, so I can't see the reason for the increase. 

Open a shell and run this:

 

du -ah --max-depth=1 /mnt/user/appdata | sort -hr

 

It will tally up the consumed space from the provided directory, in a human readable format and then sort it. It might take some time to run... my appdata is huge and it took 3 minutes

[pika]

On 6/15/2021 at 8:59 AM, Stubbs said:

Pretty sure I'd be more comfortable backing my USB Drive locally than sending my data elsewhere.

exactly my thoughts...

would re-implementing USB backup cause problems?

[Squid]

As stated, it's not being removed.

[jj_uk]

The plugin stops all dockers, then begins the backup. I'm assuming this is done so there's a complete snapshot at that point in time?

 

My problem is the backup is now taking 4 hrs to complete. I was wondering if it would be possible to stop a container, back it up, then start it, move on to the next. If that's a good idea or not I don't know? 

 

Or maybe stop docker, make incremental backup (fast), start docker, then recreate the tar from the incremental and delete the incremental?

 

There's not really a good time to stop docker for 4 hrs. It takes down my smart home. If one of the kids get up in the night during the backup, the lights don't work. 

 

I can't think of a solution?

Edited June 25, 2021 by jj_uk

[acosmichippo]

Hi all, I have a small suggestion.  I just noticed that my backups have not been running for a while, and after some investigating, i realized that when i changed the name of my appdata cache around the 6.9.0 upgrade, I forgot to update the "appdata share (source)" file path in the backu plugin.  Apparently the plugin doesn't notice a wrong/missing share path as an error state, so the backups were "completing" successfully even though it was never actually doing any backups.

 

So my suggestion would be to add a new error state for a "missing" appdata share.  This way people who use the "errors only" notification setting will see if they have a typo or outdated file path in the "appdata share (source)" setting.

 

Thanks for reading and thanks for making this plugin!

Edited July 1, 2021 by acosmichippo

[tinyaim]

I have an issue where I was playing with the network settings of a docker container.  It was set to a custom "docker" network and was changed to "host". The upshot of that was the server seems to be working but the Web GUI just hangs.  I have the USB boot disk backed up to a remote server using this plugin.  I have 2 questions.

 

1) will restoring the boot disk back prior to this change resolve the issue or not ?

2) if 1) is true what is the best way to restore that remote folder backup back to the USB boot disk.

 

I suspect the answer to 1 is no but currently I have a crippled Unrad server that is driving me nuts to get back working properly.  I am also struggling to get the diagnostics zip file off the server.  I have tried sftp which just hangs but is there any other way to get this off the server so that I can interrogate properly.

 

thanks

 

 

Edited July 2, 2021 by tinyaim

[tinyaim]

15 hours ago, tinyaim said:

I have an issue where I was playing with the network settings of a docker container.  It was set to a custom "docker" network and was changed to "host". The upshot of that was the server seems to be working but the Web GUI just hangs.  I have the USB boot disk backed up to a remote server using this plugin.  I have 2 questions.

 

1) will restoring the boot disk back prior to this change resolve the issue or not ?

2) if 1) is true what is the best way to restore that remote folder backup back to the USB boot disk.

 

I suspect the answer to 1 is no but currently I have a crippled Unrad server that is driving me nuts to get back working properly.  I am also struggling to get the diagnostics zip file off the server.  I have tried sftp which just hangs but is there any other way to get this off the server so that I can interrogate properly.

 

thanks

 

 

I have an update.  Not totally sure I did regarding the Docker networking but it certainly interfered with the web hosting for the Unraid web UI.  To recover the UI I had to SSH into the server and initially I ran docker commands to stop all the running containers.  To be safe I completely removed the offending container that I changed to cause this to go wrong.  Once I had docker stopped the UI started working.  To stop getting into issues I have reviewed the Docker networks and reverted back to the the standard Bridge and host networks.  I am now in the process of carefully bringing back each of the Docker containers.

[s449]

What's the format for excluding folders with spaces in the path? And multiple paths? I'm trying to exclude:

• /mnt/user/appdata/binhex-plex/Plex Media Server/Cache
• /mnt/user/appdata/binhex-plex/Plex Media Server/Media

Currently I have this:

"/mnt/user/appdata/binhex-plex/Plex Media Server/Cache","/mnt/user/appdata/binhex-plex/Plex Media Server/Media"

Is that the correct format? Also it doesn't look like you can use spaces in the field. I can only copy and paste space.

[jj_uk]

4 hours ago, s449 said:

What's the format for excluding folders with spaces in the path? And multiple paths? I'm trying to exclude:

• /mnt/user/appdata/binhex-plex/Plex Media Server/Cache
• /mnt/user/appdata/binhex-plex/Plex Media Server/Media

Currently I have this:

"/mnt/user/appdata/binhex-plex/Plex Media Server/Cache","/mnt/user/appdata/binhex-plex/Plex Media Server/Media"

Is that the correct format? Also it doesn't look like you can use spaces in the field. I can only copy and paste space.

 

Did you get this working? I'm thinking of doing the same. Plex database is just too large for backups.

 

[KnifeFed]

8 hours ago, s449 said:

What's the format for excluding folders with spaces in the path? And multiple paths? I'm trying to exclude:

• /mnt/user/appdata/binhex-plex/Plex Media Server/Cache
• /mnt/user/appdata/binhex-plex/Plex Media Server/Media

Currently I have this:

"/mnt/user/appdata/binhex-plex/Plex Media Server/Cache","/mnt/user/appdata/binhex-plex/Plex Media Server/Media"

Is that the correct format? Also it doesn't look like you can use spaces in the field. I can only copy and paste space.

 

4 hours ago, jj_uk said:

 

Did you get this working? I'm thinking of doing the same. Plex database is just too large for backups.

 

 

I've got the paths listed without quotes, like so:

binhex-plexpass/Plex Media Server/Crash Reports/,binhex-plexpass/Plex Media Server/Logs/,binhex-plexpass/Plex Media Server/Media/,binhex-plexpass/Plex Media Server/Metadata/

They get quoted during runtime so it works fine.

 

You can also do exclusions like:

.cache,.caches,.DS_Store,*.log,*.log.*,*.tmp,cache,Cache,caches,Caches,.Recycle.Bin

 

[s449]

On 7/7/2021 at 1:27 AM, jj_uk said:

 

Did you get this working? I'm thinking of doing the same. Plex database is just too large for backups.

 

 

Yes I did finally! Thanks to this:

 

On 7/7/2021 at 7:18 AM, KnifeFed said:

 

 

I've got the paths listed without quotes, like so:

binhex-plexpass/Plex Media Server/Crash Reports/,binhex-plexpass/Plex Media Server/Logs/,binhex-plexpass/Plex Media Server/Media/,binhex-plexpass/Plex Media Server/Metadata/

They get quoted during runtime so it works fine.

 

You can also do exclusions like:

.cache,.caches,.DS_Store,*.log,*.log.*,*.tmp,cache,Cache,caches,Caches,.Recycle.Bin

 

 

Looks like you don't need the /mnt/user/ part of the path. Ended up adding some exclusions and my full excluded folders path is this:

binhex-plex/Plex Media Server/Crash Reports/,binhex-plex/Plex Media Server/Logs/,binhex-plex/Plex Media Server/Media/,binhex-plex/Plex Media Server/Cache/,.DS_Store,*.log,*.log.*,*.tmp,.Recycle.Bin

 

Adding that made my appdata backup go from 4 hours to 15 minutes, and 149 GB (compressed) to 11.1 GB (compressed).

[jj_uk]

It appears that the backup looks for a match in the path, rather than an absolute path. Maybe the developer could confirm this?

 

I've added this, which should be the same for all plex dockers regardless of maintainer. My backup runs at 2:30am, so we'll see tomorrow what the results are.

 

Plex Media Server/Cache/,Plex Media Server/Crash Reports/,Plex Media Server/Logs/,Plex Media Server/Media/,.DS_Store,*.log,*.log.*,*.tmp,.Recycle.Bin

 

Edited July 9, 2021 by jj_uk

[benflux]

Recently, my docker containers have not restarted after the backup has finished, any tips?

[Squid]

Assuming the backup finished (it would be logged), then no.  The odd person has reported that over the years.  If I could replicate it, I'd fix it.  It's never failed for me

[benflux]

it could have been that the backup hadn't finished, I had a panic and restarted them. I've just kicked it off again without plex to see if that was the issue

[benflux]

it completed in 5 mins this time, i'll run with this and live without Plex 🙂

[abra8c]

I have a second appdata location. Is there any way to backup it too?

[CyberMew]

I am getting this warning:

 

Quote

Deprecated plugin ca.backup.plgThis plugin has been deprecate and should no longer be used due to the following reason(s): While fully functional, development has ceased on this plugin and been replaced by v2. See HERE While this plugin should still be functional, it is no recommended to continue to use it.

 

How can I migrate over to v2 without loss to my old backups and retain the settings and still being able to be restored? What is the correct upgrade steps? I think currently the /mnt/user/appdata/ is being saved to a /mnt/user/cabackup folder, basically copied pasted the appdata folder together with some extra files and dumped into a dated folder in this cabackup i made last time.

Edited July 30, 2021 by CyberMew

[Squid]

There is no upgrade path.  Uninstall, install v2 (which uses a tarball instead of individual files) and run a backup.

[Krzaku]

Does this plugin handle stopping dockers differently from unraid gui? I have an issue with Deluge, where when it is started by the backup plugin it is started in some bugged state where the web gui is broken. I have to restart it manually from unraid gui. Stopping/starting manually using unraid gui does not cause this issue. Any ideas what might be causing this?

Edited July 31, 2021 by Krzaku

[Squid]

it issues docker stop / docker start.  The command line versions of what the GUI does.  Only real reason I can think of is if you've got custom networking (Deluge is running through another container) because the plugin does not honor the startup order currently

[CyberMew]

On 7/31/2021 at 6:06 AM, Squid said:

There is no upgrade path.  Uninstall, install v2 (which uses a tarball instead of individual files) and run a backup.

Thanks. One thing I would note is that v1 settings is not ported over so luckily I took a screenshot of existing settings beforehand. Also noticed the compression option is off by default and I have turned it on, hoping to use less space.

 

One more question, if I don't use unraid.net (My Servers), can I use the same destination folder for USB Backup Destination and libvirt.img Destination like how it was previously? In v1 the destination folder was the same for all and it created a folder named Community_Applications_USB_Backup and 

Community_Applications_VM_XML_Backup in the e.g. CommunityApplicationsAppdataBackup/2021-07-20@05.00 folder.

[Squid]

There's no change in how the flash and libvirt backups are done.  Appdata should be to a separate share / folder.

[ullibelgie]

May I get some advice please ?

I am a new user of unraid and have started with two dockers:

- Paperless ng (already containing about 300 documents)

- Calibri with Calibri web (to read books via the home network on my tablet - about 200documents and some really large pdf files)

 

Now I wonder how to best backup the application data folder and the databases of these applications.

 

I have no cache drives in the server (just starting) - only the array itself (3x HDD plus 1 parity). I am therefore not sure if the Backup-restore plug-in is the right way for the backup. It seems that it is made a nice backup for people using cache drives (SSD) for speed up their dockers/VM and which are concerned of the the data outside their array.

So may be I should do the backups in my case different ?

 

Best, I would build a backup scheme following the 3-2-1 rule, to do the things right from beginning.

(3 copies, 2 different media and 1 different location)

 

The Calibri database seem to be stored outside of application data folder, but in the media share (where also all the e-books are). I was thinking to backup this folder with the script of Marc Gutt. (user script plug-in.... great explanation Marc in the German forum!)

The question is however remaining, how to backup the application folder ?!

 

And is it enough in case of fatal crash ? How about the dockers themselves ?!

 

Sorry for my stupid questions and thanks for chance to learn from experts.