[Plugin] CA Appdata Backup / Restore v2


Squid

Recommended Posts

Feature Request: @Squid

 

Now that we have multi Cache Pools, I have some Dockers App Data stored on other Cache Drives, I have 3 in total now.

 

Can we get support for Multi Cache Pools where Docker App Data is stored on other Cache Drives.

    Example: I have all my normal AppData stored on my Cache Drive, But then I also have a Plex_AppData stored on a seperate cache drive, and then Nextcloud stored on another seperate Cache Drive as NextCloud_AppData.

 

Then is there a way that when backing up dockers AppData, that only that docker (Or linked dockers, Example we link Nextcould to the database docker) are stopped and then started backup once they are backed up and then start the next docker, so only the docker that is currently being backed up is being stopped, others continue to run until its there time for backing up.

 

Then also make it so we can restore one docker(linked dockers), instead of all or nothing. And if were stopping and starting dockers like I mentioned, then the individual or if linked together are backed into individual backups. by name_date_time. 

 

Since some of use have so many dockers that one backup could be close to 1TB and if we need only one app restored it takes forever, and appdata is only growing.

 

This would allow our dockers to only be down for a short time, just enought for it to be back up and running, and then restore individual backups.

 

Also I would like the ability to back up sets of dockers in schedules,

   Example:

         Daily Backups Mark dockers for daily backups.

         Weekly Backups Mark dockers for weekly backups.

         Monthly Backups Mark dockers for weekly backups.

 

         and an option for backup up marked now!

 

As I don't need all my dockers to backup on the same schedule.  A few I need backed up on a daily basis, then most others done weekly, and a few only monthly.

 

 

My backups can take anywhere from 2-3 hours at times to back up and thats a long time for them all to be down.

 

I know its asking a lot but I feel it would be a huge benefit to the community. 

 

  • Like 4
Link to comment
  • 3 weeks later...

I have three identical Unraid servers.  Each uses syncthing to synchronize all data so that all three are identical.  I use appdata backup to keep all the docker containers backed up and, of course, those backups are synchronized to the other servers.

 

I've seen some requests here to change the single backup file into multiple backup files, one for each container.  I like this not only because it makes sense and would be easier to identify the backup and its app but syncthing doesn't handle large files nicely.  The bigger the file, the longer it takes.

 

https://forum.syncthing.net/t/poor-performance-on-big-file/8934/4

 

Quote

 

calmhJakob BorgSyncthing Maintainer

Jan '17

I don’t think tweaking settings will help you much. There’s a couple of things going on here.

The hasher is parallel and concurrent for multiple files, but any given file is hashed on a single thread. With your six core / twelve thread CPU and how Syncthing (and Windows) shows CPU usage, you should expect to see about 1/12 = 8.3% CPU usage while it’s doing this. 27 minutes is 75*1024/27/60 = 47 MB/s which is a tad on the slow side. I would expect your CPU to do about 150 MB/s or so of hashing so it’s a factor three or so off. The rest is probably overhead, read latency, etc., hard to say. We’re counting on the filesystem doing readahead for us, otherwise the cycle of read-hash-read-hash-etc incurs a couple of milliseconds for each block.

Syncing the file when it has changed goes like this, on the receiving side:

Create a temporary file

Read the previous version of the file and compute weak hashes of the blocks there.

For each block in the file that ought to be the same, copy it while also hashing it and verifying the hash. Uses the weak hashes to find blocks in the old version of the file, otherwise a database lookup to find matching blocks in other files locally.

Pull the blocks that we didn’t find locally from some other device and hash them, write them.

Rename the temporary to the final name.

During the copy phase you’ll see essentially no data flowing, just a few Kbps of index updates. The original file will be read twice; once for weak hashing, and once when copying. It’s too large to fit in disk cache, so this will cause a lot of disk access. Copying the blocks within the same disk will also cause quite a lot of seeks and so on so it’s not a super efficient thing to do for files as large as this.

TL;DR: Large files can be painful.

To find out what it’s doing, if you think it’s CPU or memory allocation related slowness, grab a profile 55. I’ll help interpret it.

 

 

Link to comment
52 minutes ago, TimTheSettler said:

I've seen some requests here to change the single backup file into multiple backup files, one for each container.  I like this not only because it makes sense and would be easier to identify the backup and its app but syncthing doesn't handle large files nicely.  The bigger the file, the longer it takes.

You might want to check out this thread-

https://forums.unraid.net/topic/81233-unraid-671-easy-way-to-backup-and-restore-specific-dockers/

Its what I’ve been using to split them up.

Link to comment

@Squid I need to extract ONE backup file for my Unifi controller that would be in the /mnt/cache/unifi folder from a backup created by this plugin. I keep getting an error when running the command below and even without all the extra tags and also just the folder name itself.  Are ALL my backups crap or am I missing something? I used one of the backups to restore my dockers that is NOT working for this command and I can't seem to extract that either. What am I missing? Does the plugin use another flag or otherwise know how to pull the data? The file is quite large as it includes my plex, radarr and sonarr info. 
 

tar -xzvf CA_backup.tar.gz --ignore-failed-read --exclude="Plex-Media-Server" --exclude="config" --exclude="binhex-radarr" --exclude="binhex-sonarr"

image.png.d6aadd1affc2e9659e10eb4e5b4d1bc8.png

Edited by dja
Link to comment
2 hours ago, FlyingTexan said:

I use the same USB that's already in the system with an OS or if I want to use a secondary USB

Why would you want to make a backup of the flash drive to the same flash drive????  The plugin is open ended and allows you to select any destination you want for any of the options.  Doesn't really help for disaster recovery to store the backup on what you're backing up on the same device

 

How would you change the naming of the various options?

 

Link to comment
15 hours ago, Squid said:

The options the plugin uses on a restore is -xvaf 

I'm still getting the error below...is my tar backup bad? 
 

Plex-Media-Server/Library/Application Support/Plex Media Server/Metadata/Albums/a/92fd63c59b1620e87a3bdc7cbc78c6ecd46f7e7.bundle/Contents/tv.plex.agents.music/posters/3901c63bbf7d6e1dba7fa5c8eef75b0fc2c11b4b
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

 

Link to comment

I've successfully run appdata backup in the past, but today when I launched a manual backup, it stopped the dockers, ran - but didn't back up anything - and then restarted the dockers. I confirmed the paths and made sure nothing was excluded.  My appdata folder is 400gb and takes several hours to run, usually. 

The status gave this message: "Backup/Restore Complete. tar Return Value: 0"

 

My unRAID server is otherwise working great. I am up-to-date for unRAID OS and CA Backup / Restore Appdata.

 

Any thoughts about how to troubleshoot this?

 

Link to comment
7 hours ago, dja said:

I'm still getting the error below...is my tar backup bad? 

What it looks like.  I don't have any other suggestions though.  I test my server on average about every 6 months with this stuff.

Link to comment

Long story short i had some problems with my cache drive, so now my appdata folder has been copied to the array and the cache drive is formatted. I am trying to restore the appdata from the disk back on to the cache but it says "No Backup Sets Found" which i assume is because the appdata folder has been copied and not created with the plugin. What do i need to do to be able to restore it? Do i have to create a backup with the tool using the appdata folder on the disk as source?

Link to comment
20 hours ago, Squid said:

What it looks like.  I don't have any other suggestions though.  I test my server on average about every 6 months with this stuff.

Thanks. I did a fresh backup with and without compression yesterday and was able to extract just fine. Weird. 

On the topic of extraction- is it possible to add an option to create tars for each docker vs. one giant one? With stuff like Plex, Radarr and Config folders it can be pretty intensive to work with a backup, esp. if you only need one thing. I know the intended use of this is to automate things, but there are probably more than a few folks who would appreciate the option(s). 

Thanks for the work on this plugin btw!!

Link to comment

Good Afternoon, 

 

I'd like to create a user script that monitors the duration of an appdata backup job, and notifies me if it crosses a certain runtime.. The reason is I recently had some server issues (corrupted cache drive) that (among other things) prevented dockers from responding correctly. I had an appdata backup job that was trying to run for over a week, but was hung due to the docker/cache drive issue. I was able to recover, but only from a very old appdata backup. Since the backup job technically never failed, I was not alerted. 

 

I've created a script to alert me if my appdata backups are stale, but I'd also like something to monitor my backup job runtime. Adding a runtime watchdog such as this will be part of my belt and suspenders approach to (hopefully) ensure that I never have a missing appdata backup again. 

 

 

I need some help with the "appdata backup job started 5 hours ago and is still running" logic for the script (see below) 

Where/how can I query the status of and runtime of a CA backup job? 

 

 

Here's what I'm envisioning - 

 

if [[ $(appdata backup job started 5 hours ago and is still running) ]]; 
then
     echo "CA Backup runtime is unhealthy" 
     /usr/local/emhttp/webGui/scripts/notify -s "CA Backup runtime is unhealthy" -i "alert" -m "This alert was triggered by the ca_backup_time_watchdog script. Backup runtime is unhealthy"  -d "A CA backup job was started over 5 hours ago and still has not completed. Check AppData backup for hung jobs or other issues."
else
     echo "Recent CA Backup runtime is healthy"
     /usr/local/emhttp/webGui/scripts/notify -s "CA Backup runtimes is healthy" -i "normal" -m "Backup runtimes look good" -d "CA AppData Backup runtime is/was less than 5 hours. All is well with the cosmos." 

fi

 

 

Any help is appreciated! 

Thank you

 

Link to comment

@Squid

As posted in the other thread(https://forums.unraid.net/topic/47266-plugin-ca-fix-common-problems/?do=findComment&comment=1086953),  I'm investigating cause of the permission/ownership change in my appdata folder.  I ran the backup again last night, confirming the ownership of my test folder have changed this morning again.  Is this is an expected behaviour after running a back up?

before 

drwxrwxrwx 1 root   root        0 Jan 19 10:38 test/
drwxrwxrwx 1 root   root        0 Jan 19 10:37 test_delete/
after

drwxrwxrwx 1 nobody users       0 Jan 19 10:38 test/
drwxrwxrwx 1 nobody users       0 Jan 19 10:37 test_delete/

Thanks.

 

Edited by LeoRX
added link to previous post.
Link to comment
3 hours ago, Squid said:

What are those folders? The source or the destination?  Destination permissions are irrelevant

they are source.  I created 2 empty folders in appdata to see if if the ownership gets changed overnight.

I just can't think of anything else that was schedule to run.  I'll recreate the folder and run backup again to confirm.

Link to comment

umm.. confirming the ownerships are being changed to nobody:users.  I ran stat ./test before and after the backup process.


root@Node:/mnt/user/appdata# root@Node:/mnt/user/appdata# stat ./test
  File: ./test
  Size: 0               Blocks: 0          IO Block: 4096   directory
Device: 0,52    Inode: 14073748865542626  Links: 1
Access: (0777/drwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-01-21 13:19:40.528879285 +1100
Modify: 2022-01-21 13:19:40.528879285 +1100
Change: 2022-01-21 13:19:40.528879285 +1100
 Birth: -
root@Node:/mnt/user/appdata# stat ./test
  File: ./test
  Size: 0               Blocks: 0          IO Block: 4096   directory
Device: 0,52    Inode: 14073748865542626  Links: 1
Access: (0777/drwxrwxrwx)  Uid: (   99/  nobody)   Gid: (  100/   users)
Access: 2022-01-21 13:19:40.528879285 +1100
Modify: 2022-01-21 13:19:40.528879285 +1100
Change: 2022-01-21 13:55:23.523670898 +1100
 Birth: -
 

Edited by LeoRX
Link to comment

I chmod 0777 the destinations for easy access via SMB.  Source I don't touch.

 

Mine don't change:  Here's my before and after.  The first root:root after andrew:users is a new directory I created, and it didn't change either modify time or permissions.  Are you referencing that test directory in a container anywhere?

image.thumb.png.cc802e9ce394aa6b99a61d304e921915.png

Link to comment

The test path is /mnt/user/appdata/test, under Unraid OS.

I just tested it again just to make sure I'm not crazy.
It is very strange no one noticed this behaviour.. which means its either something else that's causing it, something specific to my computer, settings or I'm just crazy. 

can you please confirm if your destination share path is default or customised path?  If its default, will you mind test it with a custopm path please?
2060582995_Screenshot2022-01-23110315.thumb.png.0e4242ca5abe00f490f5aebb445ebbed.png

Link to comment

There are no preset paths in the plugin.  They are all user entered (except for the source which is populated by what ever is in Docker Settings - Default appdata)

 

The way to test is that it is logged the exact command used to backup.  Stop all the containers, look at the permissions, run the command and look at the permissions after its complete.

 

Link to comment

umm.. I remember seeing the default destination path was something along the line of \mnt\user\CA**Appdata***.

What you've described is exactly how I tested.  I did in the following step.
1. create test folder

2. verify test folder is root

3. run backup using backup now button. wait for it to complete

4. check test folder ownership. and confirm it has changed to nobody.

 

1.thumb.png.8ef43c5f0417f39d3a21143765b49594.png2.thumb.png.b7966880117a18754f3fc6eb66bab813.png

 

in this case, test folder is zzz.  as you can see the owner was changed in a short amount of time, after running backup.


oh well.. i'm lost for word.. I understand if it can't be reproduced, it'll be difficult to reproduce.  I've migrated my mariadb and nextcloud container to use linuxserver.io so PUID and PGID can be set so it is not longer of an issue even with the ownership change.

Link to comment

So I think I did something absolutely stupid. I ran a backup after my cache drive seemed to have been dying it ran fine but I had changed the backup path to something new as to not over write the last one completed back on 1/17. Well now doing so and replacing my cache drive with a new one I noticed that the restore appdata tab shows that there are no backup sets found. Please tell me there is a way to restore from my previously completed backup to this new cache drive. 

Thanks

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.