[Plugin] CA Appdata Backup / Restore v2

[mortenmoulder]

17 hours ago, Squid said:

Advanced Settings.

 

Reason why stopping is is the default is because assume this scenario

 

Docker X is accessing files 1 & 2 concurrently.  It decides to write changes to both files (and the changes are dependent upon each other)

In the middle of all the happening, a backup is also proceding.  Because of timing, say the original version of file 1 gets backed up, but the replacement version of file 2 gets backed up.

 

Now on a restore those 2 files are out of sync with each other.  Will it cause problems?  Impossible to say for sure.  The only sure fire way to do any backup / restore is to ensure that all processes that can potentially write to those files are stopped.  

 

What are the odds of it happening?  Probably small, but the defaults are set to stop the containers for the safest backup.  Whether you choose to override it or not is entirely up to you.

 

Very well written. I gave it a go and deselected a few containers that I know need to run 24/7. Took a backup this morning at 6 AM and it worked perfectly.

 

Thank you! And wow it's fast!

[Squid]

3 hours ago, TRusselo said:

how do I restore USB drive with the backups I made with this app?

do I restore appdata backup to restore USB data too?

Procedure all really depends upon where you were backing it up to....

 

If backing up to the array, then you have to create a new USB key, assign all of the drives as data drives, then you'll have access to the share you stored the backup on.  Copy it over the the new flash and reboot and the system will be back and you'll simply have to transfer your registration key.

 

It's not pretty to do that (far easier to restore from a place you stored outside of the array), but this plugin was doing backups (automated) for years prior to any solution from LT.  I do recommend that you look at the flash backup system available when you install the MyServers plugin from LT

[TRusselo]

yes last night i got it back up with a new USB and a trial key.

had to re-assign all my drives.  parity rebuilding now......
 

and yes my backup was on the array.... lesson learned.

so now, just copy/paste flash backup to flash drive and boot?  

[TRusselo]

yup. copy/paste to usb drive, with the aid of another pc.

 

boot up,  re-assign drives,  input new key file link,  parity rebuilding again...  

 

Happy camper.  VM's, dockers, home assistant, cameras, all back online.

Edited August 9, 2021 by TRusselo

[TRusselo]

so through the USB flash fail / rebuild,  I updated to 6.9.2 from 6.7.2

 

hmmm whats unraid.net login....?  login

hmmm whats myserver plugin....?  install

hmmm wha....  awwww.... fffffffff........  so flash backup.... eh?   ...enable....

 

(feels like a moron for not updating unraid sooner)

and i LOVE updating things...  dunno why i put off undraid update for months.....

Edited August 11, 2021 by TRusselo

[Groto]

Hi,

 

Is there a way to restore just 1 specific docker in case I screw something up? I can't see a scenario where I would want to restore everything unless the drive dies. I would think a specific docker restore would be a use case that would get used much more.

 

Thanks!

[mkayze]

How do you backup 2 cache pools with this plugin? Anyone have a solution?

[Squid]

4 hours ago, mkayze said:

How do you backup 2 cache pools with this plugin? Anyone have a solution?

The plugin only supports a single appdata share

[paaland]

Needed my first backup today since one of my SSD's on my cache pool died.

Apparently it does not backup symbolic links properly. All my linked files were restored as a 0 bytes large normal file.

Big problem

[Squid]

It damn well should...  I've never had a problem with it and symlinks.  That's the whole point of the backup being valid, and was tested.

[paaland]

56 minutes ago, paaland said:

Needed my first backup today since one of my SSD's on my cache pool died.

Apparently it does not backup symbolic links properly. All my linked files were restored as a 0 bytes large normal file.

Big problem

 

I stopped the array and reformatted the cache, then did a new restore, this time linked files worked. So it must have been some issue with restoring on top of existing files or something like that.

[Madman2012]

Is there a way to partially restore app data, ie select the apps I want to restore? 

 

I ask because I ran into trouble with cache filling up and crashed all the dockers. When that happened I reconfigured the cache and share config and now the cache drives are separate pools based on IO function, like VMs and Docker instead of one giant 4tb BTRFS pool per SpaceInvaders latest 6.9 configuration video. The issue is the app file restore data file now saturates the docker cache pool as it shrunk from one 4tb pool to 3 separate drives, 2x 1tb NVME and 1x 2tb SSD. 

 

Help! 

 

Past my depth right now.

 

Thank you in advance! 

 

 

[mgutt]

@Squid

I think I found something which could be interesting for your appdata backup plugin:

 

If the docker service is stopped and the user install your plugin, it creates the following in /var/lib/docker:

 

Not sure how important those files are for your plugin, but they aren't present anymore after the docker service has been started (as the mount /var/lib/docker > /mnt/user/system/docker/docker "overwrites" the path).

 

Now I hope you can help me as well:

I'm working on my own plugin to create a RAM disk for /var/lib/docker/containers. By that this happens after the docker service has been started:

• unraid mounts /mnt/user/system/docker/docker to /var/lib/docker
• my enhancement creates a backup of /var/lib/docker/containers
• then it creates a ram disk through tmpfs of /var/lib/docker/containers
• then it restores the backup into the ram disk
• all containers start as usual, but this time they write logs/json files to the RAM instead of the SSD
• the cache SSDs are sleeping until the backup schedule is executed

This works perfectly, except your plugin creates /var/lib/docker/unraid while the docker service wasn't running. This time, after enabling the docker service, everything seems to work as usual: Unraid mounts the docker path, my enhancement creates the ram disk..., but now the cache SSDs are not sleeping. Everything which is written to /var/lib/docker/containers, is written to the cache SSD's /mnt/cache/system/docker/docker/containers?! I really don't understand why this is happening. Do you?

 

 

[Squid]

It's all actually correct.  dockerMan itself creates those icons because the plugin uses them and if the image isn't mounted it creates it in memory.  The same thing that the logs for the plugin are also stored in memory if docker isn't mounted.  By and large, this is a edge case because it's doubtful that anyone will ever run the script without docker running, but it has to handle that situation.

[datun1c0rn]

Hi there,

 

is there a way to edit the config file for the plugin by hand? I want prevent some containers to be powered down that are spawned by a custom script.
Another question would be if there is a way to backup the plugin settings?

[Squid]

The plugin's settings are backed up by it's USB backup (and the MyServers flash backup feature)

 

Nothing's stopping you from editing the config files, so long as you maintain the proper json formatting

[Cessquill]

On 8/24/2021 at 10:49 PM, Squid said:

The plugin only supports a single appdata share

Is there any way around this?   I need to move Plex onto its own pool, but still need it backed up.

 

Or might there be a different setup that puts Plex onto its own drive but still in the same pool?

[Squid]

It logs the command it uses.  Just run a user script and modify to suit.

[3dee]

On 5/16/2021 at 2:49 AM, Richamc01 said:

 

Did you ever get an answer to this? I just ran into this same issue when trying to disable the docker service.  

 

  

On 6/7/2020 at 8:32 PM, zzgus said:

Any idea why I get this error:

Array Started•Warning: posix_kill() expects parameter 1 to be int, string given in /usr/local/emhttp/plugins/ca.backup2/scripts/killRsync.php on line 27

Thankyou

 

 

@zzgus @Richamc01 I have this issue right now, how did you solve it? I don't want to kill my docker containers if possible. The array doesn't stop, I can't disable docker and "docker stop $(docker ps -a -q)" does not work either..

 

 

Edit: I can't even do

 

halt

shutdown -h now

reboot

su -c poweroff

 

Edited September 18, 2021 by 3dee

[KnifeFed]

On 4/27/2021 at 7:02 AM, AwesomeAustn said:

 

I did install a docker manually on accident while trying to run the initialize command for one. I'll have to figure out how to delete it. Thanks for your answer and reply.

 

Edit: It doesn't look like it's installed, but still shows up in CA Appdata Backup.

 

I ran:

docker rm dazzling_panini
Error: No such container: dazzling_panini

 

docker rmi dazzling_panini
Error: No such image: dazzling_panini

Was there ever a fix for this?

I had some garbage containers created due to partially failed `docker build` commands. Performing a complete `docker system prune --force --all --volumes` got rid of them everywhere except from CA Appdata Backup/Restore and CA Application Auto Update.

[danioj]

Hey @Squid hope you're doing ok man!

 

I just - randomly - went to the CA AppData Backup / Restore Settings Page and saw this:

 

"NOTE: USB Backup is deprecated on Unraid version 6.9.0 It is advised to use the Unraid.net plugin instead"

 

I kind of get this, given unRAID's own new functionality. However, I would prefer to keep using your functionality instead. I assume (given the settings are still there and the backup keeps working each night) that this feature isn't truly depreciated.

 

I hope you plan on keeping things going!! A local backup is desirable to me and your plugin makes it very easy.

[JonathanM]

1 hour ago, danioj said:

I hope you plan on keeping things going!! A local backup is desirable to me and your plugin makes it very easy.

Read from here.

https://forums.unraid.net/topic/61211-plugin-ca-appdata-backup-restore-v2/?do=findComment&comment=961160

 

[3dee]

On 9/18/2021 at 2:07 PM, 3dee said:

 

  

 

 

@zzgus @Richamc01 I have this issue right now, how did you solve it? I don't want to kill my docker containers if possible. The array doesn't stop, I can't disable docker and "docker stop $(docker ps -a -q)" does not work either..

 

 

Edit: I can't even do

 

halt

shutdown -h now

reboot

su -c poweroff

 

 

 

Ok idk what this is but I think this plugin screwed up my server again. The bottom line says " Array Started•Warning: posix_kill() expects parameter 1 to be int, string given in /usr/local/emhttp/plugins/ca.backup2/scripts/killRsync.php on line 27" and I can't stop Docker, I can't stop the array and I can't shut down the server properly.

 

This happened 5 days ago, this corrupted my docker.img and my cache was read-only after that. I had to clean my cache drives, put the backup back on it and setup docker again due to corrupted docker.img.

 

I really loved the idea of this plugin but it's no option for me to re-setup my cache drives and the docker every couple of days.

 

 

The diagnostics are attached, hoping this issue can be solved. Until that, the plugin is being uninstalled.

serverpc-diagnostics-20210923-1058.zip

[apraetor]

Ok, I'm completely lost. I've got a perfectly generic UnRaid config, trying to back up /mnt/cache/appdata/ to /mnt/user/CommunityApplicationsAppData/appdata/

 

It runs the backup without errors, and creates the dated subdirectory... but it's empty. No .tar, no nothing. I've tried with compression both on and off. I've got to be missing something obvious, right? Changing it to /mnt/user/appdata/ results in a full backup.. but the data lives on the cache anyway, so I'm not sure why the default value isn't working!

 

I didn't change anything else in the CA AppData Backup configuration, no excluded folders, etc. It stops the dockers fine, ought to be backing up binhex-plexpass's data. Total size on cache is 2.65 GB.

 

[Romany]

On 8/18/2021 at 9:25 PM, Groto said:

Hi,

 

Is there a way to restore just 1 specific docker in case I screw something up? I can't see a scenario where I would want to restore everything unless the drive dies. I would think a specific docker restore would be a use case that would get used much more.

 

Thanks!

I have the same question also - can you just restore a specific docker and leave the others alone?   I just had some issue with my unifi docker - which I run when I need it.   Could not web to the GUI - console showed the DB trying to start and failing.   I finally just went to my oldest backup created by this script - un-tarred to a temp directory - and cp -a -r the unifi directory over to the appdata directory - that fixed my issue.   If I had to do a "global" restore with this script I would have lost a lot of recent changes in my other dockers....I can work around this limitation - if indeed there is no way in the script to do that...

 

..Romany