[Plugin] CA Appdata Backup / Restore v2.5

[KluthR]

3 hours ago, macason said:

would it be appropriate to include a configurable rsync/scp/ftp step inside the tool

To do what? Transfer the final backup after creation? How about the Post-Backup script option?

 

1 hour ago, KermitJr said:

I'm not sure how painful that would be to implement

Already done for the new plugin

 

1 hour ago, KermitJr said:

any chance of having a progress bar or percentage report for backup and verification (and restore?)

Thats on my list but I have to play with the tools a bit.

[macason]

1 hour ago, KluthR said:

To do what? Transfer the final backup after creation? How about the Post-Backup script option?

Yes thanks, Ive already implemented with a post-run script and removed the compression as it messes with rsync without the correct flags. it can all be handled manually but I wanted to ask the question as it feels like the next useful step in an already very useful tool.

[KluthR]

Please describe what exactly should be done with your request.

[loheiman]

What does "Create separate archives (one per folder)?" option do exactly? Does it create separate archiver per folder in the appdata folder and thus allow the user to restore a single app's appdata folder?

 

Also why does the Verify Backups option say it can lead to broken backups? Where can I learn more about the pros/cons of enabling this? Thanks

Edited May 9, 2023 by loheiman

[macason]

18 minutes ago, KluthR said:

Please describe what exactly should be done with your request.

Nothing at all, Id find it useful. Sorry for the wasted time.

[wgstarks]

4 hours ago, loheiman said:

What does "Create separate archives (one per folder)?" option do exactly? Does it create separate archiver per folder in the appdata folder and thus allow the user to restore a single app's appdata folder?

Yes (although I don’t believe that option is yet available in the plugin).

 

4 hours ago, loheiman said:

Also why does the Verify Backups option say it can lead to broken backups?

Typically, backups done while the container is still running will fail the verification because the data changed during the backup process. This will result in no backup being created. There have also been a few reports of Plex data changing even though the container is stopped. Possibly some other container is changing that data.

 

4 hours ago, loheiman said:

Where can I learn more about the pros/cons of enabling this?

There is quite a bit of discussion regarding this in this support thread.

Edited May 9, 2023 by wgstarks

[shpitz461]

Hi @KluthR,

I've upgraded Unraid and installed latest plugin.

I'm getting a message on some of the containers "does not have any volume to back up".

This causes the container to not get backed-up at all.

e.g. EmbyServer:

Quote

[09.05.2023 07:50:07][Main][info] 👋 WELCOME TO APPDATA.BACKUP!!
[09.05.2023 07:50:07][Main][debug] plugin-version: 2023.04.27
[09.05.2023 07:50:07][Main][debug] unraid-version: Array
(
    [version] => 6.12.0-rc4.1
)

 

Quote

[09.05.2023 07:50:07][Main][info] 👋 WELCOME TO APPDATA.BACKUP!!
[09.05.2023 07:50:07][Main][info] Backing up to: /mnt/user/backup/Unraid/AppdataBackup/ab_20230509_075007
[09.05.2023 07:50:07][Main][info] Saving container XML files...
[09.05.2023 07:50:07][Main][info] Method: Stop/Backup/Start
[09.05.2023 07:51:55][EmbyServer][info] Stopping EmbyServer... 
[09.05.2023 07:51:59][EmbyServer][info] done! (took 4 seconds)
[09.05.2023 07:51:59][EmbyServer][info] EmbyServer does not have any volume to back up! Skipping
[09.05.2023 07:51:59][EmbyServer][info] Starting EmbyServer... (try #1)

 

Quote

[09.05.2023 07:50:07][Main][info] Backing up to: /mnt/user/backup/Unraid/AppdataBackup/ab_20230509_075007
[09.05.2023 07:50:07][Main][debug] Not executing script: Not set!
[09.05.2023 07:50:07][Main][debug] Containers: Array
(
    [31] => Array
        (
            [Image] => emby/embyserver:latest
            [ImageId] => 51d5b09746bf
            [Name] => EmbyServer
            [Status] => Up 2 days
            [Running] => 1
            [Paused] => 
            [Cmd] => /init
            [Id] => 60069ae43a02
            [Volumes] => Array
                (
                    [0] => /mnt/user/media/:/mnt:rw
                    [1] => /mnt/user/Transcodes:/transcodes:rw
                    [2] => /mnt/cache_nvme/appdata/EmbyServer:/config:rw
                )

            [Created] => 2 months ago
            [NetworkMode] => dockernet
            [CPUset] => 
            [BaseImage] => 
            [Icon] => https://raw.githubusercontent.com/MediaBrowser/Emby.Resources/master/images/Logos/logoicon.png
            => http://[IP]:[PORT:8096]/
            [Shell] => 
            [Ports] => Array
                (
                    [0] => Array
                        (
                            [IP] => 192.168.20.6
                            [PrivatePort] => 8096
                            [PublicPort] => 8096
                            [NAT] => 1
                            [Type] => tcp
                        )

                )

        )

Here's my Docker page with Emby:

 

Same thing with AList container:

Quote

[09.05.2023 07:50:14][AList][info] No stopping needed for AList: Not started!
[09.05.2023 07:50:14][AList][info] AList does not have any volume to back up! Skipping
[09.05.2023 07:50:14][AList][info] AList is being ignored, because it was not started before (or should not be started).

 

Prior appdata-backup plugin backed up all appdata folders and not skip anything, any idea what is happening now?

Thanks

[KluthR]

9 hours ago, macason said:

Nothing at all, Id find it useful. Sorry for the wasted time.

??

=>

11 hours ago, macason said:

I wanted to ask the question as it feels like the next useful step in an already very useful tool

So, please describe more clear, I dont fully got, what you mean.

 

9 hours ago, loheiman said:

What does "Create separate archives (one per folder)?" option do exactly?

It creates one folder per found appdata folder NOT PER CONTAINER since the "v2.5" ist not capable of detecting container based folders. The all-new appdata.backup does.

 

9 hours ago, loheiman said:

Also why does the Verify Backups option say it can lead to broken backups?

Additional to @wgstarks answer: DISabling this can lead to broken backups, because ignoring verification errors could lead to broken data, since any error will be discarded then.

 

4 hours ago, wgstarks said:

Where can I learn more about the pros/cons of enabling this?

The advantage is obvious: A 100% working backup (in matter of valid data) is always our goal, so we have to make sure:

 

* Containers are stopped during their backup, to ensure it does not change data during backup

* Verification is an extra layer of certainty: It checks the original data again against the just created archive. If only one bit differs, it throws an error. Sometimes containers share shares between each others, this would bring it up to your attention (in that example case one want to exclude it in one of the containers).

 

A backup while the to backing up files are being modified CAN work, but dont have to. This is up to every user by configuring it as needed. Default settings have all checks enabled.

 

3 hours ago, shpitz461 said:

Prior appdata-backup plugin backed up all appdata folders and not skip anything, any idea what is happening now?

Your post seem to have broken styling - apart from that:

The old plugin(s) were backing up folder by folder. The new backup does not: It reads every docker container volume mapping and uses that - and only that information.

 

Please post your debug logfile to get this confimred: It seems you have "Backup external volumes?" disabled, leading to skip any container volume mapping thats outside your configured "Allowed source paths" setting. With no volume left, it skips the backup for that container. Please open the help blocks for "Allowed source paths" to get more infos how the plugin differentiate between Internal and External volumes.

 

EDIT: The last post is about the new plugin, this thread is for the old one. If more help is needed, please switch to the new one (see 1st post)

Edited May 9, 2023 by KluthR

[shpitz461]

7 hours ago, KluthR said:

Your post seem to have broken styling - apart from that:

The old plugin(s) were backing up folder by folder. The new backup does not: It reads every docker container volume mapping and uses that - and only that information.

 

Please post your debug logfile to get this confimred: It seems you have "Backup external volumes?" disabled, leading to skip any container volume mapping thats outside your configured "Allowed source paths" setting. With no volume left, it skips the backup for that container. Please open the help blocks for "Allowed source paths" to get more infos how the plugin differentiate between Internal and External volumes.

You da man! That was it, I only had mnt/user/appdata listed, I added /mnt/cache_nvme/appdata and now everything is getting backed up

Thanks much!

[Zac Lavcanski]

Hi there,

 

In the setting to run a custom script option, can an option be added to run the custom script Path To Custom Post Start Script?

 

At the moment, it has only to run the script before the backup starts.

 

Thanks.

[macason]

16 hours ago, KluthR said:

??

=>

So, please describe more clear, I dont fully got, what you mean.

1. Compression in the tool is not compatible with rsync ("--rsyncable" flag required) - maybe a check box to activate flag or customisation.

2. Once the backup is complete and verified a configuration to PUSH the backup offsite/remote would be ideal. - It should be configurable to use SCP/SFTP/RSYNC. etc.. SCP $CURRENT_BACKUP $REMOTE_SITE:/"backupLocation"

 

[KluthR]

1 hour ago, Zac Lavcanski said:

In the setting to run a custom script option, can an option be added to run the custom script Path To Custom Post Start Script?

The new appdata.backup plugin has several custom scripts options.

[Zac Lavcanski]

Yes I see it has 3 options for custom scripts but all of those run pre-backup. Can an option be added to run the script post-backup? So once the backup has completed, then the custom script can be run?

Current 3 options are:

 

Path To Custom Stop Script:

Path To Custom Pre-start Script:

Path To Custom Start Script:

 

Would be good to have another option Path To Custom Post-Start Script.

[KluthR]

No, you are talking about ca.backup2. I talk abot the new appdata.backup. It has those options. See 1st post.

 

[Zac Lavcanski]

Thank you for your reply.

 

I have checked the apps and this is the only one that appears and that is the one that I have installed, and I am using. I do not see any Post-run Script

 

 

 

 

[KluthR]

Please pay attention to the details: Appdata.backup is available starting with Unraid 6.12+ only. Appdatabackup/Restore 2.5 is deprecated since 6.12+

[Zac Lavcanski]

Thank you. Currently running Ver 6.11.5. Have not update to 6.12 RC5.

 

Appreciate your responses.

[Eddie Seelke]

Will the new version add an option to stop and start each container as it's appdata folder is being tarred, instead of stopping all docker containers for the entire backup process?

[KluthR]

It has the option, yes.

[buee]

I searched and see this has been reported. Add me to the list of failing backups due to the "Image cannot be deleted, in use by other containers" problem. The container in question is prowlarr and, as far as I know, it's not in use by other containers. Quite the contrary, other containers depend on it for communications, but as far as I know there's no image dependency. Anyone know how I can check that?

 

The really weird thing is that it's stopped and left in a stopped state, so about 6 hours after the backup runs, I start getting health check notifications that it's down and I just manually start it again. Why not bring it back up? I'd like to see containers returned to the state they were found in after the backup runs if possible.

[KluthR]

Please retest with the new plugin (see first post) after Unraid 6.12 is released.

Edited May 21, 2023 by KluthR

[Bjur]

On 1/3/2023 at 3:08 AM, BlueLight said:

My appdata got deleted, I think?

 

I upgraded to v3 around the same time as upgrading to 6.11.5 and set backups for 4am on the 1st of every month. Even with the red warning,s I misinterpreted it as in USB flash backup has been deprecated ( and taken out on the docker side), so i mindlessly filled in the usb backup location and bing bang boom...

After the backup on the 1st, I woke up to finding that unraid is fine, but something happened within appdata (my suspicion).

Some apps can't start, most apps are running but not making connection to the server while in their GUI.

ex.: My NginxProxyManager docker starts up and takes me to login, but cannot verify my credentials to login. // Myplex app works fine, but can't connect to libraries. // Krusader opens up but don't see any of the folders/shares configured.

 

I've tried to restore using the backup file that was generated, and it did not work.

 

I copied my appdata folder to make a duplicate. This backup was right before upgrading to 6.11.5... 

I've tried copying this duplicate folder over and I don't see that it made any difference.

What are my options moving forward? Is there any advice on how to make my duplicate appdata folder usable?

 

Thanks for any help in advanced! Happy 2023!

 

 

Hi thanks for the plugin.

This post below scares me a little bit.

I've just installed appdata backup 2.5 and Unraid 6.11.5.

 

1. If I want to backup USB flash drive and put in a folder in USB backup destination, would I then have my appdata folder deleted?

 

2. If I want to have 2 destinations. 1 locally which will be a folder in Rclone that syncs with cloud and a second one that will be a USB drive attaced?

 

3. Would it also be possible to have docker templates backed up?

Edited May 23, 2023 by Bjur

[KluthR]

1. Normally, nothing gets deleted, but some users have "special" setups.

 

To be honest: I would disable USB backup for now, wait for Unraid 6.12 stable and migrate to

 

I just made the errors popping up correctly, the rest is the same. The new plugin is a complete rewrite. The ca.backup plugin will stop working with Unraid 6.12 (It still works for Unraid up to 6.11.5 though)

[Bjur]

37 minutes ago, KluthR said:

1. Normally, nothing gets deleted, but some users have "special" setups.

 

To be honest: I would disable USB backup for now, wait for Unraid 6.12 stable and migrate to

 

I just made the errors popping up correctly, the rest is the same. The new plugin is a complete rewrite. The ca.backup plugin will stop working with Unraid 6.12 (It still works for Unraid up to 6.11.5 though)

 

Thanks for the quick answer:)

 

What about 

2. If I want to have 2 destinations. 1 locally which will be a folder in Rclone that syncs with cloud and a second one that will be a USB drive attaced?

Would it be possible to backup to two destinations?

 

3. Would it also be possible to have docker templates backed up?

 

Also would it be possible to not have the as tar files if it's uncompressed in regards to deduplicating backup software?

[murkus]

I am going through a very frustrating moment because of the way CA backup is ordering the stopping and starting of containers. normally, as I understood unraid, for starting the order of the containers in the UI list is used. And of course this has a deeper meaning, if you have container dependencies.

 

Why the '*/%§§ is CA backup using an alphabetical order, which is of course not the same as in the UI container list.

 

This caused that Mariadb was stopped before the depending application and so the database was corrupted.

 

I would strongly mandate that CA backup should use gthe same order as unraid does for starting containers, and the inverse order for stopping the containers (i.e. the same order that unraid uses to stop the containers). Anything else will lead to disaster for some users!