[Plugin] CA Appdata Backup / Restore v2.5


KluthR

Recommended Posts

securityIcon-WithShadow.png.ef32284851a33176398055071ae5642b.png

CA Appdata Backup / Restore v2.5

Welcome to a new epsiode of the appdata backup plugin :)

 

The previous thread is located here: https://forums.unraid.net/topic/61211-plugin-ca-appdata-backup-restore-v2/ 

 

IMPORTANT!

Starting with Unraid 6.12, this plugin will be deprecated. As already stated, there will be a complete rewrite of it. Please have a look at the new thread for it

 

-----------------------------------------------------------------------------------------

What happened?

I stumbled upon some bugs while using Squids plugin and made pull requests. Squid told me, he has only limited time and asked me if I want to publish a new version of the plugin. So we decided to deprecate his version in favor of "my" version.

 

For now, the plugin has fixed for error detection and some other new features (see changelog). More to come.

 

Anything I should consider?

YES!

Since the updated version shows you error during backup/verification (and even while stopping/starting containers), it COULD happen, that you see errors after update which were not there before. In most cases, these errors/warnings were there before but supressed.

Errors like "File changed as we read it" or "Contents differ" (during verification) tells you, that some files are still in use. Post here (with your log) in that case.

Edited by KluthR
  • Like 9
Link to comment

Known issues & planned features

Known issues

  • If container autoupdate is enabled, the container gets started and producing a "Already started" info
    • Currently being accepted as ok - produces some debug info inside the log but does not treat it as error
  • 02.2023: Some users facing "tar verify failed!" issues although all dockers are stopped.
    • Currently searching possible root causes.

 

Planned features

  • Complete overhaul of the code including PHP8 compatibility
  • ✔️Files/Folder config per container
    • ✔️Maybe extra sources apart from containers?
  • ✔️Option (separate) to disable encoutered errors (containers, tar backup, tar verify)
  • ✔️Multicore backup (zstdmt)
  • ✔️PreRun/PreBackup/PostBackup/PostRun custom scripts
  • ✔️Restore single container data
  • Treesize view of all source data?
  • ✔️Include the docker templates in backups
  • ✔️Some easy diag functions
    • Share diag docker infos and the config
  • Some anonymous statistic collection
    • Opt-In

 

Not now / Never / Not planned

✔️ Done

In progress

 

Notes

  • Multiple appdata volume sources possible
  • Save to single archive removed
    • The backup will always create seperate files now
  • Flash backup is back!
    • It uses unraids native way for doing it
  • VMs backup also included
    • Basic operation. More to come - maybe
Edited by KluthR
State update
  • Like 7
Link to comment
9 hours ago, Transient said:

Would it be possible to migrate the config from the old one?

Just copy the .json file from /boot/config/plugins/ca.backup2 to a sage place on your pc and after reinstall back to that location.

 

An automated migration is not implemented.

Link to comment

HI

I am getting errors after installing the new version

CA Backup: 25-12-2022 03:06

appData Backup
Backup of appData complete - Errors occurred
But how do i know what the error was ?

 

I can see some error in log interface see here

 

Line 39 to 46 show some issue but that path it says is not there is wrong its a shared backup path i have used for years for appdata ???

You can see libvert no issue but appdata nothing from 11 Dec 

I removed the Flashdrive backup as i usethe UNraid.net plugin as suggested

 

 

MuzN7aA.png

Edited by OsoPolar
  • Like 1
Link to comment

Merry Christmas!

 

Just made the switch a few days ago and I believe I chose the same settings. However, my first backup using v3 is currently running and its saving the backups in seperate .tar files per docker container. On v3, I just got a single .tar file with the whole backup.

 

Is there a benefit to having it do the backup like this or did I choose the wrong setting during setup? I do my backups on the 25th of every month at 3AM, for some reason, Plex is taking HOURS and is holding up the process. 😅

 

Just wanted to ask :) Thank you so much for maintaining this feature. 

Edited by DazedAndConfused
Link to comment
On 12/24/2022 at 2:36 PM, blaine07 said:

Like 70gb compared to a normal ~15gb

Are you sure, that your settings are the same? libvirt backup enabled?

 

7 hours ago, OsoPolar said:

I am getting errors after installing the new version

Yes, as you already wrote: The plugin says, the source folder is not there. Are you sure /mnt/CACHE/appdata is correct? Not /mnt/USER/appdata?

 

49 minutes ago, DazedAndConfused said:

Is there a benefit to having it do the backup like this or did I choose the wrong setting during setup?

Its on you. If you need something single from a backup, you can pick the right tar.gz instead open the whole backup. In future, more container specific options will come, like single recovery etc. The backup job is the same in both variants - just a split for every folder found in the source folder.

Link to comment

Hi @KluthR, backups have been successfull every night since the 23rd. Thanks forall your work on this! Here is the log as requested showing container already started debug info:

[26.12.2022 06:00:01] Backup of appData starting. This may take awhile
[26.12.2022 06:00:01] Stopping adminer... done! (took 0 seconds)
[26.12.2022 06:00:01] Stopping authelia... done! (took 0 seconds)
[26.12.2022 06:00:01] Not stopping auto-m4b: Not started! [ / Exited (2) 7 days ago]
[26.12.2022 06:00:01] Stopping bazarr... done! (took 5 seconds)
[26.12.2022 06:00:06] Stopping binhex-delugevpn... done! (took 3 seconds)
[26.12.2022 06:00:09] Not stopping binhex-krusader: Not started! [ / Exited (0) 3 weeks ago]
[26.12.2022 06:00:09] Stopping binhex-lidarr... done! (took 2 seconds)
[26.12.2022 06:00:11] Stopping binhex-prowlarr... done! (took 1 seconds)
[26.12.2022 06:00:12] Stopping binhex-radarr... done! (took 1 seconds)
[26.12.2022 06:00:13] Stopping binhex-readarr... done! (took 1 seconds)
[26.12.2022 06:00:14] Stopping binhex-readarr-2... done! (took 0 seconds)
[26.12.2022 06:00:14] Stopping binhex-sonarr... done! (took 1 seconds)
[26.12.2022 06:00:15] Stopping bitwarden... done! (took 0 seconds)
[26.12.2022 06:00:15] Not stopping calibre: Not started! [ / Created]
[26.12.2022 06:00:15] Not stopping code-server: Not started! [ / Created]
[26.12.2022 06:00:15] Stopping frigate... done! (took 3 seconds)
[26.12.2022 06:00:18] Not stopping HandBrake: Not started! [ / Exited (0) 5 months ago]
[26.12.2022 06:00:18] Stopping Home-Assistant-Core... done! (took 7 seconds)
[26.12.2022 06:00:25] Stopping mariadb... done! (took 4 seconds)
[26.12.2022 06:00:29] Stopping mosquitto... done! (took 1 seconds)
[26.12.2022 06:00:30] Not stopping Network-Speed-Test: Not started! [ / Created]
[26.12.2022 06:00:30] Stopping nextcloud... done! (took 3 seconds)
[26.12.2022 06:00:33] Stopping NodeRed... done! (took 0 seconds)
[26.12.2022 06:00:33] Stopping ombi... done! (took 4 seconds)
[26.12.2022 06:00:37] Not stopping PASTA: Not started! [ / Exited (0) 7 days ago]
[26.12.2022 06:00:37] Stopping PlexMediaServer... done! (took 7 seconds)
[26.12.2022 06:00:44] Stopping redis... Error while stopping container! Code: Container already started
[26.12.2022 06:00:44] Stopping swag... done! (took 4 seconds)
[26.12.2022 06:00:48] Stopping tautulli... done! (took 0 seconds)
[26.12.2022 06:00:48] Stopping unifi-controller... done! (took 8 seconds)
[26.12.2022 06:00:56] Backing up libvirt.img to /mnt/user/BackupVM/
[26.12.2022 06:00:56] Using Command: /usr/bin/rsync -avXHq --delete --log-file="/var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log" "/mnt/user/system/libvirt/libvirt.img" "/mnt/user/BackupVM/" > /dev/null 2>&1
2022/12/26 06:00:56 [5235] building file list
2022/12/26 06:01:05 [5235] sent 123 bytes received 19 bytes 14.95 bytes/sec
2022/12/26 06:01:05 [5235] total size is 1,073,741,824 speedup is 7,561,562.14
[26.12.2022 06:01:31] Backing Up appData from /mnt/user/appdata/ to /mnt/user/Backup Appdata/[email protected]
[26.12.2022 06:01:31] Separate archives disabled! Saving into one file.
[26.12.2022 06:01:31] Backing Up
[26.12.2022 06:19:24] Verifying Backup
[26.12.2022 06:37:21] done
[26.12.2022 06:37:21] Starting redis... (try #1) Hmm - container is already started! DEBUG Info following...
[26.12.2022 06:37:21] BEFORE backing up container status: {"Image":"bitnami\/redis:latest","ImageId":"57de284f5635","Name":"redis","Status":"Up 23 hours","Running":true,"Paused":false,"Cmd":"\/opt\/bitnami\/scripts\/redis\/entrypoint.sh \/opt\/bitnami\/scripts\/redis\/run.sh","Id":"ef036282f03e","Volumes":null,"Created":"2 days ago","NetworkMode":"proxynet","CPUset":"","BaseImage":false,"Icon":"https:\/\/raw.githubusercontent.com\/A75G\/docker-templates\/master\/templates\/icons\/redis.png","Url":false,"Shell":false,"Ports":[{"IP":"172.19.0.2","PrivatePort":"6379","PublicPort":"6379","NAT":true,"Type":"tcp"}]}
[26.12.2022 06:37:21] AFTER backing up container status: {"Image":"bitnami\/redis:latest","ImageId":"7ce22299b7ca","Name":"redis","Status":"Up 36 minutes","Running":true,"Paused":false,"Cmd":"\/opt\/bitnami\/scripts\/redis\/entrypoint.sh \/opt\/bitnami\/scripts\/redis\/run.sh","Id":"35c577a5a171","Volumes":null,"Created":"36 minutes ago","NetworkMode":"proxynet","CPUset":"","BaseImage":false,"Icon":"https:\/\/raw.githubusercontent.com\/A75G\/docker-templates\/master\/templates\/icons\/redis.png","Url":false,"Shell":false,"Ports":[{"IP":"172.19.0.3","PrivatePort":"6379","PublicPort":"6379","NAT":true,"Type":"tcp"}]}
[26.12.2022 06:37:23] Starting mariadb... (try #1) done!
[26.12.2022 06:37:24] Waiting 10 seconds before carrying on
[26.12.2022 06:37:34] Starting adminer... (try #1) done!
[26.12.2022 06:37:36] Starting authelia... (try #1) done!
[26.12.2022 06:37:38] Starting swag... (try #1) done!
[26.12.2022 06:37:38] Waiting 10 seconds before carrying on
[26.12.2022 06:37:48] Starting bitwarden... (try #1) done!
[26.12.2022 06:37:50] Starting nextcloud... (try #1) Hmm - container is already started! DEBUG Info following...
[26.12.2022 06:37:50] BEFORE backing up container status: {"Image":"linuxserver\/nextcloud:latest","ImageId":"699e4779f9ed","Name":"nextcloud","Status":"Up 23 hours","Running":true,"Paused":false,"Cmd":"\/init","Id":"e6160240a456","Volumes":["\/mnt\/user\/nextcloud\/:\/data:rw","\/mnt\/user\/:\/shares:rw","\/mnt\/user\/appdata\/nextcloud:\/config:rw"],"Created":"6 days ago","NetworkMode":"proxynet","CPUset":"","BaseImage":false,"Icon":"https:\/\/raw.githubusercontent.com\/linuxserver\/docker-templates\/master\/linuxserver.io\/img\/nextcloud-icon.png","Url":"https:\/\/[IP]:[PORT:443]\/","Shell":false,"Ports":[{"IP":"172.19.0.6","PrivatePort":"443","PublicPort":"444","NAT":true,"Type":"tcp"}]}
[26.12.2022 06:37:51] AFTER backing up container status: {"Image":"linuxserver\/nextcloud:latest","ImageId":"f2e246dffe1e","Name":"nextcloud","Status":"Up 36 minutes","Running":true,"Paused":false,"Cmd":"\/init","Id":"da63929d871d","Volumes":["\/mnt\/user\/appdata\/nextcloud:\/config:rw","\/mnt\/user\/nextcloud\/:\/data:rw","\/mnt\/user\/:\/shares:rw"],"Created":"36 minutes ago","NetworkMode":"proxynet","CPUset":"","BaseImage":false,"Icon":"https:\/\/raw.githubusercontent.com\/linuxserver\/docker-templates\/master\/linuxserver.io\/img\/nextcloud-icon.png","Url":"https:\/\/[IP]:[PORT:443]\/","Shell":false,"Ports":[{"IP":"172.19.0.2","PrivatePort":"443","PublicPort":"444","NAT":true,"Type":"tcp"}]}
[26.12.2022 06:37:53] Starting unifi-controller... (try #1) done!
[26.12.2022 06:37:55] Starting mosquitto... (try #1) done!
[26.12.2022 06:37:57] Starting frigate... (try #1) done!
[26.12.2022 06:37:57] Waiting 20 seconds before carrying on
[26.12.2022 06:38:17] Starting Home-Assistant-Core... (try #1) done!
[26.12.2022 06:38:20] Starting NodeRed... (try #1) done!
[26.12.2022 06:38:22] Starting binhex-delugevpn... (try #1) done!
[26.12.2022 06:38:22] Waiting 45 seconds before carrying on
[26.12.2022 06:39:07] Starting binhex-prowlarr... (try #1) done!
[26.12.2022 06:39:09] Starting binhex-radarr... (try #1) done!
[26.12.2022 06:39:12] Starting binhex-sonarr... (try #1) done!
[26.12.2022 06:39:12] Waiting 60 seconds before carrying on
[26.12.2022 06:40:12] Starting binhex-lidarr... (try #1) done!
[26.12.2022 06:40:14] Starting bazarr... (try #1) done!
[26.12.2022 06:40:16] Starting binhex-readarr... (try #1) done!
[26.12.2022 06:40:18] Starting binhex-readarr-2... (try #1) done!
[26.12.2022 06:40:20] Starting PlexMediaServer... (try #1) done!
[26.12.2022 06:40:23] Starting ombi... (try #1) done!
[26.12.2022 06:40:25] Starting tautulli... (try #1) done!
[26.12.2022 06:40:27] Deleting Dated Backup set: /mnt/user/Backup Appdata/[email protected]
[26.12.2022 06:40:27] Backup / Restore Completed

Hope you had a great Christmas!

Happy New Year to all!

Cheers,

Tim

Link to comment
23 hours ago, KluthR said:

Are you sure, that your settings are the same? libvirt backup enabled?

 

Yes, as you already wrote: The plugin says, the source folder is not there. Are you sure /mnt/CACHE/appdata is correct? Not /mnt/USER/appdata?

 

Its on you. If you need something single from a backup, you can pick the right tar.gz instead open the whole backup. In future, more container specific options will come, like single recovery etc. The backup job is the same in both variants - just a split for every folder found in the source folder.

So it appears my huge backup and maybe all my issues were caused by recently famed Kasm(& I evidently installed around same time as I went v2 to V3) everyone has been talking about lately. Evidently it’s huge. Later today I’ll switch back to V3 and give this a go again. Finally got a clean backup on V2 though today. 🤞🏻

Link to comment
20 hours ago, MothyTim said:

Here is the log as requested showing container already started debug info

These containers were clearly started during backup from something else. It seems, that you do not have autoupdate enabled? So, something else is starting the containers. Please check what it could be.

 

20 hours ago, MothyTim said:

Hope you had a great Christmas!

Happy New Year to all!

Same to you :)

 

Link to comment

Thank you @KluthR to put effort into this plugin. I'm building a backup server and use my backups to bringback the data. I upgraded the plugin on December 18. I can bring back any backup set prior to this date, but get a TAR archive error for the backup done after the 18 done with the new version of the plugin. Here is the log of the restore that doesn't work.

 

[27.12.2022 08:54:25] Restore of appData starting. This may take awhile
[27.12.2022 08:54:25] Stopping duplicati... done! (took 4 seconds)
[27.12.2022 08:54:29] Restoring appData from to
[27.12.2022 08:54:29] Restoring: backup.log
/usr/bin/tar: This does not look like a tar archive
/usr/bin/tar: Skipping to next header
/usr/bin/tar: Exiting with failure status due to previous errors
[27.12.2022 08:54:29] User aborted backup!
[27.12.2022 08:54:29] done
[27.12.2022 08:54:29] Starting duplicati... (try #1) done!
[27.12.2022 08:54:32] Restore finished. Ideally you should now restart your server
[27.12.2022 08:54:32] Backup / Restore Completed

 

As I was writing this post, I saw that the restore was trying to read backup.log and stopped working as this is not a TAR file. I removed the backup.log file from the source directory and started again the restore. I thought it was the culprit, but then, the system showed that I aborded the backup, which of course, I did not. 
 

[27.12.2022 09:39:50] Restore of appData starting. This may take awhile
[27.12.2022 09:39:50] Not stopping duplicati: Not started! [ / Exited (0) 12 minutes ago]
[27.12.2022 09:39:50] Restoring appData from to
[27.12.2022 09:39:50] Restoring: CA_backup_binhex-krusader.tar.gz
[27.12.2022 09:40:35] User aborted backup!
[27.12.2022 09:40:35] done
[27.12.2022 09:40:35] Restore finished. Ideally you should now restart your server
[27.12.2022 09:40:35] Backup / Restore Completed

 

I don't know where to look now. Any thoughts?

 

Thank you.

Edited by Yanos!
Link to comment

I'm traveling at the moment so can't get the logs, but was just curious if this is a known issue.......manual backups WORK, but the automated daily at 5am is not working. I didn't notice it when I swapped v2 for v3 (because I was doing manuals at the time during the switch). All my settings are the same as v2 and everything works as expected except the daily automatic execution. 

 

If no one has heard of this I'll grab the logs tomorrow and post them when I get back from traveling.

 

Edit:

Well, never mind. I changed the auto backup time settings, saved, then changed them back to the original settings and backup ran fine this morning. All is good.

 

Thanks!

Edited by jmbailey2000
Link to comment
  • KluthR changed the title to [Plugin] CA Appdata Backup / Restore v2.5

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.