[Plugin] CA Appdata Backup / Restore v2

[itimpi]

1 minute ago, Squid said:

if either of these files exist

 

/tmp/ca.backup2/tempFiles/backupInProgress

/tmp/ca.backup2/tempFiles/restoreInProgress

Thanks, that should be easy enough to handle.

Do you agree this would be better handled in the Parity Check Tuning plugin rather than trying to make it a feature in the CA Backup2 plugin?

[Squid]

Absolutely, because I could only stop a backup if a parity check is in operation prior to the beginning of it, and can't do anything once it starts and I'm not sure that postponing a backup is a wise thing to do.

[itimpi]

15 minutes ago, Squid said:

Absolutely, because I could only stop a backup if a parity check is in operation prior to the beginning of it, and can't do anything once it starts and I'm not sure that postponing a backup is a wise thing to do.

OK I will go ahead on that basis.   

 

The Parity Check Tuning plugin runs a background monitor task so it does not need to know when the backup is scheduled  to run as long as it can detect the backup task is running/finished and the presence or absence of the files you mention makes that easy.  There might be few minutes while they run in parallel before the monitor task triggers a pause/resume but that should not be a problem.

[DesertCookie]

Is there a way to have Appdata Backup create multiple backup archives?

 

I am experiencing an issue with my backup which is an 11GB tar.gz file. Most of that is my Jellyfin appdata with >5GB. When trying to open the tar.gz in Midnight Commander I always get an out of space error, even though I have TB of space. I'd like to create a separate backup of my Jellyfin appdata if possible to get two more manageable files.

[Squid]

Its a memory thing that's the cause (presumably) (or mc doesn't support gz that large).

 

Not currently possible to separate things out, but it's on the drawing board.  I'd suggest opening it via WinRar on a Windows Box

[dja]

4 hours ago, Squid said:

Not currently possible to separate things out, but it's on the drawing board. 

By drawing board, do you mean the ability to create different backup configs (jobs)? Such as how the VM backup plugin works? That would seem like an easy approach since you could selectively include what apps you want in one backup vs. another. 

I happen to use Plex and see the same  huge files. I can pull the info I need when required, but I don't need to back that up as often as my other dockers..   An option for scheduled jobs would be awesome! 

[Squid]

No, separate archives for each folder.

[Yousty]

So last night it appears my nvme cache drive just completely stopped working. I've tried everything but Unraid won't even recognize it. Thanks to this amazing app I have a backup of my cache drive from 3 days ago. The problem is I have no idea how to restore it. Could somebody please give me step by step instructions of what to do once I install the new cache drive and boot up Unraid?

[Squid]

[Bulletoverload]

The program will not lunch my startup script, location "/mnt/user/Dir/Scripts/Onedrive_Appdata_Sync.ph". Am I doing something wrong? it's just a simple rclone command to sync the new backup.

[Squid]

37 minutes ago, Bulletoverload said:

/mnt/user/Dir/Scripts/Onedrive_Appdata_Sync.ph

The assumption the plugin makes is that everything is a bash script (because most people would store it on the flash drive), so it executes it as /bin/bash

 

Create another script (.sh) that then calls it

 

(Or if that is a typo (ph), then make sure you're using Linux style line endings and not dos - Config File Editor / Notepad++ will be able to convert for you

[thor2002ro]

Works great i have some suggestions...option to backup docker.img its useful of you have lots of custom dockers..... And i think flash backup could use compression.....

 

The rest looks good. Good job 👌

[Atreides]

Whats the recommendation for backing up Plex if it's on its own in an Unassigned Disk SSD? Can I create two Backup tasks?

[Squid]

4 minutes ago, Atreides said:

Can I create two Backup tasks?

No

[Hammy Havoc]

On 3/31/2022 at 7:41 PM, Squid said:

No

Would love if this was possible. Even just having a separate/second instance of the plugin would be hugely useful.

[AustinCNunn]

I seem to be having an issue with the "Use Compression" function, as well as the "Update Applications On Restart" function. In that, neither of them change what the plugin is doing, regardless of which setting I choose. The plugin will keep compressing everything regardless of setting, and the Applications never get updated when that setting is enabled (and yes, the "CA Auto Update Applications" plugin is installed). 

 

I can toggle these functions on and off all day long, but they never seem to do anything. 

 

Anyone have any ideas?

[Squid]

 Compression simply adds another option to the tar command which compresses the archive.  Personally, I never use this as the extra time it takes to do this is insane, and since we're all media whores at heart, we've learned that "storage is cheap"

 

The "update" thing I'd have to look at.

[AustinCNunn]

On 4/10/2022 at 2:05 PM, Squid said:

 Compression simply adds another option to the tar command which compresses the archive.  Personally, I never use this as the extra time it takes to do this is insane, and since we're all media whores at heart, we've learned that "storage is cheap"

 

The "update" thing I'd have to look at.

Understood on the compression thing. It's really not an issue for me, just figured it may be a bug. 

 

Thanks on the update issue.

[vw-kombi]

Shouldn't the exclude folders option in this great tool NOT then stop the docker container ?

I have tdarr running and converting, and I wanted to backup all the others, so i exceluded this one folder, mistakenly thinking that it would then not be stopped........  

I often cant do a backup of all as on container is doing something and cant be stopped.

Would be great if we have that capability.

[Squid]

On 4/13/2022 at 6:28 PM, vw-kombi said:

Shouldn't the exclude folders option in this great tool NOT then stop the docker container ?

I have tdarr running and converting, and I wanted to backup all the others, so i exceluded this one folder, mistakenly thinking that it would then not be stopped........  

I often cant do a backup of all as on container is doing something and cant be stopped.

Would be great if we have that capability.

No, that just excludes a folder from the backup.

 

You want Advanced options at the bottom to keep the app from stopping.

[Ryonez]

Just had the plugin fail to backup.

Looking at the logs, everything worked fine, verification was good, but the backup file just does not exist.

 

Apr 18 02:10:41 Atlantis CA Backup/Restore: docker stop -t 60 atlantis-yoko
Apr 18 02:10:41 Atlantis CA Backup/Restore: Backing up USB Flash drive config folder to 
Apr 18 02:10:41 Atlantis CA Backup/Restore: Using command: /usr/bin/rsync  -avXHq --delete  --log-file="/var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log" /boot/ "/mnt/user/Backups/machine_specific_backups/atlantis_flash/" > /dev/null 2>&1
Apr 18 02:10:44 Atlantis CA Backup/Restore: Changing permissions on backup
Apr 18 02:10:44 Atlantis CA Backup/Restore: Backing up libvirt.img to /mnt/user/Backups/machine_specific_backups/atlantis_libvirt/
Apr 18 02:10:44 Atlantis CA Backup/Restore: Using Command: /usr/bin/rsync  -avXHq --delete  --log-file="/var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log" "/mnt/user/system/libvirt/libvirt.img" "/mnt/user/Backups/machine_specific_backups/atlantis_libvirt/" > /dev/null 2>&1
Apr 18 02:10:44 Atlantis CA Backup/Restore: Changing permissions on backup
Apr 18 02:10:44 Atlantis CA Backup/Restore: Backing Up appData from /mnt/user/appdata/ to /mnt/user/Backups/server_backups/atlantis_docker_appdata/[email protected]
Apr 18 02:10:45 Atlantis CA Backup/Restore: Using command: cd '/mnt/user/appdata/' && /usr/bin/tar -cvaf '/mnt/user/Backups/server_backups/atlantis_docker_appdata/[email protected]/CA_backup.tar'  --exclude "alteria-plex/Library/Application Support/Plex Media Server/Metadata" --exclude "0-pihole" --exclude "alteria-postgres" --exclude "alteria-hydrus-server"  * >> /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log 2>&1 & echo $! > /tmp/ca.backup2/tempFiles/backupInProgress
Apr 18 02:10:45 Atlantis CA Backup/Restore: Backup Complete
Apr 18 02:10:45 Atlantis CA Backup/Restore: Verifying backup
Apr 18 02:10:45 Atlantis CA Backup/Restore: Using command: cd '/mnt/user/appdata/' && /usr/bin/tar --diff -C '/mnt/user/appdata/' -af '/mnt/user/Backups/server_backups/atlantis_docker_appdata/[email protected]/CA_backup.tar' > /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log & echo $! > /tmp/ca.backup2/tempFiles/verifyInProgress
Apr 18 02:10:45 Atlantis kernel: br-55ef392db062: port 1(veth2d7642e) entered blocking state

Apr 18 02:11:46 Atlantis kernel: br-55ef392db062: port 70(veth7449b16) entered forwarding state
Apr 18 02:11:46 Atlantis CA Backup/Restore: #######################
Apr 18 02:11:46 Atlantis CA Backup/Restore: appData Backup complete
Apr 18 02:11:46 Atlantis CA Backup/Restore: #######################
Apr 18 02:11:46 Atlantis CA Backup/Restore: Backup / Restore Completed

Looking at the folder the backup should be in, it's empty.

 

Any idea what happened?

[Ryonez]

Second week in a row:

 

Apr 25 02:10:37 Atlantis CA Backup/Restore: docker stop -t 60 atlantis-yoko
Apr 25 02:10:37 Atlantis CA Backup/Restore: Backing up USB Flash drive config folder to 
Apr 25 02:10:37 Atlantis CA Backup/Restore: Using command: /usr/bin/rsync  -avXHq --delete  --log-file="/var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log" /boot/ "/mnt/user/Backups/machine_specific_backups/atlantis_flash/" > /dev/null 2>&1
Apr 25 02:10:42 Atlantis CA Backup/Restore: Changing permissions on backup
Apr 25 02:10:42 Atlantis CA Backup/Restore: Backing up libvirt.img to /mnt/user/Backups/machine_specific_backups/atlantis_libvirt/
Apr 25 02:10:42 Atlantis CA Backup/Restore: Using Command: /usr/bin/rsync  -avXHq --delete  --log-file="/var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log" "/mnt/user/system/libvirt/libvirt.img" "/mnt/user/Backups/machine_specific_backups/atlantis_libvirt/" > /dev/null 2>&1
Apr 25 02:10:42 Atlantis CA Backup/Restore: Changing permissions on backup
Apr 25 02:10:42 Atlantis CA Backup/Restore: Backing Up appData from /mnt/user/appdata/ to /mnt/user/Backups/server_backups/atlantis_docker_appdata/[email protected]
Apr 25 02:10:42 Atlantis CA Backup/Restore: Using command: cd '/mnt/user/appdata/' && /usr/bin/tar -cvaf '/mnt/user/Backups/server_backups/atlantis_docker_appdata/[email protected]/CA_backup.tar'  --exclude "alteria-plex/Library/Application Support/Plex Media Server/Metadata" --exclude "0-pihole" --exclude "alteria-postgres" --exclude "alteria-hydrus-server"  * >> /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log 2>&1 & echo $! > /tmp/ca.backup2/tempFiles/backupInProgress
Apr 25 02:10:42 Atlantis CA Backup/Restore: Backup Complete
Apr 25 02:10:42 Atlantis CA Backup/Restore: Verifying backup
Apr 25 02:10:42 Atlantis CA Backup/Restore: Using command: cd '/mnt/user/appdata/' && /usr/bin/tar --diff -C '/mnt/user/appdata/' -af '/mnt/user/Backups/server_backups/atlantis_docker_appdata/[email protected]/CA_backup.tar' > /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log & echo $! > /tmp/ca.backup2/tempFiles/verifyInProgress
Apr 25 02:10:42 Atlantis kernel: br-55ef392db062: port 1(veth391173c) entered blocking state

Apr 25 02:11:43 Atlantis kernel: br-55ef392db062: port 70(vetha9a6916) entered forwarding state
Apr 25 02:11:44 Atlantis CA Backup/Restore: #######################
Apr 25 02:11:44 Atlantis CA Backup/Restore: appData Backup complete
Apr 25 02:11:44 Atlantis CA Backup/Restore: #######################
Apr 25 02:11:44 Atlantis CA Backup/Restore: Deleting /mnt/user/Backups/server_backups/atlantis_docker_appdata/[email protected]
Apr 25 02:11:45 Atlantis CA Backup/Restore: Backup / Restore Completed

That's two weeks of missed backups, this is a big issue.

[UnKwicks]

Is it possible to have the Backup not in a .gz or .tar.gz file but just 1:1 copied?

Also without the folder with current date.

This would save a lot of disk space when backing it up further with duplicati.

To let duplicati do the versioning magic I just need a copy of the appdata folder as it is

Edited May 7, 2022 by UnKwicks

[Squid]

That would be v1, which is long deprecated in favour of the tarballs.  Problem with that method is a fair amount of users suffered crashes of the system when the backup ran due to just bombing the system with potentially a million symlinks when doing the backups of certain apps (ie: Plex)

 

You can always run an appropriate cp command yourself on a schedule.

[Tolete]

i made a cp of the CA_backup.tar.gz

can i access a file insde the CA_backup.tar.gz?

 

tar -xvf CA_backup.tar.gz

i guess how do i tar a specific directory from inside CA_backup.tar.gz?

Edited May 21, 2022 by Tolete