[Plugin] CA Appdata Backup / Restore v2


Squid

Recommended Posts

Well - tested it myself by initiating a manual backup.  It does not purge old backups until the running one is complete - as expected and safe.  Is it a big ask to have an option where the backups are purged according to the plan prior to the backup?

 

TY.

 

Kev.

Link to comment
On 11/28/2020 at 4:09 PM, Squid said:

If it doesn't come with Unraid, I'm not using it in the plugin.

Okay. Fair decision, your call after all. Yet it still causes a significant downtime for me when the backup runs if compression only runs on one thread. 

I found a way to use pigz without needing to modify the plugin tho. I installed pigz via nerd tools and replaced the gzip binary with a symlink to pigz, so it's used by

tar --auto-compress

with the .tar.gz ending like CA Backupv2 invokes it. It yields substantial performance improvements with my 8C/16T CPU.

Currently that's a reduction down to 6min from 37min (with verification of the backup file).

 

If anyone stumbles upon this and thinks they can profit from parallel compression, it's fairly simple to do, just don't expect @Squid to fix any problems that should come up with this configuration and keep it in mind if you should ever run into a problem with CA Backupv2 to revert this before filing any bug report.


Just move the original gzip binary (I renamed it to ogzip so if you should ever need it, it's still there) and then create the link:

mv /bin/gzip /bin/ogzip
ln -s /usr/bin/pigz /bin/gzip

'Official' support via a checkbox if pigz is detected would probably be better but this works for me and can be applied via the /boot/config/go script just fine.

  • Like 1
  • Thanks 1
  • Upvote 1
Link to comment

Hi, I am using this plugin for month and it works perfectly. Thanks for that.

I recently installed a nvme disk and put some containers on it. By doing this, I suppose the Autobackup plugin will not detect the new location for the containers' data. Does a symlink would solve this problem? Something like ln -s /mnt/disks/nvme/appdata/plex /mnt/user/appdata/plex ?

How will behave unraid with a symlink in the appdata folder?

Thank you for your answers.

Denis

Link to comment

Yeah, IDK how everything would respond.  Probably it would work though.  Reasoning for only dealing with the single folder is that 

  1. It's a hell of a lot simpler to do
  2. Not all containers utilize a /config mapping (or have multiple mappings to appdata), so automatically determining what to backup is a PITA
  3. I don't use my system like that 😎
Link to comment
3 hours ago, Unraid_Noob said:

Does a symlink would solve this problem?

As CA Backupv2 doesn't use the --dereference (-h) operator for tar it would only backup the symbolic link itself, not the files it points to.

Even with the -h option I'd advise against it tho. Just put all your config folders from your docker containers to the same folder for a clean backup and easy restore.

Edited by Thx And Bye
Link to comment

I have a quick question about verifying backups. 

 

Mainly creating the backup takes about 10 minutes.

 

Dec 20 01:02:02 Tower CA Backup/Restore: Backing Up appData from /mnt/user/appdata/ to /mnt/user/backups/appdatabackup/[email protected]
Dec 20 01:12:38 Tower CA Backup/Restore: Backup Complete

 

Verifying the backup takes over an hour. 

Dec 20 01:12:38 Tower CA Backup/Restore: Verifying backup
Dec 20 02:27:07 Tower CA Backup/Restore: appData Backup complete

Is this normal behaviour? 

 

Also, what would happen if a parity check happened to be running at the same time as the backup script? 

 

This is a fantastic plugin btw!

Link to comment

Permissions issue, maybe? If I browse to the folder where the Tower Backups are being stored, I can see the individual folders based on the dates completed. But if I try to browse into any of them, from my Windows computer, I get the error that I don't have permission.

Yet I can browse into the USB Backups, Libvirt Backups folders without any issue.

Link to comment
3 hours ago, Whaler_99 said:

Permissions issue, maybe? If I browse to the folder where the Tower Backups are being stored, I can see the individual folders based on the dates completed. But if I try to browse into any of them, from my Windows computer, I get the error that I don't have permission.

Yet I can browse into the USB Backups, Libvirt Backups folders without any issue.

Works for me.  Does your user have access to the share?

Link to comment
17 minutes ago, Squid said:

Works for me.  Does your user have access to the share?

This is what I have for settings

 

image.png.0023d1cbfa7ff9adc79254952fb98a02.png

 

If I browse to \\tower\backups I can access USB and Libvirt folders fine, but not access the subfolders under "Tower Backups". My user does have read/write. I have not done anything for permissions other than what's done on the Shares page within unraid.  

Link to comment
  • 2 weeks later...

Hi there. So I somehow wiped my cache pool while upgrading from 2 x 500GB SSDs to 2 x 1TB SSDs (I did them one at a time, but here we are anyway). But when I try to restore from a backup, it had a bunch of errors and seemed to finish way too quickly. Here is the last of the message output:

"Backup / SettingsRestore AppdataBackup / Restore Status
Backup / Restore Status: Not Running
/usr/bin/tar: binhex-preclear/home/.icons/BLACK-Ice-Numix-FLAT: Directory renamed before its status could be extracted
/usr/bin/tar: binhex-preclear/home/.icons: Directory renamed before its status could be extracted
/usr/bin/tar: binhex-preclear/home: Directory renamed before its status could be extracted
/usr/bin/tar: binhex-preclear: Directory renamed before its status could be extracted
/usr/bin/tar: airsonic-advanced/transcode: Directory renamed before its status could be extracted
/usr/bin/tar: airsonic-advanced: Directory renamed before its status could be extracted
/usr/bin/tar: Exiting with failure status due to previous errors
Backup/Restore Complete. tar Return Value:
Restore finished. Ideally you should now restart your server
Backup / Restore Completed"

 

I've attached diagnostics if that helps. My most recent backup is from Jan 4th, that's the one I'm trying to restore, but I have backups from the 12 previous weeks.

I was able to copy the CA_backup.tar file to my Mac and unzip it, so I can copy everything back manually I hope? Just thought I'd ask first. Thanks for any help!

tower-diagnostics-20210109-1541.zip

Link to comment

The Exclude Folders option does not seem to be working for me.  I'm trying to exclude some subfolders in one of my docker container's appdata area (Plex for reference).  I've tried both absolute paths and relative paths.
Relative:

Plex-Media-Server/Library/Application Support/Plex Media Server/Cache/ ,Plex-Media-Server/Library/Application Support/Plex Media Server/Media/ ,Plex-Media-Server/Library/Application Support/Plex Media Server/Metadata/

Absolute:

/mnt/cache/appdata/Plex-Media-Server/Library/Application Support/Plex Media Server/Cache/ ,/mnt/cache/appdata/Plex-Media-Server/Library/Application Support/Plex Media Server/Media/ ,/mnt/cache/appdata/Plex-Media-Server/Library/Application Support/Plex Media Server/Metadata/


 

Link to comment

I just setup the CA Backup/Restore for 1st time.

 

Left all my Docker apps unchecked but binhex-krusader & DiskSpeed didn't get backed up, they are in a Folder done by Docker Folder plugin.

image.thumb.png.4a5c3be639d44dc8eb934adaab8bd168.png

 

How they look in my Docker tab.

image.png.6c799d1b1394e9f05690b67d1e9ef5ca.png

 

2nd question, does CA Backup/Restore back everything in the appdata share if I don't exclude any?

Even if that Docker is not currently unstalled? i.e. binhex-delugevpn isn't installed but I have a folder for if I installed again.

image.png.e81fdabf1c6fa7db781db9b659f94c82.png

Edited by Paul_Ber
Link to comment
On 12/28/2020 at 11:44 AM, Whaler_99 said:

So turns out I did update it yesterday, running version 2020.12.27 and backup ran this morning at 1:00 am and still cannot access unfortunately...

Update, last two backups I can now browse into the folder and see the ca_backup.tar files, but still messed up permissions. Cannot copy that file. Weird as the libvirt backup is fine, USB Backup is fine. Just the tower backups are inaccessible...

 

Link to comment

This has been requested atleast once before by @aptalca back in 2018 however this still doesn't seem to have been implemented?

 

Basically instead of one tarball of all the appdata directories, please could we have the option to have it seperated per directory? This will make a restore much easier for people who only want to restore one application. Currently they would have to manually untarball the file and move the directory.

 

So the feature request is:

 

Backup:

In the backup config page have a option called "Backup Directories Seperately"

If this option is selected it will backup each directory in the provided appdata directory into a time and dated folder within the provided backup directory.

In addition to this if the "Compression" option is ticked then compress each directory.

 

Restore:

When someone chooses the restore option, along with the date/time selection provide a directory selection to restore (using checkboxes to allow for multiple selections and an "all" option to restore everything)

 

Hopefully this is enough detail for the feature request to be actioned... @Squid

  • Like 4
Link to comment
  • Squid locked this topic
Guest
This topic is now closed to further replies.