[Plugin] Appdata.Backup

[MammothJerk]

2 hours ago, Jaycedk said:

Hi there.

First of thanks for making this plugin.

I keep getting backup failed, but can't seem to figure out why.

Can I get someone to look at this log ?

 

Thanks i advance

backup.debug.log 161.8 kB · 0 downloads

[17.06.2023 10:14:07][GoAccess-NPM-Logs][debug] Tar out: tar: /mnt/user/appdata/Nginx-Proxy-Manager-Official/data/logs/proxy-host-2_access.log: file changed as we read it; tar: /mnt/user/appdata/Nginx-Proxy-Manager-Official/data/logs/proxy-host-3_access.log: file changed as we read it
[17.06.2023 10:14:07][GoAccess-NPM-Logs][error] tar creation failed! More output available inside debuglog, maybe.

 

[Jaycedk]

51 minutes ago, MammothJerk said:

[17.06.2023 10:14:07][GoAccess-NPM-Logs][debug] Tar out: tar: /mnt/user/appdata/Nginx-Proxy-Manager-Official/data/logs/proxy-host-2_access.log: file changed as we read it; tar: /mnt/user/appdata/Nginx-Proxy-Manager-Official/data/logs/proxy-host-3_access.log: file changed as we read it
[17.06.2023 10:14:07][GoAccess-NPM-Logs][error] tar creation failed! More output available inside debuglog, maybe.

 

Thank you, don't know how I missed that

[dopeytree]

Hi did a appdata backup using this new plugin in 6.12.

 

I then created some a new ZFS pool for the ssd date like appdata.

 

Have now gone to restore / set up docker folder on the new zfs share.

 

This app is refusing to restore the backup as not in gzip format. (if this is required for the plugin to work please remove the compression option in the settings)

 

At the moment this plugin does do a backup but cannot restore said backup.

 

[17.06.2023 15:16:12][Main][debug] Final tar command: tar -C '/' -x -f '/mnt/disk1/backups/appdata/ab_20230615_101252/binhex-get-iplayer.tar' -z
[17.06.2023 15:16:12][Main][debug] Tar out: ; gzip: stdin: not in gzip format; tar: Child returned status 1; tar: Error is not recoverable: exiting now
[17.06.2023 15:16:12][Main][error] restore failed! More output available inside debuglog, maybe.

 

Edited June 17, 2023 by dopeytree

[KluthR]

10 minutes ago, dopeytree said:

This app is refusing to restore the backup as not in gzip format.

Restoring uncompressed backup is currently broken. See second post, known bug which I fix after vacation.

Edited June 17, 2023 by KluthR

[dopeytree]

Cool - enjoy your holiday 🍻

[leige]

Hi

Thanks in advanced

 

 

I think the plugin has a bug. I have a script run 2 times that only runs the first time. you can see that in "Pre-backup script" the script is executed, but in "Post-backup script" it is not executed.

ab.debug.log

[KluthR]

Interesting. Will check that as well after vacation.

[Renegade605]

Just installed after 6.12 migration. Here are some notes/issues I've put together:

 

Button UX

The Save button's location makes it unclear that it is for the whole page. Since it's under a heading, global Unraid UX design suggests it's only for saving the additional backup paths, not the whole page.

Suggest placing a Save button at the bottom of every section (and/or in the detailed setup of every container).

 

The Discard button is always active, making it unclear if changes have been made or not.

Suggest disabling (or hiding) the Discard button until changes have been made.

 

The Manual Backup button is always enabled, so it's unknown if changes will be saved before running.

Suggest disabling (or hiding) the Manual Backup button if changes are detected.

 

Folder picker UX

If a field with a folder picker is clicked, the folder list appears and cannot be dismissed until the Add To List button is clicked. If deciding not to add folders (or for example, we select the field to remove the folder from the list) the folder picker gets stuck unless we click the Add To List button with nothing selected.

Suggest dismissing the folder picker when the field loses focus or adding a cancel button.

 

General Note location

Suggest moving this note.

 

Backup/update order

Backup is executed then updates, meaning containers with updates will stop and restart twice. (Stop, backup, start for each container ON)

Suggest that checking for updates should occur after backup but before restart for each container.

 

Excluded directories in logs

Excluded directories appear to only be in logs if they are a root container path. Eg. /mnt/user/appdata/container/ is a path, but /mnt/user/appdata/container/big_unnecessary_directory/ is excluded, no messages in the log indicate that is the case.

 

Ambiguity in 'skip'

It's unclear if skip container will skip both the backup and the update, or just the backup.

 

Backup folder structure

CA Appdata Backup allowed for putting flash and docker backups in different folders. Suggest restoring this functionality or making subfolders for these backups (plus vm_meta).

Saving the my-container.xml is nice, but makes navigating a backup strange due to sorting. Suggest putting this in the tarball or making subfolders for each container.

 

Backup folder name

Suggest a more human readable folder name. ie. ab_20230618_150259 -> [email protected]

 

libvrt.img

CA Appdata Backup also backed up libvrt.img while this does not. The backup of /etc/libvirt/qemu is new. Are these comparable?

 

A nice to have

Allow choosing the order of stop, backup, update, and restart manually. ie. some containers depend on others, so ideal order could be:

- Stop container A

- Backup A

- Update A

- Restart A

- Stop D, C, B

- Backup B, C, D

- Update B, C, D

- Restart B, C, D

- etc.

 

Minor tweak to log info

Many log messages say hopefully, or probably. Worth a chuckle, but potentially frightening to an end user.

Suggest that if errors can be detected, be specific, otherwise don't bother saying this.

Edited June 18, 2023 by Renegade605

[House Of Cards]

Any chance this can add options like the original had to select directories for different things.  For example, to specify the location of the flash backup that is different from the docker location?  The old version also backed up the VM image and let you put it somewhere different.

 

Thanks for all the hard work!

[Nogami]

I have a few dockers with data inside the appdata folder, however getting an error that there's nothing inside the volume and it refuses to back anything up.

 

[18.06.2023 19:18:20][jackett][info] Stopping jackett...
[18.06.2023 19:18:25][jackett][info] done! (took 5 seconds)
[18.06.2023 19:18:25][jackett][debug] Backup jackett - Container Volumeinfo: Array
(
    [0] => /mnt/apps/jackett_downloads/:/downloads:rw
    [1] => /mnt/user/appdata/jackett:/config:rw
)

[18.06.2023 19:18:25][jackett][debug] usorted volumes: Array
(
    [0] => /mnt/user/appdata/jackett
    [1] => /mnt/apps/jackett_downloads
)

[18.06.2023 19:18:25][jackett][info] Should NOT backup external volumes, sanitizing them...
[18.06.2023 19:18:25][jackett][info] jackett does not have any volume to back up! Skipping
[18.06.2023 19:18:25][jackett][info] Starting jackett... (try #1)

 

/mnt/apps/jacket_downloads is indeed empty

 

for

 

/mnt/user/appdata/jackett, there's 21MB of files and riirectories in there.

 

Note that if I enable "save external volumes" for that docker it does back it up, however the data is inside the appdata folder, so it shouldn't show as an external volume, should it?

 

Maybe related to having my appdata on a ZFS drive in 6.12?

 

One other small question - it's creating backups with user root, group root.  Shouldn't it be nobody/users when creating on shares?

Edited June 19, 2023 by Nogami

[Kilrah]

Did you mess with the "Appdata Sources" setting?

[Omen]

There are no global excludes, right?

[KluthR]

16 hours ago, Renegade605 said:

Just installed after 6.12 migration. Here are some notes/issues I've put together:

Thanks for the input have to review it later. Iam on vacation currently.

one word to linvirt.img: linvirt.img is the image behind the qemu folder. So I decoded to change the way how to backup it.

 

14 hours ago, House Of Cards said:

Any chance this can add options like the original had to select directories for different things.

What would be the benefit?

 

9 hours ago, Nogami said:

I have a few dockers with data inside the appdata folder, however getting an error that there's nothing inside the volume and it refuses to back anything up.

as @Kilrah said. And: Please review the info help block for the sources config option. I believe this answers your question.

 

(Sidenote: I see that I have to improve the onboarding process)

 

5 hours ago, Omen said:

There are no global excludes, right?

Not yet. See the first posts (known issues, planned features).

Edited June 19, 2023 by KluthR

[Sacred]

@KluthR can you possibly design your code to be multi-language capable?

 

Currently, you write your texts hard where you want them to be. If you had them in a language file, I'm sure there would be users here and there who would help with a translation.  😁

 

I know that this is not a dramatically important thing. But somehow it would be nice if that were possible for the future.

Now, as long as the code is still "small", it is easier to implement. 😅

[itimpi]

3 hours ago, Sacred said:

@KluthR can you possibly design your code to be multi-language capable?

 

Currently, you write your texts hard where you want them to be. If you had them in a language file, I'm sure there would be users here and there who would help with a translation.  😁

 

I know that this is not a dramatically important thing. But somehow it would be nice if that were possible for the future.

Now, as long as the code is still "small", it is easier to implement. 😅

It is quite easy to convert a plugin that has hard-coded text strings to one that has them in an external file that can form the basis of translation.   Limetech has a document on the steps involved.   The plugin author does not need to understand any of the foreign language as it is the English text that acts as the key to the relevant strings in the translation file for any other language. 
 

My guess is that as soon as the plugin appears to be stable that this activity will be taken to externalize all strings needed.   If not done directly by the plugin maintainer then third parties can easily submit pull requests to add this capability.

[KluthR]

1 hour ago, itimpi said:

My guess is that as soon as the plugin appears to be stable

It should BE stable 😂. Betas are gone.

 

but I already have a branch for i18n and already know how the i18n system inside unraid is working. I dont know why its an own system (apart from gettext) but ok. The next weeks should bring up full i18n support as well as german translation.

[House Of Cards]

On 6/19/2023 at 10:08 AM, KluthR said:

| 

|  Any chance this can add options like the original had to select directories for different things.

| 

 

What would be the benefit?

 

One benefit is that it would be cleaner, and allow more flexibility.

 

For example, I used to keep the Flash Drive in a separate folder and sync that folder to another location.  The reason is that because if UnRAID's flash drive dies, it would be more difficult to gain access to the backup on the server because I wouldn't be able to boot the server until I fixed the flash drive.  In this scenario I'd have a flash copy somewhere else that is always up to date.

 

The way it is now I'd have to keep all the docker backups and everything in sync just to have the latest flash drive, because the plugin keeps all the flash backups in separate folders for each backup version.  Same with the VM image.  It just makes recovery a lot easier, and lets us selectively archive things if we'd like.

 

Thanks for all the work you do!

[Kilrah]

Your script could only copy *.zip from the directory.

 

I sync the whole appdata backup anyway so it could be restored on a completely different machine if necessary.

[el_fredo]

On 6/19/2023 at 4:08 PM, KluthR said:

as @Kilrah said. And: Please review the info help block for the sources config option. I believe this answers your question.

 

(Sidenote: I see that I have to improve the onboarding process)

That would be great, because I still don't get it, unfortunately.

 

I only use the cache pool in Unraid, no more array. All my shares (thus also the appdata folder) are set to "Exclusive". So I don't use the FUSE layer.

 

How do I have to set the source in the plugin? Is it /mnt/user/appdata/ or is it /mnt/cache-disk/appdata/ ? By the way, I'm not interested in backing up any external volumes...

 

Also, the matter of "VM meta" is not quite clear. Does it back up all VMs? I used to use the plugin "VM Backup" for this, but it disappeared on my settings page (maybe incompatible with Unraid 6.12)

[Kilrah]

/mnt/user is provided by FUSE, so if you bypass it for everything you might want to use /mnt/cache-disk/appdata.

If you can see the contents by going to /mnt/user/appdata it should work, in which case there may be something to check in the plugin when a share is exclusive due to the symlink.

 

2 hours ago, el_fredo said:

Also, the matter of "VM meta" is not quite clear. Does it back up all VMs?

That backs up the VM XMLs, not the vdisks.

 

2 hours ago, el_fredo said:

I used to use the plugin "VM Backup" for this, but it disappeared on my settings page (maybe incompatible with Unraid 6.12)

I still have it on 6.12 and it did work as normal on its last run.

Edited June 22, 2023 by Kilrah

[Aman Saroha]

Getting this error at the end while trying to backup.ab.log

[el_fredo]

19 hours ago, Kilrah said:

I still have it on 6.12 and it did work as normal on its last run.

Indeed. I had to re-install the plugin and now it's back in my settings page.

[Gibbles0779]

I want to say thank you for all your hard work on this awesome plugin. This little gem has saved me from myself more times than i care to admit.

 

Keep up the great work!

[ramair02]

I am on 6.12.1. I didn't realize the CA Backup plugin was deprecated. My cache drive is corrupt / failed (Unmountable: Unsupported or no file system).

 

I only have backups from CA Backup plugin (from 6.11.5), not from your new plugin. I understand from your pinned post that the CA backups cannot be restored with your new Appdata Backup plugin. Is there a workaround? 

 

I was also able to mount the failing cache drive to /temp and copy off all of the contents. Perhaps that is useful in rebuilding a new cache drive? 

Edited June 23, 2023 by ramair02

[Renegade605]

Just now, ramair02 said:

I am on 6.12.1. I didn't realize the CA Backup plugin was deprecated. My cache drive is corrupt / failed.

 

I only have backups from CA Backup plugin (from 6.11.5), not from your new plugin. I understand from your pinned post that the CA backups cannot be restored with your new Appdata Backup plugin. Is there a workaround? 

 

I was also able to mount the failing cache drive to /temp and copy off all of the contents. Perhaps that is useful in rebuilding a new cache drive? 

You can manually access CA backup files and extract them. Basically just copy paste all the files to restore.