[Plugin] Appdata.Backup

[schreibman]

On 2/27/2024 at 4:53 AM, KluthR said:

Known an I (we) are on it. Currently collecting debug info from affected users as I cant reproduce it on my end.

 

have 6.12.8 w/ 

WARN: Multi mapping detected!

attached "tower-diagnostics"

tower-diagnostics-20240303-1657.zip

[BartyB]

3 hours ago, ConnerVT said:

Known bug.  He's working on it.

oh whoops. I did not realize it's a known bug. By the paragraph I quoted I thought it was a feature and I just had to exclude certain path containers. Am I ok to use the plugin or does this bug cause issues with the backup? 

[KluthR]

Though its a bug, it seems like @BartyB sees a part of working warning. The data dir IS used in more than one container, so excluding it should do the trick.

 

@schreibmanIn this case I dont need Unraids diagnostics. It would be great if you could PM me, I need the output of your browsers developer console (F12).

[Sptz87]

On plex container backup, on Library/Application Support/Plex Media Server/Cache I understand it's something that's just recreated easily.

 

But if I have a crap ton of media that takes days to recreate video preview thumbnails etc and if I also want to keep Posters etc, is there any logical reason not to backup the below?

 

Library/Application Support/Plex Media Server/Media

Library/Application Support/Plex Media Server/Metadata

 

[ConnerVT]

9 hours ago, KluthR said:

Though its a bug, it seems like @BartyB sees a part of working warning. The data dir IS used in more than one container, so excluding it should do the trick.

 

 

I respectfully disagree.  The plugin is "Backup/Restore Appdata", not "Backup Docker Ecosystem".  I cannot think of a use case I would wish for this plugin to back up the 20+ TB of files in my Plex Media share.

 

This message should only be applied to check paths which are explicitly listed in the plugin's Main Setting Appdata source(s).  If there is something other than that not in the conventional appdata location, this explicit location can also be included here, and then would be captured by the plugin and generate a warning (if used by multiple containers).

 

Anyone using the arrs (likely the most common usage by Unraid servers) will certainly have multiple containers which point to the same location/path.

[KluthR]

I fully understood now, what the issue was. I must be blind

 

However: The fix includes supports now to respect excludes as well as skipping external volumes (if the per-container setting does not wish to backup those). The two new ideas are featured by noirsoldats.

 

For anyone who wants to test the fix on their affected setup: I created a small script that patches the file in question. Its a one-liner and it will do the job for you. Running twice will undo the fix!

 

Open a terminal and paste the WHOLE LINE:

curl -s https://raw.githubusercontent.com/Commifreak/unraid-appdata.backup/volume_dup_check_fix/vol_dup_fix.sh | bash

 

Please reload the settings page and check if the warnings are now gone or - if shown - correct!

[KluthR]

On 3/3/2024 at 1:23 AM, Sptz87 said:

Since it looks like this plugin isn't attempting to create the folder if it's missing, just populate it

It should print a red, ylear error then:

 

But yes, it could create the dir. Noted.

 

On 3/3/2024 at 4:53 PM, tenpaiyomi said:

I figured I could just add that directory to the global exclusions, but when I started a manual backup and it got to the first container which included that directory, it did not include the message about that particular folder matching an exclusion and included it in the calculated volumes to be backed up.  Only when I aborted and then added the exclusion to each container exclusion individually did it

Damn. My fault. The global exclusion list gets ignored during container mapping determination. It will be checked in the next update. But note: The global exclusion list also supports tar's exclusion syntax, but only exact path matches are supported!

[abhi.ko]

 I'm getting a lot of

tar verification failed 

errors in the log after my first run (switched from old plugin just last week). Some of the containers don't have the errors, but most of them do.

 

Not sure what if anything I can do to resolve these? I am also confused about what the previous errors referred to in the due to previous errors statement is.

 

Log snippet below:

[04.03.2024 02:38:10][ℹ️][radarr] Stopping radarr...  done! (took 4 seconds)
[04.03.2024 02:38:14][ℹ️][radarr] Should NOT backup external volumes, sanitizing them...
[04.03.2024 02:38:14][ℹ️][radarr] Calculated volumes to back up: /mnt/user/appdata/radarr
[04.03.2024 02:38:14][ℹ️][radarr] Backing up radarr...
[04.03.2024 02:38:14][ℹ️][radarr] Backup created without issues
[04.03.2024 02:38:14][ℹ️][radarr] Verifying backup...
[04.03.2024 02:38:14][❌][radarr] tar verification failed! Tar said: tar: /mnt/user/appdata/radarr: Not found in archive; tar: Exiting with failure status due to previous errors
[04.03.2024 02:38:29][ℹ️][radarr] Starting radarr... (try #1)  done!
[04.03.2024 02:38:31][ℹ️][Plex-Media-Server] Stopping Plex-Media-Server...  done! (took 7 seconds)
[04.03.2024 02:38:38][ℹ️][Plex-Media-Server] '/mnt/user/appdata/Plex-Media-Server/Transcode' is within mapped volume '/mnt/user/appdata/Plex-Media-Server'! Ignoring!
[04.03.2024 02:38:38][ℹ️][Plex-Media-Server] Should NOT backup external volumes, sanitizing them...
[04.03.2024 02:38:38][ℹ️][Plex-Media-Server] Calculated volumes to back up: /mnt/user/appdata/Plex-Media-Server
[04.03.2024 02:38:38][ℹ️][Plex-Media-Server] Backing up Plex-Media-Server...
[04.03.2024 02:38:38][ℹ️][Plex-Media-Server] Backup created without issues
[04.03.2024 02:38:38][ℹ️][Plex-Media-Server] Verifying backup...
[04.03.2024 02:38:38][❌][Plex-Media-Server] tar verification failed! Tar said: tar: /mnt/user/appdata/Plex-Media-Server: Not found in archive; tar: Exiting with failure status due to previous errors
[04.03.2024 02:47:16][ℹ️][Plex-Media-Server] Starting Plex-Media-Server... (try #1)  done!
[04.03.2024 02:47:19][ℹ️][nzbget] Stopping nzbget...  done! (took 5 seconds)
[04.03.2024 02:47:24][ℹ️][nzbget] Should NOT backup external volumes, sanitizing them...
[04.03.2024 02:47:24][ℹ️][nzbget] Calculated volumes to back up: /mnt/user/appdata/nzbget
[04.03.2024 02:47:24][ℹ️][nzbget] Backing up nzbget...
[04.03.2024 02:47:24][ℹ️][nzbget] Backup created without issues
[04.03.2024 02:47:24][ℹ️][nzbget] Verifying backup...
[04.03.2024 02:47:24][❌][nzbget] tar verification failed! Tar said: tar: /mnt/user/appdata/nzbget: Not found in archive; tar: Exiting with failure status due to previous errors
[04.03.2024 02:47:28][ℹ️][nzbget] Starting nzbget... (try #1)  done!
[04.03.2024 02:47:30][ℹ️][sabnzbd] Stopping sabnzbd...  done! (took 4 seconds)
[04.03.2024 02:47:34][ℹ️][sabnzbd] Should NOT backup external volumes, sanitizing them...
[04.03.2024 02:47:34][ℹ️][sabnzbd] Calculated volumes to back up: /mnt/user/appdata/sabnzbd/config
[04.03.2024 02:47:34][ℹ️][sabnzbd] Backing up sabnzbd...
[04.03.2024 02:47:34][ℹ️][sabnzbd] Backup created without issues
[04.03.2024 02:47:34][ℹ️][sabnzbd] Verifying backup...
[04.03.2024 02:47:34][❌][sabnzbd] tar verification failed! Tar said: tar: /mnt/user/appdata/sabnzbd/config: Not found in archive; tar: Exiting with failure status due to previous errors
[04.03.2024 02:47:38][ℹ️][sabnzbd] Starting sabnzbd... (try #1)  done!
...
...
...
[04.03.2024 04:04:11][ℹ️][Main] Backing up the flash drive.
[04.03.2024 04:05:23][ℹ️][Main] Flash backup created!
[04.03.2024 04:05:23][ℹ️][Main] Copying the flash backup to '/mnt/user/Other Media/unRAID/Flash Backup/' as well...
[04.03.2024 04:05:28][ℹ️][Main] VM meta backup enabled! Backing up...
[04.03.2024 04:05:28][ℹ️][Main] Done!
[04.03.2024 04:05:28][⚠️][Main] An error occurred during backup! RETENTION WILL NOT BE CHECKED! Please review the log. If you need further assistance, ask in the support forum.
[04.03.2024 04:05:28][ℹ️][Main] DONE! Thanks for using this plugin and have a safe day ;)
[04.03.2024 04:05:28][ℹ️][Main] ❤️

 

Here is the debug log id for a more recent manually triggered backup with same errors but different timestamps - f73d93f6-bece-4fe4-97b6-8477b9440833

[ConnerVT]

2 hours ago, KluthR said:

I fully understood now, what the issue was. I must be blind

 

No worries.  One of my roles at work is supporting a production application used in multiple sites around the world, with developers and QA testers in yet another country.  Each week we have a Teams meeting with User/Site support, Program Management and the Dev team.  We all look at the application in a slightly different way, and sometimes it takes a few tries for someone to see what you are seeing.

 

I ran the patch, and it no longer displays the warnings related to the External Volumes (which are not part of the backup).  I can't run a backup at this time, as I shut down Docker and need my services to stay up.  But I have high hopes all will work as expected.

 

Thanks for finding a resolution, as well as for your work on this important plugin.

[whitephoenix117]

I'm have all Docker's disabled on purpose. Any chance of disabling this error, or at least making it configurable? It ends up emailing me every time it runs.

 

[jcofer555]

would be nice to have a keep only so many flash backups when using a seperate folder for the flash backup to go. also a way to not have the flash backup also go in the appdata backup path when you have it set to a seperate folder

Edited March 5 by jcofer555

[KluthR]

7 hours ago, ConnerVT said:

I can't run a backup at this time

Not an issue as well because there were no changes.

 

3 hours ago, whitephoenix117 said:

I'm have all Docker's disabled on purpose. Any chance of disabling this error, or at least making it configurable?

The main purpose of this plugin is docker backup. Using it just for Flash backup is actually working but not intended. I will see what I can do.

 

10 hours ago, abhi.ko said:

Here is the debug log id for a more recent manually triggered backup with same errors but different timestamps

You added "/mnt/user" in many containers to exclude. Since I work with tar's exclusions syntax, Iam not always able to 100% check things for logical correctness. However, you should fix the exclusions to NOT exclude whole "/mnt/user" and want the backup to include "/mnt/user/appdata/*" things.

[BartyB]

17 hours ago, KluthR said:

I fully understood now, what the issue was. I must be blind

 

However: The fix includes supports now to respect excludes as well as skipping external volumes (if the per-container setting does not wish to backup those). The two new ideas are featured by noirsoldats.

 

For anyone who wants to test the fix on their affected setup: I created a small script that patches the file in question. Its a one-liner and it will do the job for you. Running twice will undo the fix!

 

Open a terminal and paste the WHOLE LINE:

curl -s https://raw.githubusercontent.com/Commifreak/unraid-appdata.backup/volume_dup_check_fix/vol_dup_fix.sh | bash

 

Please reload the settings page and check if the warnings are now gone or - if shown - correct!

 

sorry for the dumb question but is this the fix for my issue of getting the "used in multiple containers" warning? (new user trying to follow and understand)

[KluthR]

Yes. U can try the hotfix if wanted.

[abhi.ko]

13 hours ago, KluthR said:

You added "/mnt/user" in many containers to exclude. Since I work with tar's exclusions syntax, Iam not always able to 100% check things for logical correctness. However, you should fix the exclusions to NOT exclude whole "/mnt/user" and want the backup to include "/mnt/user/appdata/*" things.

Thank you for a taking a look. I am sure it is user error.

 

The reason why I excluded that was in an effort to get rid of the Multi-mapping detected warning. It did not help, but any pointers on what I can do differently to avoid the warning and have the backup run without errors?

 

 

Are these containers still backed up even though they failed verification since I am getting a Backup created without issues message right before the verification starts? 

 

[Kilrah]

19 minutes ago, abhi.ko said:

The reason why I excluded that was in an effort to get rid of the Multi-mapping detected warning.

The warning didn't work and had erroneous detections, see a few posts up.

 

 

[oddz]

Where is the documentation for performing a restore? I'm not seeing it on the Github repo. It just points me to this thread. But I don't see it at the start of this thread either.

https://github.com/Commifreak/unraid-appdata.backup

[abhi.ko]

31 minutes ago, Kilrah said:

The warning didn't work and had erroneous detections, see a few posts up.

 

 

Thanks for pointing that out. 

[KluthR]

1 hour ago, Kilrah said:

The warning didn't work and had erroneous detections, see a few posts up.

Did anyone tested the fix

[abhi.ko]

1 hour ago, KluthR said:

Did anyone tested the fix

I am happy to - is there a beta plg file or something that I need to download?

BTW - I removed the exclusions to /mnt/user/ and it ran without errors.

[BartyB]

3 hours ago, KluthR said:

Did anyone tested the fix

I just installed the hot fix and it appears to fix the errors. Although I have not run a actual backup yet since I am still trying to figure out/understand the plex exclusion suggestions. 

 

 

edit: I just ran the back up and I did not see any errors in the log. As far as I can tell this ran flawlessly! 

 

I had to do a reboot and the errors returned. I am assuming this is as expected? 

Edited March 5 by BartyB

[ConnerVT]

3 hours ago, KluthR said:

4 hours ago, Kilrah said:

The warning didn't work and had erroneous detections, see a few posts up.

Did anyone tested the fix

 

I have.  Ran a backup early this morning, all works as expected.  I had installed your patch yesterday.

[KluthR]

8 hours ago, oddz said:

Where is the documentation for performing a restore?

There is none. The restore tab should be self explanatory.

[Kilrah]

11 hours ago, KluthR said:

Did anyone tested the fix

Yes, all the duplicate warnings disappeared which seems correct.

[monarc]

Hey all. I just noticed that failed backup jobs won't be deleted if they are older than specified in the configuration. Is this an expected or unexpected behaviour?