[Plugin] Appdata.Backup

KluthR · April 15, 2023

3 minutes ago, isvein said:

I have some dockers i want to backup say ones a month and some I want to backup daily. 🤔

Multiple schedules is not available (yet).

anibis2 · April 15, 2023

1 minute ago, KluthR said:

No, that cant be the case, since its a completely new feature

Open the per container settings by clicking on its name. There you will find it.

Not explicitly enabling it means, it should be off. Im curious what its current state is.

Ah, maybe it was just backup to an external drive.

Regardless, the setting is set to no. Kinda hard to read with the coloring issues.

image.png.9969ac65b40e0741eec00abbb11cb7a8.png

KluthR · April 15, 2023

There will be a new update within the next 10 minutes. Could you update and run a manual backup and share the debug log again?

anibis2 · April 15, 2023

38 minutes ago, KluthR said:

There will be a new update within the next 10 minutes. Could you update and run a manual backup and share the debug log again?

That's definitely much better. Exclusions appear to be working as expected, as is the "Save external volumes" setting. The coloring is better as well.

I had a different issue when it tried to backup Grafana. The debug log is attached. Not sure if it's app related, or something with my configuration.

ab.debug.log

KluthR · April 15, 2023

You should exclude those for grafana as well:

/etc

/proc
/var/run/docker.sock

/var/run/utmp

/sys

as such are mapped to the container and would be backed up too (ext volume backup enabled)

anibis2 · April 15, 2023

4 hours ago, KluthR said:

You should exclude those for grafana as well:

/etc

/proc
/var/run/docker.sock

/var/run/utmp

/sys

as such are mapped to the container and would be backed up too (ext volume backup enabled)

When I try to exclude my Grafana data directory it slows down to a crawl. The .tar was growing slowly as time went on, and there was I/O activity on the disks. The Grafana tar was 500MB after 20 minutes of running Plex, which is 32GB and has many more files, takes about 90 seconds. I just turned off the external part on Grafana and it's working well now, takes about a minute.

*EDIT*

I wasn't paying attention with Grafana, I didn't realize how the docker path mappings were configured. I was excluding my DB's, but there were other container paths pointing to the root Unraid file system. I didn't catch that even though you pointed it out, but now I get it. The plugin is working fine for me nothing out of the ordinary.

Edited April 16, 2023 by darcon

KluthR · April 16, 2023

10 hours ago, darcon said:

pointing to the root Unraid file system.

I saw those mapping (/:/rootfs). Those end up being empty in the final tar command. With latest plugin update, those will be excluded from the mapping. Should not have before worked anyway.

Anym001 · April 16, 2023

I have an error in the backup log because of the volume mapping too. (SWAG Container)

My volume mappings:

/config -> /mnt/cache/appdata/swag
/var/log/nextcloud/ -> /mnt/cache/appdata/nextcloud/log/
/var/log/jellyfin/ -> /mnt/cache/appdata/Jellyfin/config/log/
/var/log/vaultwarden/vaultwarden.log -> /mnt/cache/appdata/vaultwarden/vaultwarden.log

Is there a way to do these mappings without getting an error message?

MothyTim · April 16, 2023

Hi, not sure what the problem is but since installing all 3 backups done so far say failed on them! Looks like its a couple of containers that are failing, hoping someone can help me work out the issue? Cheers Tim

ab.debug.log

KluthR · April 16, 2023

8 hours ago, Anym001 said:

I have an error in the backup log because of the volume mapping too

Please show the debug log.

1 hour ago, MothyTim said:

Hi, not sure what the problem is but since installing all 3 backups done so far say failed on them!

code-server has a volume mapping to appdata, which will be backed up. Is that mapping correct??

tautullis backup verification fails, because shfs is accessing the Logs (/mnt/cache/appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Logs)

Since tautulli has a mapping from Plex to backup and Plex is already running again, the files are being changed.

Anym001 · April 16, 2023

41 minutes ago, KluthR said:

9 hours ago, Anym001 said:

I have an error in the backup log because of the volume mapping too

Please show the debug log.

backup.debug.log

KluthR · April 16, 2023

@Anym001Duplicacy and jellyfin both having volumes within volumes (cache on each). This is working for docker but messes with the tar verification. The plugin script does not handle this currently. It will be, in future version.

Edited April 16, 2023 by KluthR

Anym001 · April 16, 2023

41 minutes ago, KluthR said:

@Anym001Duplicacy and jellyfin both having volumes within volumes (cache on each).

I solved this problems with die jellyfin and Duplicacy Container but not for the swag one.

Is there any solution for this or should I wait for the future version?

Edited April 16, 2023 by Anym001

MothyTim · April 16, 2023

4 hours ago, KluthR said:

Please show the debug log.

code-server has a volume mapping to appdata, which will be backed up. Is that mapping correct??

tautullis backup verification fails, because shfs is accessing the Logs (/mnt/cache/appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Logs)

Since tautulli has a mapping from Plex to backup and Plex is already running again, the files are being changed.

Thanks, yes need to map appdata to access config files? Can I stop backing up that bit? I don't think the old plugin did that? But this version has a lot more options, which is great! Thanks for your work! Same with Tautulli i guess? It'd be nice to be able to group apps I guess, so they stop and start together? Also to check for updates while each app/container is stopped for backup, rather than at the end? Thanks again, this is a great upgrade!

halorrr · April 16, 2023

I updated the plugin so it is now 2023.04.15a and now when I run the manual backup I get the message:

[16.04.2023 18:45:45][warning] Backing up EXTERNAL volumes, because its enabled!

So I aborted the backup, but from what I can tell it isn't enabled on any of my containers when I check them?

image.png.8b0bc6e333527ae08f43e7de1edcf264.png

My config.json for the plugin is:

{
    "destination": "/mnt/user/Backups/Appdata Backup",
    "allowedSources": "/mnt/user/appdata\r\n/mnt/cache/appdata",
    "compression": "no",
    "defaults": {
        "verifyBackup": "yes",
        "updateContainer": "no"
    },
    "flashBackup": "yes",
    "backupVMMeta": "yes",
    "deleteBackupsOlderThan": "7",
    "backupFrequency": "daily",
    "backupFrequencyWeekday": "4",
    "backupFrequencyDayOfMonth": "2",
    "backupFrequencyHour": "5",
    "backupFrequencyMinute": "0",
    "backupFrequencyCustom": ""
}

Anym001 · April 17, 2023

11 hours ago, KluthR said:

@Anym001Duplicacy and jellyfin both having volumes within volumes (cache on each). This is working for docker but messes with the tar verification. The plugin script does not handle this currently. It will be, in future version.

Sorry i posted the wrong debug log above.

Here is the current debug log, where the problems with duplicacy and jellyfin have been fixed.
Error for swag remains.

ab.debug.log

KluthR · April 17, 2023

11 hours ago, Anym001 said:

but not for the swag one

I dont see any errors for swag inside the log.

8 hours ago, MothyTim said:

Can I stop backing up that bit?

I dont think so. Adding the appdata path to exclusions would not backup any other appdata folder for that container. I think, I have to extend the plugins capabilities by that

7 hours ago, halorrr said:

So I aborted the backup, but from what I can tell it isn't enabled on any of my containers when I check them?

Thats the whole config.json? So, I guess you did not saved it for a long time. There are container parts missing like their config and the start order. Which is not a problem because you are working with defaults. But its worth mentioning.

Please confirm that this posted config IS the complete file content. If so, we have a bug here which prevents applying default defaults (yes) to older config versions. I will look into that.

59 minutes ago, Anym001 said:

Sorry i posted the wrong debug log above.

Ah I see.

swag backups contents of Jellyfin, maybe you want to exclude them (/mnt/cache/appdata/Jellyfin/log) as well as the other container mappings (vaultwarden?).

halorrr · April 17, 2023

Quote

Thats the whole config.json? So, I guess you did not saved it for a long time. There are container parts missing like their config and the start order. Which is not a problem because you are working with defaults. But its worth mentioning.

Please confirm that this posted config IS the complete file content. If so, we have a bug here which prevents applying default defaults (yes) to older config versions. I will look into that.

Yup that is the whole config.json

KluthR · April 17, 2023

Ok, thanks. I will fix all current issues within this week, including your config issue. Please leave your snippet here. You can try to save the settings once, that should do it for you.

It will take some days as Iam a bit busy currently.

Stay tuned!

halorrr · April 17, 2023

2 hours ago, KluthR said:

Ok, thanks. I will fix all current issues within this week, including your config issue. Please leave your snippet here. You can try to save the settings once, that should do it for you.

It will take some days as Iam a bit busy currently.

Stay tuned!

Saving definitely added a lot more to my config and I was able to successfully run a backup.

One issue I did notice is that my verification of tautulli failed. Looking at the debug for it:

I was able to pick out that it was backing up a plex folder as part of the tautulli backup. This seems to be because I have a /plexlogs mapping so that it is easy to tell tautulli where my plex logs live. The plugin considers this an Internal volume (I'm guessing that currently the logic considers anything in /appdata internal, even if it belongs to a different app?):
image.png.04be75f53ec958317f119606da9e5f8a.png

Also worth nothing that the plugin doesn't deduplicate volumes from the list if multiple container folders have been mapped to the same path.

Anym001 · April 18, 2023

22 hours ago, KluthR said:

23 hours ago, Anym001 said:

Sorry i posted the wrong debug log above.

Ah I see.

swag backups contents of Jellyfin, maybe you want to exclude them (/mnt/cache/appdata/Jellyfin/log) as well as the other container mappings (vaultwarden?).

I have now tried to exclude the affected mappings.
Unfortunately I still get an error message.
The tar verifications failed again.

Debug log attached

ab.debug.log

KluthR · April 18, 2023

9 hours ago, halorrr said:

even if it belongs to a different app?

Yes, because I cant tell which mapping originates from which container.

9 hours ago, halorrr said:

Also worth nothing that the plugin doesn't deduplicate volumes from the list if multiple container folders have been mapped to the same path.

Exact same reason

I could remove a mapping once it appeared but it could be backed up within wrong container context which causes confusion.

2 hours ago, Anym001 said:

I have now tried to exclude the affected mappings.
Unfortunately I still get an error message.

hmm. Have to check tars behavior here. Maybe I have to adjust some details.

halorrr · April 19, 2023

In a failed verification, the error I got emailed contained non-error lines:

Here is the debug log for the associated failure:

ab.debug (1).log

halorrr · April 19, 2023

Also on:

Quote

Yes, because I cant tell which mapping originates from which container.

Can we get an option to manually designate a mapping as external?

KluthR · April 19, 2023

3 hours ago, halorrr said:

Can we get an option to manually designate a mapping as external?

For what reason? To exclude it? If yes, the exclusion filter will adapt a more things in the next update.

[Plugin] Appdata.Backup

Recommended Posts

Link to comment

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

KluthR

KluthR

KluthR

Posted Images

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation