[Plugin] Appdata.Backup

[greenflash24]

Hi,

 

i am trying to run a Post-backup script (from the User-Scripts plugin), but the backup failed, because the script does not exists, which is not true, see:

 

[01.01.2024 03:29:34][❌][Main] /boot/config/plugins/user.scripts/scripts/my-script2/script is not existing! Skipping!

 

root@server:~# file /boot/config/plugins/user.scripts/scripts/my-script2/script
/boot/config/plugins/user.scripts/scripts/my-script2/script: Bourne-Again shell script, ASCII text executable

root@Home-Server:~# ls -la /boot/config/plugins/user.scripts/scripts/my-script2/script
-rw------- 1 root root 229 Dec 30 18:32 /boot/config/plugins/user.scripts/scripts/my-script2/script

 

 

Am I doing anything wrong with my configuration?

Edited January 1 by greenflash24

[Kilrah]

Scripts on the boot flash are not executable (FAT32 storage doesn't support linux permissions). The User Script plugin copies scripts to /tmp/user.scripts/tmpScripts/ before running them. 

[warwolf7]

On 12/30/2023 at 2:51 PM, warwolf7 said:

Hi,

I've set this appdatabackup plugin a couple of month ago, it was working fine until a few backups ago, maybe 2-3weeks.

I get this error, tar verification failed

[30.12.2023 05:13:10][ℹ️][Dropbox-by-otherguy] Stopping Dropbox-by-otherguy... done! (took 10 seconds)
[30.12.2023 05:13:20][ℹ️][Dropbox-by-otherguy] Should NOT backup external volumes, sanitizing them...
[30.12.2023 05:13:20][ℹ️][Dropbox-by-otherguy] Calculated volumes to back up: /mnt/user/appdata/dropbox
[30.12.2023 05:13:20][ℹ️][Dropbox-by-otherguy] Backing up Dropbox-by-otherguy...
[30.12.2023 05:28:07][ℹ️][Dropbox-by-otherguy] Backup created without issues
[30.12.2023 05:28:07][ℹ️][Dropbox-by-otherguy] Verifying backup...
[30.12.2023 05:32:48][❌][Dropbox-by-otherguy] tar verification failed! Tar said: tar: Removing leading `/' from member names; mnt/user/appdata/dropbox/events/store: Contents differ
[30.12.2023 05:32:48][ℹ️][Dropbox-by-otherguy] Starting Dropbox-by-otherguy... (try #1) done!

all the other containers are backedup properly and succed. Only this one is giving me trouble. 

 

I did sent the error log to the developper, this is the ID "4024f2b1-e1ae-4c4b-b225-5db3bfbbfc99"

I excluded the folder dropbox/events from the backup and everything went back to normal. That folder was pretty big (80Gb according to "du -sh" however the backup was never that big. close to 8Gb and is now under 500mb )

[jj_uk]

On 12/23/2023 at 8:50 AM, KluthR said:

The log os copied as last step I think. But the missing config file is interesting. Let me test some things.

 

Is the config file needed?

[Cornflake]

Hey

is it possible to get a notification when the backup process starts ?

For my case the backup process takes 4 hours and I am always confused to see my services offline forgetting that it’s currently doing a backup

 

happy new year

[Dustjunky]

I am backing up to an SMB share using this plugin and I am finding that after each backup anything older than my set retention is placed into a .recycle folder.  These backups are being kept indefinitely, causing my backup share to fill up.  I don't have any separate recycle bin plugins installed and I can't see any settings in the plugin for this.  What am I missing?

[trurl]

50 minutes ago, Dustjunky said:

backing up to an SMB share using this plugin and I am finding that after

Do you mean by using the Remote Shares feature of Unassigned Devices plugin? 

[kstrike155]

I have been getting errors for days/weeks with my backups saying that there's no space left on device, when I have over 1 TB free.

 

Debug key 372c1d4e-122a-4194-bf1c-a9189c65da2e

 

Example:

[02.01.2024 09:50:25][ℹ️][traefik] Stopping traefik... done! (took 13 seconds)
[02.01.2024 09:50:38][ℹ️][traefik] Should NOT backup external volumes, sanitizing them...
[02.01.2024 09:50:38][ℹ️][traefik] Calculated volumes to back up: /mnt/user/appdata/homelab/traefik-docker/certs, /mnt/user/appdata/homelab/traefik-docker/rules.yaml, /mnt/user/appdata/homelab/traefik-docker/letsencrypt
[02.01.2024 09:50:38][ℹ️][traefik] Backing up traefik...
[02.01.2024 09:50:38][❌][traefik] tar creation failed! Tar said: tar (child): /mnt/user/backups/homer/mnt/user/appdata/ab_20240102_094256/traefik.tar.gz: Cannot open: No space left on device; tar (child): Error is not recoverable: exiting now; tar: /mnt/user/backups/homer/mnt/user/appdata/ab_20240102_094256/traefik.tar.gz: Wrote only 4096 of 10240 bytes; tar: Child returned status 2; tar: Error is not recoverable: exiting now
[02.01.2024 09:50:40][ℹ️][traefik] Starting traefik... (try #1) done!
[02.01.2024 09:50:45][ℹ️][unifi] Stopping unifi... done! (took 6 seconds)
[02.01.2024 09:50:51][ℹ️][unifi] Should NOT backup external volumes, sanitizing them...
[02.01.2024 09:50:51][ℹ️][unifi] Calculated volumes to back up: /mnt/user/appdata/unifi, /mnt/user/appdata/homelab/unifi-docker/config.gateway.json
[02.01.2024 09:50:51][ℹ️][unifi] Backing up unifi...
[02.01.2024 09:50:51][❌][unifi] tar creation failed! Tar said: tar (child): /mnt/user/backups/homer/mnt/user/appdata/ab_20240102_094256/unifi.tar.gz: Cannot open: No space left on device; tar (child): Error is not recoverable: exiting now; tar: /mnt/user/backups/homer/mnt/user/appdata/ab_20240102_094256/unifi.tar.gz: Wrote only 4096 of 10240 bytes; tar: Child returned status 2; tar: Error is not recoverable: exiting now
[02.01.2024 09:50:53][ℹ️][unifi] Starting unifi... (try #1) done!

 

But looking at free space:

root@homer:~# df -h /mnt/user/backups/homer/mnt/user/appdata/
Filesystem      Size  Used Avail Use% Mounted on
shfs             11T   10T 1002G  92% /mnt/user

 

[Kilrah]

12 minutes ago, kstrike155 said:

I have been getting errors for days/weeks with my backups saying that there's no space left on device, when I have over 1 TB free.

Check the Min Free Space setting on the share in question.

[kstrike155]

1 hour ago, Kilrah said:

Check the Min Free Space setting on the share in question.

 

It's set to 390.5 GB.

 

EDIT: OK I think what's happening here is that I don't have more than that amount of space available on any of my disks. The language for free space makes it seem like it's at the share level, but because Unraid writes entire files to disk it needs to ensure that there is that much free space available on *some* disk in the array. I reduced this to 10 GB for all shares and will try again.

Edited January 2 by kstrike155
epiphany

[Kilrah]

17 minutes ago, kstrike155 said:

The language for free space makes it seem like it's at the share level

It's a "for this share, don't fill any drive more than X"

[KluthR]

On 12/23/2023 at 9:50 AM, KluthR said:

The log os copied as last step I think. But the missing config file is interesting. Let me test some things.

@jj_uk Just like I thought: the post-run is being executed before copy the log and config. This will be fixed in the next version.

 

On 12/26/2023 at 1:08 AM, Revan335 said:

Yes, this is a temp Path, but its the original Path from the Template.

I thought docker creates such mapping paths right after starting the container, so I would say it should exist, hmm. When npm is running, could you check the existence of that path?

 

On 12/26/2023 at 8:51 PM, urbanracer34 said:

I was wondering how do I configure this plugin?

Does anyone have a guide?

There are helpblocks on several places. Its not that complicated. If you need further help, post back.

 

On 12/27/2023 at 1:39 PM, ffhelllskjdje said:

wow just looked and having the same problem

Could you two share a debug log? Without ANY information, I cant do anything.

 

On 12/30/2023 at 8:51 PM, warwolf7 said:

I get this error, tar verification failed

Is the mapping in question used by another container by chance? The error means that during the backup some contents were changed. So something accessed it during backup.

 

On 12/31/2023 at 1:49 PM, whitephoenix117 said:

I'm not sure if this is the correct place for a feature request. I would like to use this plugin to only backup my flash drive. It would be nice to have a global option to skip/disable docker & appdata.

Noted

 

On 12/31/2023 at 10:18 PM, unham said:

Hello, is there any reason that the schedule I set would stop working randomly?

Never observed that behavior. The schedulung is done via cron and is registered once you hit SAVE.

[KluthR]

On 12/23/2023 at 9:50 AM, KluthR said:

The log os copied as last step I think. But the missing config file is interesting. Let me test some things.

@jj_uk Just like I thought: the post-run is being executed before copy the log and config. This will be fixed in the next version.

 

On 12/26/2023 at 1:08 AM, Revan335 said:

Yes, this is a temp Path, but its the original Path from the Template.

I thought docker creates such mapping paths right after starting the container, so I would say it should exist, hmm. When npm is running, could you check the existence of that path?

 

On 12/26/2023 at 8:51 PM, urbanracer34 said:

I was wondering how do I configure this plugin?

Does anyone have a guide?

There are helpblocks on several places. Its not that complicated. If you need further help, post back.

 

On 12/27/2023 at 1:39 PM, ffhelllskjdje said:

wow just looked and having the same problem

Could you two share a debug log? Without ANY information, I cant do anything.

 

On 12/30/2023 at 8:51 PM, warwolf7 said:

I get this error, tar verification failed

Is the mapping in question used by another container by chance? The error means that during the backup some contents were changed. So something accessed it during backup.

 

On 12/31/2023 at 1:49 PM, whitephoenix117 said:

I'm not sure if this is the correct place for a feature request. I would like to use this plugin to only backup my flash drive. It would be nice to have a global option to skip/disable docker & appdata.

Noted

 

9 hours ago, Cornflake said:

is it possible to get a notification when the backup process starts ?

Write a pre-run script that executes:

/usr/local/emhttp/webGui/scripts/notify -e "Appdata Backup" -s "Starting backup..." -d "The backup just begun" -i "normal"

 

 

[KluthR]

Did anyone tested the current beta?

[kstrike155]

2 hours ago, Kilrah said:

It's a "for this share, don't fill any drive more than X"

 

Here's the copy:

Quote

The minimum free space available to allow writing to any disk belonging to the share.

 

I read that as: "if there's this much free space available on disks belonging to the share, allow writing".

 

Looks like that was the problem as now it's fixed, thanks for the tip!

[Revan335]

5 hours ago, KluthR said:

I thought docker creates such mapping paths right after starting the container, so I would say it should exist, hmm. When npm is running, could you check the existence of that path?

Yes. I have started the Docker Container.

ls -la /tmp/Nginx-Proxy-Manager-Official/var/log/
total 0
drwxr-xr-x 3 root root 60 Jan  2 19:37 ./
drwxr-xr-x 3 root root 60 Jan  2 19:37 ../
drwxr-xr-x 2 root root 60 Jan  2 19:37 nginx/

 

Can you add a included only Option for tests?

For example only one Docker Container like Nginx must create a backup for a quick Test.

 

This simplifies this for many Docker containers, which do not have to be individually excluded and then included again.

 

[02.01.2024 23:17:16][ℹ️][Main] 👋 WELCOME TO APPDATA.BACKUP!! :D
[02.01.2024 23:17:16][ℹ️][Main] Backing up from: /mnt/user/appdata, /mnt/cache/appdata
[02.01.2024 23:17:16][ℹ️][Main] Backing up to: /mnt/user/Pfad/Appdata Backup/ab_20240102_231716
[02.01.2024 23:17:16][ℹ️][Main] Selected containers: Nginx-Proxy-Manager-Official
[02.01.2024 23:17:16][ℹ️][Main] Saving container XML files...
[02.01.2024 23:17:16][ℹ️][Main] Method: Stop all container before continuing.
[02.01.2024 23:17:16][ℹ️][Nginx-Proxy-Manager-Official] Stopping Nginx-Proxy-Manager-Official... done! (took 4 seconds)
[02.01.2024 23:17:20][ℹ️][Main] Starting backup for containers
[02.01.2024 23:17:20][ℹ️][Nginx-Proxy-Manager-Official] Should NOT backup external volumes, sanitizing them...
[02.01.2024 23:17:20][ℹ️][Nginx-Proxy-Manager-Official] Calculated volumes to back up: /mnt/cache/appdata/Nginx-Proxy-Manager-Official/data, /mnt/cache/appdata/Nginx-Proxy-Manager-Official/letsencrypt
[02.01.2024 23:17:20][ℹ️][Nginx-Proxy-Manager-Official] Backing up Nginx-Proxy-Manager-Official...
[02.01.2024 23:17:20][ℹ️][Nginx-Proxy-Manager-Official] Backup created without issues
[02.01.2024 23:17:20][ℹ️][Nginx-Proxy-Manager-Official] Verifying backup...
[02.01.2024 23:17:20][ℹ️][Main] Set containers to previous state
[02.01.2024 23:17:20][ℹ️][Nginx-Proxy-Manager-Official] Starting Nginx-Proxy-Manager-Official... (try #1) done!
[02.01.2024 23:17:22][ℹ️][Main] Backing up the flash drive.
[02.01.2024 23:17:52][ℹ️][Main] Flash backup created!
[02.01.2024 23:17:52][ℹ️][Main] VM meta backup enabled! Backing up...
[02.01.2024 23:17:52][ℹ️][Main] Done!
[02.01.2024 23:17:52][ℹ️][Main] Starting Docker auto-update check...
[02.01.2024 23:18:24][ℹ️][Main] Auto-Update for 'Nginx-Proxy-Manager-Official' is enabled but no update is available.
[02.01.2024 23:18:24][ℹ️][Main] Docker update check finished!
[02.01.2024 23:18:24][ℹ️][Main] Backing up extra files...
[02.01.2024 23:18:24][ℹ️][Main] Backup created without issues
[02.01.2024 23:18:24][ℹ️][Main] Checking retention...
[02.01.2024 23:18:24][ℹ️][Main] DONE! Thanks for using this plugin and have a safe day ;)
[02.01.2024 23:18:24][ℹ️][Main] ❤️

Its working.

 

I tested again, when the Docker Container not running/starting, after a Reboot.

 

OK, i changed the Log Path to a none tmp Path. This should fix the error in the future even after a reboot.

Edited January 2 by Revan335

[kelmino]

Thanks for the updated app, love the ability to do rolling app starts/stops during the backups.

Question though, if I exclude the Cache, Media, and Metadata folders in the Plex appdata folder, what's the worst case scenario in case of a needed restore?

I lost my Plex database a few months ago and needed to rebuild my manual collections and custom posters, also lost some play history for some of my users.  I wanted to avoid that if at all possible.

Thanks

[K1LLA_KING_KING]

Is there anyway to backup and restore plugins? Recently I've had issues with a couple plugins and I wanted to revert to previous settings, instead need to remove, reinstall and set up again

Edited January 3 by K1LLA_KING_KING

[Revan335]

I have the Issue that maybe the Status/Log are empty, but the Backup its running in the past.

 

He said he is empty or not existing.

 

I can't share a debug log that he said Logfile does not exist! By the debug log ID.

Edited January 3 by Revan335

[SiGz]

Is there a way to backup the docker.img with the plugin?

 

I put the path in the extra folders but looks like it starts the docker service prior

[KluthR]

There is no need to backup the docker.img.

[trurl]

15 minutes ago, SiGz said:

Is there a way to backup the docker.img with the plugin?

 

7 minutes ago, KluthR said:

There is no need to backup the docker.img.

 

https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file

 

https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications

[scubieman]

Question, sorry for the 100th time but i do not showing a leading / in my member. What am I missing?

 

[04.01.2024 03:14:54][❌][HomeAssistantCore] tar verification failed! Tar said: tar: Removing leading `/' from member names; mnt/user/appdata/HomeAssistantCore/home-assistant.log: Mod time differs; mnt/user/appdata/HomeAssistantCore/home-assistant.log: Size differs; mnt/user/appdata/HomeAssistantCore/home-assistant_v2.db-wal: Mod time differs; mnt/user/appdata/HomeAssistantCore/home-assistant_v2.db-wal: Contents differ; mnt/user/appdata/HomeAssistantCore/home-assistant_v2.db-shm: Mod time differs; mnt/user/appdata/HomeAssistantCore/home-assistant_v2.db-shm: Contents differ

 

[SirCadian]

I've been looking into remote backup of my Flash and Appdata backups.  My Appdata folder is quite large (~65Gb) largely thanks to the PhotoPrism folder (34Gb).  As a consequence, I'm reluctant to store multiple copies of the backups remotely as remote storage can be costly.  I'd rather have a few daily backups locally and one remotely for that "just in case the house burns down" scenario. 

 

The issue is that the Appdata plugin stores the Appdata backups in a unique date and timestamped directory (format: ab_'datestamp'_'timestamp').  Whilst the flash backup file itself is stored in the unique backup directory, it also has a unique filename that is date/timestamped (format: 'server-name'-flash-backup-'datestamp'-'timestamp'.zip).  All other docker appdata backup files are uniformly named and don't change ('container-name'.tar.gz & my-'container-name'.xml).  This means, if I try to send the backups to a Backblaze B2 storage bucket, a full copy of the Appdata and Flash backups will be kept per day.  That's going to quickly get pretty expensive.

 

With that in mind, I've created a small bash script to run after the backups complete which will identify the latest backup directory and copy it to a static directory ('ab_latest').  It then renames the Flash backup in that directory to a static filename ('server-name'-flash-backup.zip).  After that has completed a scheduled Duplicacy backup will run and send an encrypted version of 'ab-latest' up to the Backblaze B2 bucket.  The bucket is configured to only store one version of each file to keep storage size down.

 

I'm in the process of running the initial backup now and will schedule future ones to run maybe monthly.  Appdata seems pretty volatile and the size of most of the backups seem to change each night.  I don't want to engage in a +60Gb upload more regularly than that. 

 

My question is this; is this the most sensible way to approach remote backup of Appdata & Flash?  It seems a little bit more complex than necessary for something I imagine a lot of people want to do?  Is there an easier way to handle this in the Appdata plugin so that I can retain multiple nightly backups and back up the most recent remotely?

[urbanracer34]

Yes I do require further help. I am completely lost. I literally need a video on how to use this version of AppData backup.

 

There are some dockers I wish to keep backed up but don't want them turned on during the backup process. The dockers since have been deprecated by the developers and may conflict with ones I DO USE.

 

Thanks!