13 hours ago, KluthR said:

Thats also my plan. I check native ways without sanoid but incremental zfs snapshots are my goal for the next update.

?

user script:

the triky part is getting all the pool names and datasets...

In theory:
 

#!/bin/bash

#v0.4 - Updated to automatically detect all datasets and recursive snapshots
########################simple-snapshot-zfs#######################
###################### User Defined Options ######################

# Automatically gather all ZFS datasets
DATASETS=( $(zfs list -H -o name) )

# Set Number of Snapshots to Keep
SNAPSHOT_QTY=5

###### Don't change below unless you know what you're doing ######
##################################################################

timestamp=$(date "+%Y-%m-%d-%H:%M")
echo "Starting Snapshot ${timestamp}"
echo "_____________________________________________________________"

# Function to create snapshot if there is changed data
create_snapshot_if_changed() {
  local DATASET="$1"
  local WRITTEN
  WRITTEN=$(zfs get -H -o value written "${DATASET}")

  if [[ "${WRITTEN}" != "0" ]]; then
    local TIMESTAMP
    TIMESTAMP="$(date '+%Y-%m-%d-%H%M')"
    # Use -r for recursive snapshots
    zfs snapshot -r "${DATASET}@${TIMESTAMP}"
    echo "Recursive snapshot created: ${DATASET}@${TIMESTAMP}"
  else
    echo "No changes detected in ${DATASET}. No snapshot created."
  fi
}

# Function to prune snapshots
prune_snapshots() {
  local DATASET="$1"
  local KEEP="${SNAPSHOT_QTY}"
  
  local SNAPSHOTS=( $(zfs list -t snapshot -o name -s creation -r "${DATASET}" | grep "^${DATASET}@") )
  local SNAPSHOTS_COUNT=${#SNAPSHOTS[@]}

  echo "Total snapshots for ${DATASET}: ${SNAPSHOTS_COUNT}"

  local SNAPSHOTS_SPACE
  SNAPSHOTS_SPACE=$(zfs get -H -o value usedbysnapshots "${DATASET}")
  echo "Space used by snapshots for ${DATASET}: ${SNAPSHOTS_SPACE}"

  if [[ ${SNAPSHOTS_COUNT} -gt ${KEEP} ]]; then
    local TO_DELETE=$((SNAPSHOTS_COUNT - KEEP))
    for i in "${SNAPSHOTS[@]:0:${TO_DELETE}}"; do
      zfs destroy "${i}"
      echo "Deleted snapshot: ${i}"
      echo "_____________________________________________________________"
    done
  else
   echo "_____________________________________________________________"
  fi
}

# Iterate over each dataset and call the functions
for dataset in "${DATASETS[@]}"; do
  create_snapshot_if_changed "${dataset}"
  prune_snapshots "${dataset}"
done

echo "----------------------------Done!----------------------------"

Should make a recursive zfs snapshot. to all pools and datasets.

Hello,

 

I have a quick question. Is the flash backup function exactly the same as when I click on flash backup under my USB stick on flash disk settings?

 

Thanks

Infosucher

@Infosucher Yes

On 11/3/2024 at 9:28 AM, KluthR said:

Is the plex volume in use by another container?

No it's not and all are stopped during the backup. At first I thought it was an issue with a failed container stoppage before backup, but that wasn't it. It's worked fine for a week, and again last night it failed out of the blue and a manual backup this morning succeeded.

[09.11.2024 04:32:50][ℹ️][Plex-Media-Server] Verifying backup...
[09.11.2024 04:33:02][❌][Plex-Media-Server] tar verification failed! Tar said: tar: Removing leading `/' from member names; ; gzip: stdin: invalid compressed data--crc error; mnt/user/appdata/Plex-Media-Server/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db-2024-11-04: Contents differ; tar: Child returned status 1; tar: Error is not recoverable: exiting now

Edited November 9, 20241 yr by NegZero

Can anyone help me figure out why the backups won't run when scheduled but work fine when run manually? Am I not filling out the time in the right format or something?

 

 

 

Not sure this will help, but this is the log from the last manual backup: 88c50cb6-03c9-4dce-a55c-b89f00238621 

 

It should've run multiple times since then, and I even changed it to daily and set the time to be about to occur. You can see it's set for 10:30 and I set that at 10:28 and waited, but nothing ran. I originally had it set at 4 in the morning on Mondays.

In case anyone has the same question as me and can't find a solution, the answer was that I had minute blank in my original settings when it was set to weekly. Then, when I was troubleshooting and made it supposed to run in a couple minutes, I didn't realize my Unraid system time was set to the wrong timezone. So two separate errors, but the one that could probably trip up other people was leaving minute blank. Setting it to 0 or to 1 should work.

On 11/5/2024 at 7:29 AM, KluthR said:

Because online snapshots are not consistent. There will be imcremental zfs snapshots in future but the recommendation to stop the services for the snapshot will still be there.

Recommended by whom? ZFS was an enterprise solution before OpenZFS came along and you can bet that enterprise customers weren't shutting down their services every hour to run snapshots. The official documentation doesn't have any such recommendation. 

The snapshot can be made, yes, but a backup from a running database as example seems not a good thing to me. This can work in restore cases but it feels wrong. Maybe database engine dependent? Idk

1 hour ago, Renegade605 said:

enterprise customers weren't shutting down their services every hour to run snapshots

They are also not typically running software that chokes if the file wasn't closed during backup. Many consumer level software packages behave poorly if the recovered file is in a state that indicates it's currently in use.

1 hour ago, Renegade605 said:

Recommended by whom? ZFS was an enterprise solution before OpenZFS came along and you can bet that enterprise customers weren't shutting down their services every hour to run snapshots. The official documentation doesn't have any such recommendation. 

I don't think anyone here was suggesting to shut down dockers every hour for snapshots? I mean, why not a combination of both? You can have snapshots every hour for live dockers, and maybe once a week you have a snapshot scheduled when the dockers are shut down (via this plugin).

 

At the very least, on the off chance that a snapshot does not properly restore for whatever reason, having a snapshot of a shut down docker container will rule out the possibility that the restore did not work properly because the snapshot was taken while the docker was live.

13 hours ago, KluthR said:

The snapshot can be made, yes, but a backup from a running database as example seems not a good thing to me. This can work in restore cases but it feels wrong. Maybe database engine dependent? Idk

Databases should typically be flushed but that can be done online.

 

12 hours ago, JonathanM said:

They are also not typically running software that chokes if the file wasn't closed during backup. Many consumer level software packages behave poorly if the recovered file is in a state that indicates it's currently in use.

That's a fair point. But idk, I've been running snapshots online for years and have had to use them, and haven't had a problem. Maybe lucky, but with hourly snapshots you could go back one more hour if you got unlucky.

 

 

Also, one of my use cases is high-availability. I take a snapshot of Nginx Proxy Manager and replicated it to a second machine every 4 minutes with zrep. If one machine goes down, the other starts an identical container and takes over and it's never failed to work. Maybe the average Unraid user should exercise more caution than I do for one reason or another, but it often feels like Unraid (both the OS itself and the community) is pushing users towards doing things in an intentionally inefficient way for unclear reasons.

6 hours ago, Renegade605 said:

it often feels like Unraid (both the OS itself and the community) is pushing users towards doing things in an intentionally inefficient way for unclear reasons.

Data safety, wide compatibility and ease of use have been the primary drivers for many, many years. Raw speed and specialization to specific server tasks have never been at the top of the list.

 

There are plenty of distro's that cater to other niches, Unraid's focus has been an easy to use safe home NAS. The fact that it CAN be used in many other roles is largely coincidental.

This plugin stopped working during that major update and has never completed cleanly since then. I have tried changing settings dozens of times and nothing works.

 

Currently set to stop all containers, backup, and start all containers. I keep getting error mess for the following, but it is creating the tar and it looks fine. Any insight would be helpful. Thank you.

 

[Plex-Media-Server] tar creation failed! Tar said: tar: /mnt/cache/appdata/Plex-Media-Server/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db-2023-01-29: File shrank by 175224832 bytes; padding with zeros

backup.debug.log backup.log

2 hours ago, Mysteic said:

I keep getting error mess for the following

That message is something I never understood. The debug log states that nothing uses the file. I dont know how it can shrink. Maybe some tar expert here?

3 hours ago, KluthR said:

That message is something I never understood. The debug log states that nothing uses the file. I dont know how it can shrink. Maybe some tar expert here?

I have multicore compression on. Should I try normal compression or turn it off?

Hi

 

I ran a restore for all containers and it was successful.

 

However in Docker there's still 0 containers.

 

debug ID is f37207f7-5f83-4498-a9c8-c33d5117a639

 

Thanks!

Restore does not create docker containers. It restores the xml. Create the desired docker from „create container“ and select the right template. Check paths and create it.

Thank you! Found it.

 

Lots of more buttons to click. Hope this part can also me automated

The restore needs more love, yes, but I never wanted it to create containers. Users should check the template first. A restore under existing system does not need a container creation. And a new system maybe needs a template review, so a container creation is also not needed.

 

in the case the nee system is 1:1 set up like a new one, there could be a container creation, yes.

55 minutes ago, StylishQuoter said:

Lots of more buttons to click. Hope this part can also me automated

If you do it from apps->previous apps you can select multiple/all and run it in one go.

Hi, i have an issue i don't understand:
I setup some paths, hit save and ran inot the issue of having external volumes. Ok fine, just switch back and set the checkbox "External too" and whoops. where is the checkbox ?

 

Hey, today i got a error at the backup, the debug log ID is:

f6f359d8-f506-4265-bf16-654ea99ddd0b

It worked for months without issues, but today i added another folder to the extra folders list, its nothing special, just backup folder from another system, nothing is executed from there.

I tried to delete it from the list again, any change.

 

My extra folders:

/mnt/user/domains
/mnt/user/system
/mnt/user/cloud
/mnt/user/backup

 

The error suggest that it has problems to backup stuff in here:

/mnt/user/system/docker/installed/zfs/graph
/mnt/user/system/docker/installed/containers

 

Not sure if i should exclude this, but it worked before without doing it 😅

 

 

Another thing i noticed that the link in the email notification leads to the wrong DNS.
Is there a way to change it?

Edited November 18, 20241 yr by Darkestnoir

Do not backup /mnt/user/system when the docker image is there.

4 hours ago, KluthR said:

Do not backup /mnt/user/system when the docker image is there.

Just wonder why it worked for so long, but now not anymore.

 

Also is there any encryption planned?

Im wondering as well

 

no, no encryption planned. You could use post-run scripts to build something own