Hi

I plan to use @SpaceInvaderOne‘s ZFS snapshot and replication script, to replace my current backup setup. 
I have two pools

1. data (HDD pool for main data)

2. cache (SSD pool for appdata etc)

 

For data I want to

- Snapshot my data pool

- Replicate the data snapshots to a remote server (zfs)

This is possible straightforward with the script

 

For cache I want to

- Snapshot my cache pool

- Replicate my cache pool to the data pool (locally)

- AND replicate my cache pool the remote server (zfs)

 

As far as I understand the script, I have to choose if I want to replicate locally OR replicate to a remote destination. 
Can it be configured to replicate to both destinations?

 

Another scenario could be pretty handy. 
When I attach a local USB disk I want to only replicate my latest snapshots feom data AND cache to the USB disk without creating a new snapshot. But this may be a whole new script I guess

Edited April 21, 20242 yr by UnKwicks
Added second scenario

The script has had a change that it can do the following
1. Snapshot and replicate either to a remote server or a local server. There can only be one destination in the script at present.
2. The script can snapshot and replicate a whole zfs pool. You can add exclusions for datasets which you dont want included.
3. It can replicate just one dataset specified
4. It can replicate with zfs replication or also to a location using rsync if the destination doesnt have zfs. (source location still must be zfs as for the rsync replication it mounts the newest snapshot the rsyncs that to destination)

I will not add much more functionality to the script otherwise it just becomes to complicated to set up. I do plan on converting this (maybe) to a plugin at some point in the future when then i will add features
 

On 4/30/2024 at 9:14 PM, SpaceInvaderOne said:

The script has had a change that it can do the following
1. Snapshot and replicate either to a remote server or a local server. There can only be one destination in the script at present.
2. The script can snapshot and replicate a whole zfs pool. You can add exclusions for datasets which you dont want included.
3. It can replicate just one dataset specified
4. It can replicate with zfs replication or also to a location using rsync if the destination doesnt have zfs. (source location still must be zfs as for the rsync replication it mounts the newest snapshot the rsyncs that to destination)

I will not add much more functionality to the script otherwise it just becomes to complicated to set up. I do plan on converting this (maybe) to a plugin at some point in the future when then i will add features
 

 

Ed Nice work on the script as always! 

 

if i change the "update paths" at the end of the script like the photos im posting, to not make a folder like: pool/zfsfolder/pool_zfs_folder and make it like this: pool/zfsfolder to make an exact mirror on the other side? is that a problem? im doing it correctly on the script? 

Edited September 28, 20241 yr by tgiannak

On 9/28/2024 at 10:08 AM, tgiannak said:

 

Ed Nice work on the script as always! 

 

if i change the "update paths" at the end of the script like the photos im posting, to not make a folder like: pool/zfsfolder/pool_zfs_folder and make it like this: pool/zfsfolder to make an exact mirror on the other side? is that a problem? im doing it correctly on the script? 

I'm trying to do the same thing, make an exact mirror of my appdata to a new pool, and I'm wondering if you ever figured this out.

On 1/22/2025 at 10:18 PM, nicduv said:

I'm trying to do the same thing, make an exact mirror of my appdata to a new pool, and I'm wondering if you ever figured this out.

 

yes this way you can make an exact mirror! it doesnt create the "extra dir" hence it puts all directories at the parent zfs pool.

 

the think i cant get it to work is how to make it with one script, and not making different scripts for every folder! i tried to put auto select datasets, it makes the autosnapshot just fine but it gives me an error when trying to replicate...

Edited January 24, 20251 yr by tgiannak

For me, the replication script also seems not to be working:

I created two scripts, one for "disk1" (I run ZFS in the array) and one for "cache" (also a ZPool). I tried pointing them to the local ZFS pool "backup" (2 slots, but currently only one drive) with the mutual parent dataset "replication" /created with ZFS Master, not with Add Share). Snapshots of all datasets were created (I used the autodetect).

 

But for replication, I go an error for all datasets:

cannot receive new filesystem stream: pool must be upgraded to receive this stream.
mbuffer: error: outputThread: error writing to at offset 0x40000: Broken pipe
mbuffer: warning: error during output to : Broken pipe

I am not sure whether I have to "upgrade" to Unraid 7 for this to work... (I avoided that for now, until I have a replication as a true backup).

 

I also read in the script that the strict-mirror mode will delete dataset not present in source - which should be a problem with two separate scripts pointing to the same destination dataset. So I created two separate parent datasets (disk1-backup, cache-backup) and added them to their respective script. I also tried to have the ZPool Backup with only 1 slot. Neither helps - The error persists.

 

I am not sure whether it is the same issue as the OP - If not, let me know and I will open a separate thread.

On 4/30/2024 at 6:14 PM, SpaceInvaderOne said:

The script has had a change that it can do the following
1. Snapshot and replicate either to a remote server or a local server. There can only be one destination in the script at present.
2. The script can snapshot and replicate a whole zfs pool. You can add exclusions for datasets which you dont want included.
3. It can replicate just one dataset specified
4. It can replicate with zfs replication or also to a location using rsync if the destination doesnt have zfs. (source location still must be zfs as for the rsync replication it mounts the newest snapshot the rsyncs that to destination)

I will not add much more functionality to the script otherwise it just becomes to complicated to set up. I do plan on converting this (maybe) to a plugin at some point in the future when then i will add features
 

Hello, @SpaceInvaderOne wanted to thank you for all your work on this script. Been using the script for local zfs snapshots and replication and it works great!

Just wondering if it's safe to run 2 separate instances of the script on one unraid server?
1 for local zfs replication and 1 for remote rsync replication.

Thanks again!

On 3/6/2025 at 6:28 PM, chungus-mcbrando said:

Hello, @SpaceInvaderOne wanted to thank you for all your work on this script. Been using the script for local zfs snapshots and replication and it works great!

Just wondering if it's safe to run 2 separate instances of the script on one unraid server?
1 for local zfs replication and 1 for remote rsync replication.

Thanks again!

Just wanted to say that yes, this is as safe as running the commands manually. Ergo, I assume you would want to be sure that when you do replicate it, you've not already got another running instance going on, otherwise you might potentially end up replicating an incomplete running job, so perhaps off-set the cron schedule relative to how long you think it'll take to run, which is relative to the scale of your dataset and I/O throughput of drives and NIC.

new to Unraid

im looking at moving to Unraid from Synology

i dont have huge storage needs

I currently have a mirrored data pool for data

single disk for camera/tv recordings

pair of ssd's for docker and a few things

im lookinf at either the teramaster f4-424 pro or the f6-424 max

i may or may not need 6 bays

my plan is to have

1 paraty disk

1 main storage area disk

1 main video/camera recording disk

1 zfs replication/backup disk

all in the array

with a mirrored raid-z cache

i plan to allow appdata and system and domains to live in cache

while having some shares drop on the cache and sync VIA mover to specific spinners

what i need to know is, can i run raid sync/backup from the cache drive to a replication disk in the array

if so, then i do NOT need the 6 bay

as the only thing i would add is 2 SSD's in another raid z mirror

if my other disks ever get too small i will replace them

i would rather use the 4 bay f4-424 pro with i3-305 than the f6-424 max with i5-1235u for power savings

the i3 is plenty faster than my current j4125 in my synology 1520+

if i must use the additional SSD's then i will use the f6-424

i dont mind paying extra, if its needed, but i dont think it will e needed

Please advise on your thoughts