[Plugin] Snapshots

[SimonF]

On 11/11/2022 at 7:15 PM, TexasUnraid said:

 

1: The ability to convert a folder to a subvolume in the GUI would be very nice

I didn't include this feature at present due to potential data loss if the process failed. maybe add in the future.

 

On 11/11/2022 at 7:15 PM, TexasUnraid said:

The easiest option for this would be an extra checkbox at the "root snapshots folder" where it is blank now.

Will look to add in a future release.

 

On 11/11/2022 at 7:15 PM, TexasUnraid said:

4: while trying to setup everything from scratch I desperately could use a way to apply settings to multiple volumes at once. Maybe like the way it is done in the shares section of unraid? Or at least some "global" options that could use some generic relative options like {SUBVOLUMENAME} in the path that could use the same relative paths for all the snapshots.

 

For example I find I prefer my snapshots in the root dir under the same snapshot dir ordered by date.

 

EG

/mnt/cache/.snapshots/Snap-{D-m-d-Y}/temp

/mnt/cache/.snapshots/Snap-{D-m-d-Y}/data

/mnt/cache/.snapshots/Snap-{D-m-d-Y}/pictures

 

and the same structure for all drives.

Would you be able to provide screen shots to me via PM of how it looks on your system? I would need to test to see if can be setup this way i.e. substitution is supported anywhere in the path.

 

On 11/11/2022 at 7:15 PM, TexasUnraid said:

6: How does the send snapshot work? I am guessing this is for backup purposes to send the updated data to another drive / server? I knew this was possible but never looked into it. It would be super cool if the plugin could be expanded to include a backup section that would make backups a simple process of connecting an unassigned drive, mounting it and then sending over the snapshots, verifying they are correct etc. This would save a ton of time and make backups way smoother.

With BTRFS snapshots they can only be to the same Volume. the send option allows you to send to another volume with in the system or to a remote system via a SSH tunnel. The send function also allows incremental transfers after the first seed send.

 

@JorgeBScripts where the basis for the development so he may be able to explain additional things if required. The plugin is still classed as Beta, I was looking to create VM snapshots at some point. Also there is a "Simple Guide" on the first page. But thanks for your feedback and suggestions.

 

Could you provide a copy of the script you use?

[TexasUnraid]

2 hours ago, SimonF said:

I didn't include this feature at present due to potential data loss if the process failed. maybe add in the future.

 

I can understand that, maybe add a "settings" section and allow the conversion process to be enabled as an advanced feature with a warning?

  

2 hours ago, SimonF said:

Will look to add in a future release.

 

Cool, seems like it would be fairly simple and make it a lot easier to take a snapshot of everything before doing maintenance / backups etc.

  

2 hours ago, SimonF said:

Would you be able to provide screen shots to me via PM of how it looks on your system? I would need to test to see if can be setup this way i.e. substitution is supported anywhere in the path.

 

I think this is an easier way to view them then each share having it's own snapshot folder so you have to scroll through all the shares when looking for a date.

 

This way you have all the snapshot dates listing and under each date folder are all the shares that were snapshotted at that time.

 

/mnt/cache/.snapshots/daily snapshot 11-12-22/share 1

/mnt/cache/.snapshots/daily snapshot 11-12-22/share 2

/mnt/cache/.snapshots/daily snapshot 11-12-22/share 3

 

Then you have the next day snapshot folder and under it all the shares

 

/mnt/cache/.snapshots/daily snapshot 11-13-22/share 1

/mnt/cache/.snapshots/daily snapshot 11-13-22/share 2

/mnt/cache/.snapshots/daily snapshot 11-13-22/share 3

 

2 hours ago, SimonF said:

With BTRFS snapshots they can only be to the same Volume. the send option allows you to send to another volume with in the system or to a remote system via a SSH tunnel. The send function also allows incremental transfers after the first seed send.

 

Yeah, I knew this was possible but never looked into how to actually use it. It would be really cool it have a seprate section for backups that would automate the process somewhat.

 

For example in my server I keep all my backup drives in a fire safe and only plug them in when I am doing the backup, extends the life of the drives and protects them, I have a feeling that many home users do it similar with external drives they plug in etc.

 

So the backup section could

 

1: find the matching volume on the backup, ideally by the drive/volume ID so that all you have to do it mount the drive via UD

2: compare the data to make sure you are sending it to the right drive.

3: let you know how much will be transferred

4: automate taking snapshots of the backup drives before updating the data

5: If using a remote backup over the network, handle all that setup and run the same checks as above.

 

I do not know how it works, so can't really give much in the way of detailed suggestions lol.

  

2 hours ago, SimonF said:

 

@JorgeBScripts where the basis for the development so he may be able to explain additional things if required. The plugin is still classed as Beta, I was looking to create VM snapshots at some point. Also there is a "Simple Guide" on the first page. But thanks for your feedback and suggestions.

 

Could you provide a copy of the script you use?

 

Yeah, I saw it was still beta, thats why I figured some opinions from a new user might be helpful, you can only have a first impression once lol. I used to do some beta testing back in the day so I got in the habit of jotting down notes when first looking at a new piece of software.

 

The scripts I use are posted here, I tweaked the OP script some further down the thread, pretty sure I have not changed them since then.

 

  

I did see the simple guide, it answered most of the basic questions but didn't go into any detail on how the send snapshot works, having never used it in any form I have no idea how it would work in any form. Guessing on a local system I would just input another volume and it would send it over there but I am sure there is more to it then that. Particularly for remote backups.

 

Overall it is a really really good start! A few tweaks here and there and I can see this being the missing piece I have been looking for with my unraid experience.

 

Right now the biggest issue is being able to set settings in bulk, I just don't have the metal fortitude to go setup each and every share on each drive individually. That would be hundreds of cases of entering the same data over and over and what if I needed to change something, oh the horror! lol.

 

Having the option to edit each sub-volume on each drive individually is fantastic!

 

I am going to guess that 95% of users will be just fine using the same relative settings for everything though.

 

Having a system to apply settings in bulk with check boxes for all that they will apply to and relative paths would be much easier.

 

GREAT work so far!

 

[Peter Braun]

I am using unraid Version: 6.11.5 
Since version 2022.12.04 of snapshot, I can't see the snapshots anymore. Only the settings and the schedules are visible.

The snapshot function itself works perfectly.

...and is it only me that is affected?

[Peter Braun]

25 minutes ago, Peter Braun said:

I am using unraid Version: 6.11.5 
Since version 2022.12.04 of snapshot, I can't see the snapshots anymore. Only the settings and the schedules are visible.

The snapshot function itself works perfectly.

...and is it only me that is affected?

Sorry, my mistake!
It's a problem of Firefox. If I use a chromium based web browser, everything is O.K.

[Bizquick]

Has anyone done a video on how to use this or how we could use this as a somewhat equivalent to ZFS snapshots?
I got my array all changed to use BTFS so I could try to get snap shots going mostly because snap shots was the only thing unraid doesnt have that other NAS servers using ZFS has. TrueNAS has a great setup on the Array but everything else it sucks really bad.
App's and VM setup is terrible. TrueNAS Scale showed some promise but its way to much in the infant stages and needs some basic interface support to help control the memory usage etc.. Not to mention the docker support kind of forcing people to use kubernetics and just network control is lacking. And TrueNAS Core is FreeBSD and all kinds of wrong on that.

[SimonF]

20 hours ago, Bizquick said:

Has anyone done a video on how to use this or how we could use this as a somewhat equivalent to ZFS snapshots?
I got my array all changed to use BTFS so I could try to get snap shots going mostly because snap shots was the only thing unraid doesnt have that other NAS servers using ZFS has. TrueNAS has a great setup on the Array but everything else it sucks really bad.
App's and VM setup is terrible. TrueNAS Scale showed some promise but its way to much in the infant stages and needs some basic interface support to help control the memory usage etc.. Not to mention the docker support kind of forcing people to use kubernetics and just network control is lacking. And TrueNAS Core is FreeBSD and all kinds of wrong on that.

There are no videos I know about at present are you looking to snapshot whole disks or just specific shares on those disks.  Shares normally just create standard shares and snapshots only work on subvols.

[Bizquick]

Yeah looks like this will not work for what I'm looking for I guess I just need to play with trying to setup ZFS Native in unraid. Space Invader has a video on it I guess I just need to watch a few times and figure it out a little bit.

 

[TexasUnraid]

32 minutes ago, Bizquick said:

Yeah looks like this will not work for what I'm looking for I guess I just need to play with trying to setup ZFS Native in unraid. Space Invader has a video on it I guess I just need to watch a few times and figure it out a little bit.

 

Why is that? You con convert shares into sub volumes so they can be snapshotted indivdually quite easily. IMHO it should be added to the plugin but it can be done manually with a few basic commands.

[SimonF]

1 hour ago, TexasUnraid said:

IMHO it should be added to the plugin

Plan is to include something in the next enhancement release next year. Was not included because of potential data loss if it went wrong and creating issue with share settings.

[TexasUnraid]

52 minutes ago, SimonF said:

Plan is to include something in the next enhancement release next year. Was not included because of potential data loss if it went wrong and creating issue with share settings.

I can understand the reasoning, same reason I made it optional on one of the scripts I posted sometime back.

 

I think that under some kind of advanced menu with a warning it should be ok though.

[neurocis]

Hi @SimonF, Is there a GitHUB repo for this plugin if one wanted to contribute?

 

Also my snapshots do not seem to be pruning by the retention settings for me, is there a known issue there? let me know what details I can provide.

 

 

 

 

Lastly, I am sure I am misunderstanding but I am having issues sending snapshots as well, so far just local from Cache root to a spinny disk. I get the subfolder but the contents are all wonky. Once again let me know what details I can provide.

 

 

 

Folders, but no data in the sent subvolume, it is like an incremental but options are set for full.

 

 

 

Cheers!

Edited December 22, 2022 by neurocis

[SimonF]

37 minutes ago, neurocis said:

GitHUB repo

https://github.com/SimonFair/Snapshots I have started working on some of TexasUnraid's suggestions, but have not publish them to the repo as yet but will be in branch Enhancement Release

 

37 minutes ago, neurocis said:

pruning by the retention settings for me, is there a known issue there? let me know what details I can provide.

No issues that I know of.

 

Do you get log messages like:

 

Dec 22 22:12:37 computenode snapshots: Start snapping process /mnt/cache/vol Slot:0
Dec 22 22:12:37 computenode snapshots: btrfs subvolume snapshot -r '/mnt/cache/vol' '/mnt/cache/snaps/vol-202212222212.Hourly' OK  Create a readonly snapshot of '/mnt/cache/vol' in '/mnt/cache/snaps/vol-202212222212.Hourly'
Dec 22 22:12:37 computenode snapshots: Tag: Hourly Count: 6 Occurences: 5 Days: 2 Percentage:0 For /mnt/cache/vol Slot:0
Dec 22 22:12:37 computenode snapshots: Date to remove to:Tuesday 20th December (2022-12-20-22-12-37)
Dec 22 22:12:37 computenode snapshots: Deleted Snapshot by occurence: /mnt/cache/snaps/vol-202212211900.Hourly
Dec 22 22:12:47 computenode snapshots: End snapping process /mnt/cache/vol
Dec 22 22:12:47 computenode snapshots: Manual Run '/mnt/cache/vol' 0 

 

Will need to check the send process. When have you specified the send location?

 

 

Edited December 22, 2022 by SimonF

[neurocis]

So yes, this is interesting:

 

 

Send location is specified as the default:

 

 

I will review the repo tonight probably, see what I can contribute.

 

Thanks!

Edited December 23, 2022 by neurocis
Addn'l detail from Disks

[SimonF]

10 hours ago, neurocis said:

So yes, this is interesting:

 

 

Send location is specified as the default:

 

 

I will review the repo tonight probably, see what I can contribute.

 

Thanks!

Will  need to check root path was added after the initial build as to see if that is used. I was all subvol based initially.

 

Looks like the process is running ok for me, but not checked data as yet. Ignore the additional buttons there are not in the current release.

 

Dec 23 08:12:09 computenode snapshots: Start snapping process /mnt/disk1/ Slot:0
Dec 23 08:12:09 computenode snapshots: btrfs subvolume snapshot -r '/mnt/disk1/' '/mnt/disk1/disk1-back/20221223081209' OK  Create a readonly snapshot of '/mnt/disk1/' in '/mnt/disk1/disk1-back/20221223081209'
Dec 23 08:12:10 computenode snapshots: btrfs snapshot send /mnt/disk1/disk1-back/20221223081209 To /mnt/testpool/disk1 OK  At subvol 20221223081209
Dec 23 08:12:10 computenode snapshots: Tag:  Count: 0 Occurences:  Days: 1 Percentage: For /mnt/disk1/ Slot:0
Dec 23 08:12:10 computenode snapshots: Date to remove to:Thursday 22nd December (2022-12-22-08-12-10)
Dec 23 08:12:20 computenode snapshots: End snapping process /mnt/disk1/
Dec 23 08:12:20 computenode snapshots: Manual Run '/mnt/disk1/' 0 

 

 

Latest code is here but not released. https://github.com/SimonFair/Snapshots/tree/Enhancement_release

Edited December 23, 2022 by SimonF

[SimonF]

On 12/22/2022 at 11:04 PM, neurocis said:

So yes, this is interesting:

 

 

Send location is specified as the default:

 

 

I will review the repo tonight probably, see what I can contribute.

 

Thanks!

I have a fix for the deletes, just need to look at incremental sends.

 

this is in release 2022.12.25

Edited December 25, 2022 by SimonF

[neurocis]

9 hours ago, SimonF said:

I have a fix for the deletes, just need to look at incremental sends.

 

this is in release 2022.12.25

 

And here I don't have anything for you for Christmas! Is there a place I can buy you a coffee? Updated and pruning is now working.

 

Cheers m8.

[SimonF]

19 hours ago, neurocis said:

 

And here I don't have anything for you for Christmas! Is there a place I can buy you a coffee? Updated and pruning is now working.

 

Cheers m8.

Thanks for the bug report appreciated, there is a donate link on the plugins in CA.

[MyUnraidName]

What is the current state of this plugin? What I need is a previous version "shadow copy" in Windows for a share. Sheduled once a day for instance. With zfs, this works perfectly. But I want to run a different file system using Unraid.

 

How practical is this plugin? Is it reliable?

 

Lets say I have 5 disks in an array. My plan is to create a share that uses all disks in the array. Will snapshots work in this scenario? 

[TexasUnraid]

9 hours ago, MyUnraidName said:

What is the current state of this plugin? What I need is a previous version "shadow copy" in Windows for a share. Sheduled once a day for instance. With zfs, this works perfectly. But I want to run a different file system using Unraid.

 

How practical is this plugin? Is it reliable?

 

Lets say I have 5 disks in an array. My plan is to create a share that uses all disks in the array. Will snapshots work in this scenario? 

Taking snapshots of each disk is simple. They will also be combined together just like unraid does to everything else if viewing remotely.

 

You are saying you want them exported to windows so you can see them under the previous versions? That can be done but sadly I have not been able to get it to work personally.

 

If someone knows how it would be FANTASTIC if this plugin could make the necessary config changes automatically.

[SimonF]

4 hours ago, TexasUnraid said:

Taking snapshots of each disk is simple. They will also be combined together just like unraid does to everything else if viewing remotely.

 

You are saying you want them exported to windows so you can see them under the previous versions? That can be done but sadly I have not been able to get it to work personally.

 

If someone knows how it would be FANTASTIC if this plugin could make the necessary config changes automatically.

I have started playing with it but you need to update the Samba settings but the change is per share.

 

https://wiki.gentoo.org/wiki/Samba_shadow_copies

13 hours ago, MyUnraidName said:

What is the current state of this plugin? What I need is a previous version "shadow copy" in Windows for a share. Sheduled once a day for instance. With zfs, this works perfectly. But I want to run a different file system using Unraid.

 

How practical is this plugin? Is it reliable?

 

Lets say I have 5 disks in an array. My plan is to create a share that uses all disks in the array. Will snapshots work in this scenario? 

Plugin works fine. It will not action a share but you have to do each disk. or if it is a pool you can just snap the folder if it is a subvol.

[neurocis]

Hi, just checking in on Snapshot Send functionality and issue I had reported earlier. Let me know if I can assist in any way! Thanks!

 

[SimonF]

6 hours ago, neurocis said:

Hi, just checking in on Snapshot Send functionality and issue I had reported earlier. Let me know if I can assist in any way! Thanks!

 

Can you try to run from command line.

 

btrfs send {snapshotname} | btrfs receive {destination file}

[neurocis]

23 hours ago, SimonF said:

Can you try to run from command line.

 

btrfs send {snapshotname} | btrfs receive {destination file}

 

Succeeded on my one unraid box, failed on the other. digging deeper:

 

 

Looks like I have some corruption to address & maybe some memory to test as it is a RAID 1. Tried to send on my other unraid, worked a charm, so alright then, not the plugin's fault there.

 

Now have to figure out how configuring incremental sends would work as I am confused on automating that cycle with the way the dialogues are. I can get a full to send, and send manually, but incremental eludes me other than having to send the first (full) manually.

 

i.e. How to automate the Full - Incr - Incr - Incr - Full - Incr -Incr - Incr --- cycle?

 

Edit: Nevermind, I have to change my classical thinking. If I manually seed the last snap all will proceed well, it then just becomes a question of pruning the export target snaps.

 

Thanks!

Edited January 28 by neurocis

[primeval_god]

Is there a way or would it be possible to add a way to exclude custom paths from appearing in the interface? In my case i am using urBackup on one of my array disks and it produces a lot of subvolumes and snapshots which clutter the interface and are not something i have a need to manage outside of urBackup.  

[SimonF]

6 minutes ago, primeval_god said:

Is there a way or would it be possible to add a way to exclude custom paths from appearing in the interface? In my case i am using urBackup on one of my array disks and it produces a lot of subvolumes and snapshots which clutter the interface and are not something i have a need to manage outside of urBackup.  

Not currently I have something similar for docker folders, would you be able to share examples via DM?