btrfs snapshot management + VM


Friis

Recommended Posts

I have started to create a plugin for Snapshots, Currently this is for BTRFS based snapshots only. 

 

It is still very early in the development process, so I would still class as Alpha, and not all functions are in place.

 

image.thumb.png.0c2a350f462d340fdec0b80f4eb99a53.png

 

I have been working with JorgeB to work through initial issues of formatting etc so thanks for the feedback and help, but there may be use cases outside of our current environments.

 

If you use Docker folder support instead of image there could be a lot of subvols displayed.

 

If people want to look at the current development, you can download the plugin  at your own risk, any suggestions are welcome, but I will not be able to provide support on this at present until it is released as a beta.

 

Schedule function is not developed, its just the edit page at present.

 

{YMD} if used in the path prefix it will replace with date and time.

 

Send does not support remote systems or incremental at this time.

 

plugin download location https://github.com/SimonFair/Snapshots/raw/main/snapshots.plg

 

access is on tools page under system information.

Edited by SimonF
  • Like 4
  • Thanks 1
Link to comment
  • 3 weeks later...
  • 2 weeks later...
  • 2 weeks later...

My plugin is still a WIP.

 

I have added the ability to create schedule slots so you can setup a weekly, daily monthly etc. But at present the remove based on days/occurences is for all snapshots, So may need to look at making the delete dependant on schedule.

 

image.thumb.png.3480c463c9986150351e5853ea9537dc.png

 

image.thumb.png.0062f74b71df186c910e4f4c80cb488b.png

Working Parts.

 

Days to run.

Send local and remote work, but not Incremental at this time. Will complete incremental soon.

   Issue with incremental for remotes will be checking if the seed is their.

Click + and - collaspes and explodes list of snaps.

 

/mnt/cache/snaps/vol-{YMD} The {YMD} is converted to YMDHMS at run time, could look at other values.

 

VMs will shutdown, Suspend.

 

Retention works, but as stated previously it is for all snaps for that volume, not per schedule currently.

 

Tasks/Ideas to complete.

Tidy Recieved, Incremental to remove info not required on View

Remove Snap/Sends function. Currently only removes snaps on local machine.

Help Text

Error Checking

Notifications of failures, show in syslog only at present.

Add Utilities Page to convert directory/folder to subvol

Add counts for number of snaps etc.

Add QEMU snaphosts, maybe as another tab showing by VM.

Add ZFS Snaps also

 

 

Please let me know any feedback on things you like/dont like.

 

Any suggestions welcome.

 

Regards

Simon

 

  • Like 1
Link to comment

Nice work, seems to be working good enough for a beta, couple of suggestions so far:

 

On 11/28/2021 at 12:20 PM, SimonF said:

/mnt/cache/snaps/vol-{YMD} The {YMD} is converted to YMDHMS at run time, could look at other values.

Would like at least an option without seconds, or make schedule snaps show 00 in the seconds, or they won't be OCD appropriate :)

imagem.png.ba5efa719b7a20ace3a580b68d157465.png

 

On 11/28/2021 at 12:20 PM, SimonF said:

Retention works, but as stated previously it is for all snaps for that volume, not per schedule currently.

If possible I would like a retention option based on available space, that's how I currently clean up some of my sent/receive snapshots, have an hourly script that checks the pool for available space and when below a certain threshold it deletes the oldest snapshot, if it's an option difficult to implement or there's not enough interest by others it's not a big deal, I can keep using my clean up script.

 

 

On 11/28/2021 at 12:20 PM, SimonF said:

Notifications of failures, show in syslog only at present.

 

I did a test that I knew was going to fail, since I made a manual snapshot between schedule incremental sends, so the parent snapshot didn't exist on target, send failed but not seeing the error in the syslog.


 

Nov 30 11:00:01 Tower15 snapshots: Start snapping process /mnt/disk1/temp Slot:0
Nov 30 11:00:01 Tower15 snapshots: btrfs subvolume snapshot -r '/mnt/disk1/temp' '/mnt/disk1/snaps/temp-202111301900'
Nov 30 11:00:01 Tower15 snapshots: btrfs snapshot send -p /mnt/disk1/snaps/temp-20211130180418 /mnt/disk1/snaps/temp-202111301900 To /mnt/disk2/ 1 At snapshot temp-202111301900
Nov 30 11:00:11 Tower15 snapshots: End snapping process /mnt/disk1/temp

 

 

Besides some of the things not working yet, I also miss the option to delete several snapshots together, like a check-mark were I could select for multiple deletes.

 

That's all I can think of so far, keep up the good work.

  • Thanks 1
Link to comment
30 minutes ago, JorgeB said:

Would like at least an option without seconds, or make schedule snaps show 00 in the seconds, or they will be OCD appropriate

Automated is not using seconds missed s of the string.

 

I can change the {} processing to use the values people put in {YmdHis}

 

Will add checkbox for deletes and will take look at errors. I think i have only include first error line, will fix that.

 

Sent PM re clean up.

 

Thanks for the feedback.

 

 

Link to comment
  • 4 weeks later...
On 12/27/2021 at 3:21 PM, SimonF said:

I have put my initial Beta release into CA now. Please post questions in this feature reques

 

image.thumb.png.ede9c35f91dd0a5f59dff5ffb2f70c8a.png

How to use Snapshots.mhtUnavailable

get this error?

 

Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/snapshots/include/lib.php on line 499
Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/snapshots/include/lib.php on line 499

 

array(1) { ["/mnt/cache"]=> NULL }

 

Link to comment
3 minutes ago, malac said:

get this error?

 

Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/snapshots/include/lib.php on line 499
Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/snapshots/include/lib.php on line 499

 

array(1) { ["/mnt/cache"]=> NULL }

 

Is that on the debug tab?

Link to comment
18 minutes ago, SimonF said:

Thats fine. The BTRFS page should be showing no subvolumes. And the array will populate as things are added. The Debug tab will be removed when out of beta.

 

Will look to remove the warnings. Debug page if just for info if I need it.

ok, thx. Is it possible that pictures are missing in howto.mht?

Link to comment
  • 2 weeks later...

I installed the beta plug-in.  When I create a sub-volume nothing is changed/added

 

log:

Jan  6 18:05:02 ZF7-SERVER snapshots: btrfs subvolume create /mnt/data/domains 1 
Jan  6 18:05:21 ZF7-SERVER snapshots: btrfs subvolume create /mnt/data/domains 1 
Jan  6 18:05:40 ZF7-SERVER snapshots: btrfs subvolume create /mnt/data/domains 1 

 

but no sub volume appears on gui

Link to comment
6 hours ago, arturovf said:

I installed the beta plug-in.  When I create a sub-volume nothing is changed/added

 

log:

Jan  6 18:05:02 ZF7-SERVER snapshots: btrfs subvolume create /mnt/data/domains 1 
Jan  6 18:05:21 ZF7-SERVER snapshots: btrfs subvolume create /mnt/data/domains 1 
Jan  6 18:05:40 ZF7-SERVER snapshots: btrfs subvolume create /mnt/data/domains 1 

 

but no sub volume appears on gui

Looks like the command is failing. Does that path already exist?

Link to comment

I jut came across this and will be using this.

 

I found it because I was looking for a tool to control compression and deduplication for my btrfs cache volumes. Is that a set of features you would be willing to look at adding? even if it's just at the folder level, it would be extremely helpful.

Link to comment
1 hour ago, eagle470 said:

I jut came across this and will be using this.

 

I found it because I was looking for a tool to control compression and deduplication for my btrfs cache volumes. Is that a set of features you would be willing to look at adding? even if it's just at the folder level, it would be extremely helpful.

Hi @eagle470

 

Happy  to have a look at your requirements. 

 

Regards

Simon.

Link to comment

First off awesome work! Glad to see someone tackling this as a plugin. I have been playing around with it a bit and i have run into a few oddities. Every Snapshot or Send action i take from the GUI returns a popup that says "Fail" even when the action obviously succeeds (snapshot is created and page updates to show the new snapshot). Also it seems that the plugin seems to choke when the destination directory is a tree of non existing directories (I would expect it to create the directories). 

 

Finally a point of order, despite the currently conversation being in this thread doesnt conversation about your plugin belong under its support thread rather than in the feature requests subforum?

Link to comment
54 minutes ago, primeval_god said:

First off awesome work! Glad to see someone tackling this as a plugin. I have been playing around with it a bit and i have run into a few oddities. Every Snapshot or Send action i take from the GUI returns a popup that says "Fail" even when the action obviously succeeds (snapshot is created and page updates to show the new snapshot). Also it seems that the plugin seems to choke when the destination directory is a tree of non existing directories (I would expect it to create the directories). 

 

Finally a point of order, despite the currently conversation being in this thread doesnt conversation about your plugin belong under its support thread rather than in the feature requests subforum?

Thanks, Yeah it still in beta, I haven't completed the status reporting correctly in the GUI.

 

Can you give me an example of the directory tree issue and I will look into it.

 

Will move to support thread soon, just wanted to make sure that the bases where covered from a functionality perspective.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.