[VM Backup Plugin]

On 2/3/2020 at 9:57 AM, Fiservedpi said:

how would i go about restoring? from said back-up 

Do I just replace the vdisk1.img with the backed up one of my choice?

Generally that should work, but if you run into issues, you may also need to replace the config and/or nvram.

[VM Backup Plugin]

On 2/3/2020 at 11:13 AM, UncleStu said:

Settings:

Enable Snapshots - No

Compress Backups - No

 

Other Settings:

Compare files during backup - No

Disable delta syncs for backups - No

Only use rsync for backups - No

 

Danger Zone:

Fall back to standard backups - No

Pause VMs instead of shutting down - No

Thanks,

I'm having trouble tracking down what might be causing this issue, so I will need to find time to set up an NFS share in my test environment.

 

I did notice there was a recent update to unassigned devices relating to NFS shares... it didn't seem like it would be related to your issue, but possibly it will correct it.

 

I'm hoping to have an opportunity to set up a share for testing sometime in the next week, but I'm out of town the next two weekends, so it may take me longer than that if time doesn't permit during the week.

[VM Backup Plugin]

On 1/30/2020 at 9:21 PM, UncleStu said:

I am getting an error message about the vdisk failing to copy over, yet when I look at the destination location I can see the file there. My backup location set to a NFS mount point on a different NAS and it is successfully mounted via Unassigned Devices. Below is my log for the backup of my domain controller. However I am getting the same error on both VM's when they copy over. My DC is 80 GB and my other VM is 500 GB.

 

Could you let me know what you have the following set to?

 

Settings:

Enable Snapshots

Compress Backups

 

Other Settings:

Compare files during backup

Disable delta syncs for backups

Only use rsync for backups

 

Danger Zone:

Fall back to standard backups

Pause VMs instead of shutting down

 

[VM Backup Plugin]

@dodgypast it looks like there is an issue with whatever data the script is pulling from the config. are you able to DM/send me the config (xml file) for that VM?

 

Thanks,

JTok

[VM Backup Plugin]

yeah I was worried I wasn't making sense - I'd just woken up!  What I was trying to say (hopefully better this time!) is that when you create a profile, it'd be useful if there was an option to 'use current settings'.  I thought this was the behaviour when I'd tweaked all my settings and then got to the last tab and created my first profile 'w10' - I thought it was saving my current settings to w10, rather than creating a new profile alongside the existing default profile.

I see what you mean, I’ll try to find a way to make this clearer and/or more intuitive.

 

The Default button on the first tab needs to say something like 'create new profile' as that's what it does currently i.e. load blank settings, not the default profile that any initial changes are saved to.  i.e. if I on first run make changes and don't create as a new profile then those settings are saved to the 'default profile' but clicking the 'Default' button doesn't load those settings, but resets all.

 

Update: clicking the Default button actually reset my 'default' profile not loaded my defaults so it's really a Reset button.  It also applied the reset rather than letting me review, so it wiped out my settings

This is the design paradigm used by the rest of unRAID, so I don’t think it is a good idea for me make my plugin the exception as that could cause further confusion. The functions of those buttons are defined by the plugin system, and while I’ve bypassed those default functions to implement them in a specific way, I kept the basic functionality the same for consistency across all the other plugins/forms.

 

Here is a breakdown of how the buttons work by default in unRAID:

The Default button resets the form to defaults. The Apply button submits the form. The Done button takes you out of the current plugin/screen. The Done button also changes to Reset if something on the form has been altered but the Apply button hasn’t been hit yet. When it is a Reset button, it will reset the changes in the form back to the last saved settings.

 

 

Also, you should be getting a warning pop up when you hit the Default button asking you if you are sure you want to reset to Defaults.

 

 

 

Sent from my iPad using Tapatalk

[VM Backup Plugin]

14 minutes ago, DZMM said:

Creating a profile should be the first thing done, and an option to start it with the current settings or to use the default settings.  I was surprised to see I had 3 profiles (default, w10 and pfsense) to choose from on the front page when I'd only created two.  Hope that makes sense.

I get what you're saying, but I don't want to make creating a profile necessary to get started... or going to any other tabs for that matter.

I want it to be as close to working out of the box as possible, with as few fields as possible to fill in to get started, and all of them in the same place.

 

To give you an idea of where my head is at, my design philosophy is that the main tab (i.e. Settings tab) is the only tab you actually need to have a basic functioning backup with reasonable settings. The other tabs are for more advanced configurations that a basic user really may not need. A user shouldn't have to go to any of the other tabs unless they want to explore. I'm trying to avoid intimidating non-technical users because the technical ones will go exploring anyway.

I understand that this may not be ideal for every user and situation, but I would prefer to cater to the lowest common denominator and have the plugin be accessible to as many users as possible instead of throwing a wall of options at everyone and forcing them to sort through it.

 

All of that is a big reason why you started with a 'default' config. The other, more practical reason, is backwards compatibility. Everyone with a previous version of the plugin had an existing config that needed to go somewhere, and I don't think anyone would have been thrilled with me if an update wiped it out 😅

 

 

Again though, I absolutely agree that configs are very confusing in their current form, so I have already made some small changes to the upcoming version that should help clear up how the configs work some for new users, and I will keep refining it as we move forward.

 

 

14 minutes ago, DZMM said:

@JTok AHH found it - yeah it really needs to be at the top of the page.

I may move it to a different place at some point when the plugin state is a bit more stable, but for now I'm considering it an Advanced Setting and going to leave where it is. More or less for the reasons above. (Although I have some ideas on where to move it later once things are stable.)

 

14 minutes ago, DZMM said:

and an option to start it with the current settings or to use the default settings.

I'm not exactly sure what you mean by this? 🤔

 

[VM Backup Plugin]

@DZMM

You create and manage the configs on the Manage Configs tab, but to edit them you choose the "current config" from the drop-down box on the Settings tab. From there you edit the config like normal. When you want to edit a different config, you go back to the drop-down box on the Settings tab.

 

Also, default config cannot be renamed or removed, and will not show up on the Manage Configs tab. So you already start with that config active, but from the Manage Configs tab you can add/edit additional configs.

In retrospect, it isn't very obvious how it works, so I'll make some adjustments in the next version.

[unraid-autovmbackup: automate backup of virtual machines in unRAID - v0.4]

Another big thanks to @sjerisman for adding this feature.

 

Again, for anyone using the plugin, the features that get added here will make their way into the plugin before too long.

 

v1.3.1 - 2020/01/21

So Say We All

 

- added option to create a vm specific log in each vm's sub-folder.

 

Script here:

https://github.com/JTok/unraid-vmbackup/tree/v1.3.1

 

-JTok

[VM Backup Plugin]

My apologies for long delay since the last release, for those that care about why, I'll leave it in the postscript.

 

In this version there are some major changes in addition to the usual bug fixes. A big bug fix, is that the plugin should now be able to handle parenthesis in your VM names (thanks to squid for suggesting a solution to that).

New features include the ability to have multiple configs so that you can run backups on different schedules, as well as the ability to run pre and post scripts with those configs. Compression now uses pigz for multi-threaded compression (though a faster inline compression using zstandard is still coming).

Full change-log below.

 

 

v0.2.0 - 2020/01/21
The Resistance

- fixed new config options not always getting added to user config.
- fixed parenthesis in vm names.
- added ability to run pre and post scripts.
- added option to run backup without array started.
- added confirmation dialogs to destructive or dangerous buttons.
- added support for multiple configs/schedules.
- added option to allow multiple configs to run simultaneously.
- added option to set compression level.
- added option to set the number of threads used during compression.
- changed compression program from gzip to pigz to support multi-threading.
- changed alert dialogs to use sweet alerts.
- updated help for custom cron.

 

https://github.com/JTok/unraid.vmbackup/tree/v0.2.0

 

-JTok

 

The delay in this release was caused by me overhauling some of the code to prevent the pages from needed refreshed to update content, and I wound up needing to implement my own version of unRAID's markdown and inline-help to make that work the way I wanted.

[VM Backup Plugin]

Thanks for an awesome plugin! Probably a dumb question and apologies if this is clear or explained elsewhere: my question is under the 'Set backup location' - is there support for remote file shares like a NAS drive? I tried setting up a backup to a NAS drive that I have with SMB file share access at \\192.168.1.2\VMs. Could this be something that could be developed to work - if not are there better tools do this?

 

Thanks

 

I’m out of town this weekend and away from my server, so I’m not sure if there’s a better way (or if this is easily accomplished in unraid)... but off the top of my head, can you mount the share locally in unraid first, then backup to the mount point?

 

 

Sent from my iPhone using Tapatalk

[unraid-autovmbackup: automate backup of virtual machines in unRAID - v0.4]

Big fat thanks to @sjerisman for doing all the work to get a zstandard compression option added to the script!

 

This is going to be significantly faster than using the existing compression option, as well as much more efficient. However, it is not backwards compatible, so if you switch you will want to manually trim any old compressed backups.

 

For those of you that are currently using the plugin, this is coming, but it will be a bit longer. With the plugin being so new, there are some additional kinks to work out as well.

 

v1.3.0 - 2020/01/15
Better than standard

- added option to use zstd inline compression.

 

Script here:

https://github.com/JTok/unraid-vmbackup/tree/v1.3.0

 

-JTok

[VM Backup Plugin]

@sjerisman I will take a look at the PR today, and follow up with you through there.

[VM Backup Plugin]

5 minutes ago, Dati said:

Sorry, what a stupid mistake! I haven't read the error message correctly. I've just looked into the file /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php

The problem was a paranthesis. It was my mistake!

It's not your fault. It should work honestly, but it seems to be an issue with parse_ini_file in PHP as far as I can tell. I'm trying to find a workaround, but have been unsuccessful so far.

At the very least, I'll try to make it clear they are causing an issue in a future update. At least until I can get it resolved.

[VM Backup Plugin]

42 minutes ago, sjerisman said:

I assume most people host their VM image files on faster storage (i.e. SSD or NVMe cache or unassigned devices) and write their backups to the array.  The I/O performance bottleneck is mostly going to be with the array.  Currently, the script copies the image files from source to destination and then afterwards compresses them.  This results in writing uncompressed image files to the array, then reading uncompressed image files from the array, compressing them in memory, and finally writing the compressed result back to the array.  (i.e. READ from cache -> WRITE to array -> READ from array -> COMPRESS in memory -> WRITE to array)

Sorry, I wasn't clear. My tests were from an SSD cache array to the Parity Array. So that's also the bottleneck I was referring to.

I was attempting to also point out that, to anyone interested in improving throughput, the biggest improvements will come from changing storage around to cut out the Parity Array. i.e. running the VMs on an NVMe unassigned device and backing up to an SSD cache array or vice versa.

 

You're right though, that order of operations does seem a bit excessive doesn't it? lol

 

42 minutes ago, sjerisman said:

So, what about an option to use inline (zstd) compression per image file and eliminating the entire post compression step?  This would mean that all reads go against the faster storage tier and are compressed in memory prior to writing to the slower array tier.  (READ from cache -> COMPRESS in memory -> WRITE to array).

This seems viable, but there are some issues that I would need to handle related to backwards compatibility before switching compression algorithms outright.

 

42 minutes ago, sjerisman said:

Or, is there a reason the image files have to first be copied and later compressed?

I'm going from memory here, so possibly the details are wrong, but I believe it came down to being able to turn the VM back on sooner.

Essentially I couldn't guarantee the speed of the system that unRAID would be running on, so I decided to compress after copy because it meant the VM might be able to be turned on sooner (though I honestly can't remember if I tested this or not).

Esentially the logic was: Turn off VM -> Copy files -> Turn on VM -> compress files; would result in the VM being off for less time.

 

With snapshots though, this is far less efficient. So I think it will be a good behavior to make configurable in the future.

 

Thanks for looking into this btw!

[VM Backup Plugin]

On 1/13/2020 at 7:45 AM, Dati said:

I have a Syntax Error that is displayed above the basic settings:

Warning: syntax error, unexpected '(' in /boot/config/plugins/vmbackup/vdisk-list.txt on line 5 in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(501) : eval()'d code on line 55

 

I am able to change some settings - but clicking "apply" or "backup now" doesn't do anything.

I've had the same issue with 6.8.0 snd 6.8.1.

Do you have parenthesis in any of your VM paths? I've run into some issues with that.

[VM Backup Plugin]

On 1/13/2020 at 6:07 AM, Blacksus said:

Since updating to unraid 6.8.1, the plugin leaves a process running indefinitely with high cpu usage. Other than the screenshot below implies, it actually ran for 8 hours on my machine after backup completion - I terminated it and manually started a backup to confirm it was the culprit, then took the screenshot. Im fairly sure this did not happen before 6.8.1. 

 

Hope this will be okay eventually, loving the plugin!

 

 

That's going to be a fun one. I don't have 6.8.1 yet because I am using the nvidia plugin, but I'll see if I can figure out what is going on anyway and get back to you.

[VM Backup Plugin]

41 minutes ago, jpowell8672 said:

I had snapshots enabled and it broke my pfsense vm and had to manually restore. Going to disable vm backup plugin for now and wait for you to work out the bugs. Thanks @JTok

In what way did it break it? It is difficult to fix bugs if I don't know what happened.

 

Did you make sure to change the vdisk path in your VM config before using snapshots?

 

-JTok

[VM Backup Plugin]

On 1/7/2020 at 9:28 PM, nextgenpotato said:

Can I suggest using zstd for compression? This will cut down the time to compress the image by like 10 times.

Or lbzip2 if have many cores this will also make it much faster.

Or can we have options for these? Both of these are already on unraid 6.8 by default.

I looked into this a little today, but this is by no means conclusive. In my tests so far I/O has been the biggest bottleneck, not the compression algorithm or number of threads. So using the parity array vs the cache array, or an unassigned device, is probably going to have the biggest effect on performance.
Honestly, all things being equal, I only saw about a 15-20% performance improvement with my test VM (though I understand that there could be more pronounced differences with other use cases). I tested using zstd, lbzip2, and pigz.

That being said, since there are some performance improvements with a multi-threaded compression utility, I am looking into a good way to integrate something.
I suspect, that at least initially, I will stick with pigz because of backwards compatibility issues. Though I may look into adding an option for the other two later on.

[VM Backup Plugin]

On 1/7/2020 at 9:50 AM, alien said:

ps. During backup if you change setting it may not save properly, you will have to wait for the current backup finish and reload the page before you can actually change setting...atleast that's what happen to me...

I'll have to try and replicate that and see what is going on. Thanks for letting me know

[VM Backup Plugin]

@jpowell8672

At first glance, it looks like it is not seeing any extensions for your vdisks. Can you confirm that they do have extensions, and if so, what are they?

[VM Backup Plugin]

16 hours ago, jpowell8672 said:

I got a notification of a failed error from vm backup when it tried to do a backup from a set schedule. When going to vm backup in settings this error is shown at the top of vm backup settings tab right above basic settings.

Are you able to get the error message from the error log? It will be saved in the log folder inside your backup location.

 

Thanks,

JTok

[VM Backup Plugin]

Warning: parse_ini_file(/boot/config/plugins/vmbackup/vdisk-list.txt): failed to open stream: No such file or directory in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(501) : eval()'d code on line 55

 

 

Where are you seeing the warning, and what are you doing when it occurs?

 

Does anything happen after you get the warning?

 

Thanks,

JTok

 

 

Sent from my iPhone using Tapatalk

[VM Backup Plugin]

@queueiz @Stupifier

 

 

With cron I have found it extremely difficult to validate for every possible input it will accept because there are so many.

By default I use a more restrictive validation for user input to reduce errors. Since I realize that can prevent valid input that might be desired, there is an option in the Danger Zone tab to remove the validation.

 

I will update the help to try to clarify what kind of input is accepted by default.

[Unraid Forum 100K Giveaway]

I love the flexibility to manipulate the OS with plugins and other things from the community.

I would be really interested in seeing snapshots implemented for KVM.



Sent from my iPhone using Tapatalk

[VM Backup Plugin]

This new update is a lot of code optimization to reduce the number of writes to the flash drive, as well as increase the speed that the page loads after the initial run. There are some additional features added as noted in the change-log.

 

 

v0.1.9 - 2019/12/31

Flame Broiled

 

- fixed snapshot extension form validation.

- added file versions to reduce writes to thumb drive.

- added option to run backup even during parity check.

- added alert if enable backups is set to 'No' when apply button is hit.

- added ability to view latest log file.

- updated buttons and pages to use XHR (ajax) for asynchronous loading to reduce page refreshes.

- updated code to reduce page load times.

- implemented md5 hash checks for package files during install.

- removed "vms to backup while running" option due to difficulties with the menu item.

 

 

https://github.com/JTok/unraid.vmbackup/tree/v0.1.9

 

-JTok