Jump to content
JTok

VM Backup Plugin

85 posts in this topic Last Reply

Recommended Posts

5 minutes ago, Dati said:

Sorry, what a stupid mistake! I haven't read the error message correctly. I've just looked into the file /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php

The problem was a paranthesis. It was my mistake!

It's not your fault. It should work honestly, but it seems to be an issue with parse_ini_file in PHP as far as I can tell. I'm trying to find a workaround, but have been unsuccessful so far.

At the very least, I'll try to make it clear they are causing an issue in a future update. At least until I can get it resolved.

Share this post


Link to post
5 hours ago, JTok said:

Sorry, I wasn't clear. My tests were from an SSD cache array to the Parity Array. So that's also the bottleneck I was referring to.

I was attempting to also point out that, to anyone interested in improving throughput, the biggest improvements will come from changing storage around to cut out the Parity Array. i.e. running the VMs on an NVMe unassigned device and backing up to an SSD cache array or vice versa.

Yep, that makes sense.  I hadn't really considered backing up directly to the cache tier because for a lot of people that means their VMs and backups are on the same storage device(s) and it could fill up the cache quickly and add a lot of wear to the SSDs.  In thinking about it, I agree that backing up to a share that has cache: Yes, cache: Prefer, or cache: Only would definitely help with the I/O performance bottleneck.  But I think the script would still be doing things a bit inefficiently (including wearing out the cache faster) and would still be slower than inline compression.

 

5 hours ago, JTok said:

This seems viable, but there are some issues that I would need to handle related to backwards compatibility before switching compression algorithms outright.

Yes, I agree that backwards compatibility would be nice to maintain.  My suggestion is to introduce a new option that, when set performs inline zstd compression, but when not set still behaves like before.

 

5 hours ago, JTok said:

I'm going from memory here, so possibly the details are wrong, but I believe it came down to being able to turn the VM back on sooner.

That makes sense.  I think a large part of the point of zstd compression, however, is that most modern processors can easily handle real-time compression.  So, technically it will be faster to do inline compression because the bottleneck will still usually be I/O, but the amount of data that even needs to be written will usually be reduced.

5 hours ago, JTok said:

Thanks for looking into this btw!

Yep, no problem.

 

Here is a preliminary pull request to your script only GitHub repo: https://github.com/JTok/unraid-vmbackup/pull/23/files?utf8=✓&diff=split&w=1

 

I still need to finish some testing (i.e. I have no idea what will happen with snapshots enabled), as well as implement the removal of old .zst archives based on age or count limits, but the base code seems to be working well on my test system.

 

My current test VM is a single vdisk Windows 10 image, 30GB raw, 8 GB sparse, and compresses down to about 6.2GB.  With the old compression turned on it took between 2-3 minutes to copy the 8 GB sparse vdisk over to the array (from a NVMe cache tier), and about 8 minutes to compress it.  With the new compression turned on, it took between 1-2 minutes to perform the inline compression over to the array and then it was done.  The CPU usage was marginally higher, but in either case the primary bottleneck was array I/O (turbo write enabled).  This is running on my test UnRaid server which has a 4th Gen quad-core CPU (with not much else going on) and a 2x 2.5" 500 GB disk single parity array (writes cap out at around 80-90 MB/s).  Obviously results will vary depending on other hardware configs and VM disks.  I have .log files if interested.

 

Share this post


Link to post
57 minutes ago, sjerisman said:

My current test VM is a single vdisk Windows 10 image, 30GB raw, 8 GB sparse, and compresses down to about 6.2GB.  With the old compression turned on it took between 2-3 minutes to copy the 8 GB sparse vdisk over to the array (from a NVMe cache tier), and about 8 minutes to compress it.  With the new compression turned on, it took between 1-2 minutes to perform the inline compression over to the array and then it was done.  The CPU usage was marginally higher, but in either case the primary bottleneck was array I/O (turbo write enabled).  This is running on my test UnRaid server which has a 4th Gen quad-core CPU (with not much else going on) and a 2x 2.5" 500 GB disk single parity array (writes cap out at around 80-90 MB/s).  Obviously results will vary depending on other hardware configs and VM disks.  I have .log files if interested.

 

And here is another test that is closer to real world (and even more impressive)...

 

I took a 'real' Windows 7 VM with a 20 GB raw sparse img file (18 GB allocated) and ran it through the old and new compression code.

 

With the old compression code, it took 3-4 minutes to copy the 18 GB image file from a NVMe UD over to a HDD dual-parity array, and then another 14-15 minutes to .tar.gz compress it down to 8.4 GB.

 

With the new inline compression code, it only took 2-3 minutes to copy from the NVMe UD and compress (inline) over to the HDD dual-parity array with a slightly smaller 8.2 GB compressed output size.  The CPU usage was marginally higher than during the .tar.gz step but was over much quicker.

Share this post


Link to post

And, I repeated the same Windows 7 'real' VM test one more time, but this time used the SSD cache tier as the destination instead of the HDD...

 

With the old compression code, it took 1-2 minutes to copy the 18 GB image file from the NVMe UD over to the dual SSD cache, and then still took 13-14 minutes to further .tar.gz compress it down to 8.4 GB.  The compression step definitely seems CPU bound (probably single threaded) instead of I/O bound with this test.

 

With the new inline compression code, it still only took about 1-2 minutes to copy from the NVMe UD and compress (inline) over to the dual SSD cache and still produced a slightly smaller 8.2 GB output file.  The CPU was definitely hit harder, and probably became the bottleneck (over I/O), but I'm really happy with these results and would gladly trade off higher CPU for a few minutes for much lower disk I/O, much less disk wear, and much faster backups.

Share this post


Link to post

@JTok - I was able to do some more coding and testing on my open pull request: https://github.com/JTok/unraid-vmbackup/pull/23/files?utf8=✓&diff=split&w=1

 

Additional changes:

* I added seconds to the generated timestamps and logged messages for better granularity

* I refactored the existing code that deals with removing old backup files (both time based as well as count based) to make it more consistent and easier to follow

* I added support for removing old .zst archives (both time based as well as count based) using the refactored code above

* I did a bunch of additional testing (including with snapshots on) and I think everything is working properly

 

Let me know if you would like to see any other changes on this PR.  I can make a similar PR for the GUI/Plugin version after this one is approved and after I figure out how to properly test changes to the GUI/Plugin.

Share this post


Link to post
On 1/8/2020 at 8:37 PM, JTok said:

I looked into this a little today, but this is by no means conclusive. In my tests so far I/O has been the biggest bottleneck, not the compression algorithm or number of threads. So using the parity array vs the cache array, or an unassigned device, is probably going to have the biggest effect on performance.
Honestly, all things being equal, I only saw about a 15-20% performance improvement with my test VM (though I understand that there could be more pronounced differences with other use cases). I tested using zstd, lbzip2, and pigz.

That being said, since there are some performance improvements with a multi-threaded compression utility, I am looking into a good way to integrate something.
I suspect, that at least initially, I will stick with pigz because of backwards compatibility issues. Though I may look into adding an option for the other two later on.

Thanks for looking into it!

 

I just read the whole thread.

Thanks to @sjerisman as well.

Compression looks promising

Edited by nextgenpotato

Share this post


Link to post
16 hours ago, nextgenpotato said:

Thanks to @sjerisman as well.

Compression looks promising

Yep, no problem.

 

Results are definitely looking promising.  More details on the other thread: 

 

Hopefully these changes will be integrated into this GUI plugin sometime next week.

 

Share this post


Link to post

Thanks for an awesome plugin! Probably a dumb question and apologies if this is clear or explained elsewhere: my question is under the 'Set backup location' - is there support for remote file shares like a NAS drive? I tried setting up a backup to a NAS drive that I have with SMB file share access at \\192.168.1.2\VMs. Could this be something that could be developed to work - if not are there better tools do this?

 

Thanks

Share this post


Link to post
Thanks for an awesome plugin! Probably a dumb question and apologies if this is clear or explained elsewhere: my question is under the 'Set backup location' - is there support for remote file shares like a NAS drive? I tried setting up a backup to a NAS drive that I have with SMB file share access at \\192.168.1.2\VMs. Could this be something that could be developed to work - if not are there better tools do this?

 

Thanks

 

I’m out of town this weekend and away from my server, so I’m not sure if there’s a better way (or if this is easily accomplished in unraid)... but off the top of my head, can you mount the share locally in unraid first, then backup to the mount point?

 

 

Sent from my iPhone using Tapatalk

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.