JTok

That's a good idea, I was a little worried about messing with people's backups, so that's why I was leaning towards the more hands-off approach. Though I suppose it would be easy enough to make it optional 🤔, and even include a button in Danger Zone to do it manually at a later time if it's opted out of it during the initial transition.

Sorry, I was unclear. What I meant was that once you switch to the new structure, backups made under the old structure will need manually cleaned out until only new structure backups exist. Backups made under the new structure would still be able to be cleaned out. I will update the original post to clarify this. Sent from my iPhone using Tapatalk

I am looking to get some feedback on the following: I'd like to change how the plugin saves backups, but it would be a breaking change. I want to change the timestamp from being part of the filename to being a sub-folder. Instead of this structure: /backup_dir/vm_name/20200204_1555_vm_name.xml/backup_dir/vm_name/20200204_1555_asdf564asd12sd5dfsd.fd/backup_dir/vm_name/20200204_1555_vdisk1.img/backup_dir/vm_name/20200204_1555_vdisk2.img/backup_dir/vm_name/20200205_1555_vm_name.xml/backup_dir/vm_name/20200205_1555_asdf564asd12sd5dfsd.fd/backup_dir/vm_name/20200205_1555_vdisk1.img/backup_dir/vm_name/20200205_1555_vdisk2.img Backups would have this structure: /backup_dir/vm_name/20200204_1555/vm_name.xml/backup_dir/vm_name/20200204_1555/asdf564asd12sd5dfsd.fd/backup_dir/vm_name/20200204_1555/vdisk1.img/backup_dir/vm_name/20200204_1555/vdisk2.img/backup_dir/vm_name/20200205_1555/vm_name.xml/backup_dir/vm_name/20200205_1555/asdf564asd12sd5dfsd.fd/backup_dir/vm_name/20200205_1555/vdisk1.img/backup_dir/vm_name/20200205_1555/vdisk2.img this should come with a few advantages: implementing restore functionality should be much easier and therefore quicker to implement. structure of backups will be less cluttered. handling files not part of a normal vm structure (such as in the case of SpaceInvaderOne's Macinabox) will be easier to handle. However, this will break the existing file cleaning functions. So you would need to manually clean out old backups until all your backups were part of the new structure. So once you switch to the new structure, only new structure backups would be cleaned by the plugin. So with that in mind, here is my proposal: Upon installation of whatever version contains this new structure, the first time you open the plugin page it will prompt you to choose which structure you want to use with a warning message (and a link to further explanation). Don't worry if the plugin auto-updates before you choose a structure, it will continue to use the old structure so your backups will still run even if you haven't picked one yet. Moving forward, development will focus on the new structure. the features that are part of the old structure will stay and keep working, but will not be updated to support new functionality. This will almost certainly include restoring and scenarios like Macinabox. This way, everything continues to work as it has in the past, and gives everyone time to transition at their own pace. Then, somewhere down the line a discussion regarding removing the old structure entirely may take place if it becomes necessary. Thoughts? Concerns? Thanks, JTok

I'll have to make some changes to the script to handle that scenario, but it should be reasonably easy. I think I've got a general way to handle this that should work for other similar situations down the line. I'll add that to my list of things to do.

Generally that should work, but if you run into issues, you may also need to replace the config and/or nvram.

Thanks, I'm having trouble tracking down what might be causing this issue, so I will need to find time to set up an NFS share in my test environment. I did notice there was a recent update to unassigned devices relating to NFS shares... it didn't seem like it would be related to your issue, but possibly it will correct it. I'm hoping to have an opportunity to set up a share for testing sometime in the next week, but I'm out of town the next two weekends, so it may take me longer than that if time doesn't permit during the week.

Could you let me know what you have the following set to? Settings: Enable Snapshots Compress Backups Other Settings: Compare files during backup Disable delta syncs for backups Only use rsync for backups Danger Zone: Fall back to standard backups Pause VMs instead of shutting down

@dodgypast it looks like there is an issue with whatever data the script is pulling from the config. are you able to DM/send me the config (xml file) for that VM? Thanks, JTok

I see what you mean, I’ll try to find a way to make this clearer and/or more intuitive. This is the design paradigm used by the rest of unRAID, so I don’t think it is a good idea for me make my plugin the exception as that could cause further confusion. The functions of those buttons are defined by the plugin system, and while I’ve bypassed those default functions to implement them in a specific way, I kept the basic functionality the same for consistency across all the other plugins/forms. Here is a breakdown of how the buttons work by default in unRAID: The Default button resets the form to defaults. The Apply button submits the form. The Done button takes you out of the current plugin/screen. The Done button also changes to Reset if something on the form has been altered but the Apply button hasn’t been hit yet. When it is a Reset button, it will reset the changes in the form back to the last saved settings. Also, you should be getting a warning pop up when you hit the Default button asking you if you are sure you want to reset to Defaults. Sent from my iPad using Tapatalk

I get what you're saying, but I don't want to make creating a profile necessary to get started... or going to any other tabs for that matter. I want it to be as close to working out of the box as possible, with as few fields as possible to fill in to get started, and all of them in the same place. To give you an idea of where my head is at, my design philosophy is that the main tab (i.e. Settings tab) is the only tab you actually need to have a basic functioning backup with reasonable settings. The other tabs are for more advanced configurations that a basic user really may not need. A user shouldn't have to go to any of the other tabs unless they want to explore. I'm trying to avoid intimidating non-technical users because the technical ones will go exploring anyway. I understand that this may not be ideal for every user and situation, but I would prefer to cater to the lowest common denominator and have the plugin be accessible to as many users as possible instead of throwing a wall of options at everyone and forcing them to sort through it. All of that is a big reason why you started with a 'default' config. The other, more practical reason, is backwards compatibility. Everyone with a previous version of the plugin had an existing config that needed to go somewhere, and I don't think anyone would have been thrilled with me if an update wiped it out 😅 Again though, I absolutely agree that configs are very confusing in their current form, so I have already made some small changes to the upcoming version that should help clear up how the configs work some for new users, and I will keep refining it as we move forward. I may move it to a different place at some point when the plugin state is a bit more stable, but for now I'm considering it an Advanced Setting and going to leave where it is. More or less for the reasons above. (Although I have some ideas on where to move it later once things are stable.) I'm not exactly sure what you mean by this? 🤔

@DZMM You create and manage the configs on the Manage Configs tab, but to edit them you choose the "current config" from the drop-down box on the Settings tab. From there you edit the config like normal. When you want to edit a different config, you go back to the drop-down box on the Settings tab. Also, default config cannot be renamed or removed, and will not show up on the Manage Configs tab. So you already start with that config active, but from the Manage Configs tab you can add/edit additional configs. In retrospect, it isn't very obvious how it works, so I'll make some adjustments in the next version.

Another big thanks to @sjerisman for adding this feature. Again, for anyone using the plugin, the features that get added here will make their way into the plugin before too long. v1.3.1 - 2020/01/21 So Say We All - added option to create a vm specific log in each vm's sub-folder. Script here: https://github.com/JTok/unraid-vmbackup/tree/v1.3.1 -JTok

My apologies for long delay since the last release, for those that care about why, I'll leave it in the postscript. In this version there are some major changes in addition to the usual bug fixes. A big bug fix, is that the plugin should now be able to handle parenthesis in your VM names (thanks to squid for suggesting a solution to that). New features include the ability to have multiple configs so that you can run backups on different schedules, as well as the ability to run pre and post scripts with those configs. Compression now uses pigz for multi-threaded compression (though a faster inline compression using zstandard is still coming). Full change-log below. v0.2.0 - 2020/01/21 The Resistance - fixed new config options not always getting added to user config. - fixed parenthesis in vm names. - added ability to run pre and post scripts. - added option to run backup without array started. - added confirmation dialogs to destructive or dangerous buttons. - added support for multiple configs/schedules. - added option to allow multiple configs to run simultaneously. - added option to set compression level. - added option to set the number of threads used during compression. - changed compression program from gzip to pigz to support multi-threading. - changed alert dialogs to use sweet alerts. - updated help for custom cron. https://github.com/JTok/unraid.vmbackup/tree/v0.2.0 -JTok The delay in this release was caused by me overhauling some of the code to prevent the pages from needed refreshed to update content, and I wound up needing to implement my own version of unRAID's markdown and inline-help to make that work the way I wanted.

I’m out of town this weekend and away from my server, so I’m not sure if there’s a better way (or if this is easily accomplished in unraid)... but off the top of my head, can you mount the share locally in unraid first, then backup to the mount point? Sent from my iPhone using Tapatalk

Big fat thanks to @sjerisman for doing all the work to get a zstandard compression option added to the script! This is going to be significantly faster than using the existing compression option, as well as much more efficient. However, it is not backwards compatible, so if you switch you will want to manually trim any old compressed backups. For those of you that are currently using the plugin, this is coming, but it will be a bit longer. With the plugin being so new, there are some additional kinks to work out as well. v1.3.0 - 2020/01/15 Better than standard - added option to use zstd inline compression. Script here: https://github.com/JTok/unraid-vmbackup/tree/v1.3.0 -JTok

@sjerisman I will take a look at the PR today, and follow up with you through there.

It's not your fault. It should work honestly, but it seems to be an issue with parse_ini_file in PHP as far as I can tell. I'm trying to find a workaround, but have been unsuccessful so far. At the very least, I'll try to make it clear they are causing an issue in a future update. At least until I can get it resolved.

Sorry, I wasn't clear. My tests were from an SSD cache array to the Parity Array. So that's also the bottleneck I was referring to. I was attempting to also point out that, to anyone interested in improving throughput, the biggest improvements will come from changing storage around to cut out the Parity Array. i.e. running the VMs on an NVMe unassigned device and backing up to an SSD cache array or vice versa. You're right though, that order of operations does seem a bit excessive doesn't it? lol This seems viable, but there are some issues that I would need to handle related to backwards compatibility before switching compression algorithms outright. I'm going from memory here, so possibly the details are wrong, but I believe it came down to being able to turn the VM back on sooner. Essentially I couldn't guarantee the speed of the system that unRAID would be running on, so I decided to compress after copy because it meant the VM might be able to be turned on sooner (though I honestly can't remember if I tested this or not). Esentially the logic was: Turn off VM -> Copy files -> Turn on VM -> compress files; would result in the VM being off for less time. With snapshots though, this is far less efficient. So I think it will be a good behavior to make configurable in the future. Thanks for looking into this btw!

Do you have parenthesis in any of your VM paths? I've run into some issues with that.

That's going to be a fun one. I don't have 6.8.1 yet because I am using the nvidia plugin, but I'll see if I can figure out what is going on anyway and get back to you.

In what way did it break it? It is difficult to fix bugs if I don't know what happened. Did you make sure to change the vdisk path in your VM config before using snapshots? -JTok

I looked into this a little today, but this is by no means conclusive. In my tests so far I/O has been the biggest bottleneck, not the compression algorithm or number of threads. So using the parity array vs the cache array, or an unassigned device, is probably going to have the biggest effect on performance. Honestly, all things being equal, I only saw about a 15-20% performance improvement with my test VM (though I understand that there could be more pronounced differences with other use cases). I tested using zstd, lbzip2, and pigz. That being said, since there are some performance improvements with a multi-threaded compression utility, I am looking into a good way to integrate something. I suspect, that at least initially, I will stick with pigz because of backwards compatibility issues. Though I may look into adding an option for the other two later on.

I'll have to try and replicate that and see what is going on. Thanks for letting me know

@jpowell8672 At first glance, it looks like it is not seeing any extensions for your vdisks. Can you confirm that they do have extensions, and if so, what are they?

Are you able to get the error message from the error log? It will be saved in the log folder inside your backup location. Thanks, JTok