Jump to content
JTok

VM Backup Plugin

178 posts in this topic Last Reply

Recommended Posts

Just did a test with a VM that I created using @SpaceInvaderOne Macinabox docker. The docker creates a VM with this structure:

root@Truesource:/mnt/user/domains/Stedding# find .
.
./icon
./icon/catalina.png
./icon/highsierra.png
./icon/mojave.png
./icon/osx.png
./ovmf
./ovmf/OVMF_CODE.fd
./ovmf/OVMF_VARS.fd
./macos_disk.img
./Clover.qcow2
./Catalina-install.img

VMBackup only backs up the following files:

root@Truesource:/mnt/user/Saidar/Backups/Stedding# find . -name '202*'
./20200204_1952_Stedding.xml
./20200204_1952_OVMF_VARS.fd
./20200204_1952_Clover.qcow2
./20200204_1952_Catalina-install.img
./20200204_1952_macos_disk.img

Is there a way to get it to back up the rest of the files? I can probably survive without the icons, but I'm not sure if the VM will work without the others.

Share this post


Link to post
On 2/3/2020 at 11:13 AM, UncleStu said:

Settings:

Enable Snapshots - No

Compress Backups - No

 

Other Settings:

Compare files during backup - No

Disable delta syncs for backups - No

Only use rsync for backups - No

 

Danger Zone:

Fall back to standard backups - No

Pause VMs instead of shutting down - No

Thanks,

I'm having trouble tracking down what might be causing this issue, so I will need to find time to set up an NFS share in my test environment.

 

I did notice there was a recent update to unassigned devices relating to NFS shares... it didn't seem like it would be related to your issue, but possibly it will correct it.

 

I'm hoping to have an opportunity to set up a share for testing sometime in the next week, but I'm out of town the next two weekends, so it may take me longer than that if time doesn't permit during the week.

Share this post


Link to post
On 2/3/2020 at 9:57 AM, Fiservedpi said:

how would i go about restoring? from said back-up 

Do I just replace the vdisk1.img with the backed up one of my choice?

Generally that should work, but if you run into issues, you may also need to replace the config and/or nvram.

Share this post


Link to post
19 hours ago, DBJordan said:

Is there a way to get it to back up the rest of the files? I can probably survive without the icons, but I'm not sure if the VM will work without the others.

I'll have to make some changes to the script to handle that scenario, but it should be reasonably easy. I think I've got a general way to handle this that should work for other similar situations down the line. I'll add that to my list of things to do.

Share this post


Link to post

I am looking to get some feedback on the following:

 

I'd like to change how the plugin saves backups, but it would be a breaking change. I want to change the timestamp from being part of the filename to being a sub-folder.

 

Instead of this structure:

/backup_dir/vm_name/20200204_1555_vm_name.xml/backup_dir/vm_name/20200204_1555_asdf564asd12sd5dfsd.fd/backup_dir/vm_name/20200204_1555_vdisk1.img/backup_dir/vm_name/20200204_1555_vdisk2.img/backup_dir/vm_name/20200205_1555_vm_name.xml/backup_dir/vm_name/20200205_1555_asdf564asd12sd5dfsd.fd/backup_dir/vm_name/20200205_1555_vdisk1.img/backup_dir/vm_name/20200205_1555_vdisk2.img
 

 

Backups would have this structure:

/backup_dir/vm_name/20200204_1555/vm_name.xml/backup_dir/vm_name/20200204_1555/asdf564asd12sd5dfsd.fd/backup_dir/vm_name/20200204_1555/vdisk1.img/backup_dir/vm_name/20200204_1555/vdisk2.img/backup_dir/vm_name/20200205_1555/vm_name.xml/backup_dir/vm_name/20200205_1555/asdf564asd12sd5dfsd.fd/backup_dir/vm_name/20200205_1555/vdisk1.img/backup_dir/vm_name/20200205_1555/vdisk2.img
 

 

this should come with a few advantages:

 

  1. implementing restore functionality should be much easier and therefore quicker to implement.
  2. structure of backups will be less cluttered.
  3. handling files not part of a normal vm structure (such as in the case of SpaceInvaderOne's Macinabox) will be easier to handle.

However, this will break the existing file cleaning functions. So you would need to manually clean out old backups until all your backups were part of the new structure. So once you switch to the new structure, only new structure backups would be cleaned by the plugin.  

 

 

So with that in mind, here is my proposal:

Upon installation of whatever version contains this new structure, the first time you open the plugin page it will prompt you to choose which structure you want to use with a warning message (and a link to further explanation).

Don't worry if the plugin auto-updates before you choose a structure, it will continue to use the old structure so your backups will still run even if you haven't picked one yet.

Moving forward, development will focus on the new structure. the features that are part of the old structure will stay and keep working, but will not be updated to support new functionality. This will almost certainly include restoring and scenarios like Macinabox.

 

This way, everything continues to work as it has in the past, and gives everyone time to transition at their own pace.

 

Then, somewhere down the line a discussion regarding removing the old structure entirely may take place if it becomes necessary.

 

 

Thoughts? Concerns?

 

Thanks,

JTok

Edited by JTok
clarity

Share this post


Link to post

I like the cleaner file structure, but would not want to lose the ability to clean up my older backups. Thinking out loud here. Could you take a listing of the folders within the backup directory and convert that to a date or variable to use? Say the backup location is /mnt/user/backups and then the VM Backup app creates dates as directories from there. I know within Linux you can get a listing of files or directories that are older than X days. Is there a way to generate this list on the fly and then purge out old directories that are older than X? Within the app the user can define the value for X.

Share this post


Link to post
I like the cleaner file structure, but would not want to lose the ability to clean up my older backups. Thinking out loud here. Could you take a listing of the folders within the backup directory and convert that to a date or variable to use? Say the backup location is /mnt/user/backups and then the VM Backup app creates dates as directories from there. I know within Linux you can get a listing of files or directories that are older than X days. Is there a way to generate this list on the fly and then purge out old directories that are older than X? Within the app the user can define the value for X.

 

Sorry, I was unclear. What I meant was that once you switch to the new structure, backups made under the old structure will need manually cleaned out until only new structure backups exist.

Backups made under the new structure would still be able to be cleaned out.

I will update the original post to clarify this.

 

 

Sent from my iPhone using Tapatalk

Share this post


Link to post

Sweet. I'm all in for the change. I'll also test my backups again since the update to the Unassigned Devices plugin. I'll also create a new unRAID share and test backing up there. Ideally I would like the VM backups off box though.

Edited by UncleStu

Share this post


Link to post

Thoughts? Concerns?
 
Thanks,
JTok


Sounds like a great idea.


Sent from my iPad using Tapatalk

Share this post


Link to post
22 hours ago, JTok said:

Sorry, I was unclear. What I meant was that once you switch to the new structure, backups made under the old structure will need manually cleaned out until only new structure backups exist.

Just a thought... What if any old backup files were converted to the new format after a plugin upgrade or after opting into the new naming convention? (i.e. use the filenames read and then remove the timestamps and move the files to new subfolders)

Share this post


Link to post
1 hour ago, sjerisman said:

Just a thought... What if any old backup files were converted to the new format after a plugin upgrade or after opting into the new naming convention? (i.e. use the filenames read and then remove the timestamps and move the files to new subfolders)

I agree if it's not hard, but since the VMBackup plugin is beta free software, I'll take whatever is easiest.

Share this post


Link to post
8 hours ago, sjerisman said:

Just a thought... What if any old backup files were converted to the new format after a plugin upgrade or after opting into the new naming convention? (i.e. use the filenames read and then remove the timestamps and move the files to new subfolders)

 

That's a good idea, I was a little worried about messing with people's backups, so that's why I was leaning towards the more hands-off approach. Though I suppose it would be easy enough to make it optional 🤔, and even include a button in Danger Zone to do it manually at a later time if it's opted out of it during the initial transition.

 

Share this post


Link to post
2020-02-05 04:08 information: backup of hassos_ova-3.7.qcow2 vdisk to /mnt/user/Unraid Backup VM/HomeAssistant/20200205_0400_hassos_ova-3.7.qcow2 complete.
2020-02-05 04:08 information: the extensions of the vdisks that were backed up are qcow2.
2020-02-05 04:08 information: vm_state is shut off. vm_original_state is running. starting HomeAssistant.
Domain HomeAssistant started

2020-02-05 04:08 information: backup of HomeAssistant to /mnt/user/Unraid Backup VM/HomeAssistant completed.
2020-02-05 04:08 information: number of days to keep backups set to indefinitely.
2020-02-05 04:08 information: cleaning out backups over 2 in location /mnt/user/Unraid Backup VM/HomeAssistant/
2020-02-05 04:08 information: removed '/mnt/user/Unraid Backup VM/HomeAssistant/20200122_0400_HomeAssistant.xml'.
2020-02-05 04:08 information: removed '/mnt/user/Unraid Backup VM/HomeAssistant/20200122_0400_79ef5362-0561-ac5a-c8a7-9c80b790294c_VARS-pure-efi.fd'.
2020-02-05 04:08 information: did not find any image files to remove.
2020-02-05 04:08 information: removing local HomeAssistant.xml.
removed 'HomeAssistant.xml'

I have set number of backups to keep to 2, but's it's not deleting the actual image, only the .xml and .fd files. Is it because of the qcow2 extension? 

Share this post


Link to post

I have set number of backups to keep to 2, but's it's not deleting the actual image, only the .xml and .fd files. Is it because of the qcow2 extension? 



It shouldn’t matter what the extension is, but I’ll create a qcow2 drive this afternoon and run some tests.


Sent from my iPhone using Tapatalk

Share this post


Link to post
On 2/10/2020 at 9:05 PM, JTok said:

 

 


It shouldn’t matter what the extension is, but I’ll create a qcow2 drive this afternoon and run some tests.


Sent from my iPhone using Tapatalk

 

 

Did you find anything interesting, or do you know how I should troubleshoot this problem?

Share this post


Link to post

Fantastic Plugin! Thank you so much for all your work!

 

It was mentioned once before, but I didn't see the response (My apologies if it has already been brought up). I was wondering if support for passthrough disk backup is in the cards? I understand Snapshots wouldn't work (The vm will need to be shutdown), but it would be great if the disk could be backed up to an img using something like qemu-img convert...

 

Thank you for your consideration!

Share this post


Link to post
On 2/14/2020 at 10:03 AM, HammerNL said:

Did you find anything interesting, or do you know how I should troubleshoot this problem?

I think I might have an idea. How are you having the backups cleaned?
Are you using "Number of days to keep backups" or "Number of backups to keep"?

 

Because it looks like you are using "Number of backups to keep". In order for that to properly function, the vdisks must be named in a specific way.

If you expand the help, it should give the following info:
"If a VM has multiple vdisks, then they must end in sequential numbers in order to be correctly backed up (i.e. vdisk1.img, vdisk2.img, etc.)."

 

-JTok

Edited by JTok

Share this post


Link to post

Some behind the scenes changes thanks to @sjerisman that include zstandard compression and updates to the logging.

Make sure you read the help if you are going to be switching from legacy gzip compression to zstandard.

 

I also made some changes to how configs are shown to the users that should make it clearer how they work, and what is currently being edited.

 

Full change-log below.

 

 

v0.2.1 - 2020/02/20
Pika Pika

- merged changes from unraid-vmbackup script v1.3.1.
- added seconds to logs.
- added option to use zstandard compression.
- added option to create a vm specific log in each vm's subfolder.
- added config drop-down selection to the top of each tab.
- updated method used to determine cpu thread count.

 

https://github.com/JTok/unraid.vmbackup/tree/v0.2.1

 

-JTok

Share this post


Link to post

I am currently focusing on implementing a restore function next.

 

Based on feedback, I have decided that in order for restore functionality to work, you will need to allow the plugin to convert the existing backup structure. This will be optional, but without it, you will need to perform restores manually.

 

As development progresses, I will work out additional details and reach back out here for feedback.

 

Share this post


Link to post
On 2/17/2020 at 2:29 PM, DoeBoye said:

It was mentioned once before, but I didn't see the response (My apologies if it has already been brought up). I was wondering if support for passthrough disk backup is in the cards? I understand Snapshots wouldn't work (The vm will need to be shutdown), but it would be great if the disk could be backed up to an img using something like qemu-img convert...

That's definitely something I would be willing to implement at some point, but there is other functionality that I would like to ensure is working before I try to mess with it.

Unfortunately, I cannot promise more than that right now.

Share this post


Link to post
On 2/5/2020 at 12:58 PM, UncleStu said:

Sweet. I'm all in for the change. I'll also test my backups again since the update to the Unassigned Devices plugin. I'll also create a new unRAID share and test backing up there. Ideally I would like the VM backups off box though.

I ran a test backup after the updates to Unassigned Devices and no other changes. The backup completed just fine. The only difference this time was that I powered off my VM manually vs. letting the script do it. I want to test this more as it appeared to copy my previous backup file to the current date and then actually backup the VM to the new dated file as well. It definitely took longer and the logs read funny.

I/E: (My recollection of the logs, I don't have access to them at the moment.)

Copying backup VMimage_01-31-2020 to VMimage_02-05-2020 started.

Copying backup VMimage_01-31-2020 to VMimage_02-05-2020 completed.

Copying VMimage to VMimage_02-05-2020 started.

Copying VMimage to VMimage_02-05-2020 completed.

 

The good news though is that it finished with no errors.

Share this post


Link to post

kinda a noob question, but does this plugin automatically shutdown the VM to do the backup? What is the snapshot feature about? I can't find any documentation on this. Sorry if this has been asked, i couldn't find this in the forum.

Share this post


Link to post

Yes, it will shutdown the VM to do the backup. Not sure about the snapshot feature.

Share this post


Link to post

After installing the plugin, I receive the following error in the system terminal upon starting Unraid.

error: failed to connect to the hypervisor
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory

This does not get logged in the syslog ... only output to an attached terminal.  I have had this error since Unraid 6.8.0 but I just traced it here on a new 6.8.3 install.

 

This has been experienced by other users too, but up till now, no one knew what was causing it.

 

-JesterEE

Share this post


Link to post

I'm installing and using this for the first time and have a question. You've talked about a breaking change to the file structure. Since I have no existing backups is there a way use this new structure from the start? Or is it not implemented yet?

 

Also, I want to use compression, so is Zstandard the recommended setting?

 

Thanks!

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.