unraid-autovmbackup: automate backup of virtual machines in unRAID - v0.4


Recommended Posts

  • 2 months later...

Hi @danioj

 

Started to use the script. I like the option to turn vm back on once backupped but it seems vm's that were already down are also started after the backup. The vm's are down for a reason.. My unRaid server is dimensioned to run 4 vm's - but I have a lot more which are on and off from time to time. I guess you know what happend with the unRaid server after it tried to start the 7th vm.. No more memory.


Could you tell me how to modify the script so that vm's that are down stay down (or make a 0.5?)

 

Thanks

Sjoerd

Link to comment
Hi @danioj   Started to use the script. I like the option to turn vm back on once backupped but it seems vm's that were already down are also started after the backup. The vm's are down for a reason.. My unRaid server is dimensioned to run 4 vm's - but I have a lot more which are on and off from time to time. I guess you know what happend with the unRaid server after it tried to start the 7th vm.. No more memory.

 

Could you tell me how to modify the script so that vm's that are down stay down (or make a 0.5?)

 

Thanks

Sjoerd

 

 

Hi Mate, 

 

The script I wrote ended up being more of a Proof of Concept that, while a good foundation, was something I didn’t have the time to take forward into something more functional and stable.

 

It appears that it has been taken over by the community (got to love open source code) and has since morphed into a vastly more functional plugin that I am happy to have inspired but can neither take credit for nor know much about. I think the name of the unRAID community member now maintaining the resulting plugin is called Jtok. See above posts.

 

I’d suggest searching for this plugin in Community Applications (from which there will no doubt be a support thread linked where you can ask functional questions) and start there.

 

Ta

 

Daniel

 

 

Sent from my iPhone using Tapatalk

  • Thanks 1
Link to comment
1 hour ago, sjoerd said:
Oh right -   Thank you - I run into this script by the on of the many tutorials of "our" @SpaceInvaderOne.  

I will go fetch that plug-in then

 

 

 

 

 

  1. You definitely were not using the latest version of the script.... You were using the version that is on the original post to this topic.... That is very outdated and others have improved immensely on it. Just a few posts before yours someone posted a modified/improved version of the script.... Really should look at just through the last couple of posts here. The latest version was just a couple posts above your comment man.....
    https://forums.unraid.net/topic/46281-unraid-autovmbackup-automate-backup-of-virtual-machines-in-unraid-v04/?do=findComment&comment=814187 
  2. The latest version of the script doesn't have the issue you are mentioning.
  3. The plugin is extreme alpha. I wouldn't really call it ready for primetime. The script version I linked is solid as a rock. I'd recommend using the script over the plugin. There are many things to work out on the plugin still.

 

 

Edited by Stupifier
  • Thanks 1
Link to comment
I was (ofc) not aware of that. I used the plug-in and seems to be working. I will pick up the new version of autovmbackup too
Thanks for bringing that to my attention
Just be careful with that plugin. When it comes to VM backups, I really want to be 100% confident in whatever tool I use. That's why I use the script instead
Link to comment
  • 1 month later...

I read through all nine pages of this, only saw one poster ask about backing up a passed-through disk. I actually don't want to back-up my pass-through, just the vdisk the OS is running on (vdisk1.img in attached image).

 

Because I'm passing through the disk using 'by-id', which just references the device name, I can't add or change an extension on it as far as I know. That would allow me to use the scripts 'skip extension' feature. 

 

Instead I get this when it runs:

2020-07-18 14:42:21 information: able to perform snapshot for disk /mnt/cache/domains/gameboy/vdisk1.img on gameboy. use_snapshots is 1. vm_state is running. vdisk_type is raw
2020-07-18 14:42:21 information: qemu agent found. enabling quiesce on snapshot.
error: unsupported configuration: source for disk 'hdd' is not a regular file; refusing to generate external snapshot name

2020-07-18 14:42:21 failure: snapshot command failed on vdisk1.snap for gameboy.
2020-07-18 14:42:21 failure: snapshot_fallback is 0. skipping backup for gameboy to prevent data loss. no cleanup will be performed for this vm.

Any idea how to get the script to skip backups where the source is not a file?

 

TIA!

 

 

 

Capture.PNG

Link to comment

Hi, I wish someone confirms to me if everything is fine with my backup.

This is my config:

2020-07-24 08:41:41 information: send_notifications is 1. notifications will be sent.
2020-07-24 08:41:41 information: only_send_error_notifications is 0. normal notifications will be sent if send_notifications is enabled.
2020-07-24 08:41:42 information: keep_log_file is 1. log files will be kept.
2020-07-24 08:41:42 information: number_of_log_files_to_keep is 1. this is probably a sufficient number of log files to keep.
2020-07-24 08:41:42 information: enable_vm_log_file is 0. vm specific logs will not be created.
2020-07-24 08:41:42 information: backup_all_vms is 0. only vms listed in vms_to_backup will be backed up.
2020-07-24 08:41:42 information: use_snapshots is 0. vms will not be backed up using snapshots.
2020-07-24 08:41:42 information: kill_vm_if_cant_shutdown is 0. vms will not be forced to shutdown if a clean shutdown can not be detected.
2020-07-24 08:41:42 information: set_vm_to_original_state is 1. vms will be set to their original state after backup.
2020-07-24 08:41:42 information: number_of_days_to_keep_backups is 0. backups will be kept indefinitely. be sure to set number_of_backups_to_keep to keep backups storage usage down.
2020-07-24 08:41:42 information: number_of_backups_to_keep is 0. an infinite number of backups will be kept. be sure to set number_of_days_to_keep_backups to keep backups storage usage down.
2020-07-24 08:41:42 information: inline_zstd_compress is 1. vdisk images will be inline compressed but will not be compared afterwards or post compressed.
2020-07-24 08:41:42 information: zstd_level is 3.
2020-07-24 08:41:42 information: zstd_threads is 2.
2020-07-24 08:41:42 information: use_snapshots disabled, not adding snapshot_extension to vdisk_extensions_to_skip.
2020-07-24 08:41:42 information: snapshot_fallback is 0. snapshots will fallback to standard backups.
2020-07-24 08:41:42 information: pause_vms is 0. vms will be shutdown for standard backups.
2020-07-24 08:41:42 information: enable_reconstruct_write is 0. reconstruct write will not be enabled by this script.
2020-07-24 08:41:42 information: backup_xml is 1. vms will have their xml configurations backed up.
2020-07-24 08:41:42 information: backup_nvram is 1. vms will have their nvram backed up.
2020-07-24 08:41:42 information: backup_vdisks is 1. vms will have their vdisks backed up.
2020-07-24 08:41:42 information: start_vm_after_backup is 0. vms will not be started following successful backup.
2020-07-24 08:41:42 information: start_vm_after_failure is 0. vms will not be started following an unsuccessful backup.
2020-07-24 08:41:42 information: disable_delta_sync is 0. rsync will be used to perform delta sync backups.
2020-07-24 08:41:42 information: rsync_only is 0. cp will be used when applicable.
2020-07-24 08:41:42 information: actually_copy_files is 1. files will be copied.
2020-07-24 08:41:42 information: clean_shutdown_checks is 20. this is probably a sufficient number of shutdown checks.
2020-07-24 08:41:42 information: seconds_to_wait is 30. this is probably a sufficient number of seconds to wait between shutdown checks.
2020-07-24 08:41:42 information: keep_error_log_file is 1. error log files will be kept.
2020-07-24 08:41:42 information: number_of_error_log_files_to_keep is 10. this is probably a sufficient error number of log files to keep.

And this is the attempt to backup one of the vm´s:

2020-07-24 08:41:42 information: c3po can be found on the system. attempting backup.
2020-07-24 08:41:42 information: creating local c3po.xml to work with during backup.
2020-07-24 08:41:42 information: /mnt/user/backup/unraid/vm/c3po exists. continuing.
2020-07-24 08:41:42 information: copy of backup of /mnt/user/backup/unraid/vm/c3po/20200422_1928_vdisk1.img vdisk to /mnt/user/backup/unraid/vm/c3po/20200724_0841_20200422_1928_vdisk1.img starting.
2020-07-24 08:51:28 information: copy of /mnt/user/backup/unraid/vm/c3po/20200422_1928_vdisk1.img to /mnt/user/backup/unraid/vm/c3po/20200724_0841_20200422_1928_vdisk1.img complete.
2020-07-24 08:51:29 information: skip_vm_shutdown is false. beginning vm shutdown procedure.
2020-07-24 08:51:29 information: c3po is shut off. vm desired state is shut off. can_backup_vm set to y.
2020-07-24 08:51:29 information: actually_copy_files is 1.
2020-07-24 08:51:29 information: can_backup_vm flag is y. starting backup of c3po configuration, nvram, and vdisk(s).
2020-07-24 08:51:29 information: copy of c3po.xml to /mnt/user/backup/unraid/vm/c3po/20200724_0841_c3po.xml complete.
2020-07-24 08:51:29 information: c3po does not appear to have an nvram file. skipping.
2020-07-24 08:54:47 information: copy of /mnt/cache/domains/c3po/20200422_1928_vdisk1.img to /mnt/user/backup/unraid/vm/c3po/20200724_0841_20200422_1928_vdisk1.img.zst complete.
2020-07-24 08:54:47 information: backup of /mnt/cache/domains/c3po/20200422_1928_vdisk1.img vdisk to /mnt/user/backup/unraid/vm/c3po/20200724_0841_20200422_1928_vdisk1.img.zst complete.
2020-07-24 08:54:47 information: the extensions of the vdisks that were backed up are img.
2020-07-24 08:54:47 information: vm_state is shut off. vm_original_state is shut off. not starting c3po.
2020-07-24 08:54:47 information: backup of c3po to /mnt/user/backup/unraid/vm/c3po completed.
2020-07-24 08:54:47 information: number of days to keep backups set to indefinitely.
2020-07-24 08:54:47 information: number of backups to keep set to infinite.
2020-07-24 08:54:47 information: removing local c3po.xml.

Now, I get this output in my backup folder:

imagen.png.13937f7564eaecd920286589d0179aac.png

It seems everything works as axpected although I have not tried to recover my vm using the backup file (vdisk) yet. What do you think?

 

What is actually the .ZST file. I see the actual vdisk of the vm (20200724_0841_20200422_1928_vdisk1.img) and the associated ZST file, but do not understand their relationship and how should be the vm restored (if I need to)...will it work just copying over to the vm the 20200724_0841_20200422_1928_vdisk1.img?

 

 

 

Link to comment
55 minutes ago, luca2 said:

What is actually the .ZST file

ZST files are compresses vdisk files if you're using the "Zstandard compression" option. Usually the vdisk files is copied first, compressed into a ZST file and the vdisk copy is been removed after compression. In your case there is an older backup in the same folder which isn't compressed. You have to clean your backup destination first or you will end up in 2 big files, wasting space. Check the help!

 

grafik.png.20056a01d4757fa8b085e03f4629af14.png

Link to comment
57 minutes ago, bastl said:

ZST files are compresses vdisk files if you're using the "Zstandard compression" option. Usually the vdisk files is copied first, compressed into a ZST file and the vdisk copy is been removed after compression. In your case there is an older backup in the same folder which isn't compressed. You have to clean your backup destination first or you will end up in 2 big files, wasting space. Check the help!

 

grafik.png.20056a01d4757fa8b085e03f4629af14.png

Hi, where do I find this option your pic shows? I am running just the script (JTOK) and don´t have this option neihter remember to see it somewhere else.

Link to comment
27 minutes ago, bastl said:

@luca2 Sry, i thought you're using the plugin, what basically has the same functionality and is based on the older script. In the script you can find the zst compession options starting at line 75

 

EDIT: forgot to ask how to uncompress the zst file? I want to try the integrity of the compressed backup

ok thx. I just re-run the script and now I get this ... definitly a space saver!

imagen.png.e62a2a73d804a71b10d3002db528b4d0.png

Edited by luca2
how to uncompress
  • Haha 1
Link to comment

@luca2 You can use a modified 7zip version for example. Either you install that modified version or like I did, only use the desired zstd codec in the already installed mainline 7zip version.

 

https://github.com/mcmilk/7-Zip-zstd#zstandard-codec-plugin-for-mainline-7-zip

 

You should be also able to decompress the file using tar

 

https://stackoverflow.com/questions/45355277/how-can-i-decompress-an-archive-file-having-tar-zst

Link to comment
27 minutes ago, xylon said:

Help

The developer for this script and the associated plugin hasn't been around for a long time....you're gonna have to think outside the box when you run into issues like this.

Work around your issue. Turn off the function in the script which checks if the VM is on/off before running.....then simply create another separate script to schedule your pfsense vm to turn off PRIOR to the VM backup script running.

This way, your pfsense will always be OFF when the VM backup script starts.....

a VERY quick google for "unraid script turn off vm" shot out exactly how to turn on/off VMs via command-line or script. Super easy.

 

Link to comment
On 7/31/2020 at 4:15 PM, Stupifier said:

The developer for this script and the associated plugin hasn't been around for a long time....you're gonna have to think outside the box when you run into issues like this.

Work around your issue. Turn off the function in the script which checks if the VM is on/off before running.....then simply create another separate script to schedule your pfsense vm to turn off PRIOR to the VM backup script running.

This way, your pfsense will always be OFF when the VM backup script starts.....

a VERY quick google for "unraid script turn off vm" shot out exactly how to turn on/off VMs via command-line or script. Super easy.

 

Thanks didn't think of that. It is not the cleanest solution however it works and that is all that counts.

Link to comment
4 hours ago, xylon said:

Thanks didn't think of that. It is not the cleanest solution however it works and that is all that counts.

Why even bother backing up the VM's vdisk? It's a very simple procedure to do a clean pfsense reinstall that automatically restores a backup file.

https://docs.netgate.com/pfsense/en/latest/backup/automatically-restore-during-install.html

All you need to do is make a backup of the xml for the VM in unraid, and do a backup from the pfsense GUI whenever you change something.

Link to comment
  • 11 months later...

All,

 

I have been backing up my PFSense VM for a while now and just stumbled into a situation where I need to restore a backup.

Should I use rsync to sync them back to production after shutting down the PFSense instance?

 

Running an Unraid server (latest) and use the auto_vm_backup script.

 

The backup has 3 directories:

NVRAM
XML

IMG

...with various files in each.....

 

I think though that I am having permission, ownership and rights issues.

 

I did the restore of the xml and libvirt files using rsync:

rsync -a --no-o /mnt/user/backups/auto_vm_backup/_07_Jul_2021/xml/* /etc/libvirt/qemu/*

rsync -a --no-o /mnt/user/backups/auto_vm_backup/_07_Jul_2021/img/* /mnt/user/system/libvirt/*

...but had to use the "CP" to copy over the .pf files since when I used:

rsync -a --no-o /mnt/user/backups/auto_vm_backup/_07_jul_2021/nvram/* /etc/libvirt/qemu/nvram/*

...it didn't work.

 

Has anyone attempted this with rsync? should I use a different set of switches- a?v?z?

 

I did note that as a final step in the backup process it runs a:

chmod -R 777 $dir 

...which alters the permissions etc.

 

When I used the above I restarted the PFSense instance and nothing changed (am attemptiong to move the PFSense version back to ver. 2.5.1 from 2.5.2 since it has dynamic dns issues)

 

On 8/2/2020 at 9:25 PM, jonathanm said:

Why even bother backing up the VM's vdisk? It's a very simple procedure to do a clean pfsense reinstall that automatically restores a backup file.

https://docs.netgate.com/pfsense/en/latest/backup/automatically-restore-during-install.html

All you need to do is make a backup of the xml for the VM in unraid, and do a backup from the pfsense GUI whenever you change something.

 

This is an interesting alternative but would need to find the version 2.5.1 iso in order to implement since the backups include the XML file.

 

Any help > direction to head in> or previous experience is appreciated.

 

TIA

toolmanz

Spoiler

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.