bastl Posted March 6, 2020 Share Posted March 6, 2020 @JTok Quick question, is it possible to create encrypted compressed backups? Quote Link to comment
TheSnotRocket Posted May 15, 2020 Share Posted May 15, 2020 Not sure if I missed this or not... but can I backup to remote SMB share? Â .... I added a remote SMB share as a mount point but not seeing it in my drop down. Â Â Quote Link to comment
sjoerd Posted May 20, 2020 Share Posted May 20, 2020 Hi @danioj  Started to use the script. I like the option to turn vm back on once backupped but it seems vm's that were already down are also started after the backup. The vm's are down for a reason.. My unRaid server is dimensioned to run 4 vm's - but I have a lot more which are on and off from time to time. I guess you know what happend with the unRaid server after it tried to start the 7th vm.. No more memory. Could you tell me how to modify the script so that vm's that are down stay down (or make a 0.5?)  Thanks Sjoerd Quote Link to comment
danioj Posted May 20, 2020 Author Share Posted May 20, 2020 Hi @danioj  Started to use the script. I like the option to turn vm back on once backupped but it seems vm's that were already down are also started after the backup. The vm's are down for a reason.. My unRaid server is dimensioned to run 4 vm's - but I have a lot more which are on and off from time to time. I guess you know what happend with the unRaid server after it tried to start the 7th vm.. No more memory. Could you tell me how to modify the script so that vm's that are down stay down (or make a 0.5?)  Thanks Sjoerd  Hi Mate,  The script I wrote ended up being more of a Proof of Concept that, while a good foundation, was something I didnât have the time to take forward into something more functional and stable.  It appears that it has been taken over by the community (got to love open source code) and has since morphed into a vastly more functional plugin that I am happy to have inspired but can neither take credit for nor know much about. I think the name of the unRAID community member now maintaining the resulting plugin is called Jtok. See above posts.  Iâd suggest searching for this plugin in Community Applications (from which there will no doubt be a support thread linked where you can ask functional questions) and start there.  Ta Daniel  Sent from my iPhone using Tapatalk 1 Quote Link to comment
sjoerd Posted May 20, 2020 Share Posted May 20, 2020 Oh right - Â Thank you - I run into this script by the on of the many tutorials of "our" @SpaceInvaderOne. Â I will go fetch that plug-in then đ Quote Link to comment
Stupifier Posted May 20, 2020 Share Posted May 20, 2020 (edited) 1 hour ago, sjoerd said: Oh right - Â Thank you - I run into this script by the on of the many tutorials of "our" @SpaceInvaderOne. Â I will go fetch that plug-in then [emoji846] Â Â Â Â Â You definitely were not using the latest version of the script.... You were using the version that is on the original post to this topic.... That is very outdated and others have improved immensely on it. Just a few posts before yours someone posted a modified/improved version of the script.... Really should look at just through the last couple of posts here. The latest version was just a couple posts above your comment man..... https://forums.unraid.net/topic/46281-unraid-autovmbackup-automate-backup-of-virtual-machines-in-unraid-v04/?do=findComment&comment=814187Â The latest version of the script doesn't have the issue you are mentioning. The plugin is extreme alpha. I wouldn't really call it ready for primetime. The script version I linked is solid as a rock. I'd recommend using the script over the plugin. There are many things to work out on the plugin still. Â Â Edited May 20, 2020 by Stupifier 1 Quote Link to comment
sjoerd Posted May 20, 2020 Share Posted May 20, 2020 I was (ofc) not aware of that. I used the plug-in and seems to be working. I will pick up the new version of autovmbackup too Thanks for bringing that to my attention Quote Link to comment
Stupifier Posted May 20, 2020 Share Posted May 20, 2020 I was (ofc) not aware of that. I used the plug-in and seems to be working. I will pick up the new version of autovmbackup too Thanks for bringing that to my attentionJust be careful with that plugin. When it comes to VM backups, I really want to be 100% confident in whatever tool I use. That's why I use the script instead Quote Link to comment
sjoerd Posted May 20, 2020 Share Posted May 20, 2020 (edited) Script is running as we speak.. Disabled the plugin's cron till it claimed ready Edited May 20, 2020 by sjoerd Quote Link to comment
jortan Posted May 21, 2020 Share Posted May 21, 2020 (edited) Really appreciate this script/plugin. Â [snapshot failures] Â I've moved this post to the plugin thread as that's what I'm using: Â Edited May 27, 2020 by ConnectivIT edited for clarity Quote Link to comment
Stupifier Posted May 21, 2020 Share Posted May 21, 2020 As far as I recall...the snapshot backup features were all beta. I personally wouldn't rely on it. Just my 2 cents Quote Link to comment
baccula Posted July 18, 2020 Share Posted July 18, 2020 I read through all nine pages of this, only saw one poster ask about backing up a passed-through disk. I actually don't want to back-up my pass-through, just the vdisk the OS is running on (vdisk1.img in attached image).  Because I'm passing through the disk using 'by-id', which just references the device name, I can't add or change an extension on it as far as I know. That would allow me to use the scripts 'skip extension' feature.  Instead I get this when it runs: 2020-07-18 14:42:21 information: able to perform snapshot for disk /mnt/cache/domains/gameboy/vdisk1.img on gameboy. use_snapshots is 1. vm_state is running. vdisk_type is raw 2020-07-18 14:42:21 information: qemu agent found. enabling quiesce on snapshot. error: unsupported configuration: source for disk 'hdd' is not a regular file; refusing to generate external snapshot name 2020-07-18 14:42:21 failure: snapshot command failed on vdisk1.snap for gameboy. 2020-07-18 14:42:21 failure: snapshot_fallback is 0. skipping backup for gameboy to prevent data loss. no cleanup will be performed for this vm. Any idea how to get the script to skip backups where the source is not a file?  TIA!    Quote Link to comment
luca2 Posted July 24, 2020 Share Posted July 24, 2020 Hi, I wish someone confirms to me if everything is fine with my backup. This is my config: 2020-07-24 08:41:41 information: send_notifications is 1. notifications will be sent. 2020-07-24 08:41:41 information: only_send_error_notifications is 0. normal notifications will be sent if send_notifications is enabled. 2020-07-24 08:41:42 information: keep_log_file is 1. log files will be kept. 2020-07-24 08:41:42 information: number_of_log_files_to_keep is 1. this is probably a sufficient number of log files to keep. 2020-07-24 08:41:42 information: enable_vm_log_file is 0. vm specific logs will not be created. 2020-07-24 08:41:42 information: backup_all_vms is 0. only vms listed in vms_to_backup will be backed up. 2020-07-24 08:41:42 information: use_snapshots is 0. vms will not be backed up using snapshots. 2020-07-24 08:41:42 information: kill_vm_if_cant_shutdown is 0. vms will not be forced to shutdown if a clean shutdown can not be detected. 2020-07-24 08:41:42 information: set_vm_to_original_state is 1. vms will be set to their original state after backup. 2020-07-24 08:41:42 information: number_of_days_to_keep_backups is 0. backups will be kept indefinitely. be sure to set number_of_backups_to_keep to keep backups storage usage down. 2020-07-24 08:41:42 information: number_of_backups_to_keep is 0. an infinite number of backups will be kept. be sure to set number_of_days_to_keep_backups to keep backups storage usage down. 2020-07-24 08:41:42 information: inline_zstd_compress is 1. vdisk images will be inline compressed but will not be compared afterwards or post compressed. 2020-07-24 08:41:42 information: zstd_level is 3. 2020-07-24 08:41:42 information: zstd_threads is 2. 2020-07-24 08:41:42 information: use_snapshots disabled, not adding snapshot_extension to vdisk_extensions_to_skip. 2020-07-24 08:41:42 information: snapshot_fallback is 0. snapshots will fallback to standard backups. 2020-07-24 08:41:42 information: pause_vms is 0. vms will be shutdown for standard backups. 2020-07-24 08:41:42 information: enable_reconstruct_write is 0. reconstruct write will not be enabled by this script. 2020-07-24 08:41:42 information: backup_xml is 1. vms will have their xml configurations backed up. 2020-07-24 08:41:42 information: backup_nvram is 1. vms will have their nvram backed up. 2020-07-24 08:41:42 information: backup_vdisks is 1. vms will have their vdisks backed up. 2020-07-24 08:41:42 information: start_vm_after_backup is 0. vms will not be started following successful backup. 2020-07-24 08:41:42 information: start_vm_after_failure is 0. vms will not be started following an unsuccessful backup. 2020-07-24 08:41:42 information: disable_delta_sync is 0. rsync will be used to perform delta sync backups. 2020-07-24 08:41:42 information: rsync_only is 0. cp will be used when applicable. 2020-07-24 08:41:42 information: actually_copy_files is 1. files will be copied. 2020-07-24 08:41:42 information: clean_shutdown_checks is 20. this is probably a sufficient number of shutdown checks. 2020-07-24 08:41:42 information: seconds_to_wait is 30. this is probably a sufficient number of seconds to wait between shutdown checks. 2020-07-24 08:41:42 information: keep_error_log_file is 1. error log files will be kept. 2020-07-24 08:41:42 information: number_of_error_log_files_to_keep is 10. this is probably a sufficient error number of log files to keep. And this is the attempt to backup one of the vmÂŽs: 2020-07-24 08:41:42 information: c3po can be found on the system. attempting backup. 2020-07-24 08:41:42 information: creating local c3po.xml to work with during backup. 2020-07-24 08:41:42 information: /mnt/user/backup/unraid/vm/c3po exists. continuing. 2020-07-24 08:41:42 information: copy of backup of /mnt/user/backup/unraid/vm/c3po/20200422_1928_vdisk1.img vdisk to /mnt/user/backup/unraid/vm/c3po/20200724_0841_20200422_1928_vdisk1.img starting. 2020-07-24 08:51:28 information: copy of /mnt/user/backup/unraid/vm/c3po/20200422_1928_vdisk1.img to /mnt/user/backup/unraid/vm/c3po/20200724_0841_20200422_1928_vdisk1.img complete. 2020-07-24 08:51:29 information: skip_vm_shutdown is false. beginning vm shutdown procedure. 2020-07-24 08:51:29 information: c3po is shut off. vm desired state is shut off. can_backup_vm set to y. 2020-07-24 08:51:29 information: actually_copy_files is 1. 2020-07-24 08:51:29 information: can_backup_vm flag is y. starting backup of c3po configuration, nvram, and vdisk(s). 2020-07-24 08:51:29 information: copy of c3po.xml to /mnt/user/backup/unraid/vm/c3po/20200724_0841_c3po.xml complete. 2020-07-24 08:51:29 information: c3po does not appear to have an nvram file. skipping. 2020-07-24 08:54:47 information: copy of /mnt/cache/domains/c3po/20200422_1928_vdisk1.img to /mnt/user/backup/unraid/vm/c3po/20200724_0841_20200422_1928_vdisk1.img.zst complete. 2020-07-24 08:54:47 information: backup of /mnt/cache/domains/c3po/20200422_1928_vdisk1.img vdisk to /mnt/user/backup/unraid/vm/c3po/20200724_0841_20200422_1928_vdisk1.img.zst complete. 2020-07-24 08:54:47 information: the extensions of the vdisks that were backed up are img. 2020-07-24 08:54:47 information: vm_state is shut off. vm_original_state is shut off. not starting c3po. 2020-07-24 08:54:47 information: backup of c3po to /mnt/user/backup/unraid/vm/c3po completed. 2020-07-24 08:54:47 information: number of days to keep backups set to indefinitely. 2020-07-24 08:54:47 information: number of backups to keep set to infinite. 2020-07-24 08:54:47 information: removing local c3po.xml. Now, I get this output in my backup folder: It seems everything works as axpected although I have not tried to recover my vm using the backup file (vdisk) yet. What do you think? Â What is actually the .ZST file. I see the actual vdisk of the vm (20200724_0841_20200422_1928_vdisk1.img) and the associated ZST file, but do not understand their relationship and how should be the vm restored (if I need to)...will it work just copying over to the vm the 20200724_0841_20200422_1928_vdisk1.img? Â Â Â Quote Link to comment
bastl Posted July 24, 2020 Share Posted July 24, 2020 55 minutes ago, luca2 said: What is actually the .ZST file ZST files are compresses vdisk files if you're using the "Zstandard compression" option. Usually the vdisk files is copied first, compressed into a ZST file and the vdisk copy is been removed after compression. In your case there is an older backup in the same folder which isn't compressed. You have to clean your backup destination first or you will end up in 2 big files, wasting space. Check the help! Â Quote Link to comment
luca2 Posted July 24, 2020 Share Posted July 24, 2020 57 minutes ago, bastl said: ZST files are compresses vdisk files if you're using the "Zstandard compression" option. Usually the vdisk files is copied first, compressed into a ZST file and the vdisk copy is been removed after compression. In your case there is an older backup in the same folder which isn't compressed. You have to clean your backup destination first or you will end up in 2 big files, wasting space. Check the help! Â Hi, where do I find this option your pic shows? I am running just the script (JTOK) and donÂŽt have this option neihter remember to see it somewhere else. Quote Link to comment
bastl Posted July 24, 2020 Share Posted July 24, 2020 @luca2 Sry, i thought you're using the plugin, what basically has the same functionality and is based on the older script. In the script you can find the zst compession options starting at line 75 Â Quote Link to comment
luca2 Posted July 24, 2020 Share Posted July 24, 2020 (edited) 27 minutes ago, bastl said: @luca2 Sry, i thought you're using the plugin, what basically has the same functionality and is based on the older script. In the script you can find the zst compession options starting at line 75 Â EDIT: forgot to ask how to uncompress the zst file? I want to try the integrity of the compressed backup ok thx. I just re-run the script and now I get this ... definitly a space saver! Edited July 24, 2020 by luca2 how to uncompress 1 Quote Link to comment
luca2 Posted July 24, 2020 Share Posted July 24, 2020 1 hour ago, luca2 said: ok thx. I just re-run the script and now I get this ... definitly a space saver! Forgot to ask how to decompress this file .zst? Quote Link to comment
bastl Posted July 24, 2020 Share Posted July 24, 2020 @luca2 You can use a modified 7zip version for example. Either you install that modified version or like I did, only use the desired zstd codec in the already installed mainline 7zip version.  https://github.com/mcmilk/7-Zip-zstd#zstandard-codec-plugin-for-mainline-7-zip  You should be also able to decompress the file using tar  https://stackoverflow.com/questions/45355277/how-can-i-decompress-an-archive-file-having-tar-zst Quote Link to comment
xylon Posted July 28, 2020 Share Posted July 28, 2020 @JTok I have some weird bug that senses the vm_original_state incorrectly, it's state according to the program was off (but it was actually on because it is the router) and thus doesn't turn the VM back on. Please HELP TIA Quote Link to comment
Stupifier Posted July 31, 2020 Share Posted July 31, 2020 27 minutes ago, xylon said: Help The developer for this script and the associated plugin hasn't been around for a long time....you're gonna have to think outside the box when you run into issues like this. Work around your issue. Turn off the function in the script which checks if the VM is on/off before running.....then simply create another separate script to schedule your pfsense vm to turn off PRIOR to the VM backup script running. This way, your pfsense will always be OFF when the VM backup script starts..... a VERY quick google for "unraid script turn off vm" shot out exactly how to turn on/off VMs via command-line or script. Super easy.  Quote Link to comment
xylon Posted August 2, 2020 Share Posted August 2, 2020 On 7/31/2020 at 4:15 PM, Stupifier said: The developer for this script and the associated plugin hasn't been around for a long time....you're gonna have to think outside the box when you run into issues like this. Work around your issue. Turn off the function in the script which checks if the VM is on/off before running.....then simply create another separate script to schedule your pfsense vm to turn off PRIOR to the VM backup script running. This way, your pfsense will always be OFF when the VM backup script starts..... a VERY quick google for "unraid script turn off vm" shot out exactly how to turn on/off VMs via command-line or script. Super easy.  Thanks didn't think of that. It is not the cleanest solution however it works and that is all that counts. Quote Link to comment
JonathanM Posted August 3, 2020 Share Posted August 3, 2020 4 hours ago, xylon said: Thanks didn't think of that. It is not the cleanest solution however it works and that is all that counts. Why even bother backing up the VM's vdisk? It's a very simple procedure to do a clean pfsense reinstall that automatically restores a backup file. https://docs.netgate.com/pfsense/en/latest/backup/automatically-restore-during-install.html All you need to do is make a backup of the xml for the VM in unraid, and do a backup from the pfsense GUI whenever you change something. Quote Link to comment
toolmanz Posted July 11, 2021 Share Posted July 11, 2021 All,  I have been backing up my PFSense VM for a while now and just stumbled into a situation where I need to restore a backup. Should I use rsync to sync them back to production after shutting down the PFSense instance?  Running an Unraid server (latest) and use the auto_vm_backup script.  The backup has 3 directories: NVRAM XML IMG ...with various files in each.....  I think though that I am having permission, ownership and rights issues.  I did the restore of the xml and libvirt files using rsync: rsync -a --no-o /mnt/user/backups/auto_vm_backup/_07_Jul_2021/xml/* /etc/libvirt/qemu/* rsync -a --no-o /mnt/user/backups/auto_vm_backup/_07_Jul_2021/img/* /mnt/user/system/libvirt/* ...but had to use the "CP" to copy over the .pf files since when I used: rsync -a --no-o /mnt/user/backups/auto_vm_backup/_07_jul_2021/nvram/* /etc/libvirt/qemu/nvram/* ...it didn't work.  Has anyone attempted this with rsync? should I use a different set of switches- a?v?z?  I did note that as a final step in the backup process it runs a: chmod -R 777 $dir ...which alters the permissions etc.  When I used the above I restarted the PFSense instance and nothing changed (am attemptiong to move the PFSense version back to ver. 2.5.1 from 2.5.2 since it has dynamic dns issues)  On 8/2/2020 at 9:25 PM, jonathanm said: Why even bother backing up the VM's vdisk? It's a very simple procedure to do a clean pfsense reinstall that automatically restores a backup file. https://docs.netgate.com/pfsense/en/latest/backup/automatically-restore-during-install.html All you need to do is make a backup of the xml for the VM in unraid, and do a backup from the pfsense GUI whenever you change something.  This is an interesting alternative but would need to find the version 2.5.1 iso in order to implement since the backups include the XML file.  Any help > direction to head in> or previous experience is appreciated.  TIA toolmanz Spoiler   Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.