JTok

Ahhh. Sorry, I was unclear. The the vdisk names must be unique per VM, not system-wide. They will be backed up into a folder corresponding with the VM name. e.g. /backup_location/VM1/vdisk1.img /backup_location/VM1/vdisk2.img /backup_location/VM2/vdisk1.img /backup_location/VM2/vdisk2.img So with the script as it currently stands, if you have the following vdisks attached to a VM: /mnt/user/domains/VM1/vdisk1.img /mnt/disk1/VM1/vdisk1.img They will both be copied to the /backup_location/VM1/ folder and one will overwrite the other and/or skip it. While I personally prefer a flatter structure for my backups, I like the idea of having the option to backup using the full path. That would resolve the issue with unique names. I will add it to my to-do list for the next version.

Unfortunately having vdisks with the same name wasn't something I took into consideration with the script. Since the both vdisks will be backed up to the same location the second vdisk will overwrite or ignore (depending on which options are used) the first vdisk because the script will think they are the same one. Renaming one of the disks should be a relatively safe operation, but if you are concerned you could try one of these ways: 1) Make a copy of one of the vdisks manually before renaming it and changing the VM configuration. If you are feeling extra concerned, you may also want to make a copy of the existing VM XML before changing the config to point to the new vdisk name. 2) Run the script with the vdisks_to_skip field filled out for one of the vdisks so that only one gets backed up. Then rename the one vdisk you have a backup of and change the VM configuration. IMPORTANT: You will want to make sure that you rename the disk you're changing to vdisk1.img or the backup script will calculate the retention wrong when using number_of_backups_to_keep.

v1.1.4 - 2018/05/19 A Rose by Any Other Name fixed issue with vdisks not being skipped when defined in vdisks_to_skip. fixed incorrect vdisks sometimes being copied before backup when using delta sync with multiple vdisks. fixed wrong number of vdisks being kept when a VM had multiple vdisks attached. added option to automatically enable reconstruct write (a.k.a. turbo write) during backups and then disable after backups complete. I've tested to the best of my ability, but I cannot guarantee everything will work flawlessly. Please be sure to verify your backups are running correctly and let me know of any issues. Script here: https://github.com/JTok/unraid-vmbackup -JTok

I wanted to give a quick update. I was able to replicate this issue, and I have a fix I'm testing. It also led me to the discovery of a couple other specific use case issues that I want to resolve (details below). Once I have all 3 issues resolved I'll post an updated version of the script. I am going to try to include the option to turn on turbo write during backups if I can figure it out quickly enough. The issues I am addressing in the next update: not skipping vdisks backups with multiple vdisks when using delta syncing (default) are slow because sometimes the wrong vdisk is chosen (finished backups appear to be correct, but the intermediate steps are done in the wrong order) number of backups to keep not keeping the correct number with multiple vdisks (i.e. keeping 2 vdisks on a VM with two vdisks will keep 1 copy of each because it counts both towards the total to keep instead of keeping 2 copies of each) -JTok

Thanks! I was wondering about that after reading the question. I will have to look into implementing that feature. Sent from my iPhone using Tapatalk

Weird. I am in Spain right now, but I should have some time before next week to run a few tests to see if I can replicate the issue. Just to make sure, are you putting the full path of the vdisk to skip? Also, are you able to pm me the log file? Thanks. Sent from my iPhone using Tapatalk

v1.1.3 - 2018/04/16 From Here to Infirmary included inline variable to prevent script from running if a parity check is in progress. fixed spelling errors in script. added advanced option to ignore iso files during vdisk check. #UNTESTED fixed so tarballs so files are added without full paths. updated user selectable options so that they only generate a warning in the log, not a full error log file. As always, use at your own risk with no guarantees. Script here: https://github.com/JTok/unraid-vmbackup -JTok

@joelones Sorry for the delay, I somehow missed the notification for this. It does look like everything completed to me as well. The script is throwing an error on the iso file listed here: 2018-04-11 11:30 warning: /mnt/user/kvm/isos/mini.iso of srv-ubuntu is not a vdisk. skipping. It sees that it is not a vdisk and generates a warning. I can add an option to the next version to ignore iso files. As for the compressed files containing the full path, that is something I was aware of, but just didn't get around to changing before the last update. I am planning on having it compress without the path in the next version. -JTok

Okay, let's try this again. I've put a new version out. It addresses the concerns of @joelones and @PieQuest as well as a few other fixes and enhancements. Added option to only receive error notifications. Added option to receive detailed notifications. Sends notifications when VM backups are started and stopped. Sends notifications when old backups are deleted. Sends notifications when old log files are deleted. Added option to backup nvram. Enabled by default. Fixed issues with notification system. Not all failures and warnings were using notifications. send_notifications variable wasn't disabling many notifications. Fixed issues with tarball backups not removing original files. Fixed issues with log files not being removed. Fixed logging issues. Detailed logging for tarballs. Logging for rm commands. Updated notification message language. Updated script comments/directions. Same deal as the previous updates. I've tested as best I can, but I can't guarantee everything. The script can still be found here: https://github.com/JTok/unraid-vmbackup -JTok

It's because I made a mistake with quotation marks I think. I'm on my phone, so it's a little tough to tell for sure, but I'm pretty sure that's what happening. I will make sure it's fixed in the release for next week. [emoji15] Sorry about that. Sent from my iPhone using Tapatalk

That should be easy enough to add. I'll add it to my to-do list. I'm planning on having the next version out sometime next week, and I think I should be able to get this into that release.

I have made a few updates to the script again. Added the option to disable delta syncs. Added option to only use rsync. Dry-runs can be used with the rsync_only option. Dates and times are now included in log messages. Log files can be generated (enabled by default) Error logs can be managed separately from regular logs. Option to choose the number of logs to keep. Log messages can be sent through unRAID notification system. As with the last update, I've tested as best I can, but I can't guarantee everything. The script can still be found here: https://github.com/JTok/unraid-vmbackup -JTok

Hello, I just happened to notice that after using this that it seems to de-sparsify (un-sparsify?) files. Rsync has a --sparse option, but I'm not sure what effect that would have on non-sparse files. Specifically my vdisk backups were affected. If I had moved docker.img, or libvirt.img, they would presumably have been affected as well. I also have some dockers that create sparse files. Disclaimer: Unfortunately, I forgot to double-check the files it de-sparsed (de-sparsified?) before I removed them, so maybe I'm wrong about this. It would be worth double-checking. You could run a check like this to return a list of sparse files. The command searches sub-directories. find /directory/to/be/copied -type f -printf "%S\t%p\n" | gawk '$1 < 1.0 {print}' Hopefully this information is useful to someone. Thanks, JTok Edit: I forgot to include a link to the code source: https://www.thegeekdiary.com/how-to-find-all-the-sparse-file-in-linux/

I have updated the script on github to include the following changes: Moved variables close to the top to make them easier to access. Added the ability to skip specific vdisks. Restoring the VMs to their previous state can be overridden. Backing up xml and/or vdisks can be skipped. Number of days to keep backups can be set. Changed to use mmin instead of mtime because of issues with accuracy. see: https://unix.stackexchange.com/a/92351 Number of backups to keep can be set. Always checks for img and qcow2 files by default. Timestamps are always used except when the number of backups being kept is set to 1. Toggle for compressing backups. Option to compare backups to original file and retry once. A few other minor things. Currently I've tested everything as well as I can, but I can't guarantee everything will work, so please feel free to let me know if you find any issues. I've tried to make it easy and customizable as possible. You can find the script here: https://github.com/JTok/unraid-vmbackup -JTok

I have created a variation on the original script that uses cp for the initial copy and then uses rsync for subsequent copies. I did a whole lot of testing on my own system and I found that while dd was slightly faster than cp, it did not create sparse files very well. Rsync was so much slower that it was nearly pointless to run. It did sparse files about as well as dd though. All of my tests were from my SSD cache array to my storage array. I also found that using rsync to delta sync was about as fast as re-copying the whole file using cp/dd when using --inplace. However, since the inplace argument creates some risk, and I did want to keep one rolling backup, I have the script make a copy first. Effectively the script does this: First run: uses cp to create a sparse copy of your vdisk. It also makes a copy of your xml. Second run: BEFORE shutting down the VM it makes a copy of the backup from step one using cp. Then it shuts down the VM and uses rsync to perform a delta sync with the inplace argument. Subsequent runs: BEFORE shutting down the VM it uses rsync to perform a delta sync of the latest backup to the second copy with the inplace argument. Then it shuts down the VM and uses rsync to perform a delta sync with the inplace argument. I also changed the script to make a note of the state of the VM when it starts the backup so that it can restore the VM to that state. I have some VMs that I don't keep running, so I don't want them started after a backup. Paused VMs will just be left in the stopped state. I *think* the timestamps will still work with this, but it will just use the cp option for everything. I'm also pretty sure it will keep making them forever without clearing them out. I am working on fixing that. You can find the script here: https://github.com/JTok/unraid-vmbackup Please be sure to read the Change Log and To-Do List as they contain important notes about the state of the script currently. -JTok Edit 1: forgot to mention: the original script does not make sparse copies. Additional explanation: When you create a VM you choose a disk size. Let's say we pick 40GB. Then we install an OS. The OS is taking up 10GB of that 40GB, so the vdisk will say it is 40GB, but only take up 10GB of actual space on the storage disk. This is highly desirable with lots of VMs. However, the original script didn't use the --sparse argument with rsync so it would create a 40GB vdisk that took up 40GB of actual space on the storage disk even though 30GB of that was actually just zeros. So sparse files are basically files that the file system knows are smaller than they appear to be, because they don't have all the unnecessary zeros in them. Edit 2: I have tested restoring the files using cp from the shell to put them back and they do work. However, I have not tested restoring them via the shares directly. You have been warned.

I have been reading about how this plugin works, and I have an alternative method that might work as well (rumor has it that @Squid likes feature creep). Hopefully I didn't miss a post in this thread where someone already talked about this method; I tired to read through the whole thing before posting. I have some experience detecting ransomware in Windows environments, and we use a slightly different method to detect it on file shares that could be appealing to some here. Instead of using bait files we just monitor files against a list of known extensions and file-names used by ransomware, and use a white-list to override it for application specific extensions that we might have in our environment. The pros: - You know when I file gets modified without relying on the ransomware to pick your bait files/shares first - You don't have bait files/shares The cons: - One of your actual files has to be encrypted before you know it's happening (though if a bait file is not picked first, you may still be in the same boat or worse) - You are relying on a list to be kept up-to-date I would love to now offer you some code and a more concrete explanation of how I think this could be implemented, but it has been some time since I had to do anything beyond the most basic scripting in Linux. I'm also assuming that this is possible, and not a burden on the system; I just don't have enough experience to guess at at this point. Widows reference materials (the first link maintains a list of known bad files/extensions here): https://fsrm.experiant.ca/ https://community.spiceworks.com/how_to/100368-cryptolocker-canary-detect-it-early Another alternative is some kind of behavior monitoring (i.e. watching for encryption), but I suspect that is far more difficult to implement. (We use this as well, but pay handsomely for the feature.) Regardless, thank you for the plugin (and all the others). -JTok Quick Edit: Forgot to mention that I've seen fail2ban be used for something similar by reviewing samba logs. The original is in German, so here is the google translate link and the original. Not sure it makes sense to use in this case, but hey, I'm told that knowledge is power. Translated: https://translate.google.com/translate?sl=auto&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=http%3A%2F%2Fwww.heise.de%2Fsecurity%2Fartikel%2FErpressungs-Trojaner-wie-Locky-aussperren-3120956.html&edit-text Original: https://www.heise.de/security/artikel/Erpressungs-Trojaner-wie-Locky-aussperren-3120956.html