JTok

Yes, that is exactly how it works. I was able to replicate your issue, and I'm working on it now. I'm hoping to have it fixed in the next few hours.

I know it's been a while, but I made some big-ish changes to the script that some of you may find useful. I haven't really read back through the thread yet, so I'm not sure if someone else has made any additions not included here. The biggest changes are 1. vdisk extensions are pulled directly from the config file, so you can name them anything you want (except .snap). This should prevent the issue @rorton was having. 2. SNAPSHOTS!! Snapshots are now able to be used to allow live backups, but there are some caveats so be sure to pay attention to the notes in the script when enabling it. BIG thanks to @Dikkekop for that one. 3. any extension can be blacklisted now, not just iso. v1.1.5 - 2019/11/21 In Loving Memory of Net Neutrality - fixed some errors not generating an error log file. - updated comments. - updated documentation to include important information. - reverse changelog order. - added support for more vdisk extensions by parsing them directly from the config. - added user variable (case-insensitive) for blacklisting extensions. iso included by default. - fixed issue with where files would not be removed after compressing when config files were not set to be backed up. - updated comments to use more consistent terms. - added option to allow specific vms to be backed up without shutting down. - added option to allow vms to be backed up using snapshots. disabled by default. - fixed script trying to remove leftover files that weren't part of the backup. This thing has gotten rather complex, and I only use some of the functions in my backups, so as always: I've tested it as best as I am able to, but cannot guarantee it will work for you. Please verify your backups are running correctly when using the script. Script here: https://github.com/JTok/unraid-vmbackup/tree/v1.1.5 -JTok

So... I honestly I haven't looked at this for more than a few months since it has been working for my needs and my personal life has been rather busy. However, I did start working on (and am planning to eventually finish) some updates to this script as well as a version that works with btrfs snapshots. Things have slowed down a little lately, so I've started getting back into these kind of projects. I will take another look at the vdisk skipping, plus I'm working on adding the ability to parse the vdisk names directly from the xml file as well as a few other things. Unfortunately I don't have a firm timeline right now, but I expect to be able to devote a decent amount of time over the next month or so. -JTok

@jonathanm Are you booting directly to the vdi image, or did you convert it to another format first? I only ask because since I based this on the original script, it only looks for img and qcow2 files when backing up virtual disks. In hindsight, this is rather fragile, so I think I'll try to switch over to parsing the VM configuration to get the attached disks and their extensions. (Note to self: Also include a blacklist variable so users can exclude extensions they don't want backed up like iso.) Thanks, JTok

Ahhh. Sorry, I was unclear. The the vdisk names must be unique per VM, not system-wide. They will be backed up into a folder corresponding with the VM name. e.g. /backup_location/VM1/vdisk1.img /backup_location/VM1/vdisk2.img /backup_location/VM2/vdisk1.img /backup_location/VM2/vdisk2.img So with the script as it currently stands, if you have the following vdisks attached to a VM: /mnt/user/domains/VM1/vdisk1.img /mnt/disk1/VM1/vdisk1.img They will both be copied to the /backup_location/VM1/ folder and one will overwrite the other and/or skip it. While I personally prefer a flatter structure for my backups, I like the idea of having the option to backup using the full path. That would resolve the issue with unique names. I will add it to my to-do list for the next version.

Unfortunately having vdisks with the same name wasn't something I took into consideration with the script. Since the both vdisks will be backed up to the same location the second vdisk will overwrite or ignore (depending on which options are used) the first vdisk because the script will think they are the same one. Renaming one of the disks should be a relatively safe operation, but if you are concerned you could try one of these ways: 1) Make a copy of one of the vdisks manually before renaming it and changing the VM configuration. If you are feeling extra concerned, you may also want to make a copy of the existing VM XML before changing the config to point to the new vdisk name. 2) Run the script with the vdisks_to_skip field filled out for one of the vdisks so that only one gets backed up. Then rename the one vdisk you have a backup of and change the VM configuration. IMPORTANT: You will want to make sure that you rename the disk you're changing to vdisk1.img or the backup script will calculate the retention wrong when using number_of_backups_to_keep.

v1.1.4 - 2018/05/19 A Rose by Any Other Name fixed issue with vdisks not being skipped when defined in vdisks_to_skip. fixed incorrect vdisks sometimes being copied before backup when using delta sync with multiple vdisks. fixed wrong number of vdisks being kept when a VM had multiple vdisks attached. added option to automatically enable reconstruct write (a.k.a. turbo write) during backups and then disable after backups complete. I've tested to the best of my ability, but I cannot guarantee everything will work flawlessly. Please be sure to verify your backups are running correctly and let me know of any issues. Script here: https://github.com/JTok/unraid-vmbackup -JTok

I wanted to give a quick update. I was able to replicate this issue, and I have a fix I'm testing. It also led me to the discovery of a couple other specific use case issues that I want to resolve (details below). Once I have all 3 issues resolved I'll post an updated version of the script. I am going to try to include the option to turn on turbo write during backups if I can figure it out quickly enough. The issues I am addressing in the next update: not skipping vdisks backups with multiple vdisks when using delta syncing (default) are slow because sometimes the wrong vdisk is chosen (finished backups appear to be correct, but the intermediate steps are done in the wrong order) number of backups to keep not keeping the correct number with multiple vdisks (i.e. keeping 2 vdisks on a VM with two vdisks will keep 1 copy of each because it counts both towards the total to keep instead of keeping 2 copies of each) -JTok

Thanks! I was wondering about that after reading the question. I will have to look into implementing that feature. Sent from my iPhone using Tapatalk

Weird. I am in Spain right now, but I should have some time before next week to run a few tests to see if I can replicate the issue. Just to make sure, are you putting the full path of the vdisk to skip? Also, are you able to pm me the log file? Thanks. Sent from my iPhone using Tapatalk

v1.1.3 - 2018/04/16 From Here to Infirmary included inline variable to prevent script from running if a parity check is in progress. fixed spelling errors in script. added advanced option to ignore iso files during vdisk check. #UNTESTED fixed so tarballs so files are added without full paths. updated user selectable options so that they only generate a warning in the log, not a full error log file. As always, use at your own risk with no guarantees. Script here: https://github.com/JTok/unraid-vmbackup -JTok

@joelones Sorry for the delay, I somehow missed the notification for this. It does look like everything completed to me as well. The script is throwing an error on the iso file listed here: 2018-04-11 11:30 warning: /mnt/user/kvm/isos/mini.iso of srv-ubuntu is not a vdisk. skipping. It sees that it is not a vdisk and generates a warning. I can add an option to the next version to ignore iso files. As for the compressed files containing the full path, that is something I was aware of, but just didn't get around to changing before the last update. I am planning on having it compress without the path in the next version. -JTok

Okay, let's try this again. I've put a new version out. It addresses the concerns of @joelones and @PieQuest as well as a few other fixes and enhancements. Added option to only receive error notifications. Added option to receive detailed notifications. Sends notifications when VM backups are started and stopped. Sends notifications when old backups are deleted. Sends notifications when old log files are deleted. Added option to backup nvram. Enabled by default. Fixed issues with notification system. Not all failures and warnings were using notifications. send_notifications variable wasn't disabling many notifications. Fixed issues with tarball backups not removing original files. Fixed issues with log files not being removed. Fixed logging issues. Detailed logging for tarballs. Logging for rm commands. Updated notification message language. Updated script comments/directions. Same deal as the previous updates. I've tested as best I can, but I can't guarantee everything. The script can still be found here: https://github.com/JTok/unraid-vmbackup -JTok

It's because I made a mistake with quotation marks I think. I'm on my phone, so it's a little tough to tell for sure, but I'm pretty sure that's what happening. I will make sure it's fixed in the release for next week. [emoji15] Sorry about that. Sent from my iPhone using Tapatalk

That should be easy enough to add. I'll add it to my to-do list. I'm planning on having the next version out sometime next week, and I think I should be able to get this into that release.

Ahh yeah. I see that the Quadro cards are more capable. That's good to know. Plex also has some limitations listed in the link I posted earlier. They only support decoding on Nvidia in Windows. To decode on Linux you will need Intel. I'm not sure if this is a limitation in Plex or something else (I would assume it's Plex though, so presumably support will get better). Of course, this may not be a major issue for your use case. Sent from my iPhone using Tapatalk

There are some limitations associated with Plex and hardware acceleration that you may want to familiarize yourself with before spending a lot of money on hardware. For one, Nvidia limits encoding to 2 streams on at least some (if not all) of their GPUs. There are also potential issues with dockers, if that's how you are running it. I would read through this page as well as the Plex forums to get a feel for the best purchase for what you want to do. It may be in your best interest to re-ask this question there as well. https://support.plex.tv/articles/115002178853-using-hardware-accelerated-streaming/ Sent from my iPhone using Tapatalk

I have made a few updates to the script again. Added the option to disable delta syncs. Added option to only use rsync. Dry-runs can be used with the rsync_only option. Dates and times are now included in log messages. Log files can be generated (enabled by default) Error logs can be managed separately from regular logs. Option to choose the number of logs to keep. Log messages can be sent through unRAID notification system. As with the last update, I've tested as best I can, but I can't guarantee everything. The script can still be found here: https://github.com/JTok/unraid-vmbackup -JTok

A few other words of advice: Avoid using the DMZ on your router. Only forward the ports needed to the device that needs it. Make sure that whatever username/password you use for accessing your shares isn't the same username/password you use to manage your game servers. (Honestly, I highly recommend using a unique username/password for accessing shares. Depending on your level of paranoia you can tell your desktop to remember it or not.) I don't know your level of technical expertise with networking, but you could consider VLANs (potentially complicated)... or if you will have two NICs on your server you could set up two physical networks (potentially less complicated). Either option could be set up in such a way that it isolates your game servers from the rest of your network to mitigate some of the risks. Both options could carry some additional costs if you don't already have the equipment available.

Hello, I just happened to notice that after using this that it seems to de-sparsify (un-sparsify?) files. Rsync has a --sparse option, but I'm not sure what effect that would have on non-sparse files. Specifically my vdisk backups were affected. If I had moved docker.img, or libvirt.img, they would presumably have been affected as well. I also have some dockers that create sparse files. Disclaimer: Unfortunately, I forgot to double-check the files it de-sparsed (de-sparsified?) before I removed them, so maybe I'm wrong about this. It would be worth double-checking. You could run a check like this to return a list of sparse files. The command searches sub-directories. find /directory/to/be/copied -type f -printf "%S\t%p\n" | gawk '$1 < 1.0 {print}' Hopefully this information is useful to someone. Thanks, JTok Edit: I forgot to include a link to the code source: https://www.thegeekdiary.com/how-to-find-all-the-sparse-file-in-linux/

I have updated the script on github to include the following changes: Moved variables close to the top to make them easier to access. Added the ability to skip specific vdisks. Restoring the VMs to their previous state can be overridden. Backing up xml and/or vdisks can be skipped. Number of days to keep backups can be set. Changed to use mmin instead of mtime because of issues with accuracy. see: https://unix.stackexchange.com/a/92351 Number of backups to keep can be set. Always checks for img and qcow2 files by default. Timestamps are always used except when the number of backups being kept is set to 1. Toggle for compressing backups. Option to compare backups to original file and retry once. A few other minor things. Currently I've tested everything as well as I can, but I can't guarantee everything will work, so please feel free to let me know if you find any issues. I've tried to make it easy and customizable as possible. You can find the script here: https://github.com/JTok/unraid-vmbackup -JTok

I have created a variation on the original script that uses cp for the initial copy and then uses rsync for subsequent copies. I did a whole lot of testing on my own system and I found that while dd was slightly faster than cp, it did not create sparse files very well. Rsync was so much slower that it was nearly pointless to run. It did sparse files about as well as dd though. All of my tests were from my SSD cache array to my storage array. I also found that using rsync to delta sync was about as fast as re-copying the whole file using cp/dd when using --inplace. However, since the inplace argument creates some risk, and I did want to keep one rolling backup, I have the script make a copy first. Effectively the script does this: First run: uses cp to create a sparse copy of your vdisk. It also makes a copy of your xml. Second run: BEFORE shutting down the VM it makes a copy of the backup from step one using cp. Then it shuts down the VM and uses rsync to perform a delta sync with the inplace argument. Subsequent runs: BEFORE shutting down the VM it uses rsync to perform a delta sync of the latest backup to the second copy with the inplace argument. Then it shuts down the VM and uses rsync to perform a delta sync with the inplace argument. I also changed the script to make a note of the state of the VM when it starts the backup so that it can restore the VM to that state. I have some VMs that I don't keep running, so I don't want them started after a backup. Paused VMs will just be left in the stopped state. I *think* the timestamps will still work with this, but it will just use the cp option for everything. I'm also pretty sure it will keep making them forever without clearing them out. I am working on fixing that. You can find the script here: https://github.com/JTok/unraid-vmbackup Please be sure to read the Change Log and To-Do List as they contain important notes about the state of the script currently. -JTok Edit 1: forgot to mention: the original script does not make sparse copies. Additional explanation: When you create a VM you choose a disk size. Let's say we pick 40GB. Then we install an OS. The OS is taking up 10GB of that 40GB, so the vdisk will say it is 40GB, but only take up 10GB of actual space on the storage disk. This is highly desirable with lots of VMs. However, the original script didn't use the --sparse argument with rsync so it would create a 40GB vdisk that took up 40GB of actual space on the storage disk even though 30GB of that was actually just zeros. So sparse files are basically files that the file system knows are smaller than they appear to be, because they don't have all the unnecessary zeros in them. Edit 2: I have tested restoring the files using cp from the shell to put them back and they do work. However, I have not tested restoring them via the shares directly. You have been warned.

I have been reading about how this plugin works, and I have an alternative method that might work as well (rumor has it that @Squid likes feature creep). Hopefully I didn't miss a post in this thread where someone already talked about this method; I tired to read through the whole thing before posting. I have some experience detecting ransomware in Windows environments, and we use a slightly different method to detect it on file shares that could be appealing to some here. Instead of using bait files we just monitor files against a list of known extensions and file-names used by ransomware, and use a white-list to override it for application specific extensions that we might have in our environment. The pros: - You know when I file gets modified without relying on the ransomware to pick your bait files/shares first - You don't have bait files/shares The cons: - One of your actual files has to be encrypted before you know it's happening (though if a bait file is not picked first, you may still be in the same boat or worse) - You are relying on a list to be kept up-to-date I would love to now offer you some code and a more concrete explanation of how I think this could be implemented, but it has been some time since I had to do anything beyond the most basic scripting in Linux. I'm also assuming that this is possible, and not a burden on the system; I just don't have enough experience to guess at at this point. Widows reference materials (the first link maintains a list of known bad files/extensions here): https://fsrm.experiant.ca/ https://community.spiceworks.com/how_to/100368-cryptolocker-canary-detect-it-early Another alternative is some kind of behavior monitoring (i.e. watching for encryption), but I suspect that is far more difficult to implement. (We use this as well, but pay handsomely for the feature.) Regardless, thank you for the plugin (and all the others). -JTok Quick Edit: Forgot to mention that I've seen fail2ban be used for something similar by reviewing samba logs. The original is in German, so here is the google translate link and the original. Not sure it makes sense to use in this case, but hey, I'm told that knowledge is power. Translated: https://translate.google.com/translate?sl=auto&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=http%3A%2F%2Fwww.heise.de%2Fsecurity%2Fartikel%2FErpressungs-Trojaner-wie-Locky-aussperren-3120956.html&edit-text Original: https://www.heise.de/security/artikel/Erpressungs-Trojaner-wie-Locky-aussperren-3120956.html

Follow-up: I apparently spoke too soon. The issue has returned. I'm still reasonably certain that this isn't a hardware problem, but I have spare components I can swap in (despite the current hardware having passed every test I've run). It will be about a week or two before I'm able to get everything swapped and tested, but I will report back once done in case it matters.

This seemed to fix the issue for me as well. For the sake of complete records: I am running Turnkey Linux Core (Debian based); https://www.turnkeylinux.org.