JTok

Community Developer
  • Posts

    100
  • Joined

  • Last visited

  • Days Won

    1

Everything posted by JTok

  1. @ncohafmuta and @IronBeardKnight you can bypass pretty much every safety check that won't actually break the script in the Danger Zone tab. Scroll all the way to the bottom and look for Disable restrictive validation and set it to Yes, after that you can set Number of backups to keep to 1 without the script complaining (screenshots attached). I am intending to update the help to note which fields can be deactivated and where to do that in a future update, but for now hopefully this helps. fwiw, the plugin is very customizable and I didn't want to prevent anyone from doing something just because I considered it non-standard (this is one of my biggest pet-peeves with software!). Because of that, I made sure to include options to disable anything that wasn't explicitly breaking. The only reason it doesn't let you do it out of the box is because it is too easy for someone who just wants an easy-to-use backup plugin to make a mistake that doesn't protect them or causes data issues. I highly recommend reading through the help for the more advanced options if you ever run into something like this again... or even just for good measure - there may be other useful things in there you want to use!
  2. This is something I found right after the last release. It will be fixed in the next version. Glad you figured it out though. If anyone else runs into this, you can also fix it by running this from the terminal (which will force the vmbackup plugin to fully uninstall). removepkg vmbackup*
  3. Yeah, that should be the only thing you need to change. If it isn't, then I have something to fix haha
  4. @CS01-HS Your issue can be fixed by going to the Danger Zone tab and setting "Disable restrictive validation" to "Yes" @peter76 your issue can be fixed by going to the Danger Zone tab and setting "Disable custom cron validation" to "Yes" Additional Information: @CS01-HS There are limits in place to prevent people from choosing values that could be dangerous without making sure they understand them. Also, if you only need 1 backup, why not just use "Number of Backups to Keep" and set it to 1 instead of using "number of days to keep backups"? @peter76 With cron it is very complicated to validate all possible options, so I went with something that would cover most scenarios while leaving the disable switch to allow for more complex ones like yours. I've been trying to make the plugin's default state the bare minimum amount of restrictive necessary to help protect less experienced users from themselves, while still giving power users the ability to remove those restrictions and be more adventurous. Nearly every setting can can be overridden somewhere, but I know that isn't very obvious right now. Making that more apparent is on my list of things to do in a future release.
  5. Normally I would have, but I just didn't have a chance to yet. My life is still a bit of a mess after moving (I'm currently sitting on the floor while I type this because most of my furniture won't arrive for another week haha). I'm planning on incrementing the version later today though, but I wanted to get the quick and dirty fix out sooner rather than later.
  6. I finally found the issue. For some reason, several of the files had their EOL character changed when I pushed the last update (I'm blaming GitHub). As long as you have removed the broken version this should work. If you get an error saying that the plugin is already installed when you try to install this version, you will need to remove the the old version. If you have already removed the old version and are still getting an error saying that the plugin is already installed, run the following terminal command and try again. removepkg vmbackup*
  7. I just wanted to check in and say I am still working on this. Unfortunately my new ISP couldn’t get my Internet going yesterday, and had to schedule another appointment for earlier today. Which meant I wasn’t able to start on this until now. Hopefully it is a quick fix [emoji1696] Sent from my iPhone using Tapatalk
  8. Dang. Sorry for the issues with the latest version. I wasn’t having them on my server (although mine was still on 6.8.3), but I must have messed something up when I uploaded the latest version to GitHub. I am arriving at my new place this afternoon, so I’ll update my server and try to get a fix out tonight. Sent from my iPhone using Tapatalk
  9. Whoops! typed the wrong filename. I think it should work now.
  10. In light of the official release of 6.9.0 I have removed the max version restriction. I am not able to confirm how well it works this week because I am moving across several time-zones. I am also still working on a new version. As a quick heads up, the new version will be dropping support for pigz, but gzip will still work for anyone using legacy compression (this change should not be breaking or require any user input). I also fixed a few bugs, and am adding a few new features.
  11. Hello, long time no see. I am truly sorry to see so many of you have had an issue with this plugin, and it was not my intention to abandon it for as long as I have. Sadly, life had other plans (as it often does). I've recently found myself with time to tinker again, and as such I've released an update that does a few things to try and address some of the issues I am aware of. Unfortunately, I haven't been able to replicate many of the issues others are having, so my ability to test has been limited. I also can see that the operation of some of the advanced functions isn't immediately clear either. A few of the issues people have had could have been resolved with a settings change in one of the more advanced settings tabs (usually Danger Zone). I'll try to find a way to make some of those things clearer as I progress on addressing the larger issues. The big issue I'm attempting to address in the new release is the issue some people have had with the array not wanting to stop. I have adjusted how the backup scripts are checked and it will hopefully be able kill stuck ones more readily. The other thing I have done is set a max version of 6.8.3 until I have time to test on 6.9. I currently do not have a test server, so it could be a bit before I have a chance to spin one up for troubleshooting. Again, I do sincerely apologize for those that feel I left them in the lurch. Thankfully, as others have rightly pointed out, the script that the plugin uses on the back-end is much more stable and a solid way to go. I would also like to add that I am more that willing to add contributors to the project if others would like to help me maintain it. Best, JTok
  12. For what it is worth, the direct mount did not actually fix it for me, just obfuscate it by making it so loop2/3 didn't show up in iotop. When I checked the SMART status I wound up with just as many writes as before, so I would be interested to hear your results. It seems like I might be an outlier here, so possibly I have a different issue affecting my setup.
  13. A note on this from my experience. I actually did this and it worked, but when I went to make changes to the dockers it failed and they would no longer start. I think I could have fixed it by clearing out the mountpoints, but I opted to just wipe the share and then re-add all the dockers from "my templates" which worked just fine. So, can confirm --would not recommend. I did copy the libvirt folder though and have not noticed any ill effects (...yet. haha)
  14. It's funny you mention that. I haven't rolled the workaround back yet, but I'm starting to wonder if it isn't fixing the issue so much as obfuscating it. I'm planning on rolling the fix back later today/tonight to see what happens. I got some metrics by basically doing the same thing as your script (except manually). According to SMART reporting, my cache drives are writing 16.93GB/hr. This is even though when I implemented the workaround I also moved several VMs and a few docker paths to my nvme drives just to reduce the writes further. I'd be curious to know what others are seeing.
  15. For anyone else that needs it, I was having more issues with libvirt/loop3 than docker/loop2, so I adapted @S1dney's solution from here for libvirt. A little CYA: To reiterate what has already been said, this workaround is not ideal and comes with some big caveats, so be sure to read through the thread and ask questions before implementing. I'm not going to get into it here, but I used S1dney's same basic directions for the docker by making backups and copying files to folders in /boot/config/. Create a share called libvirt on the cache drive just like for the docker instructions. edit rc.libvirt 's start_libvirtd method as follows: start_libvirtd() { if [ -f $LIBVIRTD_PIDFILE ];then echo "libvirt is already running..." exit 1 fi if mountpoint /etc/libvirt &> /dev/null ; then echo "Image is mounted, will attempt to unmount it next." umount /etc/libvirt 1>/dev/null 2>&1 if [[ $? -ne 0 ]]; then echo "Image still mounted at /etc/libvirt, cancelling cause this needs to be a symlink!" exit 1 else echo "Image unmounted succesfully." fi fi # In order to have a soft link created, we need to remove the /etc/libvirt directory or creating a soft link will fail if [[ -d /etc/libvirt ]]; then echo "libvirt directory still exists, removing it so we can use it for the soft link." rm -rf /etc/libvirt if [[ -d /etc/libvirt ]]; then echo "/etc/libvirt still exists! Creating a soft link will fail thus refusing to start libvirt." exit 1 else echo "Removed /etc/libvirt. Moving on." fi fi # Now that we know that the libvirt image isn't mounted, we want to make sure the symlink is active if [[ -L /etc/libvirt && -d /etc/libvirt ]]; then echo "/etc/libvirt is a soft link, libvirt is allowed to start" else echo "/etc/libvirt is not a soft link, will try to create it." ln -s /mnt/cache/libvirt /etc/ 1>/dev/null 2>&1 if [[ $? -ne 0 ]]; then echo "Soft link could not be created, refusing to start libvirt!" exit 1 else echo "Soft link created." fi fi # convert libvirt 1.3.1 w/ eric's hyperv vendor id patch to how libvirt does it in libvirt 1.3.3+ sed -i -e "s/<vendor id='none'\/>/<vendor_id state='on' value='none'\/>/g" /etc/libvirt/qemu/*.xml &> /dev/null # remove <locked/> from xml because libvirt + virlogd + virlockd has an issue with locked sed -i -e "s/<locked\/>//g" /etc/libvirt/qemu/*.xml &> /dev/null # copy any new conf files we dont currently have cp -n /etc/libvirt-/*.conf /etc/libvirt &> /dev/null # add missing tss user account if coming from an older version of unRAID if ! grep -q "^tss:" /etc/passwd ; then useradd -r -c "Account used by the trousers package to sandbox the tcsd daemon" -d / -u 59 -g tss -s /bin/false tss fi echo "Starting libvirtd..." mkdir -p $(dirname $LIBVIRTD_PIDFILE) check_processor /sbin/modprobe -a $MODULE $MODULES /usr/sbin/libvirtd -d -l $LIBVIRTD_OPTS } Add this code the the go file in addition to the code for the docker workaround: # Put the modified libvirt service file over the original one to make it not use the libvirt.img cp /boot/config/service-mods/libvirt-service-mod/rc.libvirt /etc/rc.d/rc.libvirt chmod +x /etc/rc.d/rc.libvirt
  16. That's definitely something I would be willing to implement at some point, but there is other functionality that I would like to ensure is working before I try to mess with it. Unfortunately, I cannot promise more than that right now.
  17. I am currently focusing on implementing a restore function next. Based on feedback, I have decided that in order for restore functionality to work, you will need to allow the plugin to convert the existing backup structure. This will be optional, but without it, you will need to perform restores manually. As development progresses, I will work out additional details and reach back out here for feedback.
  18. Some behind the scenes changes thanks to @sjerisman that include zstandard compression and updates to the logging. Make sure you read the help if you are going to be switching from legacy gzip compression to zstandard. I also made some changes to how configs are shown to the users that should make it clearer how they work, and what is currently being edited. Full change-log below. v0.2.1 - 2020/02/20 Pika Pika - merged changes from unraid-vmbackup script v1.3.1. - added seconds to logs. - added option to use zstandard compression. - added option to create a vm specific log in each vm's subfolder. - added config drop-down selection to the top of each tab. - updated method used to determine cpu thread count. https://github.com/JTok/unraid.vmbackup/tree/v0.2.1 -JTok
  19. I think I might have an idea. How are you having the backups cleaned? Are you using "Number of days to keep backups" or "Number of backups to keep"? Because it looks like you are using "Number of backups to keep". In order for that to properly function, the vdisks must be named in a specific way. If you expand the help, it should give the following info: "If a VM has multiple vdisks, then they must end in sequential numbers in order to be correctly backed up (i.e. vdisk1.img, vdisk2.img, etc.)." -JTok
  20. @torch2k Hmmmm 🤔. Okay, just did some checking and it looks like docker looks for pigz by default. That said, when the plugin installs pigz, it does add /usr/bin/unpigz. Possibly your installation was corrupted somehow, since I just checked my dockers and two of them needed updated. I then ran Update All without issue. It seems like reinstalling the plugin could fix the issue for you... unless I'm missing something? -JTok
  21. Will push a fix. Thx. Sent from my iPhone using Tapatalk
  22. It shouldn’t matter what the extension is, but I’ll create a qcow2 drive this afternoon and run some tests. Sent from my iPhone using Tapatalk
  23. That's a good idea, I was a little worried about messing with people's backups, so that's why I was leaning towards the more hands-off approach. Though I suppose it would be easy enough to make it optional 🤔, and even include a button in Danger Zone to do it manually at a later time if it's opted out of it during the initial transition.
  24. Sorry, I was unclear. What I meant was that once you switch to the new structure, backups made under the old structure will need manually cleaned out until only new structure backups exist. Backups made under the new structure would still be able to be cleaned out. I will update the original post to clarify this. Sent from my iPhone using Tapatalk
  25. I am looking to get some feedback on the following: I'd like to change how the plugin saves backups, but it would be a breaking change. I want to change the timestamp from being part of the filename to being a sub-folder. Instead of this structure: /backup_dir/vm_name/20200204_1555_vm_name.xml/backup_dir/vm_name/20200204_1555_asdf564asd12sd5dfsd.fd/backup_dir/vm_name/20200204_1555_vdisk1.img/backup_dir/vm_name/20200204_1555_vdisk2.img/backup_dir/vm_name/20200205_1555_vm_name.xml/backup_dir/vm_name/20200205_1555_asdf564asd12sd5dfsd.fd/backup_dir/vm_name/20200205_1555_vdisk1.img/backup_dir/vm_name/20200205_1555_vdisk2.img Backups would have this structure: /backup_dir/vm_name/20200204_1555/vm_name.xml/backup_dir/vm_name/20200204_1555/asdf564asd12sd5dfsd.fd/backup_dir/vm_name/20200204_1555/vdisk1.img/backup_dir/vm_name/20200204_1555/vdisk2.img/backup_dir/vm_name/20200205_1555/vm_name.xml/backup_dir/vm_name/20200205_1555/asdf564asd12sd5dfsd.fd/backup_dir/vm_name/20200205_1555/vdisk1.img/backup_dir/vm_name/20200205_1555/vdisk2.img this should come with a few advantages: implementing restore functionality should be much easier and therefore quicker to implement. structure of backups will be less cluttered. handling files not part of a normal vm structure (such as in the case of SpaceInvaderOne's Macinabox) will be easier to handle. However, this will break the existing file cleaning functions. So you would need to manually clean out old backups until all your backups were part of the new structure. So once you switch to the new structure, only new structure backups would be cleaned by the plugin. So with that in mind, here is my proposal: Upon installation of whatever version contains this new structure, the first time you open the plugin page it will prompt you to choose which structure you want to use with a warning message (and a link to further explanation). Don't worry if the plugin auto-updates before you choose a structure, it will continue to use the old structure so your backups will still run even if you haven't picked one yet. Moving forward, development will focus on the new structure. the features that are part of the old structure will stay and keep working, but will not be updated to support new functionality. This will almost certainly include restoring and scenarios like Macinabox. This way, everything continues to work as it has in the past, and gives everyone time to transition at their own pace. Then, somewhere down the line a discussion regarding removing the old structure entirely may take place if it becomes necessary. Thoughts? Concerns? Thanks, JTok