Use symlinks to make libvirt storage domain a persistent location

[scorcho99]

So I followed space invader one's great guide to link unraid to virt-manager and have converted my disks to qcow2. Snapshots on the disks work fine. I also hibernate/save a VM so that I can go straight back to where I was instead of rebooting it.

 

The issue I discovered is that snapshots disappear from virt-manager after the unraid host is rebooted. I also had a hibernate fail the other day. These things both have the same root cause I believe, which is that the libvirt folder is stored in memory on unraid. This means that while the qcow2 disk snapshots still exist (I've confirmed this) the greater vm snapshot disappears and thus virt-manager can't see beyond that. And hibernation/saving the VM is actually silly because it just copies it from ram to a ram disk. Which if you don't have enough ram left will obviously fail.

 

So I think the simplest solution is to create a symlink of the relevant folders where snapshots and vm state information are stored to the cache drive. I'd probably than put that symlink command in the go file. It seems like some users on here have done some things like this so but is this a stupid idea for some reason I'm not aware of?

 

I'm also a little unsure which folders I should symlink or if it would be best to symlink /var/lib/libvirt in its entirety. I don't know where the vm state is stored but I've seen the snapshots folder so I think I've found that.

[scorcho99]

So after doing some reading I came up with the following script based on one I saw dmacias use for a similar purpose. It appears to work exactly how I want, snapshots and save states persist after reboots and are stored on the cache drive instead of in ram. I chose to only link the save and snapshot folders rather than trying out the entire qemu directory.  There's also nvram, ram, dump folders that might be of interest but I'll just cross those bridges when I get to them.

 

How do I assure this script runs at the right time? If I link to a /mnt/ location for a script in the go file will it work? And will it run after libvirt has been unpacked?

 

 

 

# S00slqemu.sh

#!/bin/bash
# stops libvirt and moves the save and snapshot folders creating a symlink to them
#

LVSAVEDIR=/mnt/cache/domains/save
LVSNAPSHOTDIR=/mnt/cache/domains/snapshot

if [ -f /var/run/libvirt/libvirtd.pid ];then
    /etc/rc.d/rc.libvirt stop
fi

if [ $LVSAVEDIR != "/var/lib/libvirt/qemu/save" ];then
    if [ ! -L /var/lib/libvirt/qemu/save ];then
        if [ ! -e $LVSAVEDIR ];then
            mv /var/lib/libvirt/qemu/save $LVSAVEDIR
        else
            rm -r /var/lib/libvirt/qemu/save
        fi                
    fi    
    ln -s $LVSAVEDIR /var/lib/libvirt/qemu/save
fi

if [ $LVSNAPSHOTDIR != "/var/lib/libvirt/qemu/snapshot" ];then
    if [ ! -L /var/lib/libvirt/qemu/snapshot ];then
        if [ ! -e $LVSNAPSHOTDIR ];then
            mv /var/lib/libvirt/qemu/snapshot $LVSNAPSHOTDIR
        else
            rm -r /var/lib/libvirt/qemu/snapshot
        fi                
    fi    
    ln -s $LVSNAPSHOTDIR /var/lib/libvirt/qemu/snapshot
fi

/etc/rc.d/rc.libvirt start

 

[BobPhoenix]

Use the User Scripts plugin.  Probably set it to run at array start.

[scorcho99]

Thanks, yeah I ended up there anyway and did exactly that. Seems to work fine with array start. I'm not sure if shutting down libvirt like in the script will handle vms that are running but that would only be a potential problem if I set them to auto start.

 

Only remaining issue I have left is I can't copy the save and snapshot files off because I don't have permissions. I can change them but it doesn't seem to take, and whenever a new file is created it sets it to root owned anyway. Its a pretty minor issue honestly, I don't necessary need this backed up it would just be nice.

[BobPhoenix]

You should be able to copy them as root.  So you should be able to login to the console and copy them.  I use MC to copy files on unRAID.  Not sure what user the User Scripts run under but if it is root then it should copy just fine.  If you can't copy them as root then they must be open somewhere.

[scorcho99]

Yeah, I can copy them as root but when I try to access them over smb I can't even read the files.

 

I'm pretty sure the scripts run under root. I may just use another script to help the backup along.

[BobPhoenix]

OK I thought you said you were having problems copying them.  To access them over SMB you need to set the permissions appropriately in your script that copies them.  Array files have to be nobody:users.  I would tell you to use new permissions but that is for drives in the array and the cache.

[itimpi]

Actually if you are using a script then you can use

 

newperms path

 

against any valid path (including ones set up by the UD plugin) and it will work just fine.

 

[Squid]

44 minutes ago, BobPhoenix said:

Array files have to be nobody:users.

Actually they can be any user or group.  But the world permission should be "7"

 

ie: chmod 0777 -R /path/to/folder

[scorcho99]

I suppose I'm getting a bit off the original topic though. I solved the symlink to make qemu components persistent. I'm really just planning a backup script at this point.

 

So, I'm not a little confused about permissions. But I'm not the best with linux permissions to start with. Does unraid use its own layer on top of the standard linux permissions?

 

20 hours ago, BobPhoenix said:

OK I thought you said you were having problems copying them.  To access them over SMB you need to set the permissions appropriately in your script that copies them.  Array files have to be nobody:users.  I would tell you to use new permissions but that is for drives in the array and the cache.

I did chown (with -R or whatever recursive option is) nobody and I believe nobody:users. The permissions from ls -l looked the same for the other files in the directory aside from the owner. IIRC I still had the problem so wasn't sure what was going on at that point. One related issue was that I noticed the permissions wouldn't "stick" since creation of a new .save file was done by root and set the permissions to root again.

 

20 hours ago, itimpi said:

Actually if you are using a script then you can use

 

newperms path

 

against any valid path (including ones set up by the UD plugin) and it will work just fine.

 

I'll try that. Googling it's a custom script for fixing permissions included with unraid.

 

20 hours ago, Squid said:

Actually they can be any user or group.  But the world permission should be "7"

 

ie: chmod 0777 -R /path/to/folder

Yeah, I noticed a file I plopped in there manually through smb a connection had my user name instead of nobody.

[munchies2x]

scorcho99, did you manage to make this work? I want to change the Hibernate button to Saved state. That way I can just start the VM and get back to my work straight away...

 

Is there a way to change how the default setting works?

[scorcho99]

On 1/28/2019 at 2:13 AM, munchies2x said:

scorcho99, did you manage to make this work? I want to change the Hibernate button to Saved state. That way I can just start the VM and get back to my work straight away...

 

Is there a way to change how the default setting works?

Pretty old post, sorry I don't check these forums to often. Its possible that script is kind of buggy and I've updated it but I believe it at least mostly worked.

 

This has been working well for me for quite awhile. I'm not sure about the hibernate button, but through virt-manager and I believe some scripts I use for backups I can trigger the save of the VM state to disk. IIRC its similar to a guest driven hibernate but not quite the same. I vaguely remember giving up on guest driven hibernate/save for some reason or other. The guest time will be incorrect upon restore of the vm save state, but it corrects itself within a minute or two probably due to NTP time server.

 

One gotcha I recall: snapshots would not work with UEFI based VMs. There was some bug or limitation in how nvram states were stored that prevent it from working and last time I checked no one had fixed it. This didn't bother me, I just run everything with seabios.

[scorcho99]

Some one reported some issues with the script I wrote in another thread. I pulled the one I've been using off the server and I think it corrected the bug. No warranties on this, I wouldn't say I'm a bash expert. Note: LVSAVEDIR and LVSNAPSHOTDIR are probably different on your machine so change those to what you want.

 

# S00slqemu.sh

#!/bin/bash
# stops libvirt and moves the save and snapshot folders creating a symlink to them
#

LVSAVEDIR=/mnt/cache/VMs/qemu_nv/save
LVSNAPSHOTDIR=/mnt/cache/VMs/qemu_nv/snapshot

if [ -f /var/run/libvirt/libvirtd.pid ];then
    /etc/rc.d/rc.libvirt stop
fi

if [ $LVSAVEDIR != "/var/lib/libvirt/qemu/save" ];then
    if [ ! -L /var/lib/libvirt/qemu/save ];then
        if [ ! -e $LVSAVEDIR ];then
            if [ ! -e /var/lib/libvirt/qemu/save ];then
                mkdir $LVSAVEDIR
            else
                mv /var/lib/libvirt/qemu/save $LVSAVEDIR
            fi
        else
            rm -r /var/lib/libvirt/qemu/save
        fi                
    fi    
    ln -s $LVSAVEDIR /var/lib/libvirt/qemu/save
fi

if [ $LVSNAPSHOTDIR != "/var/lib/libvirt/qemu/snapshot" ];then
    if [ ! -L /var/lib/libvirt/qemu/snapshot ];then
        if [ ! -e $LVSNAPSHOTDIR ];then
            if [ ! -e /var/lib/libvirt/qemu/shapshot ];then
                mkdir $LVSNAPSHOTDIR
            else
                mv /var/lib/libvirt/qemu/snapshot $LVSNAPSHOTDIR
            fi
        else
            rm -r /var/lib/libvirt/qemu/snapshot
        fi                
    fi    
    ln -s $LVSNAPSHOTDIR /var/lib/libvirt/qemu/snapshot
fi

/etc/rc.d/rc.libvirt start

 

[arturovf]

On 7/25/2019 at 6:25 AM, scorcho99 said:

Some one reported some issues with the script I wrote in another thread. I pulled the one I've been using off the server and I think it corrected the bug. No warranties on this, I wouldn't say I'm a bash expert. Note: LVSAVEDIR and LVSNAPSHOTDIR are probably different on your machine so change those to what you want.

 

# S00slqemu.sh

#!/bin/bash
# stops libvirt and moves the save and snapshot folders creating a symlink to them
#

LVSAVEDIR=/mnt/cache/VMs/qemu_nv/save
LVSNAPSHOTDIR=/mnt/cache/VMs/qemu_nv/snapshot

if [ -f /var/run/libvirt/libvirtd.pid ];then
    /etc/rc.d/rc.libvirt stop
fi

if [ $LVSAVEDIR != "/var/lib/libvirt/qemu/save" ];then
    if [ ! -L /var/lib/libvirt/qemu/save ];then
        if [ ! -e $LVSAVEDIR ];then
            if [ ! -e /var/lib/libvirt/qemu/save ];then
                mkdir $LVSAVEDIR
            else
                mv /var/lib/libvirt/qemu/save $LVSAVEDIR
            fi
        else
            rm -r /var/lib/libvirt/qemu/save
        fi                
    fi    
    ln -s $LVSAVEDIR /var/lib/libvirt/qemu/save
fi

if [ $LVSNAPSHOTDIR != "/var/lib/libvirt/qemu/snapshot" ];then
    if [ ! -L /var/lib/libvirt/qemu/snapshot ];then
        if [ ! -e $LVSNAPSHOTDIR ];then
            if [ ! -e /var/lib/libvirt/qemu/shapshot ];then
                mkdir $LVSNAPSHOTDIR
            else
                mv /var/lib/libvirt/qemu/snapshot $LVSNAPSHOTDIR
            fi
        else
            rm -r /var/lib/libvirt/qemu/snapshot
        fi                
    fi    
    ln -s $LVSNAPSHOTDIR /var/lib/libvirt/qemu/snapshot
fi

/etc/rc.d/rc.libvirt start

 

thanks for the script!  where do you put it? I put it at array start and ended with /snapshot/snapshot/snapshot. etc

[scorcho99]

1 hour ago, arturovf said:

thanks for the script!  where do you put it? I put it at array start and ended with /snapshot/snapshot/snapshot. etc

 

I put mine as "at array first start"

 

I remember having a lot of problems with the chained directories and can't remember if I figured it out. I had to clean them all out before I got things appearing to work OK again though. I think the symlinks were messed up when the script was run to many times so it kept creating redundant ones. Its been to long since I messed with this unfortunately.

 

If I were to improve the script I would make it better handle checking for the symlinks existing.

[arturovf]

Yes I recall this to work properly when I first tried it, however I don't use snapshots that much so I let it be.   Checking it today I had nested directories 

Edited July 1, 2021 by arturovf