Hoping someone can help here so that I do not lose all of my VM machines.

 

 

Used mover to empty cache ssd to check it with Crucial windows program. Looked like it was empty. Checked it in windows machine. All ok. Put it back into Unraid and used mover to add docker and kvm (libvirt files) back onto the cache.

 

Restarted Docker & recovered all containers.

 

But now libvirt service fails to start.

 

Libvirt Log

2020-03-14 14:01:57.797+0000: 23130: info : libvirt version: 5.10.0
2020-03-14 14:01:57.797+0000: 23130: info : hostname: TOWER
2020-03-14 14:01:57.797+0000: 23130: error : qemuStateInitialize:837 : unable to set ownership of '/var/lib/libvirt/qemu/save' to 0:100: No such file or directory
2020-03-14 14:01:57.797+0000: 23130: error : virStateInitialize:668 : Initialization of QEMU state driver failed: unable to set ownership of '/var/lib/libvirt/qemu/save' to 0:100: No such file or directory
2020-03-14 14:01:57.797+0000: 23130: error : daemonRunStateInit:839 : Driver state initialization failed

 

Diagnostics attached.

tower-diagnostics-20200314-1020.zip

Rebooting should fix it.

Rebooted 2x. Does not fix the problem. I also noticed that the system gets hung trying to unmount disks whn rebooting. Have to force reboot, which obviously causes unclean shutdown.

 

Any other ideas?

 

Thanks in advance.

On the diags posted libvirt started correctly at first array start:

Mar 14 10:01:57 TOWER emhttpd: shcmd (463): /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 2
Mar 14 10:01:57 TOWER kernel: BTRFS: device fsid 473e1a30-4df5-4093-9659-11deafd49e06 devid 1 transid 1987 /dev/loop3
Mar 14 10:01:57 TOWER kernel: BTRFS info (device loop3): disk space caching is enabled
Mar 14 10:01:57 TOWER kernel: BTRFS info (device loop3): has skinny extents
Mar 14 10:01:57 TOWER root: Resize '/etc/libvirt' of 'max'
Mar 14 10:01:57 TOWER kernel: BTRFS info (device loop3): new size for /dev/loop3 is 2147483648
Mar 14 10:01:57 TOWER emhttpd: shcmd (465): /etc/rc.d/rc.libvirt start
Mar 14 10:01:57 TOWER root: Starting virtlockd...
Mar 14 10:01:57 TOWER root: Starting virtlogd...
Mar 14 10:01:57 TOWER root: Starting libvirtd...

 

Then for some reason it tried to start again a few minutes later:

 

Mar 14 10:18:28 TOWER emhttpd: shcmd (552): /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 2
Mar 14 10:18:28 TOWER root: /mnt/user/system/libvirt/libvirt.img is in-use, cannot mount
Mar 14 10:18:28 TOWER emhttpd: shcmd (552): exit status: 1

 

Try rebooting in safe mode and post new diags.

I rebooted the array in safe mode. I ran diagnostics prior to starting the array. That is file -1305.zip

 

I then started the array and started the libvirt service with VM Manager. This time it started and the VM's showed up in the dashboard. Ran diags again. That is file -1307.zip

 

I guess there is a plugin or something causing the problem. I hope this helps to identify the issue.

 

I really appreciate the help.

 

tower-diagnostics-20200314-1307.zip tower-diagnostics-20200314-1305.zip

I found the issue to be a user script that was creating symlinks for virt manager persistent snapshots. It was referencing disk locations that were no longer valid after I reintroduced the ssd back into the array. It was set to run at array startup, and the script restarts the libvirt service. However, it was not completing due to the incorrect disk locations.

Stopped the script and libvirt starts every time.

Moved the specified locations back to original re-ran the script and all is ok.

 

Sorry for the fire drill, and thanks for the assistance in identifying the root cause.

👍