tomjrob Posted March 14, 2020 Share Posted March 14, 2020 Hoping someone can help here so that I do not lose all of my VM machines. Used mover to empty cache ssd to check it with Crucial windows program. Looked like it was empty. Checked it in windows machine. All ok. Put it back into Unraid and used mover to add docker and kvm (libvirt files) back onto the cache. Restarted Docker & recovered all containers. But now libvirt service fails to start. Libvirt Log 2020-03-14 14:01:57.797+0000: 23130: info : libvirt version: 5.10.0 2020-03-14 14:01:57.797+0000: 23130: info : hostname: TOWER 2020-03-14 14:01:57.797+0000: 23130: error : qemuStateInitialize:837 : unable to set ownership of '/var/lib/libvirt/qemu/save' to 0:100: No such file or directory 2020-03-14 14:01:57.797+0000: 23130: error : virStateInitialize:668 : Initialization of QEMU state driver failed: unable to set ownership of '/var/lib/libvirt/qemu/save' to 0:100: No such file or directory 2020-03-14 14:01:57.797+0000: 23130: error : daemonRunStateInit:839 : Driver state initialization failed Diagnostics attached. tower-diagnostics-20200314-1020.zip Quote Link to comment
JorgeB Posted March 14, 2020 Share Posted March 14, 2020 Rebooting should fix it. Quote Link to comment
tomjrob Posted March 14, 2020 Author Share Posted March 14, 2020 Rebooted 2x. Does not fix the problem. I also noticed that the system gets hung trying to unmount disks whn rebooting. Have to force reboot, which obviously causes unclean shutdown. Any other ideas? Thanks in advance. Quote Link to comment
JorgeB Posted March 14, 2020 Share Posted March 14, 2020 On the diags posted libvirt started correctly at first array start: Mar 14 10:01:57 TOWER emhttpd: shcmd (463): /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 2 Mar 14 10:01:57 TOWER kernel: BTRFS: device fsid 473e1a30-4df5-4093-9659-11deafd49e06 devid 1 transid 1987 /dev/loop3 Mar 14 10:01:57 TOWER kernel: BTRFS info (device loop3): disk space caching is enabled Mar 14 10:01:57 TOWER kernel: BTRFS info (device loop3): has skinny extents Mar 14 10:01:57 TOWER root: Resize '/etc/libvirt' of 'max' Mar 14 10:01:57 TOWER kernel: BTRFS info (device loop3): new size for /dev/loop3 is 2147483648 Mar 14 10:01:57 TOWER emhttpd: shcmd (465): /etc/rc.d/rc.libvirt start Mar 14 10:01:57 TOWER root: Starting virtlockd... Mar 14 10:01:57 TOWER root: Starting virtlogd... Mar 14 10:01:57 TOWER root: Starting libvirtd... Then for some reason it tried to start again a few minutes later: Mar 14 10:18:28 TOWER emhttpd: shcmd (552): /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 2 Mar 14 10:18:28 TOWER root: /mnt/user/system/libvirt/libvirt.img is in-use, cannot mount Mar 14 10:18:28 TOWER emhttpd: shcmd (552): exit status: 1 Try rebooting in safe mode and post new diags. Quote Link to comment
tomjrob Posted March 14, 2020 Author Share Posted March 14, 2020 I rebooted the array in safe mode. I ran diagnostics prior to starting the array. That is file -1305.zip I then started the array and started the libvirt service with VM Manager. This time it started and the VM's showed up in the dashboard. Ran diags again. That is file -1307.zip I guess there is a plugin or something causing the problem. I hope this helps to identify the issue. I really appreciate the help. tower-diagnostics-20200314-1307.zip tower-diagnostics-20200314-1305.zip Quote Link to comment
tomjrob Posted March 15, 2020 Author Share Posted March 15, 2020 I found the issue to be a user script that was creating symlinks for virt manager persistent snapshots. It was referencing disk locations that were no longer valid after I reintroduced the ssd back into the array. It was set to run at array startup, and the script restarts the libvirt service. However, it was not completing due to the incorrect disk locations. Stopped the script and libvirt starts every time. Moved the specified locations back to original re-ran the script and all is ok. Sorry for the fire drill, and thanks for the assistance in identifying the root cause. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.