sonofapharmacist

UNRAID_LIBVIRT_VMSERVICE_FIX.md My journey today to begin using VMs again on this old box. Long-story short: libvirt.img was corrupted and recreating the image through the existing scripts in the OS did not seem to work right and so had to directly rename libvirt.img to libvirt-xfs.img in the domain.cfg path. I included the markdown file of my troubleshooting steps and how to diagnose should you have the same issues (attached) and may as well make it scrapeable: # Unraid LibVirt VM Service Failure — Troubleshooting Guide ## Overview This document describes a specific, confirmed bug/misconfiguration that causes the **VM Service (LibVirt) to fail on startup** in Unraid 7.x. It was diagnosed and resolved on Unraid **7.2.4** (upgraded from **7.2.3**). The issue may have been introduced during or prior to the 7.2.3 → 7.2.4 upgrade cycle, with the earliest confirmed failure logged on March 18, 2026. Docker continued to work normally throughout. Only the VM Service was affected. --- ## Symptom - VM Service shows red/failed in the Unraid GUI after array start - Syslog contains: ``` emhttpd: shcmd: /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 1 root: mount: /etc/libvirt: wrong fs type, bad option, bad superblock on /dev/loop3 ``` or: ``` root: mount: /etc/libvirt: can't read superblock on /dev/loop3 ``` - Kernel log shows BTRFS being detected on loop3 with a consistent UUID — even after the array is restarted or the server is rebooted - rc.libvirt reports: libvirt daemon... No image mounted at /etc/libvirt --- ## Root Cause (Two-Part) ### Part 1 — The libvirt.img file was created as BTRFS The Unraid helper script /usr/local/sbin/mount_image determines the filesystem type to use based on the **filename suffix** of the image file: | Filename ends with | Filesystem used | |--------------------|----------------| | -xfs.img | XFS | | anything else | **BTRFS** | Because the default image is named libvirt.img (no -xfs suffix), mount_image always uses BTRFS — both when creating the file and when mounting it. At some point the libvirt.img on this system became corrupted (BTRFS superblock bad fsid errors), making it unmountable even as BTRFS. ### Part 2 — Replacing the file with ext4 does not work A natural troubleshooting step is to delete the corrupted image and create a fresh one using mkfs.ext4. This produces a valid ext4 image that file and od confirm as correct — but mount_image will still fail because it always calls: ```bash mount -o noatime,space_cache=v2 libvirt.img /etc/libvirt ``` These are BTRFS-specific mount options. Mounting an ext4 image with space_cache=v2 causes the kernel to reject it with "wrong fs type." The ext4 image is never the problem — the mount options are. ### Why the kernel always reports the old BTRFS UUID The kernel BTRFS scanner caches device UUIDs. Even after replacing the file, the kernel may report the old UUID in dmesg because the loop device is being presented with the same size (2097152 sectors = 1GiB). This is a red herring — the file on disk can be perfectly valid ext4 while the kernel log still references old BTRFS metadata from a prior mount attempt in the same session. --- ## Diagnosis Steps ### 1. Check the image file type ```bash file /mnt/cache/system/libvirt/libvirt.img ``` - If it reports BTRFS → original corrupted image still present - If it reports ext4 → replacement was made but won't work due to mount_image logic ### 2. Verify the magic bytes directly ```bash # BTRFS magic at offset 0x10040 — should be zeros if NOT btrfs od -A x -t x1z -j $((0x10040)) -N 8 /mnt/cache/system/libvirt/libvirt.img # ext4 magic at offset 0x438 — should be "53 ef" if ext4 od -A x -t x1z -j $((0x438)) -N 2 /mnt/cache/system/libvirt/libvirt.img ``` ### 3. Confirm which file is actually being mounted The system share is typically cache=only, meaning the real file is on the cache drive, not the array disks. Always check: ```bash file /mnt/cache/system/libvirt/libvirt.img ls -lah /mnt/cache/system/libvirt/ find /mnt/disk1 /mnt/disk2 /mnt/disk3 /mnt/disk4 /mnt/disk5 /mnt/disk6 /mnt/disk7 \ -name "libvirt.img" 2>/dev/null ``` ### 4. Confirm the mount_image script behavior ```bash cat /usr/local/sbin/mount_image ``` Look for the filename suffix logic — if libvirt.img doesn't end in -xfs.img, BTRFS mount options will always be used regardless of actual filesystem type. ### 5. Quick proof-of-concept test (array running, LibVirt stopped) ```bash mount -o noatime /mnt/cache/system/libvirt/libvirt.img /etc/libvirt && \ echo "SUCCESS" && \ /etc/rc.d/rc.libvirt start ``` If this succeeds, it confirms the image is fine and mount_image's BTRFS options are the sole cause of failure. --- ## The Fix ### Step 1 — Edit /boot/config/domain.cfg directly (array can be stopped) ```bash sed -i 's|IMAGE_FILE=.*|IMAGE_FILE="/mnt/user/system/libvirt/libvirt-xfs.img"|' \ /boot/config/domain.cfg # Verify grep IMAGE_FILE /boot/config/domain.cfg ``` ### Step 2 — Remove the old/replacement image ```bash # If array is running: rm /mnt/cache/system/libvirt/libvirt.img rm /mnt/cache/system/libvirt/libvirt.img.bak # if present ``` ### Step 3 — Start the array On next array start, mount_image will see that libvirt-xfs.img does not exist, and will **automatically create it** using mkfs.xfs. No manual image creation needed. ### Step 4 — Verify in GUI Settings → VM Manager → Libvirt storage location should now show: /mnt/user/system/libvirt/libvirt-xfs.img VM Service should go green. LibVirt config files virtlockd.conf, virtlogd.conf) will be regenerated automatically on first successful start — the "No such file" errors for these in the log are harmless and expected on a fresh image. --- ## What Is NOT Lost - **VM disk images** .img, .qcow2) stored in /mnt/user/domains/ are completely separate from libvirt.img and are unaffected - **Docker** is unaffected throughout — it uses its own docker.img - Only the LibVirt XML configuration files (VM definitions) live inside libvirt.img If VMs were previously defined, they will need to be re-created in the GUI and pointed at their existing disk images in /mnt/user/domains/ --- ## Key Files and Paths | Item | Path | |------|------| | LibVirt image (correct name) | /mnt/user/system/libvirt/libvirt-xfs.img | | LibVirt image (real location on cache-only share) | /mnt/cache/system/libvirt/libvirt-xfs.img | | VM disk images | /mnt/user/domains/ | | Unraid VM config | /boot/config/domain.cfg | | mount_image script | /usr/local/sbin/mount_image | | LibVirt startup script | /etc/rc.d/rc.libvirt | --- ## Version Context | Field | Value | |-------|-------| | Unraid version at time of fix | 7.2.4 | | Previous Unraid version | 7.2.3 | | Kernel | 6.12.54-Unraid | | LibVirt version | 11.7.0 | | Earliest confirmed failure in logs | March 18, 2026 | | Docker image filesystem | BTRFS (working correctly — uses different image) | | Cache pool filesystem | BTRFS | --- ## Summary for LLM Context If an LLM is helping debug this issue, the key facts to establish are: 1. What does file /mnt/cache/system/libvirt/libvirt.img report? 2. What does the kernel log say about loop3 when the mount fails? 3. What does cat /usr/local/sbin/mount_image show about filesystem selection logic? 4. What is the current value of IMAGE_FILE in /boot/config/domain.cfg? The fix is always: **rename the image to end in -xfs.img** so that mount_image uses XFS filesystem operations instead of BTRFS. Edit domain.cfg to match, then let the array start create the image fresh.

Hi goobaroo, et al. How can I contribute to this? I have been trying to build a docker compose for StaTech Industry using the auto curse command using itzg/minecraft-server image but am having a heck of a time getting around the excluded mods and gettingthe thing to load properly. I'm sure you've worked all this out over a year ago and would love to be able to at least chip in on this one if you could point me in the right direction. Thx!

UNMS is discovering things OK but my devices are timing out when I try to connect them. I think the inform is still set for 443. How are you guys getting your devices to connect?