jch

I've been experimenting with LVM-backed VM storage as a middle ground between file-based images and direct device passthrough. I wanted better performance than img files but didn't want to sacrifice snapshots or lose an entire NVMe device to a single VM. The I/O performance improvement over regular img files is pretty substantial - my Windows VM feels much more responsive now. The interesting thing is that while it looks like more layers: VM → qemu → LVM → loop device → file → filesystem → storage It actually performs better than direct file access because you get proper block device I/O with better caching and alignment. Setup process: Create storage file: truncate -s 100G vdisk-lvm.img Mount via loop device: losetup /dev/loop4 vdisk-lvm.img Initialize LVM: pvcreate /dev/loop4 && vgcreate vm-storage /dev/loop4 Create logical volume: lvcreate -L 90G -n windows vm-storage Update VM XML to point to /dev/vm-storage/windows instead of the img file The bonus is you get LVM snapshots, which are really handy for things like Windows updates where you want an easy rollback option. I set up some User Scripts to handle the loop device mounting automatically when the array starts/stops. Benchmarks (all using NVME + Real World test, the disk image sits on a Samsung 990 Pro NVMe drive, encrypted XFS format): Using raw image file, "typical" setup: <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' io='native' discard='unmap' iothread='1'/> <source file='/mnt/primary/domains/Windows/vdisk1.img'/> Using LVM mounted <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native' discard='unmap' iothread='1'/> <source dev='/dev/vm-storage/windows' index='1'/> My question is: why isn't this approach discussed more often in VM guides or forums? Is there some downside I'm not seeing, or is it just the perceived complexity that keeps people using raw/qcow2 files? The performance improvement seems worth the slightly more involved setup. It's not native NVME speeds, but you retain snapshot flexibility.

While I appreciate all the functionality that this plugin provides, I wanted a very simple way of pausing any parity checks if mover was running. I noted that the `monitor` command for this plugin seems to... take a while to complete. Probably because it's doing a lot of checks? It also seems to touch a lot of files and I was looking for some simple functionality. To this end, I wrote a small script that others might find useful. I have this script run every few minutes using the CA User Scripts plugin. It only uses built-in system commands. #!/usr/bin/php #arrayStarted=true <?php // https://claude.ai/chat/41bf3384-61ac-44ab-8341-0898d78d1223 // Define constants $PIDFILE = "/var/run/mover.pid"; $LOCKFILE = "/var/run/parity-mover-monitor.lock"; $mdcmd = '/usr/local/sbin/mdcmd'; $parity_control = '/usr/local/emhttp/webGui/scripts/parity_control'; $logName = "parity-monitor"; // Function to get mdstat values function mdstat($key) { return exec("grep -Pom1 '^$key=\K.+' /proc/mdstat"); } // Log function using system logger function logMessage($string) { global $logName; $cmd = 'logger -t "' . $logName . '" "' . $string . '"'; shell_exec($cmd); } // Check if parity check is active $isParityActive = (mdstat('mdResyncPos') > 0 && substr(mdstat('mdResyncAction'), 0, 7) == 'check P'); $isParityRunning = ($isParityActive && mdstat('mdResync') > 0); $isParityPaused = ($isParityActive && mdstat('mdResync') == 0); $moverRunning = file_exists($PIDFILE); $lockExists = file_exists($LOCKFILE); // Main logic if ($moverRunning) { // If mover is running and parity is running, pause parity if ($isParityRunning) { logMessage("Mover running and parity check active - pausing parity check"); exec("$parity_control pause"); // Create lock file to indicate that this script paused the parity check file_put_contents($LOCKFILE, date('Y-m-d H:i:s')); logMessage("Created lock file to track parity pause"); } } else { // If mover is not running and parity is paused, check if we should resume if ($isParityPaused && $lockExists) { logMessage("Mover not running, parity check paused, and lock file exists - resuming parity check"); exec("$parity_control resume"); // Remove the lock file after resuming unlink($LOCKFILE); logMessage("Removed lock file after resuming parity"); } } // Cleanup: Only remove the lock file if we're back to a normal state // (mover not running AND parity check not paused) AND the lock file exists if (!$moverRunning && !$isParityPaused && $lockExists) { unlink($LOCKFILE); logMessage("Cleanup: Removed lock file because normal operation has resumed"); }

The package folders.

3rd time's the charm.

Is this it?

A @SimonF I have the exact same issue. Using an Supermicro X11SPi-TF , Version 1.02 motherboard. Here's my fan settings page (let me know if I can post any other diagnostics to help locate the issue):

There's some indication that for very high memory systems (i.e i have 256GB of memory), it would be better to set dirty_bytes / dirty_background_bytes instead (https://www.suse.com/support/kb/doc/?id=000017857). Since the % numbers are limited to integers, 1% / 2% still causes quite some freezing when the mover is running (my rough model is reads are queued behind 2.56GB of cache, so transferring at ~100MB / s means operations freeze for ~30s while mover is running). Could we add those two flags as an alternative way of setting dirty?

@EDACerton I think one of my "At Startup of Array" scripts was crashing out or misconfigured. It calls a custom `rc.service` script and inside that script i call `. /etc/rc.d/rc.library.source` but i was only including that because i copied other files in `rc.d`. i looked at that script and it looks like there's a lot of stuff in there so i opted to refactor my startup script to remove references to `rc.library.source`. haven't had an issue since i excluded it from my "At Startup of Array" scripts. the original script is here: https://gist.github.com/jasperchan/2016270e2b4f6dfab9ca409a68b4ae8e

thanks, that was helpful guidance. looks like the tailscale service wasn't started. i cleaned out some of my "on array start" scripts and now it's installing properly so it's clearly something to do with one of my scripts interfering with the tailscale install sequence. nothing to do on your end, thanks for your continued maintenance on this plugin!

The main dashboard and settings page is still not loading for me using the Preview version. This is from a clean install where I deleted any Tailscale references from the /boot/config directory and rebooted. Tower-tailscale-diag-20241020-175615.zip

The folder I've mounted is directly on the cache pool (i.e. by-passing FUSE via exclusive access or referencing /mnt/cache/<folder>/) and is on a pair of Samsung 870 EVO 2TB SSD's in an encrypted mirrored BTRFS configuration. My CrystalDiskMark results from the guest VM (this is with other concurrent writes going on -- was too lazy to do a real dedicated test) are below. It's fast enough for me and like @Shadowplay highlighted the stability is phenomenal now, I don't see any of the issues I previously had with virtiofsd.

They are all documented here: https://gitlab.com/virtio-fs/virtiofsd EDIT: Oops I misread your question. No I haven't found anything that clearly outlines the "best" setup. The document I linked has some guidelines (i.e. they were very very adamant that `--announce-submounts` is extremely important if you're passing through a mount). Given it's an open-source project, probably worth reaching out to the maintainers especially if they will become default arguments in unraid. So, I think this is a personal choice but what I've found is that if you enable caching then you have to be very careful about modifying the files in the host while the guest is running (at least when I was doing extensive testing a year ago). I have a project that I am running that concurrently accesses the files that the VM has access to but if your VM files aren't touched by any other process on the host than it's probably safe to re-enable cache. I believe `auto` in the rust virtiofsd implementation does some checking for changes but with older versions I found that I was seeing very strange file corruption, especially if mover moves the file. I disabled it because I couldn't get to the bottom of it and everything seems "fast enough" without it (I get near native writes anyway onto the nvme drive that I shared). If you want to re-enable cache, I'd suggest going with "auto" (the default for rust virtiofsd) and then do testing where you make sure there isn't any file corruption under the following circumstances: - actively write to a file on the guest, try to move file file on the host (to simulate mover) - open the file on guest (to hit the cache), then modify the file on host, then make sure those changes propagate to the guest - open the file on guest (to hit the cache), then rename the file on host and make sure those changes propagate to the guest There might be more test cases but those were the ones causing issues with older versions. Also thanks for expanding my steps to be more detailed :).

The arguments for the Rust version of virtiofsd diverges from the original version that Unraid has bundled so certain XML changes are not properly passed (i.e. cache). For absolute stability, I found the Rust version to be superior since it has various options for handling submounts and much better handling for host files changing via cache=never. For others looking for this functionality, what I did was using the latest version of the Rust virtiofsd (compiled version available at https://gitlab.com/virtio-fs/virtiofsd/-/releases) stored at /root/.virtiofsd/virtiofsd (though you can put it anywhere, just modify the script), and then combined it with this script (stored at /root/.virtiofsd/virtiofsd.sh) that passes the --fd parameter properly: #!/bin/bash # process -o option but ignore it because unraid generates the command for us VALID_ARGS=$(getopt -o o -l fd: -- "$@")i if [[ $? -ne 0 ]]; then exit 1; fi eval set -- "$VALID_ARGS" while [ : ]; do case "$1" in --fd ) FD="$2" shift 2 ;; -o ) shift 1 ;; -- ) shift; break ;; * ) shift; ;; esac done # https://gitlab.com/virtio-fs/virtiofsd /root/.virtio/virtiofsd \ --fd="$FD" \ --shared-dir="/mnt/<YOUR SHARE DIR HERE>" \ --xattr \ --cache="never" \ --sandbox="chroot" \ --inode-file-handles="mandatory" \ --announce-submounts The relevant excerpt from your XML config is (noting that this setup ignores most parameters and instead you should make argument changes directly in `virtiofsd.sh` above; through testing you do need to keep the `<target ... />` argument as that's the handle the windows driver will be looking for): <filesystem type='mount' accessmode='passthrough'> <driver type='virtiofs' queue='1024'/> <binary path='/root/.virtio/virtiofsd.sh' xattr='on'> ... </binary> ... <target dir='YOUR HANDLE HERE'/> <alias name='fs0'/> ... </filesystem> These changes resulted in effectively native file transfer speeds between VM and host (~200MB/s) and I've noted it seems to no longer have any memory issues or weird file locking issues. 100GB+ transfers between the VM and host systems daily (large and small files) so I feel good about the current setup (I also followed the instructions on upgrading to the latest v248 drivers in windows). Using this setup allows me to create a synthetic directory in the host that only contains the folders I want to pass through to the VM and consolidates them under a single drive in the VM. The --announce-submounts argument was critical in doing this in a stable way.

Is there a way to add additional stats to the footer of the WebGUI? The IPMI support plugin allows adding some fan/temp stats, but I'd like to add things like overall load, network traffic, etc.

Since libvirt.img contains all of the VM configs, is this ever updated (or does it need changing across unRAID versions)? I ask specifically because it looks like the qemu startup script file contains some vfio binding logic. I noticed with this recent upgrade to 6.12.4 that my changes to that script were retained. Aside, if anybody seeing this has an unmodified /etc/libvirt/hooks/qemu script, could you attach it here? I made changes to mine without backing it up, and I don't want to reset it because then I'd have to re-setup my VMs.