un1ty

I had the same problem, and it wasn't caused by ZFS master, docker or VMs. But it turned out, that I had a scrub process running on the ZFS drive that is taking forever, which had never been a problem for XFS or BTRFS. It wasn't just that it was constantly reading/writing, it was also slowing down all other processes such as parity checks.

I have the same problem, however I see many warnings that ZFS is not available on the newer kernels.. So I'm afraid I'm stuck at the default 6.1.74 kernel with Unraid 6.12.8. Or do you have other experiences?

Also to update from my original post at Original performance (cpu mode passthrough): Same configuration (cpu mode QEMU64): Current configuration (cpu mode passthrough, tuned) is attached. sequential is much better, but random IO barely doubled, where as it should have been closer to 6x or above to approach baremetal.

Here are the diagnostics un1ty-diagnostics-20240226-2250.zip

Okay, I will add those in a bit. I have changed some things in my configuration since the start of this post to experiment with other settings: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='3'> <name>Windows 11</name> <uuid>f40f9d6c-7709-b2e2-fc68-1660a0537782</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/> </metadata> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>24</vcpu> <iothreads>4</iothreads> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='20'/> <vcpupin vcpu='2' cpuset='5'/> <vcpupin vcpu='3' cpuset='21'/> <vcpupin vcpu='4' cpuset='6'/> <vcpupin vcpu='5' cpuset='22'/> <vcpupin vcpu='6' cpuset='7'/> <vcpupin vcpu='7' cpuset='23'/> <vcpupin vcpu='8' cpuset='8'/> <vcpupin vcpu='9' cpuset='24'/> <vcpupin vcpu='10' cpuset='9'/> <vcpupin vcpu='11' cpuset='25'/> <vcpupin vcpu='12' cpuset='10'/> <vcpupin vcpu='13' cpuset='26'/> <vcpupin vcpu='14' cpuset='11'/> <vcpupin vcpu='15' cpuset='27'/> <vcpupin vcpu='16' cpuset='12'/> <vcpupin vcpu='17' cpuset='28'/> <vcpupin vcpu='18' cpuset='13'/> <vcpupin vcpu='19' cpuset='29'/> <vcpupin vcpu='20' cpuset='14'/> <vcpupin vcpu='21' cpuset='30'/> <vcpupin vcpu='22' cpuset='15'/> <vcpupin vcpu='23' cpuset='31'/> <emulatorpin cpuset='2-3,18-19'/> <iothreadpin iothread='1' cpuset='2,18'/> <iothreadpin iothread='2' cpuset='3,19'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-7.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader> <nvram>/etc/libvirt/qemu/nvram/f40f9d6c-7709-b2e2-fc68-1660a0537782_VARS-pure-efi-tpm.fd</nvram> <smbios mode='host'/> </os> <features> <acpi/> <apic/> <kvm> <hidden state='on'/> </kvm> <ioapic driver='kvm'/> </features> <cpu mode='host-passthrough' check='none' migratable='off'> <topology sockets='1' dies='1' cores='12' threads='2'/> <cache mode='passthrough'/> <feature policy='disable' name='hypervisor'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='no'/> <timer name='tsc' present='yes' mode='native'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='writeback'/> <source file='/mnt/user/domains/Windows 11/vdisk1.snap0' index='1'/> <backingStore type='file' index='2'> <format type='qcow2'/> <source file='/mnt/user/domains/Windows 11/vdisk1.img'/> <backingStore/> </backingStore> <target dev='hdc' bus='virtio'/> <serial>vdisk1</serial> <boot order='1'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </disk> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xd'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xe'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xf'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0x10'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x11'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='11' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='11' port='0x12'/> <alias name='pci.11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:bf:12:d0'/> <source bridge='br0'/> <target dev='vnet2'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/1'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/1'> <source path='/dev/pts/1'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-3-Windows 11/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <tpm model='tpm-tis'> <backend type='emulator' version='2.0' persistent_state='yes'/> <alias name='tpm0'/> </tpm> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x10' slot='0x00' function='0x0'/> </source> <alias name='hostdev3'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x12' slot='0x00' function='0x0'/> </source> <alias name='hostdev4'/> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x15' slot='0x00' function='0x2'/> </source> <alias name='hostdev5'/> <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x15' slot='0x00' function='0x3'/> </source> <alias name='hostdev6'/> <address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x15' slot='0x00' function='0x4'/> </source> <alias name='hostdev7'/> <address type='pci' domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source startupPolicy='optional' missing='yes'> <vendor id='0x0e8d'/> <product id='0x0616'/> </source> <alias name='hostdev8'/> <address type='usb' bus='0' port='1'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain> With the hope that I can get cpu mode passthrough to get as good random IO performance as i did with emulated (QEMU64). Moreover so to make Windows no longer think it's running inside a VM. I was also running into some bluescreens when running passmark's disk tests with the configuration I posted in the OP, though unraid itself remained stable without a hickup. I've also set the PCIe ACS override to both, this resolved another issue I had where I wasn't able to see my AMD RZ616 WiFi/Bluetooth card in the list of system devices, and therefore couldn't parse it to the VM. Now it is visible and works like a charm. So I no longer have to use some dodgy USB bluetooth adapter.

You might also want to benchmark with the changes I tried here: SLOW RANDOM IO PERFORMANCE ON NVME POOL I also sadly had to struggle my way to get better random IO performance, though I am not passing through my PCI-e device, since I wanted to group my SSD's into a raid or a storage pool, but AMD Raid doesn't function properly in a VM. The best uplift in performance I saw was by moving from: cpu mode='host-passthrough' to emulated.

I never noticed the snapshot, my best bet is that it was added by the easy backup plugin. Also what kind of diagnostics would you require? from unraid itself, benchmarks or something else? And yes, I have seen that thread before, but only had marginal changes to performance.

In plain text here is my configuration which I will refer to as the default: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='5'> <name>Windows 11</name> <uuid>f40f9d6c-7709-b2e2-fc68-1660a0537782</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/> </metadata> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>24</vcpu> <iothreads>2</iothreads> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='20'/> <vcpupin vcpu='2' cpuset='5'/> <vcpupin vcpu='3' cpuset='21'/> <vcpupin vcpu='4' cpuset='6'/> <vcpupin vcpu='5' cpuset='22'/> <vcpupin vcpu='6' cpuset='7'/> <vcpupin vcpu='7' cpuset='23'/> <vcpupin vcpu='8' cpuset='8'/> <vcpupin vcpu='9' cpuset='24'/> <vcpupin vcpu='10' cpuset='9'/> <vcpupin vcpu='11' cpuset='25'/> <vcpupin vcpu='12' cpuset='10'/> <vcpupin vcpu='13' cpuset='26'/> <vcpupin vcpu='14' cpuset='11'/> <vcpupin vcpu='15' cpuset='27'/> <vcpupin vcpu='16' cpuset='12'/> <vcpupin vcpu='17' cpuset='28'/> <vcpupin vcpu='18' cpuset='13'/> <vcpupin vcpu='19' cpuset='29'/> <vcpupin vcpu='20' cpuset='14'/> <vcpupin vcpu='21' cpuset='30'/> <vcpupin vcpu='22' cpuset='15'/> <vcpupin vcpu='23' cpuset='31'/> <emulatorpin cpuset='6-7,22-23'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-7.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader> <nvram>/etc/libvirt/qemu/nvram/f40f9d6c-7709-b2e2-fc68-1660a0537782_VARS-pure-efi-tpm.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>EPYC</model> <topology sockets='1' dies='1' cores='24' threads='1'/> <feature policy='disable' name='monitor'/> <feature policy='require' name='x2apic'/> <feature policy='require' name='hypervisor'/> <feature policy='disable' name='svm'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='writeback'/> <source file='/mnt/user/domains/Windows 11/vdisk1.snap0' index='1'/> <backingStore type='file' index='2'> <format type='qcow2'/> <source file='/mnt/user/domains/Windows 11/vdisk1.img'/> <backingStore/> </backingStore> <target dev='hdc' bus='virtio'/> <serial>vdisk1</serial> <boot order='1'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </disk> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xd'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xe'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xf'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0x10'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x11'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='11' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='11' port='0x12'/> <alias name='pci.11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:bf:12:d0'/> <source bridge='br0'/> <target dev='vnet4'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-5-Windows 11/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <tpm model='tpm-tis'> <backend type='emulator' version='2.0' persistent_state='yes'/> <alias name='tpm0'/> </tpm> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x10' slot='0x00' function='0x0'/> </source> <alias name='hostdev3'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x12' slot='0x00' function='0x0'/> </source> <alias name='hostdev4'/> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x15' slot='0x00' function='0x2'/> </source> <alias name='hostdev5'/> <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x15' slot='0x00' function='0x3'/> </source> <alias name='hostdev6'/> <address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x15' slot='0x00' function='0x4'/> </source> <alias name='hostdev7'/> <address type='pci' domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source startupPolicy='optional' missing='yes'> <vendor id='0x0e8d'/> <product id='0x0616'/> </source> <alias name='hostdev8'/> <address type='usb' bus='0' port='1'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain>

Hi all, I've made a post previously about the lackluster I/O performance in my Windows 11 VM (SLOW RANDOM IO PERFORMANCE ON NVME POOL), that I've somewhat managed to resolve myself. However, it seems that that forum category is almost, if not completely, dead. I want to further optimize my Windows 11 VM experience to be almost similar to baremetal. I use my system in several different ways; - Windows workstation (main VM) - Windows gaming station (secondary VM, only online when main VM is offline) - Home assistant VM (24/7, runs on 1c/2t) - Dockers for managing libraries (24/7, runs on 2c/4t) - Backups of VM's to spinning metal (scheduled either through unraid plugins or windows native applications). Here's my hardware configuration: CPU: Ryzen 9 7950X3D RAM: 2*16GB DDR5 6000MT/s CL30-38-38-96 Storage: 3x Samsung 970 evo plus (BTRFS Raid 0 pool) for the storage of the Windows VM's virtual disk. 1x Samsung 980 pro for cache to the data array, storage of dockers and home assistant VM. GPU: Gigabyte GeForce RTX 4060 Gaming OC 8G Motherboard: Gigabyte AORUS X670E MASTER I mainly work on my workstation VM, which I work on by passing through the GPU and some USB devices. It's been mostly fine, but I think there are some configurations improvements that can be made that I'm not aware of. I work as a embedded software developer so there are 2 key aspects that I would like to fine tune; CPU performance and IO performance. Since the tooling that I'm using is heavily random IO dependant. There are also some gaming aspects that I'd like to address, since my Windows VM says it's running in a virtual machine. But that is of secondary importance, since the games I play don't involve anti-cheat software. Things that (I think) can improve: CPU performance. random read/write IOPS. BIOS/UEFI information in the OS (fan/temp/whatever). False-positive anti-cheat detection Rationale: I don't think the 3D vcache is properly being utilized, and even L1/L2 cache isn't properly reported to the guest VM to take use off. I've tried a couple of different QEMU setting regarding CPU mode. I've stopped at the current setting emulating a AMD EPYC processor. Which is as similar as it is different from the actual processor. And performance wise hasn't been too negative. There seems to be a translation bottleneck between how Windows and Linux write to the file system, which requires CPU. I'm willing to reformat/rebuild the pool in whatever configuration. Attached is the default configuration, which I will allow to benchmark overnight using passmark (will attach results later). default.xml

In my old NAS I'm still running 2 WD Red pro 4TB drives in raid 0 since late 2014. They have a online time of over 9 years without any failure, issue or sign of slowing down.

The ones I was using were 12TB HDD's (9 internal disks) from toshiba, they were making a much louder normal read/write noise (clicking/clacking) from the head assembly, compared to my old 'shucked' WD red plus which I intended to replace, and those were also producing chirping sounds as if they were spinning down when they shouldn't while not performing any better. Which resulted into complete failure, at first it was only the first drive, but eventually the second drive failed as well. TL:DR; They were louder than need be, they (completely) failed without warning, they wouldn't even show up in bios eventually. Here is the unformatted report of the failure on drive 1:

Toshiba MG07 series, one of the only non-SMR models available at 12TB (product page) for data centre/nas usage, and typically known for it's reliability. I think the post order service had been rough handling the package, even though the drives are known for being more audible, to me it felt they were too audible to make sense even though they made 'normal' sounds. This is somewhat true now, however I did test with direct passthrough and it didn't give any performance benefits, I wanted a single 'storage space' to be passed through to the VM for higher performance across all 3 drives and underlying 2 partitions that I was going to use. However I did a test with a passed through single drive and got similar results; sequential was somewhat equal, but random was aberrant. However, I think I have found the solution! and this should be probably be a template change for the Windows 11 VM (with newer CPU's?). Before I used the default CPU 'passthrough' mode, but when I ran the same test again with CPU 'emulated (QEMU64)' mode this was the result: This change is massive, and noticable even from just opening windows explorer and browsing through directories even though the random IO is still sub-par.

I've also tried to compare between 1440fx and Q35.. Haven't gone through extensive testing yet, since sadly 3/4 of my brand new harddrives failed, with luckily all of the VM's and dockers stored on SSD's and private photos/videos backup to a NAS and cloud. But also this seems to provide a minimum performance improvement of around 10%.

I have to retract this statement, the load wasn't visible due to the setting of the timescale of the graph. I do see a 100% loads on the number of threads specified in the benchmarks, so it could very well be the bottleneck. I have performed tests in several different CPU configurations; first or second CCD only, core isolation on/off, more cores towards VM or unraid etc. And the performance change of this was within 10% (so similar changes were visible from run to run). So this doesn't seem to be the solution. I have changed from btrfs to zfs and did see a bump up in sequential speeds, but practically none in the random speeds. Moreover, even when I parsed through the drives as a raid directly to the VM, I saw a negligible raise in performance.

Ps. I'm aware I should format my 1 disk NVME cache to xfs as well, but that shouldn't impact the random IO performance of the "windows" share. What I'm really asking is should I use a different file system for the windows share, or should I directly parse the disks to the OS as a raid 0 to gain the expected benefits?