thatonething

I used ASSSD which is similar to CrystalDiskMark but gives timing info as well. I also see reduced IOPS with CrystalDiskMark. I reinstalled Windows which gave me an improvement, so there was something in the OS which wasn't happy. I get about 40MB/s 4k read, and 70MB/s 4k write. Which still not raw performance but better. So FPS has been happier. I actually had better IOPS using a vdisk than using direct pass-through of NVME... I've been unable to quantify the configuration in relation to performance and it seems change significantly between unraid/kvm/cemu release. So I'll probably move away from a gaming VM in future. Currently using: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='1'> <name>Windows 11</name> <uuid>e394db12-b178-f662-36d2-9b2f5e06b228</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/> </metadata> <memory unit='KiB'>33030144</memory> <currentMemory unit='KiB'>33030144</currentMemory> <memoryBacking> <nosharepages/> <source type='memfd'/> <access mode='shared'/> </memoryBacking> <vcpu placement='static'>14</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='12'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='13'/> <vcpupin vcpu='4' cpuset='4'/> <vcpupin vcpu='5' cpuset='14'/> <vcpupin vcpu='6' cpuset='5'/> <vcpupin vcpu='7' cpuset='15'/> <vcpupin vcpu='8' cpuset='6'/> <vcpupin vcpu='9' cpuset='16'/> <vcpupin vcpu='10' cpuset='7'/> <vcpupin vcpu='11' cpuset='17'/> <vcpupin vcpu='12' cpuset='8'/> <vcpupin vcpu='13' cpuset='18'/> <emulatorpin cpuset='9'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-7.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader> <nvram>/etc/libvirt/qemu/nvram/e394db12-b178-f662-36d2-9b2f5e06b228_VARS-pure-efi-tpm.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='7' threads='2'/> <cache mode='passthrough'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.240-1.iso' index='1'/> <backingStore/> <target dev='hdb' bus='sata'/> <readonly/> <alias name='sata0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xa'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xb'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xc'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xd'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xe'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0xf'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x10'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </controller> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <filesystem type='mount' accessmode='passthrough'> <driver type='virtiofs' queue='1024'/> <binary path='/usr/libexec/virtiofsd' xattr='on'> <cache mode='always'/> <sandbox mode='chroot'/> </binary> <source dir='/mnt/user/Documents'/> <target dir='Documents'/> <alias name='fs0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </filesystem> <interface type='bridge'> <mac address='52:54:00:48:1c:ee'/> <source bridge='virbr0'/> <target dev='vnet0'/> <model type='e1000'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-Windows 11/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <tpm model='tpm-tis'> <backend type='emulator' version='2.0' persistent_state='yes'/> <alias name='tpm0'/> </tpm> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </source> <boot order='1'/> <alias name='hostdev3'/> <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc539'/> <address bus='1' device='9'/> </source> <alias name='hostdev4'/> <address type='usb' bus='0' port='1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc53d'/> <address bus='1' device='10'/> </source> <alias name='hostdev5'/> <address type='usb' bus='0' port='2'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain>

Are you using a vdisk or passing ssd/nvme directly to the vm? If you're using a vdisk make sure to allocate an IOThread. See my comment here

See my comment here for some xml config options which may help. https://forums.unraid.net/topic/139810-windows-vm-stuttering-on-upgrade-to-unraid-6115/?do=findComment&comment=1371536

Hey, I've been experiencing similar issues. I've spent days troubleshooting and testing. See my thread https://forums.unraid.net/topic/153038-poor-random-readwrite-nvme-passthrough-performance/ Here are a few things to try, some depend on your hardware, sharing what CPU you are running can help. 1. Try locked memory, this tells Unraid not to swap out the blocks of memory of other uses <memoryBacking> <locked/> <nosharepages/> </memoryBacking> 2. Isolate CPU cores from Unraid `Settings` -> `CPU Pinning` -> `CPU `Isolation`, pin all for best performance or some. This means Unraid will never use these cores. So up to you 3. Emulator Pin, puts kvm activities onto a dedicated thread outside of the VM. Better for latency, for total performance use an isolated CPU core <cputune> ... <emulatorpin cpuset='1'/> </cputune> 4. IOThread, apply a specific thread for all IO tasks, this is only applicable if you use Vdisk, not PCIE or direct disk passthrough. <iothreads>1</iothreads> <cputune> ... <iothreadpin iothread='1' cpuset='1,11'/> </cputune> 5. Use the newer faster TSC for clock. Most modern CPUs should support TSC. It is generally faster than HPET. I think you will need to be using i440fx-6.0 or higher for TSC. <clock offset='localtime'> <timer name='rtc' present='no' tickpolicy='catchup'/> <timer name='pit' present='no' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='no'/> <timer name='tsc' present='yes' mode='native'/> </clock> 6. I personally have found benefits with migration=off. Migration is useful for kvm in the enterprise space, so give it a try off <cpu mode='host-passthrough' check='none' migratable='off'> 7. You're using `pc-i440fx-3.1` try using the latest version `pc-i440fx-7.2` I don't see your GPU passed in the xml, if you do, ensure with using i440fx that you enable multifunction=on and map the addresses correctly

I've tested some more configs relating to NVMe performance which significantly impacts gaming. A - Emulator Pin + IOThread B - CPU migratable = off C - HyperV off clock tsc D - memory <locked/> xmls A - Emulator Pin + IOThread <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Windows 11 - default with io thread</name> <uuid>30e40a2a-558f-9113-b3ad-7030e96d9208</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/> </metadata> <memory unit='KiB'>33554432</memory> <currentMemory unit='KiB'>33554432</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>16</vcpu> <iothreads>1</iothreads> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='12'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='13'/> <vcpupin vcpu='4' cpuset='4'/> <vcpupin vcpu='5' cpuset='14'/> <vcpupin vcpu='6' cpuset='5'/> <vcpupin vcpu='7' cpuset='15'/> <vcpupin vcpu='8' cpuset='6'/> <vcpupin vcpu='9' cpuset='16'/> <vcpupin vcpu='10' cpuset='7'/> <vcpupin vcpu='11' cpuset='17'/> <vcpupin vcpu='12' cpuset='8'/> <vcpupin vcpu='13' cpuset='18'/> <vcpupin vcpu='14' cpuset='9'/> <vcpupin vcpu='15' cpuset='19'/> <emulatorpin cpuset='1,11'/> <iothreadpin iothread='1' cpuset='1,11'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-7.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader> <nvram>/etc/libvirt/qemu/nvram/30e40a2a-558f-9113-b3ad-7030e96d9208_VARS-pure-efi-tpm.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='8' threads='2'/> <cache mode='passthrough'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> B - CPU migratable = off <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Windows 11 - default with io thread</name> <uuid>30e40a2a-558f-9113-b3ad-7030e96d9208</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/> </metadata> <memory unit='KiB'>33554432</memory> <currentMemory unit='KiB'>33554432</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>16</vcpu> <iothreads>1</iothreads> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='12'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='13'/> <vcpupin vcpu='4' cpuset='4'/> <vcpupin vcpu='5' cpuset='14'/> <vcpupin vcpu='6' cpuset='5'/> <vcpupin vcpu='7' cpuset='15'/> <vcpupin vcpu='8' cpuset='6'/> <vcpupin vcpu='9' cpuset='16'/> <vcpupin vcpu='10' cpuset='7'/> <vcpupin vcpu='11' cpuset='17'/> <vcpupin vcpu='12' cpuset='8'/> <vcpupin vcpu='13' cpuset='18'/> <vcpupin vcpu='14' cpuset='9'/> <vcpupin vcpu='15' cpuset='19'/> <emulatorpin cpuset='1,11'/> <iothreadpin iothread='1' cpuset='1,11'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-7.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader> <nvram>/etc/libvirt/qemu/nvram/30e40a2a-558f-9113-b3ad-7030e96d9208_VARS-pure-efi-tpm.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='off'> <topology sockets='1' dies='1' cores='8' threads='2'/> <cache mode='passthrough'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> C - HyperV off clock tsc <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Windows 11 - default with io thread</name> <uuid>30e40a2a-558f-9113-b3ad-7030e96d9208</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/> </metadata> <memory unit='KiB'>33554432</memory> <currentMemory unit='KiB'>33554432</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>16</vcpu> <iothreads>1</iothreads> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='12'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='13'/> <vcpupin vcpu='4' cpuset='4'/> <vcpupin vcpu='5' cpuset='14'/> <vcpupin vcpu='6' cpuset='5'/> <vcpupin vcpu='7' cpuset='15'/> <vcpupin vcpu='8' cpuset='6'/> <vcpupin vcpu='9' cpuset='16'/> <vcpupin vcpu='10' cpuset='7'/> <vcpupin vcpu='11' cpuset='17'/> <vcpupin vcpu='12' cpuset='8'/> <vcpupin vcpu='13' cpuset='18'/> <vcpupin vcpu='14' cpuset='9'/> <vcpupin vcpu='15' cpuset='19'/> <emulatorpin cpuset='1,11'/> <iothreadpin iothread='1' cpuset='1,11'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-7.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader> <nvram>/etc/libvirt/qemu/nvram/30e40a2a-558f-9113-b3ad-7030e96d9208_VARS-pure-efi-tpm.fd</nvram> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none' migratable='off'> <topology sockets='1' dies='1' cores='8' threads='2'/> <cache mode='passthrough'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='no'/> <timer name='hpet' present='no'/> <timer name='tsc' present='yes' mode='native'/> </clock> D - memory <locked/> <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Windows 11 - default with io thread</name> <uuid>30e40a2a-558f-9113-b3ad-7030e96d9208</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/> </metadata> <memory unit='KiB'>33554432</memory> <currentMemory unit='KiB'>33554432</currentMemory> <memoryBacking> <locked/> <nosharepages/> </memoryBacking> <vcpu placement='static'>16</vcpu> <iothreads>1</iothreads> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='12'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='13'/> <vcpupin vcpu='4' cpuset='4'/> <vcpupin vcpu='5' cpuset='14'/> <vcpupin vcpu='6' cpuset='5'/> <vcpupin vcpu='7' cpuset='15'/> <vcpupin vcpu='8' cpuset='6'/> <vcpupin vcpu='9' cpuset='16'/> <vcpupin vcpu='10' cpuset='7'/> <vcpupin vcpu='11' cpuset='17'/> <vcpupin vcpu='12' cpuset='8'/> <vcpupin vcpu='13' cpuset='18'/> <vcpupin vcpu='14' cpuset='9'/> <vcpupin vcpu='15' cpuset='19'/> <emulatorpin cpuset='1,11'/> <iothreadpin iothread='1' cpuset='1,11'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-7.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader> <nvram>/etc/libvirt/qemu/nvram/30e40a2a-558f-9113-b3ad-7030e96d9208_VARS-pure-efi-tpm.fd</nvram> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none' migratable='off'> <topology sockets='1' dies='1' cores='8' threads='2'/> <cache mode='passthrough'></cache> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='no'/> <timer name='hpet' present='no'/> <timer name='tsc' present='yes' mode='native'/> </clock>

I've been attempting to debug significant performance issues on my Windows 11 VM recently. After testing many configs I found the NVMe performance to degrade significantly from bare metal. Specifically, the Random IO. It seems to be directly related to interrupts and clock configuration. But I've tried just about every config option with varying results. CPU: Intel i7-10900k NVMe: Samsung 970 Evo The drive is passed through with IMMOU binding directly to the VM via PCIe. All CPUs are isolated and there is a emulatorPin While sequential read/writes are within acceptable ranges. Random access and latency are very poor. I'm curious if anyone can recreate these results. I checked the NVMe devices are using MSI which they seem to be, https://forums.guru3d.com/threads/windows-line-based-vs-message-signaled-based-interrupts-msi-tool.378044/ This is the best performing configuration. You can omit the hyperV options with similar results. But may as well have those there. <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none' migratable='off'> <topology sockets='1' dies='1' cores='8' threads='2'/> <cache mode='passthrough'/> </cpu> <clock offset='localtime'> <timer name='rtc' present='no' tickpolicy='catchup'/> <timer name='pit' present='no' tickpolicy='delay'/> <timer name='hpet' present='no'/> <timer name='hypervclock' present='no'/> <timer name='tsc' present='yes' mode='native'/> </clock> ``` kernel /bzimage append acpi_enforce_resources=lax pcie_acs_override=downstream,multifunction video=efifb:off isolcpus=2-9,12-19 initrd=/bzroot ```

I recently upgraded my GPU and noticed significant stuttering and lower than expected performance. After trying multiple flags for the KVM settings I've been unable to get anywhere near bare-metal performance. So, in my efforts, I've spent some time testing. I'll be capturing the performance of more configuration options. If you have any ideas to try let me know. I'll write up and share the full analysis, configs and so on when I've finished.

Also seeing this issue in 6.12.2 There is this post which is related. I'm also seeing issues with networking, Started Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to create new parent process: namespace path: lstat /proc/14548/ns/net: no such file or directory: unknown