xandyedgex Posted April 6, 2019 Share Posted April 6, 2019 Hi All, Hopefully im just missing something small and y'all can help me out. I'll outline everything i have done to try to remedy this issue. The Issue at hand... i have a Dell R720, Dual Xeon E5-2680 CPUs, 64GB 1600MHZ Ram. Running UNRAID 6.7.0 rc7. I have a GTX1660 passed Through to a Windows 10 VM for gaming, The VM resides on a Samsung 960 NVME Drive, is assigned 14GB of ram, and 12 Cores the card is only giving me an average of 30-40FPS in the VM. If i boot off the NVME drive, and run bare metal it spits out 100+ at the same settings (Running UNIGINE Benchmark). So Far i have tried Q35 and 1440fx, with no change. 8-12 cpus, No Change. Seabios or OVMF, no change. I've also tried with and without the VBIOS Rom file. A word of note, i have tried multiple new installs of windows as well, and they all do this. I have all the processors assigned to NUMA node 0, The GTX is on NUMA node 0, and using NUMASTAT QEMU, all the ram is assigned from NUMA node 0. In the VM i have enabled High Performance Power mode, Disabled Hibernation, and turned off all unrelated services possible. I have also tried the card in the RISER 2 Slot, that's connected to CPU 2 with the Cores, and ram on NUMA 1 and same results. I have all the same Cores Isolated from UNRAID as well (except the emulator pin) Below is my XML for the machine, Maybe i'm missing something? <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Win</name> <uuid>278722d8-5ca6-1a44-a0d9-5fb8f8de32be</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>14680064</memory> <currentMemory unit='KiB'>14680064</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>12</vcpu> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='20'/> <vcpupin vcpu='2' cpuset='6'/> <vcpupin vcpu='3' cpuset='22'/> <vcpupin vcpu='4' cpuset='8'/> <vcpupin vcpu='5' cpuset='24'/> <vcpupin vcpu='6' cpuset='10'/> <vcpupin vcpu='7' cpuset='26'/> <vcpupin vcpu='8' cpuset='12'/> <vcpupin vcpu='9' cpuset='28'/> <vcpupin vcpu='10' cpuset='14'/> <vcpupin vcpu='11' cpuset='30'/> <emulatorpin cpuset='0,16'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-3.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/278722d8-5ca6-1a44-a0d9-5fb8f8de32be_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='6' threads='2'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/disks/VMs/Win/vdisk1.img'/> <target dev='hdc' bus='scsi'/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/template/iso/Win10.X64.en-US.ISO'/> <target dev='hda' bus='sata'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/> <target dev='hdb' bus='sata'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </controller> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:e9:85:73'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> </domain> Quote Link to comment
jonp Posted April 6, 2019 Share Posted April 6, 2019 I think the issue here is that you are using a dual socket system. If you're going to do that, you need to make sure that the logical CPUs assigned to each guest VM are aligned with the PCIe devices you are assigning to them. You can use lstopo to determine this from the command line, though I don't think we have a detailed guide just yet. Something we will work up in the future I'm sure. That said, if you do some basic Internet searching on lstopo, vfio, and dual socket CPUs, you should find the guides you need to better tune your virtual machine(s). Quote Link to comment
xandyedgex Posted April 6, 2019 Author Share Posted April 6, 2019 Thank you for the quick reply, I'll look into this deep and keep updated. And of course SI1 has a video of it, so i guess thatll be my next step! Link posted for anyone else who may need it. Quote Link to comment
bastl Posted April 7, 2019 Share Posted April 7, 2019 (edited) 14 hours ago, xandyedgex said: <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='20'/> <vcpupin vcpu='2' cpuset='6'/> <vcpupin vcpu='3' cpuset='22'/> <vcpupin vcpu='4' cpuset='8'/> <vcpupin vcpu='5' cpuset='24'/> <vcpupin vcpu='6' cpuset='10'/> <vcpupin vcpu='7' cpuset='26'/> <vcpupin vcpu='8' cpuset='12'/> <vcpupin vcpu='9' cpuset='28'/> <vcpupin vcpu='10' cpuset='14'/> <vcpupin vcpu='11' cpuset='30'/> <emulatorpin cpuset='0,16'/> </cputune> Are you sure, you're not assigning cores from both CPUs with this settings? Can you post a screen of the settings of this VM where we can see the core pairings? You have 2 CPUs each with 8 cores 16 threads. If cores 0-7 are from the first CPU you are mixing them with the second CPU if that one starts counting with 8 up to 16. You have choosen the core 4 thats in the lower half of all cores and 30 thats close to the last core. For me this looks wrong. Emulatorpin is set to 0,16. If that is the first physical/logical core I end up with the following scheme CPU1: 0 16 1 17 2 18 3 19 4 20 5 21 6 22 7 23 CPU2: 8 24 9 25 10 26 11 27 12 28 13 29 14 30 15 31 EDIT: Post a screen of the output of the following command please numactl --hardware EDIT2: Example of my Threadripper CPU. This is basically also a dual CPU. The topology Unraid shows you in the UI might be different to mine, just to get you an idea. The marked cores in my config are all cores from the second die/CPU used by one VM. Edited April 7, 2019 by bastl Quote Link to comment
xandyedgex Posted April 10, 2019 Author Share Posted April 10, 2019 Sorry it took a minute to get back to this, Here is a screenshot of my CPU pinning, and the output of numactl --hardware and lspci -v -s for the GTX, It looks like im pinned to all Node 0 for my cores and the card is also on node 0, i have also tried exactly the same setup but with the card in the riser 2 slot which puts it on Node 1 and then pinning cores to Node 1 as well, effecively putting the emulator pins on the first Node seperately. Currently as you see this the same CPU cores are also isolated from UNRAID. Quote Link to comment
bastl Posted April 10, 2019 Share Posted April 10, 2019 Ok, so your CPU pinning looks fine and your GPU is on the same node as the cores you use for that VM. Perfect. What I noticed from your numactl output, you're using a lot of RAM and only have 1.3GB free in total. In general Unraid doesn't care where the RAM is connected to, to which node. This could also cause your issue. Try the following to strictly use the RAM from a specific node. Ad the <numatune> part from the example below to your XML underneath the </cputune> element. This should prevent Unraid using RAM from the second node. Another thing you can try is to set a specific iothread and pin it to the same node as the VM, also in the example below. <vcpu placement='static'>14</vcpu> <iothreads>1</iothreads> <cputune> <vcpupin vcpu='0' cpuset='9'/> <vcpupin vcpu='1' cpuset='25'/> <vcpupin vcpu='2' cpuset='10'/> <vcpupin vcpu='3' cpuset='26'/> <vcpupin vcpu='4' cpuset='11'/> <vcpupin vcpu='5' cpuset='27'/> <vcpupin vcpu='6' cpuset='12'/> <vcpupin vcpu='7' cpuset='28'/> <vcpupin vcpu='8' cpuset='13'/> <vcpupin vcpu='9' cpuset='29'/> <vcpupin vcpu='10' cpuset='14'/> <vcpupin vcpu='11' cpuset='30'/> <vcpupin vcpu='12' cpuset='15'/> <vcpupin vcpu='13' cpuset='31'/> <emulatorpin cpuset='8,24'/> <iothreadpin iothread='1' cpuset='8,24'/> </cputune> <numatune> <memory mode='strict' nodeset='0'/> </numatune> With "numastat qemu" you can check how the current RAM usage of all running VMs is looking. For some reasons the "strict" settings not always works and some users already reported that even with this setting Unraid tends to grab a couple megs from the other node as well. I myself couldn't figure out what causing this or how to prevent that. Maybe @limetech has an idea or can provide us a solution to prevent Unraid doing that. Quote Link to comment
xandyedgex Posted April 16, 2019 Author Share Posted April 16, 2019 Still having this issue, I have moved everything to Node 1, Since thats where the NVME drive also resides. Isolated the 8 Cores from Unraid, Pinned them to the vm, Added the Numatune command for node 1, Added the iothread pin etc. Seems that the strict setting on numatune still shows 14 MB frome node 0. Im getting 60 or so FPS so i guess i'll live with it for now. Here is current xml just in case i'm missing something. <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='4'> <name>Windows 10</name> <uuid>eff3db79-a5f2-88cf-4027-8ccd0578d107</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <iothreads>1</iothreads> <cputune> <vcpupin vcpu='0' cpuset='9'/> <vcpupin vcpu='1' cpuset='25'/> <vcpupin vcpu='2' cpuset='11'/> <vcpupin vcpu='3' cpuset='27'/> <vcpupin vcpu='4' cpuset='13'/> <vcpupin vcpu='5' cpuset='29'/> <vcpupin vcpu='6' cpuset='15'/> <vcpupin vcpu='7' cpuset='31'/> <emulatorpin cpuset='7,23'/> <iothreadpin iothread='1' cpuset='7,23'/> </cputune> <numatune> <memory mode='strict' nodeset='1'/> </numatune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-3.1'>hvm</type> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='4' threads='2'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/disks/VMs/domains/Windows 10/vdisk1.img'/> <backingStore/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/> <backingStore/> <target dev='hdb' bus='sata'/> <readonly/> <alias name='sata0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x14'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x8'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x9'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:e2:29:83'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-4-Windows 10/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x42' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/mnt/user/Media/MSI.GTX1660Ti.6144.190313.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x42' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x42' slot='0x00' function='0x2'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x42' slot='0x00' function='0x3'/> </source> <alias name='hostdev3'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain> Quote Link to comment
dojesus Posted April 4, 2020 Share Posted April 4, 2020 I'm having the same issue, though I'm on a single CPU i7 3770. I think the issue is the 1660ti, as I've tested with a Radeon 480 and 580 and have zero issues. Came here to see if anyone else had a bios passthrough that would help. Quote Link to comment
xandyedgex Posted April 4, 2020 Author Share Posted April 4, 2020 This is the time I used for my setup, lemme know if it helps you out. MSI.GTX1660Ti.6144.190313.rom Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.