trivial Posted March 28, 2016 Share Posted March 28, 2016 Hi everybody! after following the guid on this threat http://lime-technology.com/forum/index.php?topic=43644.msg452464#msg452464 i managed to passthrough my MSI 950 GTX to my Windows 10 guest. After running some benchmarks it seems that my GPU is not even at 50% of its nativ performance on bare metal. Cine bench (OpenGL part): nativ: 90-95 FPS VM: 33-40 FPS I looked a bit closer and found something suspect in the nvidia systeminformation. The GPU seems to be passed through as PCIe x1 instead of x16 (see screenshot) which would be a perfect explanation of my performance issue. Mainboard and CPU can be found in the unRAID system info below. These are my current vm settings: <domain type='kvm' id='9' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Windows 10 sea bios</name> <uuid>40c557f6-29e0-1f45-6d1a-78de2dd6efbf</uuid> <metadata> <vmtemplate name="Custom" icon="windows.png" os="windows"/> </metadata> <memory unit='KiB'>12582912</memory> <currentMemory unit='KiB'>12582912</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> <vcpupin vcpu='2' cpuset='4'/> <vcpupin vcpu='3' cpuset='5'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='4' threads='1'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/cache/OS/Windows 10 sea bios/vdisk1.img'/> <backingStore/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/ISO/Windows10_64Pro.iso'/> <backingStore/> <target dev='hda' bus='ide'/> <readonly/> <boot order='2'/> <alias name='ide0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/ISO/virtio-win.iso'/> <backingStore/> <target dev='hdb' bus='ide'/> <readonly/> <alias name='ide0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:57:e8:8d'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/Windows 10 sea bios.org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc227'/> <address bus='7' device='4'/> </source> <alias name='hostdev0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x046d'/> <product id='0xc226'/> <address bus='7' device='3'/> </source> <alias name='hostdev1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x1532'/> <product id='0x0002'/> <address bus='5' device='2'/> </source> <alias name='hostdev2'/> </hostdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/boot/vbios.rom'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/> </qemu:commandline> </domain> Does anybody have a similar situation and found a solution for this ? Would be happy about any suggestions. Martin Quote Link to comment
fatalfurry Posted March 28, 2016 Share Posted March 28, 2016 What does GPU-Z say for its Bus Interface? Quote Link to comment
jonp Posted March 29, 2016 Share Posted March 29, 2016 And what about physically? What slot is the GPU plugged into? Other pcie devices on that system? Any BIOS settings you can toggle for pcie link speed? Quote Link to comment
iphillips77 Posted March 29, 2016 Share Posted March 29, 2016 It's more likely a problem with nvidia's utility. I had the same issue and was chasing my tail a bit.. Try downloading lspci for Windows and see what it tells you. Nvidia told me 1x no matter what I did, but lspci (and gpu-z, I think?) showed the correct number of lanes. Quote Link to comment
trivial Posted March 29, 2016 Author Share Posted March 29, 2016 First of all thank you for your quick response! @ fatalfurry: GPU-Z told me that the GPU is connected over x16 @ jonp: The card is physical on a x16 port (same port during in bare metal test). There are no other PCIe devices in my system and no speedlimits set in the BIOS. Maybe the NVIDIA software is just wrong and the device is indeed passed through as x16 (as told by GPU-Z) but where the hack is the performance gone? About 95 % of the native performance, as its told in nearly every threat about this topic would be absolutely perfect. Have you any other ideas where i loose such a big chunk of GPU power ? Big thanks in advance Martin Quote Link to comment
jonp Posted March 29, 2016 Share Posted March 29, 2016 Try running the latest unRAID 6.2 beta. Quote Link to comment
spylex Posted March 29, 2016 Share Posted March 29, 2016 unRAID 6.2 has fixed this issue for me... Quote Link to comment
fatalfurry Posted March 29, 2016 Share Posted March 29, 2016 unRAID 6.2 has fixed this issue for me... Was it really running at 1x causing low performance, or was the Nvidia config just reporting the incorrect number? Quote Link to comment
spylex Posted March 29, 2016 Share Posted March 29, 2016 I don't know what fatal combination caused it, but it was on my test machine, and its running 6.2 now so i don't want to go back Actually 6.2 fixed many issues for me, including SSD performance... Quote Link to comment
yesdog Posted March 29, 2016 Share Posted March 29, 2016 You're using the i440fx emulated chipset, so you dont actually have to define another pci bridge/bus. It can sit on the root PCI bus without issue. Some drivers may complain about it, but the NV ones dont seem to. The bus configuration for windows is more about managing DMA memory space and for i440fx it seems to do it just fine on the root bus. Adding more bridges/buses windows is more likely to mismanage DMA space and not assign enough for the GPU. This can be a silent killer and slow down access to the GPU preventing it from performing to its max. i would remove the ioh3420 device, as well as the bus fields on the passthrough devices. GPU-z is also better at showing the PCI speeds because it shows both the max and current connection speeds. Newer GPUs will reduce their PCI spec for a power-saving effort, so you might see a max of 3.0 x16, but current of 1.0 x8 Quote Link to comment
trivial Posted March 30, 2016 Author Share Posted March 30, 2016 I´ve found out that the NV sysinfo is indeed wrong about the number of pci lanes the GPU is connected to. After upgrading to 6.2 the tool consider that the card is connected over x0 lanes @ jonp: I upgrade to 6.2 beta 20 but did not see any difference in GPU performance. I also try messing around with stubbing the gpu and its build in soundcard as well as enabling/disabling pcie ACS override without any results. i would remove the ioh3420 device, as well as the root fields and the passthrough devices. I´ll try removing the ioh3420 device and see if there is any effect. Are there any other suggestions where to search for my lost performance? Quote Link to comment
trivial Posted March 30, 2016 Author Share Posted March 30, 2016 After i´ve upgrade to 6.2 my vm xml looks like this: <domain type='kvm'> <name>Windows 10 sea bios</name> <uuid>40c557f6-29e0-1f45-6d1a-78de2dd6efbf</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>12582912</memory> <currentMemory unit='KiB'>12582912</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> <vcpupin vcpu='2' cpuset='4'/> <vcpupin vcpu='3' cpuset='5'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='2' threads='2'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/cache/OS/Windows 10 sea bios/Windows 10 sea bios/vdisk1.img'/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/ISO/Windows10_64Pro.iso'/> <target dev='hda' bus='ide'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/ISO/virtio-win.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='nec-xhci'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:57:e8:8d'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <source mode='connect'/> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <rom file='/boot/vbios.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc226'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc227'/> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x1532'/> <product id='0x0002'/> </source> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> </devices> </domain> I also tried applying the MSI fix provided by the wiki (http://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support) Did not find any improvement so far. Quote Link to comment
yesdog Posted March 30, 2016 Share Posted March 30, 2016 hmm have you tried setting the machine as EFI (uses OVMF bios)? Quote Link to comment
trivial Posted March 31, 2016 Author Share Posted March 31, 2016 hmm have you tried setting the machine as EFI (uses OVMF bios)? If I remember correctly the GPU passthrough didn´t work with the vbios rom i actual use. But i´ll give it another try. Maybe the vbios rom i´m using is the root of all the trouble. Have anybody found another solution to pass a NV GPU to a guest if there is no other graphics card in the system (not even onboard)? Quote Link to comment
trivial Posted March 31, 2016 Author Share Posted March 31, 2016 Have tried OVMF and see an actual improvement of about 10 FPS more in Cinebench. This leads to about 55% of the bare metal GPU performance, that can´t be the max to get! Quote Link to comment
yesdog Posted March 31, 2016 Share Posted March 31, 2016 Hmm, are you sure the ROM bios matches the bios on your card? I would maybe try to leave out the rom line as OVMF is really good at getting it from your card. Quote Link to comment
trivial Posted April 1, 2016 Author Share Posted April 1, 2016 I´ll give it a try, but when i remember correctly the VM did not even start without the rom line. Quote Link to comment
trivial Posted April 3, 2016 Author Share Posted April 3, 2016 I have tested the ovmf vm without the romfile option, as i already mentioned the passthrough did not work this way. Here a short syslog dump while testing without the romfile (both sea and ovmf): Apr 3 12:01:29 matrix kernel: vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem Apr 3 12:01:30 matrix kernel: device vnet0 entered promiscuous mode Apr 3 12:01:30 matrix kernel: br0: port 2(vnet0) entered listening state Apr 3 12:01:30 matrix kernel: br0: port 2(vnet0) entered listening state Apr 3 12:01:32 matrix kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258 Apr 3 12:01:33 matrix acpid: input device has been disconnected, fd 6 Apr 3 12:01:33 matrix acpid: input device has been disconnected, fd 7 Apr 3 12:01:34 matrix kernel: vfio-pci 0000:01:00.0: Invalid ROM contents Apr 3 12:01:34 matrix kernel: vfio-pci 0000:01:00.0: Invalid ROM contents While testing i have found that the rom file i manage to extract could be the reason for my issues. I tested the card (GTX 950) in the second PCIe slot (x4 lanes) without the romfile, because its not needed there. And find out, that i have the exact same bad performance in this slot. My assumption is, that the rom i extracted for later use while the card is in a x16 slot, is some kind of a "use just x4 lanes" version of the bios. Could anyone provide a vbios extracted from a x16 slot ? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.