April 29, 20179 yr I have been messing with this all week and cannot figure it out. Let me start with some details. Model: Custom M/B: ASRock - Z87 Extreme4 CPU: Intel® Core™ i7-4790S CPU @ 3.20GHz HVM: Enabled IOMMU: Enabled Cache: 256 kB, 1024 kB, 8192 kB Memory: 28 GB (max. installable capacity 32 GB) Network: bond0: adaptive load balancing, mtu 1500 eth0: 1000 Mb/s, full duplex, mtu 1500 eth1: 1000 Mb/s, full duplex, mtu 1500 eth2: 1000 Mb/s, full duplex, mtu 1500 eth3: 1000 Mb/s, full duplex, mtu 1500 eth4: 1000 Mb/s, full duplex, mtu 1500 Kernel: Linux 4.9.19-unRAID x86_64 I have the ACS patch enabled because without it most of the devices listed below are all in group 1 PCI Devices and IOMMU Groups Warning: Your system has booted with the PCIe ACS Override setting enabled. The below list doesn't not reflect the way IOMMU would naturally group devices. To see natural IOMMU groups for your hardware, go to the VM Settings page and set the PCIe ACS Override setting to No. IOMMU group 0 [8086:0c00] 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) IOMMU group 1 [8086:0c01] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) IOMMU group 2 [8086:0c09] 00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06) IOMMU group 3 [8086:0412] 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) IOMMU group 4 [8086:0c0c] 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) IOMMU group 5 [8086:8c31] 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) IOMMU group 6 [8086:8c3a] 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) IOMMU group 7 [8086:153b] 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05) IOMMU group 8 [8086:8c2d] 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) IOMMU group 9 [8086:8c20] 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05) IOMMU group 10 [8086:8c10] 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) IOMMU group 11 [8086:8c14] 00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5) IOMMU group 12 [8086:8c26] 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) IOMMU group 13 [8086:8c44] 00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05) [8086:8c02] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) [8086:8c22] 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05) IOMMU group 14 [1002:6818] 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn XT [Radeon HD 7870 GHz Edition] [1002:aab0] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] IOMMU group 15 [111d:8018] 02:00.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e) IOMMU group 16 [111d:8018] 03:02.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e) IOMMU group 17 [111d:8018] 03:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e) IOMMU group 18 [8086:10bc] 04:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06) [8086:10bc] 04:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06) IOMMU group 19 [8086:10bc] 05:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06) [8086:10bc] 05:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06) IOMMU group 20 [1b21:0612] 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) CPU Thread Pairings cpu 0 <===> cpu 4 cpu 1 <===> cpu 5 cpu 2 <===> cpu 6 cpu 3 <===> cpu 7 USB Devices Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002: ID 8087:8008 Intel Corp. Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 002 Device 002: ID 8087:8000 Intel Corp. Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 003 Device 002: ID 0781:5571 SanDisk Corp. Cruzer Fit Bus 003 Device 003: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub Bus 003 Device 004: ID 045e:0800 Microsoft Corp. Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 004 Device 002: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub SCSI Devices [0:0:0:0] disk SanDisk Cruzer Fit 1.00 /dev/sda 16.0GB [1:0:0:0] disk ATA WDC WD10EZEX-00B 1A01 /dev/sdb 1.00TB [2:0:0:0] disk ATA WDC WD10EZEX-00B 1A01 /dev/sdc 1.00TB [3:0:0:0] disk ATA WDC WD10EZEX-00B 1A01 /dev/sdd 1.00TB [4:0:0:0] disk ATA WDC WD10EZEX-22M 1A01 /dev/sde 1.00TB [5:0:0:0] disk ATA WDC WD10EZEX-22M 1A01 /dev/sdf 1.00TB [6:0:0:0] disk ATA WDC WD5000AVVS-6 0A01 /dev/sdg 500GB [7:0:0:0] disk ATA TOSHIBA MK7559GS 8B /dev/sdh 750GB [8:0:0:0] disk ATA ST9750420AS HPM1 /dev/sdi 750GB I have created two different Windows 10 VMs, 1 is the original i have tried to create which has most of the edits I have tried in its XML. The second, well the second is fresh, I dont even recall if I ever finished installing Windows on the second one. But neither Work with any GPU. <domain type='kvm' id='1'> <name>Steam Stream 1</name> <uuid>b52aecaf-5f69-d2dc-a740-d92cae53ee1d</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='5'/> <vcpupin vcpu='2' cpuset='6'/> <vcpupin vcpu='3' cpuset='7'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/b52aecaf-5f69-d2dc-a740-d92cae53ee1d_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='2' threads='2'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='writeback'/> <source file='/mnt/user/vdisks/Steam Stream 1/vdisk1.img'/> <backingStore/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/en_windows_10_multiple_editions_x64_dvd_6846432.iso'/> <backingStore/> <target dev='hda' bus='ide'/> <readonly/> <boot order='2'/> <alias name='ide0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.126-2.iso'/> <backingStore/> <target dev='hdb' bus='ide'/> <readonly/> <alias name='ide0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='nec-xhci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:23:d2:24'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/1'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/1'> <source path='/dev/pts/1'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-Steam Stream 1/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> </devices> <seclabel type='none' model='none'/> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain> <domain type='kvm'> <name>Steam Stream 2</name> <uuid>e842a934-ba43-279a-15ab-a46cf412cdc6</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>2097152</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='1'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='3'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/e842a934-ba43-279a-15ab-a46cf412cdc6_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='2' threads='2'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='writeback'/> <source file='/mnt/user/domains/Steam Stream 2/vdisk1.img'/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/en_windows_10_multiple_editions_x64_dvd_6846432.iso'/> <target dev='hda' bus='ide'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.126-2.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='nec-xhci'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:39:97:9f'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x045e'/> <product id='0x0800'/> </source> <address type='usb' bus='0' port='1'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> </devices> </domain> Last but not least, here is my syslinux.cfg file, located @ \flash\syslinux\. As you can see I added a few extra parameters to the unRAID OS section. No luck with this either. default /syslinux/menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label unRAID OS menu default kernel /bzimage append pcie_acs_override=downstream intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot label unRAID OS GUI Mode kernel /bzimage append pcie_acs_override=downstream initrd=/bzroot,/bzroot-gui label unRAID OS Safe Mode (no plugins, no GUI) kernel /bzimage append pcie_acs_override=downstream initrd=/bzroot unraidsafemode label unRAID OS GUI Safe Mode (no plugins) kernel /bzimage append pcie_acs_override=downstream initrd=/bzroot,/bzroot-gui unraidsafemode label Memtest86+ kernel /memtest Ok, I think thats it. . So my issue is that I cannot get GPU pass through working no matter what I do. I have tried it with nvidia 660ti, and AMD HD7870. 660ti: I can see the card in Windows, but I get Error Code 43. I have tried: enabling ACS Override adding the iommu lines to the syslinux.cfg file I have two of these cards, in two different PCI-E slots, and neither worked. I had tried adding: <qemu:commandline> <qemu:arg value='-cpu'/> <qemu:arg value='host,kvm=off'/> </qemu:commandline> and <features> <kvm> <hidden state='on'/> </kvm> <hyperv> <vendor_id state='on' value='whatever'/> </hyperv> </features> to the VM XML file for Steam Stream 1 (however they dont seem to be in there now despite the fact that I didnt remove them....) Alright, now onto the HD7870. I cant get into Windows at all, simple as that. I tried on both VM1 and VM2, and nothing ever appears on the monitor at all, no Vbios, nothing. VM1 is setup with TeamViewer since I disabled VNC, and it never shows up in my list of online computers in Teamviewer, unless I go back to the nvidia card, if I use the nvidia card then I can get into windows but im back to the original issue. Both of the 660Ti work fine in other systems, and the HD7870 works just fine as well. I also have made sure that all the Virtualization settings in my BIOS are enabled, my IGPU is set as the default display in the BIOS, and Multi-Monitor Compatibility mode, as well and HDMI-IN Compatibility mode are both disabled in the BIOS aswell. I will be happy to just get this working with any card right now, but I would really prefer to get this working with both of the 660Ti's as the point of these VM's was to allow me and my wife to have our own seperate gaming VMs specifically to use Steam Streaming to computers around the house. Lastly, Any card I have tried passing through has had either a DVI monitor attached to it, OR a 1080p@60hz HDMI dummy plug attached to it. Alright, I cant think of any more details to add, but id be happy to get anymore information that may be needed. I really hope someone can help me out with this.
April 29, 20179 yr Author MORE INFO: So I just tried adding the HD7870 to a Ubuntu VM and it atleast gets me to the desktop...what can you guys make of this? In the meantime im going to make another Windows 10 VM with the same hardware config as the Ubuntu VM, just for shits and giggles because im out of ideas at this point. lol
April 29, 20179 yr Author MORE INFO 2: OK I just created a Windows 10 VM using SeaBIOS and Q35 and...IT BOOTED INTO WINDOWS SETUP!!!!! This was with the HD7870, Im shutting down now and am going to try it with the 660ti now. Wish me luck. Im gonna look like a doofus if I endup fixing this myself before anyone replies. EDIT: So far so good with the 660ti, im installing Windows 10 now. However the VM refused to work with my USB Wireless Microsoft Keyboard/touchpad combo, I had to pull a regular plain old wired USB mouse out of the closet and that worked fine. Also the machine is still using SeaBIOS, but I switch it back to i440fx-2.7 and it was still working. Edited April 29, 20179 yr by relink UPDATE
April 29, 20179 yr Author Well I seem to have solved it myself...Switching to SeaBIOS seems to have been the answer. I just successfully ran the new FFXIV Stormblood Benchmark on Standard settings through Steam in home streaming at 1080p and never dropped below 60fps. it ran buttery smooth. The only issue seems to be rebooting the VM, I sometimes have to force it to stop, and reboot again to get the GPU to show anything but a black screen.
April 29, 20179 yr Seems you found the solution yourself I can however try to explain why you have to switch. Your GPUs doesn't support uefi and using ovmf only creates problems, as you have found out. The VM templates is set up to use either seabios or uefi depending on what the operating system supports. If the operating system supports uefi OVMF will be chosen. Some of the older GPUs do have problems to reset correctly when restarting a VM. What you could try to do is to safely remove the GPU before rebooting.
April 29, 20179 yr Author Thanks for that. I never for a second considered that my GPU wouldn't support a uefi...I just didn't really think that would be a "thing", but its the only answer that makes sense. I will try removing the GPU next time I need to reboot and see if that helps. But honestly the reboot process has been pretty consistent so far. Reboot, Force shutdown, Start again, and bam it works! Im just worried about what will happen after a Windows Update, we all know how MS loves to do it without your permission anymore. Which im not worried about the updates, so much as Im worried my VM could potentially be locked up for days without me noticing, until someone goes to fire up a game in the living room and it inexplicably isn't working.
Archived
This topic is now archived and is closed to further replies.