ok so here is what I've found with one gpu
lspci - v gives me this
04:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 980] (rev a1) (prog-if 00 [VGA controller])
Subsystem: eVga.com. Corp. GM204 [GeForce GTX 980]
Flags: fast devsel, IRQ 11, NUMA node 0
Memory at f8000000 (32-bit, non-prefetchable) [disabled]
Memory at b0000000 (64-bit, prefetchable) [disabled]
Memory at c0000000 (64-bit, prefetchable) [disabled]
I/O ports at d000 [disabled]
Expansion ROM at f9000000 [disabled by cmd]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [258] L1 PM Substates
Capabilities: [128] Power Budgeting <?>
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] #19
04:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)
Subsystem: eVga.com. Corp. GM204 High Definition Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 10, NUMA node 0
Memory at f9080000 (32-bit, non-prefetchable)
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
and IOMMU groups are this
IOMMU group 26
[8086:8d47] 00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)
[8086:2822] 00:1f.2 RAID bus controller: Intel Corporation SATA Controller [RAID mode] (rev 05)
[8086:8d22] 00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05)
IOMMU group 27
[10de:13c0] 04:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 980] (rev a1)
[10de:0fbb] 04:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)
IOMMU group 28
[1b21:1242] 05:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
IOMMU group 29
[1b21:1242] 06:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
Logs SeaBIOS
Jul 21 11:44:29 Tower kernel: input: Logitech USB Optical Mouse as /devices/pci0000:00/0000:00:14.0/usb1/1-9/1-9.3/1-9.3:1.0/0003:046D:C077.000A/input/input14 Jul 21 11:44:29 Tower kernel: hid-generic 0003:046D:C077.000A: input,hidraw3: USB HID v1.11 Mouse [Logitech USB Optical Mouse] on usb-0000:00:14.0-9.3/input0 Jul 21 11:44:41 Tower kernel: vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem Jul 21 11:44:41 Tower kernel: br0: port 2(vnet0) entered blocking state Jul 21 11:44:41 Tower kernel: br0: port 2(vnet0) entered disabled state Jul 21 11:44:41 Tower kernel: device vnet0 entered promiscuous mode Jul 21 11:44:41 Tower kernel: br0: port 2(vnet0) entered blocking state Jul 21 11:44:41 Tower kernel: br0: port 2(vnet0) entered forwarding state Jul 21 11:44:42 Tower kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258 Jul 21 11:44:42 Tower kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Jul 21 11:44:42 Tower kernel: vfio-pci 0000:00:1b.0: enabling device (0000 -> 0002) Jul 21 11:44:42 Tower acpid: input device has been disconnected, fd 6 Jul 21 11:44:42 Tower acpid: input device has been disconnected, fd 7 Jul 21 11:44:44 Tower kernel: kvm: zapping shadow pages for mmio generation wraparound Jul 21 11:44:49 Tower kernel: usb 1-9.3: reset low-speed USB device number 11 using xhci_hcd Jul 21 11:44:50 Tower kernel: usb 1-9.3: reset low-speed USB device number 11 using xhci_hcd Jul 21 11:44:50 Tower kernel: usb 1-9.2: reset low-speed USB device number 6 using xhci_hcd Jul 21 11:44:51 Tower kernel: usb 1-9.2: reset low-speed USB device number 6 using xhci_hcd Jul 21 11:44:51 Tower kernel: kvm: zapping shadow pages for mmio generation wraparound
VM Logs
2017-07-21 18:44:41.766+0000: starting up libvirt version: 2.4.0, qemu version: 2.7.1, hostname: Tower LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/local/sbin/qemu -name 'guest=Windows 10 Gaming,debug-threads=on' -S -object 'secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-Windows 10 Gaming/master-key.aes' -machine pc-i440fx-2.7,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -cpu host -m 8192 -realtime mlock=off -smp 4,sockets=1,cores=2,threads=2 -uuid 3e3a2e1e-bfa3-59e0-2bde-cab34c96cd5f -display none -no-user-config -nodefaults -chardev 'socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-2-Windows 10 Gaming/monitor.sock,server,nowait' -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device nec-usb-xhci,id=usb,bus=pci.0,addr=0x7 -device ahci,id=sata0,bus=pci.0,addr=0x3 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive 'file=/mnt/user/domains/Windows 10 Gaming/vdisk1.img,format=r=0x6,romfile=/mnt/user/drivers/test.dump -device vfio-pci,host=00:1b.0,id=hostdev1,bus=pci.0,addr=0x8 -device usb-host,hostbus=1,hostaddr=11,id=hostdev2,bus=usb.0,port=1 -device usb-host,hostbus=1,hostaddr=6,id=hostdev3,bus=usb.0,port=2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 -msg timestamp=on Domain id=2 is tainted: high-privileges Domain id=2 is tainted: host-cpu char device redirected to /dev/pts/0 (label charserial0)
LOGS OVMF
Jul 21 11:52:33 Tower kernel: usb 1-9.3: USB disconnect, device number 13 Jul 21 11:52:34 Tower kernel: usb 1-9.3: new low-speed USB device number 14 using xhci_hcd Jul 21 11:52:34 Tower kernel: input: Logitech USB Optical Mouse as /devices/pci0000:00/0000:00:14.0/usb1/1-9/1-9.3/1-9.3:1.0/0003:046D:C077.0010/input/input20 Jul 21 11:52:34 Tower kernel: hid-generic 0003:046D:C077.0010: input,hidraw3: USB HID v1.11 Mouse [Logitech USB Optical Mouse] on usb-0000:00:14.0-9.3/input0 Jul 21 11:52:57 Tower kernel: vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem Jul 21 11:52:57 Tower kernel: br0: port 2(vnet0) entered blocking state Jul 21 11:52:57 Tower kernel: br0: port 2(vnet0) entered disabled state Jul 21 11:52:57 Tower kernel: device vnet0 entered promiscuous mode Jul 21 11:52:57 Tower kernel: br0: port 2(vnet0) entered blocking state Jul 21 11:52:57 Tower kernel: br0: port 2(vnet0) entered forwarding state Jul 21 11:52:59 Tower kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258 Jul 21 11:52:59 Tower kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Jul 21 11:52:59 Tower kernel: vfio-pci 0000:00:1b.0: enabling device (0000 -> 0002) Jul 21 11:52:59 Tower acpid: input device has been disconnected, fd 6 Jul 21 11:52:59 Tower acpid: input device has been disconnected, fd 7 Jul 21 11:53:06 Tower kernel: usb 1-9.3: reset low-speed USB device number 14 using xhci_hcd Jul 21 11:53:07 Tower kernel: usb 1-9.3: reset low-speed USB device number 14 using xhci_hcd Jul 21 11:53:08 Tower kernel: usb 1-9.2: reset low-speed USB device number 6 using xhci_hcd Jul 21 11:53:08 Tower kernel: usb 1-9.2: reset low-speed USB device number 6 using xhci_hcd Jul 21 11:53:10 Tower kernel: kvm: zapping shadow pages for mmio generation wraparound Jul 21 11:53:10 Tower kernel: kvm: zapping shadow pages for mmio generation wraparound
VM Logs
2017-07-21 18:52:57.121+0000: starting up libvirt version: 2.4.0, qemu version: 2.7.1, hostname: Tower LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/local/sbin/qemu -name 'guest=Windows 10,debug-threads=on' -S -object 'secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3-Windows 10/master-key.aes' -machine pc-i440fx-2.7,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -cpu host -drive file=/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/etc/libvirt/qemu/nvram/988a76c3-d21b-e56b-8e8c-a0f7c78f1c24_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 -m 20480 -realtime mlock=off -smp 8,sockets=1,cores=4,threads=2 -uuid 988a76c3-d21b-e56b-8e8c-a0f7c78f1c24 -display none -no-user-config -nodefaults -chardev 'socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-3-Windows 10/monitor.sock,server,nowait' -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device nec-usb-xhci,i,nowait' -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x6,romfile=/mnt/user/drivers/test.dump -device vfio-pci,host=00:1b.0,id=hostdev1,bus=pci.0,addr=0x8 -device usb-host,hostbus=1,hostaddr=14,id=hostdev2,bus=usb.0,port=1 -device usb-host,hostbus=1,hostaddr=6,id=hostdev3,bus=usb.0,port=2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 -msg timestamp=on Domain id=3 is tainted: high-privileges Domain id=3 is tainted: host-cpu char device redirected to /dev/pts/0 (label charserial0) 2017-07-21T18:54:12.473689Z qemu-system-x86_64: libusb_release_interface: -4 [NO_DEVICE]
I performed a cat of the rom with the GTX980 in the second pci slot which worked successfully (attached). I then tested that rom by editing the xml on 2 seperate vms. one with SeaBIOS and the other OVMF, both of which gave me a black screen. Ive tried this with the GTX 980 in the primary PCI slot and the secondary slot with the same results.
The vms tested above do work with passthrough when there are 2 GFX cards in the rig, when GT 710 is in the primary PCI slot and theGTX 980 in the secondary .
Sorry for the lengthy post.
TL;DR
Cannot get to work with 1 GPU and rom dump. Will work with 2 GPU's
GTX980test.dump