GPU passthrough help...not booting?

[slizz]

I have a Windows 10 install that is working well.  I can RDP into it when assigning the VM to VNC.  Once I assign my GPU (AMD RADEON 7970) it doesn't show up on a connected monitor and I can no longer RDP into the VM.  It may not even be booting.  I had a Windows 10 VM working fine with this GPU and pass through yesterday.  I just decided to get rid of that VM and do a clean install because of some errors that I made.  HVM and IOMMU are enabled.  PCIe ACS override is on.  The previous working VM I had was set up originally in unRaid 6.1.  I am currently using version 6.2.4.  Thanks for any insight. 

 

 

[slizz]

00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller [8086:0c08] (rev 06)

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)

00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)

00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 05)

00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 05)

00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5)

00:1c.2 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 [8086:8c14] (rev d5)

00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5)

00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05)

00:1f.0 ISA bridge [0601]: Intel Corporation C222 Series Chipset Family Server Essential SKU LPC Controller [8086:8c52] (rev 05)

00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05)

00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05)

00:1f.6 Signal processing controller [1180]: Intel Corporation 8 Series Chipset Family Thermal Management Controller [8086:8c24] (rev 05)

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]

01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970] [1002:aaa0]

02:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0086] (rev 05)

03:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 03)

04:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30)

05:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)

06:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)

IOMMU Groups

 

Warning: Your system has booted with the PCIe ACS Override setting enabled. The below list doesn't not reflect the way IOMMU would naturally group devices. To see natural IOMMU groups for your hardware, go to the VM Settings page and set the PCIe ACS Override setting to No.

/sys/kernel/iommu_groups/0/devices/0000:00:00.0

/sys/kernel/iommu_groups/1/devices/0000:00:01.0

/sys/kernel/iommu_groups/2/devices/0000:00:01.1

/sys/kernel/iommu_groups/3/devices/0000:00:14.0

/sys/kernel/iommu_groups/4/devices/0000:00:1a.0

/sys/kernel/iommu_groups/5/devices/0000:00:1c.0

/sys/kernel/iommu_groups/6/devices/0000:00:1c.2

/sys/kernel/iommu_groups/7/devices/0000:00:1c.3

/sys/kernel/iommu_groups/8/devices/0000:00:1d.0

/sys/kernel/iommu_groups/9/devices/0000:00:1f.0

/sys/kernel/iommu_groups/9/devices/0000:00:1f.2

/sys/kernel/iommu_groups/9/devices/0000:00:1f.3

/sys/kernel/iommu_groups/9/devices/0000:00:1f.6

/sys/kernel/iommu_groups/10/devices/0000:01:00.0

/sys/kernel/iommu_groups/10/devices/0000:01:00.1

/sys/kernel/iommu_groups/11/devices/0000:02:00.0

/sys/kernel/iommu_groups/12/devices/0000:03:00.0

/sys/kernel/iommu_groups/12/devices/0000:04:00.0

/sys/kernel/iommu_groups/13/devices/0000:05:00.0

/sys/kernel/iommu_groups/14/devices/0000:06:00.0

CPU Thread Pairings

 

cpu 0 <===> cpu 4

cpu 1 <===> cpu 5

cpu 2 <===> cpu 6

cpu 3 <===> cpu 7

USB Devices

 

Bus 002 Device 002: ID 8087:8000 Intel Corp.

Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 001 Device 002: ID 8087:8008 Intel Corp.

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 003 Device 005: ID 0557:2419 ATEN International Co., Ltd

Bus 003 Device 004: ID 0557:7000 ATEN International Co., Ltd Hub

Bus 003 Device 003: ID 046d:c52b Logitech, Inc. Unifying Receiver

Bus 003 Device 002: ID 0781:5571 SanDisk Corp. Cruzer Fit

Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

SCSI Devices

 

[0:0:0:0]    disk    SanDisk  Cruzer Fit      1.00  /dev/sda  8.00GB

[2:0:0:0]    disk    ATA      Samsung SSD 850  1B6Q  /dev/sdb    250GB

[4:0:0:0]    disk    ATA      WDC WD30EFRX-68E 0A82  /dev/sdc  3.00TB

[5:0:0:0]    disk    ATA      WDC WD30EFRX-68E 0A82  /dev/sdd  3.00TB

[6:0:0:0]    disk    ATA      WDC WD30EFRX-68E 0A82  /dev/sde  3.00TB

[7:0:0:0]    disk    ATA      WDC WD30EFRX-68E 0A82  /dev/sdf  3.00TB

[1812]

not an expert here:

 

but since you have 2 devices in IOMMU group 12 (03:00.0 and 04:00.0) you might make sure you're also passing through the bridge (03:00.0) as well as the gpu (04:00.0) , since KVM  wants the entire group.... or so i'm told.

 

 

 

 

 

[slizz]

What do you mean pass through the bridge?  I have br0 selected in the VMSettings as the default network bridge. 

[slizz]

Here is the log from the VM:

 

 

2016-11-28 21:17:38.694+0000: starting up libvirt version: 1.3.1, qemu version: 2.5.1, hostname: unLIZZIO

LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/local/sbin/qemu -name 'Windows 10' -S -machine pc-i440fx-2.5,accel=kvm,usb=off,mem-merge=off -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=none -drive file=/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/etc/libvirt/qemu/nvram/ed964547-aba4-4f97-5843-3e308a11416e_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 -m 8192 -realtime mlock=on -smp 6,sockets=1,cores=3,threads=2 -uuid ed964547-aba4-4f97-5843-3e308a11416e -nographic -no-user-config -nodefaults -chardev 'socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-Windows 10/monitor.sock,server,nowait' -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-hpet -no-shutdown -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x7.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x7 -device ich9-usb-uhci2,mastbvirt/qemu/channel/target/domain-Windows 10/org.qemu.guest_agent.0,server,nowait' -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x5 -device vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x6 -device usb-host,hostbus=3,hostaddr=3,id=hostdev2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on

Domain id=1 is tainted: high-privileges

Domain id=1 is tainted: host-cpu

char device redirected to /dev/pts/0 (label charserial0)

[1812]

What do you mean pass through the bridge?  I have br0 selected in the VMSettings as the default network bridge.

 

 

WAIT

 

is that the onboard graphics (04:00.0)? if so IGNORE what i put before. It was late and I glanced at your post...

 

instead when you assign this gpu:

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]

 

then you need to assign through this audio at the same time:

01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970] [1002:aaa0]

 

since both of those are in the same IOMMU group.

 

 

unless you're doing that already? please post your xml.

[thaddeussmith]

Check my recent thread for some ideas as well.. Lots of head banging before an obscure setting was suggested.

 

http://lime-technology.com/forum/index.php?topic=54059.0

[SpaceInvaderOne]

I have a Windows 10 install that is working well.  I can RDP into it when assigning the VM to VNC.  Once I assign my GPU (AMD RADEON 7970) it doesn't show up on a connected monitor and I can no longer RDP into the VM.  It may not even be booting.  I had a Windows 10 VM working fine with this GPU and pass through yesterday.  I just decided to get rid of that VM and do a clean install because of some errors that I made.  HVM and IOMMU are enabled.  PCIe ACS override is on.  The previous working VM I had was set up originally in unRaid 6.1.  I am currently using version 6.2.4.  Thanks for any insight.

 

Please try setting up a new windows 10 VM.

But choose seabios for the bios type. ( In 6.19 the unRAID version in which you set up the vm which worked, by default windows VMS defaulted to seabios. From version 6.2 they default to ovmf.)

I suspect that your GPU doesn't have an EFI bios ,which it would need to work with an ovmf VM hence the black screen.

[slizz]

Well folks...turns out I'm just not very smart.  I had assigned a static IP when accessing in VNC.  When I changed to the GPU it changed the static connection back to DHCP which is why RDP was not working.  I realized this after a restart of the VM while sitting at my TV which is connected to the GPU rather than doing it from the other room....Doh!  Thanks for all of the help as always.