Problems with GPU passtrough

[TombaDude]

Hello. I am having a hard time passing through the GPUs to VMs. I have two GPUs in my system, a HD7770 and a GTX 780. I have the 7770 in first PCI-E slot and unraid is using it. I am trying to passthrough the 780 in secondary slot to windows 10 VM. I've watched spaceinvaderone's tutorials, read forums and tried with and without binding to VFIO but nothing seems to work. I've also tried with unraid VM setting ACS override off and both. I always get either errors or no boot at all. I've also tried passing the gpu before installing windows and after installing it. No luck. Same thing with ubuntu. What I'm missing? The system supports IOMMU and VT-d and both are enabled.

System specs:
CPU: Intel Xeon E5-2620v3
MOBO: Huananzhi X99 F8
RAM: 32GB (2x16) Samsung ECC DDR4
GPU1: Radeon HD7770
GPU2: Geforce GTX 780

Edited February 5, 2021 by TombaDude

[TombaDude]

Here is the log from win10 VM. This is with ACS disabled and GTX 780 binded to VFIO. The VM seemingly starts but I get no display what so ever and can't connect with RDP configured earlier which tells me the system isn't really booted....

 

ErrorWarningSystemArrayLogin

-blockdev '{"driver":"file","filename":"/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/8c014591-3fa3-1ca8-8754-6ac44de91284_VARS-pure-efi.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-i440fx-5.1,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-cpu host,migratable=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=none,host-cache-info=on,l3-cache=off \
-m 12800 \
-overcommit mem-lock=off \
-smp 8,sockets=1,dies=1,cores=4,threads=2 \
-uuid 8c014591-3fa3-1ca8-8754-6ac44de91284 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=34,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.0,addr=0x7 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/Windows 10 - NAS/vdisk1.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device virtio-blk-pci,bus=pci.0,addr=0x3,drive=libvirt-1-format,id=virtio-disk2,bootindex=1,write-cache=on \
-netdev tap,fd=36,id=hostnet0 \
-device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:49:81:9c,bus=pci.0,addr=0x2 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=37,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-device vfio-pci,host=0000:04:00.0,id=hostdev0,bus=pci.0,addr=0x5 \
-device vfio-pci,host=0000:04:00.1,id=hostdev1,bus=pci.0,addr=0x6 \
-device usb-host,hostbus=3,hostaddr=3,id=hostdev2,bus=usb.0,port=2 \
-device usb-host,hostbus=3,hostaddr=2,id=hostdev3,bus=usb.0,port=3 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-02-05 23:40:50.832+0000: Domain id=2 is tainted: high-privileges
2021-02-05 23:40:50.832+0000: Domain id=2 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)

[meep]

Check the link in my sig to see if there are any tips there that you might have missed.

 

one trick that works well for me is to install virt manager docker, and use that to add a VNC graphics device alongside your gpu passthrough. In the vm xml , double check to ensure that the vnc adapter is first (before the gpu).

 

then, when you boot the vm, you can observe the boot process via VNc to see what’s foing on.

 

ive had instances in the past where the vm was stalling or jumping into seabios, of where windows was loading into recovery or waiting at the login screen.

 

its always a good idea to use netplwiz on windows vms to set up auto-login, it eliminates on other potential trip hazard.

 

 

 

Edited February 8, 2021 by meep
Clarification / accuracy

[TombaDude]

Just now, meep said:

Check the link in my sig to see if there are any tips there that you might have missed.

 

one trick that works well for me is to install vm manager docker, and use that to add a VNC graphics device alongside your gpu passthrough. In the vm xml , double check to ensure that the vnc adapter is first (before the gpu).

 

then, when you boot the vm, you can observe the boot process via VNc to see what’s foing on.

 

ive had instances in the past where the vm was stalling or jumping into seabios, of where windows was loading into recovery or waiting at the login screen.

 

its always a good idea to use netplwiz on windows vms to set up auto-login, it eliminates on other potential trip hazard.

 

 

 

Thanks, I'll try with those tricks. One thing I'm going to change first is the BIOS to seabios. I'll get back to here when I've tried these fixes...

[TombaDude]

One step forward, I got the gpu to show up in device manager but it has the -43 code  I'll try to get rid of it later...

 

[meep]

12 hours ago, TombaDude said:

One step forward, I got the gpu to show up in device manager but it has the -43 code  I'll try to get rid of it later...

 

That’s progress. Section 8 in my blog post suggests a solution for -43 that’s worked well for me.

[TombaDude]

So I tried to get rid of the -43 error. With seabios and gpu vbios passed through, and also the fact that it's VM hidden, the VM starts but still -43. I changed to OVMF and got following on the VM log:
 

ErrorWarningSystemArrayLogin

-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=38,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=3 \
-vnc 0.0.0.0:1,websocket=5701 \
-k fi \
-device cirrus-vga,id=video0,bus=pcie.0,addr=0x1 \
-device vfio-pci,host=0000:04:00.0,id=hostdev0,bus=pci.4,addr=0x0,romfile=/mnt/user/isos/GTX780manualdump.dump \
-device vfio-pci,host=0000:04:00.1,id=hostdev1,bus=pci.5,addr=0x0 \
-device usb-host,hostbus=3,hostaddr=5,id=hostdev2,bus=usb.0,port=1 \
-device usb-host,hostbus=3,hostaddr=2,id=hostdev3,bus=usb.0,port=2 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-02-06 19:50:54.933+0000: Domain id=16 is tainted: high-privileges
2021-02-06 19:50:54.933+0000: Domain id=16 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)
KVM internal error. Suberror: 1
emulation failure
RAX=0000000000000000 RBX=000000007f31f798 RCX=0000000000000000 RDX=000000007f3e9718
RSI=000000007f2d8e18 RDI=000000007f322698 RBP=000000007f3e9718 RSP=000000007fe89678
R8 =000000007fe896e0 R9 =0000000000000001 R10=0000000000000000 R11=000000007ea19ba0
R12=000000007f2d4c18 R13=0000000000000000 R14=000000007f31e698 R15=000000007f2d4098
RIP=00000000000b0000 RFL=00010246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
CS =0038 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
DS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
FS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
GS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA]
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT= 000000007fe16698 00000047
IDT= 000000007f858018 00000fff
CR0=80010033 CR2=0000000000000000 CR3=000000007fe28000 CR4=00000668
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000500
Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <ff> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
2021-02-06T19:55:13.919890Z qemu-system-x86_64: terminating on signal 15 from pid 4084 (/usr/sbin/libvirtd)
2021-02-06 19:55:16.321+0000: shutting down, reason=destroyed

[TombaDude]

Finally success! My problem was I had hyper-v on. Turned it off and works like a charm

Edited February 13, 2021 by TombaDude

[TombaDude]

I haven't had time to play around with the vm since I got it working but now that I do, I have noticed it has terrible performance. It's running on unassigned device NVME SSD and has 4 cores + threads assigned and isolated for it. Everything feels sluggish, Opening software takes more time than should, even file eplorer opens like 10 seconds. The windows task manager is showing barely any resource usage, same with unraid cpu monitoring. Disk reads and writes are fast (tested in VM with crystal disk mark), what's the hangup?