TombaDude Posted February 5, 2021 Share Posted February 5, 2021 (edited) Hello. I am having a hard time passing through the GPUs to VMs. I have two GPUs in my system, a HD7770 and a GTX 780. I have the 7770 in first PCI-E slot and unraid is using it. I am trying to passthrough the 780 in secondary slot to windows 10 VM. I've watched spaceinvaderone's tutorials, read forums and tried with and without binding to VFIO but nothing seems to work. I've also tried with unraid VM setting ACS override off and both. I always get either errors or no boot at all. I've also tried passing the gpu before installing windows and after installing it. No luck. Same thing with ubuntu. What I'm missing? The system supports IOMMU and VT-d and both are enabled. System specs: CPU: Intel Xeon E5-2620v3 MOBO: Huananzhi X99 F8 RAM: 32GB (2x16) Samsung ECC DDR4 GPU1: Radeon HD7770 GPU2: Geforce GTX 780 Edited February 5, 2021 by TombaDude Quote Link to comment
TombaDude Posted February 5, 2021 Author Share Posted February 5, 2021 Here is the log from win10 VM. This is with ACS disabled and GTX 780 binded to VFIO. The VM seemingly starts but I get no display what so ever and can't connect with RDP configured earlier which tells me the system isn't really booted.... ErrorWarningSystemArrayLogin -blockdev '{"driver":"file","filename":"/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \ -blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/8c014591-3fa3-1ca8-8754-6ac44de91284_VARS-pure-efi.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \ -machine pc-i440fx-5.1,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \ -cpu host,migratable=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=none,host-cache-info=on,l3-cache=off \ -m 12800 \ -overcommit mem-lock=off \ -smp 8,sockets=1,dies=1,cores=4,threads=2 \ -uuid 8c014591-3fa3-1ca8-8754-6ac44de91284 \ -display none \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=34,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=localtime \ -no-hpet \ -no-shutdown \ -boot strict=on \ -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.0,addr=0x7 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 \ -blockdev '{"driver":"file","filename":"/mnt/user/domains/Windows 10 - NAS/vdisk1.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \ -device virtio-blk-pci,bus=pci.0,addr=0x3,drive=libvirt-1-format,id=virtio-disk2,bootindex=1,write-cache=on \ -netdev tap,fd=36,id=hostnet0 \ -device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:49:81:9c,bus=pci.0,addr=0x2 \ -chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0 \ -chardev socket,id=charchannel0,fd=37,server,nowait \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \ -device usb-tablet,id=input0,bus=usb.0,port=1 \ -device vfio-pci,host=0000:04:00.0,id=hostdev0,bus=pci.0,addr=0x5 \ -device vfio-pci,host=0000:04:00.1,id=hostdev1,bus=pci.0,addr=0x6 \ -device usb-host,hostbus=3,hostaddr=3,id=hostdev2,bus=usb.0,port=2 \ -device usb-host,hostbus=3,hostaddr=2,id=hostdev3,bus=usb.0,port=3 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on 2021-02-05 23:40:50.832+0000: Domain id=2 is tainted: high-privileges 2021-02-05 23:40:50.832+0000: Domain id=2 is tainted: host-cpu char device redirected to /dev/pts/1 (label charserial0) Quote Link to comment
meep Posted February 6, 2021 Share Posted February 6, 2021 (edited) Check the link in my sig to see if there are any tips there that you might have missed. one trick that works well for me is to install virt manager docker, and use that to add a VNC graphics device alongside your gpu passthrough. In the vm xml , double check to ensure that the vnc adapter is first (before the gpu). then, when you boot the vm, you can observe the boot process via VNc to see what’s foing on. ive had instances in the past where the vm was stalling or jumping into seabios, of where windows was loading into recovery or waiting at the login screen. its always a good idea to use netplwiz on windows vms to set up auto-login, it eliminates on other potential trip hazard. Edited February 8, 2021 by meep Clarification / accuracy 1 Quote Link to comment
TombaDude Posted February 6, 2021 Author Share Posted February 6, 2021 Just now, meep said: Check the link in my sig to see if there are any tips there that you might have missed. one trick that works well for me is to install vm manager docker, and use that to add a VNC graphics device alongside your gpu passthrough. In the vm xml , double check to ensure that the vnc adapter is first (before the gpu). then, when you boot the vm, you can observe the boot process via VNc to see what’s foing on. ive had instances in the past where the vm was stalling or jumping into seabios, of where windows was loading into recovery or waiting at the login screen. its always a good idea to use netplwiz on windows vms to set up auto-login, it eliminates on other potential trip hazard. Thanks, I'll try with those tricks. One thing I'm going to change first is the BIOS to seabios. I'll get back to here when I've tried these fixes... Quote Link to comment
TombaDude Posted February 6, 2021 Author Share Posted February 6, 2021 One step forward, I got the gpu to show up in device manager but it has the -43 code I'll try to get rid of it later... Quote Link to comment
meep Posted February 6, 2021 Share Posted February 6, 2021 12 hours ago, TombaDude said: One step forward, I got the gpu to show up in device manager but it has the -43 code I'll try to get rid of it later... That’s progress. Section 8 in my blog post suggests a solution for -43 that’s worked well for me. Quote Link to comment
TombaDude Posted February 6, 2021 Author Share Posted February 6, 2021 So I tried to get rid of the -43 error. With seabios and gpu vbios passed through, and also the fact that it's VM hidden, the VM starts but still -43. I changed to OVMF and got following on the VM log: ErrorWarningSystemArrayLogin -device isa-serial,chardev=charserial0,id=serial0 \ -chardev socket,id=charchannel0,fd=38,server,nowait \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \ -device usb-tablet,id=input0,bus=usb.0,port=3 \ -vnc 0.0.0.0:1,websocket=5701 \ -k fi \ -device cirrus-vga,id=video0,bus=pcie.0,addr=0x1 \ -device vfio-pci,host=0000:04:00.0,id=hostdev0,bus=pci.4,addr=0x0,romfile=/mnt/user/isos/GTX780manualdump.dump \ -device vfio-pci,host=0000:04:00.1,id=hostdev1,bus=pci.5,addr=0x0 \ -device usb-host,hostbus=3,hostaddr=5,id=hostdev2,bus=usb.0,port=1 \ -device usb-host,hostbus=3,hostaddr=2,id=hostdev3,bus=usb.0,port=2 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on 2021-02-06 19:50:54.933+0000: Domain id=16 is tainted: high-privileges 2021-02-06 19:50:54.933+0000: Domain id=16 is tainted: host-cpu char device redirected to /dev/pts/1 (label charserial0) KVM internal error. Suberror: 1 emulation failure RAX=0000000000000000 RBX=000000007f31f798 RCX=0000000000000000 RDX=000000007f3e9718 RSI=000000007f2d8e18 RDI=000000007f322698 RBP=000000007f3e9718 RSP=000000007fe89678 R8 =000000007fe896e0 R9 =0000000000000001 R10=0000000000000000 R11=000000007ea19ba0 R12=000000007f2d4c18 R13=0000000000000000 R14=000000007f31e698 R15=000000007f2d4098 RIP=00000000000b0000 RFL=00010246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 ES =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA] CS =0038 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA] SS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA] DS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA] FS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA] GS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA] LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy GDT= 000000007fe16698 00000047 IDT= 000000007f858018 00000fff CR0=80010033 CR2=0000000000000000 CR3=000000007fe28000 CR4=00000668 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000ffff0ff0 DR7=0000000000000400 EFER=0000000000000500 Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <ff> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 2021-02-06T19:55:13.919890Z qemu-system-x86_64: terminating on signal 15 from pid 4084 (/usr/sbin/libvirtd) 2021-02-06 19:55:16.321+0000: shutting down, reason=destroyed Quote Link to comment
TombaDude Posted February 6, 2021 Author Share Posted February 6, 2021 (edited) Finally success! My problem was I had hyper-v on. Turned it off and works like a charm Edited February 13, 2021 by TombaDude 1 Quote Link to comment
TombaDude Posted February 13, 2021 Author Share Posted February 13, 2021 I haven't had time to play around with the vm since I got it working but now that I do, I have noticed it has terrible performance. It's running on unassigned device NVME SSD and has 4 cores + threads assigned and isolated for it. Everything feels sluggish, Opening software takes more time than should, even file eplorer opens like 10 seconds. The windows task manager is showing barely any resource usage, same with unraid cpu monitoring. Disk reads and writes are fast (tested in VM with crystal disk mark), what's the hangup? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.