December 22, 20178 yr I've been using a Windows 10 VM on Unraid with GPU Passthrough with no problems for the past few months. I ran into an odd issue today where my user shares disappeared (a listing on /mnt showed a bunch of question marks for the user directory). So I restarted my server and fixed the problem BUT I could no longer boot into Windows with GPU Passthrough. I can get into the VM with VNC without any issues, and I can use GPU Passthrough with a Debian VM. Here's a list of things I've attempted (all with the same result: black screen when I start the VM with GPU Passthrough): VNC into windows, turn off fast boot, gracefully shut down New VM pointing to same vdisk New VM pointing to same vdisk with Q35 New VM pointing to same vdisk with SeaBios (it wouldn't make sense for this to work, but I tried anyways) Reduce number of CPU used by VM (from 4 to 2) Reduce RAM dedicated to VM (from 8GB to 4GB) HyperV disabled I have NOT tried to create a fresh VM yet, but that is next on the list. For the record, I have 32GB of RAM with an intel core i7-7700 CPU and a NVIDIA GTX1060 card. I'm curious if anyone has run into a similar issue? What info would be good for me to provide? Below is what my original Windows 10 XML looked like. <domain type='kvm'> <name>Windows</name> <uuid>e0626207-002f-ddc3-e611-8ebe6587a672</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> <vcpupin vcpu='2' cpuset='6'/> <vcpupin vcpu='3' cpuset='7'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/e0626207-002f-ddc3-e611-8ebe6587a672_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='2' threads='2'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/Windows/vdisk1.img'/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/Windows10x64.iso'/> <target dev='hda' bus='ide'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.126-2.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:23:5c:77'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x045e'/> <product id='0x0745'/> </source> <address type='usb' bus='0' port='1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc537'/> </source> <address type='usb' bus='0' port='2'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </memballoon> </devices> </domain> Edited December 22, 20178 yr by fronzbot Marking as solved
December 22, 20178 yr just some guesswork does the vm show any activity when you start it ? what does the system log say ? any errors (call traces etc.) there for passthrough / vfio assignments when you click on start vm ? I would stop the 2 device USB passthrough (keyboard and mouse plugged in unraid usb slot ?) and switch to only one core and hyper-v and see what happens are you using a ACS patch ? GPUs should be in an isolated iommu group. unraid is not supposed to bind drivers for the dGPUs anyway. just wondering with you here.
December 22, 20178 yr Author Thanks for the suggestions! Unfortunately, none of those ended up working, but it at least gave me some more information. Here are my responses to each of your suggestions: The VM shows no activity when I start it with GPU passthrough. VNC works fine. And to be clear, when I say no activity I mean just that: I hit 'start' and my monitors stay black the whole time. I have to force stop the VM to kill it. See bottom of this post for system log stuff. Passing through only the GPU and zero USB devices does nothing. Same with one core and hyper-v. Yes, I was using the ACS override patch. Out of curiosity, I disabled it and rebooted my server and have the same results without it (Windows 10 VMw/ passthrough does not work, VNC with Windows works, Debian with passthrough works) See below for my iommu groups. I have my integrated graphics set as the boot video driver and the GPU is in its own iommu group. As for the syslog, there's nothing overly interesting (warning or error wise) but I DO have an invalid ROM header entry in the log. As far as I can tell with some googling, it isn't a problem, but maybe there's something there? Dec 22 15:17:56 baratheon kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900 Dec 22 15:17:56 baratheon kernel: vfio-pci 0000:05:00.0: enabling device (0400 -> 0402) Dec 22 15:17:58 baratheon acpid: input device has been disconnected, fd 9 Dec 22 15:17:58 baratheon acpid: input device has been disconnected, fd 10 Dec 22 15:17:58 baratheon acpid: input device has been disconnected, fd 11 Dec 22 15:17:58 baratheon acpid: input device has been disconnected, fd 6 Dec 22 15:17:59 baratheon kernel: usb 3-2: reset full-speed USB device number 2 using xhci_hcd Dec 22 15:17:59 baratheon kernel: usb 1-9: reset high-speed USB device number 5 using xhci_hcd Dec 22 15:18:00 baratheon kernel: usb 3-2: reset full-speed USB device number 2 using xhci_hcd Dec 22 15:18:00 baratheon kernel: usb 1-9: reset high-speed USB device number 5 using xhci_hcd Dec 22 15:18:03 baratheon kernel: vfio-pci 0000:01:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff Dec 22 15:18:03 baratheon kernel: vfio-pci 0000:01:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff Dec 22 15:18:04 baratheon kernel: kvm: zapping shadow pages for mmio generation wraparound Dec 22 15:18:04 baratheon kernel: kvm: zapping shadow pages for mmio generation wraparound And then my iommu groups: IOMMU group 0 [8086:591f] 00:00.0 Host bridge: Intel Corporation Device 591f (rev 05) IOMMU group 1 [8086:1901] 00:01.0 PCI bridge: Intel Corporation Skylake PCIe Controller (x16) (rev 05) [10de:1c03] 01:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1) [10de:10f1] 01:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1) IOMMU group 2 [8086:a2af] 00:14.0 USB controller: Intel Corporation 200 Series PCH USB 3.0 xHCI Controller IOMMU group 3 [8086:a2ba] 00:16.0 Communication controller: Intel Corporation 200 Series PCH CSME HECI #1 IOMMU group 4 [8086:a282] 00:17.0 SATA controller: Intel Corporation 200 Series PCH SATA controller [AHCI mode] IOMMU group 5 [8086:a2e7] 00:1b.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #17 (rev f0) IOMMU group 6 [8086:a290] 00:1c.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #1 (rev f0) IOMMU group 7 [8086:a291] 00:1c.1 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #2 (rev f0) IOMMU group 8 [8086:a292] 00:1c.2 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #3 (rev f0) IOMMU group 9 [8086:a294] 00:1c.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5 (rev f0) IOMMU group 10 [8086:a297] 00:1c.7 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #8 (rev f0) IOMMU group 11 [8086:a298] 00:1d.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #9 (rev f0) IOMMU group 12 [8086:a2c5] 00:1f.0 ISA bridge: Intel Corporation 200 Series PCH LPC Controller (Z270) [8086:a2a1] 00:1f.2 Memory controller: Intel Corporation 200 Series PCH PMC [8086:a2f0] 00:1f.3 Audio device: Intel Corporation 200 Series PCH HD Audio [8086:a2a3] 00:1f.4 SMBus: Intel Corporation 200 Series PCH SMBus Controller [8086:15d6] 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (5) I219-V IOMMU group 13 [8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) IOMMU group 14 [1102:0012] 05:00.0 Audio device: Creative Labs Sound Core3D [Sound Blaster Recon3D / Z-Series] (rev 01) IOMMU group 15 [1b21:2142] 06:00.0 USB controller: ASMedia Technology Inc. Device 2142 IOMMU group 16 [1912:0014] 07:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
December 22, 20178 yr Author Well, this is embarrassing... The problem was my integrated graphics output was not connected to a monitor. So when UnRAID started, the motherboard output all my video via my NVidia card. When starting the Window VM, it didn't like that the card I was trying to passthrough was in use and so nothing started up. I realized this after restarting unraid and noticing my screen turning off after starting my VM (if it was connected to the integrated graphics, that wouldn't happen). Remember the first two questions of IT, kids: Have you tried turning it off and on again? Are you sure it's plugged in? I'm glad my stupidity was root cause here!
December 23, 20178 yr glad it works now for you. still strange that a plugged in cable can cause this. I wonder whats happening behind the scenes. Why would unraid claim the dGPU. Would the command lspci -v -nn -s xx:xx.x (stands for the GPU bus:slot.function) show that a driver is assigned (by unraid) to the GPU ?
December 23, 20178 yr 8 hours ago, unrateable said: glad it works now for you. still strange that a plugged in cable can cause this. I wonder whats happening behind the scenes. Why would unraid claim the dGPU. Would the command lspci -v -nn -s xx:xx.x (stands for the GPU bus:slot.function) show that a driver is assigned (by unraid) to the GPU ? This is most likely a bios issue. If nothing is connected to the output, the bios chooses a card with a connected monitor as the primary adapter. There will not be a driver assigned by unraid unless you boot gui mode.
Archived
This topic is now archived and is closed to further replies.