Azarog Posted September 26, 2021 Share Posted September 26, 2021 (edited) Hello, I would like to use my GTX660Ti as an output for a windows VM. M/B: Supermicro H11SSL-i Version 2.00 - s/n: WM204S602455 BIOS: American Megatrends Inc. Version 2.1. Dated: 02/21/2020 CPU: AMD EPYC 7401P 24-Core @ 2000 MHz HVM: Enabled IOMMU: Enabled Memory: 64 GiB DDR4 Multi-bit ECC (max. installable capacity 2048 GiB) Kernel: Linux 5.10.28-Unraid x86_64 Everything starts smoothly and i'm prompted by the windows installation thingy in VNC But nothing on the real screen connected to the GPU. Here is my VM xml : <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>660 Ti</name> <uuid>a517245a-6a95-5b5a-e9c5-2a323c221f6d</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='24'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='26'/> <vcpupin vcpu='4' cpuset='4'/> <vcpupin vcpu='5' cpuset='28'/> <vcpupin vcpu='6' cpuset='6'/> <vcpupin vcpu='7' cpuset='30'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-5.0'>hvm</type> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='4' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/vms/660 Ti/vdisk1.img'/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/Windows.iso'/> <target dev='hda' bus='sata'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x15'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:31:0f:a7'/> <source bridge='br0'/> <model type='virtio-net'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='3'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x21' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/rom/NVIDIA.GTX660Ti.2048.120709.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x21' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x05ac'/> <product id='0x024f'/> </source> <address type='usb' bus='0' port='1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x1532'/> <product id='0x0043'/> </source> <address type='usb' bus='0' port='2'/> </hostdev> <memballoon model='none'/> </devices> </domain> As you can see i use the "official" rom but i also tried the SpaceInvaderOne's method from his video using the one from the card itself, nothing worked yet. Hope we can find a way to have it work ! Thanks PS: This is the only PCIe device (alongisde the NVMe on the right of the PCIe slot under the one the GPU is connected) Edited September 26, 2021 by Azarog Quote Link to comment
ghost82 Posted September 26, 2021 Share Posted September 26, 2021 Use vnc OR gpu passthrough, I don't think you can use both together. Quote Link to comment
Azarog Posted September 26, 2021 Author Share Posted September 26, 2021 Ok, since i posted i managed to have both working (vnc and gpu passtrough) for ubuntu ... Perhaps it is something related to the windows VM not being activated ? Quote Link to comment
Azarog Posted September 26, 2021 Author Share Posted September 26, 2021 Ok so everything is working nicely under ubuntu (even with no rom for the GPU) On the windows VM, the card is recognized (when i go through VNC) but nothing comes out of the HDMI port Quote Link to comment
JonathanM Posted September 26, 2021 Share Posted September 26, 2021 1 hour ago, Azarog said: nothing comes out of the HDMI port What about the other ports on the card? Quote Link to comment
Azarog Posted September 27, 2021 Author Share Posted September 27, 2021 (edited) The HDMI port is not faulty as it works well with the ubuntu VM. As of yet, i can't test the display port of the card. (i don't have good cables) Edited September 27, 2021 by Azarog mistakes Quote Link to comment
JonathanM Posted September 27, 2021 Share Posted September 27, 2021 5 hours ago, Azarog said: The HDMI port is not faulty as it works well with the ubuntu VM. I'm not saying it's faulty, but possibly not active with windows until the drivers are loaded. Quote Link to comment
Azarog Posted September 27, 2021 Author Share Posted September 27, 2021 (edited) I'm confused at which step the drivers need to be loaded ? Here is what i get BEFORE trying to install drivers from NVIDIA wesite : And this is AFTER installing the drivers from NVIDIA website: Even with everything looking alright nothing comes out of HDMI ( or at least reaches my screen) GPU-Z showing this : Edited September 27, 2021 by Azarog adding after pics and gpuz Quote Link to comment
Azarog Posted September 27, 2021 Author Share Posted September 27, 2021 (edited) And for what it's worth, the sound setting finds the monitor to which the GTX is connected to. So, windows detected the monitor and even if no sound come through, it grabbed the name of the monitor. And these are the VMs settings: Edited September 27, 2021 by Azarog adding vm settings Quote Link to comment
runamuk Posted October 2, 2021 Share Posted October 2, 2021 (edited) Do you need your PCIe ACS override to Multi-function? If so try always using "Both" as the override; I have always found better results on those machines. Have you tried using a "Graphics ROM BIOS" for the windows VM. Also it recommended you install your own VNC on a Windows system; not try to load the VNC graphic controller and NVidia at the same time. Edited October 2, 2021 by runamuk Quote Link to comment
Azarog Posted October 3, 2021 Author Share Posted October 3, 2021 I tried all the combination and settled with multi function, but it is still not working. I also tried only the NVIDIA only ... Where should i specify "Graphics ROM BIOS" ? Quote Link to comment
ghost82 Posted October 4, 2021 Share Posted October 4, 2021 NVIDIA only, no vnc in xml This: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x21' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/rom/NVIDIA.GTX660Ti.2048.120709.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x21' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> Should be replaced with this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x21' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/rom/NVIDIA.GTX660Ti.2048.120709.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x21' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </hostdev> Make sure NVIDIA.GTX660Ti.2048.120709.rom is dumped from your gpu and not downloaded elsewhere. If it's not working after these changes attach unraid diagnostics file. Quote Link to comment
ghost82 Posted October 4, 2021 Share Posted October 4, 2021 Moreover You are using seabios, I would change to ovmf, since the 660Ti is uefi capable. Quote Link to comment
Azarog Posted October 4, 2021 Author Share Posted October 4, 2021 Ok, i'll try everything you said. But i have few questions : With only nvidia, how do i know why it failed (if it does) i tried before with omvf, i switched to seabios, bc a friend of mine managed to have a nvidia card working like this. why do i have to dump NVIDIA.GTX660Ti.2048.120709.rom if it works on linux ? Thank you for your leads, i'll try them when i get back home. Quote Link to comment
ghost82 Posted October 4, 2021 Share Posted October 4, 2021 2 hours ago, Azarog said: With only nvidia, how do i know why it failed (if it does) By looking at the syslog and trying to understand why it's not working, that's why I asked for diagnostics if it will not work. 2 hours ago, Azarog said: i tried before with omvf, i switched to seabios, bc a friend of mine managed to have a nvidia card working like this. Unless you are booting unraid in legacy bios mode, which may cause some issue with some uefi gpus, I don't see why not using it; uefi is more recent, so let's use it! 2 hours ago, Azarog said: why do i have to dump NVIDIA.GTX660Ti.2048.120709.rom if it works on linux ? Linux is different than linux; it may work or it may not, if you dump your own bios you are eliminating one possible cause. Since it's the only gpu in the system the main os could use it (efifb, vesafb), or it may not be possible to attach to vfio "on the fly" if not isolated at boot, let's try one step at a time. Quote Link to comment
Azarog Posted October 8, 2021 Author Share Posted October 8, 2021 Ok, so. I'll try one more time and post diagnostic afterward but : Quote Unless you are booting unraid in legacy bios mode, which may cause some issue with some uefi gpus, I don't see why not using it; uefi is more recent, so let's use it! I cannot boot from UEFI, i got stuck with the startup.sh script error thing. Quote Since it's the only gpu in the system the main os could use it (efifb, vesafb) I fear this is the issue ... but sometimes when i start with VNC and the GPU, i have both screen (client and GPU output) working. And right after the install finished when i keep only the gpu nothing shows up and logs are ok. When i go back to VNC only, i have the "guest display not initialized (yet)" error which is pissing me of alot. Quote Link to comment
Azarog Posted October 8, 2021 Author Share Posted October 8, 2021 I can provide some updates after fiddling the whole afternoon: The problem is too big, i'm starting by solving the linux vm one. My card (i switched the 660ti for some lower consumption nvidia card, GT216) and they only support legacy. So switching to UEFI for unraid is out of the picture. The (linux) vm is starting but nothing comes out of the HDMI, the logs are not suspicious. (i've added the diagnostic) When i switch back to VNC i have the "guest display not initialized (yet)" and it's driving me crazy. I tried solutions from this but none worked (either changing memory size or recreating vm) I couldn't find how to install qm though. kamuy-huci-diagnostics-20211008-1807.rar Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.