btrcp2000 Posted November 19, 2019 Share Posted November 19, 2019 (edited) Asking because so far it has proven impossible. I have a few functioning VMs (aside from occasional shutdown attempts that crash the entire server, but I have fought that for two years and just live with it now). Windows10, Windows Server 2012 r2, and Ubuntu. Using AMD Radeon HD5450 cards as I had to give up on Nvidia due to the code43 issue. The first one was plug n play and works fine in the Win10 VM, which is a daily driver in the kitchen. The second gives various errors upon VM restart, which almost always kills UNRAID. Most recent attempt was Ubuntu where I get "XML error: Invalid PCI address 0000:02:02.0. slot must be <= 0" Second GPU is to be able to have a terminal at the server itself, since I have spare GPUs and empty slots. It will be tough to post diags because screwing with it usually brings everything down forcing a hard reboot with the power button, so I thought I would ask if this is even possible first. Edited November 19, 2019 by btrcp2000 Quote Link to comment
bastl Posted November 19, 2019 Share Posted November 19, 2019 @btrcp2000 GPU passthrough isn't a "always working, plug it in and it works out of the box" thingy. Most important thing is that the device you wanna passthrough is in it's own IOMMU group. Check under system devices how your devices are grouped together or not. Worth to mention, the PCI slots are wired differently. Some are directly attached to the CPU. In most scenarios you will have the GPU in it's own group and passthrough should work. Some slots are are connected to the chipset and share the connection to the CPU with other devices like USB controllers, sata controllers, network cards onboard audio etc. In this scenario the devices are grouped together and can't be passed through that easily. You can select them, sure, but what will happen is by starting up a VM with such a device is, that something else in the server breaks, freeze or it will crash. Only thing you can try is to use the ACS Override Patch to split the groups. For most users this is the only way to get their devices into a VM especially in systems with multiple PCI devices and VMs. If you can't get the device you wanna passthrough separated by this and selecting another slot also doesn't help there is nothing else you can do. Quote Link to comment
btrcp2000 Posted November 19, 2019 Author Share Posted November 19, 2019 Come to think of it, it's possible that many of the VM problems I've had were related to devices in that slot. The non-working card is group 40 below, o it's in its own. I had heard of the override patch but never used it, will it make a difference if its already in its own group? Quote Link to comment
bastl Posted November 19, 2019 Share Posted November 19, 2019 12 minutes ago, btrcp2000 said: will it make a difference if its already in its own group? No. It only helps if you have like I described earlier multiple different devices in the same group. In your example group 39 is the same device. You have to pass both to the VM to make it work. Not passing through the audio part can lead to issues with the VM. Group 40 in your screen is this another AMD GPU? Do you have 2 exact same models in the server or are they different? I wonder why it's not showing an HDMI audio controller as well. If it's the same generation it should show similiar results as group 39. Without ACS the audio part should show up in the same group as the card. Is there any other group maybe with the HDMI audio controller? 9 hours ago, btrcp2000 said: "XML error: Invalid PCI address 0000:02:02.0. slot must be <= 0" This error looks a bit weird to me. Check your device list for "02:02.0" if any such a device exists. The dot behind the last 0 confuses me a bit. Maybe the formating is wrong in your XML. Keep in mind, adding or removing devices or changing something in the BIOS or updating it can change the adressing. Quote Link to comment
btrcp2000 Posted November 19, 2019 Author Share Posted November 19, 2019 (edited) Yes, they were both the same gpu chip but different cards. the second one is dual DVI only, no hdmi therefore no audio. In any case I first shuffled things around with that card in a different slot and got the same result. Then I tried to go back to the nvidia gt710 card in the new slot, passing through both its video and audio addresses and it outputs video but uses the microsoft basic display driver. nvidia driver installer doesn't see it. That XML error only comes up when I try to pass the card to the Ubuntu VM, no devices match those numbers. Here are the new groupings: Here is my passthru line from the flash drive. All the other devices are working fine in their respective VMs: kernel /bzimage append vfio-pci.ids=1b21:1142,1912:0014,1b73:1100,10de:128b,10de:0e0f initrd=/bzroot Diags attached unraid-diagnostics-20191119-1442.zip Edited November 19, 2019 by btrcp2000 Quote Link to comment
btrcp2000 Posted November 20, 2019 Author Share Posted November 20, 2019 The slot doesn't seem to be the issue. Tried moving things around and booted everything with nothing there. Two different cards (radeon hd5450 and nvidia gt710) in the second VM won't come through as anything other than microsoft basic display driver. Getting Code 43 errors, with an without Hyper-V enabled I'm out of ideas, anyone else? Here is the XML <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='8'> <name>Windows Server 2012</name> <uuid>c0fbc9fe-4a5e-3ac0-0d94-18da2e6002c0</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows Server 2012" icon="windows.png" os="windows2012"/> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='18'/> <vcpupin vcpu='1' cpuset='38'/> <vcpupin vcpu='2' cpuset='19'/> <vcpupin vcpu='3' cpuset='39'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-3.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/c0fbc9fe-4a5e-3ac0-0d94-18da2e6002c0_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='2' threads='2'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/Windows Server 2012/vdisk1.img'/> <backingStore/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/9600.16384.WINBLUE_RTM.130821-1623_X64FRE_SERVER_SOLUTION_EN-US-IRM_SSSO_X64FRE_EN-US_DV5.ISO'/> <backingStore/> <target dev='hda' bus='ide'/> <readonly/> <boot order='2'/> <alias name='ide0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.126-2.iso'/> <backingStore/> <target dev='hdb' bus='ide'/> <readonly/> <alias name='ide0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:fc:d9:61'/> <source bridge='br0'/> <target dev='vnet2'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/2'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-8-Windows Server 2012/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x83' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/mnt/user/domains/vbios dumps/MSI.GT710.2048.160112.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x0403'/> <product id='0xf850'/> <address bus='2' device='5'/> </source> <alias name='hostdev3'/> <address type='usb' bus='0' port='2'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x0c16'/> <product id='0x0002'/> <address bus='2' device='6'/> </source> <alias name='hostdev4'/> <address type='usb' bus='0' port='3'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain> Quote Link to comment
btrcp2000 Posted November 21, 2019 Author Share Posted November 21, 2019 Well, this is new, but unfortunately I'm apparently the only one ever to have gotten a code 31 in the entire UNRAID Forum universe. This is the Nvidia GT710 with both audio and video stubbed. Anybody reading this stuff? unraid-diagnostics-20191121-0308.zip Quote Link to comment
bastl Posted November 21, 2019 Share Posted November 21, 2019 @btrcp2000 This is a normal scenario if you have also the VNC adapter connected to the VM. The dedicated GPU shows up as Generic Microsoft Display Adapter and the driver install often fails. Configure remote software like RDP or Teamviewer, make sure you can connect, attach a display to the GPU, remove VNC and test if it works. Quote Link to comment
btrcp2000 Posted November 21, 2019 Author Share Posted November 21, 2019 Yes, I've seen that before but tried again just now for good measure. It outputs video, but uses the Microsoft driver. Mfr GPU driver installs don't recognize it Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.