Capt.Insano Posted January 2, 2022 Share Posted January 2, 2022 (edited) Firstly: Happy New Year unRAID community I would HUGELY appreciate any help that could be offered here: I have been reading all the AMD passthrough threads that I could find but I have gotten incredibly confused by conflicting information and information that is possibly no longer valid as per versions 6.9+. I am swapping the GPU on a Win10 VM from a Nvidia 1050Ti to an AMD RX580 (Gigabyte Aorus RX580 8GB). I am having a lot of difficulty in getting the RX580 reliably passed through: I can get it to boot in the VM with a 800x600 resolution and green/blue artifacts on the screen and I am sometimes get the AMD drivers (latest from AMD website) to correctly install but it inevitably crashes and then reboots to show an "Error Code 43" in the device manager for the card. I have tried completely removing all driver components with DDU in safe mode between attempts but still struggling greatly. I recently had it working but then after a reboot of the server it reverted to Code 43 again! Artifacts disappear in safemode so I am fairly confident they are related to drivers/passthrough. Card is known working as was removed from eGPU enclosure. My Setup: M/B: Gigabyte Technology Co., Ltd. C246-WU4-CF Version x.x - s/n: Default string BIOS: American Megatrends Inc. Version F6g. Dated: 09/16/2020 CPU: Intel® Core™ i9-9900 CPU @ 3.10GHz HVM: Enabled IOMMU: Enabled Cache: 512 KiB, 2 MB, 16 MB Memory: 64 GiB DDR4 (max. installable capacity 64 GiB) Network: bond0: fault-tolerance (active-backup), mtu 1500 eth0: 1000 Mbps, full duplex, mtu 1500 eth1: 1000 Mbps, full duplex, mtu 1500 Kernel: Linux 5.14.15-Unraid x86_64 I have CSM disabled in BIOS as per some posts I read here. I have the AMD Vendor Reset Plugin from @ich777 installed: My System VM Manager Settings are as follows: I have the following VFIO bindings: VM Log: -boot strict=on \ -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \ -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \ -device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \ -device pcie-root-port,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 \ -device pcie-root-port,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 \ -device pcie-root-port,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x5 \ -device pcie-root-port,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x6 \ -device pcie-root-port,port=0xf,chassis=8,id=pci.8,bus=pcie.0,addr=0x1.0x7 \ -device pcie-pci-bridge,id=pci.9,bus=pci.1,addr=0x0 \ -device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x7.0x7 \ -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x7 \ -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x7.0x1 \ -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x7.0x2 \ -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \ -blockdev '{"driver":"file","filename":"/mnt/disks/CT2000MX500SSD1_2111E589AB92/domains/Windows 10 (BI)/vdisk_new.img","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":null}' \ -device virtio-blk-pci,bus=pci.4,addr=0x0,drive=libvirt-2-format,id=virtio-disk2,bootindex=1,write-cache=on \ -blockdev '{"driver":"host_device","filename":"/dev/disk/by-id/ata-WDC_WD20PURZ-85AKKY0_WD-WX42D609RS31","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \ -device virtio-blk-pci,bus=pci.5,addr=0x0,drive=libvirt-1-format,id=virtio-disk3,write-cache=on \ -netdev tap,fd=39,id=hostnet0 \ -device virtio-net,netdev=hostnet0,id=net0,mac=30:11:d2:80:23:6a,bus=pci.3,addr=0x0 \ -chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0 \ -chardev socket,id=charchannel0,fd=40,server=on,wait=off \ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \ -device usb-tablet,id=input0,bus=usb.0,port=3 \ -audiodev id=audio1,driver=none \ -device vfio-pci,host=0000:01:00.0,id=hostdev0,bus=pci.6,multifunction=on,addr=0x0,romfile=/mnt/user/Files/VMs/vbios/RX580.Fast.rom \ -device vfio-pci,host=0000:01:00.1,id=hostdev1,bus=pci.6,addr=0x0.0x1 \ -device usb-host,hostdevice=/dev/bus/usb/001/006,id=hostdev2,bus=usb.0,port=1 \ -device usb-host,hostdevice=/dev/bus/usb/001/003,id=hostdev3,bus=usb.0,port=2 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/2 (label charserial0) 2022-01-01T23:52:10.162457Z qemu-system-x86_64: vfio: Cannot reset device 0000:01:00.1, no available reset mechanism. 2022-01-01T23:52:10.167466Z qemu-system-x86_64: vfio: Cannot reset device 0000:01:00.1, no available reset mechanism. 2022-01-01T23:59:38.988500Z qemu-system-x86_64: vfio: Cannot reset device 0000:01:00.1, no available reset mechanism. 2022-01-01T23:59:38.994580Z qemu-system-x86_64: vfio: Cannot reset device 0000:01:00.1, no available reset mechanism. VM XML: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Windows 10 BI XML_Edits</name> <uuid>33eb8092-98b7-7fb8-c0c4-219f958f4b13</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>6</vcpu> <cputune> <vcpupin vcpu='0' cpuset='5'/> <vcpupin vcpu='1' cpuset='13'/> <vcpupin vcpu='2' cpuset='6'/> <vcpupin vcpu='3' cpuset='14'/> <vcpupin vcpu='4' cpuset='7'/> <vcpupin vcpu='5' cpuset='15'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-6.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/33eb8092-98b7-7fb8-c0c4-219f958f4b13_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='3' threads='2'/> <cache mode='passthrough'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='writeback'/> <source file='/mnt/disks/CT2000MX500SSD1_2111E589AB92/domains/Windows 10 (BI)/vdisk_new.img'/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source dev='/dev/disk/by-id/ata-WDC_WD20PURZ-85AKKY0_WD-WX42D609RS31'/> <target dev='hdd' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xd'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xe'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xf'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='9' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <interface type='bridge'> <mac address='30:11:d2:80:23:6a'/> <source bridge='br0'/> <model type='virtio-net'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='3'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/Files/VMs/vbios/RX580.Fast.rom'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x413c'/> <product id='0x2003'/> </source> <address type='usb' bus='0' port='1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x413c'/> <product id='0x3012'/> </source> <address type='usb' bus='0' port='2'/> </hostdev> <memballoon model='none'/> </devices> </domain> NOTE: I have tried it both with and without vbios files as I have read conflicting reports of their importance for AMD cards. I have ran a vbios file that I dumped myself via @SpaceInvaderOne script and have ran ones downloaded from techpowerup.com There has been no difference in success either way. I have nothing special in my syslinux.cfg: kernel /bzimage append isolcpus=5-7,13-15 pcie_acs_override=downstream initrd=/bzroot Diagnostics attached also: Thanks so much for any advice! tower-diagnostics-20220102-0013.zip Edited January 2, 2022 by Capt.Insano Update title Quote Link to comment
ich777 Posted January 2, 2022 Share Posted January 2, 2022 5 hours ago, Capt.Insano said: Artifacts disappear in safemode so I am fairly confident they are related to drivers/passthrough. Card is known working as was removed from eGPU enclosure. Can you please verify that the card is working in another computer, also try to install the drivers for the card and put a 3D load on it for about 5 to 10 minutes and see if you got no artifacts. Usually this is caused when the card is defective but in your case this can also be an issue with the Passthrough itself but I would at least verify that it works in another computer (without the enclosure). Quote Link to comment
Capt.Insano Posted January 2, 2022 Author Share Posted January 2, 2022 Seems you may be right! 😭 Booted the same unraid hardware with a Windows 10 drive and also got Code 43 errors after driver installation. What a balls! GPUs are just so expensive at the moment. Thanks for your advice Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.