wesman Posted October 16, 2019 Share Posted October 16, 2019 Hi, I am hoping I can get a little help with a Graphic Card Passthrough. Hardware: Asus Crosshair 8 AM4 Ryzen 3700 (No dedicated graphics) RTX 2070 Super (Only Graphics Card) I have setup the Windows 10 VM with VNC and it works fine. Attempting to add the Graphic card however causes this error (MORE BELOW IMAGE) Sorry the image is so large. I have confirmed that its in its own IOMMU group Also attempted to add the BIOS from https://www.techpowerup.com and Allow Unsafe Interrupts but still I get the same error. IF i change ACS override (reboot) and try again, when launching the VM, the entire system hangs requiring a full (push the button) reset. I upgraded the 6.8.0 RC1 to see if there way any help there, but no. Any ideas? Quote Link to comment
jpowell8672 Posted October 17, 2019 Share Posted October 17, 2019 Did you remove the header from the bios? Quote Link to comment
wesman Posted October 17, 2019 Author Share Posted October 17, 2019 Indeed, I edited the ROM and moved the header. but no joy, same effect, the whole system crashes Quote Link to comment
jpowell8672 Posted October 17, 2019 Share Posted October 17, 2019 Do you have HVM & IOMMU enabled in the bios and not on auto which makes a difference? Do you have the latest bios? Newest bios agesa have caused problems with passthrough. Try another video bios close to your card or dump your actual bios. Try multifunction acs. What is machine type & bios type set to in the vm config? Quote Link to comment
Squid Posted October 17, 2019 Share Posted October 17, 2019 You're not passing through the USB controller or the serial that's also on the card (why your passthrough isn't viable). Quote Link to comment
wesman Posted October 17, 2019 Author Share Posted October 17, 2019 45 minutes ago, Squid said: You're not passing through the USB controller or the serial that's also on the card (why your passthrough isn't viable). I did see the other two, but didn't know how to pass them through. There is a "Other PCI Devices" but it was not editable, do I have to edit the XML and if so, what would that syntax look like? Quote Link to comment
jpowell8672 Posted October 17, 2019 Share Posted October 17, 2019 1 hour ago, wesman said: I did see the other two, but didn't know how to pass them through. There is a "Other PCI Devices" but it was not editable, do I have to edit the XML and if so, what would that syntax look like? I have Nvidia RTX card also and I just set ACS to both which split them into there own seperate iommu groups and has worked great. Quote Link to comment
jpowell8672 Posted October 17, 2019 Share Posted October 17, 2019 1 hour ago, wesman said: I did see the other two, but didn't know how to pass them through. There is a "Other PCI Devices" but it was not editable, do I have to edit the XML and if so, what would that syntax look like? Also you can pass the usb through by adding vfio-pci.ids=10de:1ad8,10de:1ad9 to your syslinux config for them to show up in the VM template in unRAID. Quote Link to comment
wesman Posted October 17, 2019 Author Share Posted October 17, 2019 9 hours ago, jpowell8672 said: I have Nvidia RTX card also and I just set ACS to both which split them into there own seperate iommu groups and has worked great. I did try that but it made no difference. @jpowell8672 with regards to passing through, is this the correct syntax? <qemu:commandline> <<<----- add the two device in this group on the XML File <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=0b:00.2,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=0b:00.3,bus=root.1,addr=00.1'/> </qemu:commandline> or do I need to use the 10de:1ad8 and 10de:1ad9 numbers? Quote Link to comment
jpowell8672 Posted October 17, 2019 Share Posted October 17, 2019 Not your xml, Unraid's Syslinux configuration. Goto MAIN>Flash - Syslinux configuration - Unraid OS - Add vfio-pci.ids=10de:1ad8,10de:1ad9 Quote Link to comment
wesman Posted October 17, 2019 Author Share Posted October 17, 2019 (edited) Update: @jpowell8672 I was wrong about it working, the Window 10 VM got a green Play Icon and I assumed it was working but it wasn't. I think I had a black screen, I followed the second video with updating the ROM, but still black screen. I looked at the VM Log and was seeing this.. 0000:0b:00.0 is my Nvidia card 2019-10-18T02:02:08.239896Z qemu-system-x86_64: vfio_region_write(0000:0b:00.0:region1+0x12a738, 0x0,8) failed: Device or resource busy Edited October 18, 2019 by wesman Quote Link to comment
wesman Posted October 18, 2019 Author Share Posted October 18, 2019 3 hours ago, wesman said: Update: @jpowell8672 I was wrong about it working, the Window 10 VM got a green Play Icon and I assumed it was working but it wasn't. I think I had a black screen, I followed the second video with updating the ROM, but still black screen. I looked at the VM Log and was seeing this.. 0000:0b:00.0 is my Nvidia card 2019-10-18T02:02:08.239896Z qemu-system-x86_64: vfio_region_write(0000:0b:00.0:region1+0x12a738, 0x0,8) failed: Device or resource busy Any thing else I can try? is this only an issue if the system has a only one graphic card (primary)? Quote Link to comment
jpowell8672 Posted October 18, 2019 Share Posted October 18, 2019 (edited) 35 minutes ago, wesman said: Any thing else I can try? is this only an issue if the system has a only one graphic card (primary)? Can't help you with so little info, post your current vm xml, screenshot of your vm template, syslinux config, Tools>System Devices & Tools>Diagnostics. Edited October 18, 2019 by jpowell8672 add system devices Quote Link to comment
jpowell8672 Posted October 18, 2019 Share Posted October 18, 2019 On 10/16/2019 at 9:54 PM, jpowell8672 said: Do you have HVM & IOMMU enabled in the bios and not on auto which makes a difference? Do you have the latest bios? Newest bios agesa have caused problems with passthrough. Try another video bios close to your card or dump your actual bios. Try multifunction acs. What is machine type & bios type set to in the vm config? You never even answered these questions to be able to properly help you. Quote Link to comment
Sleepyme Posted October 18, 2019 Share Posted October 18, 2019 On 10/16/2019 at 8:55 PM, wesman said: Hi, I am hoping I can get a little help with a Graphic Card Passthrough. Hardware: Asus Crosshair 8 AM4 Ryzen 3700 (No dedicated graphics) RTX 2070 Super (Only Graphics Card) I have setup the Windows 10 VM with VNC and it works fine. Attempting to add the Graphic card however causes this error (MORE BELOW IMAGE) Sorry the image is so large. I have confirmed that its in its own IOMMU group Also attempted to add the BIOS from https://www.techpowerup.com and Allow Unsafe Interrupts but still I get the same error. IF i change ACS override (reboot) and try again, when launching the VM, the entire system hangs requiring a full (push the button) reset. I upgraded the 6.8.0 RC1 to see if there way any help there, but no. Any ideas? You might have a similiar issue as me. When i boot into my vm. It doesn't hang. Instead i get a black screen for about 10 minutes and the screen comes on. During that time cpu usuage is at 100% but only one 1 core. Quote Link to comment
wesman Posted October 18, 2019 Author Share Posted October 18, 2019 11 hours ago, jpowell8672 said: You never even answered these questions to be able to properly help you. Hi @jpowell8672 Thanks for all your assistance, below is the information you asked for. The last thing I tried was to Dump my own BIOS from the Video Card (GPU-Z), mod it, and use that in the passthrough. Yes, I downgraded the BIOS to each previous level and tried on it, currently, I am down to the release BIOS for the this board ASUS Crosshair Hero 8 Yes, i made sure the BIOS Virtualization was set to enabled, not just "auto" **** SYSLINUX default menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label Unraid OS menu default kernel /bzimage append initrd=/bzroot vfio-pci.ids=144d:a802,10de:1ad8,10de:1ad9,10ec:8125 label Unraid OS GUI Mode kernel /bzimage append initrd=/bzroot,/bzroot-gui label Unraid OS Safe Mode (no plugins, no GUI) kernel /bzimage append initrd=/bzroot unraidsafemode label Unraid OS GUI Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot,/bzroot-gui unraidsafemode label Memtest86+ kernel /memtest *** VM Config <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Windows 10 DualBoot</name> <uuid>52af3a68-d904-87f5-3ac3-04d9E4573ac2</uuid> <description>Dual Bootable Bare Metal and via Unraid</description> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>24641536</memory> <currentMemory unit='KiB'>24641536</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>14</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='9'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='10'/> <vcpupin vcpu='4' cpuset='3'/> <vcpupin vcpu='5' cpuset='11'/> <vcpupin vcpu='6' cpuset='4'/> <vcpupin vcpu='7' cpuset='12'/> <vcpupin vcpu='8' cpuset='5'/> <vcpupin vcpu='9' cpuset='13'/> <vcpupin vcpu='10' cpuset='6'/> <vcpupin vcpu='11' cpuset='14'/> <vcpupin vcpu='12' cpuset='7'/> <vcpupin vcpu='13' cpuset='15'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-3.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/52af3a68-d904-87f5-3ac3-04d9f5873ac2_VARS-pure-efi.fd</nvram> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='14' threads='1'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:cc:63:4b'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/domains/ASUSRTX2070SuperROM_ORG_Modded.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc045'/> </source> <address type='usb' bus='0' port='2'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x0518'/> <product id='0x0001'/> </source> <address type='usb' bus='0' port='3'/> </hostdev> <memballoon model='none'/> </devices> </domain> valyria-diagnostics-20191018-1337.zip Quote Link to comment
wesman Posted October 18, 2019 Author Share Posted October 18, 2019 5 hours ago, Sleepyme said: You might have a similiar issue as me. When i boot into my vm. It doesn't hang. Instead i get a black screen for about 10 minutes and the screen comes on. During that time cpu usuage is at 100% but only one 1 core. Interesting, I tried waiting 10 minutes, went and ate breakfast and came back, the Windows VM was in a Paused state, trying to resume it just made it move to a Stopped state with the Error that CONT was not an option. Quote Link to comment
wesman Posted October 18, 2019 Author Share Posted October 18, 2019 Does anyone know, is this completely related to the fact this is the Primary and Only Graphics card in the system? or is this a Ryzen sort of issue? Quote Link to comment
Tise Posted November 4, 2019 Share Posted November 4, 2019 (edited) I Think I had a similar problem with my 980ti, that it was busy. What I did was to put this in my sys linux conf under the unraid os boot option. append vfio-pci.ids=mygf:xcard What this did was half way through the boot of unraid my gfx card was disabled from being used by unraid and it was no longer busy for the vm. Edited November 4, 2019 by Tise Quote Link to comment
wesman Posted November 5, 2019 Author Share Posted November 5, 2019 @Tise I did that too, no joy. although, I think my issues are more to due with the newness of X570, its got a patch via third party, but, unstable https://forums.unraid.net/topic/82625-kernelcustom-kernel-build-with-some-useful-patches-20191018-update-support-for-680rc1/ Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.