Bagpuss Posted April 19, 2020 Share Posted April 19, 2020 (edited) Hi All, I've recently build a new Ryzen 3950X system based on the Asus Crosshair VIII Hero (WiFi) motherboard (latest BIOS v1302). System has a Gigabyte RTX2080 OC graphics card, Mellanox ConnectX/2 dual NIC, 2x32GB Corsair Vengeance RBG Pro DIMMS, and 4 x SSD (2 x M.2 and 2 x SATA). I had some random lockup issues, but these are now fixed with the 6.0-beta1. I'm now trying to create a Windows 10 VM where I passthrough my RTX2080 and a USB controller. I initially had some major issues with the VM locking up, but from searching here, I discovered that you can't passthrough the onboard audio and one specific USB controller. I've used Skital's VFIO-PCI plugin (thanks Skital), and have now managed to successfully passthrough just the GPU, so I can now boot into Windows 10. My current config is as follows: IOMMU group 0: [1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 1: [1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 2: [1022:1483] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 3: [1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 4: [1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 5: [1022:1483] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 6: [1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 7: [1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 8: [1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 9: [1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 10: [1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 11: [1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 12: [1022:1484] 00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 13: [1022:1484] 00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 14: [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61) [1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) IOMMU group 15: [1022:1440] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1441] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1442] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1443] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1444] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1445] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1446] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1447] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 IOMMU group 16: [1987:5016] 01:00.0 Non-Volatile memory controller: Phison Electronics Corporation E16 PCIe4 NVMe Controller (rev 01) IOMMU group 17: [1022:57ad] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream IOMMU group 18: [1022:57a3] 03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 19: [1022:57a3] 03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 20: [1022:57a3] 03:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 21: [1022:57a3] 03:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 22: [1022:57a3] 03:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 23: [1022:57a4] 03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:1485] 09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:149c] 09:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 09:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller IOMMU group 24: [1022:57a4] 03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:7901] 0a:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 25: [1022:57a4] 03:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:7901] 0b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 26: [1987:5012] 04:00.0 Non-Volatile memory controller: Phison Electronics Corporation E12 NVMe Controller (rev 01) IOMMU group 27: [15b3:6750] 05:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0) IOMMU group 28: [10ec:8125] 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller IOMMU group 29: [8086:1539] 07:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) IOMMU group 30: [8086:2723] 08:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a) IOMMU group 31: [10de:1e87] 0c:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1) [10de:10f8] 0c:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1) [10de:1ad8] 0c:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1) [10de:1ad9] 0c:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1) IOMMU group 32: [1022:148a] 0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function IOMMU group 33: [1022:1485] 0e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 34: [1022:1486] 0e:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP IOMMU group 35: [1022:149c] 0e:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller IOMMU group 36: [1022:1487] 0e:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller IOMMU group 37: [1022:7901] 0f:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 38: [1022:7901] 10:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) For unRAID boot menu entry, I have: kernel /bzimage append video=efifb:off initrd=/bzroot With VFIO-PCI, I'm currently just passing through the GPU with: BIND=0c:00.0 0c:00.1 0c:00.2 0c:00.3 Output from plugin is as follows: I know that I can't passthrough the 0e:00.0 as this will cause the VM to crash the system with the following errors: Apr 18 15:51:41 Tower kernel: vfio-pci 0000:0e:00.4: not ready 1023ms after FLR; waiting Apr 18 15:51:43 Tower kernel: vfio-pci 0000:0e:00.4: not ready 2047ms after FLR; waiting Apr 18 15:51:46 Tower kernel: vfio-pci 0000:0e:00.4: not ready 4095ms after FLR; waiting Apr 18 15:51:51 Tower kernel: vfio-pci 0000:0e:00.4: not ready 8191ms after FLR; waiting Apr 18 15:52:01 Tower kernel: vfio-pci 0000:0e:00.4: not ready 16383ms after FLR; waiting Apr 18 15:52:18 Tower kernel: vfio-pci 0000:0e:00.4: not ready 32767ms after FLR; waiting Apr 18 15:52:54 Tower kernel: vfio-pci 0000:0e:00.4: not ready 65535ms after FLR; giving up Apr 18 15:52:54 Tower kernel: clocksource: timekeeping watchdog on CPU16: Marking clocksource 'tsc' as unstable because the skew is too large: Apr 18 15:52:54 Tower kernel: clocksource: 'hpet' wd_now: 9510c1c6 wd_last: 9357bec0 mask: ffffffff Apr 18 15:52:54 Tower kernel: clocksource: 'tsc' cs_now: 19b24db3682 cs_last: 19a510945b0 mask: ffffffffffffffff Apr 18 15:52:54 Tower kernel: tsc: Marking TSC unstable due to clocksource watchdog Apr 18 15:52:54 Tower kernel: TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'. Apr 18 15:52:54 Tower kernel: sched_clock: Marking unstable (474034170098, -113600877)<-(474295664568, -375107169) I tried passing through just the 09:00.0 and 09:00.1 devices, but this prevented the unRAID USB stick from being recognised, as it's on 09:00.3. I'm now a bit stuck on how to passthrough the USB that I need. Do I have to use pcie_acs_override? My VM XML config is as follows: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Windows 10</name> <uuid>a0e0b769-3ca4-ea6f-e2b0-d80d9519c029</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>16</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='17'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='18'/> <vcpupin vcpu='4' cpuset='3'/> <vcpupin vcpu='5' cpuset='19'/> <vcpupin vcpu='6' cpuset='4'/> <vcpupin vcpu='7' cpuset='20'/> <vcpupin vcpu='8' cpuset='5'/> <vcpupin vcpu='9' cpuset='21'/> <vcpupin vcpu='10' cpuset='6'/> <vcpupin vcpu='11' cpuset='22'/> <vcpupin vcpu='12' cpuset='7'/> <vcpupin vcpu='13' cpuset='23'/> <vcpupin vcpu='14' cpuset='8'/> <vcpupin vcpu='15' cpuset='24'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/a0e0b769-3ca4-ea6f-e2b0-d80d9519c029_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='8' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/Windows 10/vdisk1.img'/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/Win10_1909_EnglishInternational_x64.iso'/> <target dev='hda' bus='ide'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.173-2.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:01:81:be'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <sound model='ich9'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </sound> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0c' slot='0x00' function='0x0'/> </source> <rom file='/mnt/disk1/isos/TU104.stripped.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0c' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0c' slot='0x00' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0c' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x3'/> </hostdev> <memballoon model='none'/> </devices> </domain> I've mapped out the USB ports, and they look like this: Ultimately, I want to use this as my gaming VM, and so need to passthrough enough USB for the mouse, keyboard and Oculus Rift sensors. Would really appreciate some pointers on where to go next. Thanks, Andy. Edited April 20, 2020 by Bagpuss Quote Link to comment
meep Posted April 19, 2020 Share Posted April 19, 2020 Why pass through 0e.00.0? 0e.00.3 is the usb controller? Quote Link to comment
Bagpuss Posted April 19, 2020 Author Share Posted April 19, 2020 19 minutes ago, meep said: Why pass through 0e.00.0? 0e.00.3 is the usb controller? It is, and according to other posts on here, there is a Linux kernel bug which causes this to fail on X570 platforms. The onboard audio controller is similarly affected. I'm told there is a patch for the audio controller, but I've not seen anything for the USB. In both cases, this isn't yet available in a 5.x kernel for unRAID. Quote Link to comment
meep Posted April 19, 2020 Share Posted April 19, 2020 Hmm, you say you can’t pass through 0e.00.0, and show an error that seems to indicate 0e.00.4. Are you saying you e tried 0e.00.3 and it produces the same error? And this is due to a known kernnal issue on the x570 platform? have you tried acs override? Could you drop in a pcie USB adapter and pass that through? Quote Link to comment
Bagpuss Posted April 19, 2020 Author Share Posted April 19, 2020 10 minutes ago, meep said: Hmm, you say you can’t pass through 0e.00.0, and show an error that seems to indicate 0e.00.4. Are you saying you e tried 0e.00.3 and it produces the same error? And this is due to a known kernnal issue on the x570 platform? have you tried acs override? Could you drop in a pcie USB adapter and pass that through? Check this thread for more details on the problems with X570: I've tried all combinations I could think of, and it wouldn't work. I guess I could get a separate PCI USB adapter, but I'd prefer not to. One of the reasons I went with the Crosshair MB was for the extra USB ports (it's got 12). Quote Link to comment
meep Posted April 19, 2020 Share Posted April 19, 2020 Can you swap your boot device into one of the usb 3.0 ports on 0e. and passthrough all the 09 controllers, or do you have the same issue there? Otherwise it might be a discrete adapter pending a fix for the issue at hand. Quote Link to comment
Bagpuss Posted April 20, 2020 Author Share Posted April 20, 2020 Just a quick message to say that I've solved the problem now. Mapping out the USB ports helped me a lot. I've now moved the unRAID USB stick to the 0e:00 USB controller, and am now able to passthrough both USB controllers on 09:00. I also made an elementary mistake when editing the XML to make the GPU a multifunction device. I put the multifunction='on' in the wrong place. Doh! I'm very happy now. 😀 Just need to work out how to put my Windows install on the Sabrent Rocket NVMe drive now. Any pointers would be much appreciated. Quote Link to comment
rachid596 Posted April 30, 2020 Share Posted April 30, 2020 (edited) On 4/20/2020 at 2:37 PM, Bagpuss said: Just a quick message to say that I've solved the problem now. Mapping out the USB ports helped me a lot. I've now moved the unRAID USB stick to the 0e:00 USB controller, and am now able to passthrough both USB controllers on 09:00. I also made an elementary mistake when editing the XML to make the GPU a multifunction device. I put the multifunction='on' in the wrong place. Doh! I'm very happy now. 😀 Just need to work out how to put my Windows install on the Sabrent Rocket NVMe drive now. Any pointers would be much appreciated. Hello, i need some help can you send a screenshot of your VFIO-PCI Config to know what usb do you passthrough. Thank you very much. edit: it's fine now Edited June 7, 2020 by rachid596 Quote Link to comment
ssinseeme Posted June 28, 2020 Share Posted June 28, 2020 On 4/30/2020 at 2:33 PM, rachid596 said: Hello, i need some help can you send a screenshot of your VFIO-PCI Config to know what usb do you passthrough. Thank you very much. edit: it's fine now Hey brother. Me again. Before I post this issue about USB pass through. I see that is resolved but I can’t get it to work. With the new option to pass thought with 8.9 update I still get my Unraid crash when I pass thought USB controller on a VM. What is your work around? Quote Link to comment
rachid596 Posted June 28, 2020 Share Posted June 28, 2020 What motherboard do you have ? Envoyé de mon HD1913 en utilisant Tapatalk Quote Link to comment
ssinseeme Posted June 28, 2020 Share Posted June 28, 2020 Asus Rog Crosshair VIII Quote Link to comment
rachid596 Posted June 28, 2020 Share Posted June 28, 2020 59 minutes ago, ssinseeme said: Asus Rog Crosshair VIII Look your MP. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.