Passthrough USB Controller hard locks unraid


Xerby

Recommended Posts

Hi, 

 

I am on my extended trial for unraid and one thing that's stopping me from buying is trying to pass through a USB Controller.

 

So far things work well when it works, Windows 10 VM has passed through GPU successfully, when passing through just USB devices works fine but the issue is when USB devices disconnect and then have go through the web interface to re-attach the device or reboot the vm. 

 

System specs: 

M/B: Gigabyte Technology Co., Ltd. X570 GAMING X Version x.x  (latest bios)

BIOS: American Megatrends Inc. Version F11. Dated: 12/06/2019

CPU: AMD Ryzen 9 3900X 12-Core @ 3800 MHz

HVM: Enabled

IOMMU: Enabled

Cache: 768 KiB, 6144 KiB, 65536 KiB

Memory: 32 GiB DDR4

 

IOMMU group 0:	[1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
[1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
[1022:1483] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
[1bb1:5016] 01:00.0 Non-Volatile memory controller: Seagate Technology PLC Device 5016 (rev 01)
[1022:57ad] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream
[1022:57a3] 03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
[1022:57a3] 03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
[1022:57a3] 03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
[1022:57a3] 03:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
[1022:57a4] 03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
[1022:57a4] 03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
[1022:57a4] 03:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
[144d:a808] 04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
[10de:13bb] 05:00.0 VGA compatible controller: NVIDIA Corporation GM107GL [Quadro K620] (rev a2)
[10de:0fbc] 05:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)
[10ec:8168] 06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16)
[1b21:0612] 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
[1022:1485] 08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
[1022:149c] 08:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
[1022:149c] 08:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
[1022:7901] 09:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
[1022:7901] 0a:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
IOMMU group 1:	[1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 2:	[1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
[1022:1483] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
[10de:1187] 0b:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 760] (rev a1)
[10de:0e0a] 0b:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
IOMMU group 3:	[1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 4:	[1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 5:	[1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 6:	[1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 7:	[1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 8:	[1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 9:	[1022:1484] 00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 10:	[1022:1484] 00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 11:	[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 12:	[1022:1440] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0
[1022:1441] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1
[1022:1442] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2
[1022:1443] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3
[1022:1444] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4
[1022:1445] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5
[1022:1446] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6
[1022:1447] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7
IOMMU group 13:	[1022:148a] 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 14:	[1022:1485] 0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU group 15:	[1022:1486] 0d:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
IOMMU group 16:	[1022:149c] 0d:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
IOMMU group 17:	[1022:1487] 0d:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
IOMMU group 18:	[1022:7901] 0e:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
IOMMU group 19:	[1022:7901] 0f:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

 

So
IOMMU group 16:    [1022:149c] 0d:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

is the back of the Mobo USB ports, looks greats its in its own group with no other devices everything I've seen/read sounds like this is the best thing i could hope for.

 

IOMMU Group 0:  [1022:149c] 08:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

This controller has the unRaid USB

 

Issues start when i try to pass through 0d:00.3, first way i tried was append vfio-pci.ids=1022:149c but this caused unraid to not boot at all i was wondering if it was because both controllers shared the same number. i have tried the pcie_acs_override=downstream,multifunction with this as well but still unraid wouldn't boot. 

 

Next way i tried is VFIO-PCI CFG which seemed to work, it got unraid booted and i was able to select the USB Controller in Other PCI devices.

Then i tried to start the VM:

Jan 29 18:55:49 XiamoUnraid kernel: br0: port 2(vnet0) entered blocking state
Jan 29 18:55:49 XiamoUnraid kernel: br0: port 2(vnet0) entered disabled state
Jan 29 18:55:49 XiamoUnraid kernel: device vnet0 entered promiscuous mode
Jan 29 18:55:49 XiamoUnraid kernel: br0: port 2(vnet0) entered blocking state
Jan 29 18:55:49 XiamoUnraid kernel: br0: port 2(vnet0) entered forwarding state
Jan 29 18:55:50 XiamoUnraid kernel: vfio-pci 0000:0d:00.3: enabling device (0000 -> 0002)
Jan 29 18:55:51 XiamoUnraid avahi-daemon[3024]: Joining mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fef7:df28.
Jan 29 18:55:51 XiamoUnraid avahi-daemon[3024]: New relevant interface vnet0.IPv6 for mDNS.
Jan 29 18:55:51 XiamoUnraid avahi-daemon[3024]: Registering new address record for fe80::fc54:ff:fef7:df28 on vnet0.*.
Jan 29 18:56:03 XiamoUnraid kernel: vfio-pci 0000:0d:00.3: not ready 1023ms after FLR; waiting
Jan 29 18:56:05 XiamoUnraid kernel: vfio-pci 0000:0d:00.3: not ready 2047ms after FLR; waiting
Jan 29 18:56:08 XiamoUnraid kernel: vfio-pci 0000:0d:00.3: not ready 4095ms after FLR; waiting
Jan 29 18:56:13 XiamoUnraid kernel: vfio-pci 0000:0d:00.3: not ready 8191ms after FLR; waiting
Jan 29 18:57:14 XiamoUnraid kernel: clocksource: timekeeping watchdog on CPU7: Marking clocksource 'tsc' as unstable because the skew is too large:
Jan 29 18:57:14 XiamoUnraid kernel: clocksource: 'hpet' wd_now: 9f6a0fcd wd_last: 9d88d3d5 mask: ffffffff
Jan 29 18:57:14 XiamoUnraid kernel: clocksource: 'tsc' cs_now: f36a63b2100 cs_last: f3596a25a74 mask: ffffffffffffffff
Jan 29 18:57:14 XiamoUnraid kernel: tsc: Marking TSC unstable due to clocksource watchdog
Jan 29 18:57:16 XiamoUnraid kernel: TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.
Jan 29 18:57:16 XiamoUnraid kernel: sched_clock: Marking unstable (4386715655544, -38594027)<-(4386979681370, -302640679)

Once vfio-pci 0000:0d:00.3: not ready 1023ms after FLR; waiting starts unraid just starts to break quickly then i have to hard power off the PC. 

 

I have tried a PCI-E USB Card but the issue with that is the rest of my pci-e slots all are part of IOMMU Group 0 which i have tried ACS downstream with but when passing through i still get vfio-pci : not ready 1023ms after FLR; waiting and unraid hard locks. 

 

Any help with this would be great.

 

Thanks, 

 

Xerby

 

xiamounraid-diagnostics-20200130-1658.zip xiamounraid-syslog-20200130-2153.zip

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.