Hello everyone,
maybe someone can point me in the right direction. I don't know anything else to do.
The last couple of days I was trying to passthrough a USB PCIe card for my VMs.
My setup:
- Unraid 6.8.3
According to the A2SDi manual the board supports Intel Virtualization Technology for Directed I/O (Intel VT-X, VT-d)
Cards I tried so far:
- KALEA-INFORMATIQUE Controller Card PCI Express (PCI-E) to 3.1 USB (Chipset: Asmedia ASM1142)
- Inateck USB 3.0 PCIe Karte KT4006 (Chipset: FL1100)
The following steps were done with both cards:
After plugging in a card in the PCIe x4 slot I boot up Unraid without a problem. The USB card is detected by the system and has its own IOMMU group 16.
Following the tutorial from @SpaceInvaderOne I double-checked if the card is in its own group and if it is able to reset. (Which both cards are)
Then I set the vfio-pci.ids=1b73:1100 (for the fresco card) according to the video in the syslinux file. Later I also tried using the "VFIO-PCI Config" Plugin.
Both ways seem to work as expected and after a restart the card is bound.
For the last step I then select the card in one of my VMs and start the VM it. This is when things startet to act strange...
-------------------------------------
1) KALEA-INFORMATIQUE Controller Card PCI Express (PCI-E) to 3.1 USB (Chipset: Asmedia ASM1142):
I start one of my VMs and the following messages pop up in my log files:
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio_cap_init: 0000:02:00.0 hiding cap 0xff
Nov 11 21:19:32 Tower kernel: vfio-pci 0000:02:00.0: timed out waiting for pending transaction; performing function level reset anyway
Nov 11 21:19:33 Tower kernel: vfio-pci 0000:02:00.0: not ready 1023ms after FLR; waiting
Nov 11 21:19:35 Tower kernel: vfio-pci 0000:02:00.0: not ready 2047ms after FLR; waiting
Nov 11 21:19:37 Tower kernel: vfio-pci 0000:02:00.0: not ready 4095ms after FLR; waiting
Nov 11 21:19:41 Tower kernel: vfio-pci 0000:02:00.0: not ready 8191ms after FLR; waiting
Nov 11 21:19:50 Tower kernel: vfio-pci 0000:02:00.0: not ready 16383ms after FLR; waiting
Nov 11 21:20:07 Tower kernel: vfio-pci 0000:02:00.0: not ready 32767ms after FLR; waiting
Nov 11 21:20:43 Tower kernel: vfio-pci 0000:02:00.0: not ready 65535ms after FLR; giving up
So far my research tells me that these messages appear when the card can not be resetted (which it normally should do)
In order to try again I then have to restart my server, because of the "Unknown PCI header type '127' for device '0000:02:00.0'"
As I understand: the card isn't resetting...
-------------------------------------
2) Inateck USB 3.0 PCIe Karte KT4006 (Chipset: FL1100):
The Inateck card looked more promising, but failed anyways with a slight change in the log messages.
Nov 12 13:57:20 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:20 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:20 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:20 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:20 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:20 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:20 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:20 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:20 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:20 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:21 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:21 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:21 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:21 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:21 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:21 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
Nov 12 13:57:21 Tower kernel: vfio_bar_restore: 0000:02:00.0 reset recovery - restoring bars
In the end, I also have to restart my server because the card isn't resetting itself.
So far I wasn't able to find an answer what exactly those messages try to tell me.
-------------------------------------
What I tried to solve the problem:
Set PCIe overdrive to downstream, multifunction or both
Enabled VFIO allow unsafe interrupts (and disabled it in the bios)
Looked in the bios if any setting is disabled/enabled which could cause the error.
Any help is really appreciated. Maybe one of you is able to help me.
If any further information helps just tell me and I will post it as soon as possible!
Best regards
tatec