QNAP QXG-2G4T-I225 VM passthrough issues

I've been having issues trying to get a new QNAP QXG-2G4T-I225 card passed through into a pfSense/freeBSD VM.

Using the Intel I225-LM, it's a PCIEx4 card with 4 x 2.5Gbe ports, each with it's own adaptor paired with a ASMedia Technology Inc. Device 1812 PCI bridge (although there seem to be 7 bridges across 7 IOMMU groups?).

The system is using an ASUS Z390-e mobo with VT-d and IO-SV enabled in BIOS (although I've tried the below with both IO-SV enabled and disabled).

IOMMU groups as shown (truncated list, just showing the ones associated with the card).

Quote

 

IOMMU group 11:				[1b21:1812] 03:00.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01)

IOMMU group 12:				[1b21:1812] 04:00.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01)
 					[8086:15f2] 05:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-LM (rev ff)

IOMMU group 13:				[1b21:1812] 04:02.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01)
 					[8086:15f2] 06:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-LM (rev ff)

IOMMU group 14:				[1b21:1812] 04:03.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01)

IOMMU group 15:				[1b21:1812] 04:08.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01)
 					[8086:15f2] 08:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-LM (rev ff)

IOMMU group 16:				[1b21:1812] 04:0a.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01)
 					[8086:15f2] 09:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-LM (rev ff)

IOMMU group 17:				[1b21:1812] 04:0b.0 PCI bridge: ASMedia Technology Inc. Device 1812 (rev 01)

 

Any time I've tried, I get the dreaded "internal error: Unknown PCI header type '127' for device '0000:0X:00.0'".

Here's what I've tried so far:

• Binding the individual Ethernet Controllers to VFIO through the System Devices page.
  • When doing this, the ethernet controllers show up with "!!! Unknown header type 7f" when hovering over the device on the list, even after boot.
  • NOTE: When not binding the controllers to VFIO, they present as: 

    [8086:15f2] 09:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-LM (rev 03) and work fine. Note the revision change, not sure why that is.

  • Composite image below shows 3 out of the 4 controllers being bound to VFIO.

  • 

• Put "pci-stub.ids=8086:15f2" in the syslinux config file.
  • When I do this, the controllers seem to start fine as shown in the System Devices  *until* I try and start a VM with them, then they show "!!! Unknown header type 7f"
  • I also tried "pci-stub.ids=8086:15f2,1b21:1812" (added the PCI bridge)
• Put "xen-pciback.hide=(05:00.0)(06:00.0)(08:00.0)(09:00.0) in the syslinux config file.
  • Same issues as the pci-stubs.ids method: seems to work *until* I try to start the pfSense
  • Also tried adding in the PCI bridges as well and it has the same effect.
• Tried using the PCIe ACS overrides to no avail, although it *does* separate the ethernet controllers and PCI bridges to separate IOMMU groups.
• Tried VFIO allow safe interrupts to no avail (not that I'd want to have this on 24/7)

I haven't had any issues with the 'other devices within the same IOMMU group are in use' issues that you can have and the card works fine in Unraid itself.

I had no issues with my Dell THGMP Intel I350-T4 Quad Port card in the same setup, be it passing in the whole card or half the ports, so I don't think it's related to my mobo and VT-d support.

I'm stumped. Anyone got any ideas what's up?

Edited June 5, 20215 yr by Hazza64
Added in additional fix attempts