Hi all,

 

I have an ASUS motherboard with a Thunderbolt 4 header: https://www.newegg.com/p/N82E16813119540?Item=N82E16813119540

as well as this Thunderbolt 4 PCI expansion card, the ThunderboltEX 4: https://www.newegg.com/asus-90mc09p0-m0aay0-pci-express-to-thunderbolt/p/N82E16815293048?Description=thunderboltex 4&cm_re=thunderboltex_4-_-15-293-048-_-Product&quicklink=true

running under Unraid 6.10.2 . The motherboard bios has been updated.

 

I plugged the card into the first PCIe slot on the motherboard, and connected both the Thunderbolt 4 and USB headers to the mobo (I didn't connect the molex power b/c I believe it is optional for enabling quick charging Power Delivery, which I don't need). In unraid's System Devices I can see a lot of "Thunderbolt" items, but only the USB controllers have checkboxes next to them. I've tried booting with all 4 ACS modes and it always looks the same (see attached image).

 

Why am I unable to select the PCI bridge IOMMU groups to bind to VFIO at boot?

 

EDIT: The command

cat /sys/bus/thunderbolt/devices/domain0/security

outputs "nopcie". I couldn't find any Thunderbolt security settings in the motherboard bios to change this. But perhaps this is part of the reason?

 

Thanks for the help,

 

-- Matt

Edited June 2, 20224 yr by matthewmalek
Add unraid version

Update:

 

I now believe the IOMMU configuration above is OK. The NHI is what a VM is going to talk to, and the system will use PCIe over USB 4 to implement Thunderbolt (hence why it's a USB controller).

 

I made sure PCIe tunneling over USB4 was enabled in my BIOS (it was by default), I attached the two Thunderbolt devices to my Windows 11 VM, and booted it up. Note that when booting with the IOMMU groups bound to vfio, there is no longer a 'security' file for the device because it is no longer being mounted by unraid.

 

I had one unrecognized device in Windows so I installed this Intel Thunderbolt software (https://www.intel.com/content/www/us/en/download/19402/thunderbolt-3-and-4-dch-driver-for-windows-10-for-intel-nuc.html) which has an overview page (attached).

 

It seems to think that PCIe tunneling is disabled and the tooltip says it can be enabled in the BIOS. Why would both Unraid and the Windows VM think that PCIe tunneling is disabled and how can I troubleshoot this further?

 

Thanks!

multivac-diagnostics-20220601-1847.zip

Edited June 2, 20224 yr by matthewmalek
Attach sys logs

4 hours ago, matthewmalek said:

It seems to think that PCIe tunneling is disabled and the tooltip says it can be enabled in the BIOS. Why would both Unraid and the Windows VM think that PCIe tunneling is disabled and how can I troubleshoot this further?

I don't do VMs, but I guess that the function would need to be enabled in the VM BIOS/UEFI also ? Hopefully, someone who actually knows VMs can chime in.  

I'm having issues getting the ThunderboltEX 4 and mobo to play nicely on a bare metal Win 10 installation, so I think I'm going to call it quits. I'm going to return some of my recent hardware purchases and instead build two separate systems.

I know this is an old post and I am only providing this info for future people who may need it. One thing I will point out that I have found with the ASUS ThunderboltEX 4 and getting it to work on my "Asus ROG Strix B550-F Gaming WiFi II" motherboard and with windows is it HAD to be in the last slot of my motherboard for it to work. When I tried it in the first slot, it refused to work. The second issue I found is while in the last slot, if I had certain other x1 slots filled with anything, the same problem happened and it would not work. It seems it uses that last slot and a few others to share its resources with. While I have it working fine in a true Windows boot environment, I was never able to get it to work in unraid though. Any time I tried to boot it with the drives plugged in, it would just crash and reboot, and same for after it was booted and I plugged them in.

I still suspect it has to do with authorizing the devices in Unraid before it boots, but I do not know enough about Linux to figure it out, and I was not able to get any help here in the forums in another post I made. Also here is a picture of some of the settings in my bios I had to set for it to work. Note that the "Host Chipset" for this card as to be Maple Ridge, if you have the EX 3, then it needs to be the previous option that I cannot recall the name of. I am not sure if I kept "Data Link Feature Exchange" enabled or not though. That setting was disabled by default after first enabling TB and I was trying different things to try to get it working with Unraid. There are a few other settings that need to be on too like enable "Above 4G Decoding" and "Resize BAR". I hope this helps some others at least.

Edited January 16, 20242 yr by jalari