gromp

Hello! First post here Mellanox Plugin I've been struggling with a new Mellanox NIC, and wanted to make sure I understood what the correct settings were to set this up properly. The main issue I ran into is getting VFS exposed/visible. I was able to update the firmware normally using mstflint and the Mellanox Plugin you so graciously created but I am not able to see any evidence of the VFS or SRIOV in the /sys directory. Some of the documentation I was able to find from Nvidia states that for Linux it's normally required to manually "echo >" some chosen value into the appropriate directory, but this file is not visible on my system and I'm getting permission issues when I try to do it anyway: ~# echo 13 > /sys/class/net/eth2/device/sriov_numvfs -bash: /sys/class/net/eth2/device/sriov_numvfs: Permission denied I've tried multiple flash kernel parameters disabling enabling ASPM (that's a whole rabbit hole) ACS, I'm explicitly enabled all virtualization based features in my BIOS, I've also most recently tried this as well: ============================================= append pcie_aspm=off pci=assign-busses,realloc pci=nocrs pcie_acs_override=downstream,multifunction video=efifb:off isolcpus=10-15,18-23,28-31,42-47,50-55,60-63 initrd=/bzroot ============================================= I've tried enabling and disabling Resizable BAR, CSM and allowing IOMMU as "Auto" vs. "Enabled" - SRIOV is explicitly enabled in the bios I also tried removing bridging from the interfaces, and checking for VFS in the "Tools" and via lspci/mstconfig without success. All of these attempts had reboots between in an effort to reload. dmesg shows some information - like that other interfaces on the motherboard support SRIOV for example which is somewhat helpful in understanding what might cause this issue. Pretty novice when it comes to pcie, subsystems - I've made sure to configure SRIOV and VFS on the card as well - any assistance is deeply appreciated! Some clues: PCIe x4 seems to be negotiated - even though it's on a 16 slot which supports 16 (limited by another chipset PCIe device/switch however, not due to capacity issues) For context: Mellanox PCIe 21:00.0 & 21:00.1 represent PF/physical SFP 25Gb interfaces lspci -t ================================== +-[0000:20]-+-00.0 | +-00.2 | +-01.0 | +-02.0 | +-03.0 | +-03.1-[21]--+-00.0 | | \-00.1 | +-04.0 | +-05.0 | +-07.0 | +-07.1-[22]----00.0 | +-08.0 | \-08.1-[23]--+-00.0 | +-00.1 | +-00.3 | \-00.4 ================================ lspci (I can share more - this is just what seemed relevant to the mellanox) ================================ 20:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex 20:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU 20:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge 20:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge 20:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge 20:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge 20:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge 20:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge 20:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge 20:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] 20:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge 20:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] 21:00.0 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx] 21:00.1 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx] ================================= Happy to include any logs as needed, I wasn't sure which information was best to include for troubleshooting this problem. My hope is that this post might help other Unraid users using Mellanox cards to set up some of the cooler features more easily. Thank you!!