Hi all,

Recently bought a couple of Mellanox MCX312B-XCCT CX312B, these are dual ports and one is used for connecting a pfsense box between WAN and LAN (1G SFP on the WAN side and a 10G SFP+ DAC cable on the LAN side).
The other i put in my Unraid box (One port unused and one connected to LAN with a 10G SFP+ DAC) but it won't show up in the list of NICs in the "/Settings/NetworkSettings -> Interface Rules and the drop down lists.

Checking the logs gives me these warnings and "0000:04:00.0" being my Mellanox card indicates that everything is not all right.

Jan 22 19:39:48 Tower kernel: ACPI: Early table checksum verification disabled
Jan 22 19:39:48 Tower kernel: pci 0000:04:00.0: BAR 9: failed to assign [mem size 0x04000000 64bit pref]
Jan 22 19:39:48 Tower kernel: floppy0: no floppy controllers found
Jan 22 19:39:48 Tower kernel: ACPI Error: No handler for Region [SYSI] (000000001bc0de87) [IPMI] (20220331/evregion-130)
Jan 22 19:39:48 Tower kernel: ACPI Error: Region IPMI (ID=7) has no handler (20220331/exfldio-261)
Jan 22 19:39:48 Tower kernel: ACPI Error: Aborting method \_SB.PMI0._GHL due to previous error (AE_NOT_EXIST) (20220331/psparse-529)
Jan 22 19:39:48 Tower kernel: ACPI Error: Aborting method \_SB.PMI0._PMC due to previous error (AE_NOT_EXIST) (20220331/psparse-529)
Jan 22 19:39:48 Tower kernel: ACPI: \_SB_.PMI0: _PMC evaluation failed: AE_NOT_EXIST
Jan 22 19:39:48 Tower kernel: mlx4_core 0000:04:00.0: Failed to initialize queue pair table, aborting
Jan 22 19:39:48 Tower kernel: mlx4_core: probe of 0000:04:00.0 failed with error -5
Jan 22 19:52:02 Tower root: Fix Common Problems: Warning: Deprecated plugin ca.backup2.plg
Jan 22 19:52:08 Tower root: Fix Common Problems: Other Warning: dynamix.file.manager Support Updated

I have updated the FW to the latest i could find "2.42.5000" but i have the error/issue. I have also deleted network-rules.cfg and network.cfg and rebooted but no dice.
The two cards i have have been swapped around between the pfsense box and the unraid one where both worked as intended in pfsense.

Last idea i have right now is to swap in the card in another PCIe port but wanted to ask here first.

Bit of googeling on "mlx4_core 0000:04:00.0: Failed to initialize queue pair table, aborting" but that doesn't seem like something 'i' can configure?
https://support.mellanox.com/s/article/MLNX2-117-2389kn

Someone have any good ideas?

tower-diagnostics-20230122-2125.zip

14 hours ago, Cael817 said:

Last idea i have right now is to swap in the card in another PCIe port but wanted to ask here first.

Try this, you can also try the other NIC.

Good news, changing the PCIe port worked.

 

This is a Dell R720xd and i had the Mellanox card in riser 2, slot 5 which indicates that its connected to CPU 1. Moved it around to riser 1, slot 3 and that slot says CPU 2. Show right up after starting the server, set it to 'eth0' and did a restart and viola, everything works.

 

Thanks @JorgeB.

Hey, I am trying to use this same card as above. One SFP+ port on the Mellanox card via Linkbasic Direct Attached Copper 1m 10G SFP+ Uplink Cable to SFP+ port 11 on UDM Pro. The card shows in the unraid network settings setup. However when i try configure the device shows online for a few seconds then drops. eth0 is existing 1000mbps port. I am not sure if this is a configuration thing of hardware is cable or pcie slot. i also had a short lived issue with getting stuck in the flexboot initialisation as the USB was not reading. Port one was connected to the UDM at the start. @Cael817 how did you configure? I have set rules for the port to go to the same vlan as the existing GBE. I have set the SFP to a new ip address manually with ETH1 mac address so when it joins the network it will be at an ip i know. then i can replace it with the existing static one services call to.

I hope this is of some help, it was about a year ago i fiddled with this, these are my settings (starting to think that having a 1GBe as backup may hurt my performance but im not sure, system is up so my settings don't seem to show the same screen as yours but i can stop my VMs tomorrow if you need better screens).

This is the cables i bought from fs.com

Im not sure i explained my connections thoroughly and im not sure this will help but..

Wan is connected to a pfSense box with 1GBe fiber, pfSense box has one one the Mellanox cards and a 1GBe SFP in the Wan port, The second (LAN) port has one of the DAC cables connecting to my switch (Cisco, that's why i bought a Cisco coded cable, i think you can disable the check but price was low and id didn't want to screw around with that, fortunately the Mellanox card didn't complain (and shouldn't to my understanding)), Then from the switch i have the other DAC cable connected to my Unraid and eth0 using the other Mellanox card (second SFP port on this Mellanox card is unused).

 

Edited February 26, 20242 yr by Cael817

Thanks for the support @Cael817, taking the time to look out the information to help me is very kind of you. Through some experimentation I think you made the 10g port the eth0 using the "Interface Rules" and the 1GBE your eth1. I have replicated making NIC p1 = eth0, NIC p2 = eth1, GBE = eth2. I have bonded eth0 and eth2. eth0 picks up a ip from DHCP on UDM and Unraid is accessible by this DHCP provided address in addition to the static address assigned by the UDM router to the GBE /eth2 port. If i pull eth2 Unraid is inaccessible. In addition i have the warning "interface ethernet port 0 is down. check cable!" I'm still uncertain if it is a config or cable issue. perhaps you could share your config of your eth1. Perhaps there is something i am overlooking with my inexperience with sfp and NIC.

 

EDIT: to include the interface dashboard view. It will alternate on/off eth0.

Edited February 27, 20242 yr by Be-Art
Addition of screenshots

Hey @Cael817, do you mind sharing your config info with me, i am stumped. I have got SFP+ fiber transceivers now because of the check cable statement and still no communication. Greatly appreciated if i could know if there is anything special configured on your eth1 before bonding to eth0. 

Crap, genuinely sorry. I completely forgot about this when i got some cool suff for my 3D printer build.

 

Anyway, i don't remember doing anything special, heres a screen with eth1 removed from the bond

 

And here's when re adding it

 

Now, i have to admit that i have not tested if the backup is working but im going to pull the 10GB in a few minutes and see what happens 😃

 

Few minutes later, failover worked

Edited February 28, 20242 yr by Cael817

Thanks for the feedback above, Been many months. I have left it and re-visited from time to time. I recently went through this issues with another PCIE card. 

Then i re-visited the PCIE configurations to try get this working and still no joy. Actually This Open Speed Test report is slower when setting 'active backup' on for the GBE port on the motherboard and the Mellanox running the main link.