Mellanox ConnectX-3 support

[Bladedude]

8 hours ago, JorgeB said:

Are use using fiber with transceivers or DAC cables?

DAC cables

[JorgeB]

I'm sorry, I don't see how that can be an Unraid problem, driver is not detecting a link, and it won't work until it does, and find it very strange that TrueNAS Scale would detected a link with exactly the same hardware, it uses the same driver.

 

 

[Bladedude]

7 minutes ago, JorgeB said:

I'm sorry, I don't see how that can be an Unraid problem, driver is not detecting a link, and it won't work until it does, and find it very strange that TrueNAS would detected a link with exactly the same hardware.

Thanks, but help me understand: If I reboot the server, without touching the server or hardware, and only change the boot device, TrueNas Scale boots and detects the Mellanox NIC with DAC cables and receives a reserved IP. Unraid does not. Same HW, one OS brings up NIC ok, but another OS does not. Sounds to me like software.

Edited March 4 by Bladedude

[JorgeB]

Not sure, you missed my edit, TrueNAS scale uses the same driver, so don't see how that is possible, but not saying it's not happening.

[Bladedude]

Yup. Fair enough. Let me 100% validate on my end again. I'll reboot into TrueNas Scale and grab a screenshot and dmesg on boot and see if that can help.

[simo241]

On 1/28/2024 at 12:00 PM, simo241 said:

I been able to show the Mellanox 10gb Card trought the below procedures:

 

# lspci | grep Mellanox 01:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX IB QDR, PCIe 2.0 5GT/s] (rev b0)

#echo ib > /sys/bus/pci/devices/0000\:01\:00.0/mlx4_port1

#echo eth > /sys/bus/pci/devices/0000\:01\:00.0/mlx4_port2

 

Configured the Eth1 to work with windows Pc , only 3 uge problem:

1 - After reboot Eth2 disappear , configuration need to be done again.

2 - if i resume from sleep or i dont use server for a while  port eth1 is going down.

3 - Transfer from Unraid to Windows is very slow only 120 Mb/s , from windows to unraid the speed is about 1,5Gb/s

Hi everyone , anybody can help me on point # 2 , i been able to resume the ETH port trough a script but how can i do it in automatically after resume from sleep and not manually every time , any suggestion?

Also i still experience very slow transfer speed  

Edited April 24 by simo241

[simo241]

On 4/24/2024 at 2:54 PM, simo241 said:

Hi everyone , anybody can help me on point # 2 , i been able to resume the ETH port trough a script but how can i do it in automatically after resume from sleep and not manually every time , any suggestion?

Also i still experience very slow transfer speed  

Any help please

 

[Bladedude]

Data when booting server and Mellanox ConnectX-3 Pro into TrueNas scale and ethernet comes up, but ethernet does come up when I reboot into Unraid.

admin@supermicro[~]$ uname -srm
Linux 6.1.63-production+truenas x86_64

admin@supermicro[~]$ /sbin/ifconfig
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.X.201  netmask 255.255.255.0  broadcast 192.168.X.255
        inet6 XX prefixlen 64  scopeid 0x20<link>
        inet6 XX prefixlen 64  scopeid 0x0<global>
        ether 9c:dc:X  txqueuelen 1000  (Ethernet)
        RX packets 47076  bytes 49359473 (47.0 MiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 16362  bytes 6274827 (5.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

admin@supermicro[~]$ sudo dmesg | grep -i enp1s0          
[    9.744788] mlx4_core 0000:01:00.0 enp1s0: renamed from eth0
[    9.836760] mlx4_core 0000:01:00.0 enp1s0d1: renamed from eth0
[   11.154076] mlx4_en: enp1s0: Link Up
[   48.685128] mlx4_en: enp1s0: Steering Mode 1
[   48.740749] mlx4_en: enp1s0: Link Up

admin@supermicro[~]$ netstat -i
Kernel Interface table
Iface             MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
enp1s0           1500    57271      0      1 0         20159      0      0      0 BMRU

admin@supermicro[~]$ lspci | grep Ethernet
01:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

 

Edited May 4 by Bladedude

[bmartino1]

 to use Mellanox ConnectX cards, you need to install the plugin and run the firmware:



while the infiniband for the ring form 1 dac to mutil doesn't work in unraid.  a single interface direct sac to sac fiber line can still carry the full bandwidth when setup correctly.

 

[Bladedude]

13 minutes ago, bmartino1 said:

 to use Mellanox ConnectX cards, you need to install the plugin and run the firmware:



while the infiniband for the ring form 1 dac to mutil doesn't work in unraid.  a single interface direct sac to sac fiber line can still carry the full bandwidth when setup correctly.

 

Thanks @bmartino1. I'm not using infiniband, just 10GbE ethernet via DAC cable. Will that work as is? Or do I need fiber transceiver and cable?

Edited May 4 by Bladedude

[bmartino1]

yes ich777 made that plugin for that exact setup.
Tested with a freinds melnox 10G fiber and dak to a mikrotek switch.

install plugin via app, may need to reboot settings confirm via firmware that ist up and check unraid setting netwrok for a igbc interface.

[Bladedude]

@bmartino1Thanks. I had already installed the Mellanox Firmware Tools plugin. I assumed the interface would be enabled by default on install. It wasn't, but all I had to do was enable the interface. The board has 4 GbE + 2 SFP+, so I missed that on the network page. 🤦‍♂️ Once enabled, it came right up with no issues. Thanks again!

Feature request / bug fix would be to enable, by default, on install any/all interfaces.

Edited May 5 by Bladedude