8 hours ago, JorgeB said:

Are use using fiber with transceivers or DAC cables?

DAC cables

I'm sorry, I don't see how that can be an Unraid problem, driver is not detecting a link, and it won't work until it does, and find it very strange that TrueNAS Scale would detected a link with exactly the same hardware, it uses the same driver.

 

 

7 minutes ago, JorgeB said:

I'm sorry, I don't see how that can be an Unraid problem, driver is not detecting a link, and it won't work until it does, and find it very strange that TrueNAS would detected a link with exactly the same hardware.

Thanks, but help me understand: If I reboot the server, without touching the server or hardware, and only change the boot device, TrueNas Scale boots and detects the Mellanox NIC with DAC cables and receives a reserved IP. Unraid does not. Same HW, one OS brings up NIC ok, but another OS does not. Sounds to me like software.

Edited March 4, 20242 yr by Bladedude

Not sure, you missed my edit, TrueNAS scale uses the same driver, so don't see how that is possible, but not saying it's not happening.

Yup. Fair enough. Let me 100% validate on my end again. I'll reboot into TrueNas Scale and grab a screenshot and dmesg on boot and see if that can help.

On 1/28/2024 at 12:00 PM, simo241 said:

I been able to show the Mellanox 10gb Card trought the below procedures:

 

# lspci | grep Mellanox 01:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX IB QDR, PCIe 2.0 5GT/s] (rev b0)

#echo ib > /sys/bus/pci/devices/0000\:01\:00.0/mlx4_port1

#echo eth > /sys/bus/pci/devices/0000\:01\:00.0/mlx4_port2

 

Configured the Eth1 to work with windows Pc , only 3 uge problem:

1 - After reboot Eth2 disappear , configuration need to be done again.

2 - if i resume from sleep or i dont use server for a while  port eth1 is going down.

3 - Transfer from Unraid to Windows is very slow only 120 Mb/s , from windows to unraid the speed is about 1,5Gb/s

Hi everyone , anybody can help me on point # 2 , i been able to resume the ETH port trough a script but how can i do it in automatically after resume from sleep and not manually every time , any suggestion?

Also i still experience very slow transfer speed  

Edited April 24, 20242 yr by simo241

On 4/24/2024 at 2:54 PM, simo241 said:

Hi everyone , anybody can help me on point # 2 , i been able to resume the ETH port trough a script but how can i do it in automatically after resume from sleep and not manually every time , any suggestion?

Also i still experience very slow transfer speed  

Any help please

 

Data when booting server and Mellanox ConnectX-3 Pro into TrueNas scale and ethernet comes up, but ethernet does come up when I reboot into Unraid.

admin@supermicro[~]$ uname -srm
Linux 6.1.63-production+truenas x86_64

admin@supermicro[~]$ /sbin/ifconfig
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.X.201  netmask 255.255.255.0  broadcast 192.168.X.255
        inet6 XX prefixlen 64  scopeid 0x20<link>
        inet6 XX prefixlen 64  scopeid 0x0<global>
        ether 9c:dc:X  txqueuelen 1000  (Ethernet)
        RX packets 47076  bytes 49359473 (47.0 MiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 16362  bytes 6274827 (5.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

admin@supermicro[~]$ sudo dmesg | grep -i enp1s0          
[    9.744788] mlx4_core 0000:01:00.0 enp1s0: renamed from eth0
[    9.836760] mlx4_core 0000:01:00.0 enp1s0d1: renamed from eth0
[   11.154076] mlx4_en: enp1s0: Link Up
[   48.685128] mlx4_en: enp1s0: Steering Mode 1
[   48.740749] mlx4_en: enp1s0: Link Up

admin@supermicro[~]$ netstat -i
Kernel Interface table
Iface             MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
enp1s0           1500    57271      0      1 0         20159      0      0      0 BMRU

admin@supermicro[~]$ lspci | grep Ethernet
01:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

 

Edited May 4, 20242 yr by Bladedude

 to use Mellanox ConnectX cards, you need to install the plugin and run the firmware:



while the infiniband for the ring form 1 dac to mutil doesn't work in unraid.  a single interface direct sac to sac fiber line can still carry the full bandwidth when setup correctly.

 

13 minutes ago, bmartino1 said:

 to use Mellanox ConnectX cards, you need to install the plugin and run the firmware:



while the infiniband for the ring form 1 dac to mutil doesn't work in unraid.  a single interface direct sac to sac fiber line can still carry the full bandwidth when setup correctly.

 

Thanks @bmartino1. I'm not using infiniband, just 10GbE ethernet via DAC cable. Will that work as is? Or do I need fiber transceiver and cable?

Edited May 4, 20242 yr by Bladedude

yes ich777 made that plugin for that exact setup.
Tested with a freinds melnox 10G fiber and dak to a mikrotek switch.

install plugin via app, may need to reboot settings confirm via firmware that ist up and check unraid setting netwrok for a igbc interface.

@bmartino1Thanks. I had already installed the Mellanox Firmware Tools plugin. I assumed the interface would be enabled by default on install. It wasn't, but all I had to do was enable the interface. The board has 4 GbE + 2 SFP+, so I missed that on the network page. 🤦‍♂️ Once enabled, it came right up with no issues. Thanks again!

Feature request / bug fix would be to enable, by default, on install any/all interfaces.

Edited May 5, 20242 yr by Bladedude