December 3, 20205 yr Interested as well... I scored a ConnectX-3 card a little while back and been wondering if i should toss it into my NAS box. A quick search revealed that even pfSense V1.2 supports it, so hopefully unraid can as well. *EDIT: Card in question is the HP 764736-001 - HPE InfiniBand FDR/Ethernet 2-Port 544+QSFP/FLR-QSFP and 547FLR-QSFP Adapter. these cards are readily available on eBay for under $100 Edited December 3, 20205 yr by _Adrian_
December 4, 20205 yr Community Expert 10 hours ago, _Adrian_ said: so hopefully unraid can as well. Ethernet ConnectX-3 are supported for a long time, but Unraid doesn't support Infiniband.
December 5, 20205 yr On 12/4/2020 at 12:51 AM, JorgeB said: Ethernet ConnectX-3 are supported for a long time, but Unraid doesn't support Infiniband. I have no interest in infiniband but rather if unRaid can bifurcate the 40G ports ( QSFP - 4x 10GBe ) in order to eliminate the need for a core network switch and use my unraid box as a Router/10G switch. I have 3 Miktotik CSS610-8G+2S+ ( 8 Gigabit + 2SFP+ ports ) that are linked to my pfSense Box. Im soon adding a 10Gigabit card to the pfSense box and would like to upgrade the backbone to 10Gigabit but that would require a 10Gigabit switch. IF i can achieve this with my unRaid box in software then i wont bother purchasing a $400 Switch ( Mikrotik CRS309-1G-8S+ ) and can sink it into something like a 3080
February 23, 20215 yr On 12/5/2020 at 10:57 PM, _Adrian_ said: I have no interest in infiniband but rather if unRaid can bifurcate the 40G ports ( QSFP - 4x 10GBe ) in order to eliminate the need for a core network switch and use my unraid box as a Router/10G switch. As far as I am aware this isn't even a question if UnRaid supports it or not. The network cards don't support it. Splitting 40G into 4x 10G is reserved for Switches and some of the newer models. I am currently getting around 1.8 -1.5 GB/s to my NVMe RAID 1. So fairly close to @Siren. On Windows Server I can get to around 3 GB/s with RDMA and RSS working correctly which as near as makes no difference maxes out the drives. I suspect 2 GB/s is going to be a fairly hard limit without either RDMA or RSS.
February 23, 20215 yr Community Expert On 12/6/2020 at 5:57 AM, _Adrian_ said: I have no interest in infiniband but rather if unRaid can bifurcate the 40G ports ( QSFP - 4x 10GBe ) in order to eliminate the need for a core network switch and use my unraid box as a Router/10G switch. I have 3 Miktotik CSS610-8G+2S+ ( 8 Gigabit + 2SFP+ ports ) that are linked to my pfSense Box. Im soon adding a 10Gigabit card to the pfSense box and would like to upgrade the backbone to 10Gigabit but that would require a 10Gigabit switch. IF i can achieve this with my unRaid box in software then i wont bother purchasing a $400 Switch ( Mikrotik CRS309-1G-8S+ ) and can sink it into something like a 3080 Its not bifurcation in this context, it's SR-IOV that you want to look at. Its supported in Linux, and by the X3 cards (see link), but in an Unraid context I don't know of anyone that's been successful, or if we have all the prerequisites in place, even with the Mellanox toolset installed... maybe @ich777 knows? https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connectx-3-with-kvm--ethernet-x
February 24, 20215 yr On 2/23/2021 at 4:45 AM, tjb_altf4 said: maybe @ich777 knows? No, haven't done anything related to SR-IOV but after reading through the link that @tjb_altf4 has posted it should be possible if you install the Mellanox Firmware Tools if you are on 6.9.0RC2 from the CA App (be sure to replace 'mlxconfing' with 'mstconfig' my package includes the open source version of the Mellanox Firmware Tools).
June 2, 20215 yr JFYI: I've been playing with a cx3, the (mainline) in-kernel driver is v4.0.0 which contains a bug which makes sriov not possible for windows guests (other guests work well). This bug was fixed with mlx4 v4.3.x which is only available via OFED.
October 24, 20232 yr Sorry to rescue this old thread. I was about to purchase this melanox model but seen that is not fully compatible is a problem. Is there any other 40G card or better that is fully compatible with unraid and can be purchased used with a good price? I have found the intel XL710-QDA1 but it cost almost 3 times more in ebay Edited October 24, 20232 yr by L0rdRaiden
October 26, 20232 yr Community Expert On 10/24/2023 at 2:40 PM, L0rdRaiden said: Sorry to rescue this old thread. I was about to purchase this melanox model but seen that is not fully compatible is a problem. Is there any other 40G card or better that is fully compatible with unraid and can be purchased used with a good price? I have found the intel XL710-QDA1 but it cost almost 3 times more in ebay There are SKUs of the mellonox cards that are infiniband only, but don't think there are any in the MCX353A line up. You can confirm support in the product manual https://network.nvidia.com/pdf/user_manuals/ConnectX-3_VPI_Single_and_Dual_QSFP_Port_Adapter_Card_User_Manual.pdf MCX353A working in Jerod's video below (after some troubleshooting)
October 26, 20232 yr 2 hours ago, tjb_altf4 said: There are SKUs of the mellonox cards that are infiniband only, but don't think there are any in the MCX353A line up. You can confirm support in the product manual https://network.nvidia.com/pdf/user_manuals/ConnectX-3_VPI_Single_and_Dual_QSFP_Port_Adapter_Card_User_Manual.pdf MCX353A working in Jerod's video below (after some troubleshooting) So any melanox card based on QSFP+ will be compatible with Unraid but if it's based on infinitiband I could have problems? like these would be compatible? Mellanox CX314A MCX314A-BCCT ConnectX-3 Pro 40GbE Dual-Port QSFP PCIe Card | eBay Mellanox ConnectX-3 MCX354A-FCBT CX354A VPI 40/56GbE Dual-Port QSFP Adapter 789398906032 | eBay Edited October 26, 20232 yr by L0rdRaiden
October 26, 20232 yr Community Expert 2 hours ago, L0rdRaiden said: So any melanox card based on QSFP+ will be compatible with Unraid but if it's based on infinitiband I could have problems? like these would be compatible? Mellanox CX314A MCX314A-BCCT ConnectX-3 Pro 40GbE Dual-Port QSFP PCIe Card | eBay Mellanox ConnectX-3 MCX354A-FCBT CX354A VPI 40/56GbE Dual-Port QSFP Adapter 789398906032 | eBay Just don't get an infiniband only card, QSFP+ is just a connector/cable and can be used for either protocol. This one does both modes, you may to change modes to use when setting up. This one is ethernet only. Note the final character of the model (T or S) means the bracket height (Tall or Short)
January 11, 20242 yr Guys please i just purchased 2 Mcx354A -FCBT card but i am not able to get working on unraid , in network configuration is not showing , is present in system device. i belive is because is set to infiniband but i am not able to change , i also tried on windows to do taht but installying all the software and driver but the option is not showing . Can you please help me
January 12, 20242 yr Community Expert 13 hours ago, simo241 said: i belive is because is set to infiniband but i am not able to change See if this helps: https://forums.unraid.net/topic/78406-mellanox-connectx-3-support/?do=findComment&comment=761855
January 13, 20242 yr No, i already tryed all, i tried with both card, i installed both card on two different machine , using linux and windows and in all case when i try mst status , i got device not found . The card are perfectly working in infiniband , but i cannot converth to eterneth Its a shame the card are new, i have to buy another 2 card but this time i will purchase the ethernet one Is this card will be good ? Mellanox MCX311A-XCAT CX311A ConnectX-3 EN 10G Ethernet 10GbE SFP Network Card. What if i buy only one card and connect the infiniband on windows and this MCX311A on unraid with a cable QSFP to SFP+ will work ? Edited January 13, 20242 yr by simo241
January 13, 20242 yr Community Expert 2 hours ago, simo241 said: Mellanox MCX311A-XCAT CX311A ConnectX-3 EN 10G Ethernet 10GbE SFP Network Card. Should be, I have several of those. 2 hours ago, simo241 said: What if i buy only one card and connect the infiniband on windows and this MCX311A on unraid with a cable QSFP to SFP+ will work ? IDK
January 13, 20242 yr Author 12 hours ago, simo241 said: No, i already tryed all, i tried with both card, i installed both card on two different machine , using linux and windows and in all case when i try mst status , i got device not found . So it seems like it could be 2 things: - Drivers are messed up somewhere and you might need to reinstall them - Firmware issue on the card (happened to mine, which were also brand new at the time) On your windows machine, if you have the cards set to Infiniband, can you run mst status and post a screenshot? I want to try and map out your issue. If you cant, You might need to use WinMFT and re-flash the firmware. Here's a link for the firmware: http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip Link to WinMFT: https://www.mellanox.com/downloads/MFT/WinMFT_x64_4_26_1_3.exe Steps on how to burn the firmware: https://network.nvidia.com/support/firmware/nic/ Your ID's should be the same as mine since I have the same card, but with 1 physical port instead of 2. Since then, I've got the card to work on UnRAID on my Dell server, but I'm planning on redoing my main server in the future and had to pull the card out to fit multiple GPU's.
January 14, 20242 yr 14 hours ago, Siren said: So it seems like it could be 2 things: - Drivers are messed up somewhere and you might need to reinstall them - Firmware issue on the card (happened to mine, which were also brand new at the time) On your windows machine, if you have the cards set to Infiniband, can you run mst status and post a screenshot? I want to try and map out your issue. If you cant, You might need to use WinMFT and re-flash the firmware. Here's a link for the firmware: http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip Link to WinMFT: https://www.mellanox.com/downloads/MFT/WinMFT_x64_4_26_1_3.exe Steps on how to burn the firmware: https://network.nvidia.com/support/firmware/nic/ Your ID's should be the same as mine since I have the same card, but with 1 physical port instead of 2. Since then, I've got the card to work on UnRAID on my Dell server, but I'm planning on redoing my main server in the future and had to pull the card out to fit multiple GPU's. I cannot install new firmware with windows because when i type smt status , i got " no mst device " However i did upgrade to latest firmware troug mellanox plug in on unraid instead of use the link othe plug in i use the link of the firmware belong to my card. Even after upgrade on unraid and move the card to windwos still i got " no mst device' I also tried with ubuntu on 2 different machine and still i got no mst device found , i also use mlxconfig and i got unable to open the device i also just i case disable secure boot and enable SR-iov as reported on same post around Edited January 14, 20242 yr by simo241
January 22, 20242 yr On 1/14/2024 at 4:12 AM, simo241 said: I cannot install new firmware with windows because when i type smt status , i got " no mst device " However i did upgrade to latest firmware troug mellanox plug in on unraid instead of use the link othe plug in i use the link of the firmware belong to my card. Even after upgrade on unraid and move the card to windwos still i got " no mst device' I also tried with ubuntu on 2 different machine and still i got no mst device found , i also use mlxconfig and i got unable to open the device i also just i case disable secure boot and enable SR-iov as reported on same post around are you sure sure you installed the correct drivers? connectx-3 and below have diff drivers than connecx-4 and up
January 22, 20242 yr 57 minutes ago, bandit8623 said: are you sure sure you installed the correct drivers? connectx-3 and below have diff drivers than connecx-4 and up The driver for windows are automatically installed , the inly thing i can install is winmft, and i installed the one linked above . i also removed all drivers with clean unistaller , reboot and install again , but my card is not recognised, actually my boot cards are not reconised.
January 22, 20242 yr Ok i been able to properly install the driver using winof on windows and change the card protocol to eth . Now the problem is that still when i use mst status i am sill not able to find any devise, so i believe once i move the card to unraid it will still be on infiniband Edited January 22, 20242 yr by simo241
January 28, 20242 yr I been able to show the Mellanox 10gb Card trought the below procedures: # lspci | grep Mellanox 01:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX IB QDR, PCIe 2.0 5GT/s] (rev b0) #echo ib > /sys/bus/pci/devices/0000\:01\:00.0/mlx4_port1 #echo eth > /sys/bus/pci/devices/0000\:01\:00.0/mlx4_port2 Configured the Eth1 to work with windows Pc , only 3 uge problem: 1 - After reboot Eth2 disappear , configuration need to be done again. 2 - if i resume from sleep or i dont use server for a while port eth1 is going down. 3 - Transfer from Unraid to Windows is very slow only 120 Mb/s , from windows to unraid the speed is about 1,5Gb/s
March 1, 20242 yr # lspci | grep Mellanox 01:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] # lspci -vv -s 01:00.0 | grep "Part number" -A 3 [PN] Part number: 779793-B21 [EC] Engineering changes: B-5718 [SN] Serial number: IL271700F5 [V0] Vendor specific: PCIe 10GbE x8 6W New install of Unraid. Adaptor will not link up out of the box. I installed Mellanox Firmware Tools plug-in, and it's still not up. If I reboot into Truenas Scale (I'm wiping my existing Scale and installing Unraid), it came up from the default Truenas Scale installation.
March 1, 20242 yr Community Expert 33 minutes ago, Bladedude said: New install of Unraid. Adaptor will not link up out of the box. I installed Mellanox Firmware Tools plug-in, and it's still not up. If I reboot into Truenas Scale (I'm wiping my existing Scale and installing Unraid), it came up from the default Truenas Scale installation. Please post the diagnostics, if it works with TrueNAS it should work with Unraid, assuming it's in Ethernet mode.
March 3, 20242 yr Diagnostics attached. Yes, it's in ethernet mode. supermicro-diagnostics-20240303-1147.zip Edited March 3, 20242 yr by Bladedude
March 4, 20242 yr Community Expert NIC appears to be working correctly, except there' no link, doesn't look like a software problem: Link detected: no driver: mlx4_en Are use using fiber with transceivers or DAC cables?
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.