_Adrian_ Posted December 3, 2020 Share Posted December 3, 2020 (edited) Interested as well... I scored a ConnectX-3 card a little while back and been wondering if i should toss it into my NAS box. A quick search revealed that even pfSense V1.2 supports it, so hopefully unraid can as well. *EDIT: Card in question is the HP 764736-001 - HPE InfiniBand FDR/Ethernet 2-Port 544+QSFP/FLR-QSFP and 547FLR-QSFP Adapter. these cards are readily available on eBay for under $100 Edited December 3, 2020 by _Adrian_ Quote Link to comment
JorgeB Posted December 4, 2020 Share Posted December 4, 2020 10 hours ago, _Adrian_ said: so hopefully unraid can as well. Ethernet ConnectX-3 are supported for a long time, but Unraid doesn't support Infiniband. Quote Link to comment
_Adrian_ Posted December 5, 2020 Share Posted December 5, 2020 On 12/4/2020 at 12:51 AM, JorgeB said: Ethernet ConnectX-3 are supported for a long time, but Unraid doesn't support Infiniband. I have no interest in infiniband but rather if unRaid can bifurcate the 40G ports ( QSFP - 4x 10GBe ) in order to eliminate the need for a core network switch and use my unraid box as a Router/10G switch. I have 3 Miktotik CSS610-8G+2S+ ( 8 Gigabit + 2SFP+ ports ) that are linked to my pfSense Box. Im soon adding a 10Gigabit card to the pfSense box and would like to upgrade the backbone to 10Gigabit but that would require a 10Gigabit switch. IF i can achieve this with my unRaid box in software then i wont bother purchasing a $400 Switch ( Mikrotik CRS309-1G-8S+ ) and can sink it into something like a 3080 Quote Link to comment
intel64gamer Posted February 23, 2021 Share Posted February 23, 2021 On 12/5/2020 at 10:57 PM, _Adrian_ said: I have no interest in infiniband but rather if unRaid can bifurcate the 40G ports ( QSFP - 4x 10GBe ) in order to eliminate the need for a core network switch and use my unraid box as a Router/10G switch. As far as I am aware this isn't even a question if UnRaid supports it or not. The network cards don't support it. Splitting 40G into 4x 10G is reserved for Switches and some of the newer models. I am currently getting around 1.8 -1.5 GB/s to my NVMe RAID 1. So fairly close to @Siren. On Windows Server I can get to around 3 GB/s with RDMA and RSS working correctly which as near as makes no difference maxes out the drives. I suspect 2 GB/s is going to be a fairly hard limit without either RDMA or RSS. Quote Link to comment
tjb_altf4 Posted February 23, 2021 Share Posted February 23, 2021 On 12/6/2020 at 5:57 AM, _Adrian_ said: I have no interest in infiniband but rather if unRaid can bifurcate the 40G ports ( QSFP - 4x 10GBe ) in order to eliminate the need for a core network switch and use my unraid box as a Router/10G switch. I have 3 Miktotik CSS610-8G+2S+ ( 8 Gigabit + 2SFP+ ports ) that are linked to my pfSense Box. Im soon adding a 10Gigabit card to the pfSense box and would like to upgrade the backbone to 10Gigabit but that would require a 10Gigabit switch. IF i can achieve this with my unRaid box in software then i wont bother purchasing a $400 Switch ( Mikrotik CRS309-1G-8S+ ) and can sink it into something like a 3080 Its not bifurcation in this context, it's SR-IOV that you want to look at. Its supported in Linux, and by the X3 cards (see link), but in an Unraid context I don't know of anyone that's been successful, or if we have all the prerequisites in place, even with the Mellanox toolset installed... maybe @ich777 knows? https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connectx-3-with-kvm--ethernet-x Quote Link to comment
ich777 Posted February 24, 2021 Share Posted February 24, 2021 On 2/23/2021 at 4:45 AM, tjb_altf4 said: maybe @ich777 knows? No, haven't done anything related to SR-IOV but after reading through the link that @tjb_altf4 has posted it should be possible if you install the Mellanox Firmware Tools if you are on 6.9.0RC2 from the CA App (be sure to replace 'mlxconfing' with 'mstconfig' my package includes the open source version of the Mellanox Firmware Tools). 1 Quote Link to comment
efschu86 Posted June 2, 2021 Share Posted June 2, 2021 JFYI: I've been playing with a cx3, the (mainline) in-kernel driver is v4.0.0 which contains a bug which makes sriov not possible for windows guests (other guests work well). This bug was fixed with mlx4 v4.3.x which is only available via OFED. Quote Link to comment
L0rdRaiden Posted October 24, 2023 Share Posted October 24, 2023 (edited) Sorry to rescue this old thread. I was about to purchase this melanox model but seen that is not fully compatible is a problem. Is there any other 40G card or better that is fully compatible with unraid and can be purchased used with a good price? I have found the intel XL710-QDA1 but it cost almost 3 times more in ebay Edited October 24, 2023 by L0rdRaiden Quote Link to comment
tjb_altf4 Posted October 26, 2023 Share Posted October 26, 2023 On 10/24/2023 at 2:40 PM, L0rdRaiden said: Sorry to rescue this old thread. I was about to purchase this melanox model but seen that is not fully compatible is a problem. Is there any other 40G card or better that is fully compatible with unraid and can be purchased used with a good price? I have found the intel XL710-QDA1 but it cost almost 3 times more in ebay There are SKUs of the mellonox cards that are infiniband only, but don't think there are any in the MCX353A line up. You can confirm support in the product manual https://network.nvidia.com/pdf/user_manuals/ConnectX-3_VPI_Single_and_Dual_QSFP_Port_Adapter_Card_User_Manual.pdf MCX353A working in Jerod's video below (after some troubleshooting) Quote Link to comment
L0rdRaiden Posted October 26, 2023 Share Posted October 26, 2023 (edited) 2 hours ago, tjb_altf4 said: There are SKUs of the mellonox cards that are infiniband only, but don't think there are any in the MCX353A line up. You can confirm support in the product manual https://network.nvidia.com/pdf/user_manuals/ConnectX-3_VPI_Single_and_Dual_QSFP_Port_Adapter_Card_User_Manual.pdf MCX353A working in Jerod's video below (after some troubleshooting) So any melanox card based on QSFP+ will be compatible with Unraid but if it's based on infinitiband I could have problems? like these would be compatible? Mellanox CX314A MCX314A-BCCT ConnectX-3 Pro 40GbE Dual-Port QSFP PCIe Card | eBay Mellanox ConnectX-3 MCX354A-FCBT CX354A VPI 40/56GbE Dual-Port QSFP Adapter 789398906032 | eBay Edited October 26, 2023 by L0rdRaiden Quote Link to comment
tjb_altf4 Posted October 26, 2023 Share Posted October 26, 2023 2 hours ago, L0rdRaiden said: So any melanox card based on QSFP+ will be compatible with Unraid but if it's based on infinitiband I could have problems? like these would be compatible? Mellanox CX314A MCX314A-BCCT ConnectX-3 Pro 40GbE Dual-Port QSFP PCIe Card | eBay Mellanox ConnectX-3 MCX354A-FCBT CX354A VPI 40/56GbE Dual-Port QSFP Adapter 789398906032 | eBay Just don't get an infiniband only card, QSFP+ is just a connector/cable and can be used for either protocol. This one does both modes, you may to change modes to use when setting up. This one is ethernet only. Note the final character of the model (T or S) means the bracket height (Tall or Short) 1 Quote Link to comment
simo241 Posted January 11 Share Posted January 11 Guys please i just purchased 2 Mcx354A -FCBT card but i am not able to get working on unraid , in network configuration is not showing , is present in system device. i belive is because is set to infiniband but i am not able to change , i also tried on windows to do taht but installying all the software and driver but the option is not showing . Can you please help me Quote Link to comment
JorgeB Posted January 12 Share Posted January 12 13 hours ago, simo241 said: i belive is because is set to infiniband but i am not able to change See if this helps: https://forums.unraid.net/topic/78406-mellanox-connectx-3-support/?do=findComment&comment=761855 Quote Link to comment
simo241 Posted January 13 Share Posted January 13 (edited) No, i already tryed all, i tried with both card, i installed both card on two different machine , using linux and windows and in all case when i try mst status , i got device not found . The card are perfectly working in infiniband , but i cannot converth to eterneth Its a shame the card are new, i have to buy another 2 card but this time i will purchase the ethernet one Is this card will be good ? Mellanox MCX311A-XCAT CX311A ConnectX-3 EN 10G Ethernet 10GbE SFP Network Card. What if i buy only one card and connect the infiniband on windows and this MCX311A on unraid with a cable QSFP to SFP+ will work ? Edited January 13 by simo241 Quote Link to comment
JorgeB Posted January 13 Share Posted January 13 2 hours ago, simo241 said: Mellanox MCX311A-XCAT CX311A ConnectX-3 EN 10G Ethernet 10GbE SFP Network Card. Should be, I have several of those. 2 hours ago, simo241 said: What if i buy only one card and connect the infiniband on windows and this MCX311A on unraid with a cable QSFP to SFP+ will work ? IDK 1 Quote Link to comment
Siren Posted January 13 Author Share Posted January 13 12 hours ago, simo241 said: No, i already tryed all, i tried with both card, i installed both card on two different machine , using linux and windows and in all case when i try mst status , i got device not found . So it seems like it could be 2 things: - Drivers are messed up somewhere and you might need to reinstall them - Firmware issue on the card (happened to mine, which were also brand new at the time) On your windows machine, if you have the cards set to Infiniband, can you run mst status and post a screenshot? I want to try and map out your issue. If you cant, You might need to use WinMFT and re-flash the firmware. Here's a link for the firmware: http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip Link to WinMFT: https://www.mellanox.com/downloads/MFT/WinMFT_x64_4_26_1_3.exe Steps on how to burn the firmware: https://network.nvidia.com/support/firmware/nic/ Your ID's should be the same as mine since I have the same card, but with 1 physical port instead of 2. Since then, I've got the card to work on UnRAID on my Dell server, but I'm planning on redoing my main server in the future and had to pull the card out to fit multiple GPU's. Quote Link to comment
simo241 Posted January 14 Share Posted January 14 (edited) 14 hours ago, Siren said: So it seems like it could be 2 things: - Drivers are messed up somewhere and you might need to reinstall them - Firmware issue on the card (happened to mine, which were also brand new at the time) On your windows machine, if you have the cards set to Infiniband, can you run mst status and post a screenshot? I want to try and map out your issue. If you cant, You might need to use WinMFT and re-flash the firmware. Here's a link for the firmware: http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip Link to WinMFT: https://www.mellanox.com/downloads/MFT/WinMFT_x64_4_26_1_3.exe Steps on how to burn the firmware: https://network.nvidia.com/support/firmware/nic/ Your ID's should be the same as mine since I have the same card, but with 1 physical port instead of 2. Since then, I've got the card to work on UnRAID on my Dell server, but I'm planning on redoing my main server in the future and had to pull the card out to fit multiple GPU's. I cannot install new firmware with windows because when i type smt status , i got " no mst device " However i did upgrade to latest firmware troug mellanox plug in on unraid instead of use the link othe plug in i use the link of the firmware belong to my card. Even after upgrade on unraid and move the card to windwos still i got " no mst device' I also tried with ubuntu on 2 different machine and still i got no mst device found , i also use mlxconfig and i got unable to open the device i also just i case disable secure boot and enable SR-iov as reported on same post around Edited January 14 by simo241 Quote Link to comment
bandit8623 Posted January 22 Share Posted January 22 On 1/14/2024 at 4:12 AM, simo241 said: I cannot install new firmware with windows because when i type smt status , i got " no mst device " However i did upgrade to latest firmware troug mellanox plug in on unraid instead of use the link othe plug in i use the link of the firmware belong to my card. Even after upgrade on unraid and move the card to windwos still i got " no mst device' I also tried with ubuntu on 2 different machine and still i got no mst device found , i also use mlxconfig and i got unable to open the device i also just i case disable secure boot and enable SR-iov as reported on same post around are you sure sure you installed the correct drivers? connectx-3 and below have diff drivers than connecx-4 and up Quote Link to comment
simo241 Posted January 22 Share Posted January 22 57 minutes ago, bandit8623 said: are you sure sure you installed the correct drivers? connectx-3 and below have diff drivers than connecx-4 and up The driver for windows are automatically installed , the inly thing i can install is winmft, and i installed the one linked above . i also removed all drivers with clean unistaller , reboot and install again , but my card is not recognised, actually my boot cards are not reconised. Quote Link to comment
simo241 Posted January 22 Share Posted January 22 (edited) Ok i been able to properly install the driver using winof on windows and change the card protocol to eth . Now the problem is that still when i use mst status i am sill not able to find any devise, so i believe once i move the card to unraid it will still be on infiniband Edited January 22 by simo241 Quote Link to comment
simo241 Posted January 28 Share Posted January 28 I been able to show the Mellanox 10gb Card trought the below procedures: # lspci | grep Mellanox 01:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX IB QDR, PCIe 2.0 5GT/s] (rev b0) #echo ib > /sys/bus/pci/devices/0000\:01\:00.0/mlx4_port1 #echo eth > /sys/bus/pci/devices/0000\:01\:00.0/mlx4_port2 Configured the Eth1 to work with windows Pc , only 3 uge problem: 1 - After reboot Eth2 disappear , configuration need to be done again. 2 - if i resume from sleep or i dont use server for a while port eth1 is going down. 3 - Transfer from Unraid to Windows is very slow only 120 Mb/s , from windows to unraid the speed is about 1,5Gb/s Quote Link to comment
Bladedude Posted March 1 Share Posted March 1 # lspci | grep Mellanox 01:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] # lspci -vv -s 01:00.0 | grep "Part number" -A 3 [PN] Part number: 779793-B21 [EC] Engineering changes: B-5718 [SN] Serial number: IL271700F5 [V0] Vendor specific: PCIe 10GbE x8 6W New install of Unraid. Adaptor will not link up out of the box. I installed Mellanox Firmware Tools plug-in, and it's still not up. If I reboot into Truenas Scale (I'm wiping my existing Scale and installing Unraid), it came up from the default Truenas Scale installation. Quote Link to comment
JorgeB Posted March 1 Share Posted March 1 33 minutes ago, Bladedude said: New install of Unraid. Adaptor will not link up out of the box. I installed Mellanox Firmware Tools plug-in, and it's still not up. If I reboot into Truenas Scale (I'm wiping my existing Scale and installing Unraid), it came up from the default Truenas Scale installation. Please post the diagnostics, if it works with TrueNAS it should work with Unraid, assuming it's in Ethernet mode. Quote Link to comment
Bladedude Posted March 3 Share Posted March 3 (edited) Diagnostics attached. Yes, it's in ethernet mode. supermicro-diagnostics-20240303-1147.zip Edited March 3 by Bladedude Quote Link to comment
JorgeB Posted March 4 Share Posted March 4 NIC appears to be working correctly, except there' no link, doesn't look like a software problem: Link detected: no driver: mlx4_en Are use using fiber with transceivers or DAC cables? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.