So it's my first network that wasn't plugged in via a telephone cable or a Cat Cable and my first server (Unraid is really cool) I'm really enjoying having a file server and VM's to do my video production remotely from downstairs without making the Downstairs Computer basically unusable. Except all my Traffic, both file and internet is running through my poor Router because my ConnectX-3 cards show as cable unplugged.

 

Downstairs is a https://www.asus.com/us/motherboards-components/motherboards/prime/prime-z690-a/techspec/ and it's in the lowest slot (Windows 10 / PopOS machine)
Upstairs is a https://www.msi.com/Motherboard/Z370-SLI-PLUS/Specification and it's in the lowest slot (Unraid)

 

2x Mellanox CX354A ConnectX-3 FDR Infiniband 40GbE PCIex8
Prolabs OM3-24C-MPO/F-13.72M Fibre Optic Cable
Finisar FTL414QL2C-E5 QSFP 4X14Gpbs FDR Transceiver

 

pic of port 2 https://imgur.com/J7H3hnf Unraid

pic of port 1 https://imgur.com/5QIHNec Windows10 / PopOS (since moved it to port 2 just to try)

 

Transceivers plugged in with a good 'click' and won't pull backwards out without the tab pull

Cable plugged in with a small 'click'

 

Will the 'network display cable disconnected' error show up even if everything is plugged in correctly, but I haven't made a connection to the second computer as a settings error? or is cable disconnected more of an immediate problem for each computer like the hardware is faulty like the cable? I made downstairs IP 10.10.20.196 and subnet 255.255.255.0 rest blank, and upstairs 10.10.20.198/24 in unraid

 

Both cards were flashed and everything on the same windows 10 same day

running mstconfig -d 03:00.0 q on Unraids terminal gives me:
Device type: ConnectX3
Device: 03:00.0

Configurations: Next Boot
SRIOV_EN False(0)
NUM_OF_VFS 8
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)
LOG_BAR_SIZE 3
BOOT_PKEY_P1 0
BOOT_PKEY_P2 0
BOOT_OPTION_ROM_EN_P1 True(1)
BOOT_VLAN_EN_P1 False(0)
BOOT_RETRY_CNT_P1 0
LEGACY_BOOT_PROTOCOL_P1 PXE(1)
BOOT_VLAN_P1 1
BOOT_OPTION_ROM_EN_P2 True(1)
BOOT_VLAN_EN_P2 False(0)
BOOT_RETRY_CNT_P2 0
LEGACY_BOOT_PROTOCOL_P2 PXE(1)
BOOT_VLAN_P2 1
IP_VER_P1 IPv4(0)
IP_VER_P2 IPv4(0)
CQ_TIMESTAMP True(1)

Win10 Downstairs running mlxconfig -d /dev/mst/mt4099_pci_cr0 query gives the exact same results as above. running flint -d mt4099_pci_cr0 query full on downstairs gives:

 

Image Type: FS2
FW Version 2.42.5000
FW Release date 5.9.2017
MiC Version 2.0.0
Config Sectors 1
PRS Name cx3-1_MCX354A_fdr_09v.prs
Product Version 02.42.50.00
Rom Info: type=PXE Version=3.4.752
Device ID:4099
GUIDS: lots of numbers
Macs Lots of numbers
VSD: -blank here-
PSID MT_1090120019

 

Added a couple of other pictures here, but honestly i'm ready to try any advice, the website which suggested ConnectX-3 as something i should get don't really seem to have any help they wanted to give me. I was really hoping these would be plug'n'play and simple, and this has been making me miserable for a month and i don't know how to help myself other than basically buying a second set of everything and swapping it all out, which still might even do anything if it was settings.

 

some device manager pic https://imgur.com/2YqFtRq

unraids network page pic https://imgur.com/KeGAadP

network settings window10 https://imgur.com/G2weRqL

 

Not sure Unraid has a driver for that.

 

Attach diagnostics to your NEXT post in this thread

Here you go!tower-diagnostics-20230703-0017.zip 

 

Edit: I tried to follow along mostly with this Spaceinvader One video about using this brand of network cards with unraid and others 

 

And I have a plugin called "Mellanox Firmware Tools" from "ich777" but I wasn't overly sure if i needed it, or if i did need it if i needed to click anything.

Edited July 2, 20233 yr by Terrace

10 hours ago, Terrace said:

Infiniband

Did you change the NICs to Ethernet mode? Unraid doesn't support Infiniband

Yep, Ethernet mode, both are as above:

 

LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)

 

default was Auto (VPI?)which I read doesnt work, and the third option is infiniband(INF) which I also read wasn't right. both have been rebooted and checked that ETH is set

Edited July 3, 20233 yr by Terrace

Please post the diagnostics.

Are the diagnostics not visible from the last time i posted them? trurl asked me to post them as the first post, and then i immediately posted them. Is my attachment not visible to others?

Sorry, only checked the 1st post and didn't see them there, diags show that the NICs are installed and initialized correctly, just not detecting a link, so most likely a transceiver/cable issue.

Could also be a bad NIC of course, I assume your doing a direct connection, so if one of the NICs is bad there's won't be a link.

If 10.10.20.198 /24 is valid on the unraid and 10.10.20.196 and 255.255.255.0 is valid on the downstairs computer (because i've never ran two computers before, I wouldn't know) and everything else is ok, then I guess i'll have to bite the bullet and buy a second cable. If it's possible it got crushed in transit or if it was never working (purchased used on ebay a year ago, no way of testing it at the time of arrival) I guess if i start with a cable first then atleast I only need to buy one thing to test.

I'll wait a day before i buy just in case any other ideas come up.

IPs are valid.

Trying to find the right cable is a bit of a pain, the ones i'm finding are '8 core' instead of '24 core' and i'm not sure if i need a 'patch cable' or a 'trunk cable' for a direct connection or one of the other types. And then there's 'type B Chanel 1 to Chanel 12' and I guess something else out there. Ugh, i wish the cable I have worked.

 

https://www.fiber-optic-transceiver-module.com/difference-between-fiber-trunk-harness-and-patch-cables.html seems like Patch Cable is fine.

 

Seems like the difference between and A and a B type is important and i have no idea which i need

 

https://community.fs.com/news/type-b-polarity-mtp-cable-user-guide.html I think I need a Type-B Trunk Cable from reading this, I might be completely incorrect though

Edited July 3, 20233 yr by Terrace

You just need a standard Fiber Patch Cable with the needed length.

I think standard could mean either of them, they're just listed on ebay for example 'ect ect type a cable' £60 and 'ect ect type b cable' £60 no colour change on the cable, no big description change, it's just as likely you'd hit an A or a B if a person went with top of page result

My cable just arrived today from China and the network was picked up immediately upon plugging it in, I needed a 12 core cable instead of a 24 core cable, and a type B cable instead of a type A!

Thank you for the help here with narrowing down what could go wrong and the IP range validation and such!