I'm building my second unRAID server as a backup to my primary unRAID server. I want to have a point to point netword connect between the two servers just for backups. The initial backup is going to talk a long time, ~100TB, and the faster connection the better.

 

I've seen many 40G Mellanox NICs on eBay under $40 and looking for people with first hand experience with 40G. Would like a card that's reliable and doen't require additional configuration. Model number would be appreciated.

 

TIA,

 

Mike

40GE can be a bit of a rat's nest.  The physical standards are primarily 4-lane interfaces (4 copper lanes, or 4 optical lanes).

That usually makes the transceivers a bit expensive.

 

Many of the initial deployments of 40GE used it as 4 independent 10GE links with a special break out cable arrangement.

This was done as a way to increase the density of 10GE interfaces on a piece of equipment, not as 40GE links.

 

For 40GE use those usually had link aggregation (x4) to treat the bundle as a single 40GE point-to-point.  If your application

is as a single-40GE-link (which is what it sounds like), you need to be concerned with compatibility of the two

ends, both physically and with link aggregation compatibility.  Expect to need a fair amount of configuration effort.

 

There were a number of link aggregation issues back in the 2010-2015 time frame where the two ends just would not

talk between vendors.  Whether older equipment was eventually harmonized I don't know (and is probably vendor and

model number dependent).

 

-- Tom

 

 

Tom3,

 

Thanks for taking time to reply.

 

My use case is point-to-point (no switch). I wonder if you could explain your statement about "compatibility of the two ends". I plan to buy two identical NICs and configure them the same. I assume two identical cards would be compatible with each other. Maybe you mean there could be issues in the configuration beyond what can be configured in the unRAID GUI.

 

Also, thanks for explaining that these cards are not a single 40G link, but rather 4x10G links aggregated.

 

I did find this and ran it thru Google Translate. The OP lists Mellanox 40G cards in the setup and I have to assume they work.

 

 

Mike

Edited April 20, 20233 yr by subagon

Hi Mike - early on, the equipment on each end used Link Aggregation to package 4 10GE links as a single 40 GE link.

This is where some issues cropped up.

 

The ratified version of the 802.3 ba standard (June 2010) for 40GE & 100GE inserted something called

Multi Lane Distributor (MLD 64/66b) to handle all this in hardware chips on the Ethernet ASIC

interfaces ahead of the physical interfaces. (The physical interfaces generally don't know about this).

Your Mellanox interface is one end, the ethernet switch that the cable is plugged into is the other end.

 

So the question is what is the age of the equipment on each end of the link?  If both ends are listed as

being 802.3 ba compliant, then you should be OK.  Practically most equipment took several years after the

standard rataification to adopt everything.

 

-- Tom

 

On 4/19/2023 at 7:38 PM, subagon said:

I'm building my second unRAID server as a backup to my primary unRAID server. I want to have a point to point netword connect between the two servers just for backups. The initial backup is going to talk a long time, ~100TB, and the faster connection the better.

 

I've seen many 40G Mellanox NICs on eBay under $40 and looking for people with first hand experience with 40G. Would like a card that's reliable and doen't require additional configuration. Model number would be appreciated.

 

TIA,

 

Mike

I've had a good experience with MCX354A-FCBTs (likely the dirt cheap model you saw on eBay). I haven't been able to test file copying accurately because my NAS is currently on a pretty outdated CPU, but I'm getting 28-32Gbit over iperf3 (MTU 9000, 10 parallel streams, virtualized client). Keep in mind that the cards aren't supported anymore, so if any issues were to pop up, you'd be out of luck in terms of updates or support.

Thanks guys for the replies. For now I'm going to wait on any 40G NICs. Two weeks ago I bought 2 Intel X520-DA2 NICs (10G) and then I saw the 40G NICs at about the same price. I just assumed anything faster that 10G would be more than I wanted to spend (wrong). Anyway, yeaterday I received a second cable for the X520's and bonded the two interface to get a 20G point-to-point connection. I think that's good enough for what I have planned.

 

Mike