10gbe...how in the world do I do this?

[Johnny Utah]

I have searched forum after forum and watched video after video but still cannot wrap my head around this process.  My goal is to simply get a direct 10gb connection between my Unraid server and a Windows PC for file transfers.  I see everyone talking about Mellanox cards and HP cards with SFP, copper, and/or Fiber and so on.  Is there any plain jane easy to understand, explain to me like I don't know what a computer is, answer to the question?  If I bought 2 Mellanox x2 cards and placed one in each of my machines, which cable do I need to connect the two if they are roughly 60ft apart?  Do I need a transceiver or adapter of some sort or something on each end?  What is the difference between SFP, SFP+, and QSFP?  What is OM4, OM3, and OS2?  Please help a desperate man lol!

[tjb_altf4]

For your distance, fiber is the appropriate choice, direct attach copper (DAC) is a cheap and effective alternative when dealing with short distances.

This video should cover the rest.

 

 

 

[Ford Prefect]

17 hours ago, Johnny Utah said:

If I bought 2 Mellanox x2 cards and placed one in each of my machines,

...what cards exactly are you refering to.

A Mellanox ConnectX-3 is most likely your best and cheapest option...they are PCIe-3 x4 models.

ConnectX-2 are older and PCIe-2 x8.

17 hours ago, Johnny Utah said:

which cable do I need to connect the two if they are roughly 60ft apart? 

You cannot use a DAC Cable for this length. You should either go for Copper (RJ45, CAT6a) or Fiber.

 

17 hours ago, Johnny Utah said:

Do I need a transceiver or adapter of some sort or something on each end? 

Yes, you need a tranceiver for the correct "technology" in between both endpoints.

Just be warnded. Although a RJ45 copper tranceiver is tempting, as cabling in between is easy, note that these get quite hot as they consume a lot of energy, compared to fiber based ones.

Also each Transceiver needs to be "coded" in its Firmware, identifying it as compatible with the make&model of the card.

For Mellanox Cards, of course mellanox but also cisco coded transceivers will work fine.

17 hours ago, Johnny Utah said:

What is the difference between SFP, SFP+, and QSFP?

SFP is for 1Gbps, SFP+ for 10Gbps (downward compatible with SFP tranceivers).

QSFP is Quad-SFP, as well as QSFP+ is Quad-SFP+ ... for times the speed of single link.

For these, you also can get breakout-cables, so one QSFP+ Port can be transfered / made into a fan-out of four SFP+ links, 10GBE each.

17 hours ago, Johnny Utah said:

 

  What is OM4, OM3, and OS2?  Please help a desperate man lol!

These are different fiber optics standards/grades....also there are single-mode and multi-modes fibers.

Depending on the standard used, you can reach for longer or shorter range....but we are talking miles, not feet here.

With a fiber transceiver, you need a match for the required optics/cable - for example OM4/multimode - on both sides to get a working link.

 

For a pair of mellaonox cards, and fiber connection, you would want something like this:

 

Tranceiver(s): https://www.fs.com/products/65334.html?attribute=106&id=219237

Fiber-patchcable: https://www.fs.com/products/40256.html?attribute=213&id=163177

[Dan M]

Bumping an old thread....

I'm wanting to go the 10Gbps route as well.  Going Fiber is blazing a new trail for me - lots of technology I'm not familiar with.  It's a long way from the old 2.5Mbps ARCNet I used to run.

I would like to have 10G NIC in my Unraid server as well as my main Media Server/Workhorse.  I would also like both to have access to everything on my 1G LAN and vice versa.   Since I don't have a lot of PCIe slots available on my main Media Server (I only have 3, and the GPU with the cooling fan takes up two slots, leaving me with a single useable PCIe X4 slot).  The Unraid server has 4 (1 being used) PCIe X16 slots, 1 PCIe X1 slot, and one plain old PCI slot.  
So this is what I was thinking.

10G NIC in Unraid server  (Melllanox MCX353A-QCBT CONNECTX-3)
10G NIC in my Media Server  (Melllanox MCX353A-QCBT CONNECTX-3)
Both connected to a 10G Switch which also has 1G ports. (10Gtek 5-Port Fast Ethernet Desktop Fiber Switch, with 2 Ports Dual SC Fiber - https://www.amazon.com/gp/product/B0895XB1SD)

First, will that work?

Second, I understand I'll need some kind of transceiver to slide into the Mellanox cards.   And I'll need some cables (thinking 2M) to go from the Mellanox/Transceivers to the switch. 

What would I need?  

 

I suppose, though, it would be less expensive to directly between the two computers at 10G and use the built-in NICs on the motherboard (Realtek) to connect to the 1G network.  In that case, I believe it would be easier to wire, but again, what would I need?  Transceivers and cable.

 

 

[Aran]

IMO, it's easier to just connect your workstation directly to you server. I have those asus 10gbit xg c100c cards in my two systems connected with a cat6a copper cable.
Works as it should.