10g Setup Question


crowdx42

Recommended Posts

Hi all,

 so I am in the process of setting up a 10g connection between my main server and backup server. My thought is, what I need is a 10g switch and then connect both servers via the switch and they should connect at 10g, correct?
I am looking at the TP-Link TL-SX3008F which has 8 sfp+ 10g ports, to connect the two servers I believe I can add a couple of 10g NICs with SFP+ ports and then two SFP+ 10g cables.("Mellanox MCX312B-XCCT CX312B ConnectX-3 EN Pro 10GbE SFP+ Dual-Port PCIe NIC")
My router has only 2.5g ports, but I don't think it will impact, correct?? As long as I have the two servers connected via the 10g switch?|

Patrick

Link to comment

In general you are on a good way.

 

Buy one 10Gbe (RJ45) module, put it into the switch an connect it to your router. It will automatically step down to 2,5Gig (but there are no modules on the market that just do 2,5gig because it is just "10Gig with 3 pauses" (data-pause-pause-pause) to allow the usage of less quality cables.

(make sure you activate "flow control" at least for this port!)

 

For the PCs you should consider other types of module. RJ45 gets very hot and consumes more than 10 times the Watts compared to fiber or direct attach.

 

Personally I go for fiber, the modules are rather cheap, the cables (ready made) too and widely available.

I have bad experiences with "direct attach" cables (2 modules with a fixed cable). Only one out of 10 survived here the 1st year (and the survivor is only 20cm "long")

But surely, RJ45 is the worst choice you can do: Most cables dont work (its hard to buy a correct one), very hot and very greedy for current. If the distance is over 30m, its pure luck to get a connection and even more luck if it is stable for a longer time.

 

Link to comment

Thanks for the insight, I can go fiber for the two servers as they are in the same rack. For the rest of the network which I am also in the process of upgrading, my livingroom PC is in a different room and so Cat6/6a is my only choice. In general the livingroom PC would just be used to drop files onto the server and so there would be no long sustained loads on that connection.
I was unaware that Cat6 (or is it just the Cat6 transceivers?) get hot and use a lot more power?
Patrick

Link to comment

No, sorry, you have misunderstood me (usual excuse: "english is not my natural language" 🙂 )

 

What I wanted to tell you is to avoid CAT cables totally.

For short connections inside the rack you can use the short "direct attach" cables (everything <1m is fine, but they also sell you up to 10m ones and I would stay away if > 1m). These are cheap because both needed modules are fixed to the cable, you buy one thing and get 3.

 

But for the PC in a different room you should use fiber if it is possible somehow (good installations have spare pipes where you can sqeeze them through even with plugs attached already)

 

Link to comment

8J9A1639.thumb.jpg.5a361e75563a0e248386ade5c4268591.jpg

I've just made you a pic of my desktop here to explain better what I mean.

on the very left are 6 fiber outlets that go to differnt rooms (4 to the garage with the rack and the other two to cellar and guest room)

next to them are older CAT7 outlets with the same destinations. (I just added the fiber cables last year).

The I have 2 old 1Gbe Switches for handling old devices (partly 100Mbits only like telephones) and 2 10G switches with almost all ports in use.

(all these switches are fanless, so I can use them on my desk. But that is also the heat problem with CATs what I have meantioned. I've started with all CAT modules, the switch overheated. I was ordered to use only every 2nd port for CAT, but it still overheated. So at the end I have almost no fast CAT anymore. the only one remaining is a 2,5G line to my wifes PC. Its a NUC and cannot handle a normal PCIe card because it does not have any slot)

Note the direct attach connection between the 2 10G switches (the uplink to the 1G at the botton is a standard CAT cable plugged into the 1G Management Port of the 10G Switch)

 

Link to comment

So with idle connected Cat to the 10g switch, it overheated? Is it the Cloud Router Switch in the picture? What brand is it? How old? I am wondering if it is just a design flaw in that model. The switch I was considering was the TP-Link TL-SX3008F which has 8 SFP ports but is a full 1U rack size, so it may also have a fan inside to keep it cool, which is not an issue as it would sit below my Unraid server which already has fans that would be loud enough regardless.

 

Link to comment

its a mikrotik, fairly new stuff look at https://mikrotik.com/ the larger one is also rack-mountable (the 4+1 on top is almost the same, just with fewer ports)

They just came out with totally new devices, i havent seen through to the new stuff for now. But I think the new ones are all in the 100Gig region.

 

Almost ANY 10G device has a fan and I dont like them here on my table. So I waited for the first fanless devices.

but beware! these are not for the fainting heart! If the devicename contains an "S" its a fairly simple switch, just plug and play. But with an "R" its a router, having a very complicated os that allows almost anything a router could ever need. The problem is that it becomes so complicated, without a proper training you cant get it up and running (yeah, it contains a "wizzard", but what it creates is more or less a base to start with). Mine are "RS" which means they come with 2 Os-ses and you can select between simple switch and complicated router.

 

It did not overheat with idle lines already, but I tend to transfer data (a lot 🙂 ) over them, so yeah, copying over a few videos made it glow and finally shut off the port .

 

Link to comment

That SX3008F switch is "fiber only" and will not work with RJ45 SFP+ due to the power required by them.  It will only work with optical SFP's (either 1g or 10g), as well as twinaxial DAC transceivers.  

Also something to keep in mind, most 10gbe gear will drop back to 1gbe, but will not negotiate 2.5 or 5gbe rates.  I saw mention of your 2.5gbe router in your OP.  If you have gig internet (or less) that is typically a non issue, but if you have 2gig+ internet that uses a multigig (2.5/5) port, you'll need to find some intermediary solution.


If you want 10gbe throughout the house, you'll need to spend a good bit more money on 10gbe switches that have 10gbe RJ45 onboard, or that support RJ45 SFP's.  Or start pulling fiber through your house.  Single mode fiber is cheap and futureproof.

 

Otherwise, a more basic 12-24 port gig switch that has one or two 10gbe uplink ports for the server would still prove very beneficial.  That would still give the server full 1gbe access to your internet, while allowing full 1gbe access for each of nine other clients. Grab something like a Intel x520-SR2 that has dual SFP+ cages and connect that up to the switch with a 10gbe DAC cable.  Super cheap, low power and reliable.

Link to comment

On the TP-Link switch, I did see one person say the switch was fiber only but I then saw several others say that it worked fine with RJ45 transceivers. I also did a quick Reddit search on the switch and there are also folk using the switch with 10gbe RJ45 tranceivers. From reading on Reddit it seems that it depends more on the transceiver rather than the switch to get things working. Here is the link to the Reddit discussion I read:

 

Link to comment

Btw, there is only a single machine in another room that I would like to hook up, the main 3 machines are all in the same office, so fiber would be an option for those machines. I am in the process of getting new cat6a drops in the room I have that PC, but the main contender for doing the work has stated they don't work with fiber. I am wondering if I bought a run of fiber cable would they just pull it at the same time as the other drop of cat6 and no need for termination etc. 
From this thread I think I will explore that option.

Link to comment

if you plan to pull fiber into your home, consider a "ready made" "build on order" cable. You can tell them to use 8 wires (you need 6 for your 3 pcs in multimode and I would recommend to add a spare pair for the future). They put everything into ONE cable and glue the plugs on both sides and coverthem with a protection box with a hook to pull at the end.

Problem here is that the top with the pull is quite wide (32mm ususally) and, depending on how many lines are put in, also quite long and stiff at that part. This means you need quite large wholes and a lot of space for curves. But only for getting the cable in, afterwards even 20 lines are smaller than 1 CAT cable.

Optical cables also have a kind of "CAT" quality description. Here it is called "OM" (optical media) and today is usually OM3, thats good enough for 10G, but if you install a new cable I would put in OM4 already which is good for 100G and supposely 1000G someday.

I disagree with @Brandon_K about single mode cable. I would recommend multimode. Yeah, you need 2 lines for a connection, but the power consumption is (just a bit) lower and the used frequency is low enough that you can see it (faint dark red glowing) with your eyes already. But the drawback is, you cannot connect a straigth line, you need to do a crossover connection. But patchcables usually know this and do the crossing for you.

 

BTW, you talk about Cat 6e. This is fine for up to 1G, maybe 2,5 and 5G work too. But for 10G you need at least Cat 7, even better choice would be Cat 8.

Link to comment
19 minutes ago, BRiT said:

Cat 6A works fine for 10G.

short sighted yes (or "maybe", or "hopefully"). But if I put new cables into the wall I always take the most elaborated available. They maybe a bit more expensive those days, but I can then usually skip the next one or two speed improvements before I need to exchange them again.

 

At the end "more expensive" turns out to be cheaper and more reliable than "just what I need now".

 

Reminds me of a joke that was going around at my university days in the early 80s:  (electrical engineering)

Job: "Build a power supply that delivers 10 Amperes with 5 Volts"

 

Chinese Solution: "calculate the needed parts, subtract 15% from the values and use the cheapest parts available". Result: works! But only between 3 Months and the end of the guarantee (1yr or 2yrs). Then blows up in a massive explosion. (even with 30% to exchange they make money). Replacements !?!?!? never heard that word, the next day a new model is in production.

 

American Solution: "calculate the needed parts, use the resulting value, leave out any "not needed" parts". Result: works! Survives the guarantee time an maybe another year or two, then dies without big bang. no replacements or repairs available, 3 new models are out already.

 

German Solution: "calculate the needed parts, add 25% as a reserve to any part". Result: works!, after 5yrs: still works. (three times the price of the chinese one of course), after 15yrs, most still work you can still get replacements or parts.

 

I'm german 😁😁😁

Edited by MAM59
Link to comment

So the more I think about this the more I think for the moment I will have cat6a used for the new drops, if in the future they need to be upgraded I can do that myself, once the new drops are in plae, it is not a huge job to pull new cable with the old drops already in place.
I also think I may just go with 2.5g NICs and switch, I have watched my main PC simply copy files between two spinning drives and the speeds do not justify a 10g network right out the gate. 
Thank you all for your input :)

Link to comment

these are ok (though I dont know if they can do 2,5 too, it is such a stupid speed, not common in real life).

But what you see is a very very old and outdated version of that card. It needs 8 Lanes of PCIe 2.0 to run. 8x slots are rare in customer PCs, most of the time you have to use the 16x slot you usally need for the video card. Server motherboards had 8x slots.

 

But there is a newer version of this card using 4 lanes PCIe 3.0 slot connector. Most of them only have one port, but that does not real matter. Dual port cards are barely useful in client computers.

 

Link to comment
6 hours ago, crowdx42 said:

Can you recommend a card that will work with PCIe 3.0?

its the same card, just a different version.

But if you have a spar 8x slots to offer, dont worry and just use it.

(I dont use ebay, so I cant give you a link)

 

 

Edited by MAM59
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.