10gb switch

[PhantomICEMAN]

Hi im in the market for buying a 10 gig switch with a minimum 2 10gb ports. I thought id found the ideal switch for my specific setup in  the QNAP QSW-2104-2T but reading the review below has got me really concerned , im sure ive read something similar in other posts..I predominatley need 2 10 gig ports, the rest can be 2.5gb . I dont necessarily need all 10gb as i will only have 2 devices that will make use of it an i dont want to spend extra money on something , when that money could easily go towards a 10gig card. Can anyone recommend a reliable switch to go for, that isnt going to start playing up after some use.

 

 

[MAM59]

What people forget, or do not know at all, heat is bad! And 10G over Twisted Pair cables produces a lot of heat (and draws a lot of electricity). So it is not uncommon that fanless switches with 10G get crazy if used heavily.

Also these "lockups" are normal if you use not very good cables. 10G depends on flow control, and if there is a transmission error and a port was stopped before, it may never get the "you may continue now" message, resulting in a full stop until the cows come home.

 

Also this small 2+4 Switches are usually not fast enough to allow full speed between the 2 10G ports. Expect not more than 6,5G here (the used chips are mostly still equipped with a PCIe 2.0 bus interface, they cannot run faster)

 

All switches in this class are this way. If you want better performance, you have to look higher (in price and in hardware).

 

But with "normal" usage, nothing bad will happen, just do not stress them too much... Remember, its the "toy class"...

 

[BRiT]

1 hour ago, PhantomICEMAN said:

Hi im in the market for buying a 10 gig switch with a minimum 2 10gb ports. I thought id found the ideal switch for my specific setup in  the QNAP QSW-2104-2T but reading the review below has got me really concerned , im sure ive read something similar in other posts..I predominatley need 2 10 gig ports, the rest can be 2.5gb . I dont necessarily need all 10gb as i will only have 2 devices that will make use of it an i dont want to spend extra money on something , when that money could easily go towards a 10gig card. Can anyone recommend a reliable switch to go for, that isnt going to start playing up after some use.

 

 

I would advise to look at Serve The Home as they have reviewed a metric ton of mixed 2.5G / 10G switches which includes 10 GBase-T and SFP+ model variants.

 

[_cjd_]

rj45 or SFP+? ... if you can, the latter is a MUCH better way to go if it's an option.

I use a Unifi Aggregation switch - 8 SFP+ ports. I do have two UI rj45 adapters (one only does 10g, the other will do 2.5/5/10g but running at 10) for a couple use cases where DAC or fiber weren't an option - one fiber DAC, the rest close enough for copper DAC. All ports full, no heat issues, no stability issues. It just works.

[PhantomICEMAN]

Its most likely going to be RJ45 though if theres other options im willing to explore them, i wouldnt mind exploring fibre in the future.I dont have a whole lot of devices to connect to it.I have wondered what the difference is between having an SFP+ port over a standard port, besides being able to change betwen connectors are there any other performance benefits. All i currently have is my main machine my unraid server and my work laptop and i have an ethernet going to my ISP router (2.5gb port) that brings my internet feed and i can use it to stream my plex library. I primarily want the 10gb connectivity between my unraid sever and machine. Hence investing in a expensive large switch wouldnt make sense. I did have a look at the mickrotik switches though im not very savvy when it comes to adminstering a network hence why i opted for a unmanaged switch. Is there any particular one you would recommend.

[MAM59]

8 hours ago, PhantomICEMAN said:

I did have a look at the mickrotik switches though im not very savvy when it comes to adminstering a network hence why i opted for a unmanaged switch

They come with two different OSs from which you can pick the one you need.

"RouterOS" is hardcore, you can do almost ANYTHING what you need to set up a router. All that complicated stuff, and even for a network engineer it is not possible to be handled without weeks of additional training.

"SwitchOS" is simple, it just offers stuff you need to build a switch. If you leave everything default, you instantly get something like an unmanaged switch up and running. But you can control certain stuffs like bonding/aggragation, Flow Control, Speed selections and so on. There is (hope it still is, very old already, maybe not build anymore?) a simple and cheap 4+1 (4*SFP+, 1*1G) one called CRS-305 or his bigger 8+1 brother (8*SFP+,1*1G) CRS-309. I use both of them, they are great, fanless, but sensible to overheating if you use RJ45 Modules (can only use maximum of 2 on the smaller and 4 on the bigger one, due to heat problems. all ports can run fiber or direct attach without heat problems).

But they may have newer devices already, it helps to understand the naming scheme: "R"=RouterOS, "S"=SwitchOS "3"=type "05"/"09"=number of ports.

For your needs, you can avoid Devices with "R", they offer some devices only with one OS "CS" or "CR". They are a bit cheaper then compared to the same Device with "CRS" multi-OSs.

 

[PhantomICEMAN]

On 3/7/2024 at 6:03 AM, MAM59 said:

They come with two different OSs from which you can pick the one you need.

"RouterOS" is hardcore, you can do almost ANYTHING what you need to set up a router. All that complicated stuff, and even for a network engineer it is not possible to be handled without weeks of additional training.

"SwitchOS" is simple, it just offers stuff you need to build a switch. If you leave everything default, you instantly get something like an unmanaged switch up and running. But you can control certain stuffs like bonding/aggragation, Flow Control, Speed selections and so on. There is (hope it still is, very old already, maybe not build anymore?) a simple and cheap 4+1 (4*SFP+, 1*1G) one called CRS-305 or his bigger 8+1 brother (8*SFP+,1*1G) CRS-309. I use both of them, they are great, fanless, but sensible to overheating if you use RJ45 Modules (can only use maximum of 2 on the smaller and 4 on the bigger one, due to heat problems. all ports can run fiber or direct attach without heat problems).

But they may have newer devices already, it helps to understand the naming scheme: "R"=RouterOS, "S"=SwitchOS "3"=type "05"/"09"=number of ports.

For your needs, you can avoid Devices with "R", they offer some devices only with one OS "CS" or "CR". They are a bit cheaper then compared to the same Device with "CRS" multi-OSs.

 

Cheers mate i think im going to go for the CRS 305 its nice and compact,can support 10g accross all port and most importantly its cheap. I did look at the unifi flex switch which is close to £300 not factoring the cost of the nics.ive got some CAT 8 but i think im going to down the fibre route as its cheaper and lower latency.Im probably looking at two fibre dacs, and two RJ45 (1 running 10gb spf+ running at 2.5gb to start off with and one that can a be a gig).  I assume based on your above comment that fibre isnt conducive to the same heating issues as RJ45 both on the switch side as well the nic.I need to find a decent spf+ nic that will work with windows 11 and of course unraid. Any recommendations?

[MAM59]

25 minutes ago, PhantomICEMAN said:

decent spf+ nic that will work with windows 11 and of course unraid. Any recommendations?

best choices are used "Mellanox Connect-X3" cards. There are several different versions, the very old one uses a PCie2.0X8 interface, the newer one a PCIe3.0X4 Interface. Make sure you have an appropriate spare slot available on your Motherboards. They are rock solid.

Either on Windows,

or on UNRAID:

 

There are other cards from ASUS or other vendors. Most of them use an Aquatic Chipset. I would not recommend these for server usage, they do not offer several features like buffer controls, virtualisation and so on. The Mellanox is a pure server card, a bit of overdressed for windows, but of course, works well there too.

 

BTW1: DAC connections are the cheapest of all (because they contain both modules fixed to a single cable) and work very well on shorter distances (3m is ok, more is risky).

BTW2: there is no special 2,5 RJ45 Module for the Mikrotik, only a 10G one is available. Rather expensive and very hot! This is because neither 2.5G or 5G really exist. They were "invented" after it turned out that some hardware was not able to run the full 10G and before they threw the chips away they relabled them. 2.5G is "10G with 3 pauses" in reality. The pauses are inserted to give the chip and the cable time to calm down.

Dont wonder why a 2.5G connection is still shown as 10G in the microtik status screen.

 

 

I have used Intel X540 and X550 cards before, they were problematic. Either instantly, or they have worked for some weeks and suddenly showed link losses, line resets and so on. I have all kicked them out, went to Mellanox and they are running smoothly for years now.

 

Edited March 9 by MAM59

[wuudogg]

Excelent advice here.  Also try to use passive DACs for your short connections as it will result in significant power and heat savings.

[MAM59]

One Addon: its cheaper and more reliable if you buy a 2nd 4*2.5+1SFP+ unmananged switch (they are less than 70€). Connect one SFP of the microtik with a 20cm DAC and you have 4 RJ45 Ports that work well and cost less.

Looks like this

(Sorry its a bit full in the moment, I am building a new PC currently, the desk is full of new parts :-)))

You see 2 Mikrotik 309 (connected with 2*DAC as a trunk to each other) and a cheap 4*2.5+2*SFP noname unmanaged switch on top.

As long as there is ONE managed switch, flow control will work because the unmanaged ones pass it on.

 

[PhantomICEMAN]

Cheers mate your advice is invaluable..  I'm going to start pricing this up and get ordering, going of that picture looks like your using a hell of a lot of fibre in your setup.. Does the 309 offer anything more over the 305 besides the additional ports. Also are you able to manage the switch via the sfp ports or do you have to plug something into the dedicated management port to access the management software?.

[MAM59]

1 minute ago, PhantomICEMAN said:

Does the 309 offer anything more over the 305 besides the additional ports. Also are you able to manage the switch via the sfp ports or do you have to plug something into the dedicated management port to access the management software?.

No, just more ports and different case (also rack-mountable). You can configure from which ports configuration is allowed, by default its "ALL".

The "management port" is just a 1G port, not dedicated. Can be used for Devices too.

But of course, in a "serious installation" the management port belongs to a different vlan and admin access is restricted to this port only.

 

But, be aware. The devices are always shipped with routerOS. You need to start it up with this, then search through the complicated menus until you can select "boot with switchOS". Its not that easy, but once you have made it, things become simple.

 

[Zonediver]

On 3/4/2024 at 8:05 PM, PhantomICEMAN said:

I dont necessarily need all 10gb as i will only have 2 devices

 

If so, for what purpos is the switch?

You can directly link both machines with additional 10GBit-NICs and done 👍

[PhantomICEMAN]

sorry as in i will only have 2 evices that will be 10gb abled. my work laptop is 1gb and i have a CAT 6A cable going to my internet router which is 2.5gb enabled. essentially i will be using all 4 ports on the switch.

 

Im just looking at some mellanox cards, i noticed they selling them on Aliexpress for a reasonalble price but are they legit or do you have to look for ones exclusively pulled from a server.Its always minefield buying these type of cards as you never know if they are cheap replicas or the real deal.

[MAM59]

I dont know about any replicas, but yeah, I would not buy from aliexpress or ebay at all too.

 

There are special shops that sell "used server hardware" all over. Here in Germany I prefer HaenleinIT. You can find them on Amazon https://www.amazon.de/s?me=A2TDDLK6I16CEM&marketplaceID=A1PA6795UKMFR9

 

Direct link is https://www.its-haehnlein.com/Shop/ but prices are identical.

 

They offer a nice bundle with 2 cards and one DA-Cable for less than 130€. Remember: you want the PCIe3.0x4 ones!!!

 

But I am sure, there is a alike shop near you too, search for "used server" or something alike.

 

Edited March 9 by MAM59

[PhantomICEMAN]

Would this work mate , Its x8 not not x4 but i know these guys are legit and they based not to far from me either. My motherboard on my main machine has 3 pci 4.0 x16 ports one of them has got my 3090 in it  the rest are free.

 

https://www.bargainhardware.co.uk/mellanox-connectx-3-pro-mcx312b-xccs-dual-port-sfp-10gbe-fh-pcie3-0-x8-ethernet

[MAM59]

the problem with these 3*16 ports is that they usually are fakes.

For instance, i have 3*16, BUUUUT: if I put a card into the 2nd slot, slots 1&2 become 8x only. and slot 3 is 4 lanes only (or even 2 lanes only if i add the 4th nvme drive).

 

So read your MoBo's manual carefully! Servers have 8x slots, but normal motherboards are much more limited. Thats why you should get a 4x card, they work almost everywhere.

 

 

[Zonediver]

8 hours ago, PhantomICEMAN said:

Im just looking at some mellanox cards

 

I cant recommend this cards because the cheap ones are old and used.

I would use 2x TP-Link TX401 and a propper cable. 

This card is similar to the Asus XG-C100C what i use but with a WOL-function.

The Asus 10GBit-NIC is working since Oct.2019 without issues.

Edited March 10 by Zonediver

[MAM59]

5 hours ago, Zonediver said:

would use 2x TP-Link TX401 and a propper cable. 

This card is similar to the Asus XG-C100C what i use but with a WOL-function.

I have one of them onboard in my new pc I am currently building on my table. But I guess, it won't last for long. No native drivers in Windows. And (mine) is RJ45 only, once more a risky connection. All these cards are Aqantic-based. (of course there is a windows driver, but you have to download and install it manually).

I will disable it and use an "old and used" Mellanox with fiber instead.

[MAM59]

Just an Update: I've stresstested the onboard Aquatic 10G Card (Mobo: ASUS Proart 670 WiFi), but it did not long until she freaked out, reseted line and finally blocked the port completely. Did not even took an hour to kill her.

It "works" until you put heavy load (100Gig+ Files) on her.

 

Will be disabled and replaced by a just ordered "new" Mellanox X3 with SFP+

 

[MAM59]

@Zonediver: what works for you, does not work for me. This does not mean one is right and the other is wrong. Maybe there are different situations/settings/traffic that have impact on the result. I dont know, I can only say what happens here.

[PhantomICEMAN]

On 3/11/2024 at 10:15 AM, MAM59 said:

 

 

Will be disabled and replaced by a just ordered "new" Mellanox X3 with SFP+

 

Where did you get it from mate?

[MAM59]

3 minutes ago, PhantomICEMAN said:

Where did you get it from mate?

as told before already: from amazon (marketplace of the used hardware shop).

Was here within a day and is "used" (but still shrink wrapped).

Plugged in: works like a charm (driver is included in almost every os).

 

[PhantomICEMAN]

no worries just wondering if you sourced them from elsewhere. I've found the deal you are referring uptop.  just wanted to ask do you to have to use specific sfp transceivers  with the card.  I've read on another post you need a qsfp one. can you suggest a fibre aoc that will go with this card or would a standard dac be sufficient. Some of these cables are expensive and  I  don't want to order the wrong one.

Edited March 12 by PhantomICEMAN

[MAM59]

8 hours ago, PhantomICEMAN said:

just wanted to ask do you to have to use specific sfp transceivers  with the card.  I've read on another post you need a qsfp one. can you suggest a fibre aoc that will go with this card or would a standard dac be sufficient.

The X3 is 10G only and uses SFP+ cages. "normal" DACs and modules are used.

But she is not built anymore and the successor, the X4, uses QSFP cages and goes from 25G up to 100G. These are really expensive.

 

The X3 is the most used card of the 10G era. Almost EVERY (SFP+) module from any vendor works with her. You only need to avoid modules that at labled "HP only" or something, but they are rare.

 

DAC cables of shorter length are usually passive and cheap. If a DAC is expensive, you shoul avoid it. Fiber Modules are usually the cheapest of all (but you need 2 of them and the cable too, so in sum, DACs are cheaper). For normal inhouse usage Multimode Modules with up to 300m cable length are best.  I usually buy modules from 10GTek, good quality, good price and I had only one that broke after some months. But just talking with the support girl, sending her a photo of the barcode of the module brought me 2 replacement modules within a week (She said, they have no single module packages, thats why I've got 2). Bougth them also on Amazon (before you ask).

Edited March 13 by MAM59