10gb switch

[PhantomICEMAN]

Evening . So I received the x3 cards today,  the next thing on the shopping list is the switch and the rj45 spf transceiver. Just had 1 quick question that  I touched upon earlier.  I've got my sights on crs305 and I'm planning to get a 10gb spf to connect my broadband feed with the switch. The switch will have a 10g connector but the broadband router only supports upto 2.5. Will it a negotiate okay and  what speed will it actually run at ?

[Amane]

Hi PhantomICEMAN

Up to 10Gbit is possible locally if all the entire hardware supports 10Gbit, on the Internet (router) only 2.5Gbit is possible. Basically all speeds are compatible. You just have to make sure that there are no bottlenecks anywhere..

 

10Gbit cards can get very hot, I always have bad experiences with temperatures. (If you have problems, keep this in mind, when debugging.)

 

Greetings

Edited March 19 by Amane

[MAM59]

7 hours ago, PhantomICEMAN said:

Will it a negotiate okay and  what speed will it actually run at ?

I guess I have written it above already? If not, I repeat:

 

The CRS will negotiate 2,5 speed, but will always show 10G. You have to make sure, that both partners have activated "flow control" on these ports because 2,5 is just "10g with pauses". And they need to be aware which part of the frametime is data and which ones are pauses.

 

And again the warning: 10G RJ45 SFP+ will get very hot!!! Never even dare to try more than one or two in that switch, even with an external fan it will crash the port after a period!!!

Think of buying a seperate cheap (they are <70€ now already, cheaper than the RJ45 SFP module!) 2.5 switch with one or two SFP+ ports. Connect this to the CRS via a cheap direct attach line. Gets you rid of the heat and also handles the flow control.

 

For real 10G connections use DA or fiber if possible. Longer RJ45 connections will fail, not directly but after a time they will become unstable and you will get more and more strange errors...

Edited March 20 by MAM59

[PhantomICEMAN]

Evening so i have finally set everything up this evening so the way eveything is setup is the ethernet for my internet goes into a nicgiga cheap switch the same as yours mame and then i have an spf+ dac line that goes into the CRS305 switch(running switch OS). i then have two spf+ dac lines going to my main machine(windows 11) and my unraid server connected to via mellonox connect-x3 cards on both sides. I have flashed the latest firmware on both cards to ensure they are upto date.However the issue i have found is that i am not hitting the 1+Gigabytes a second 10gb is supposed to acheive. I seem to hover beween around 90mb to 700mb , mostly the lowest figure but cannot understand why. Both machines have a very fast nvme which is the WD SN850, unraid is set to write to nvme cache pool before moving to the array. Im i missing something.. why am i not able to hit the higher speed and hold it consistently instead of it dropping to double digits most of the time.Ive check in unraid and it says 10000mb full duplex and windows says my connection is 10gbs. I have also ran iperf and that stated that i had just under 10gb worth of bandwidth so unsure what im missing.

Edited April 12 by PhantomICEMAN

[MAM59]

Patience 😁

What you see is the difference between gross and net.

You HAVE a 10G connection, but depending on the files you transfer, you do not reach the upper end.

Small files will be slow because the overhead needed to adjust the folder content.

Try to transfer a 1G+ file (a movie for instance).

Also note that heavy load on UNRAID (like dockers or VMs) can slow down the whole machine.

 

To be sure that all connections are ok, take a look at the status & error pages of the CRS305. If errors sum up a lot, there is a problem and if there are a lot of timeouts on the stats page, you know that something slows down the net (don't panic if this port connects to the 2.5G switch, it is totally normal that it stops the 10G line quite often, the cheap switches do not handle speed correction themselfs, they put the load onto the upper switch)

 

Also keep in mind that high speeds only can be achived when writing NEW files to UNRAID only. File on and from the array are limited to real disk speeds.

 

Better you create a new share ("TEST") directly and only on the NVME and use it for speed tests. This should work in both directions. But do not expect to see 1,1Gb/s, 700-900Mb/s are ok.

 

[PhantomICEMAN]

Hi Does anyone know how you can enable the Wake up on Lan on the connect X3 card on unraid. I usually put the server to sleep during periods when its not being used and re-awaken it using an app on my phone. I was able to do it on the ethernet  port thats on the motherboard but cannot do it with the card. i have seen some instructions to enable it  that are somewhat convoluted . Does anyone have anything simple that works.