Afternoon all. 

 

Recently got in to the unRAID realm and have 2 unRAID servers (replacing my very old Synology setup) currently with a 2.5GB link between them both. Main server running all docker containers and VM's and replica server purely for file backup. Both WILL be located in different buildings (Currently sit next to each other connected to a new Yuanley 5 port switch for initial testing and config) but want to upgrade the link between the 2 unRAID servers for 10GB networking and later on down the line my workstations. 

 

My Gateway is a Dream Machine Special Edition with a 1GB up and down internet connection but next week will be upgraded to 2GB up and down so now looking to upgrade my home lab networking that has not really been touched for over 7 years other than a few Unifi WiFi 6AP's.

 

I have invested in Unifi over the years and just put an order in for a USW-AGGREGATION switch and will be using this as a centre point to connect switches and my 2 unRAID hosts. Due to initial spend for the 2 new hosts and HDD's, can't afford the top tier switches Unifi offer so got some temp 2.5GB switches to help me along.

 

So main USW-AGG connecting via 10GTek SFP to

 - 10GB SFP YuanLey 9 Port 2.5G PoE Switch

 - 10GB SFP 6 Port 2.5G Umanaged Ethernet Switch

 - 10GB SFP Unifi UDM SE

 - - 1GB SFP connection to existing 8 port Unifi PoE switch Office

 - - 1GB SFP connection to existing 8 port Unifi PoE switch Garage

 

Now getting to my main question, what would be a good SFP nic run in my 2 hosts that will be compatible with unRAID. I have a spare PCIe slot in both of the hosts and just need some recommendations on known working/reliable cards I could use.

 

If anyone can help out with some recommendations, that would be great.

 

Thanks,

 

Hal 

1 hour ago, hallamnet said:

Both WILL be located in different buildings

 

What is your plan for getting 10GB link between the different buildings?

whatever you buy, be prepared that both of these switches never will transfert 10Gbe. Tests here in Germany have shown a maximum throuput of around 6,6Gbs. The cheaper ones ran even slower than that.

 

For longer lines forget about Twisted Pair cabling directly, between the buildings you will need fiber. Up to 300m are done even with cheap modules.

 

Edited July 28, 20241 yr by MAM59

Depends on the price you got, Mellanox ConnectX3 or Intel 82599 is my favourite.

 

Recently use back us-16-xg because it have 12 SFP+ port, also add some SFP+ Ethernet converter, so now have 2.5Gbps device running in the network and upgrade switch trunk from 1G to 10G ( use existing 15m ethernet CAT3 cable )

Edited July 28, 20241 yr by Vr2Io

I have Mellanox Connectx3 and Intel x710-da2. Both worked out of the box, whether copper or fiber DAC to a Unifi USW-Aggregation. I also have my backup Unraid and a windows system connected via cat6 and a Unifi 10gbe SFP+ adapter - though hoping to run fiber when we have to open up walls for a 50a to the kitchen as those adapters are far more power hungry...

I had to tune the windows/rj45 (forget which Intel card I have there, x550 I think) but with jumbo frames set up properly I see transfer rates approach theoretical max in iperf3. Backing up Unraid to unraid is disk speed constrained - it doesn't need the 10gbe for the main array. Not using jumbo frames drops to ~6Gb

I got the Intel x710 as it supports lower power states better.

Hey all and thanks for the replies. 

 

@JonathanM - I have bought some external grade fibre to run from my comms cupboard out the side of the building in a cable channel, down in to some underground ducting and in to outbuilding. Bit of a palaver but currently having some ground work done so included it in that.

 

@MAM59 - I know these aren't the best options but when I have the chance to sort the switches out, will be going for Unifi switches but for the time being, these will kind of be ok.

 

@_cjd_ - I know the disks are going to be a limiting factor but good to have the core cabling installed ready for when (if I do it) I upgrade the HDD's to SSD's/NVME's.

 

Funds are a bit tight present but just want to find out what is known working so I can then carry out my own research on compatible chipsets and cards. 

 

 

Just to update this thread. 

I have bought some Intel X520-DA2 Dual port 10GB SFP+ PCIe Adapter's and they seem o be working AOK. Just had 2GB internet installed and I can run a speed test to it's ull potential so happy about that. Only limiting factor now is my Spinning Rust. I am doing away with a lot of my older drives and updating to Seagate Exos x18 18TB drives. i know I am not going to be getting a massive speed difference over my older 8 year old WD Red's but it will help a bit. 

 

Both NAS's have these cards installed along with my 3 active workstations and all terminate back to my Unifi USW Aggregation.

All in all, not too bad on a bit of a budget. 

On 7/28/2024 at 9:17 AM, Vr2Io said:

Depends on the price you got, Mellanox ConnectX3 or Intel 82599 is my favourite.

 

Recently use back us-16-xg because it have 12 SFP+ port, also add some SFP+ Ethernet converter, so now have 2.5Gbps device running in the network and upgrade switch trunk from 1G to 10G ( use existing 15m ethernet CAT3 cable )

Since I have all the new expansion slots on my new board, I'm going to get an SFP+ card to take over the non-IPMI on my unraid server.  I see people keep recommending Mellanox but I'd probably rather go with Intel.  What would your choice be for a single direct connected SFP+ card to my switch with a SFP+ interface?

 

This is the one I was planning on picking up: https://www.amazon.com/gp/product/B06WWB5RT7/ref=ox_sc_saved_title_2?smid=A2WW81I15Z72MC&th=1

Edited September 17, 20241 yr by prongATO

4 hours ago, prongATO said:

Since I have all the new expansion slots on my new board, I'm going to get an SFP+ card to take over the non-IPMI on my unraid server.  I see people keep recommending Mellanox but I'd probably rather go with Intel.  What would your choice be for a single direct connected SFP+ card to my switch with a SFP+ interface?

 

This is the one I was planning on picking up: https://www.amazon.com/gp/product/B06WWB5RT7/ref=ox_sc_saved_title_2?smid=A2WW81I15Z72MC&th=1

As said, I haven't found much different between Mellanox between Intel. I have two Mellanox ConnecX3-pro and 3 or 4 82599, the 2nd Mellanox was used and I buy recently, I haven't put it to live because no immediate benefit. But when I test it I got TWO trouble to make it work, after half-day work, I identify it is because SRIOV issue, when it disable in BIOS ( or enable in BIOS, I can't remember that ) then its driver will stop working, Another trouble relate to mobo, when it connect the NIC in PCIe 3.0 it will got error but no error if I downgrade the slot to PCIe 2.0 .....

 

BTW, you would found different possible issue for Mellanox or Intel card, but in simple say, Intel should less trouble from my experiences.

 

The only reason for me to choice between them is PCIe interface related , 82599 are PCIe 2.0x8 , ConnectX-3 are PCIe 3.0x4, so ConnectX-3 are the winner.

 

4 hours ago, prongATO said:

direct connected SFP+ card to my switch with a SFP+ interface?

Which 10G switch you own ? If still haven't, you may deeply consider use SFP+ or not. ( I don't mean not use SFP+ ).

Edited September 18, 20241 yr by Vr2Io

38 minutes ago, Vr2Io said:

Which 10G switch you own ? If still haven't, you may deeply consider use SFP+ or not. ( I don't mean not use SFP+ ).

It's a cheap one I got from Amazon Vine.  We will see if it causes issues or not.  I'm about to upgrade my router on the other side of my AT&T router to a Ubiquiti Cloud Gateway Max or Dream Machine.  I haven't decided how much I want to allot for that purchase yet.  I was looking at bifurcation on the new motherboard and can choose PCIE 1.0, 2.0 or 3.0 per slot in the BIOS.

21 minutes ago, prongATO said:

It's a cheap one I got from Amazon Vine.

This fine, I also do that when I first try on 10G, start from a cheap NIC then other expensive equipment.

 

22 minutes ago, prongATO said:

I was looking at bifurcation on the new motherboard

This also my concern in pass, but it have many limitation and not that important now. A mobo have much PCIe slot already enough.