Unraid 10gbps Network Troubleshooting

[UnBeastRaid]

Hello, thanks for reading into my post.

This is post does not specifically target one issue, it is generally aimed at 10gbps network performance and how to actually achieve 10gbps transfer throughput.

 

I guess the problem that I'm having is that I'm not fully taking advantage of my 10gbps hardware and I do not really know where to start in order to fix it. On average, If I transfer a (10GB) cached (NVMEe) file from my Unraid media server, to my workstation destination which is also NVMe storage, the max speed that I have been able to achieve is roughly ~370MB/s or 2.96gbps. I know with the hardware that I am using, I should theoretically reach 8gbps transfer speeds. I don't really know where the bottleneck is.

 

The hardware I am using (lvl1-2 OSI) is as follows:

- CAT-8 Ethernet cable (max distance is 20')

- 2x ASUS 10gig NICs (updated firmware)

- MikroTik 5-Port Desktop Switch, 1 Gigabit Ethernet Port, 4 SFP+ 10Gbps Ports (CRS305-1G-4S+IN)

- 10G SFP+ RJ45 Copper Transceivers, 10GBase-T Module for Cisco SFP-10G-T-S

- Western Digital WDS500G3X0C 500GB NVMe M.2 SSD

- Samsung 970 Pro 500GB NVMe M.2 SSD

 

If you know of any diagnosing/troubleshooting tips to help me achieve higher transfer speeds, by all means please post the info down below.

Also, if I have missed any information that you need, I will provide it.

 

- Thanks for your help

 

 

 

[JorgeB]

Good to run a single stream iperf test to confirm bandwidth then you can also try transferring to/from a disk share instead of a user share, some users have a very hight user share overhead.

[UnBeastRaid]

Here are the results, not sure as to why the bitrate is so low...

[JorgeB]

That suggests the network is the problem, you should get close to line speed, I get around 8 to 9Gbits/s.

[UnBeastRaid]

Any advice on where to start to fix the issue? New NICs?

[JorgeB]

Basically you need to try swapping the network components one at a time, start with whatever is easier, cables/NICs/Switch/Transceivers

[UnBeastRaid]

Thanks for the help Jorge. I will post results when I manage 8gbps+

[Zonediver]

25 minutes ago, UnBeastRaid said:

Any advice on where to start to fix the issue? New NICs?

Remove the switch and try a "direct-connection" and see if this helps - and check your cables (using the "right" type?)

My 10GBit connection reaches exactly 1200MB/s or 9600MBit

Edited September 18, 2020 by Zonediver