I have a Mellanox 40 gbit network card running in ethernet mode that's been installed in my Unraid server for some time. Before I installed Unraid, I was running Ubuntu server and able to get most of the speed out of my 40 gbit card. After switching to Unraid the speed dropped to around 4 gbits. I tried fixing the problem but eventually gave up due to other things I needed to take care of. After making some upgrades to my server I've decided to try and resolve this issue again so I can make full use of the high speed drives in my server.

 

Unraid version is currently 6.11.5, but this problem has existed for the last few versions. Basically, I have no idea where to start diagnosing the issue. I do recall I was able to get 10 gbits out of the card during my attempts to resolve the issue in the past, but I really can't recall what I may have done to get that to happen.

Edited April 30, 20233 yr by supacupa
spelling

look at your settings for "flow control" (you may need the plugin "tips & tweaks" to see them). Both directions should be ON.

Also make sure, the other side of the line has turned them on too (switch setting?)

 

One of the steps I've recently taken was to disable flow control on my desktop, 40 gbit switch, and server. I went ahead and reenabled it on everything again and I can't seem to see any difference in it being on or off. I've found that file transfers run faster with jumbo frames enabled, though.

if all devices on the network are the same (40G) speed, FC does not matter. But if there are slower devices (and I bet there are!!!) FC is needed to give them a way not to be overrun by packets.

Jumbo Frames usually do not do anything "better" today. They were introduced with 1Gbe LANs, and made sense then because they took off the higher cpu load because there were fewer packets to calculate.

But since then those calculations have been offloaded and are now done directly by the NICs. So there is no benefit from jumbo frames anymore.

 

Yes, there are 1 gbit devices. I'll disable jumbo frames.

Disable NIC Offload? is set to Yes in TipsAndTweaks, should I set that to No?

no no 🙂

Nic offload is usually perfectly ok. The TipsAndTweaks only allows it to be turned off, because some very cheap and stupid cards/drivers messed it up. But for intel or mellanox they are fine. And for 40G you need them desperately. Else your CPU will be glowing in the dark.

 

Thanks! I've made those changes and am not seeing any differences. Do I need to restart anything in Unraid to see a change?

Hmm, the usual problem with "seeing" is wrong expectation.

Remember, UNRAID has no read cache, therefore reading files is usually limited to the drive speed. So you should expect to see 180-200Mb/s on reading "old" files.

In case you have made a fast NVMe your "cache" drive (since today renamed to "primary storage"), you should be able to see write speeds for a certain amount of data (as long as the cache on the disk can keep up) in the range >3000Mb/s (maybe almost 4000Mb).

But these are peak values!

I only have 10Gb LAN here, and use Samsung 980Pro drives. They can hold the 1000Mb/s max speed for over a minute!

But with 40Gb... outrun will happen much earlier.

 

 

to eliminate the bottlenecks of disks, cpu and ram you should test your connection with "iperf" on both sides.

This usually tells you correct data about the LAN speed. It should be high and almost the same level in both directions (swap client and server iperfs and test again!)

 

Maybe you are just chasing ghosts...

 

That's a very good point, but it seems to be that the network interface is the bottleneck. To test I have a 2 tb NVME that's it's own share that's speed limited to about about 370 MB/s, a RAM disk I've added to the server that's limited to about 370 MB/s, and the LibreSpeed docker can't get higher than about 4000 Mbps for download and 1867 Mbps for upload.

 

If it helps: The server is a 22 core Xeon and it has 64 GB of RAM. I have several 12 tb SAS disks in the primary array and my cache array is made up of four SSDs with DRAM in RAID 1. I am able to achieve excellent speeds internally. Recently I was able to get over 1.8 GB/s out of the array while testing. VMs seem to be limited to 10 gbit which I believe is a limitation of the network driver.

I will need to install iperf. Which I'll do after I'm done preclearing a disk.

and i will go to bed now and will be back tomorrow to read your results...

 

Thank you so much for your help. I'll let you know how it goes.