Hi all, was playing around with perf3 in order to diagnose issues with upload speeds and found that an MTU of 9000 (@600 Mb/s) works better than 1500 (@350 Mb/s), but still not quite the expected 950 Mb/s.  So I decided to max out the MTU.  Now, the server is no longer reachable.

 

Is it safe to pull the USB and modify the network config on a Windows machine?  And where could I find the value to modify?

Thanks!

Jumbo frames are evil. Well, they cause a lot of problems anyway. Fix Common Problems will warn against them:

https://forums.unraid.net/topic/120220-fix-common-problems-more-information/page/2/#comment-1167702

 

To manually remove them, on the flash drive, edit config/network.cfg. Find any MTU[x] entries and change them to "1500", i.e.:

MTU[0]="1500"
MTU[1]="1500"

When you reboot you will be back to stock MTUs.

2 hours ago, floepie05 said:

Is it safe to pull the USB

I would not pull the USB drive from the server if it is running.  You should be able to do a clean shutdown with about a one second push of the power switch.  (A five second push will force a shutdown and it will probably not be clean!) 

12 hours ago, ljm42 said:

Jumbo frames are evil. Well, they cause a lot of problems anyway. Fix Common Problems will warn against them:

https://forums.unraid.net/topic/120220-fix-common-problems-more-information/page/2/#comment-1167702

 

To manually remove them, on the flash drive, edit config/network.cfg. Find any MTU[x] entries and change them to "1500", i.e.:

MTU[0]="1500"
MTU[1]="1500"

When you reboot you will be back to stock MTUs.

 

Thanks a lot - that did the trick.

So you would suggest I'm asking for trouble moving MTU up to 9000?  With that setting I'm able to almost double my upload speed.

This begs the question as to why I receive only 1/3 the expected upload speed with MTU 1500.

I've tried with multiple cat5e cables and testing them with a Windows machine.

 

Ethernet hardware on Unraid:

Intel Corporation Ethernet Connection (12) I219-V 

 

The following related drivers are in use

e1000eIntel(R) PRO/1000 Network Driver In use net/ethernet/intel/e1000e

e100Intel(R) PRO/100 Network Driver In use net/ethernet/intel

e1000Intel(R) PRO/1000 Network Driver In use net/ethernet/intel/e1000

 

20 minutes ago, floepie05 said:

This begs the question as to why I receive only 1/3 the expected upload speed with MTU 1500.

 

14 hours ago, floepie05 said:

Hi all, was playing around with perf3 in order to diagnose issues with upload speeds and found that an MTU of 9000 (@600 Mb/s) works better than 1500 (@350 Mb/s), but still not quite the expected 950 Mb/s. 

 

What are the results in 'real life'?  Remember that perf3 is an artificial test platform and its results represents that best that can be expected. 

7 minutes ago, Frank1940 said:

 

 

What are the results in 'real life'?  Remember that perf3 is an artificial test platform and its results represents that best that can be expected. 

 

Perf3 over the LAN and various docker speedtests (over the internet) are aligned at an average of 350 Mb/s.  A transfer from an SMB share over the LAN with a very large file fares a bit better at 45 MB/s. 

There's an upload bottleneck somewhere.   Downloads are rock solid at about 940-950 Mb/s.

Edited May 18, 20242 yr by floepie05

5 minutes ago, floepie05 said:

There's an upload bottleneck somewhere.   Downloads are rock solid at about 940-950 Mb/s.

How is this upload traveling?  Is a wireless link involved?  Is it local or remote? 

 

The reason for my question is that my upload and download speeds via a wired LAN connection for large files (>10GB) is above 90MB/s.  (My MTU is 1500!) 

41 minutes ago, Frank1940 said:

How is this upload traveling?  Is a wireless link involved?  Is it local or remote? 

 

The reason for my question is that my upload and download speeds via a wired LAN connection for large files (>10GB) is above 90MB/s.  (My MTU is 1500!) 

Yeah, everything's wired up with cat5e on gigabit LAN.  Live by myself, so no real congestion at all. Speeds tested across LAN (iperf and SMB transfer large file) and internet with docker speedtests, and all agree I'm seeing only ~35-40% of expected upload speeds on gigabit LAN, but downloads are at 100%.  FWIW, internet is via fiber at 2 Gbit down and up.  

 

There is only one ethernet adapter on my server and have tried also with bonding turned off.  

Edited May 18, 20242 yr by floepie05

When you speak of "upload", are you talking about to the server?

 

Do you have a cache drive on the server? 

 

Are you uploading (1) to the cache drive or (2) directly to the array?

 

11 minutes ago, Frank1940 said:

When you speak of "upload", are you talking about to the server?

 

Do you have a cache drive on the server? 

 

Are you uploading (1) to the cache drive or (2) directly to the array?

 

No, upload, as in from the server to LAN/internet.  This is too slow @ 35 MB (iperf/speedtests/SMB transfers).  LAN to server is fast and expected.

Yes, nvme drive as cache.

 

As I mentioned above, moving the MTU from 1500 to 9000 almost doubles the upload speed, still not the full gig.  So, likely something wonky with the driver or adapter or unraid, not sure.

 

Edited May 18, 20242 yr by floepie05

Not seen Unraid having problems reaching gigabit speeds in the past.

 

Could be worth rebooting the network equipment just to see if it helps.

3 hours ago, floepie05 said:

No, upload, as in from the server to LAN/internet.  This is too slow @ 35 MB (iperf/speedtests/SMB transfers).  LAN to server is fast and expected.

Yes, nvme drive as cache.

 

As I mentioned above, moving the MTU from 1500 to 9000 almost doubles the upload speed, still not the full gig.  So, likely something wonky with the driver or adapter or unraid, not sure.

 

This is extremely confusing.

 

There are no advantages to using jumbo frames on one computer, you have to use them on every computer that that computer might talk to. Typically this is not practical unless you have a multiple nics and a separate subnet that key systems use to talk. But the main network should be kept at 1500.

 

Have you changed the MTU on your router? That could explain what you are seeing. If so, return the router's MTU to 1500

1 hour ago, ljm42 said:

 

This is extremely confusing.

 

Have you changed the MTU on your router? That could explain what you are seeing. If so, return the router's MTU to 1500

No, no other MTU settings on the network have been changed from the default.  I was just playing around with MTU values on Unraid to troubleshoot.  I've seen other reports of moving MTU to 9000 helped.  

Thanks for trying to dig up what was causing the issue.

Turns out there is a setting in the BIOS which was the culprit.

I've resorted to loading optimized defaults and speeds are now as expected.  From the pic, one of those settings created the less than desirable upload speeds.

I will further test to determine which one exactly the network adapter didn't like.

 

 

 

Edited May 19, 20242 yr by floepie05

So, if anyone is still tuning into this thread and for that future somebody, the one bad apple setting was "DMI ASPM Support".  For my setup, I must have this set to Disabled to restore full upload speeds.  Curiously, I could keep the "PCH DMI ASPM Support" set to Auto (not disabled).

All others were tested by process of elimination.