Problem

I am unsure as to why the data transfer speeds are very slow. I know I can't expect 2.5G speeds as my Windows LAN is only 1G, but I expect it to be a bit faster than 15-20 MB/s. I have tested it with a fresh file added to the cache and it is still showing very slow speeds. Any other information I need to include?

 

First off, diagnostics:

lancenas-diagnostics-20250105-1607.zip

 

System Specs:

 

iPerf3:

Windows to Unraid

 

Unraid to PC

 

Transfer Speeds:

Edited January 6, 20251 yr by LanceM26
Moved problem to top

known realtek driver issues and defautl unraid behavior. Your not usiung 2.5 your using 1..
 

 

you may even need to install a driver:

1 minute ago, bmartino1 said:

known realtek driver issues and defautl unraid behavior. Your not usiung 2.5 your using 1..
 

 

you may even need to install a driver:

Oh, I guess I am reading the wrong thing. Doesn't the spec sheet for the mobo say Intel 2.5? And yes i know it will be 1G between my PC and Unraid because of the PC only having gigabit ethernet. But my router and all the switches in the house are 2.5G

let me double-check the hardware once more. Usually msi uses the relatek chip for 2.5 on their intel for that chipset. But the 2.5 advertise is not on by default for compatibility.

While the server to the pc may utilize 2.5 it negation is using 1G and will use 1G due to windows auto tcp nic options.

Apologies I was wrong. Thank you for posting the Diag file.

you nic is negations and advertising a 2.5 per the eth command:
 

Quote

Settings for eth0:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            2500baseT/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            2500baseT/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: 2500Mb/s
    Duplex: Full
    Auto-negotiation: on
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: internal
    MDI-X: off (auto)
    Supports Wake-on: pumbg
    Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
    Link detected: yes

driver: igc
version: 6.1.106-Unraid
firmware-version: 1073:8754
expansion-rom-version: 
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

 

Edited January 6, 20251 yr by bmartino1
data- correction

OK then general optimizations may be needed.

Enable "Tunable (md_write_method)" for Turbo Writes

Turbo Write, or reconstruct write, significantly improves write speeds by reading from all drives in the array to calculate parity, avoiding parity recalculations.

Steps:

Go to Settings > Disk Settings.

Find the "Tunable (md_write_method)" setting.

Set it to "reconstruct write" for maximum write performance.

Note: This increases drive activity and keeps all drives spinning, which may increase power consumption.

I see you have the plugin tips and tweaks.

Turn Off Flow Control

Flow control can introduce latency and bottlenecks in high-speed networks.

Steps:

Go to your NIC settings via the Unraid web GUI or your switch management interface.

Disable "Flow Control" for the NIC.

 

Tweak Tunables for Disk Performance

Fine-tuning Unraid's disk settings can improve read/write performance.

Steps:

Go to Settings > Disk Settings.

Adjust the following:

Tunable (md_num_stripes): Increase this to allocate more stripes for disk operations. Start with 2048 and test for stability.

Tunable (md_sync_window): Increase this (e.g., 896) to allow more concurrent sync operations.

Tunable (md_sync_thresh): Set this to a slightly lower value than md_sync_window (e.g., 192).

 

Jumbo Frames (Optional)

Enable Jumbo Frames if all devices on your network support it. This increases the Maximum Transmission Unit (MTU) to 9000 bytes, reducing CPU overhead and improving throughput.

Steps:

Set the MTU on your Unraid NIC to 9000 via the web GUI under "Network Settings."

Ensure your switch and client devices are configured for the same MTU.

Edited January 6, 20251 yr by bmartino1
data- correction

28 minutes ago, bmartino1 said:

OK then general optimizations may be needed.

Enable "Tunable (md_write_method)" for Turbo Writes

Turbo Write, or reconstruct write, significantly improves write speeds by reading from all drives in the array to calculate parity, avoiding parity recalculations.

Steps:

Go to Settings > Disk Settings.

Find the "Tunable (md_write_method)" setting.

Set it to "reconstruct write" for maximum write performance.

Note: This increases drive activity and keeps all drives spinning, which may increase power consumption.

I see you have the plugin tips and tweaks.

Turn Off Flow Control

Flow control can introduce latency and bottlenecks in high-speed networks.

Steps:

Go to your NIC settings via the Unraid web GUI or your switch management interface.

Disable "Flow Control" for the NIC.

 

Tweak Tunables for Disk Performance

Fine-tuning Unraid's disk settings can improve read/write performance.

Steps:

Go to Settings > Disk Settings.

Adjust the following:

Tunable (md_num_stripes): Increase this to allocate more stripes for disk operations. Start with 2048 and test for stability.

Tunable (md_sync_window): Increase this (e.g., 896) to allow more concurrent sync operations.

Tunable (md_sync_thresh): Set this to a slightly lower value than md_sync_window (e.g., 192).

 

Jumbo Frames (Optional)

Enable Jumbo Frames if all devices on your network support it. This increases the Maximum Transmission Unit (MTU) to 9000 bytes, reducing CPU overhead and improving throughput.

Steps:

Set the MTU on your Unraid NIC to 9000 via the web GUI under "Network Settings."

Ensure your switch and client devices are configured for the same MTU.

First of all, thank you for the in depth instructions. Unfortunately, I tried all the steps except Jumbo frames as I think some devices don't support it and nothing has changed:

1 hour ago, bmartino1 said:

OK then general optimizations may be needed.

Enable "Tunable (md_write_method)" for Turbo Writes

Turbo Write, or reconstruct write, significantly improves write speeds by reading from all drives in the array to calculate parity, avoiding parity recalculations.

Steps:

Go to Settings > Disk Settings.

Find the "Tunable (md_write_method)" setting.

Set it to "reconstruct write" for maximum write performance.

Note: This increases drive activity and keeps all drives spinning, which may increase power consumption.

I see you have the plugin tips and tweaks.

Turn Off Flow Control

Flow control can introduce latency and bottlenecks in high-speed networks.

Steps:

Go to your NIC settings via the Unraid web GUI or your switch management interface.

Disable "Flow Control" for the NIC.

 

Tweak Tunables for Disk Performance

Fine-tuning Unraid's disk settings can improve read/write performance.

Steps:

Go to Settings > Disk Settings.

Adjust the following:

Tunable (md_num_stripes): Increase this to allocate more stripes for disk operations. Start with 2048 and test for stability.

Tunable (md_sync_window): Increase this (e.g., 896) to allow more concurrent sync operations.

Tunable (md_sync_thresh): Set this to a slightly lower value than md_sync_window (e.g., 192).

 

Jumbo Frames (Optional)

Enable Jumbo Frames if all devices on your network support it. This increases the Maximum Transmission Unit (MTU) to 9000 bytes, reducing CPU overhead and improving throughput.

Steps:

Set the MTU on your Unraid NIC to 9000 via the web GUI under "Network Settings."

Ensure your switch and client devices are configured for the same MTU.

Also, after changing the settings you recommended... iperf3 seems to be slower?

weird. That stuff is pulled from this forum and other in-house test for similar speedups.

I see that you have bond and bridging enabled. I would recommend disable bridging and disable bonding. Unless you have a second nic you're using for theses interfaces.

what is your network digram what's connected to what

For Me Example
ISP Modem > unifi UX router > unfi swtich > unriad eth0 

other sites UDM Pro ? sfp+ 10 GB fiber connections > microtek switch > unraid

I do see per the diag that you have quite a few docker running. not sure if yo have a bottle neck due to traffic that concurrent things are running.

7 minutes ago, bmartino1 said:

weird. That stuff is pulled from this forum and other in-house test for similar speedups.

I see that you have bond and bridging enabled. I would recommend disable bridging and disable bonding. Unless you have a second nic you're using for theses interfaces.

what is your network digram what's connected to what

For Me Example
ISP Modem > unifi UX router > unfi swtich > unriad eth0 

other sites UDM Pro ? sfp+ 10 GB fiber connections > microtek switch > unraid

I do see per the diag that you have quite a few docker running. not sure if yo have a bottle neck due to traffic that concurrent things are running.

I am still sort of new to docker so that bridging is enabled because all my docker containers not part of the *arr network are on there, is there a different recommended way to set it up? And for the bonding, I can disable it. I don't think I am using it for anything. Also, how do I disable bond0/bonding?

 

My network diagram:

ISP Router (Technicolor NH20T) -> 2.5G unmanaged switch -> Unraid = 3 Open WRT Dumb Access Points running in wired backhaul. I don't see the need to have one of the access points run in router mode for me as the ISP router already has 10G ports.

 

And as for the docker images, I have tried moving files over with docker disabled and got the same results.

Edited January 6, 20251 yr by LanceM26
added question

Thank you. since you have 3 openwrt wifi AP, is your samba connection for test and file tranfers over wifi? as wifi will be spratic. lan connected test would be best especially in troubleshooting.

 

Will post, but is offtopic... There a multiple way to setup docker. That comes down to how you want to interact with it and what docker services are running what where. Then what ports for elevating port conflicts. I'm a fan of Macvlan as that was the Unraid default setup. Now its Unraid default usue IPVLAN. Pro cons to both.... The docker configuration shouldn't/wouldn't be causing this level of a bottleneck.

More on docker networks:

 

outside of potental testing to setup and remove some other unraid network stuff. as direct eth0 may be better then goin over a bridge connection. But that more theroy then use case.

I'm not seeing anything outright in the diag file... For a test it may be worth a try.

I'm using unriad 7 RC2 
But here our my recommend network side settings that I use between multiple unraid instance:

Web UI > Network Settings >

*Turn off bonding Turn off bridging.
bridgeing is on by default. but both are more usefull if/when you have 2 network nics in unraid.

While here, set you dns setting here:

1st one should be your gateway / the Technicolor NH20T based on your digram (unless your trying to run a dns ad block system like pihole / adguard then gateway should be the second option) the next dns option should be a public dns server. google 8.8.8.8 cloudflares 1.1.1.1 quad 9 9.9.9.9 cisco open dns 208.67.222.222 are all decent options.
 

next lets review some vm and docker setting in unraid:
Web Ui > Setting > VM Manager

Defualt Network Source should be set to vhost0
*This is more for VM having access to you Lans Subnet for VM comunication.

Next lets edit and review some docker configurations.
Web UI > Sewttings > Docker

Its more to set host custom netowoks and preserver user defined networks.
 

For Future compatibility. It is recommended to use the docker xfs image option. You may need to delete the Vdisk and recreate the Docker Vdisk
*Warning this is will empty/clear your current docker tab. (This done't remove or delete docker app configuration data)

-- to re-add existing docker after docker image created, go to docker tab add container in the drop down list select the name and click apply to bring your dockers back.

 

You don't have to change to macvlan. nor changed you Default appdata storage location its better to kepp /mnt/user/appdata for docker templates.

With these set you should now see eth0 as the only interface br0 and bond0 will be no more. if you had any docker set to a custom ip they may need netowrk recofigured to use use the correct network:

 

with docker defualt using vhost0 and vm set to vhost0 depending on ISP dhcp snopping and rouge dhcp server setting you may need a user script to treat teh vhost tab like a service and give it a different IP at first array start. see a work around script here:

 

this may help in some iperf stuff.... I think there may be a issue within your test with iper over wifi. I would ask to confirm you are runnin these test with a device connected to the unmaned switch over the lan. as wifi will muticast brodcast packets and windwos samba sppeds will suffer due to openwrt wifi ap access to the netwrok.

otherwise we wil need to lookat a differeent work around by ediing some advanded samba settings 
 

which can be added to unriad via smb extra settings without having to rewrite and replace unraids samba script.

#========Other Settings that should lookup and inspect for Performance tuning [increase and saturate your network for higher and faster file copying]
        #v -Warning Know your Numbers!
#socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072
        #^ -Warning Know your Numbers!

 

 

Otherwise a more advanced user will need to take a look.

Back on Topic:
as based on your network digram:

Quote

ISP Router (Technicolor NH20T) -> 2.5G unmanaged switch -> Unraid = 3 Open WRT Dumb Access Points running in wired backhaul. I don't see the need to have one of the access points run in router mode for me as the ISP router already has 10G ports.

the 10G may be the ISP connection coming in not the lan connection for your network. so teh ISP modem/router combo is 1 GB

2.5G switch to 2.5 G msi nic for unraid.
this i believe is using 2.5 but ti botle necks then become outgoing internet to 1GB and internal lan to 1 GB

as you have 3 openwrt AP i assume a pie like device 1 GB nic openwrt router flashed...

then you test machine is connecting to what?
and how as i have to asumme...

I asume wifi > openwrt > unmanned swtich > unraid.

to many bottle necs here which would be right per your tests. even with a lan conection with the test machine
i can only asume 1 GB due to test machine onboard nic.

as per your test. even with it not lan connected. it still be 1 GB or less due to wifi repeated traffic.

as such while the switch is geting a good connection but other device are not thus the slowdown.

if you can lan connect to the switch and run the iperf test....

16 minutes ago, bmartino1 said:

Thank you. since you have 3 openwrt wifi AP, is your samba connection for test and file tranfers over wifi? as wifi will be spratic. lan connected test would be best especially in troubleshooting.

 

Will post, but is offtopic... There a multiple way to setup docker. That comes down to how you want to interact with it and what docker services are running what where. Then what ports for elevating port conflicts. I'm a fan of Macvlan as that was the Unraid default setup. Now its Unraid default usue IPVLAN. Pro cons to both.... The docker configuration shouldn't/wouldn't be causing this level of a bottleneck.

More on docker networks:

 

outside of potental testing to setup and remove some other unraid network stuff. as direct eth0 may be better then goin over a bridge connection. But that more theroy then use case.

I'm not seeing anything outright in the diag file... For a test it may be worth a try.

I'm using unriad 7 RC2 
But here our my recommend network side settings that I use between multiple unraid instance:

Web UI > Network Settings >

*Turn off bonding Turn off bridging.
bridgeing is on by default. but both are more usefull if/when you have 2 network nics in unraid.

While here, set you dns setting here:

1st one should be your gateway / the Technicolor NH20T based on your digram (unless your trying to run a dns ad block system like pihole / adguard then gateway should be the second option) the next dns option should be a public dns server. google 8.8.8.8 cloudflares 1.1.1.1 quad 9 9.9.9.9 cisco open dns 208.67.222.222 are all decent options.
 

next lets review some vm and docker setting in unraid:
Web Ui > Setting > VM Manager

Defualt Network Source should be set to vhost0
*This is more for VM having access to you Lans Subnet for VM comunication.

Next lets edit and review some docker configurations.
Web UI > Sewttings > Docker

Its more to set host custom netowoks and preserver user defined networks.
 

For Future compatibility. It is recommended to use the docker xfs image option. You may need to delete the Vdisk and recreate the Docker Vdisk
*Warning this is will empty/clear your current docker tab. (This done't remove or delete docker app configuration data)

-- to re-add existing docker after docker image created, go to docker tab add container in the drop down list select the name and click apply to bring your dockers back.

 

You don't have to change to macvlan. nor changed you Default appdata storage location its better to kepp /mnt/user/appdata for docker templates.

With these set you should now see eth0 as the only interface br0 and bond0 will be no more. if you had any docker set to a custom ip they may need netowrk recofigured to use use the correct network:

 

with docker defualt using vhost0 and vm set to vhost0 depending on ISP dhcp snopping and rouge dhcp server setting you may need a user script to treat teh vhost tab like a service and give it a different IP at first array start. see a work around script here:

 

this may help in some iperf stuff.... I think there may be a issue within your test with iper over wifi. I would ask to confirm you are runnin these test with a device connected to the unmaned switch over the lan. as wifi will muticast brodcast packets and windwos samba sppeds will suffer due to openwrt wifi ap access to the netwrok.

otherwise we wil need to lookat a differeent work around by ediing some advanded samba settings 
 

which can be added to unriad via smb extra settings without having to rewrite and replace unraids samba script.

#========Other Settings that should lookup and inspect for Performance tuning [increase and saturate your network for higher and faster file copying]
        #v -Warning Know your Numbers!
#socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072
        #^ -Warning Know your Numbers!

 

 

Otherwise a more advanced user will need to take a look.

Back on Topic:
as based on your network digram:

the 10G may be the ISP connection coming in not the lan connection for your network. so teh ISP modem/router combo is 1 GB

2.5G switch to 2.5 G msi nic for unraid.
this i believe is using 2.5 but ti botle necks then become outgoing internet to 1GB and internal lan to 1 GB

as you have 3 openwrt AP i assume a pie like device 1 GB nic openwrt router flashed...

then you test machine is connecting to what?
and how as i have to asumme...

I asume wifi > openwrt > unmanned swtich > unraid.

to many bottle necs here which would be right per your tests. even with a lan conection with the test machine
i can only asume 1 GB due to test machine onboard nic.

as per your test. even with it not lan connected. it still be 1 GB or less due to wifi repeated traffic.

as such while the switch is geting a good connection but other device are not thus the slowdown.

if you can lan connect to the switch and run the iperf test....

Thank you again for the detailed guidance. I've done everything up to the userscripts you created as I need to read up more about when they should be run (cron) and the pros and cons of them. My docker was already using the xfs img and macvlan enabled. To add, every test I have been running has been over LAN. The equals sign just indicates that my access points, host windows pc and unraid are all connected to 2.5G unmanaged switches.

 

The access points are all Wifi 6 asus routers that have been flashed with OpenWRT

 

Your assumption here I asume wifi > openwrt > unmanned swtich > unraid.

How I have it: ISP Modem > unmanned switch - > openwrt/unraid/hostpc (all on same level)

 

Also, the port that all the switches/devices are connected to on my ISP modem is 10G:

 

Just ran another test and seems like nothing has changed:

 

 

6 hours ago, bmartino1 said:

see you have the plugin tips and tweaks.

Turn Off Flow Control

Flow control can introduce latency and bottlenecks in high-speed networks.

Steps:

never!!!

This is the worst you can do to your machines. Read the Text carefully! "Meant for mixed-speed networks". And guess what he has here? 2,5 to 1G

If this is not mixed what else should it be?

Also, 2,5G alone is "mixed" already because it is just "10G with pauses". It cannot work without flow control.

 

Turn it back on.

And make sure every device on the LAN does honor it, else there will be strange lockups, blocades, pauses...

Unmanaged switches do not honor it, they just pass on the request in the hope, the next device will be more clever. So this is not reliable. If you daisy-chain even more unmanaged devices it will become even more risky.

 

 

30 minutes ago, MAM59 said:

never!!!

This is the worst you can do to your machines. Read the Text carefully! "Meant for mixed-speed networks". And guess what he has here? 2,5 to 1G

If this is not mixed what else should it be?

Also, 2,5G alone is "mixed" already because it is just "10G with pauses". It cannot work without flow control.

 

Turn it back on.

And make sure every device on the LAN does honor it, else there will be strange lockups, blocades, pauses...

Unmanaged switches do not honor it, they just pass on the request in the hope, the next device will be more clever. So this is not reliable. If you daisy-chain even more unmanaged devices it will become even more risky.

 

 

I've put it back to default and rebooted my system

iperf results confirm the problem is with the LAN, could be a NIC, or its driver, cables switch, client PC, etc.

5 minutes ago, JorgeB said:

iperf results confirm the problem is with the LAN, could be a NIC, or its driver, cables switch, client PC, etc.

Any idea how I can troubleshoot what is causing it? I am able to download (according to speedtest) from client PC at 1GB and the same with my unraid server

 

there have been reports on this forum with 2.5 and reltek where turning off flow control increased performance.

most likley culprit is the isp modem router combo...

it has always been best practice to have the isp modem in bridge mode and use your own router.



 

1 hour ago, LanceM26 said:

Any idea how I can troubleshoot what is causing it?

Try a different NIC, cable, switch and even a different client PC if possible.

2 hours ago, bmartino1 said:

there have been reports on this forum with 2.5 and reltek where turning off flow control increased performance.

Yeah, you also believe everything a politician tells you?

You can safely turn it off (and gain a millipercentage) if all speeds are equal within the LAN. But if there is just one partner that is slower, there must be a means to stop a sender from overrunning the poor receiver. In 1G LANs this is often done within (managed) switches but then buffer problems arised and they have invented "flow control" which now allows both partners to know if the line is busy or not.

And for 2,5 in 10G LAN this is a lifesaver because (simplyfied, the reality is much more complex!) think of the time slots 1234. Only in one of these a 2,5G card can send and receive. which one?

Look at this sample setup:

Cards 2 & 4 are just directly connected to each other. But look at the stats...

#2 thinks she has a 10G connection (thats correct, because 2.5G does not exist) and #4 knows that its 2.5 only (because she can do no more)

This would produce a big mess. And here is where flow control comes in. If #2 as a sender uses the "wrong" slot, #4 halts her. This happens until the right window is guessed and repeats over and over again.

In the other direction (2,5->10) it never happens because the 10G card accepts data on every slot.

on UNRAID this is showed a bit different:

Settings for eth0:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseX/Full
                                10000baseCR/Full
                                10000baseSR/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseX/Full
                                10000baseCR/Full
                                10000baseSR/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d

 

"Symmetric Receive-only" means that it only looks for overrunning senders but is fast enough to capture all from the line (this is a precaution setting if the LAN contains 25/40/100G devices) But it advertises "Symmetric" meaning it COULD also pace the other side.

 

Anyway, this is a fundamental feature, turning it off sets you to a great risk...