Slow Docker Network?

[Nakor]

Greetings!

 

Recently (last 15 days or so), I noticed my SABnzbd downloads were never exceeding 10MBs (previously it would max out a little over 100MBs on my fiber connection).  I also have Speedtest Tracker running which has a corresponding slowdown:

 

Line 147 was last "good" result

 

I tested iperf and got this today router to unraid server:

 

SAB is also showing this:

 

I've attached my diagnostics to the post. I'm not sure what else to test or change.  Fix Common Problems comes up with nothing.  SAB is running in bridge mode.  Any insight would be appreciated.

tower-diagnostics-20230206-1455.zip

[Nakor]

Update: Using new NIC and new cable as I was out of ideas.  Same issue.  Updated diagnostics attached in case anyone has an idea before I just nuke it.

 

tower-diagnostics-20230207-2051.zip

Edited February 8, 2023 by Nakor

[Nakor]

Deteled docker image and installed everything from scratch - no fix

[Nakor]

Updated tests:

from docker network inspect:

Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        MDI-X: on (auto)
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes
iperf3 shows gig speeds and file transfers are fast as well
relevant parts of docker network inspect:
        "Name": "bridge",
        "Id": "###",
        "Created": "2023-02-09T19:36:20.054881309-05:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"

 

Host speedtest

docker run --rm --net=host tianon/speedtest --accept-license:

 

'Bridge' speedtest

docker run --rm --net=bridge tianon/speedtest --accept-license:

 

 

[Vr2Io]

1 hour ago, Nakor said:

Host speedtest

docker run --rm --net=host tianon/speedtest --accept-license:

 

 

It likely Router / ISP issue, check router WAN linkspeed too.

 

   Speedtest by Ookla

      Server: XXXXXX
         ISP: XXXXXX
Idle Latency:     2.30 ms   (jitter: 0.20ms, low: 1.98ms, high: 2.56ms)
    Download:   395.71 Mbps (data used: 178.6 MB)                                                   
                  9.17 ms   (jitter: 1.07ms, low: 2.90ms, high: 12.81ms)
      Upload:   782.11 Mbps (data used: 372.9 MB)                                                   
                  8.27 ms   (jitter: 0.84ms, low: 2.74ms, high: 11.07ms)
 Packet Loss: Not available.

 

Edited February 10, 2023 by Vr2Io

[Nakor]

No other device on the same network has any issues.  What should I look at on my router?

 

Here is docker run --rm --net=host tianon/speedtest --accept-license run directly on my router:

[Vr2Io]

Never got docker network low performance, could you try docker openspeedtest for local test, easy reach 8G actual network traffic.

 

openspeedtest/latest

 

[Nakor]

Here ya go:

[Vr2Io]

So, it likely not docker / docker network issue. You said other device no problem, speed test over 100Mbps ? Does router have QoS apply ?

[Nakor]

No QoS.  All other devices work fine. SABnzbd downloads at 100MB on other computers on the same network.

[Gragorg]

Are you running SAB through privoxy?

[Nakor]

No

[Vr2Io]

Would it be storage slow issue for writing ?

[Nakor]

I've ordered a new NIC to see if that fixes anything.  If that doesnt work, I suppose I can hold out for the next next release of unraid before just wiping and utilizing something other than unraid

[Nakor]

13 minutes ago, Vr2Io said:

Would it be storage slow issue for writing ?

I'll re-run diskspeed, but it shouldnt.  I have 2 SSDs for cache

[Vr2Io]

1 hour ago, Nakor said:

I'll re-run diskspeed, but it shouldnt.  I have 2 SSDs for cache

I miss that your last Ookla speed test got abnormal resut, so it shouldn't storage related.

Edited February 11, 2023 by Vr2Io

[Nakor]

Diskspeed:

 

HDDs:

 

SSDs:

[Faceman]

Just chiming in as i might be having a similar issue, recently upgraded from 100mbit to 1gig internet and i'm seeing my unraid networking maxing out at ~400-500mbit, and that includes an internal test from OpenSpeedTest on HOST to a VM, SAB caps out at around 55Mb/s, torrents hit the same sort of limit, and if I hit them all at once, I still only see about 500Mbit maximum total throughput. 

 

Still tinkering as it is a brand new connection and I haven't done any real tuning yet, and I know my old CX2 10g card is running on a 5GT/s bottlenecked PCIE bus at the moment, but that should be more than enough to service a 1G internet connection.. Modem to router is a solid 1G, Then router to switch to server connection is all 10G.

 

Internet speed test on the Router exceeds 900mbit, so that's fine.

 

Edit:

Some testing, if I disable docker entirely from the settings the network speeds skyrocket, If I enable docker with any settings it collapses down to 500mbit, not just the WAN speed, the local lan access drops too. Might need to rebuild my whole docker config from scratch.. somethings wrong here. after some tweaking running the openspeedtest container and accessing it from across the network I see a full speed download (docker br0 upstream speed) and 500mbit upload (docker br0 download speed)... so theres something bottlenecking the download pseeds on the bridge network, but running on host or br1 (separate NIC) or any other test doesnt seem to change the behavior.

Edited May 19, 2023 by Faceman

[faroh]

I'm having similar issues atm. Where you able to get to the bottom of the issue? @Faceman

[Faceman]

haven't gotten too far, but haven't done much testing either.

 

Seems like the connection OUT of the docker network is OK, I can access a locally hosted Openspeed test and pull a 2gb+ download speed on a remote computer, though that is on a 10gbe connection so theres still a bottleneck there. the other direction IN to the docker network varies between 500mb and 800mb in these tests, which seems very slow.

 

CPU usage isn't too bad, Previously my server ran on a pair of ancient low power Xeon 2450L processors running at 1.8ghz and could basically saturate 10gbe connections from docker network, now I'm on a 4Ghz 6700k , same NIC and struggling to hit half a gig download whether local LAN or from the web. definitely seems off.

 

I've also tested a VM, and it behaved the same as a real machine on the network, very similar throughput numbers, it is on the same bridge network... will try testing a dedicated NIC and removing the secondary bridge I have set up for a single container next.

Edited June 6, 2023 by Faceman

[kcossabo]

@Faceman - same issue here. Curious if you found a cure

[brwoodford]

Also getting this issue, and been driving me crazy on two Unraid servers..

SABNZB on host or bridged, most I can get is 10MB/s but mostly 7 ish... then I put it on BR0 as its own IP, and im hitting pretty much the speed of my ISP give of take a few MBs