Hello everyone,

I'm experiencing severe and persistent network instability on my Unraid server, leading to drastically reduced speeds (40-80 MB/s, often dropping to 0 bytes/s) and frequent "Broken pipe" errors during file transfers. This affects both my direct 5GbE connection to a Windows PC and transfers over Wi-Fi (via my UDM), indicating a server-side issue.

Crucially, this entire setup, with the exact same hardware, worked perfectly at expected high speeds (over 250 MB/s from HDDs, even higher from SSDs) until recently.

I've performed extensive troubleshooting, and the diagnostics point to a fundamental problem with the atlantic network driver and/or the kernel's handling of my Aquantia 5GbE network interfaces under load.

Here's a summary of my setup and findings:

Server Hardware:

• CPU: 12th Gen Intel® Core™ i5-1235U (on Unraid server, confirmed via uname -a)

• Terramaster F4-424 Max NAS

• Network Cards: Two integrated Aquantia 5GbE NICs (using the atlantic driver).

  • eth0 (PCI 0000:03:00.0): Connected directly to my Windows PC via 5GbE Ethernet.

  • eth1 (PCI 0000:04:00.0): Connected to my Unifi Dream Machine (UDM), serving other network traffic including Wi-Fi clients.

Unraid OS Version:

• Unraid OS 7.1.4 (confirmed via unraid.net/blog/unraid-7-1-4)

• Kernel Version: Linux Lintgracht 6.12.24-Unraid #1 SMP PREEMPT_DYNAMIC Sat May 3 00:12:52 PDT 2025 x86_64 12th Gen Intel(R) Core(TM) i5-1235U GenuineIntel GNU/Linux

• atlantic Driver Version: 6.12.24-Unraid

• Firmware Version: 1.3.30

Observed Symptoms:

1. Drastically Low Speeds: Downloads from Unraid to my Windows PC via SMB/SFTP consistently hover around 40-80 MB/s, then often drop to 0 bytes/s.

2. "Broken Pipe" Errors: Samba logs frequently show smb2_sendfile_send_data: sendfile failed for file ... (Broken pipe) ... Terminating.

3. Active Link Instability (Confirmed via dmesg): The network interfaces (eth0 and eth1) are actively losing and regaining their links under load. This is the primary confirmed issue.

   • Example dmesg snippet:

     [355282.775806] atlantic 0000:03:00.0 eth0: atlantic: link change old 10000 new 0
     [355282.776298] br0: port 1(eth0) entered disabled state
     [352583.708424] atlantic 0000:04:00.0 eth1: atlantic: link change old 5000 new 0
     [355291.064360] atlantic 0000:03:00.0 eth0: atlantic: link change old 0 new 10000
     [355291.064417] br0: port 1(eth0) entered blocking state
     [355291.064425] br0: port 1(eth0) entered forwarding state
     [355292.025375] atlantic 0000:04:00.0 eth1: atlantic: link change old 0 new 5000
     

4. Packet Drops: ip -s link show eth0 consistently shows a high and increasing count of dropped packets under the RX (receive) column (e.g., 256203 increasing to 82009 after a reset, then actively increasing during transfer).

5. High kworker CPU activity: top command shows elevated CPU usage by kworker processes during network transfers, suggesting kernel-level overhead in handling I/O.

Troubleshooting Steps Taken (with results):

1. Verified 5GbE Link Speed (Client Side): Windows PC reports a stable 5000 Mbps link.

2. Local Disk Speed on Unraid: dd test on Unraid shows 306 MB/s, confirming disks are not the bottleneck.

3. Unraid CPU: Server CPU (i5-1235U) is modern and powerful enough, not a bottleneck.

4. Unraid Network Settings (GUI): All standard and correctly configured (e.g., MTU 1500, no VLANs/bonding, bridging enabled).

5. Disabled Docker Containers/VMs: Tested with all Dockers/VMs stopped; problem persists.

6. sysctl Network Buffer Adjustments: Temporarily increased net.core.rmem_max, rmem_default, and netdev_max_backlog – no improvement.

7. ethtool -G Ring Buffer Adjustment:

   • ethtool -g eth0 showed Max RX: 8184, Current RX: 256.

   • Attempted to set ethtool -G eth0 rx 8184 tx 8184 via SSH.

   • Result: While ethtool -g eth0 then showed Current RX: 8184, the problem persisted and seemed to cause an even more immediate drop to 0 bytes/s. I have since reverted these back to 256.

8. Unraid OS Tweaks Page:

   • NIC Flow Control: Tried Yes and No.

   • Disable NIC Offloads: Tried Yes (to disable all offloads) and No.

   • Ethernet NIC Rx Buffer / Tx Buffer: Set to 8184 (reverted to 256 now).

   • Result: No improvement in speed or link stability; link change events continued.

9. Coalescing Parameters (ethtool -c eth0): Show rx-usecs: 112, tx-usecs: 510. Have not yet extensively experimented with these due to the fundamental link-drop issue.

My Analysis:

The consistent "link change" events on both atlantic interfaces (eth0 and eth1) are the direct root cause of the degraded performance and connection drops. Maybe a specific incompatibility between this driver/kernel version and the Aquantia hardware firmware, manifesting under high network load? The fact that this worked perfectly before on the same hardware is key.

Seeking Assistance: Has anyone else encountered similar "link change" issues with Aquantia/atlantic NICs on Unraid OS 7.1.4 (kernel 6.12.24)?

Given that this issue only started recently and that my hardware was performing optimally before, I'm at a loss. Would it be advisable to attempt a downgrade to a previous stable 6.12.x Unraid OS version, even though 7.1.4 is currently the latest released version? Are there any specific risks associated with such a downgrade for my setup?

Any insights or suggestions from those with deep kernel/network driver knowledge or similar experiences with Aquantia NICs would be greatly appreciated.

Thank you in advance for your help!

Best regards,

foobar1452

lintgracht-diagnostics-20250728-1015.zip

Edited July 28, 2025Jul 28 by foobar1452

as you can see from your logs, eth0 keeps on trying to establish a 10G link, why?

And, for sure, with 5G "cripple net" you need to run it with flow control turned on, else you will notice a lot of lost packets (slowdowns). But the other side also needs to honor flow control, check it out.

54 minutes ago, foobar1452 said:

This affects both my direct 5GbE connection to a Windows PC

it looks like you have a 10GbE connection on eth0 ????

Maybe a bad and not complying cable is the issue of your problems. Get a REAL (not a "raw cable" labeled!!!) CAT 8.1 on with less than 10m Length. This should work for a while. maybe for longer. Everthing below or longer (yeah I know, the marketing tells you "up to 30m". They are lying as usual) is a high risk.

btw, i cannot find boot infos in your diagnostics... maybe i am blind or you never rebooted before you took the diagnostics???

Current Status and Troubleshooting Steps:

eth0 Status: FIXED and Performing Well! Following community advice, I have successfully applied and made persistent the ethtool optimizations for eth0:

• ethtool -G eth0 rx 8184 tx 8184 (Ring Buffers set to max)

• ethtool -A eth0 tx on rx on (Flow Control enabled) These settings are correctly applied at boot via the /boot/config/go script (with a sleep 10 delay to ensure proper driver initialization). ethtool -g eth0 and ethtool -a eth0 confirm this after every reboot. My general LAN/WiFi connection via eth0 is now achieving expected high transfer speeds, and link change events or packet drops no longer seem to impact its performance. This confirms the atlantic driver can perform well with these specific parameters on my hardware.

My Network Topology: My Unraid server features two Aquantia 5GbE NICs (both using the atlantic driver), configured as follows:

• eth0 (PCI 0000:03:00.0): General LAN/WiFi Connection

  • Connection: This NIC is connected to my router (Unifi Dream Machine - UDM), handling my general LAN traffic, including my desktop's Wi-Fi connection.

  • Subnet: It operates within the 192.168.0.x subnet.

  • Bridging: It is correctly configured as a member of br0, which holds the server's main IP address (192.168.0.164).

• eth1 (PCI 0000:04:00.0): Dedicated 5GbE Direct Link

  • Connection: This NIC is directly connected via 5GbE Ethernet to my desktop PC.

  • Subnet: It is intended to operate on a dedicated 192.168.3.x subnet for high-speed direct transfers.

  • Bridging: It is currently intended to be part of a separate bridge (br1) to maintain this dedicated subnet.

eth1 Status: STILL Slow (40 MB/s) and Stuck in a Kernel Inconsistency Despite the success with eth0, my dedicated 5GbE link via eth1 is still limited to around 40 MB/s. I have applied the exact same ethtool settings to eth1 and verified them after reboot:

• ethtool -g eth1 shows RX: 8184, TX: 8184.

• ethtool -a eth1 shows RX: on, TX: on.

However, eth1 remains slow. My dmesg -w logs still show frequent link change old X new 0 events specifically for eth1, indicating that the underlying link instability persists for this interface.

The CORE Problem for eth1: Inconsistent Kernel Bridging State My attempts to configure eth1 into its desired separate bridge (br1) have revealed a deep kernel inconsistency:

• The Unraid GUI for eth1 shows Enable bridging: Yes and Bridging members of br1: eth1 (with IPv4 address: 192.168.3.1). However, this field is not editable.

• When attempting to delete eth1 from br1 via SSH: brctl delif br1 eth1 results in "bridge br1 does not exist!".

• brctl show further confirms br1 is not listed at all (only br0 with eth0 as member).

• Yet, when trying to add eth1 to br0 (or any other bridge) via SSH: brctl addif br0 eth1 results in "device eth1 is already a member of a bridge; can't add it to bridge br0.".

• ip a show eth1 shows eth1 as DOWN with inet 192.168.3.1/24 directly assigned to it.

This means eth1 is stuck in an inconsistent state where the kernel internally marks it as "bridged" (or occupied), but that bridge (br1) does not exist, and it cannot be added to any other bridge (br0). This kernel inconsistency is almost certainly the root cause of the continued link instability and slow performance on eth1.

Seeking Advice: How to Resolve this Kernel Inconsistency for eth1 and Achieve its Desired Dedicated Bridge Topology?

Given this very confusing and seemingly stuck kernel state for eth1, I am seeking advice on the best way forward.

• My goal is for eth1 to be a member of a separate bridge (e.g., br1), retaining its dedicated 192.168.3.x subnet.

• What steps would you recommend to properly clear eth1 from this inconsistent "bridged" state and successfully establish it in its own bridge (br1) with the 192.168.3.x subnet?

• Would an aggressive reset of the entire network stack (e.g., bringing down all interfaces, attempting to delete/recreate br0, then creating br1 and adding interfaces via SSH commands) be a valid and recommended approach in this situation? If so, what are the specific caveats or commands to be aware of for this atlantic driver on Unraid OS 7.1.4, especially regarding brctl's contradictory messages?

• Are there any less disruptive methods to "force-clear" a network interface's inconsistent state in the kernel?

Any further insights or guidance on resolving this persistent kernel inconsistency for eth1 would be greatly appreciated.

Thank you in advance for your help!

Best regards,

Foobar1452

lintgracht-diagnostics-20250728-1320.zip

Edited July 28, 2025Jul 28 by foobar1452

19 minutes ago, foobar1452 said:

My goal is for eth1 to be a member of a separate bridge (e.g., br1), retaining its dedicated 192.168.3.x subnet.

Hmm, what should this be good for?

With your setup the normal way would be to put both cards into ONE bridge (br0). So packets from your windows PC can be forwarded to devices on the other card without any routing tricks.

Also both cards would share ONE IP address.

I do not understand what you think you might get from your setup?

(shutdown, delete network.config and network.rules from the stick. reboot, add eth1 to br0 and you are done)