• (SOLVED) Dropped rx packets and halved 10G performance with 6.9.0-beta30


    Toskache
    • Closed

    With 6.9.0-beta30 i can observe a performance-issue with my "SoNNeT G10E-1X-E3" network card. It is based on the chipset "Aquantia AQC-107S".
    With 6.9.0-beta25 i got the following iperf3 results:

    toskache@10GPC ~ % iperf3 -c 192.168.2.4
    Connecting to host 192.168.2.4, port 5201
    [  5] local 192.168.2.199 port 50204 connected to 192.168.2.4 port 5201
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-1.00   sec  1.15 GBytes  9.90 Gbits/sec
    [  5]   1.00-2.00   sec  1.15 GBytes  9.89 Gbits/sec
    [  5]   2.00-3.00   sec  1.15 GBytes  9.89 Gbits/sec
    [  5]   3.00-4.00   sec  1.15 GBytes  9.89 Gbits/sec
    [  5]   4.00-5.00   sec  1.15 GBytes  9.89 Gbits/sec
    [  5]   5.00-6.00   sec  1.15 GBytes  9.89 Gbits/sec
    [  5]   6.00-7.00   sec  1.15 GBytes  9.89 Gbits/sec
    [  5]   7.00-8.00   sec  1.15 GBytes  9.89 Gbits/sec
    [  5]   8.00-9.00   sec  1.15 GBytes  9.90 Gbits/sec
    [  5]   9.00-10.00  sec  1.15 GBytes  9.89 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  sender
    [  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  receiver
    
    iperf Done.

    And there where no dropped packets.

     

    With the current 6.9.0-beta 30 I get a lot of dropped packets and the iperf3 performance is halved.
    But just the RX-side. TX is fine.


    Since 6.9.0-beta 29 also the plugin page is loading extremely slow (approx. 15 seconds). With 6.9.0-beta 25 the loading process took less than 2 seconds.

    Here are some network informations and attached you can find the diagnostics-file:

    root@nas:~# ifconfig
    eth0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
            ether 00:30:93:14:08:72  txqueuelen 1000  (Ethernet)
            RX packets 206238861  bytes 292428787789 (272.3 GiB)
            RX errors 0  dropped 87152  overruns 0  frame 0
            TX packets 88635371  bytes 128707642804 (119.8 GiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    root@nas:~# ethtool eth0
    Settings for eth0:
            Supported ports: [ TP ]
            Supported link modes:   100baseT/Full
                                    1000baseT/Full
                                    10000baseT/Full
                                    2500baseT/Full
                                    5000baseT/Full
            Supported pause frame use: Symmetric Receive-only
            Supports auto-negotiation: Yes
            Supported FEC modes: Not reported
            Advertised link modes:  100baseT/Full
                                    1000baseT/Full
                                    10000baseT/Full
                                    2500baseT/Full
                                    5000baseT/Full
            Advertised pause frame use: No
            Advertised auto-negotiation: Yes
            Advertised FEC modes: Not reported
            Speed: 10000Mb/s
            Duplex: Full
            Auto-negotiation: on
            Port: Twisted Pair
            PHYAD: 0
            Transceiver: internal
            MDI-X: Unknown
            Supports Wake-on: pg
            Wake-on: g
            Current message level: 0x00000005 (5)
                                   drv link
            Link detected: yes
    hermann@Hacky ~ % iperf3 -c 192.168.2.4
    Connecting to host 192.168.2.4, port 5201
    [  5] local 192.168.2.26 port 52792 connected to 192.168.2.4 port 5201
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-1.00   sec   519 MBytes  4.36 Gbits/sec
    [  5]   1.00-2.00   sec   509 MBytes  4.27 Gbits/sec
    [  5]   2.00-3.00   sec   491 MBytes  4.12 Gbits/sec
    [  5]   3.00-4.00   sec   410 MBytes  3.44 Gbits/sec
    [  5]   4.00-5.00   sec   390 MBytes  3.27 Gbits/sec
    [  5]   5.00-6.00   sec   485 MBytes  4.07 Gbits/sec
    [  5]   6.00-7.00   sec   447 MBytes  3.75 Gbits/sec
    [  5]   7.00-8.00   sec   452 MBytes  3.79 Gbits/sec
    [  5]   8.00-9.00   sec   449 MBytes  3.76 Gbits/sec
    [  5]   9.00-10.00  sec   481 MBytes  4.03 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-10.00  sec  4.53 GBytes  3.89 Gbits/sec                  sender
    [  5]   0.00-10.00  sec  4.52 GBytes  3.89 Gbits/sec                  receiver
    hermann@Hacky ~ % iperf3 -c 192.168.2.4 -P5
    Connecting to host 192.168.2.4, port 5201
    [  5] local 192.168.2.26 port 52831 connected to 192.168.2.4 port 5201
    [  7] local 192.168.2.26 port 52832 connected to 192.168.2.4 port 5201
    [  9] local 192.168.2.26 port 52833 connected to 192.168.2.4 port 5201
    [ 11] local 192.168.2.26 port 52834 connected to 192.168.2.4 port 5201
    [ 13] local 192.168.2.26 port 52835 connected to 192.168.2.4 port 5201
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-1.00   sec   200 MBytes  1.67 Gbits/sec
    [  7]   0.00-1.00   sec   199 MBytes  1.67 Gbits/sec
    [  9]   0.00-1.00   sec   199 MBytes  1.67 Gbits/sec
    [ 11]   0.00-1.00   sec   199 MBytes  1.67 Gbits/sec
    [ 13]   0.00-1.00   sec   200 MBytes  1.67 Gbits/sec
    [SUM]   0.00-1.00   sec   996 MBytes  8.36 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [  5]   1.00-2.00   sec   199 MBytes  1.67 Gbits/sec
    [  7]   1.00-2.00   sec   199 MBytes  1.67 Gbits/sec
    [  9]   1.00-2.00   sec   199 MBytes  1.67 Gbits/sec
    [ 11]   1.00-2.00   sec   199 MBytes  1.67 Gbits/sec
    [ 13]   1.00-2.00   sec   199 MBytes  1.67 Gbits/sec
    [SUM]   1.00-2.00   sec   994 MBytes  8.34 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [  5]   2.00-3.00   sec   194 MBytes  1.62 Gbits/sec
    [  7]   2.00-3.00   sec   196 MBytes  1.65 Gbits/sec
    [  9]   2.00-3.00   sec   203 MBytes  1.70 Gbits/sec
    [ 11]   2.00-3.00   sec   190 MBytes  1.59 Gbits/sec
    [ 13]   2.00-3.00   sec   186 MBytes  1.56 Gbits/sec
    [SUM]   2.00-3.00   sec   968 MBytes  8.12 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [  5]   3.00-4.00   sec   205 MBytes  1.72 Gbits/sec
    [  7]   3.00-4.00   sec   201 MBytes  1.69 Gbits/sec
    [  9]   3.00-4.00   sec   204 MBytes  1.71 Gbits/sec
    [ 11]   3.00-4.00   sec   202 MBytes  1.69 Gbits/sec
    [ 13]   3.00-4.00   sec   170 MBytes  1.43 Gbits/sec
    [SUM]   3.00-4.00   sec   982 MBytes  8.24 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [  5]   4.00-5.00   sec   200 MBytes  1.68 Gbits/sec
    [  7]   4.00-5.00   sec   200 MBytes  1.68 Gbits/sec
    [  9]   4.00-5.00   sec   200 MBytes  1.67 Gbits/sec
    [ 11]   4.00-5.00   sec   200 MBytes  1.67 Gbits/sec
    [ 13]   4.00-5.00   sec   189 MBytes  1.58 Gbits/sec
    [SUM]   4.00-5.00   sec   988 MBytes  8.29 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [  5]   5.00-6.00   sec   199 MBytes  1.67 Gbits/sec
    [  7]   5.00-6.00   sec   199 MBytes  1.67 Gbits/sec
    [  9]   5.00-6.00   sec   199 MBytes  1.67 Gbits/sec
    [ 11]   5.00-6.00   sec   198 MBytes  1.66 Gbits/sec
    [ 13]   5.00-6.00   sec   196 MBytes  1.64 Gbits/sec
    [SUM]   5.00-6.00   sec   991 MBytes  8.31 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [  5]   6.00-7.00   sec   187 MBytes  1.56 Gbits/sec
    [  7]   6.00-7.00   sec   186 MBytes  1.56 Gbits/sec
    [  9]   6.00-7.00   sec   186 MBytes  1.56 Gbits/sec
    [ 11]   6.00-7.00   sec   186 MBytes  1.56 Gbits/sec
    [ 13]   6.00-7.00   sec   187 MBytes  1.57 Gbits/sec
    [SUM]   6.00-7.00   sec   932 MBytes  7.82 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [  5]   7.00-8.00   sec   142 MBytes  1.19 Gbits/sec
    [  7]   7.00-8.00   sec   142 MBytes  1.19 Gbits/sec
    [  9]   7.00-8.00   sec   142 MBytes  1.19 Gbits/sec
    [ 11]   7.00-8.00   sec   142 MBytes  1.19 Gbits/sec
    [ 13]   7.00-8.00   sec   142 MBytes  1.19 Gbits/sec
    [SUM]   7.00-8.00   sec   708 MBytes  5.94 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [  5]   8.00-9.00   sec   139 MBytes  1.17 Gbits/sec
    [  7]   8.00-9.00   sec   139 MBytes  1.17 Gbits/sec
    [  9]   8.00-9.00   sec   139 MBytes  1.17 Gbits/sec
    [ 11]   8.00-9.00   sec   139 MBytes  1.17 Gbits/sec
    [ 13]   8.00-9.00   sec   139 MBytes  1.17 Gbits/sec
    [SUM]   8.00-9.00   sec   696 MBytes  5.84 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [  5]   9.00-10.00  sec   184 MBytes  1.54 Gbits/sec
    [  7]   9.00-10.00  sec   184 MBytes  1.54 Gbits/sec
    [  9]   9.00-10.00  sec   184 MBytes  1.54 Gbits/sec
    [ 11]   9.00-10.00  sec   184 MBytes  1.54 Gbits/sec
    [ 13]   9.00-10.00  sec   183 MBytes  1.54 Gbits/sec
    [SUM]   9.00-10.00  sec   918 MBytes  7.70 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-10.00  sec  1.80 GBytes  1.55 Gbits/sec                  sender
    [  5]   0.00-10.00  sec  1.80 GBytes  1.55 Gbits/sec                  receiver
    [  7]   0.00-10.00  sec  1.80 GBytes  1.55 Gbits/sec                  sender
    [  7]   0.00-10.00  sec  1.80 GBytes  1.55 Gbits/sec                  receiver
    [  9]   0.00-10.00  sec  1.81 GBytes  1.56 Gbits/sec                  sender
    [  9]   0.00-10.00  sec  1.81 GBytes  1.55 Gbits/sec                  receiver
    [ 11]   0.00-10.00  sec  1.79 GBytes  1.54 Gbits/sec                  sender
    [ 11]   0.00-10.00  sec  1.79 GBytes  1.54 Gbits/sec                  receiver
    [ 13]   0.00-10.00  sec  1.75 GBytes  1.50 Gbits/sec                  sender
    [ 13]   0.00-10.00  sec  1.75 GBytes  1.50 Gbits/sec                  receiver
    [SUM]   0.00-10.00  sec  8.96 GBytes  7.70 Gbits/sec                  sender
    [SUM]   0.00-10.00  sec  8.95 GBytes  7.69 Gbits/sec                  receiver
    
    iperf Done.

     

    Of course I have already tested various cables (CAT 7e). I also swapped the ports on the switch.

    I use an Xeon E3-1270 v4 @3,5GHz with 16GB ECC-Ram. 

    If necessary, I can provide access via e. g. Anydesk or TeamViewer. 




    User Feedback

    Recommended Comments

    Finaly i got it:

    In the plugin "Tips and Tweaks" the setting of the "CPU Scaling Governor" was set to "Power Save". I don't know whether I set it that way or whether it came with the beta29.

    This was the reason for the drops. When I switch to "On Demand" or "Performance" everything is fine again. No RX drops.

     

    toskache@Hacky ~ % iperf3 -c 192.168.2.4
    Connecting to host 192.168.2.4, port 5201
    [  5] local 192.168.2.26 port 50768 connected to 192.168.2.4 port 5201
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-1.00   sec  1.09 GBytes  9.38 Gbits/sec
    [  5]   1.00-2.00   sec  1.09 GBytes  9.41 Gbits/sec
    [  5]   2.00-3.00   sec  1.09 GBytes  9.40 Gbits/sec
    [  5]   3.00-4.00   sec  1.09 GBytes  9.40 Gbits/sec
    [  5]   4.00-5.00   sec  1.10 GBytes  9.41 Gbits/sec
    [  5]   5.00-6.00   sec  1.09 GBytes  9.40 Gbits/sec
    [  5]   6.00-7.00   sec  1.09 GBytes  9.40 Gbits/sec
    [  5]   7.00-8.00   sec  1.09 GBytes  9.40 Gbits/sec
    [  5]   8.00-9.00   sec  1.09 GBytes  9.40 Gbits/sec
    [  5]   9.00-10.00  sec  1.09 GBytes  9.39 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-10.00  sec  10.9 GBytes  9.40 Gbits/sec                  sender
    [  5]   0.00-10.00  sec  10.9 GBytes  9.40 Gbits/sec                  receiver
    
    iperf Done.

     

    Link to comment

    Update1:
    I can still see some RX drops, but the iperf3 performance is fine again now. I may play a little more with the RX buffers ...

     

    Update2:

    All Rx-drops are gone!!

    Setting the Rx-Buffer from 256k to 1024 did the trick.


    image.png.449dbc847449ca30375cb5caeb208f20.png

    Edited by Toskache
    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.