iperf3 - high retr between unraid servers


Recommended Posts

This is weird and I know it but if any one can help I'd appreciate it.

 

I'm getting high retr when iperf3 from my unraid r720 to my unraid r710 constantly:

Quote

4] 325.00-326.00 sec   112 MBytes   937 Mbits/sec    0    235 KBytes       
[  4] 326.00-327.00 sec   106 MBytes   890 Mbits/sec  1468    195 KBytes       
[  4] 327.00-328.00 sec  97.7 MBytes   819 Mbits/sec  1273    195 KBytes       
[  4] 328.00-329.00 sec  97.8 MBytes   820 Mbits/sec  1121    195 KBytes       
[  4] 329.00-330.00 sec   101 MBytes   850 Mbits/sec  1270    195 KBytes       
[  4] 330.00-331.00 sec   112 MBytes   938 Mbits/sec    0    195 KBytes       
[  4] 331.00-332.00 sec   112 MBytes   937 Mbits/sec    0    195 KBytes       
[  4] 332.00-333.00 sec   111 MBytes   930 Mbits/sec    0    195 KBytes       
[  4] 333.00-334.00 sec   110 MBytes   926 Mbits/sec  125    189 KBytes       
[  4] 334.00-335.00 sec  98.1 MBytes   823 Mbits/sec  1914    195 KBytes       
[  4] 335.00-336.00 sec  98.9 MBytes   829 Mbits/sec  2105    195 KBytes       
[  4] 336.00-337.00 sec   102 MBytes   852 Mbits/sec  503    195 KBytes       
[  4] 337.00-338.00 sec   104 MBytes   876 Mbits/sec  105    195 KBytes       
[  4] 338.00-339.00 sec   100 MBytes   840 Mbits/sec  708    195 KBytes       
[  4] 339.00-340.00 sec  99.0 MBytes   830 Mbits/sec  783    195 KBytes       
[  4] 340.00-341.00 sec  98.7 MBytes   828 Mbits/sec  578    195 KBytes       
[  4] 341.00-342.00 sec   100 MBytes   841 Mbits/sec  1344    195 KBytes       
[  4] 342.00-343.00 sec  99.4 MBytes   834 Mbits/sec  377    195 KBytes       
[  4] 343.00-344.00 sec   100 MBytes   842 Mbits/sec  1137    195 KBytes       
[  4] 344.00-345.00 sec   101 MBytes   845 Mbits/sec  317    195 KBytes       
[  4] 345.00-346.00 sec  98.6 MBytes   827 Mbits/sec  1618    195 KBytes       
[  4] 346.00-347.00 sec  99.8 MBytes   837 Mbits/sec  537    195 KBytes       
[  4] 347.00-348.00 sec   101 MBytes   850 Mbits/sec  733    195 KBytes       
[  4] 348.00-349.00 sec  98.0 MBytes   822 Mbits/sec  1570    195 KBytes       
[  4] 349.00-350.00 sec  98.0 MBytes   822 Mbits/sec  2215    195 KBytes       
[  4] 350.00-351.00 sec   100 MBytes   840 Mbits/sec  466    144 KBytes       
[  4] 351.00-352.00 sec   101 MBytes   846 Mbits/sec  977    195 KBytes       
[  4] 352.00-353.00 sec  99.2 MBytes   832 Mbits/sec  2081    191 KBytes       
[  4] 353.00-354.00 sec   100 MBytes   839 Mbits/sec  408    195 KBytes       
[  4] 354.00-355.00 sec  96.9 MBytes   813 Mbits/sec  2085    195 KBytes       
[  4] 355.00-356.00 sec  98.4 MBytes   826 Mbits/sec  1922    195 KBytes       
[  4] 356.00-357.00 sec  98.7 MBytes   828 Mbits/sec  990    195 KBytes      

I thought I narrowed it down to my unifi video docker (when that docker is stopped the retr's stop); however, I get retr's also when during an install or uninstall any dockers. After the install is complete it goes back to zero (when unifi-video container is stopped also).

When playing videos from plex docker (710) which accesses library files from the r720 there are no retr's (with the unifi video docker stopped).

 

I tested all the camera's individually(disconnecting), updated to :beta, changed my POE to inline for testing. If I iperf3 my 10gb connect between my r710 and r720 its fine. I also tested iperf using different 1gb nic ports from the r710 to r720 and still get retrs.

 

I do have some errors in my log:

Quote

Mar 16 00:19:34 R710 kernel: mysqld[22512]: segfault at 0 ip 000055f70275a55c sp 0000146de3cd4240 error 6 in mysqld[55f702278000+513000] Mar 16 00:19:34 R710 kernel: mysqld[22538]: segfault at 0 ip 0000560a0776755c sp 000014678981b240 error 6 in mysqld[560a07285000+513000] Mar 16 00:19:35 R710 kernel: mysqld[22626]: segfault at 0 ip 000055c3a2ab255c sp 0000153e59608240 error 6 in mysqld[55c3a25d0000+513000] Mar 16 00:19:36 R710 kernel: mysqld[22732]: segfault at 0 ip 000055e78b2f355c sp 000014b133c95240 error 6 in mysqld[55e78ae11000+513000] Mar 16 00:19:37 R710 kernel: mysqld[22816]: segfault at 0 ip 000056496853d55c sp 0000154002ed0240 error 6 in mysqld[56496805b000+513000] Mar 16 00:19:38 R710 kernel: mysqld[22936]: segfault at 0 ip 0000561d17e2555c sp 000014ef72bf3240 error 6 in mysqld[561d17943000+513000] Mar 16 00:19:39 R710 kernel: mysqld[23061]: segfault at 0 ip 00005563bf45e55c sp 0000146c5d66c240 error 6 in mysqld[5563bef7c000+513000] Mar 16 00:19:40 R710 kernel: mysqld[23093]: segfault at 0 ip 00005590dce7a55c sp 0000153b3b3a7240 error 6 in mysqld[5590dc998000+513000] Mar 16 00:19:41 R710 kernel: mysqld[23127]: segfault at 0 ip 0000559a3005255c sp 0000146183cdd240 error 6 in mysqld[559a2fb70000+513000] Mar 16 00:19:42 R710 kernel: mysqld[23163]: segfault at 0 ip 000055f7927dd55c sp 0000149ca0856240 error 6 in mysqld[55f7922fb000+513000] Mar 16 00:19:43 R710 kernel: mysqld[23199]: segfault at 0 ip 000056400fb7355c sp 0000151cd6bb1240 error 6 in mysqld[56400f691000+513000] Mar 16 00:19:44 R710 kernel: mysqld[23259]: segfault at 0 ip 00005604a444155c sp 00001523a8072240 error 6 in mysqld[5604a3f5f000+513000] Mar 16 00:19:45 R710 kernel: mysqld[23297]: segfault at 0 ip 000055e285c1555c sp 0000147c8fbd6240 error 6 in mysqld[55e285733000+513000] Mar 16 00:19:46 R710 kernel: mysqld[23325]: segfault at 0 ip 0000560e743c755c sp 000014737ba9d240 error 6 in mysqld[560e73ee5000+513000] Mar 16 06:40:05 R710 kernel: WARNING: CPU: 0 PID: 32599 at net/netfilter/nf_nat_core.c:420 nf_nat_setup_info+0x94/0x666 [nf_nat] Mar 16 06:40:05 R710 kernel: CPU: 0 PID: 32599 Comm: kworker/0:3 Tainted: G W I 4.19.107-Unraid #1 Mar 16 06:40:05 R710 kernel: Call Trace: Mar 16 07:40:36 R710 kernel: WARNING: CPU: 0 PID: 32599 at net/netfilter/nf_nat_core.c:420 nf_nat_setup_info+0x94/0x666 [nf_nat] Mar 16 07:40:36 R710 kernel: CPU: 0 PID: 32599 Comm: kworker/0:3 Tainted: G W I 4.19.107-Unraid #1 Mar 16 07:40:36 R710 kernel: Call Trace: Mar 16 09:09:42 R710 kernel: CPU: 0 PID: 8837 Comm: nginx Tainted: G W I 4.19.107-Unraid #1 Mar 16 09:09:42 R710 kernel: Call Trace: Mar 16 09:51:07 R710 kernel: WARNING: CPU: 9 PID: 12016 at net/netfilter/nf_nat_core.c:420 nf_nat_setup_info+0x94/0x666 [nf_nat] Mar 16 09:51:07 R710 kernel: CPU: 9 PID: 12016 Comm: kworker/9:1 Tainted: G B W I 4.19.107-Unraid #1 Mar 16 09:51:07 R710 kernel: Call Trace:

I have an SSD in my array which causes some errors.

 

I've been messing around with network setting, dockers, hardware. I've lost a little focus and I'm not sure where to go now. I'm tempted just to reinstall the unifi-video and ignore all the retr's. Would that be so bad? I'm not an IT person and it could be a simple error I've made in playing or setting up my homelab.

I'm not including any info on my r720 I've spend a lot of time looking at it also but I believe its somehow network / docker on the 710 related but I don't know.

 

 

r710-diagnostics-20200316-0957.zip

Link to comment

So I thought I'd just solve this the old fashioned way... I uninstalled the unifi-video docker from the r710 and install it on my r720.

 

I get not more retr's from the iperf client from r720 to the iperf3 server on the r710.

BUT now I get retr's when I -R (or reverse the packets) WTF? My r720 server is fairly clean but I get the same issue on a completely different server with different (newer) hardware.

 

I might just create a VM just for the unifi video.

 

Quote

Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 20 second test
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  5]   0.00-1.00   sec   109 MBytes   913 Mbits/sec  111    221 KBytes       
[  5]   1.00-2.00   sec   109 MBytes   913 Mbits/sec  215    221 KBytes       
[  5]   2.00-3.00   sec   111 MBytes   935 Mbits/sec    6    221 KBytes       
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec   29    221 KBytes       
[  5]   4.00-5.00   sec   111 MBytes   929 Mbits/sec   17    221 KBytes       
[  5]   5.00-6.00   sec   109 MBytes   912 Mbits/sec   97    221 KBytes       
[  5]   6.00-7.00   sec   111 MBytes   931 Mbits/sec    9    221 KBytes       
[  5]   7.00-8.00   sec   111 MBytes   934 Mbits/sec   41    221 KBytes       
[  5]   8.00-9.00   sec   111 MBytes   932 Mbits/sec    8    221 KBytes       
[  5]   9.00-10.00  sec   111 MBytes   928 Mbits/sec   58    221 KBytes       
[  5]  10.00-11.00  sec  91.0 MBytes   764 Mbits/sec   36    232 KBytes       
[  5]  11.00-12.00  sec   111 MBytes   932 Mbits/sec   44    232 KBytes       
[  5]  12.00-13.00  sec   111 MBytes   930 Mbits/sec   20    232 KBytes       
[  5]  13.00-14.00  sec   109 MBytes   917 Mbits/sec   37    232 KBytes       
[  5]  14.00-15.00  sec   111 MBytes   930 Mbits/sec   30    229 KBytes       
[  5]  15.00-16.00  sec   111 MBytes   930 Mbits/sec    6    195 KBytes       
[  5]  16.00-17.00  sec   111 MBytes   931 Mbits/sec    0    195 KBytes       
[  5]  17.00-18.00  sec   111 MBytes   929 Mbits/sec   10    195 KBytes       
[  5]  18.00-19.00  sec   109 MBytes   917 Mbits/sec    1    195 KBytes       
[  5]  19.00-20.00  sec   110 MBytes   924 Mbits/sec    8    195 KBytes       
[  5]  20.00-20.04  sec  4.41 MBytes   980 Mbits/sec    0    195 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-20.04  sec  2.14 GBytes   918 Mbits/sec  783             sender
[  5]   0.00-20.04  sec  0.00 Bytes  0.00 bits/sec                  receiver
CPU Utilization: local/sender 3.0% (0.0%u/2.9%s), remote/receiver 1.3% (0.1%u/1.2%s)

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.