[fixed] read/write speed drops after a few seconds

[Sledgehamma]

Alright, so I finally built my new server with decent hardware:

 

• ASRock Rack X570D4U-2L2T
• Ryzen 5600X
• 2x 32GB ECC RAM
• 2x 1TB Seagate IronWolf 530 NVMe SSDs
• 2x Seagate Exos 18TB HDD (no parity right now)
• latest stable 6.9.2

 

After I set up everything the way I wanted I started to copy the files (movies, TV shows) from my old Qnap 453A. The problem is that after just a few seconds the speed drops from around 110MB/s to 8MB/s and then even under 1 MB/s. 

 

Here a few things I tried with a 50GB movie file without any luck:

 

• disabled the NVMe Cache so that it wrights straight to the array
• used Krusader Docker
• user rclone from the unRaid server
• mounted the unRaid share on my Qnap via SMB and FTP
• activated tunable
• transferred Windows iso from Macbook, same behavior 
• tried 10Gbit NIC and 1Gbit NIC

 

The thing is when I use NZBGet (Docker) for example I get max Gbit speed for the whole download. I am out of ideas at this point and hope somebody can help me. 

Maybe its some weird BIOS setting I'm missing?

Edited November 12, 2021 by Sledgehamma

[trurl]

diagnostics?

[Sledgehamma]

Here you go. Don't mind the USB reconnecting issue. That was my UPS,  disconnecting it didn't help either. 

stormy-diagnostics-20211101-1929.zip

[Sledgehamma]

I wanted to show a new graph, because the one I posted above made it look like the speeds go up again while in fact it shows three separate copy commands (I aborted them after the speed dropped). 

That said, the write speeds stayed longer this time. For about 35GB I had full speed and then they dropped to 9MB/s.

stormy-diagnostics-20211102-0704.zip

 

//edit:

I just tested the speed on a Windows 10 VM (located on two 240GB Ironwolf 510 NMVe SSDs at raid 1) on unRaid and its the same behavior.  

 

//edit2:

In order to see that its not really an issue with I/O of HDDs I tested transferring via USB 3.0 and I dont have any speed issues on this route. It really seems limited to network.

Edited November 2, 2021 by Sledgehamma

[Sledgehamma]

@trurl: Did you have timte to look into my logs? Please let me know if you need additional information.

 

Just tried another way: copying from my unRaid NAS to my Qnap NAS and I get the same behaviour (see picture). So its not only writing via network but also reading where I get the bottleneck

[Sledgehamma]

After a lot of fiddling with the settings I think I have the solution:

Settings -> Network Settings -> Network Protocol -> IPv4 + IPv6

I cant believe it....

Will test some more and update the thread accordingly. 

Edited November 3, 2021 by Sledgehamma

[Sledgehamma]

Spoke too soon.....

I didn't change anything and its back to the reduced speeds. What the heck is going on here?

[trurl]

On 11/2/2021 at 2:04 AM, Sledgehamma said:

It really seems limited to network.

 

[Sledgehamma]

17 hours ago, trurl said:

 

 

 

root@Stormy:~# iperf3 -c 192.168.178.21
Connecting to host 192.168.178.21, port 5201
[  5] local 192.168.178.83 port 32928 connected to 192.168.178.21 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   112 MBytes   938 Mbits/sec   54    351 KBytes       
[  5]   1.00-2.00   sec   110 MBytes   923 Mbits/sec    0    225 KBytes       
[  5]   2.00-3.00   sec   109 MBytes   912 Mbits/sec    0    225 KBytes       
[  5]   3.00-4.00   sec   109 MBytes   912 Mbits/sec    0    225 KBytes       
[  5]   4.00-5.00   sec   109 MBytes   912 Mbits/sec    0    311 KBytes       
[  5]   5.00-6.00   sec   109 MBytes   912 Mbits/sec    0    220 KBytes       
[  5]   6.00-7.00   sec   110 MBytes   923 Mbits/sec    0    331 KBytes       
[  5]   7.00-8.00   sec   111 MBytes   933 Mbits/sec    0    396 KBytes       
[  5]   8.00-9.00   sec   109 MBytes   912 Mbits/sec    0    334 KBytes       
[  5]   9.00-10.00  sec   110 MBytes   923 Mbits/sec    0    288 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.07 GBytes   920 Mbits/sec   54             sender
[  5]   0.00-10.00  sec  1.07 GBytes   915 Mbits/sec                  receiver
iperf Done.
root@Stormy:~# iperf3 -c 192.168.178.21
Connecting to host 192.168.178.21, port 5201
[  5] local 192.168.178.83 port 32932 connected to 192.168.178.21 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   115 MBytes   967 Mbits/sec    0    259 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   944 Mbits/sec    0    248 KBytes       
[  5]   2.00-3.00   sec   111 MBytes   933 Mbits/sec    0    305 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   944 Mbits/sec    0    245 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   944 Mbits/sec   65    319 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   944 Mbits/sec    0    336 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   944 Mbits/sec    0    245 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   944 Mbits/sec    0    305 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   944 Mbits/sec    0    282 KBytes       
[  5]   9.00-10.00  sec   111 MBytes   933 Mbits/sec    0    331 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   944 Mbits/sec   65             sender
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver

Seems like its retrying?

 

Here is the crit log of my mainboard (but its just the IPMI port, right?):

<130>  2021-11-03T22:02:07.065678+01:00 AMID05099F720F4 spx_restservice: spx_restservice - -  [12024 : 12024 CRITICAL][libipmi_AMIOEM.c:4212]Error in Get Service Configuration in raw IPMI command layer::403 - 
<130>  2021-11-03T22:02:07.065994+01:00 AMID05099F720F4 spx_restservice: spx_restservice - -  [12024 : 12024 CRITICAL][libipmi_AMIOEM.c:4300]Error While Getting service configuration 403 - 

 

Edited November 4, 2021 by Sledgehamma

[Sledgehamma]

@trurl I tried some Wireshark (me= network noob) and found this:

 

Does this help?

Edited November 4, 2021 by Sledgehamma

[trurl]

19 minutes ago, Sledgehamma said:

(me= network noob)

I'm not that good myself. Other than maybe changing cables I don't know what to suggest. Let's see if @JorgeB will take a look.

[JorgeB]

Not a big networking guy, but it does look like a network issue, try different cables/switch/NIC.

[Sledgehamma]

Alright, so its fixed. Not a big fan oof ASRock that's for sure...

 

So it wasn't enough to disable network bonding which I already had:

I also had to disable eth1 (eth0 is the IPMI):

 

It really didn't help that the MAC address of eth1 wasn't listed in unRaid (not blaming Limetech, probably ASRocks fault). Eth0 I can understand because its the IPMI port.

 

Then I used one of the 10GBe ports in unRaid (disabled bonding as well) and put it as eth0 (this is not the same eth0 as the IPMI port) and disabled all other ethernet ports.

If anybody else has this problem or this board, for the love go God do not disable eth0! I did and while eth1 was set to DHCP it didn't get an IP address. While stupid me saw the 0.0.0.0 IP in the BMC I thought it was a display error because DHCP for sure would work....of course it didn't and I had to buy a VGA to HDMI adapter in order to access the BIOS or the BMC because I couldn't access it anymore over LAN.

All in all a pretty bad implementation for a board that costs 500€

 

 

 

Edited November 12, 2021 by Sledgehamma