Low write and read speeds to/from unRAID (after sleep?)

[GerryGER]

Hi,

 

for a few days now, my unRAID had been bugging me:

Sometimes when I write to a share, my write speed gets stuck on 1,1MB/s. At other times the write speed starts out good (40-110MB/s), but then drops. Reading can get slow too, but more like 10-15MB/s. And sometimes it just works!

I tried different PCs and they all suffer the same problem. Writing to a Windows share from PC to PC works fine. Mounting an external SMB share to unRAID and copying files from it to the array, suffers from the same problem.

 

It doesn't matter what HDD is getting accessed inside unRAID and the SSD Cache has the same problem.

Rebooting unRAID or the PCs/SMB share sometimes helps. No change if I turn all Dockers off.

 

My network hardware hasn't been touched (PCs <-> GBit Switch <-> a good 10m Cat. 7 cable <-> Router <-> unRAID connected to Router), the only thing that changed with unRAID was a new drive (Toshiba MG09 18TB, pre-cleared and then used as Parity) - a quick parity checked showed 140MB/s+ speed. Router reboot doesn't help. Connection from my PC to the router gives full 1GBit/s to the internet, so these cables are fine. I also switched out the cable from unRAID to the router - that worked for a day, next day (after S3 Sleep) the write speed started out ok and later dropped to 1.1MB/s again... I have a feeling that the NIC gets stuck on 10MBit/s after S3 Sleep, but ethtool gives me advertised link mode as 1000baseT, partner too, NIC is supposedly in "1000MBit/s" Mode.

 

I have absolutely no idea what the heck is going on...

 

 

 

tower-diagnostics-20240324-1219.zip

[JorgeB]

40 minutes ago, GerryGER said:

I have a feeling that the NIC gets stuck on 10MBit/s after S3 Sleep, but ethtool gives me advertised link mode as 1000baseT, partner too, NIC is supposedly in "1000MBit/s" Mode.

In that case unlikely that the NIC is the problem, but you can run iperf to confirm. I suspect more a compatibility issue with the S3 plugin.

[OrdinaryButt]

I just got the same problem very recently. My server never sleeps and I don't think it is network related, here is what I changed:
I had 1 single NVMe for cache, it was btrfs
Added 2x SATA NVME in ZFS mirror for apps/vm cache

Added a second NVMe and made a ZFS mirror for storage cache

Shortly after adding them I noticed write speeds to cache down to 20mb/s, but writing to array was above 70 and reading was above 100, so it isn't network. T/s a bit to no avail. Rebooted the server - writes to either cache jump back to 110 saturating 1G NIC.

Next day, writes to cache were again down to 20mb/s. I am suspecting something with ZFS and/or the mirrors but I don't know what. 6.12.8

[JorgeB]

2 hours ago, OrdinaryButt said:

I just got the same problem very recently. My server never sleeps

So it doesn't look like the same problem at all, I would suggest starting your own thread.

[GerryGER]

9 hours ago, JorgeB said:

I suspect more a compatibility issue with the S3 plugin.

I wonder what could cause this? I have been using the S3 plugin for the last 10 years and never experienced something like this.

 

Just now, again it started out perfectly fine, even topping out at 110MB/s, but then gets stuck at 1,xxMB/s...

[itimpi]

It might be worth posting new diagnostics taken while this is happening.   

[JorgeB]

49 minutes ago, GerryGER said:

Just now, again it started out perfectly fine, even topping out at 110MB/s

This is after waking up?

[GerryGER]

23 hours ago, itimpi said:

It might be worth posting new diagnostics taken while this is happening.   

So far I couldn't reproduce the mid transfer drop, but attached a log where it got stuck on 1-3MB/s again. Before going to sleep, I copied some files and it worked fine...

 

23 hours ago, JorgeB said:

This is after waking up?

Yes.

tower-diagnostics-20240325-2301.zip

[JorgeB]

And it never happens after a reboot, if the server doesn't go to sleep?

[GerryGER]

On 3/26/2024 at 10:08 AM, JorgeB said:

And it never happens after a reboot, if the server doesn't go to sleep?

So far, no. Only after sleep it seems. But at the other hand... I never kept the server running... I will keep the server running for today and tomorrow and check what happens.

 

Edit: Ironically, that just happened. I rebooted the server, deactivated S3 sleep, copied files, worked for a few GB and then stuck at 1.1MB/S again.

Diagnostic attached.

tower-diagnostics-20240328-1317.zip

Edited March 28 by GerryGER

[GerryGER]

Meh, I actually wanted to say "Latest unRaid update fixed this", but the problem just occurred again, after working for the last few days...

Diagnostics is right after it happened again, first the files were written to unRaid with like 50-60MB/s and after a few GB written dropped to 1,3MB/s.

tower-diagnostics-20240404-1559.zip

[JorgeB]

I'm sorry but if this only happens after S3 sleep don't have any other suggestion other than disabling it.