Low disk write speeds (kb/s)

[broadcastthebadger]

Hi all, 

 

I'm facing some problems with one of the disks in my array. 

 

Symptoms

- When performing and read/write operations I get speeds largely in the kb/s.
- The issue exists for mover, parity check and rsync operations. 

- SMB shares on that disk will mount but are very slow to populate and unmounts when trying to write data

- A reboot will temporarily resolve this issue seeing speeds closer to 60MB/s but quickly drop back down to the kB/s range. 

- Other disks in the array seem to be fine but the performance has dropped since noticing this issue

- General Unraid performance has been negatively impacted with some apps struggle to load (these apps don't store data this disk)

 

Testing

- SMART test reports no errors 

- A disk speed test (jbartlett777/diskspeed) shows decent speeds in the 100-180MB/s range

- Netdata reports high IO Utilisation on the disk but speeds in the kb/s

hades-diagnostics-20221223-1136.zip

 

Disk affected: ST6000DM003-2CY186_WCT3EA77-20221223-1136 disk1 (sdh)

Edited December 23, 2022 by broadcastthebadger
adding impacted disk information

[JorgeB]

Try this to see if it stops the log spam:

https://forums.unraid.net/topic/118286-nvme-drives-throwing-errors-filling-logs-instantly-how-to-resolve/?do=findComment&comment=1165009

 

Then post new diags during a slow copy

[broadcastthebadger]

As requested, I have added that line to my config. New diagnostics attached.

 

On reboot, mover initially started at 50MB/s (very low compared to what it had been) with the IO limiting disk being the same disk mentioned previously. Mover then dropped down to 1MB/s

hades-diagnostics-20221223-1348.zip

 

After posting this diagnostic, I ran 'mover stop' in CLI and began a parity check which reported speeds of <2MB/s

Edited December 23, 2022 by broadcastthebadger

[JorgeB]

Disk1 is SMR and that specif Seagate family is known to sometimes have very bad write performance, suggest replacing it with a CMR disk, or try to use that disk for mainly WORM data, especially larger files.

[broadcastthebadger]

Thank you for your help. 

I'd just like to understand this a little better if possible.

 

What I'm struggling to understand is why this is only recently a problem and previously that disk had very good performance. This particular disk has been in my array for a while and typically has performed better than the rest in my array.

Further, why has the performance is fine initially but degrades significantly over time. Even for an SMR drive, <5MB/s feels alarming to me

Edited December 23, 2022 by broadcastthebadger

[JorgeB]

1 hour ago, broadcastthebadger said:

Further, why has the performance is fine initially but degrades significantly over time. Even for an SMR drive

That's typical for SMR drives, they usually have a small CMR cache, when that fills up you hit the SMR wall, I have an array with SMR drives only and every few minutes for a couple of minutes writes stay below 5MB/s.

[broadcastthebadger]

I just wanted to provide an update for anyone else who might come across this in the future. 
 

Further investigations revealed that the files I was trying to move (both manually and with mover) had some how ended up partially copied across two disks. New files moved at a regular speed but once these files came to be moved, the speed would drop down to the kb/s range and would never progress.
 

The array became unstable during this process, mover/file transfers struggled to cancel and even when cancelled the array would refuse to reboot. 
 

Solution: Deleting these files from the array (on all disks) resolved this issue. I did this over SSH but there is a file browser app that will do the same thing for anyone less comfortable. 

 

A few days after I resolved this, my flash drive failed. I’m leaning towards this being caused by the multiple unclean shutdowns that had to be forced as the array had been running fine between the initial fix and the drive failing. 
 

Either way, all up and running now. Hope this helps anyone who may come across the same issue.