broadcastthebadger Posted December 23, 2022 Share Posted December 23, 2022 (edited) Hi all, I'm facing some problems with one of the disks in my array. Symptoms - When performing and read/write operations I get speeds largely in the kb/s. - The issue exists for mover, parity check and rsync operations. - SMB shares on that disk will mount but are very slow to populate and unmounts when trying to write data - A reboot will temporarily resolve this issue seeing speeds closer to 60MB/s but quickly drop back down to the kB/s range. - Other disks in the array seem to be fine but the performance has dropped since noticing this issue - General Unraid performance has been negatively impacted with some apps struggle to load (these apps don't store data this disk) Testing - SMART test reports no errors - A disk speed test (jbartlett777/diskspeed) shows decent speeds in the 100-180MB/s range - Netdata reports high IO Utilisation on the disk but speeds in the kb/s hades-diagnostics-20221223-1136.zip Disk affected: ST6000DM003-2CY186_WCT3EA77-20221223-1136 disk1 (sdh) Edited December 23, 2022 by broadcastthebadger adding impacted disk information Quote Link to comment
JorgeB Posted December 23, 2022 Share Posted December 23, 2022 Try this to see if it stops the log spam: https://forums.unraid.net/topic/118286-nvme-drives-throwing-errors-filling-logs-instantly-how-to-resolve/?do=findComment&comment=1165009 Then post new diags during a slow copy Quote Link to comment
broadcastthebadger Posted December 23, 2022 Author Share Posted December 23, 2022 (edited) As requested, I have added that line to my config. New diagnostics attached. On reboot, mover initially started at 50MB/s (very low compared to what it had been) with the IO limiting disk being the same disk mentioned previously. Mover then dropped down to 1MB/s hades-diagnostics-20221223-1348.zip After posting this diagnostic, I ran 'mover stop' in CLI and began a parity check which reported speeds of <2MB/s Edited December 23, 2022 by broadcastthebadger Quote Link to comment
JorgeB Posted December 23, 2022 Share Posted December 23, 2022 Disk1 is SMR and that specif Seagate family is known to sometimes have very bad write performance, suggest replacing it with a CMR disk, or try to use that disk for mainly WORM data, especially larger files. Quote Link to comment
broadcastthebadger Posted December 23, 2022 Author Share Posted December 23, 2022 (edited) Thank you for your help. I'd just like to understand this a little better if possible. What I'm struggling to understand is why this is only recently a problem and previously that disk had very good performance. This particular disk has been in my array for a while and typically has performed better than the rest in my array. Further, why has the performance is fine initially but degrades significantly over time. Even for an SMR drive, <5MB/s feels alarming to me Edited December 23, 2022 by broadcastthebadger Quote Link to comment
JorgeB Posted December 23, 2022 Share Posted December 23, 2022 1 hour ago, broadcastthebadger said: Further, why has the performance is fine initially but degrades significantly over time. Even for an SMR drive That's typical for SMR drives, they usually have a small CMR cache, when that fills up you hit the SMR wall, I have an array with SMR drives only and every few minutes for a couple of minutes writes stay below 5MB/s. Quote Link to comment
broadcastthebadger Posted January 3, 2023 Author Share Posted January 3, 2023 I just wanted to provide an update for anyone else who might come across this in the future. Further investigations revealed that the files I was trying to move (both manually and with mover) had some how ended up partially copied across two disks. New files moved at a regular speed but once these files came to be moved, the speed would drop down to the kb/s range and would never progress. The array became unstable during this process, mover/file transfers struggled to cancel and even when cancelled the array would refuse to reboot. Solution: Deleting these files from the array (on all disks) resolved this issue. I did this over SSH but there is a file browser app that will do the same thing for anyone less comfortable. A few days after I resolved this, my flash drive failed. I’m leaning towards this being caused by the multiple unclean shutdowns that had to be forced as the array had been running fine between the initial fix and the drive failing. Either way, all up and running now. Hope this helps anyone who may come across the same issue. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.