Unraid Data Drive Rebuild Fluctuating Speed / Performance.

[Aceriz]

Hello All, 

 

I have attached my diagnostics, and photo below.  

 

I had a drive recently fail so purchased a replacement drive. Ran it through 3x preclear with it passing without issues. During the preclear I think it was averaging around 230-240mb/sec. 

 

I added the drive to the array to rebuild the failed drive.   I noticed initially that the rebuild appeared to be writing at around 250mb/s which was the speed that it appeared to run at while doing the pre-clear. (see graph below)

 

However after a period of time into the rebuild it has slowed right down to like 5-7mb/s.  This appeared to fluctuate with times where it spikes up to 125mg/second for a brief period of time. Then again drops down to 5-7mb/sec

 

I am trying to figure out why this is occurring and what I can do to help with process.   I have attached a photo of the stats page which shows the performance of the drives over time since the start of the re-build.  As I was looking things up online found a number of posts that suggest to ensure VMs and dockers etc are all closed. I had the VMs all off from very early on, and then turned off the dockers 1/2 way through without any significant change or improvement in the speed. 

 

rizznetunraid-diagnostics-20230825-0019.zip

[Kilrah]

14 minutes ago, Aceriz said:

As I was looking things up online found a number of posts that suggest to ensure VMs and dockers etc are all closed. I had the VMs all off from very early on, and then turned off the dockers 1/2 way through without any significant change or improvement in the speed. 

It's not "just dockers/VMs", it's "anything that accesses the array".

The only way to go full speed is for nothing whether internal or external to do operations on the array.

[Aceriz]

1 minute ago, Kilrah said:

It's not "just dockers/VMs", it's "anything that accesses the array".

The only way to go full speed is for nothing whether internal or external to do operations on the array.

At this point there shouldnt be anything accessing the array.  I have had no other computers on the network connecting to it.  Not accessing it from my laptop other than to look at the progress page through the browser

 

Is there a way to see if anything else is accessing the array that I might not be aware of?

[JorgeB]

Could be a disk with slow zones, try running the diskspeed docker when the rebuild finishes.

[Aceriz]

On 8/25/2023 at 2:50 AM, JorgeB said:

Could be a disk with slow zones, try running the diskspeed docker when the rebuild finishes.

Okay perfect I will run that once finished.  The initial rebuild of the lost drive finished.  (but i started up the building of my 2nd parity after I recovered the lost drive. Had been on list of to dues.    

 

I have noticed the same behaviour with the building of the second parity drive. 

 

 

When I run Diskspeed what am I looking for specifically ? That way I can perhaps post helpful things here to help with diagnosing any issues 

 

[JorgeB]

27 minutes ago, Aceriz said:

When I run Diskspeed what am I looking for specifically ?

A fluctuating speed graph for one of the disks.

[Aceriz]

8 hours ago, JorgeB said:

A fluctuating speed graph for one of the disks.

So if I have understood and interpreted the graphs correctly I think the drive that is causing my issues is the disk 8 in the graphs below.   it tanks at the last 1 tb each time in terms of read speed as compared to the other drives.    I re tested with the original scanning settings on its own and the same behaviour occurs.

When I test it with more frequent testing zones 2% as opposed to the default 10% it plummeted even further in the last TB.  to speeds that were in keeping to what I was seeing during the data rebuild and build of the 2nd parity drive. It also appeared to stall at the 98% with it sitting there for 20 min (even with the speed check option unchecked)

 

I note that I tested some of the other drives ex (drive 10) which was showing a bit of early variation in the grouped test in the first graph and it smoothed out on retesting, and disk 9 which paused for some time at 61% finished without issues.  

 

Going to run a few more tests on the disk 9 and 10 (at the 2% increments... But thought I would ask 

mind letting me know your thoughts is it likely the disk 8 in your opinion causing my issues ?  before I start a warranty process wanted to ensure I am  reading the graphs correctly lol 

 

 

 

 

 

 

 

 

Edited August 27, 2023 by Aceriz
adding additional details

[JorgeB]

Could be disk8, but since the test only scans a few sectors it's not always easy to be sure, there are utils that do a full surface scan while reporting the speed, that would be better to confirm but you'd need to take the disk out.