getting sub 10mb rebuild speeds

[Jurak]

I was getting UDMA CRC error rate errors on most of my drives, this caused one of the drive to be failed out. I have verified that drive is still good. I have replaced the cables from the back plane to the LSI card and reseated every drive. Been over 12 hours with no new errors. Now its doing a rebuild on that failed drive but im getting sub 10mb for the rebuild at at times below 1mb. Need some help on why its going so slow. Attached is my diagnostics. There are times some drives will jump up to 80 to 100mb but then drop back down. 

This is just one test on one of the drives  but most of them come back with similar speeds.
 

hdparm -Tt /dev/sds

/dev/sds:
 Timing cached reads:   20482 MB in  2.00 seconds = 10252.49 MB/sec
 Timing buffered disk reads: 590 MB in  3.01 seconds = 196.24 MB/sec

So the drives are able to work faster.

heimdall-diagnostics-20211231-0929.zip

[JorgeB]

Dec 31 09:19:22 Heimdall kernel: sd 7:0:16:0: Power-on or device reset occurred

 

There are multiple errors like this for disk1, this is usually a connection/power problem.

[Jurak]

12 minutes ago, JorgeB said:

Dec 31 09:19:22 Heimdall kernel: sd 7:0:16:0: Power-on or device reset occurred

 

There are multiple errors like this for disk1, this is usually a connection/power problem.

So are these ongoing right now? Its not showing any errors for that drive at this time. I will check it again.

[Jurak]

13 minutes ago, Jurak said:

26 minutes ago, JorgeB said:

Dec 31 09:19:22 Heimdall kernel: sd 7:0:16:0: Power-on or device reset occurred

 

There are multiple errors like this for disk1, this is usually a connection/power problem.

So are these ongoing right now? Its not showing any errors for that drive at this time. I will check it again.

So shut down pulled the drive cleaned everything and put it back in and still getting slow speeds. All of my hard drives are plugged into BPN-SAS2-846EL1 backplane. Could that be going bad?

 

[JorgeB]

Yep, normal for the GUI to not show any errors since the connection is recovering, but it causes timeouts.

 

7 minutes ago, Jurak said:

Could that be going bad?

Are there still similar errors in the log?

[Jurak]

4 minutes ago, JorgeB said:

Are there still similar errors in the log?

Yes

[Jurak]

Now have many drive starting to fail. Shutting server down. I really think its the back plane that is causing this.

 

[JorgeB]

16 hours ago, Jurak said:

I really think its the back plane that is causing this.

Could be, or a cable/power issue.

[Jurak]

Ok back plane replaced. So far everything looks good but still getting very slow rebuild speeds. Am I still having the drives having power issues?

heimdall-diagnostics-20220111-1038.zip

[JorgeB]

There's something writing to the array, it's normal for the rebuild to slow down by a lot during any array access.

[Jurak]

Just out of curiosity can a array drive be rebuild in maintenance mode? Also found that my new LSI card was the culprit for the slow rebuild speeds. Put the old one back in and now getting 80mbs to 120mbs speeds.

[itimpi]

28 minutes ago, Jurak said:

ust out of curiosity can a array drive be rebuild in maintenance mode?

Yes.

[JorgeB]

30 minutes ago, Jurak said:

Just out of curiosity can a array drive be rebuild in maintenance mode?

Yes.