Extremely slow disk rebuild


4 posts in this topic Last Reply

Recommended Posts

First thanks in advance for your input.

Due to its age (more than 7 years old), one of my disks reported some reallocated sectors and has been replaced with a brand new WD40PURZ.

Initially the rebuild started with unusual slow speed of 30 MB/s only to find out that it additionally slowed down to 3-4 MB/s.

One of the other drives in the array (disk3-3tb) is showing concerning smart results as well (read raw errors) and scheduled to be exchanged next.

While rebuilding I can see disk3 doubled the read raw errors. Both Disk1 and Disk3 are connected to 16 port LSI controller and sharing the same SFF cable.

It deserved to be mentioned that during the disk exchange I haven't moved any cables as drive cages are involved. All cables in the system are Supermicro branded and seated properly. The PSU is 850W Corsair.

The parity drive is connected to the MB Sata controller. 16 of the array disks are connected to a 16 port LSI controller and 2 disks to a Dell Perc H310.

My usual parity check speeds were close to 130-140 MB/s.

Can I blame disk3 to be the bottleneck (due to read raw errors) or this is more like a controller/cable/PSU related issue?

Unraid 6.8.2

Diagnostics attached.

 

 

 

 

unraid-diagnostics-20200307-1753.zip

Link to post
On 3/7/2020 at 6:54 PM, Alexandro said:

Can I blame disk3 to be the bottleneck (due to read raw errors) or this is more like a controller/cable/PSU related issue?

I would guess disk3 is the most likely culprit, but it can get better at any point since these are usually bad areas, though it can also end up giving some read errors.

 

Unrelated to this, you have some filesystem complaining of low space:

 

Mar  6 20:13:58 UNRAID kernel: Filesystem "md11": reserve blocks depleted! Consider increasing reserve pool size.
Mar  6 20:13:58 UNRAID kernel: XFS (md11): Per-AG reservation for AG 0 failed.  Filesystem may run out of space.
Mar  6 20:13:58 UNRAID kernel: XFS (md11): Per-AG reservation for AG 1 failed.  Filesystem may run out of space.
Mar  6 20:13:58 UNRAID kernel: XFS (md11): Per-AG reservation for AG 2 failed.  Filesystem may run out of space.

 

There are a various disks with less than 50MB free, you might have issues if you need for example to run a filesystem check in any of those, you should leave a few GB free, like 10/20GB.

 

Filesystem      Size  Used Avail Use% Mounted on
/dev/md2        2.8T  2.8T  4.4M 100% /mnt/disk2
/dev/md6        2.8T  2.8T   45M 100% /mnt/disk6
/dev/md7        2.8T  2.8T   46M 100% /mnt/disk7
/dev/md9        2.8T  2.8T   48M 100% /mnt/disk9
/dev/md11       2.8T  2.8T   40M 100% /mnt/disk11
/dev/md12       3.7T  3.7T   38M 100% /mnt/disk12
/dev/md16       3.7T  3.7T   35M 100% /mnt/disk16


 

 

 

 

 

 

 

Link to post

Dear johnnie.black,

Thank you very much for your reply.

You are right. The disc was the problem. Once the 3tb has been passed the rebuild went with full speed.

Although some really weird results. I have already rebuild successfully Disk3 and decided to preclear the faulty disc. To my surprise the preclear went smoothly reaching almost 120MB/sec. I haven't noticed any change in the parameters mentioned earlier after the preclear. I have decided to give it a  try and again put it under operation in the array. The results were similar. So this disk is obviously dead now.

 

Thank you also mentioning my full discs explaining the warnings in the log. I'll make sure to free up some space according to your recommendation.

Best regards

Link to post
12 hours ago, Alexandro said:

To my surprise the preclear went smoothly reaching almost 120MB/sec.

If that's the writing part of preclear it's normal, when disks get these slow sectors the problem is on reads, writes usually work normally.

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.