Rebuild after disc failure super slow, other disk problem any fixes?

[tippo88]

Hi, I have a 10 disk array with one parity and disk one ( 2TB) started showing as no disk, I have replaced disk one  but the rebuild speed is approx 90 kb/s estimated time 200 days.

Looking at the log the problem seems to be  constant disk 3 read errors although disk 3 shows as healthy on the main page.

Am wondering what possible solutions there are for this?

Thanks in advance.

 

[itimpi]

You are likely to get better informed feedback if you attach your system’s diagnostics zip file to your next post in this thread.

[tippo88]

Thanks and apologies , diagnostics attached

tower-diagnostics-20230503-1113.zip

[JorgeB]

Disk3 dropped offline so there's no SMART, check/replace cables and post new diags after array start.

[tippo88]

21 minutes ago, JorgeB said:

Disk3 dropped offline so there's no SMART, check/replace cables and post new diags after array start.

Thanks, sorry if dumb question but just to be sure I pause rebuild, stop array, check cable , restart array, no need to reboot is that correct?

 

[tippo88]

20 minutes ago, tippo88 said:

Thanks, sorry if dumb question but just to be sure I pause rebuild, stop array, check cable , restart array, no need to reboot is that correct?

 

Or do I need to cancel the rebuild as pausing doesn't give me the option to stop the array?

[JorgeB]

Cancel the rebuild and power down.

[tippo88]

Ok so I paused the rebuild, had a look at the server during which I moved it a couple of inches, and then restarted and now saying will be rebuilt in 18 hrs so I I'm guessing cable luckily reseated  when the server moved. I did however get a message saying read error on 3 disks, I want to get the server solid, obvioulsy I need to double check the cable on disk 3  but what else to I need to do?

 

Thanks in advance

tower-diagnostics-20230503-1743.zip

[JorgeB]

DIsk3 is still offline.

[tippo88]

Ok thanks should I leave it going as its going fast now or will I inevitably have to start again?

[JorgeB]

You only have one parity disk, so cannot successfully rebuild a disk with another one missing or with errors.

[tippo88]

Thanks much appreciated, will give it another go tomorrow


Sent from my SM-G985F using Tapatalk

[tippo88]

Ok a bit strange, i had left the office with the rebuild running as above, without doing anything with disk 3. My intention was to stop the rebuild and try and fix disk 3 today, However when I got to the office this morning it seems the rebuild successfully completed which is very strange, So am wondering if all good now and I should shut down and check disk 3 cables or if the rebuild appears to be complete but somehow isn't??

tower-diagnostics-20230504-1151.zip

[tippo88]

After shutting down and playing with disk 3 cables every time I start I seem to get different disk errors. Latest time with disk 6 as well  as 2 3 and 8. Wondering what the best way forward is?

 

Thanks in advance

[JorgeB]

4 hours ago, tippo88 said:

it seems the rebuild successfully completed which is very strange,

Rebuild will still complete with errors in another disk, but those sectors are skipped, so the rebuild disk will likely have some (or a lot) of corruption.

[tippo88]

Ok so I replaced all the  disk cables in my system with better quality locking ones and now the system is behaving more consistently but seem to have problem with disk 6 now. I guess only solution is to replace this? Then still have errors showing on 3 other disks so anything I can do about these?

 

Thanks in advance

tower-diagnostics-20230505-1209.zip

[JorgeB]

For now run an extended SMART test on disk8 and also check filesystem on disk1, then post new diags

[tippo88]

thanks will do

 

[tippo88]

Here is the smart test on disk 8 and diagnostics after running the file system check on disk 8

thanks!

tower-diagnostics-20230505-1837.zip tower-smart-20230505-1732.zip

[JorgeB]

Test failed, so the disk needs to be replaced, of course since there's already a disabled disk and you only have single parity some data loss can occur.

[tippo88]

Sorry was a typo in my last message diagnostics after system check on disk one not 8.

Sorry being a bit dumb but I assume its disk 8 needs replacing as failed the smart test correct? so I need to replace disks 6 and 8.  Do i need to do some file system repair on disk 1 first?

I assume I replace 1 disk at a time, first 8 then 6 or doesn't it matter?

 

Thanks again.

[JorgeB]

13 minutes ago, tippo88 said:

Sorry being a bit dumb but I assume its disk 8 needs replacing as failed the smart test correct?

See above.

 

14 minutes ago, tippo88 said:

Do i need to do some file system repair on disk 1 first?

Yes, but don't need to replace the disk, at least not for now.

 

Rebuild disk6 first, then replace disk8, like mentioned there might be some data loss due to disk8 also being bad, do you still have old disk6?

[tippo88]

Yep still have old disk 6 is still in the system haven't replaced it yet

So just to be sure

Repair disk 1 

replace disk 6 and rebuild then replace disk 8 and rebuild again correct?

 

Very much appreciated!

 

[JorgeB]

2 hours ago, tippo88 said:

Repair disk 1 

replace disk 6 and rebuild then replace disk 8 and rebuild again correct?

 

Correct, keep old disk6 intact and see how many errors there are on disk8 during the rebuild.

[tippo88]

On log from repairing disk 1 i get some entries like this

So I guess these file have gone and will need replacing?

Also shows this:

These are some of my shares and I'm wondering if I need to be trying to fix this before putting in new disk 8 and rebuilding or should just go ahead and rebuild a new disk 8.

 

Thanks!