(Solved) massive read errors on Parity during rebuild of another drive

[elbobo]

The parity drive is about 3 months old and had no signs of issues during pre-clear, I believe my issue is a bad cable on a 4 drive cage causing the Parity drive to have read errors. The concern I have is I am currently doing a rebuild of drive # 3 in my system, I can't imagine it will finish correctly with 32mil+ read errors on parity. 

What is the best way to handle this? 

The log file is 100% filled with read error lines. 

Do I shutdown, reseat cables and restart the drive rebuild? 

[JorgeB]

6 hours ago, elbobo said:

I can't imagine it will finish correctly with 32mil+ read errors on parity. 

It can't, unless you have dual parity.

 

6 hours ago, elbobo said:

Do I shutdown, reseat cables and restart the drive rebuild?

It worth a try, you should also post the diags.

[elbobo]

Before I could check for replies it claims it has completed:

Quote

 

Total size:4 TB

Elapsed time:8 hours, 3 minutes

Current position:4 TB (100.0 %)

Estimated speed:753.0 MB/sec

Estimated finish:completed

 

It does claim parity is valid, I'm doubtful. 

 

I'm assuming it's because of the 866mil "read error" messages but when I try to gather diagnostics I get this:

Quote

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 134094880 bytes) in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(418) : eval()'d code on line 73

 

I don't know how it (Disk 3) could be complete with only 843.382 writes either there is 2.24TB of data on that drive. 

[JorgeB]

8 minutes ago, elbobo said:

I don't know how it (Disk 3) could be complete with only 843.382 writes either there is 2.24TB of data on that drive. 

The number of writes is meaningless, but with so many errors the rebuilt disk will be mostly garbage, see if you can get the diagnostics on the console by typing diagnostics

[elbobo]

diagnostics worked, thank you. Attached

thank you also for all of your help on this!

tower-diagnostics-20190329-0905.zip

[JorgeB]

Problem appears to be the Marvell controller, besides parity it also dropped one of your cache devices and was very close to dropping the other one, Marvell controllers are not recommended for Unraid for some time now, this is one the reasons, they tend to drop disks.

 

You can try rebuilding again but risk the same problem.

 

Also see here for what to do about the cache pool:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=700582

 

[elbobo]

Is there a controller (or even just a manufacturer) you’d recommend? I built this about 5 years ago and haven’t really looked at changes and recommendations since then. 

If I replace that is it as simple as replacing the card, reseating cables and booting up, then removing and readding drive 3 to rebuild again?

drive 3 was on the card prior to a swap out now it’s using the MB SATA. 

[JorgeB]

6 minutes ago, elbobo said:

Is there a controller (or even just a manufacturer) you’d recommend?

Any LSI with a SAS2008/2308/3008 chipset in IT mode, e.g., 9201-8i, 9211-8i, 9207-8i, 9300-8i, etc and clones, like the Dell H200/H310 and IBM M1015, these latter ones need to be crossflashed.

 

7 minutes ago, elbobo said:

If I replace that is it as simple as replacing the card, reseating cables and booting up, then removing and readding drive 3 to rebuild again?

Basically, unassign disk3, start array, stop array, re-assign disk3 and start array again.

[itimpi]

34 minutes ago, elbobo said:

If I replace that is it as simple as replacing the card, reseating cables and booting up, then removing and readding drive 3 to rebuild again?

drive 3 was on the card prior to a swap out now it’s using the MB SATA. 

I switched from a Marvell controller to a LSI one and once I had installed the card and replugged the drives to the new controller Unraid booted up, recognised the drives on the new controller, and everything was in the same state as before changing the card.

[elbobo]

I have ordered an LSI controller which will arrive today, I have kept my system off and will follow your advice once I have the new card and have it installed. 

 

Thank you for all of your help and guidance!

[elbobo]

 

Sorry, hopefully the last post on this:

Got the new LSI Card today and installed it. Booted up but due to the read errors issue with the parity drive it is in the disabled state.

Drive 3 is in an enabled state but because it "rebuilt" with millions of read errors from the parity drive I do not trust that it is actually rebuilt.

Where do I go from here? 

Thank you

tower-diagnostics-20190331-1707.zip

[elbobo]

Adding one more comment in case it helps with a solution:

Drive 3 that I replaced was throwing a ton of errors, the same as parity, the only SMART issue was UDMA CRC error count so I am assuming that this was also related to the controller. If that's the case, i have that drive as it was when I removed it from the system, so I could possibly rebuild parity off of that (Except: it was a 3TB drive that when I replaced I upgraded to a 4TB drive) 

Just throwing that out there so all of my information is available... I won't do anything until I hear back.

[JorgeB]

10 hours ago, elbobo said:

was UDMA CRC error count so I am assuming that this was also related to the controller.

CRC errors are usually a bad SATA cable.

 

You can re-enable parity and force another disk3 rebuild:

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Assign any missing disk(s) if needed
-Important - After checking the assignments leave the browser on that page, the "Main" page.

-Open an SSH session/use the console and type (don't copy/paste directly from the forum, as sometimes it can insert extra characters):

mdcmd set invalidslot 3 29

-Back on the GUI and without refreshing the page, just start the array, do not check the "parity is already valid" box, disk3 will start rebuilding, disk should mount immediately but if it's unmountable don't format, wait for the rebuild to finish and then run a filesystem check

 

 

[elbobo]

Sorry for the late reply, had some other issues and wasn't able to give this a crack. It worked perfectly, thank you so much for all of your help!