Drive error while building parity drive

[publicENEMY]

Drive 12 have error. so i replace drive 12. after few hours, i successfully build drive 12. then parity drive error. so i replace parity drive. this happens when i rebuilding parity drive. attached logs.

 

Please help.

 

 

syslog.zip

Edited July 2, 2019 by publicENEMY

[JorgeB]

Please post the diagnostics: Tools -> Diagnostics

[publicENEMY]

How long does it normally take to get diagnostics. Its been a few minutes already.

[JorgeB]

It's usually under a minute, if it doesn't work reboot, you'll need to anyway since disk12 dropped offline, and grab diags then.

[publicENEMY]

I have rebooted. and still cant get the diagnostics.

[publicENEMY]

its much worst now. i see this messages non stop.

[JorgeB]

That's filesystem corruption on disk12, if you can't get the diags get the SMART report for disk12, also is disk12 still enable (green icon)?

[publicENEMY]

disk 12 device disabled. contents emulated. red x icon in disk 12.

[publicENEMY]

im curious about the state of the parity drive. disk 12 have problems during parity drive rebuilding.

[publicENEMY]

I run check filesystem status on disk 12. i have attached the full logs.

 

xfs_repair_status.zip

[JorgeB]

10 minutes ago, publicENEMY said:

disk 12 device disabled. contents emulated

OK, so fs corruption is expected since parity is invalid, need SMART report for disk12.

[publicENEMY]

tower-smart-20190702-1938.zip

[publicENEMY]

xfs_repair_status -nv.zip

This time, i run the check filesystem with -nv

[JorgeB]

1 minute ago, publicENEMY said:

This time, i run the check filesystem with -nv 

No point in running xfs_repair on the emulated disk, there will be a lot of data corruption because parity isn't valid.

 

Run an extended SMART test on disk12 and post new SMART report when done, it will take a few hours.

[publicENEMY]

did i have a faulty drive? do i have valid parity drive? if i replace disk 12, would i recover the data?

[JorgeB]

Disk12 doesn't look very good, lots of read raw errors and a pending sector, but let's wait for the SMART test to confirm.

 

You can't currently replace disk12 since parity is invalid, you might be able to re-sync parity (completely or mostly) depending on if disk12 is really failing or not, and if yes how bad it is, part of the problem might also be the SASLP controller that completely crashed (likely when the read error was detected) and dropped the disk offline (these controllerers are not recommended for some time now), using a different controller should allow you to sync most of parity even if the disk is starting to fail, another option would be to clone the disk with ddrescue and then re-sync parity, either way you should be able to recover most data.

[publicENEMY]

Before rebuilding parity, i have working disk12. does that mean that disk 12 might have the data?

[JorgeB]

Data is still there, but if the disk is failing you might not be able to get it all.

[publicENEMY]

Ok. So the parity drive is unreliable right now right? If disk 12 failed, data is lost right?

[JorgeB]

Already said that, parity is invalid, disk12 is likely failing, but most data should be recoverable, wait for the SMART test to finish to see how best to proceed.

[publicENEMY]

Here are the smart report

tower-smart-20190702-2018.zip

[publicENEMY]

can i recover the data in disk 12?

[JorgeB]

Disk is really failing, you now have a few options to try and recover as much data as possible, in order of what I consider best to worst:

 

1)-use ddrescue to clone the disk to a new one then rebuild parity

2)-copy all the data you can from that disk to another then rebuild parity

3)-connect that disk to a different controller, sync parity, it should finish with some read errors, after parity is synced replace disk, IMHO this is the worst option because there's no way to know which files are corrupt unless you have checksums.

[publicENEMY]

How can i do option number 2?

 

If i want to change controller, what controller would you recommend?

[JorgeB]

37 minutes ago, publicENEMY said:

How can i do option number 2?

-Tools -> New Config -> Retain current configuration: All -> Apply

-assign a new disk12, start the array, use for example UD to mount old disk and copy to new disk (array) before or after syncing parity

 

40 minutes ago, publicENEMY said:

If i want to change controller, what controller would you recommend?

Any LSI with a SAS2008/2308/3008 chipset in IT mode, e.g., 9201-8i, 9211-8i, 9207-8i, 9300-8i, 9400-8i, etc and clones, like the Dell H200/H310 and IBM M1015, these latter ones need to be crossflashed.