3rd Data Disk errors during rebuild

[scuppasteve]

So, i had 1 failed drive. I also needed more space, so i removed a failed drive and another drive to do an upgrade. I precleared them and i added them to the array in place of the previous 2 drives. During the rebuild, a 3rd drive has a large number of read errors, it has never had issues before this. Is there anything that can be done to save this?

[scuppasteve]

If it matters i still have one of the original disks that i pulled that had no issues.

[JorgeB]

Please post the diagnostics.

[scuppasteve]

Sorry forgot about that part. For what it is worth Disk 15 and 21 are the two i pulled. I still have the original working disk 15 if that helps in anyway.

tvserver01-diagnostics-20221121-0951.zip

[JorgeB]

Post a SMART report from old disk15, looks like it might be failing also, if you didn't write to the array since the initial replacement don't for now.

[scuppasteve]

The old disk 15 is not in the server currently, but i ran an extended SMART test and it passed with no errors, as i planned to put it in a different server. Attached is the report that i did with HDDScan, if there is more detail i can run it if need be.

[JorgeB]

Yes, that disk looks OK, nothing written to the serve since you started the replacement?

[scuppasteve]

I am not sure, but i would think there probably was. Nothing that i particularly care about. But probably.

 

EDIT: Actually i know there were things written to it. I stand by my point nothing i care about. But, i think that may make things less likely to work.

Edited November 21, 2022 by scuppasteve

[JorgeB]

Problem is that might make recovery not possible, or at least cause some data loss, you can still try but keep old disk11 intact, that way you can also try and use ddrescue on it.

 

To re-add old disk 15 and try to rebuild both disks you can do this:

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed, including old disk 15 and new disks 11 and 21, replacement disks should be same size or larger than the old ones
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign disks 11 and 21
-Start array (in normal mode now)

 

Grab and post new diags.

 

 

[scuppasteve]

So just for clarity it was drive 21 that was healthy and drive 15 that was bad at first and drive 11 is the one that just had read errors. I substituted your instructions based on me giving you the wrong info in the first place. So i did the following:

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed, including old disk 21 and new disks 11 and 15, replacement disks should be same size or larger than the old ones
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign disks 11 and 15
-Start array (in normal mode now)

 

Just to verify i understood. Anyway diagnostic is attached. Array is only with 11 and 15 not installed.

tvserver01-diagnostics-20221121-1130.zip

[JorgeB]

47 minutes ago, scuppasteve said:

So just for clarity it was drive 21 that was healthy and drive 15 that was bad

That makes sense, like I mentioned disk 15 was logged as a disk problem.

 

Filesystem was not detected on disks 11 and 15, not a very good sign, but click on both disks with the array stopped and change the filesystem from auto to xfs, then post new diags after array start.

 

 

 

 

[scuppasteve]

11 and 15 are not currently mounted and are brand new disks. i would think that would be the expected value. They started a rebuild that was stopped, what data is there should be garbage.

[JorgeB]

The emulated disks should still mount, before starting the rebuild.

[scuppasteve]

They will mount, i guess i was just saying 11 and 15 have no usable data, i wouldnt think them having a filesystem would matter. You told me to unassign 11 and 15, so there would be no filesystem.

[JorgeB]

Yes, but by unassigning them parity together with the other drives should emulate the old contents, assuming parity is still valid, and it might not be since you mentioned there were writes to the array.

[scuppasteve]

Currently it says the contents are emulated. What is the next step? Should a add the drives back and start the array?

[scuppasteve]

Sorry i forgot about the XFS settings for the two disks

tvserver01-diagnostics-20221122-0629.zip

[JorgeB]

Check filesystem on the emulated disks 11 and 15, run it without -n or nothing will be done, if it asks for -L use it.

[scuppasteve]

I am guessing no good

 

 

Phase 1 - find and verify superblock... superblock read failed, offset 0, size 524288, ag 0, rval -1 fatal error -- Input/output error

[JorgeB]

27 minutes ago, scuppasteve said:

Input/output error

this suggest a device problem, or wrong capacity, post new diags after running it.

[scuppasteve]

tvserver01-diagnostics-20221122-0753.zip

[JorgeB]

Nov 22 07:07:32 TVServer01 kernel: md: disk18 read error, sector=41446120
Nov 22 07:07:32 TVServer01 kernel: md: disk18 read error, sector=41446128
Nov 22 07:07:32 TVServer01 kernel: md: disk18 read error, sector=41446136
Nov 22 07:07:32 TVServer01 kernel: md: disk18 read error, sector=41446144
Nov 22 07:07:32 TVServer01 kernel: md: disk18 read error, sector=41446152
.....
Nov 22 07:14:12 TVServer01 kernel: md: disk10 read error, sector=8
Nov 22 07:14:12 TVServer01 kernel: md: disk10 read error, sector=16
Nov 22 07:14:12 TVServer01 kernel: md: disk10 read error, sector=24
Nov 22 07:14:12 TVServer01 kernel: md: disk10 read error, sector=32
Nov 22 07:14:12 TVServer01 kernel: md: disk10 read error, sector=40
Nov 22 07:14:12 TVServer01 kernel: md: disk10 read error, sector=48
Nov 22 07:14:12 TVServer01 kernel: md: disk10 read error, sector=56
Nov 22 07:14:12 TVServer01 kernel: md: disk10 read error, sector=64

 

See what's going with those disks, looks like they dropped offline and reconnected, this is usually a power/connection problem, since there are two emulated disks all other disks must work for Unraid to be able to emulate them.

[scuppasteve]

Its more drive failures, its fucked at this point. 10 i was able to reseat, but 18 is bad, i pulled it and tried to test it on another machine. Can i start the array at this point with the data that is on the functioning drives. If i understand correctly, nothing is affected on the existing data drives. This is due to a series of fan failures and drives overheated. Not much that can be done now i think.

[JorgeB]

3 minutes ago, scuppasteve said:

Can i start the array at this point with the data that is on the functioning drives. If i understand correctly, nothing is affected on the existing data drives.

You can, you can do a new config (Tools -> New config) with the remaining good drives.

[scuppasteve]

Thanks for the patience. This really went sideways.