Rebuild help needed

[Lignumaqua]

Server very dead, would appreciate help. I think it’s recoverable but I don’t know how. Here's what happened. 
 

1. All running fine. Using 2 x 4TB parity disks and 7 x 4Tb data disks. 
 

2. Need to increase size so add new 8TB disk with intention of this becoming new parity disk.
 

3. Stop array and disable parity disk 2

 

4. Start array with one 4TB parity disk

 

5. Stop array and designate 8TB disk as new parity disk (replacing 4Tb drive)

 

6. Start array. Parity rebuild automatically restarts. Life is good. 
 

7. A few hours in to the parity rebuild, errors show up on disk 6 and parity rebuild fails. 
 

I still have the original 2 parity disks in the system as unassigned devices and untouched.  Can I reassign them as the parity and rebuild the data on the now failed disk? As it is the whole array is stalled. 😞

Edited October 11, 2021 by Lignumaqua

[Lignumaqua]

We're back. I went to Tools/New Config and reassigned everything back as it had ben in the first place, with the two original parity drives and removed the new 8TB drive. I then checked the "Parity is already valid" checkbox, crossed my fingers, and started the array. So far, so good, I think everything is working again.

 

That was pretty scary. The GUI was frozen, and nothing was working. I had to boot into Safe Mode to get anything going again.

Edited October 11, 2021 by Lignumaqua

[itimpi]

Did you capture your system's diagnostics zip file when the failure happened?  That might give some insight into why you got the failure.  At that point since you still had one valid parity disk then your system should have been able to handle one data disk failing.

 

Have you actually run a parity check to confirm that parity really WAS valid when you put back the original drives?  You do not want another drive to fail and then find out parity was not really valid so recovery is compromised.

[Lignumaqua]

Thank you @itimpi, that’s helpful. Yes, I am currently running a parity check with the old drives. It found a lot of errors not surprisingly. 🙂
 

I also managed to trace the original problem back to a cable (isn’t it always) which triggered CRC errors causing Unraid to disable the drive. I now don’t believe there was an actual disk failure at all. It was a train of issues caused by the cable failure. That triggered the CRC failures which caused Unraid to mark the drive disabled, which, in turn, caused everything else to fail.

 

Unraid behaved as it should with one exception I believe. It shouldn’t have then locked up the GUI so that I was unable to get in to fix anything. 
 

Assuming this parity check finishes in an hour as scheduled, I have two questions please:

 

1. General question - when a parity check fixes errors I assume it does so by changing the parity drive, not by changing the data drives. Is that correct? 

 

2. When I build up the courage to try this again can you confirm that I am following the right procedure to replace dual parity 4TB drives with a single parity 8TB drive? (So I can in the future replace some of my 4TB data drives with 8TB versions.)

 

a. Stop array and remove 2nd parity drive from configuration

b. Restart array

c. Stop array and replace 1st parity drive with new 8TB drive

d. Restart array and rebuild parity onto new 8TB drive (this is where I had the failure last time)

e. Stop array and assign the two old 4TB parity drives as new data drives

f. Restart array

g. Sleep peacefully.

 

[itimpi]

28 minutes ago, Lignumaqua said:

when a parity check fixes errors I assume it does so by changing the parity drive, not by changing the data drives. Is that correct? 

Yes - Unraid always corrects parity to match the drives.

 

29 minutes ago, Lignumaqua said:

a. Stop array and remove 2nd parity drive from configuration

b. Restart array

c. Stop array and replace 1st parity drive with new 8TB drive

You can combine these steps to remove parity2 and insert the new 8TB parity1 into a single step but what you wrote would work as well.

The rest looks fine but keep the old 4TB parity drives intact until you have successfully built new parity on the 8TB drive just in case you get a failure again doing this.

[trurl]

4 hours ago, itimpi said:

keep the old 4TB parity drives intact until you have successfully built new parity on the 8TB drive

but note that if there are any writes to the array while building new parity, the old parity will no longer be valid.

[Lignumaqua]

All worked this time, thank you. BTW I wasn't able to both remove parity2 and replace parity1 in one operation. Unraid complained that was too many changes. Took two steps, but all is fine. Thanks everyone for your help! 🙂