(SOLVED) Drive went to disabled while performing a parity upgrade.

[kmorris]

Hi,

 

Yesterday I did a full parity check and everything was great on unRAID 6.1.6.

 

After parity check finished this morning I powered down and replaced the 3TB parity with a 4TB; and started a rebuild. About 6 hours into it my system started showing a large number of errors on the 4th drive so unRAID warned me that parity was (obviously) invalid. It didn't occur to me to grab a system log.

 

I stopped the rebuild, powered down and checked all the cables; turns out that the new parity drive is directly next to the drive showing errors, not only on the physical power cable, but on the pciE controller card as well.

 

From there I removed a couple un-needed power splitters and SATA cables from the system, re-seated everything and brought the system back online.

 

Now I've got Disk 4 showing on my system as disabled, but because the parity wasn't rebuilt I don't have it to rebuild the offline drive.

 

The 3TB drive I swapped out contained valid parity (checked valid yesterday before I started) is still available and no changes have been written to the array from anything other than unRAID.

 

Any idea what my options are?

 

Is there any way to mount the offline drive on another system and copy everything off it?

[garycase]

If you're CERTAIN that nothing's been written to the array since you did the swap, you can do a New Config;  assign the original data drives and the OLD 3TB parity drive; and check the "Parity is already valid" box  (this is known as the "Trust Parity" option).

 

Then Start the array;  then Stop the array and unassign the drive you want to rebuild;  then Start the array again so it shows as missing;  Stop the array and change the options so it does NOT auto-start.    Then shut down and replace the drive.    Reboot;  assign the replacement drive; and then Start the array.    It will now rebuild the drive.

 

 

[kmorris]

I'm certain nothing's been written to it from an external source, only from anything written when assigning a new parity drive to the array and starting the rebuild.

[garycase]

Then you should be good to go with the process I outlined above.

 

[kmorris]

I've put the 3TB back in, and assigned it, but it doesn't give me the option to trust the parity disk.

 

Message under Array Operation is "Too many wrong or missing disks!"

 

 

 

BTW, thank you for your help.

[garycase]

You can't just re-assign the parity disk -- you have to do a NEW CONFIG  (on the Tools tab), and then assign all of your disks and the old parity drive => you'll then see the "Parity is already valid" checkbox.

 

When you do that, disk #4 won't be disabled, as you won't have tried any operations on it.  But you can then force it to be missing (as I outlined above) and can then replace it and its data will be rebuilt onto the new disk.    [Or, if you're confident that the issue was just cabling or power, you could rebuild it onto itself by following the same process]

 

[kmorris]

Done, thanks.

 

Doing a parity check with no write corrections to check first, and will assess the next step from there.

[garycase]

If disk #4 is in fact bad, you'll likely see a bunch of indicated "errors" in parity => but these won't actually be parity errors ... they'll be due to the erroneous data on disk #4.

 

If the parity check completes error-free, then there's nothing wrong with the data on disk #4 ... so no rebuild is necessary.    If that's the case it was likely just a cabling or power issue -- I'd just check the SMART data, and if all looks good then consider the disk fine.

 

[kmorris]

Is a sync error the same as a parity error? Its been running 30 minutes and has 1700+ of them.

[trurl]

Is a sync error the same as a parity error? Its been running 30 minutes and has 1700+ of them.

Yes

[garycase]

If you're confident that you did not in fact do any writes to the array after you had removed the original (3TB) parity drive, then those errors are almost certainly NOT actual parity errors, but are in fact errors in the data on disk #4.    Assuming that's the case, you should get a new disk and replace #4 with it (following the steps I outlined earlier) and it will be rebuilt.

 

I'd do the rebuild on a NEW disk -- NOT on the original #4 -- that way you still have the original disk just in case the issue is something else and you need to attempt data recovery from the original disk #4.    But it's very likely that the disk simply failed, and the rebuilt disk will be just fine.

 

[kmorris]

7.5 hours and 13.6% complete into it, and no additional sync errors since 30 min into it...

 

I'm betting you're right that it's drive 4, it had a couple remapped sectors on the SMART status.

 

Actually a few drives (all Seagate 1.5TBs) started showing low counts of remapped sectors over the past couple months and the parity upgrade was the start of replacing all of them exhibiting a non-ok status with 4TB WD's I picked up recently.

 

Stress... gotta love it.

[kmorris]

Odd, an hour ago it was at 13%, I just refreshed and its dropped back out to maintenance mode and not doing parity check.

 

Disk 4 shows 384 writes, and 768 errors.

 

Every other drive including parity are at lovely 0 errors, but Main tab doesn't show me if parity is valid in maintenance mode, and Dashboard says Data is invalid under Parity Status.

 

If I put in a 4TB to rebuild #4 but the parity is only at 3TB, will I be able to reclaim the extra space later?

 

 

[trurl]

If I put in a 4TB to rebuild #4 but the parity is only at 3TB, will I be able to reclaim the extra space later?

unRAID will not let you start if any data drive is larger than parity so you won't be able to do this at all.

[garycase]

As trurl noted, you can't replace the bad drive with a drive larger than parity.    At this point it seems pretty likely that disk #4 is indeed bad, and you need to replace it.    The best course of action is to rebuild that drive -- but you'll need a new drive of at least 1.5TB and no larger than 3TB.    I'd simply stop the parity check;  Stop the array;  unassign the bad drive;  Start the array so it's shown as "missing";  then Stop the array again.  As I noted earlier, then be sure the Auto-Start option is NOT enabled;  and then shut down.  Now replace the drive;  boot;  assign the new drive; and Start the array => then wait for the rebuild to finish.

 

NOW you'll be ready to replace your 3TB parity with a 4TB drive, and can then start the processing of one-at-a-time replacement of the drives you want to replace.  Note, however, that a few remapped sectors doesn't mean a drive is bad or needs to be replaced.  "Pending" sectors are bad -- reallocated sectors are a normal thing with modern drives ... that's why they have spare sectors.

 

[kmorris]

Took quite a bit of effort, but everything worked out.

 

Replaced original parity drive, rebuilt 4, then did full parity check with no errors.

 

Decided to grab a 6tb for the parity since I was going through all the trouble and it's 12 hours into upgrading now.

 

All good. Thank you folks.