[Solved] Shrink Array with Dual Parity Drives?

[AgentXXL]

Now that I've direct cabled my unRAID setup and removed the failing backplanes in my Norco enclosure, I have two older drives that are indeed failing. I have used the unBalance plugin to move the data off these drives so they're both empty. I'm now ready to shrink the array to remove those drives as I have enough free space for the interim.

 

My question is related to the 'Shrink array' procedure in the Wiki. I will be using the 'Remove drives then rebuild parity' method as described here. However, I also want to re-organize the array while I do this to better group the drives of similar capacity (it's an OCD thing). In the unRAID FAQ here in this forum, the section on re-ordering drives states this:

Quote

Can I reorder my drives within my array?

 

Yes you can UNLESS you have dual parity drives. If you have 2 parity drives, you will have to unassign the second parity drive, then rearrange them as desired and restart the array, then stop the array and reassign the second parity drive and rebuild parity on it again.

 

Warning: Do NOT try to add or remove any drives during this, as that would invalidate parity. And do NOT try this procedure if one or more drives has failed or been removed.

 

Assuming you only have one parity drive assigned, reassign your drives as desired and start the array. You may have to put a check in the box Parity is already valid.

 

The reorder section seems to indicate you can move drives from one slot to another without doing the 'New Config'. Do you have to let the array rebuild parity on the 1st parity drive, then stop the array and assign the 2nd parity drive and again rebuild parity? Or can I just do the 'Shrink array' procedure, re-organize my drives using the 'New Config' with both parity drives still assigned?

 

Can any of the experts clarify the best way for me to both re-org and remove my failing drives? Thanks!

 

[JorgeB]

Since you're going to re-sync parity anyway you can re-order the drives before doing it.

[itimpi]

By default using New Config invalidates parity.      As long as you are happy with this you can re-order and add/remove drives to get to the layout you want.     Then when you start the array Unraid will start building new parity on both parity1 and parity2.    Your data will not be protected until the new parity has been successfully built.

[AgentXXL]

8 minutes ago, johnnie.black said:

Since you're going to re-sync parity anyway you can re-order the drives before doing it.

Sorry, that doesn't answer whether I can just use New Config with dual parity, but @itimpi's reply seems to indicate I can just do the new config and re-order my drives leaving both parity drives assigned.

13 minutes ago, itimpi said:

By default using New Config invalidates parity.      As long as you are happy with this you can re-order and add/remove drives to get to the layout you want.     Then when you start the array Unraid will start building new parity on both parity1 and parity2.    Your data will not be protected until the new parity has been successfully built.

Yes, I was aware that New Config would require a full parity rebuild. Since I want to re-order and remove the drives that are failing, I assume I just leave 'Preserve current assignments' set to none and then proceed with adding the drives to the new config in the order I want.

 

Thanks!

 

[AgentXXL]

One other clarification. The 'Shrink array' procedure says I need to set all shares and the Global share settings to include only the drives I wish to keep. One drive has failed completely and the other is definitely having issues, but I was able to use unBalance to move the data from both the emulated drive and the 2nd failing drive. Since both failing/failed drives are empty and won't be part of the new config/layout, do I need to set the includes for all shares?

 

[JorgeB]

9 minutes ago, AgentXXL said:

Sorry, that doesn't answer whether I can just use New Config with dual parity

After the new config in this case will need to sync parity again, it doesn't matter if you sync one or both.

[JorgeB]

6 minutes ago, AgentXXL said:

Since both failing/failed drives are empty and won't be part of the new config/layout, do I need to set the includes for all shares?

You just need to do that if any of the shares use only some of the remaining disks, disks that aren't going to be in the array don't matter.

[AgentXXL]

Thanks @johnnie.black and @itimpi! I'll do the 'Shrink array' and leave out the failing disks, but know that the rest of the drives won't be protected until my dual parity drives get rebuilt.

[AgentXXL]

The 'New Config' and shrink/removal of the two drives appears to be successful, other than waiting for the parity rebuild. One thing I noticed is that  the number of writes to each parity drive differs. So far it's never more that 5 writes difference, but sometimes Parity 1 has more writes, and other times Parity 2 has more writes. No errors are being reported, UDMA CRC or otherwise. Is this considered normal?

[JorgeB]

Number of writes can be very different between disks doing the same thing, nothing to worry about.

[AgentXXL]

New question but related to this topic: I used xfs_repair on one of the 2 drives that were failing/failed. The other drive died completely and wasn't seen by any OS or even at the BIOS level.

 

I made an image of the 4TB XFS partition using ddrescue before attempting use of xfs_repair.  As reported here, that image isn't mountable. While the xfs_repair run did appear to recover the data from the drive into the 'lost+found' folder, there are still a number of files/folders that are missing.

 

I've restored the image I made with ddrescue to a working 4TB drive that was sitting on the shelf, but the filesystem is still reported as unmountable. In the 'Check Disk Filesystems' wiki article the section on 'Redoing a drive formatted with XFS', it mentions that if xfs_repair failed to recover all files/folders, you can use Testdisk (free) or File Scavenger (paid) to try and recover missing and/or important files.

 

I've got the restored drive attached to my Ubuntu system and have installed Testdisk using 'sudo apt install testdisk'. This added Testdisk 7.0 to my /usr/bin folder under Ubuntu and I'm currently running it. It looks like it's going to take about 12 hrs to scan the disk for partitions/files. In the meantime I thought I'd ask here if others have used any of the paid tools that claim to be able to recover data from XFS formatted drives.

 

Here's some of the ones I've found via some DuckDuckGo searches:

 

File Scavenger - Windows based - $59 US

ReclaiMe - Windows based - $199.95 US

Raise Data Recovery - Windows based - General license €14.95 + XFS support module €12.95 (30 days support)

UFS Explorer - Win/Mac/Linux based - Standard edition €49.95

 

Note that some of these can be discounted at ColorMango: https://www.colormango.com/utilities/data-recovery-file-repair/index.html?filter3=XFS-Recovery

 

I'll certainly let Testdisk run and see what it can find, but curious if others have tried any of the above paid tools, or can recommend something else that has worked for them.

 

 

Edited August 8, 2019 by AgentXXL

[AgentXXL]

So it appears all my issues have now been resolved. I was able to recover the data from the 4TB disk image I made with ddrescue. I went with UFS Explorer Standard for the recovery since the ddrescue image was unmountable. Both it and Raise Data Recovery are made by the same developer - SysDev. I chose UFS Explorer as it has a native Linux version whereas all the others are Windows primarily.

 

Now the only thing that's left is to finish migrating the rest of my data from UD attached and mounted drives to being part of the protected array. 2 x 10TB and 1 x 4TB drives being precleared, with 3 x 10TB worth of data left to import. In any case, thanks again to all those that helped me along the way!