I had a system failure that completely put the USB drive in a 100% unusable state. I could not recover any of the config folder but was able to reflash the drive. Im trying to understand if I can recover my array. This is a DL380 and I did have a parity drive. can I rebuild this or is it a total loss ? The array most likely has no issues itself but I dont know the order of the drives to put it back correctly , is there a way to see the drive labels ? I'm trying to move slowly so I dont make the issue any worse that it is.

Edited September 6Sep 6 by Dooz Owings

Can I assume that

Dev 2 slot 1 is my Parity Drive
Slot 2 is blank
Dev 8 use to be my SSD Pool I believe

Dev 5 was never used, its internal to the system board.

Is there a way to tell how Dev 1, 3, 4, 6, 7, 9 use to be ordered ?

Am I way off base in having any hope in putting this array back together ?

This should be no problem. UNRAID does not depend on a certain order for the array.

But you should take a bit of care with this operation.

I assume, your array is built on XFS or BTFRS filesystems, not on ZFS!

If it was ZFS, forget the rest of this message and wait for somebody else to help you out! (but with a parity drive it is unlikely ZFS)

---

First the easy part: Dev 8 (Samsung SSD 980...) is likely your "cache" drive. So create a Pool named "cache" for it.

You think the 16Tb Disk in Dev 2 is your parity drive? ok, leave it alone for now, it will only slow down your progress yet.

Create an array containing all those 8Tb drives (I assume all of them were members before?) WITHOUT PARITY FOR NOW!!!

Start the array

This will likely fail because of dirty file systems from the crash before. Nothing to be afraid of, its normal.

If not, skip the next intended lines.

stop the array

start the array in maintanace mode

go to the "start" page and select each disk one by one

hit the "check" button in the middle of the page

pray :-)

once all disks are "cleaned", stop the array, restart the array in normal mode again

Now go to the file manager page and check if your files are visible and complete (or guess they are)

Create Shares and so on, add Users, assign Permissions and so on...

Once everything is in place and works as expected:

stop the array

assign the parity drive to the array

start the array

The parity will be rebuild now, this takes a long time (but it would have taken much longer if you did it right from the start because every change to the disks would have to be written to the parity too, much stepping!)

Hope I did not forget a major step. In doubt, ask Jorge :-)

48 minutes ago, MAM59 said:

This should be no problem. UNRAID does not depend on a certain order for the array.

But you should take a bit of care with this operation.

I assume, your array is built on XFS or BTFRS filesystems, not on ZFS!

If it was ZFS, forget the rest of this message and wait for somebody else to help you out! (but with a parity drive it is unlikely ZFS)

---

First the easy part: Dev 8 (Samsung SSD 980...) is likely your "cache" drive. So create a Pool named "cache" for it.

You think the 16Tb Disk in Dev 2 is your parity drive? ok, leave it alone for now, it will only slow down your progress yet.

Create an array containing all those 8Tb drives (I assume all of them were members before?) WITHOUT PARITY FOR NOW!!!

Start the array

This will likely fail because of dirty file systems from the crash before. Nothing to be afraid of, its normal.

If not, skip the next intended lines.

stop the array

start the array in maintanace mode

go to the "start" page and select each disk one by one

hit the "check" button in the middle of the page

pray :-)

once all disks are "cleaned", stop the array, restart the array in normal mode again

Now go to the file manager page and check if your files are visible and complete (or guess they are)

Create Shares and so on, add Users, assign Permissions and so on...

Once everything is in place and works as expected:

stop the array

assign the parity drive to the array

start the array

The parity will be rebuild now, this takes a long time (but it would have taken much longer if you did it right from the start because every change to the disks would have to be written to the parity too, much stepping!)

Hope I did not forget a major step. In doubt, ask Jorge :-)

Good News / Maybe bad news ????

This server has been up for a while and I could not remember the file system ....

I did what you said, and the array started BUT it looks like it is ZFS ........

Even as ZFS, I can see my data and at least do a backup on my critical stuff I think ? Is there any risk as is ? this array went down gracefully, before the failure. The power down was due to the utility company working on the power to our subdivision. I gracefully shut the server down before they cut the power. The server had been up for so long, I think there was some underlying corruption on the USB boot drive.

SO ...... I think my next step unless there is any objection is to recreate the SMB shares to my NAS and get current with my data back ups. After that I can start to rebuild my docker apps ? Do you see any issues with any of that before I bring the parity drive back into the mix ?

uh, you have a very unhealthy configuration :-(

zfs + Unraid Parity do not really go well together, they fight for ressources and the system slows down to almost unusable.

EITHER use zfs OR parity.

But ok, you have got back your data, thats the most importent part.

I would not recommend to add back parity, instead you should someday:

• move off all data to a backup location/server

• reformat all drives either with XFS+Parity or ZFS-Raid

• move back all data

Yeah this takes a long time, but believe me, the performance will increase dramatically.

(ZFS Raid creates its own Parity internally and splits it among all drives, so in both cases one disk can fail and the data is still accessible and can be recreated onto a new replacement drive)

10 hours ago, MAM59 said:

uh, you have a very unhealthy configuration :-(

zfs + Unraid Parity do not really go well together, they fight for ressources and the system slows down to almost unusable.

EITHER use zfs OR parity.

But ok, you have got back your data, thats the most importent part.

I would not recommend to add back parity, instead you should someday:

• move off all data to a backup location/server

• reformat all drives either with XFS+Parity or ZFS-Raid

• move back all data

Yeah this takes a long time, but believe me, the performance will increase dramatically.

(ZFS Raid creates its own Parity internally and splits it among all drives, so in both cases one disk can fail and the data is still accessible and can be recreated onto a new replacement drive)

That is so strange... I don't understand why I didn't see any issues. This was a production system and ran with zero issues for a LONG LONG time. The array now says it is running in unprotected mode which concerns me. I wont add parity back but dont that leave the array unprotected ? I consider this somewhat fixed . Thank you for your help.

Edited September 6Sep 6 by Dooz Owings

1 minute ago, Dooz Owings said:

I don't understand why I didn't see any issues

It may work for you if your HD controller is clever enough to cache and reorder the write commands.

The problem is:

• ZFS is autotuning and optimizing

• if the disk is "idle", ZFS starts optimizing it like defragmentation or so. It moves blocks around. This is harmless for a single disk.

• The UNRAID parity disk has to follow these write commands

• if you have an array with a lot of ZFS drives, they likely start all at the same time but optimize different blocks.

So the heads of disk 1 are send to track 53, disk 3 works on track 1000, disk 4 on 12 for instance

The poor parity drive has to follow all of them, so it starts stepping around like hell.

This can sum up so that the parity disk is so busy the whole system comes down to its knees.

But if you have a controller with a battery back up ram cache like your HP box likes has onboard, all those writes are cached, reordered and put out in a sequence that uses the smallest amounts and steps for the heads, you may not even notice a performance problem.

But "normal people" with dumb controllers (the mass of us :-) ) suffers a lot.

Solution is either to use no ZFS within an UNRAID array, or use a single ZFS based array (RAIDZ) without a parity drive. ZFS supposely is faster because no need for a 2nd write (to parity), it contains a builtin read cache and so on. Drawback is that those array disks depend on each other, repairs like the one you have done just now are not that easy then.

2 minutes ago, MAM59 said:

It may work for you if your HD controller is clever enough to cache and reorder the write commands.

The problem is:

• ZFS is autotuning and optimizing

• if the disk is "idle", ZFS starts optimizing it like defragmentation or so. It moves blocks around. This is harmless for a single disk.

• The UNRAID parity disk has to follow these write commands

• if you have an array with a lot of ZFS drives, they likely start all at the same time but optimize different blocks.

So the heads of disk 1 are send to track 53, disk 3 works on track 1000, disk 4 on 12 for instance

The poor parity drive has to follow all of them, so it starts stepping around like hell.

This can sum up so that the parity disk is so busy the whole system comes down to its knees.

But if you have a controller with a battery back up ram cache like your HP box likes has onboard, all those writes are cached, reordered and put out in a sequence that uses the smallest amounts and steps for the heads, you may not even notice a performance problem.

But "normal people" with dumb controllers (the mass of us :-) ) suffers a lot.

Solution is either to use no ZFS within an UNRAID array, or use a single ZFS based array (RAIDZ) without a parity drive. ZFS supposely is faster because no need for a 2nd write (to parity), it contains a builtin read cache and so on. Drawback is that those array disks depend on each other, repairs like the one you have done just now are not that easy then.

Than you for all your help. I will take these things into consideration in my next build. I have been toying with the idea of getting a 4U Dell server because this 2U was somewhat limited on the backplane and I wanted something that could support more graphic cards for AI . When I build the next system I guess I will go with BTFRS so that I have protection. I already have a second NAS I back up to but obviously I didnt trigger a backup when I should have. :( Thanks again .