Hi all.  I have a server with 20 data drives and 2 parity.  7 drives, that were in a separate box from the main box, connected via LSI 9300 controller, died on me and now show as missing.  Haven't included my logs since, I know the drives are paperweights and hopefully data can be recovered. The good news is the primary box with the parity drives and the 13 other drives are rock solid.  Looks like my PSU in the secondary box did something wonky and torched the drives.

 

I am sending the drives to a data recovery service.  I need help and advice on how I can get back up and running once I get the data back so my parity won't be destroyed during the process.

 

First, I think I need to stop the array and take the parity drives offline so they don't get corrupted.  Do I then "X" out the old drives that are shown as missing ?

 

After I get the dead drives out of the systems memory, with parity drives temporarily uninstalled, I assume I can power down and installed the replacement drives?

 

When I put in the new replacement drives, I then format for XFS but then what? Do I: 1) transfer the recovered data (HOW?) then assigned the drives? Or 2) do I assign the drives then transfer the data?

 

After getting the data transferred and then the drives assigned, Do I put the parity drives back online and go into "New Config" ensuring parity drives remain as they were and the 7 new drives come online as data drives?  I'm assuming at that point, with new config that Parity would see the data hasn't changed just the new drive assignments, yes?

 

I think I'm close on what I need to do, just not the order or how to transfer the data to the replacement drives.

 

Any help, advice, guidance is gratefully appreciated.

 

How did 7 drives just die and how are you so accepting of the fact?

Did you try connecting those presumed dead drives directly to a PC to ensure they are in fact really dead?

Are you sure your LSI is not just dropping offline because of over-heating or this perhaps being some sort of cabling issue?

 

I'd take a step back and assess the situation once again before just sending out your drives somewhere.

It'd certainly help to get better support here if you included your diagnostics package from your server, so please do.

 

Edited May 24, 20242 yr by Rysz

It sound like a potential big loss.

Have you tried to put the drive to a working Linux machine and check if you can see data?

Are you looking to start up your system with 7 replacement drives when waiting for the 7 to be recovered?

 

With 2 parity drives you can recover 2 drives out of your 7. Therefore it would really depend on how many drives could you get back from the recovery service.

- If all 7 are back good --> best. just plug them in and play.

- If 5 are back good, meaning 2 failed, --> still best since you can recover the 2 bad from using the 2 parity drives.

 

Just now, namvan said:

With 2 parity drives you can recover 2 drives out of your 7.

No you cannot - when trying to recover two failed drives you require ALL other data drives to be present and functioning without error.

 

I am also not sure that the drives from the recovery service could be used directly in the Unraid array anyway.  It is highly likely that something will have changed during the recovery that would stop this being possible so it would be necessary to copy the data off of them back to Unraid.

Oops I forgot that fact that ALL other data drives to be present good. That makes sense. Thanks @itimpi

What would be the options to rebuild in this situation? What would be the most positive outcome to expect?

 

Edited May 24, 20242 yr by namvan
syntax

1 hour ago, namvan said:

What would be the options to rebuild in this situation?

I would think your best bet is to reset the array (via Tools->New Config) and build new parity based on the remaining good drives.     You now have a healthy array based on those drives with their data intact.  You can add new drives at this point if you want using normal mechanisms.   If the Data recovery people can return any drives with their data intact then you copy the data off them back to the array and having done that decide if the returned drives are trustworthy enough to be put back into the array (as new drives).

 

Ironically enough this problem although in one sense catastrophic it shows one of the benefits of Unraid in that when you have multiple drive failures well beyond the fault tolerance level of the array you only lose the data on the failed drives.   Traditional RAID based arrays would have meant ALL your data was lost.

24 minutes ago, itimpi said:

I would think your best bet is to reset the array (via Tools->New Config) and build new parity based on the remaining good drives.     You now have a healthy array based on those drives with their data intact.  You can add new drives at this point if you want using normal mechanisms.   If the Data recovery people can return any drives with their data intact then you copy the data off them back to the array and having done that decide if the returned drives are trustworthy enough to be put back into the array (as new drives).

 

Ironically enough this problem although in one sense catastrophic it shows one of the benefits of Unraid in that when you have multiple drive failures well beyond the fault tolerance level of the array you only lose the data on the failed drives.   Traditional RAID based arrays would have meant ALL your data was lost.

 

This, but before doing that I'd make sure those drives are in fact dead and not just dropped offline because of another unrelated issue.

YES, I'm sure the drives are dead.  I did hook them up via a  USB enclosure to my working primary server.  All Drives DOA.  To answer the question of how?  Not sure but I think the PSU spiked in the secondary box since the primary was unaffected.

 

Yes, I bought new cables and tried those....yes, I bought a new PSU....as for the controller, I bought a replacement thinking it could be a controller issue.  Nothing worked. Which is why I didn't include the diagnostics as these drives are all in the secondary box cabled to the main box running unraid.  Secondary box is just drives, PSU , nothing else.  Main box has the mobo, SATA Controllers, drives, Unraid program, etc. I've already tore down the connected box to send out the drives. So running diagnostics is just going to show a working primary rig that shows missing drives.

 

That said, I talked to 3 data recovery companies.  And all they can promise is possibility of  data but not the actual drive itself.  So my question is, based upon all of your kind advice, how do I get the data to the array when I put in the new drives that I'm purchasing, not what the recovery company is providing? Just transfer data externally using Krusader or MC to the drives then rerun parity check?

Now you do a new config and rebuild parity based on the current array, when you get new drives you add them to array and format them as normal, once you get the data you load it as normal either through the network or using unassigned devices.

thanks Kilrah.  Was thinking along those lines but wanted to be sure before I messed up the array more than it is.

Well, good news, bad news....Y'all got me thinking about checking all the drives again.  Bought a new usb enclosure and did the freezer trick.  Guess what, drives were running when I put into enclosure and tested on my primary server.

 

BUT, my backup server still isn't reading the 8 (I mistyped, actually 8 all along, not 7 down). .  So, I decided, get another MOBO and see if it wasn't the PCIe slot that was bad.  Nope. The bios can see the SAS LSI9300-8i, but Unraid isn't reading them So now I've swapped out the LSI9300, all the cables that connect the controller to the 8 drives in the satellite box, and the mobo.  Unraid still shows the 8 as missing with the dreaded red "X".  I checked bios on the MOBO and it's latest.  Unraid is latest version, too. Interestingly, when I would boot, I could watch the Controller initialize and see all 8 drives, but that isn't coming up now before it goes to the bios screen.  Not sure what is going on since, 3 weeks ago, everything was fine. I keep thinking BIOS setting but not sure what since it's doing the same thing on both MOBOs and I've configured them the same.

 

MOBO: ASUS Rog Strix X299-E gaming

CPU: Intel i7-7820X

 

 

Diagnostics attached.

tower2-diagnostics-20240529-2132.zip

Both HBAs are being correctly detected and initialized, so looks to me like the problem is with how the disks are connected or powered, do you hear them spinning up?

Hello Jorge....

 

I am not sure of anything right now.  I took each drive out, put them into an USB enclosure, one by one, and attached to my primary server.  They show up as unassigned and whether I want to format or not....pretty good indicator they are spinning up without going deeper into them and possibly losing data.

 

So, I tore both boxes apart, put them back to original build. Then I took all the drives, they are shucked, so I reapplied thermal tape on all the 3 pins.  Put everything back, nothing. Tried firing them up with just one connected back to the 9300 8E....nothing.  Not sure if they are spinning or not as they sounded pretty quiet and I didn't hear anything click into park when I shut down the box. I'm out of ideas.  I suspect, I'm going to need to replace all the drives, transfer data from old drives to the new, then set up as a new config. Problem is two fold....a ton of $$ for drives and when you're talking 12TB and 18TB drives, we're talking where will I store the data until I can transfer.  Oh the fun!

Does the external enclosure use an expander or is it a direct SATA connections? If the latter, you can hold one of the drives with your hand when powering up, and you would feel if it's spinning up or not.

The enclosure connects to the server via USB connection, but internally its a single drive SATA connection.  What I did was what you suggested. I held the enclosure in my hand, turned it on and felt the centrifugal force of the spin.  That's why I think they were spinning up.  I tried powering the secondary box that the 9300-8e routes to with 2 different PSU's. Nothing! I'm getting some or those combined SATA power/data cables.  I want to see if I connect the separate cables into those then plug straight into the drives will make a difference.  It's possible, everything is so tight that I'm not getting good connections anywhere.  Low probability that is happening but I want to eliminate all that I can.  When I get those, I'm going to run and will post more diagnostic data.

Edited May 31, 20242 yr by isrdude
spelling

Did some more detective work today.  Hooked all 8 drives back into external enclosure, connected to my main server via USB.  My main uNRAID server will see each disk as unassigned. I used Krusader to check the contents. Sadly, the contents come up, show <DIR> but nothing underneath.  When we had some T-Storms, we had a power outage. When power came on, it had an unusual spike that smoked my UPS too.  I think that surge ran up the UPS into the secondary box on my backup server and caused a static surge that wiped the disk contents.  So now, my backup server shows all the disk missing, which makes sense since their is no data now. 

 

I guess only thing to do now is clear the disk from the backups historical disk, set up remaining disk as a New Config, preserving current data, then add the disk back, one by one, format and then let them rejoin the array.....since the data on the drives appear to be gone, might as well see if they are stable and useful again......

 

Thanks for all the help folks.