USB failed + 1 (data-)drive failed. How to restore data

[Druiff]

So I had a problem with my server which took down atleast the following: (probably a short-circuit as the smell of burnt plastic is very intense)

- PSU

- Motherboard

- USB

- backplane

- 1 HDD

 

Now I am looking to recover the data on the lost drive via the parity.
But as my USB is gone AND the backup was made into the (encrypte) array..

 

How can I retrieve the USB backup? Or is there an other way to restore the unraid installation WITH parity so I can recover the lost disk?
I do have the  passphrase/key to decrypt the array. 

 

Not sure if/how to decrypt the array without unraid. I could connect each disk to my linux PC and try to mount the disks one by one in search of the USB-backup.

[JonathanM]

If you know which drives were in which logical disk slots, and have a replacement drive for the burnt one the same size or larger but not larger than parity, it's fairly straightforward.

 

Glossing over the details, some of which may be very important, you would...

Set up a trial version of Unraid, get it booting and able to access the GUI on a system capable of connecting all the drives

Connect all the drives including the replacement for the burnt drive, assign them to the correct disk slots

Start the array being VERY sure to select the "parity is valid" option. If you don't do this, you will lose the data on the failed drive

Stop the array and remove the replacement drive so parity will emulate the data

Start the array and verify all your drives are mounting with the correct content

Retrieve your backup and put the content of the config folder on your trial USB, overwriting what's currently there

Start the array and see if things are working the way they should

Assign the replacement drive to the emulated disk slot and rebuild it

 

There be dragons, don't move forward unless you either a. know exactly what you are doing and why or b. get help here on the forum with explicit details on each step

 

You will lose data if things aren't done correctly, or parity wasn't in sync when things went boom. The failed drive recovery hinges on parity being perfect at the point the drive died.

[Druiff]

13 minutes ago, JonathanM said:

If you know which drives were in which logical disk slots, and have a replacement drive for the burnt one the same size or larger but not larger than parity, it's fairly straightforward.

 

I don't know exactly which disks are data and which is the parity drive. or in what sequence/diskslot they go.

That's why I think I need the backup to be able to get that running. (Failed drive is a 6TB. I will be replacing it with a drive the exact spec of the Parity drive. So should be good)

 

13 minutes ago, JonathanM said:

You will lose data if things aren't done correctly, or parity wasn't in sync when things went boom. The failed drive recovery hinges on parity being perfect at the point the drive died.

 

That is the assumption that when it said boom, it was still valid. 
So basicly at this point I just need to get the system online and see what's up.

 

What would be the right step when I am not sure which drive belongs were?

[JonathanM]

11 minutes ago, Druiff said:

I don't know exactly which disks are data and which is the parity drive. or in what sequence/diskslot they go.

Did you have alerts set up? The daily health report has that info. As would diagnostics collected before the incident.

[Druiff]

Oef... those notifications haven't come in for a while as I had some issues with it and never bothered!

 

But I was able to use the 'overheat' alerts to determine the slots! (messages were like Jan 2022, but I haven't changed anything since then.)

 

So Plan now is (as soon as the new usb is in):

1. Prepare new usb with trial version.
2. Connect all the disks to the system. 
3. Assign all the disks to there original slots. (including the replacement disk in the slot of disk that is dead)
4. Start the array (make sure "parity is valid"  is checked)
5. Stop the array
6. remove/disconnect the replacement drive
7. Start the array. (this should allow the data to be emulated.)
8. Retrieve backup of USB
9. Overwrite contents of the new/trial usb with the contents in the backup.
10. (reboot the system?)
11. Start the array, check everything is in working order
12. stop the array, and assign the replacement disk on the place where the disk dead. 
    It should start to rebuild the old data.

 

In step3 I am not sure if you want me to only physically connect the drive or also assign it to the array.

 

And would I lower the risk of faults using maintenance mode of the array.?

That way it's read only correct? Or is my understanding wrong of the maintenance mode?

 

 

 

Thanks for the help so far!

[JonathanM]

I believe you are correct, maintenance mode from step 2 until step 7 would be a good idea. In step 3 you need to assign it to the array so Unraid detects the step 6 removal and starts the emulation in step 7.

@trurl@JorgeB, anything I missed? I personally don't have any Unraid encrypted file systems in place so I may have missed something there. Possibly may need to set the file system to encrypted instead of auto on each of the data slots?

 

It wouldn't hurt to have a test drive or two, even old ones, to help validate the trial is working well before you commit to recovery and start attaching the real data disks. Starting completely over with new everything means memtest and at least some burn in hours before you should trust it. It would really suck to find out the new system mangles data while trying to recover a failed disk. I would really work over the trial system in step 1 to uncover any gremlins. After you are satisfied the new system is stable, you will need to set a "new config" to remove the test disks and prepare for step 2 and on.

 

Don't rush through any of this process. I fully expect you to pound on the trial for several days before moving on, preferably starting off with 24 hours of memtest, I'd download the latest free version from https://www.memtest86.com/ and run it.

[JorgeB]

10 hours ago, Druiff said:

Start the array (make sure "parity is valid"  is checked)

Yeah, also check maintenance mode here, rest looks OK, encrypted disk should be detected automatically at first array start, though not sure about the disk to be replaced, that one might need to be changed manually, let me test that later when I have some time, I don't use it and encryption always add another layer that can cause trouble.

[JorgeB]

On 5/30/2023 at 10:57 PM, Druiff said:

remove/disconnect the replacement drive

You can just unassign the drive here.

 

On 5/30/2023 at 10:57 PM, Druiff said:

Overwrite contents of the new/trial usb with the contents in the backup.

Don't copy super.dat or it will restore the old config, i.e. with the old disk enabled, I would first rebuild the disk and only the worry about restoring the rest of the config (again without super.dat).

 

I tested and the procedure itself works the same with encrypted disks, main difference is that the first array start will stop and ask for the encryption key/passphrase, so just don't forget to check parity is already valid and maintenance mode again.

 

 

[Druiff]

@JonathanM  @JorgeB

Thanks for the support!

 

Have been able to recover the data of the failed disk.
All the plugins are gone, but dockers and vm's were instantly up-and-running. (which made sense.. eventually)

 

But can't figure out how to "edit"  the dockers now, as none of them have edit buttons.
Maybe it's a good moment to start over

 

Thanks for the help! Much appreciated! 

[JonathanM]

On 5/30/2023 at 5:57 PM, Druiff said:

Retrieve backup of USB

 

1 hour ago, Druiff said:

All the plugins are gone, but dockers and vm's were instantly up-and-running. (which made sense.. eventually)

 

But can't figure out how to "edit"  the dockers now, as none of them have edit buttons.

All those should be in the backup of the USB, if it was current.