Parity rebuild one drive

[slize]

Hallo,

i got one failed drive in my array and need to rebuild the data on it.

Can i rebuild it without a new drive? I got enough free space, the failed drive was a 10TB. 

 

Also what is the propper way to recover the drive? 

 

 

 

Shouldn't the disk be emulated and the data immediately accessible?

 

Edit: Does unraid backup the LUKS headers of the encrypted disks?

Edited May 5, 2021 by slize

[JorgeB]

43 minutes ago, slize said:

Can i rebuild it without a new drive? I got enough free space, the failed drive was a 10TB. 

No, but you could move all the data from the emulated disk to other disk(s) then remove it from the array.

[slize]

10 minutes ago, JorgeB said:

No, but you could move all the data from the emulated disk to other disk(s) then remove it from the array.

Thanks for the fast reply. Okay the data is very important for me - im just going to get a new 10TB drive and i hope that it does lower the risk of losing anything by simplifying the process. (My backup is on year old because the rsync to an offsite server seems to have failed since a long time - my bad, my issue but lets hope i can recover the drive)

So i just put the new drive in and start the array?

Edited May 5, 2021 by slize

[JorgeB]

Just now, slize said:

So i just put the new drive in and start the array?

Assign it to that slot and start the array.

[slize]

11 minutes ago, JorgeB said:

Assign it to that slot and start the array.

Awesome, finger crossed - thank you.

[trurl]

2 hours ago, slize said:

Shouldn't the disk be emulated and the data immediately accessible?

According to the screenshot you added after those replies, the emulated disk is unmountable, which means filesystem corruption that will need repairing, possibly with some data loss.

 

2 hours ago, slize said:

the data is very important for me

You must always have another copy of anything important and irreplaceable. Parity is NOT a backup.

 

Usually we suggest repairing the emulated disk before rebuild.

 

You probably would have gotten better advice if you had attached your diagnostics in your first post.

 

Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post in this thread.

 

 

 

[slize]

56 minutes ago, trurl said:

According to the screenshot you added after those replies, the emulated disk is unmountable, which means filesystem corruption that will need repairing, possibly with some data loss.

 

You must always have another copy of anything important and irreplaceable. Parity is NOT a backup.

 

Usually we suggest repairing the emulated disk before rebuild.

 

You probably would have gotten better advice if you had attached your diagnostics in your first post.

 

Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post in this thread.

 

 

 

The rebuild is now running. How can i repair an encrypted BTRFS filesystem?

Yes, i am aware about the fact that parity is not a backup. I know, shame on me.

I added the report.

nas-diagnostics.zip

[JorgeB]

7 minutes ago, slize said:

How can i repair an encrypted BTRFS filesystem?

No valid filesystem is being detected, so there's serious damage to the superblock which may make recovery difficult, all your other disks are empty, was that the only disk with data?

[slize]

14 minutes ago, JorgeB said:

No valid filesystem is being detected, so there's serious damage to the superblock which may make recovery difficult, all your other disks are empty, was that the only disk with data?

Yes, they were empty. Yesterday i moved all the data to the missing disk (which was the newest one and running BTRFS encrypted) because i wanted to switch the other disks from XFS to BTRFS encrypted. That worked just fine. 

Today i the data disk reported that its unmountable - after a stop/start of the array.

Edit:
I just attached the "failed" disk to an other system and i think it's physically still fine. (I can read the SMART values and there are no errors) But i cant mount it because the filesystem seems to be invalid. I know that it was running BTRFS and the key that was used for the encryption. Is there a way i can recover the filesystem?

Edited May 5, 2021 by slize

[JorgeB]

23 minutes ago, slize said:

But i cant mount it because the filesystem seems to be invalid

If the disk was empty might as well reformat, but you can try to mount it with UD plugin and if it doesn't post the diags (key needs to be the same as other encrypted disks in the server).

 

 

[slize]

33 minutes ago, JorgeB said:

If the disk was empty might as well reformat, but you can try to mount it with UD plugin and if it doesn't post the diags (key needs to be the same as other encrypted disks in the server).

 

 

 

The broken dies was not empty. It contained all the data from the other disks.

UD cant mount it, the key is the same.

Okay, so i should try to mount it and than post the diags? Can i mount it on any other linux system because the rebuild of the array is running and i dont have a free bay to put the old disk in.
 

I dumped the first 10MB of the raw disk. I dont know if the header is correct - it has a lot of 0's until block 0x00010000.

I compared the headers of the "broken" disk and one of the running disks in the array - they are totally different... 

 

BrokenDisk.bin WorkingDisk.bin

Edited May 5, 2021 by slize

[JorgeB]

22 minutes ago, slize said:

Okay, so i should try to mount it and than post the diags?

Yes.

[slize]

10 minutes ago, JorgeB said:

Yes.

I will do it as soon as the rebuild is done. I tried to mount it via an external USB dock, but the button was just greyed out.

As for my understanding:
Unraid uses BTRFS as filesystem and than uses LUKS1 or LUKS2 to encrypt the whole disk?

[JorgeB]

First LUKS must be open, only then it gives access to the filesystem, and encryption will always add an extra complexity layer, I recommend only using it if really needed.

[slize]

Yes, i am trying to find the luks header. But as far as i can tell: Unraid is using LUKS2, the header is between from offset 0x0 to offset  0x00010000. Does unraid store backups of the luks header? Because it seems like the header is gone and without the header i am unable to access the data on the disk. I got my infos from here: https://security.stackexchange.com/questions/227359/how-to-determine-start-and-end-bytes-of-luks-header

[JorgeB]

Sorry, can't really help with encryption since I've never used it, but try searching the forum, seem to remember others users copying the heather from a different drive.

[slize]

First of all thank you again for your time and answers!

Now the ironic part: Yes, you can copy the header - kind of. The important values are marked here. I am comparing two valid drives. So unless unraid stores a backup of the highlighted values or the rebuild succeeds i'm definitely out of options. By getting myself locked out of the disk. I bet the data on the disk is still fine.

 

 

The part i don't understand is why the header got destroyed.

Edit:

@JorgeB do you know how the drive gets rebuilded from parity? Is it sequential from offset 0 up or random? In case its sequential i can already stop the rebuid because the header does not contain any valid luks section.

 

root@nas:~# sudo hexdump -C /dev/sdo -n 100000
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001c0  02 00 ee ff ff ff 01 00  00 00 ff ff ff ff 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
00000210  1a 05 4d cf 00 00 00 00  01 00 00 00 00 00 00 00  |..M.............|
00000220  ff ff 3f 8c 04 00 00 00  22 00 00 00 00 00 00 00  |..?.....".......|
00000230  de ff 3f 8c 04 00 00 00  3f 80 c7 a7 bb 4f da 4a  |..?.....?....O.J|
00000240  95 e0 23 75 8c ef 29 e1  02 00 00 00 00 00 00 00  |..#u..).........|
00000250  80 00 00 00 80 00 00 00  1d af a0 8f 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400  af 3d c6 0f 83 84 72 47  8e 79 3d 69 d8 47 7d e4  |.=....rG.y=i.G}.|
00000410  66 22 62 c1 7d 95 3b 40  a5 4b 11 28 01 2e 8a 8f  |f"b.}.;@.K.(....|
00000420  40 00 00 00 00 00 00 00  de ff 3f 8c 04 00 00 00  |@.........?.....|
00000430  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*

 

Thats the current header of the disk that is getting rebuilded at 20%.

 

Edited May 5, 2021 by slize

[JorgeB]

Rebuild is sequential, but you don't need to start a rebuild for that, whatever the emulated disk shows is the same it will show after the rebuild is complete.

[slize]

Thats confusing: The restored drive is neither the broken nor the intact one (as far as i can tell until now).