Do I need to put a device into the array in the empty Disk 12 position in order to do the btrfs restore?

No.

Assuming you're restoring to another disk, if restoring to disk12 position best to rebuild first instead of restoring to the emulated disk

OK forget what I posted earlier, I had to re-read the thread to remind myself of the original issue, you'll need to restore from the emulated disk 12 to another disk outside the array, like so:

 

btrfs restore -v /dev/md12 /path/where/to/restore

 

 

I may still be confused, so here's my understanding:

I need to put the actual device that became unmountable (GY9P) into the array, rebuild it, then restore that device to a different device outside of the array (probably 1DAC in this screenshot). Then I can format GY9P and copy the data into the array.

 

Okay, thank you! I understand much better now. I'll set this up and see how it went after work.

I appreciate you working through these issues with me.

Or not. This was the command and output:

root@OMNI:/dev# btrfs restore -v /dev/md12 /mnt/disks/FIXME/restore
checksum verify failed on 2868028276736 found 1C2E52C8 wanted D90661A2
checksum verify failed on 2868028276736 found 1C2E52C8 wanted D90661A2
bytenr mismatch, want=2868028276736, have=3567498873691318199
Couldn't read tree root
Could not open root, trying backup super
checksum verify failed on 2868028276736 found 1C2E52C8 wanted D90661A2
checksum verify failed on 2868028276736 found 1C2E52C8 wanted D90661A2
bytenr mismatch, want=2868028276736, have=3567498873691318199
Couldn't read tree root
Could not open root, trying backup super
checksum verify failed on 2868028276736 found 1C2E52C8 wanted D90661A2
checksum verify failed on 2868028276736 found 1C2E52C8 wanted D90661A2
bytenr mismatch, want=2868028276736, have=3567498873691318199
Couldn't read tree root
Could not open root, trying backup super

Am I out of luck now?

Note that you could also restore to the array but since to read the emulated disks all other disks will be read it's best to use a disk outside the array.

1 minute ago, elecgnosis said:

Am I out of luck now?

Looks like all superblock including the backup ones are corrupt, you can try restoring from old disk12 but it will likely have the same issue.

I decided to reassign 9GYP into the Disk 12 slot and do a rebuild. I figured the worst it could do was fail.

It's gotten farther and gone faster than it did when I was using the Marvell-based controller, but notice the status of Disk 5 (4555) in the screenshot and remember how, when all this began, it was the first drive to "fail".

I'm waiting for the rebuild to hopefully succeed, then I'll replace 4555 and try to rebuild it as well.

 

Edited April 17, 20206 yr by elecgnosis

Looks like disk5 dropped offline, check cables, also post diags.

You mentioned that Disk 12 (9GYP) was already unmountable before it was disabled. Once the Data-Rebuild was finished, it was still unmountable (no filesystem). I'm rebuilding Disk 5 (4555) now, but it may end up the same, in which case I will have lost roughly 5TB of data. I wasn't keeping track of what was on which drive so I don't even know what I've lost.

 

I've posted the current diagnostics, but I'm kind of calling this a wash at this point. I'll wait for your reply, but I have a feeling this is going to end with me formatting any drives without filesystems, doing a fresh parity sync, and setting up something to monitor the system's status better to keep this from happening again.

 

 

Edited April 17, 20206 yr by elecgnosis

2 minutes ago, elecgnosis said:

setting up something to monitor the system's status better

Have you setup Notifications?

3 minutes ago, elecgnosis said:

You mentioned that Disk 12 (9GYP) was already unmountable before it was disabled. Once the Data-Rebuild was finished, it was still unmountable (no filesystem).

That's expected, whatever is on the emulated disk is what's going to be on the rebuilt disk.

 

4 minutes ago, elecgnosis said:

I'm rebuilding Disk 5 (4555) now, but it may end up the same

You shouln't do that, disk dropped offline while rebuilding disk12, so it can't be rebuilt without corruption, it would be best to do a new config to bring online.

 

1 hour ago, trurl said:

Have you setup Notifications?

I had them setup, but I haven't received any regarding array status. I'll review my settings.

 

1 hour ago, johnnie.black said:

That's expected, whatever is on the emulated disk is what's going to be on the rebuilt disk.

 

You shouln't do that, disk dropped offline while rebuilding disk12, so it can't be rebuilt without corruption, it would be best to do a new config to bring online.

 

So I never had a hope of recovering the data on disk 12 (9GYP) to begin with, and disks lost while rebuilding another are gone for good?

Just now, elecgnosis said:

and disks lost while rebuilding another are gone for good?

With single parity if another disk fails/drops out during a rebuild you'll be rebuilding garbage, a single parity disk can only protect you from a single failure, you should read on how parity works, if you understand how it works makes it much easier to know it's limitations.

 

4 minutes ago, elecgnosis said:

So I never had a hope of recovering the data on disk 12 (9GYP) to begin with

Again, that's how parity works, if the emulated disk was corrupt beyond repair (possibly from an earlier incomplete or corrupt rebuild) the same will happen to the rebuilt disk, since they will be identical.

 

6 minutes ago, elecgnosis said:

and disks lost while rebuilding another are gone for good?

Not necessarily, in this case the other disk dropped offline, since disk12 rebuild is corrupt, if you use it to rebuild disk5 it will also be corrupt, garbage in, garbage out, you could have done a new config, disk12 would still be corrupt but disk5 could be brought online using the existing disk.

 

 

 

Last thing: Disregarding the Marvell controller issue, would it have been better to use btrfs tools to restore the lost drive's data first and treating data-rebuild as a kind of last resort?

Those are used for different scenarios, i.e., a drive fails, but the emulated disk mounts you just rebuild, a drive has filesystem corruption, i.e., it's unmountable, you use btrfs restore, now sometimes a drive can be unmountable as a direct result of the failure, in that case you repair/restore the filesystem first and then rebuild, since there's no point in rebuilding a disk if the filesystem is unfixable.