It's all broken!! D:

[JorgeB]

Just now, nexusmaniac said:

I'm not going to hit anything until somebody says what, hoping to recover disk3

Already told you, just carefully follow the instructions above.

[nexusmaniac]

Just now, johnnie.black said:

Already told you, just carefully follow the instructions above.

Trurl said press Parity already valid, you've said not to - that's why I hesitated, sorry

 

Should I do what you've said to try and recover my data?

[JorgeB]

Yes, like mentioned follow the instructions carefully, especially the part of not refreshing the GUI after the invalid slot command, or a parity sync will start instead of a disk rebuild.

[nexusmaniac]

ok cool, I'll untick parity valid then?

 

I've run the invalid slot command and not hit refresh.

 

Time to start the array?

[nexusmaniac]

@johnnie.black

 

Safe to ignore the red warning and hit start then? Then just cross my fingers that it'll still begin a rebuild in 6.6  

 

Just after a final confirmation before I do anything I'm not supposed to haha - hopefully it'll all be over and done with soon and I can stop bugging everyone

 

[JorgeB]

24 minutes ago, nexusmaniac said:

ok cool, I'll untick parity valid then?

Yes, and it has to be done before running the invalid slot command, after running the command you can only press the start button, nothing else.

[nexusmaniac]

Just now, johnnie.black said:

Yes, and it has to be done before running the invalid slot command, after running the command you can only press the start button, nothing else.

Am I safe to re-run the invalid command now that the button is unticked?

[JorgeB]

Yes

[nexusmaniac]

Just now, johnnie.black said:

Yes

[JorgeB]

Proceed

[nexusmaniac]

1 minute ago, johnnie.black said:

Proceed

OMG!  

 

Oh thank you so so much, both you and Trurl! 

 

I'm sorry for all the bother I've caused... I'll let you know once the restore is complete but that looks good??

 

[JorgeB]

It would be better if the disk mounted, but hopefully it's fixable or at least recoverable, grab and post new diags so I can see the mount error.

[nexusmaniac]

@johnnie.black

 

Diag attached

raptor-diagnostics-20180904-1004.zip

[JorgeB]

There's metadata corruption on disk3, this results from you having mounted the disks before without parity, so the transactions generations don't match, after the rebuild is finished see this thread, you should be able to use btrfs restore to recover the data but the disk will likely need to be reformatted after, since that error is usually fatal with btrfs.

[nexusmaniac]

2 hours ago, johnnie.black said:

There's metadata corruption on disk3, this results from you having mounted the disks before without parity, so the transactions generations don't match, after the rebuild is finished see this thread, you should be able to use btrfs restore to recover the data but the disk will likely need to be reformatted after, since that error is usually fatal with btrfs.

 

Took a quick look at that thread, will I need enough free space in order to do what's being suggested?

 

I.e. my array is rather full, will I need to somehow add a >3TB disk in order to copy stuff from Disk3 (if it doesn't mount properly after the data rebuild)

 

Cheers!

 

(42% restored so far, yay)

[JorgeB]

7 minutes ago, nexusmaniac said:

Took a quick look at that thread, will I need enough free space in order to do what's being suggested?

Yes, you'll need has much free space as data currently on disk3, or at least the data you want to recover, though you don't have to recover to the array, you can recover to an unassigned disk.

 

The problem was caused by starting the array without parity, there's always some filesystem housekeeping when mounting and unmounting disks read/write that didn't update parity, and with btrfs this is especially dangerous, despite the few bits of invalid parity only affecting metadata, not the data itself, the metadata on the emulated disk has an older transaction number than the disk itself, so it's invalid.

 

This wouldn't be a serious problem if you didn't then assign a data disk to the parity slot, and would only translate to a few sync errors on the first parity check, but for the future if you need to repeat the same the disks should be mounted manually in read-only mode to make sure a similar problem doesn't happen again.

[JorgeB]

Forgot to mention if you don't really have the space or a spare disk you can do it like this:

 

-wait for the rebuild to finish

-unassign disk3 and start the array, it will emulate the disk (and still be unmountable)

-format the emulated disk3

-using btrfs restore try to recover all data from the now unassigned disk3 to the emulated disk

-if successful and when all is done reassign disk3 to rebuild.

[nexusmaniac]

Cheers, loads! 

 

I can borrow a blank 4TB HDD for a couple of days and set that as an unassigned device

 

Having a spare disk sounds like it'll make the whole process a bit easier

 

I'll ping you if I have any more incredibly daft questions!

(You wouldn't think I've been working in a technical role for over 5 years would you 😅 Sadly it's mostly limited to hardware, R&D, Windows related things and some python dev work. Not much in the way of dealing with data storage / NAS etc.)

 

Truly appreciate your help & guidance! 

I've got a couple more hrs left on that data rebuild. Then I'll see what state disk3 is in and start moving things over to a 4TB UD.

[nexusmaniac]

@johnnie.black

 

Hey

 

The data rebuild 'completed' ... I don't know if it has worked? Orrrr...

 

 

There's no emulated storage or anything, thought there would be? Have I misunderstood

 

EDIT:

 

Sorry, I have to stop the array and unassign the disk first... My bad

 

Edited September 4, 2018 by nexusmaniac

[JorgeB]

5 minutes ago, nexusmaniac said:

Sorry, I have to stop the array and unassign the disk first... My bad

No need, because the filesystem is corrupt the disk won't mount, next step is to use btrfs restore like I linked earlier.

[nexusmaniac]

1 minute ago, johnnie.black said:

No need, because the filesystem is corrupt the disk won't mount, next step is to use btrfs restore like I linked earlier.

If the contents are emulate, shouldn't I have a file thing next to disk3?

 

 

I'll continue with the link you sent 🤞

[nexusmaniac]

Well this is embarrassing:

Time to sit for another 7hrs

 

3 hours ago, johnnie.black said:

-unassign disk3 and start the array, it will emulate the disk (and still be unmountable)

Eek, that read as needing to stop the array in order to unassign the disk so I did.

 

10 minutes ago, johnnie.black said:

No need, because the filesystem is corrupt the disk won't mount, next step is to use btrfs restore like I linked earlier.

Which I did before I got this, apologies.

 

Don't want to cancel this operation do I 😇

 

[JorgeB]

1 minute ago, nexusmaniac said:

Eek, that read as needing to stop the array in order to unassign the disk so I did.

That was for if you didn't have the space or a spare to btrfs restore to, since you'll have a spare disk there's no need, you should wait for the rebuild to finish again.

[nexusmaniac]

1 minute ago, johnnie.black said:

That was for if you didn't have the space or a spare to btrfs restore to, since you'll have a spare disk there's no need, you should wait for the rebuild to finish again.

Gotcha, my bad.

 

I've got the UD mounted and ready. 

 

I shall see if I can stay up late enough to kick off some of those btrfs commands

[nexusmaniac]

@johnnie.black

 

root@Raptor:~# cd /x/
root@Raptor:/x# mount -o recovery,ro /dev/sdf1 /x
mount: /x: wrong fs type, bad option, bad superblock on /dev/sdf1, missing codepage or helper program, or other error.
root@Raptor:/x# mount -o degraded,recovery,ro /dev/sdf1 /x
mount: /x: wrong fs type, bad option, bad superblock on /dev/sdf1, missing codepage or helper program, or other error.
root@Raptor:/x#

Moving onto recovery (step 2 in your thread)

Quote

2) BTRFS restore (non-destructive)