Failed Drive, successful rebuild, drive empty

[shutterbug]

After my last parity check this past weekend I found that I had a drive that had been marked 'drive disabled contents emulated'.  My smart error count was high.

 

I had a spare drive that I thought was good, popped it in and began a rebuild, it stopped rebuilding about 2 minutes in and again put this replacement drive in 'drive disabled contents emulated' mode.  So I went out and bought a brand new drive, installed it, formatted it and the array rebuild kicked off.  It ran for 10+ hrs, stats screen shows over 70,000 writes to the new drive. HOWEVER, the drive is entirely empty after the rebuild completed.

 

There are no errors that I see, it completed the rebuild successfully, but not a single byte of data is sitting on this new drive.

 

I don't get it.  I'm assuming I've lost data, but I have no idea what.  What are my next steps to examine what may have happened? I've attached my diagnostics file.

 

Edited October 2, 2018 by shutterbug

[JonathanM]

1 minute ago, shutterbug said:

formatted it

 

1 minute ago, shutterbug said:

the drive is entirely empty after the rebuild completed

Formatting writes a blank table of contents to the emulated drive slot. You formatted the emulated drive and rebuilt that fresh format to the new drive.

 

Do you still have the original drive? You may be able to salvage some data from it.

 

Moral of the story, formatting is NEVER part of a successful rebuild. The content that is emulated is what will be written to the rebuilt drive. If the drive is unmountable as well as disabled, then a separate process is needed to check the file system. Parity rebuild recreates the drive exactly as it was when it failed a write, any file system corruption will be rebuilt as well.

[shutterbug]

Just now, jonathanm said:

 

Formatting writes a blank table of contents to the emulated drive slot. You formatted the emulated drive and rebuilt that fresh format to the new drive.

 

Do you still have the original drive? You may be able to salvage some data from it.

 

Moral of the story, formatting is NEVER part of a successful rebuild. The content that is emulated is what will be written to the rebuilt drive. If the drive is unmountable as well as disabled, then a separate process is needed to check the file system. Parity rebuild recreates the drive exactly as it was when it failed a write, any file system corruption will be rebuilt as well.

When I say I formatted it, unraid forced the format.  i.e. I powered down, installed new drive, when unraid came back online the array was not started and it noted something along the lines of "to continue click here to format the drive".  Once I did that, the rebuild began. This is the same process I've used multiple times to replace a failed drive.

[JonathanM]

Take a look through a few of these threads.

https://forums.unraid.net/topic/56372-is-it-possible-to-erase-your-data-during-a-rebuild/

https://forums.unraid.net/topic/45088-disk-rebuild-lost-data/

https://forums.unraid.net/topic/60526-solved-formatted-an-unmountable-drive-now-what/

 

[itimpi]

39 minutes ago, shutterbug said:

When I say I formatted it, unraid forced the format.  i.e. I powered down, installed new drive, when unraid came back online the array was not started and it noted something along the lines of "to continue click here to format the drive".  Once I did that, the rebuild began. This is the same process I've used multiple times to replace a failed drive.

A format is never part of a rebuild process.   When you put in the new drive unRAID might have offered an option to format it as it would not at that point contain a valid file system.    However continuing with the rebuild (without doing the format) would have put back the data and the old file system.

[shutterbug]

I'm fairly certain I had no choice.  i.e. everything was ghosted out and I could not start a rebuild on the new drive until the 'format drive' box was checked.

Oh well, fortunately I have the failing drive and should be able to recover the files from it.

[JorgeB]

13 minutes ago, shutterbug said:

I'm fairly certain I had no choice.

No, that's not possible, array/rebuild start isn't dependent of formatting.

[GoldenEye22]

This is definitely possible. I just rebuilt a drive for 4 hours and then was told at the end it was successful. But after the rebuild and even after a reboot the disk still said un-mountable  (was mountable before first rebuild) and the only option to fix this was to format the drive otherwise I could not start the array. I formatted and now it is just running the same parity/rebuild process again. Pretty sure data is gone, but seems to be writing a lot again on the second build to disk, probably just zeros. Luckily I don't trust anything and backed up to the other disks in the array to a /backup share when I took the disk offline, since the disk failure seemed fake since it seems to be caused by a sleep command/event occurring during a read/write of that disk. Any clue what I did wrong? I did not pre-clear the disk before first re-build since I expected every location to be written anyways.

[itimpi]

You certainly should not have formatted the drive!   As the warning message that comes up says a format will guarantee that the data on the drive is loss.    The correct way to handle an unmountable drive would have been to run a file system check/repair

[JorgeB]

9 hours ago, GoldenEye22 said:

and the only option to fix this was to format the drive otherwise I could not start the array.

Like mentioned above array start and unmountable disks are completely unrelated, you can start the array with all disks unmountable, in fact you can only format a disk after array start, so when you format it the array must have been running.