Data loss, poor actions taken

[uaktags]

Hey guys,

 

So, I should preface this by saying that I am solely responsible for the piss-poor response I gave when I was faced with a disabled disk. I have 1 parity, and 5 disks in the array, and realized that 1 (disk 4) was disabled. SMART was showing as good, but for whatever reason the filesystem couldn't be read. I took the following steps (following no guidance, just my own trial and error, whereas I should have asked for help) I attempted to stop the array and start it again to see if that'll fix it. When that didn't work, I then had the idea that if I format the drive, it would be "enabled" and the parity would put the data back. However, format did not do this, and instead just created a blank drive.

 

I currently have the drive pulled and am running "Testdisk" to see if I can pull and recovered files from it, but have the following questions.

1) I noticed that 2 shares disappeared completely. These appear to have originally been primarily stored on this disk, but its just strange that the share config itself is missing from the GUI. Does that sound normal?

 

2) Currently I don't get any "parity-checks" any more but instead "read-checks". Are these the same thing and maybe just an update that I never noticed before?

 

3) I'm not quite sure why the parity didn't replenish the drive, nor could I find (via just google) any manual action to get the parity to force a rebuild. I thought maybe a reboot would have done this, but it didn't. The "Emulated" contents only showed about 3GB of data rather than the 1-2TB that was originally there. Is there a way to verify if the parity still knows of or is able to rebuild this data?

 

4) VMs were for some reason disabled during all of this as well, even though VMs themselves didn't appear affected by any of this. Enabling VM Management started up my two VMs automatically.

 

Currently Disk 4 is still removed from the array. Attached is my diagnostics, but I should note that its from after a few reboots and the drive being pulled a few boots ago. So I completely understand if I fubar'd the situation, but if there's like a step or something that I'm missing (particularly regarding #3 as it'll save me from waiting another 5days for TestDisk and Recuva to recover unstructured data).

tower-diagnostics-20190217-0254.zip

[JorgeB]

9 minutes ago, uaktags said:

1) I noticed that 2 shares disappeared completely. These appear to have originally been primarily stored on this disk, but its just strange that the share config itself is missing from the GUI. Does that sound normal?

If the shares only existed on that disk it's normal they are missing, shares are just top level folders.

10 minutes ago, uaktags said:

2) Currently I don't get any "parity-checks" any more but instead "read-checks". Are these the same thing and maybe just an update that I never noticed before?

Since one disk is disable you can't do a parity check, option will return after the disk is rebuilt.

12 minutes ago, uaktags said:

3) I'm not quite sure why the parity didn't replenish the drive, nor could I find (via just google) any manual action to get the parity to force a rebuild. I thought maybe a reboot would have done this, but it didn't. The "Emulated" contents only showed about 3GB of data rather than the 1-2TB that was originally there. Is there a way to verify if the parity still knows of or is able to rebuild this data?

To re-enable the disk you use this procedure:

https://wiki.unraid.net/Troubleshooting#Re-enable_the_drive

The problem was formatting the disk, format is never part of a rebuild, and parity is updated when you do it, you likely just needed to do a filesystem check, but after format a file recovery util is the only option to try and recover some data.

 

13 minutes ago, uaktags said:

4) VMs were for some reason disabled during all of this as well, even though VMs themselves didn't appear affected by any of this. Enabling VM Management started up my two VMs automatically.

As long as the system and VM shares are not on that disk they should continue to work normally.

 

 

[uaktags]

10 hours ago, johnnie.black said:

The problem was formatting the disk, format is never part of a rebuild, and parity is updated when you do it, you likely just needed to do a filesystem check, but after format a file recovery util is the only option to try and recover some data.

That's what I was afraid of. My hope right now is testdisk can find the missing partition and restore it. I've attempted Partition Wizard, but that doesn't recognize the XFS filesystem and I didn't want to restore the wrong on (it was finding ext2, 3, and 4) partitions which seems strange since the disk only had 1 partition in unraid.

 

Do you have any recommendations for recovery other than testdisk?

[JonathanM]

4 minutes ago, uaktags said:

My hope right now is testdisk can find the missing partition and restore it.

The partition isn't missing, it's been formatted, which means the list of files with disk addresses is gone.

I believe https://www.ufsexplorer.com/ has been used by others with success.

[uaktags]

You're right, perhaps that was an incorrect wording to say, but testdisk has been shown to restore deleted partitions before or atleast provide files of such so that's my current hope. I'll check out the ufsexplorer, aint nothing to lose haha.

 

Thanks both of you guys for prompt responses, and johnnie for providing the links to show the correct steps I should have taken! Appreciate it guys.

[uaktags]

So it looks like UFSExplorer may actually do the trick. Doesn't appear to be able to maintain folder structure integrity, but I am getting filenames and it has start/stop/save capabilities. Also has the ability to only search for XFS filesystems rather than all (testdisk/photorec was finding old Mac data from I guess my VMDKs on the drive?) So this may get me as far along as I can hope for. 

[uaktags]

Awesome suggestion with the UFSExplorer, best spent money yet! Majority of my files were found after having it run continuously and only look against SGI XFS. I thought the real corruption would come in terms of .isos and archives, but the ones I've gone through so far appear to be working out. Haven't yet tried the vmdks, as they're still restoring, but awesome sauce. Should have my data backed up and the drive put back into the array tonight.

 

Thanks guys

[uaktags]

@johnnie.black or @jonathanm,

 

So with the recovery of basically all of my files. Is it safe to store them onto the array right now, while that drive is still disconnected? That's where I've been storing things (I only have about 600GB of freespace on local machine), but noticed an odd thing on unraid that the Free Space of the array has never changed, so I'm worried that putting the drive 4 back in, after all of this recovery, will result in dataloss again.

[JorgeB]

Safe-ish, since it will be on the emulated disk, so if another disk fails before rebuild you'll be in trouble.

[JonathanM]

What johnnie said, I would add that now is a good time to rethink your backup strategy. Perhaps invest in another drive to attach to the local machine, and copy both to the array and the new drive.

 

As you found out, the ability to rebuild a dead drive doesn't guarantee your files integrity, you need to backup things you care about.

[uaktags]

Very true, thanks. I picked up an additional drive and currently going through rsync to move the data. Hoping that in the next 3-4hours I'll be able to bring the drive back online and get the array back into parity.

[uaktags]

Backup went way faster than the restoration via UFS. Fingers crossed, rebuild is starting now. Thus far, looking at /mnt/disk4 I definitely see all the (new) data there, so I think I'm in the clear for now. Basically not going to touch this thing for the next 8hours or so.

[uaktags]

Surprised to be able to say this, but drive was brought online, parity rebuild ran, and all my recovered data survived. While there was certainly some data loss in the form of top level directories and some random source code (found a lot of random node_modules folders) for the most part I've recovered what I believe is 99% of all my expected data. Thanks a lot guys, some definite hard lessons were learned here!