Help: accidentally formatted a full drive after restarting


Recommended Posts

Hi there, I did something really stupid. My unRAID system froze so I rebooted it. Upon restarting the array, I saw that on one of my 14TB disks, it showed unreadable file system and contents were emulated. Because I just flew back from a work trip and severely jetlagged, I did a cursory lookup of how to fix it but didn't follow instructions properly. Basically, I thought I was going to format the disk and unRAID will automatically rebuild it (instead of first taking the disk off the array first). Groggily, I checked the "reformat disk" option while unRAID was conducting a parity check after the reboot.

 

After a minute, I felt something wasn't right. I went to take a look and it, and it showed that the 14TB drive now is all free space. unRAID total disk utilization went from 83% full to 69% full in my array, meaning I lost data. As well, that disk after formatting read it's 99% free space. When it was emulated before formatting, it read 3% free space. I panicked, stopped the parity check and immediately took the array offline. Is there anything I can do now to recover the 14TB I stupidly formatted? Any help would be great. Or is there anything I can do to rebuild from parity? I'd try that, but it it looks like somehow the format also wrote into parity. I've attached diagnostics just in case.

 

My earlier thinking was that since the array was running, if I clicked format, it wouldn't have wiped 14TB from my array, but instead kept the contents emulated while formatting the unreadable drive. Would unRAID let me wipe 14TB of existing data while the array is running? I'm afraid to restart the array since I might overwrite even more data now.

 

This is really disheartening and a lesson learned. There's around 10yrs plus of family photos that I might have lost in my stupidity.

 

edit: forgot the mention that the formatted drive was an encrypted BTRFS format. I have the passkey, but not to sure how to recover the files if it comes to that.

hk-homelab-diagnostics-20191103-0409.zip

Edited by ipreferpie
Link to comment

A format with the disk in the array will always change parity (there's no "somehow" about it, that's how it should work).

So you have lost the data (the formatted disk) as well as the mean to recover it (parity is updated).

 

Now since you stopped it quickly, there is still hope of recovering some data using consumer-grade data recovery software.

However, you have jumped on the encryption bandwagon, which is beyond the capability of all consumer-grade software.

 

That means you will have to opt for commercial-grade solutions which are rather expensive.

For example, UFS Explorer (what johnnie said) "Pro" license costs €600.

And it's not at all a guarantee that you will be able to recover any data.

 

Professional data recovery service gives you the best chance of getting any thing back. However, they tend to cost per GB and I'm certain there will be significant surcharge for encrypted storage.

 

Sorry for being the grim reaper but I think it's more important you know the full picture.

  • Thanks 1
Link to comment

Thanks for all the responses so far. I was really wishing that I can find a solution to rebuild from parity, but it really looks like that as soon as I formatted while the array was online, it wiped the data off parity simultaneously. I've downloaded UFS Pro and it'll take 38hrs to scan of lost files. I really hope that I can manage to find all the ones I lost -- EUR600 is a lot, but for family photos, I'll have to bite the bullet and consider it as a very expensive lesson.

Link to comment
6 minutes ago, ipreferpie said:

but it really looks like that as soon as I formatted while the array was online, it wiped the data off parity simultaneously.

It's difficult for me to understand how this keeps happening, what would make the current warning clearer for you?

 

2145720115_Formatwarningnew.png.4c2c1ebd83c7cf93b70792d4f68958f6.png

 

8 minutes ago, ipreferpie said:

but for family photos,

You should really have a backup of any irreplaceable data, Unraid parity is for redundancy, it's not a backup.

 

 

 

Link to comment

Thanks everyone for the help so far. So the worst case scenario is real in that the format was reflected in parity. I guess there's absolutely no way to roll back the parity changes despite my taking it offline after the format. I've downloaded UFS Explorer Pro and did a lost file scan, but it all came back as wrong files (ZIPs, RARs, Apple icons) while they mostly should've been photo and video files. Is there something wrong that I'm not inputting correctly (LUKs decrypt or file name type) since this was a BTRFS encrypted format that was reformatted to BTRFS encrypted again. Many thanks -- I just trying to figure out options carefully before I do anything else.

 

Decryption and scan elements were aes256-xts-plain64 and UTF-8 Unicode

Edited by ipreferpie
Link to comment

You could try sending it off to a data recovery facility, but the results are probably going to be one of two things. Nothing recoverable, or partial recovery with a bill that has 3 zeros before the decimal. I'm not aware of any magic bullets at this stage. The addition of encryption makes a difficult job almost impossible at this stage of the game.

 

My advice? Put the drive in an anti static bag, along with a USB drive containing all the info you can think of that might facilitate recovery. On the USB drive I'd include a copy of your current unraid USB stick, encryption passphrase(s) or files, and a text copy of all the notes so far on what you've tried and how you tried it. Then, stick that anti static bag in a safe place, and wait a year or two. Either the contents will become less valuable to you as you move on, or technology and your situation changes and you can take another crack at it or send it off.

 

Or, just make the choice to move on and do things better for the future, mentally writing it off as a life lesson on technology priorities.

Link to comment

Oh, an additional thought before you throw in the towel on UFS Explorer. You need a success with your scan method before you can declare a failure on the recovery. What I mean by that is this. You need to set up a drive without file system damage that has exactly the same format and encryption, and run that through the UFS Explorer process and be able to see all the current files on the undamaged file system. That way you know the software is capable of reading that specific combination of encryption and format properly.

  • Thanks 1
Link to comment
1 hour ago, jonathanm said:

Oh, an additional thought before you throw in the towel on UFS Explorer. You need a success with your scan method before you can declare a failure on the recovery. What I mean by that is this. You need to set up a drive without file system damage that has exactly the same format and encryption, and run that through the UFS Explorer process and be able to see all the current files on the undamaged file system. That way you know the software is capable of reading that specific combination of encryption and format properly.

Much appreciated on the advice. I'll go ahead and do this last bit and see if anything works, but am mentally prepared for a full loss.

 

And to be doubly sure (which is a stupid question) -- there's no way to roll back the changes in parity after it was reflected in the format followed by an immediate shutdown, right? I haven't turned on my array yet since I'm scared of further changes in parity. But if I can't explore this option, I might as well just turn it on and see what important data I truly lost. Not knowing what is really killing me.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.