[SOLVED] Power loss, unmountable corrupt FS. How boned am I?

[DrKrieger]

I was dumb and forgot to disable power on after AC loss in BIOS after an update. There were storms in the area dropping power often. Server tried to power itself back on a few times before another loss of power killed the UPS and I didn't find out until I got home.

Thankfully no server hardware was damaged but I was staring at the array with all drives but cache showing "unsupported partition layout" after powering back on, I did verify SMART info and the drives seem fine and sound fine as well, but all are inaccessible. controller is good - plugged an old drive in to each cable and unraid saw it no problem. The ultimate "screw me" was that I planned to make a separate backup this weekend when more drives arrive

 

I've been up all night digging through these forums and other sites and hoping I am on the right track and I did this on a drive with very little data as a test to hopefully avoid too much damage:

 

- Start array in maintenance mode

- Run FS check on the drive - says the primary superblock has problems and I should try to mount it (doesn't work) and run the -L option. Same error as on this link which I tried to use the whole video as a starting point https://youtu.be/JVjB0ROdLBA?t=570 though it sees file/folder names it says the index will be removed or something like that in my logs

- run with -L a whole lot of files ended up in lost+found, some of which are super generic common names (readme, directx files sort of things) so I'll never know where they belong

- after the fs check finished add the drive to the array

- looks like the drive rebuilds on itself?

- once complete unraid reports drive is back to normal and I can indeed browse it and the disk usage is the same as before, but as mentioned most are in lost+found

 

At this point I admittedly panicked and shut the server down. I am however copying the data from the experiment above on my desktop (Win 11). Surprisingly the cache drive is fine so dockers, game servers, plex/emby metadata and so on are still there, just the actual data drives are problems.

 

Shutdown aside, am I doing it right?

 

I'm assuming I stand little chance of recovering everything to the original folders? With 40TB of data lost+found sorting feels like an impossible task.

 

Are there other tools I can try to recover data with by pulling the drives one by one on a separate system? Difficulty: I prefer Windows and I'm not super good with Linux and the like, but I'm trying to learn.

Edited October 30, 2021 by DrKrieger

[itimpi]

I would say at this point your best bet is something like UFS Explorer on Windows.

[DrKrieger]

On 9/28/2021 at 7:53 AM, itimpi said:

I would say at this point your best bet is something like UFS Explorer on Windows.

Thanks for the suggestion!

 

The fun part now will be scanning each drive one by one hoping it wasn't one of the blank ones because it's going to be 22 hours x 15 drives and then the actual recovery/copy time.

 

Looks like this is going to be a month long project all for 3 stupid mistakes.

A big lesson learned for sure as I stare at the new drives that I would've done backups on 😭

 

I did check another drive and about 20% of it ends up in lost+found. Going through all the music is going to be a nightmare!

[DrKrieger]

Final update:

 

Recovery of everything salvageable complete, I've lost around 3 out of 70TB of data, thankfully only a few shows I can make copies of again and some old game ISO's I've long since purchased digitally. Not too bad for my amateur moves at the start of this.

 

Big thanks to itimpi, UFS explorer helped a ton with some of the drives.

Another fantastic app I found was "Magic Raid Recovery" - it found lost partitions / filesystems on drives I couldn't recover through Unraid itself or any of the other dozen or so apps attempted. I was even desperate enough to try a professional recovery service, but there was no way I was going to attempt sorting through hundreds of thousands of files they output with super generic names like "12345.abc".

Edited October 30, 2021 by DrKrieger
spacing/spelling