ipreferpie Posted November 3, 2019 Share Posted November 3, 2019 (edited) Hi there, I did something really stupid. My unRAID system froze so I rebooted it. Upon restarting the array, I saw that on one of my 14TB disks, it showed unreadable file system and contents were emulated. Because I just flew back from a work trip and severely jetlagged, I did a cursory lookup of how to fix it but didn't follow instructions properly. Basically, I thought I was going to format the disk and unRAID will automatically rebuild it (instead of first taking the disk off the array first). Groggily, I checked the "reformat disk" option while unRAID was conducting a parity check after the reboot. After a minute, I felt something wasn't right. I went to take a look and it, and it showed that the 14TB drive now is all free space. unRAID total disk utilization went from 83% full to 69% full in my array, meaning I lost data. As well, that disk after formatting read it's 99% free space. When it was emulated before formatting, it read 3% free space. I panicked, stopped the parity check and immediately took the array offline. Is there anything I can do now to recover the 14TB I stupidly formatted? Any help would be great. Or is there anything I can do to rebuild from parity? I'd try that, but it it looks like somehow the format also wrote into parity. I've attached diagnostics just in case. My earlier thinking was that since the array was running, if I clicked format, it wouldn't have wiped 14TB from my array, but instead kept the contents emulated while formatting the unreadable drive. Would unRAID let me wipe 14TB of existing data while the array is running? I'm afraid to restart the array since I might overwrite even more data now. This is really disheartening and a lesson learned. There's around 10yrs plus of family photos that I might have lost in my stupidity. edit: forgot the mention that the formatted drive was an encrypted BTRFS format. I have the passkey, but not to sure how to recover the files if it comes to that. hk-homelab-diagnostics-20191103-0409.zip Edited November 3, 2019 by ipreferpie Quote Link to comment
JorgeB Posted November 3, 2019 Share Posted November 3, 2019 4 hours ago, ipreferpie said: Or is there anything I can do to rebuild from parity? No, formatting a disk updates parity, only option is to run a deleted file recovery utility like UFS explorer. Quote Link to comment
testdasi Posted November 3, 2019 Share Posted November 3, 2019 A format with the disk in the array will always change parity (there's no "somehow" about it, that's how it should work). So you have lost the data (the formatted disk) as well as the mean to recover it (parity is updated). Now since you stopped it quickly, there is still hope of recovering some data using consumer-grade data recovery software. However, you have jumped on the encryption bandwagon, which is beyond the capability of all consumer-grade software. That means you will have to opt for commercial-grade solutions which are rather expensive. For example, UFS Explorer (what johnnie said) "Pro" license costs €600. And it's not at all a guarantee that you will be able to recover any data. Professional data recovery service gives you the best chance of getting any thing back. However, they tend to cost per GB and I'm certain there will be significant surcharge for encrypted storage. Sorry for being the grim reaper but I think it's more important you know the full picture. 1 Quote Link to comment
JorgeB Posted November 3, 2019 Share Posted November 3, 2019 8 minutes ago, testdasi said: For example, UFS Explorer (what johnnie said) "Pro" license costs €600. You don't nee PRO, Standard license costs 50€, and other users have used successfully before, but obviously there are no guarantees, you can use the trial first. Quote Link to comment
testdasi Posted November 3, 2019 Share Posted November 3, 2019 14 minutes ago, johnnie.black said: You don't nee PRO, Standard license costs 50€, and other users have used successfully before, but obviously there are no guarantees, you can use the trial first. Their website said standard doesn't support encryption e.g. LUKS. Quote Link to comment
JorgeB Posted November 3, 2019 Share Posted November 3, 2019 10 minutes ago, testdasi said: Their website said standard doesn't support encryption e.g. LUKS. You're right, OP edited the post after my reply. Quote Link to comment
ipreferpie Posted November 3, 2019 Author Share Posted November 3, 2019 Thanks for all the responses so far. I was really wishing that I can find a solution to rebuild from parity, but it really looks like that as soon as I formatted while the array was online, it wiped the data off parity simultaneously. I've downloaded UFS Pro and it'll take 38hrs to scan of lost files. I really hope that I can manage to find all the ones I lost -- EUR600 is a lot, but for family photos, I'll have to bite the bullet and consider it as a very expensive lesson. Quote Link to comment
JorgeB Posted November 3, 2019 Share Posted November 3, 2019 6 minutes ago, ipreferpie said: but it really looks like that as soon as I formatted while the array was online, it wiped the data off parity simultaneously. It's difficult for me to understand how this keeps happening, what would make the current warning clearer for you? 8 minutes ago, ipreferpie said: but for family photos, You should really have a backup of any irreplaceable data, Unraid parity is for redundancy, it's not a backup. Quote Link to comment
itimpi Posted November 3, 2019 Share Posted November 3, 2019 9 minutes ago, ipreferpie said: but for family photos, I'll have to bite the bullet and consider it as a very expensive lesson. All important data should be backed up elsewhere. You should never trust a single system to never lose such data. Quote Link to comment
ipreferpie Posted November 9, 2019 Author Share Posted November 9, 2019 (edited) Thanks everyone for the help so far. So the worst case scenario is real in that the format was reflected in parity. I guess there's absolutely no way to roll back the parity changes despite my taking it offline after the format. I've downloaded UFS Explorer Pro and did a lost file scan, but it all came back as wrong files (ZIPs, RARs, Apple icons) while they mostly should've been photo and video files. Is there something wrong that I'm not inputting correctly (LUKs decrypt or file name type) since this was a BTRFS encrypted format that was reformatted to BTRFS encrypted again. Many thanks -- I just trying to figure out options carefully before I do anything else. Decryption and scan elements were aes256-xts-plain64 and UTF-8 Unicode Edited November 9, 2019 by ipreferpie Quote Link to comment
JonathanM Posted November 9, 2019 Share Posted November 9, 2019 You could try sending it off to a data recovery facility, but the results are probably going to be one of two things. Nothing recoverable, or partial recovery with a bill that has 3 zeros before the decimal. I'm not aware of any magic bullets at this stage. The addition of encryption makes a difficult job almost impossible at this stage of the game. My advice? Put the drive in an anti static bag, along with a USB drive containing all the info you can think of that might facilitate recovery. On the USB drive I'd include a copy of your current unraid USB stick, encryption passphrase(s) or files, and a text copy of all the notes so far on what you've tried and how you tried it. Then, stick that anti static bag in a safe place, and wait a year or two. Either the contents will become less valuable to you as you move on, or technology and your situation changes and you can take another crack at it or send it off. Or, just make the choice to move on and do things better for the future, mentally writing it off as a life lesson on technology priorities. Quote Link to comment
JonathanM Posted November 9, 2019 Share Posted November 9, 2019 Oh, an additional thought before you throw in the towel on UFS Explorer. You need a success with your scan method before you can declare a failure on the recovery. What I mean by that is this. You need to set up a drive without file system damage that has exactly the same format and encryption, and run that through the UFS Explorer process and be able to see all the current files on the undamaged file system. That way you know the software is capable of reading that specific combination of encryption and format properly. 1 Quote Link to comment
ipreferpie Posted November 9, 2019 Author Share Posted November 9, 2019 1 hour ago, jonathanm said: Oh, an additional thought before you throw in the towel on UFS Explorer. You need a success with your scan method before you can declare a failure on the recovery. What I mean by that is this. You need to set up a drive without file system damage that has exactly the same format and encryption, and run that through the UFS Explorer process and be able to see all the current files on the undamaged file system. That way you know the software is capable of reading that specific combination of encryption and format properly. Much appreciated on the advice. I'll go ahead and do this last bit and see if anything works, but am mentally prepared for a full loss. And to be doubly sure (which is a stupid question) -- there's no way to roll back the changes in parity after it was reflected in the format followed by an immediate shutdown, right? I haven't turned on my array yet since I'm scared of further changes in parity. But if I can't explore this option, I might as well just turn it on and see what important data I truly lost. Not knowing what is really killing me. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.