Recovering from unclean shutdown: xfs_repair worked but (full) drive now appears empty

[sysunmin]

Hi folks,

I'm trying to get back on my feet after an unclean shutdown with a failed drive. I'll try to recap the run of events as best as possible:

1. Server (12 data drives, 1 parity, 3 cache drives) shows disk error - it's done this before, usually CRC due to cabling - says it needs to be reconstructed. I turn it off and check cabling, power back on, and start the rebuild process. A minor annoyance, but no big deal.

2. Server loses power -- later determined to be a PSU that failed. Big time bummer. Took a long time to figure out - during which the server booted up and tried to restore from parity but started getting disc read errors left and right, so I terminated the parity operation. Figured out the issue, replaced PSU, parity completed. 

3. Drive (9) successfully imaged but said "unmountable - no filesystem" (or similar). I checked the filesystem in maintenance mode using xfs_repair -n on /dev/md9 and it found errors. Tried to repair using xfs_repair -v and it said something about a bad super block and froze up for the better part of a day. Perhaps stupidly, I exited with Crtl+C, having seen that this could take a few hours - not multiple days. I ran xfs_repair -v again and the disc now shows up as part of the array. 

4. To my horror, the disc showed up fine on a reboot - BUT, instead of having ~7.9TB full, it now had ~7.9TB FREE. I have checked /dev/sdh1 and /dev/md9 with ls; no files appear. So, it seems I somehow nuked the file partition table. 

 

I have since put the server in maintenance mode to tinker and, after removing the disc, so that I'm working only on an emulated copy, tried xfs_repair -v -L which despite multiple reboots has not changed anything. If I switch briefly to "on-line" mode for the array, it still shows as free space. 

 

The good news (?) is that I can use UFS explorer to pull the contents off the drive, so the data appear to be largely in-tact. But, UFS explorer can't get filenames back - and with the number and diversity of files on the drive, I'd like to salvage whatever structure I can. 

 

So, please, UnRAID gurus - what should I try next to restore the file table to /dev/md9? In the meantime, I'm off to buy another drive to copy files to in case they can be recovered. Thanks much! 

 

(Log incoming as soon as I figure out how to post it - and sadly, it's after a reboot because I didn't expect the files to disappear after the xfs_repair) 

 

 

fileserver-diagnostics-20210406-1213.zip

Edited April 6, 2021 by sysunmin
Adding log