Jump to content

Recovering from unclean shutdown: xfs_repair worked but (full) drive now appears empty

Recommended Posts

Hi folks,

I'm trying to get back on my feet after an unclean shutdown with a failed drive. I'll try to recap the run of events as best as possible:

1. Server (12 data drives, 1 parity, 3 cache drives) shows disk error - it's done this before, usually CRC due to cabling - says it needs to be reconstructed. I turn it off and check cabling, power back on, and start the rebuild process. A minor annoyance, but no big deal.

2. Server loses power -- later determined to be a PSU that failed. Big time bummer. Took a long time to figure out - during which the server booted up and tried to restore from parity but started getting disc read errors left and right, so I terminated the parity operation. Figured out the issue, replaced PSU, parity completed. 

3. Drive (9) successfully imaged but said "unmountable - no filesystem" (or similar). I checked the filesystem in maintenance mode using xfs_repair -n on /dev/md9 and it found errors. Tried to repair using xfs_repair -v and it said something about a bad super block and froze up for the better part of a day. Perhaps stupidly, I exited with Crtl+C, having seen that this could take a few hours - not multiple days. I ran xfs_repair -v again and the disc now shows up as part of the array. 

4. To my horror, the disc showed up fine on a reboot - BUT, instead of having ~7.9TB full, it now had ~7.9TB FREE. I have checked /dev/sdh1 and /dev/md9 with ls; no files appear. So, it seems I somehow nuked the file partition table. 


I have since put the server in maintenance mode to tinker and, after removing the disc, so that I'm working only on an emulated copy, tried xfs_repair -v -L which despite multiple reboots has not changed anything. If I switch briefly to "on-line" mode for the array, it still shows as free space. 


The good news (?) is that I can use UFS explorer to pull the contents off the drive, so the data appear to be largely in-tact. But, UFS explorer can't get filenames back - and with the number and diversity of files on the drive, I'd like to salvage whatever structure I can. 


So, please, UnRAID gurus - what should I try next to restore the file table to /dev/md9? In the meantime, I'm off to buy another drive to copy files to in case they can be recovered. Thanks much! 


(Log incoming as soon as I figure out how to post it - and sadly, it's after a reboot because I didn't expect the files to disappear after the xfs_repair) 




Edited by sysunmin
Adding log
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Create New...