So had my system freeze up the other day, and when I rebooted to regain access, I got a "Unmountable: No file system" warning on one of my array drives.

 

Tried removing the drive from the array and restarting the array to shift it into "emulated" mode, which got the array up and running again, but was still missing some files through the emulation (I've never seen that before with a failed drive, always emulated just fine - very strange).  I tried adding the drive back in and let it rebuild with parity.  Seemed to have completed the rebuild OK, but still shows "no file system".

 

Currently running xfs_repair on it in maintenance mode to see if it can recover anything, but shows missing superblock, so now it's scanning the entire drive to find a backup.

 

Diagnostics attached.  No SMART errors logged, so didn't pull a smart report.  Hoping someone may have some ideas to try and get things up and running again without wiping the drive and starting over. I do have a backup, but it'll be labour and time intensive to put things back, so recovering the filesystem would be awesome if possible.

 

Edited October 10, 20187 yr by dgriff
nuked diagnostic data, just for safety.

4 minutes ago, dgriff said:

Currently running xfs_repair on it in maintenance mode to see if it can recover anything, but shows missing superblock, so now it's scanning the entire drive to find a backup.

You're likely using the wrong command, see:

https://wiki.unraid.net/Check_Disk_Filesystems#Drives_formatted_with_XFS

or

https://wiki.unraid.net/Check_Disk_Filesystems#Checking_and_fixing_drives_in_the_webGui

 

5 minutes ago, dgriff said:

Tried removing the drive from the array and restarting the array to shift it into "emulated" mode, which got the array up and running again, but was still missing some files through the emulation (I've never seen that before with a failed drive, always emulated just fine - very strange).  I tried adding the drive back in and let it rebuild with parity.  Seemed to have completed the rebuild OK, but still shows "no file system".

Also, this doesn't make much sense, the rebuilt disk should be exactly the same as the emulated disk.

Thank you sir, indeed I was performing xfs_repair on the raw device instead of the md device.  Learn something new every day.

 

Later edit for clarity: As I understand it now, unRAID takes the standard Linux /dev/sdxx drives of whatever shape and size, and bundles them together into a single logical device using a customized version of the MD software.  If a hard error takes a single drive down, the system uses the parity drive(s) to emulate that individual disk until it's replaced.

 

If (in my case) however, a soft error takes out the filesystem, even though it shows 1 drive as faulting, it's actually the filesystem across the entire MD array of devices that's "damaged" and not reading properly, hence why some of my files were missing on the emulated "bad" drive.  Using xfs_repair on the /dev/sdxx drive doesn't work because it's only seeing one small and incomplete section of the overall logical drive unit.  When you run it on the /dev/md device, it's actually processing the entire filesystem across all devices and making repairs to the total unRAID array (in which case I'm not sure why you need to target a specific /md# drive as opposed to just the entire thing, guess that's just how it works).

 

Quote

Also, this doesn't make much sense, the rebuilt disk should be exactly the same as the emulated disk.

Edit: Yup, I had a bad cold when all this went down, and obviously wasn't in the right frame of mind to be problem solving, I was thinking it was a hard error on the drive instead of a soft error on the filesystem.

 

Now I'm seeing:

ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.  If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

 

Not sure exactly how to mount the filesystem manually to try and replay the log.  I can probably destroy and repair without it, but any options here that might work?

Edited October 10, 20187 yr by dgriff
clarity incase others have the problem.

Running with the -L option added is the normal way forward.   Although in theory it could mean that the last change made is lost in practice this is rarely the case.

Awesome, fixed it!  Thanks again all! Marked as solved!

Edited October 9, 20187 yr by dgriff