Unmountable No File System Error

[Fatal_Flaw]

I'm getting the error "Unmountable: No file system" for one of my drives. 

 

Syslog says

Nov  3 18:32:34 Filebox emhttpd: shcmd (86): mkdir -p /mnt/disk9
Nov  3 18:32:34 Filebox emhttpd: shcmd (87): mount -t xfs -o noatime,nodiratime /dev/md9 /mnt/disk9
Nov  3 18:32:34 Filebox kernel: XFS (md9): Mounting V5 Filesystem
Nov  3 18:32:34 Filebox kernel: XFS (md9): Corruption warning: Metadata has LSN (1:63423) ahead of current LSN (1:63186). Please unmount and run xfs_repair (>= v4.3) to resolve.
Nov  3 18:32:34 Filebox kernel: XFS (md9): log mount/recovery failed: error -22
Nov  3 18:32:34 Filebox kernel: XFS (md9): log mount failed
Nov  3 18:32:34 Filebox root: mount: /mnt/disk9: wrong fs type, bad option, bad superblock on /dev/md9, missing codepage or helper program, or other error.
Nov  3 18:32:34 Filebox emhttpd: shcmd (87): exit status: 32
Nov  3 18:32:34 Filebox emhttpd: /mnt/disk9 mount error: No file system
Nov  3 18:32:34 Filebox emhttpd: shcmd (88): umount /mnt/disk9
Nov  3 18:32:34 Filebox root: umount: /mnt/disk9: not mounted.
Nov  3 18:32:34 Filebox emhttpd: shcmd (88): exit status: 32
Nov  3 18:32:34 Filebox emhttpd: shcmd (89): rmdir /mnt/disk9

Line 4 in paste above says to run xfs_repair. Should I, or is there something else I should be trying first?

 

Diagnostics file attached.

 

Thanks

filebox-diagnostics-20181103-1844.zip

[JorgeB]

Just now, Fatal_Flaw said:

says to run xfs_repair. Should I

Yes, but the disk is disabled, you'll also need to fix that by rebuilding it after, don't do it on top of the old disk if xfs_repair doesn't fix the filesystem, and even if it does rebuilding to a new disk is safer, though SMART looks fine.

 

You're using a Marvell controller, they are known to drop disks, and not surprisingly the disabled disk is using it.

 

 

[Fatal_Flaw]

On 11/3/2018 at 6:54 PM, johnnie.black said:

Yes, but the disk is disabled, you'll also need to fix that by rebuilding it after, don't do it on top of the old disk if xfs_repair doesn't fix the filesystem, and even if it does rebuilding to a new disk is safer, though SMART looks fine.

 

Thanks for the reply, but I'm confused. I thought rebuilding the drive overwrote everything on it bit for bit regardless of the existing data or file system on it? Why would it matter if I do it before or after xfs_repair? And if I'm going to rebuild the drive, why would I run xfs_repair at all instead of just reformatting it?

 

Quote

You're using a Marvell controller, they are known to drop disks, and not surprisingly the disabled disk is using it.

These sata add-on cars have been a huge pain for years now. They do drop disks a lot when there's heavy read/write activity on them. Last time I looked at the list of "recommended" controllers, there weren't good options given my setup and requirements. That was quite a while ago now though, so I'll have to look again and see what's changed.

[itimpi]

22 minutes ago, Fatal_Flaw said:

 

Thanks for the reply, but I'm confused. I thought rebuilding the drive overwrote everything on it bit for bit regardless of the existing data or file system on it? Why would it matter if I do it before or after xfs_repair? And if I'm going to rebuild the drive, why would I run xfs_repair at all instead of just reformatting it?

If you run xfs_repair before rebuilding then you are running against the emulated disk.    If you subsequently rebuild you will get the contents of the (repaired) emulated disk written to the physical disk.    This precludes the option to run xfs_repair against the physical disk (in the state it was disabled) which might give better results if the emulated disk repair did not go well.

[trurl]

1 hour ago, Fatal_Flaw said:

instead of just reformatting it?

 

And you definitely don't want to format if you want to keep its data.

 

The thought that you might reformat the disk is due to a common misconception many people have about the meaning of format. I just commented on this in another thread today. See here:

 

https://forums.unraid.net/topic/75417-data-missing-after-replacing-failed-drive-and-adding-second-parity-drive-at-same-time/?tab=comments#comment-694435

 

[Fatal_Flaw]

5 hours ago, itimpi said:

If you run xfs_repair before rebuilding then you are running against the emulated disk.    If you subsequently rebuild you will get the contents of the (repaired) emulated disk written to the physical disk.    This precludes the option to run xfs_repair against the physical disk (in the state it was disabled) which might give better results if the emulated disk repair did not go well.

Thanks, that clarified it for me. It didn't occur to me that xfs_repair would be run against the emulated disk rather than the physical disk.

 

4 hours ago, trurl said:

 

And you definitely don't want to format if you want to keep its data.

 

The thought that you might reformat the disk is due to a common misconception many people have about the meaning of format. I just commented on this in another thread today. See here:

 

https://forums.unraid.net/topic/75417-data-missing-after-replacing-failed-drive-and-adding-second-parity-drive-at-same-time/?tab=comments#comment-694435

 

Nah, I know what formatting is. My point was that the disk was disabled with the unmountable error due to file system corruption, but the emulated disk was accessible and the data seemed to be intact. So presumably re-assigning that disabled disk as a new one in the config and rebuilding it from parity would write a useable filesystem to the disk.

 

SOLUTION: What I ended up doing

1. Because the drive wasn't being written to when the controller dropped it, I decided it was likely that the data on the physical drive was ok except for the file system corruption. I also already had everything on that drive backed up to CrashPlan as a last resort.
2. I reset the config and told it to trust the parity.
3. Started the array in maintenance mode.
4. I ran xfs_config on the previously disabled disk. It took less than 30 seconds to run and even the verbose output didn't really indicate any issues were found.
5. Restarted the array fully.
6. Drive mounted and everything on the drive is intact (I keep hashes of all the files for verification)
7. Running parity check just in case something got out of sync.

I also just ordered a LSI SAS 9207-8i to replace the absolutely awful Marvel based SYBA controller I'm currently using. This is not the first time it's dropped a disk randomly.

 

Thanks for the help everyone!

Edited November 6, 2018 by Fatal_Flaw
forgot a few words

[trurl]

11 hours ago, Fatal_Flaw said:

Nah, I know what formatting is.

But you said

17 hours ago, Fatal_Flaw said:

why would I run xfs_repair at all instead of just reformatting it?

so I wanted to make sure you didn't format a disk in the array. Even if you know what formatting is, it might not be obvious to you that parity would be updated by the format and so a rebuild from that point would just rebuild the empty filesystem the format wrote.