No Mountable Filesystem

[Auggie]

Apparently one of my SAS cables went bad after I physically moved the server’s site location.  I determined it was a bad cable by swapping cables to different ports then different backplanes and the issue followed the cable.

 

Unfortunately, unRAID did not detect any issues until I started a parity check with auto error correction and suddenly got thousands of errors on two affected drives, both resulting in No Mountable Filesystem errors.

 

I was able to correct one of them but xfs_repair failed on the other so presently I have no way of seeing any of its contents not even in emulated mode.  I’ve tried both the webGUI and terminal sessions to fix the drive to no avail.

 

So my question: should I just reformat the drive and rebuild the data directly over it or is there a possible way to somehow recover data on it?

 

If there is a reasonable way, then I’ll pull it and get a brand new drive to rebuild the data and use the old drive to compare or recover any corrupted data.  Otherwise, I won’t spend the extra coin and just rebuild over it.

Edited October 9, 2018 by Auggie

[JorgeB]

If parity is in sync rebuilding won't fix the file system, and you can see the end result of rebuilding by running xfs_repair on the emulated disk, if it fail there it will also fail on the rebuilt disk.

 

You should also post your diagnostics and the full output of xfs_repair when it fails.

[Auggie]

I ran the xfs_repair again via terminal and attached text output is what I got.

 

And this is my new syslog from startup after getting home today (didn't save the original log when the errors occurred as I felt at the time it would be all related to the bad cable and not a failure of the drive hardware-wise):

 

Oct  9 17:50:24 UnRAID kernel: XFS (md10): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x2 len 1 error 117
Oct  9 17:50:24 UnRAID kernel: XFS (md10): xfs_imap_lookup: xfs_ialloc_read_agi() returned error -117, agno 0
Oct  9 17:50:24 UnRAID root: mount: /mnt/disk10: wrong fs type, bad option, bad superblock on /dev/md10, missing codepage or helper program, or other error.
Oct  9 17:50:24 UnRAID emhttpd: /mnt/disk10 mount error: No file system

 

I figured that if the emulated drive also showed no mountable file system then the data is essentially lost unless I engage a data recovery service.

 

It's all media files and I could repopulate; a time consuming but not insurmountable task...

 

Terminal Saved Output

[JorgeB]

Yep, that looks like a very corrupt filesystem and likely beyond repair.

[JorgeB]

Also worth mentioning, you seem to be sure the problem was a cable but if you're sig is correct and you're using two SAS2LP, they are known to sometimes drop disks and that can cause filesystem corruption, can't say they were the problem, especially without the diagnostics from that time, but keep that in mind if it happens again and the affected disks are connected to one of them.

[Auggie]

4 hours ago, johnnie.black said:

Also worth mentioning, you seem to be sure the problem was a cable but if you're sig is correct and you're using two SAS2LP, they are known to sometimes drop disks and that can cause filesystem corruption, can't say they were the problem, especially without the diagnostics from that time, but keep that in mind if it happens again and the affected disks are connected to one of them.

 

Yea, I've always been wary of the SAS2LP's as there have been unexplained "drops" that eventually get cleared up by reseating or restarting.  Hence my very thorough swapping of SAS cables through multiple changes to both see if it magically corrects itself or if a dropped disk follows the cable.  In this case, it was a cable.  The problem is that the corruption was already done from the parity check with auto-correction.

 

Anywho, I'm switching to a Chenbro chassis and new mobo which has two 12GBs SAS ports builtin so I will be doing away with the SAS2LPs; the current mobo will be swapped into my backup server, which already has one LSI and with the addtion of the IBM and the expander card, I can finally rid myself of the problematic SAS2LP's entirely from my life.