Unmountable: No file system (x2)

[FRX]

One of my drives (disk 5) was failing so I replaced it. When I started the server again I got a red X on disk 2. Even after doublechecking the cables it still said the same. Stupidly enough I unselected it and then reselected it on the disk selection dropdown, figuring it might help. Now I have two disks saying "Unmountable: No file system".

 

I tried "xfs_repair -v /dev/md2" but got "Sorry, could not find valid secondary superblock" after a couple of hours.

 

What can I do? Is my data beyond saving?

 

file-server-diagnostics-20220925-0814.zip

[JorgeB]

Disk2 looks healthy so it be easy to re-enable, is the old disk5 completely dead? If not connect it and post new diags.

[FRX]

Thanks JorgeB!

 

I switched back disk5, but cant start the array because it says "Too many wrong and/or missing disks!". Attached diagnostics anyway, if thats any help!

 

file-server-diagnostics-20220925-1141.zip

Edited September 25, 2022 by FRX

[JorgeB]

Old disk5 appears to have really failed, but keep it intact for now, we can try re-enabling disk2 to rebuild disk5 again, connet new disk5 again, then:

 

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed, including old disk2 and new disk5, replacement disk should be same size or larger than the old one
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign disk5
-Start array (in normal mode now), ideally the emulated disk will now mount and contents look correct, then post new diags

 

 

 

 

 

[FRX]

Thanks for the instructions, and for helping out!

 

I did as you said above, but disk2 still says "Unmountable: No file system".

 

Posting diags!

file-server-diagnostics-20220926-2012.zip

[JorgeB]

Check filesystem on disks 2 and 5, note that you must remove -n or nothing will be done, and if it asks for -L use it.

[FRX]

Same error on disk2 again, "Sorry, could not find valid secondary superblock".

[JorgeB]

Please update to latest Unraid since it includes much newer xfs progrs and try running xfs_repair again on disk2.

[FRX]

I updated Unraid now, should I start in maintenance mode when Im running xfs_repair?

[trurl]

Just now, FRX said:

start in maintenance mode when Im running xfs_repair

No other way to do it.

 

Your question does make me wonder though. Have you been doing check filesystem from the webUI or from the command line? Easy to get the command wrong, webUI will do it right.

[FRX]

When I try to do it in the webUI I cant see "Check Filesystem Status" on drive2, its on the other drives though

[JorgeB]

Click on disk2, set the fileystem to xfs and try again.

[FRX]

Same thing again, "Sorry, could not find valid secondary superblock"

[JorgeB]

And you are using the GUI to check filesystem? Also post new diags. 

[FRX]

Yeah Im using the GUI now

file-server-diagnostics-20220929-1838.zip

[JorgeB]

Diags after array start in normal mode please.

[FRX]

Here it comes! And thanks again for helping out!

file-server-diagnostics-20220929-2001.zip

[JorgeB]

Start the array in maintenance mode and type in the console:

xfs_repair -v /dev/md2

Then post the full output from that.

[FRX]

This is what I got (but with thousands of periods)

 

Phase 1 - find and verify superblock...
couldn't verify primary superblock - not enough secondary superblocks with matching geometry !!!

attempting to find secondary superblock...

..........................................................................................................................................................................................................................................................................................................................................................................................................

Sorry, could not find valid secondary superblock
Exiting now.

[JorgeB]

This is very strange, clearly there's a xfs filesystem on disk2:

 

ep 29 20:00:48 FILE-SERVER kernel: XFS (md2): Mounting V5 Filesystem
Sep 29 20:00:48 FILE-SERVER root: mount: /mnt/disk2: mount(2) system call failed: Structure needs cleaning.
Sep 29 20:00:48 FILE-SERVER root:        dmesg(1) may have more information after failed mount system call.
Sep 29 20:00:48 FILE-SERVER kernel: XFS (md2): Log inconsistent (didn't find previous header)
Sep 29 20:00:48 FILE-SERVER kernel: XFS (md2): failed to find log head
Sep 29 20:00:48 FILE-SERVER kernel: XFS (md2): log mount/recovery failed: error -117
Sep 29 20:00:48 FILE-SERVER kernel: XFS (md2): log mount failed

 

But xfs_repair doesn't find main or a backup superblock, something weird is happening there, just one last shot in the dark, and to rule out any md driver issue, with the array stopped type in the console:

 

xfs_repair -v /dev/sdc1

 

 

[FRX]

Same thing again Im afraid.

 

Phase 1 - find and verify superblock...
couldn't verify primary superblock - not enough secondary superblocks with matching geometry !!!

attempting to find secondary superblock...
..................
..Sorry, could not find valid secondary superblock
Exiting now.

file-server-diagnostics-20221001-0812.zip

[JorgeB]

It's strange but there's nothing else I can do for that filesystem, do you have backups? If not you can try posting the problem on the xfs mailing list, it's not Unraid related since running xfs_repair on the actual disk (bypassing the Unraid md driver) still cannot fix it.

[FRX]

Thanks alot for trying!

 

No backup unfortunately.. Im thinking I should disconnect disk2 and see if I can recover the data some other way. Whats the best solution to limit the damage on unraid? Remove disk2 (and replace it when I get a new disk), format the new disk5 and rebuild?

[JorgeB]

Best to try and recover the data would be a xfs compatible file recovery util like UFS explorer.

[FRX]

Yeah I heard about UFS explorer so might give that a go. If I somehow recover the data from disk2 and copy it to unraid, will everything be back? Or will some things be forever lost since two drives failed at once?

 

Is it safe to remove disk2 and rebuild the array with one disk missing?