Unmountable file system during data rebuild [SOLVED]


Recommended Posts

I had a drive disabled due to read errors, which I understand is usually a cable issue.  I stopped the array, shutdown, changed out the cable, which is CableCreation [2 Pack] Mini SAS 36Pin (SFF-8087) Male to 4 SATA 7Pin Female Cable, Mini SAS Host/Controller to 4 SATA Target/Backplane, 1M / 3.3FT, replaced it with another of the same, and replaced the drive, because I had a drive precleared and ready to go.

 

On booting up, I selected the "new" drive in slot 3, and started the array.  Then I noticed that the new drive has "Unmountable: No file system" listed.  In the past, I have clicked on format, and lost data, which is why I am writing this, so I don't make the wrong guess and screw things up.  I think the data rebuild and unmountable file system are kind of 2 separate issues (probably same root cause), and the rebuild can continue, as in it is actually writing data to the drive, but that the fs on that drive will not work once done, because it doesn't work with the emulated contents currently.  I think I want to let it finish, then stop the array, start in maintenance mode, and check the fs?

flags-diagnostics-20200923-0809.zip

Edited by bobobeastie
solved
Link to comment
22 minutes ago, bobobeastie said:

I think I want to let it finish, then stop the array, start in maintenance mode, and check the fs?

That would work.   Whatever is showing before you start a rebuild is what will show after the rebuild.

 

if you would prefer then it is perfectly acceptable to run the file system check/repair on the emulated drive before doing a rebuild.   If that does not fix the ‘unmountable’ file system then the rebuild will not help as the rebuild simply transfers what is on the emulated drive to the physical drive.

Link to comment
5 minutes ago, itimpi said:

That would work.   Whatever is showing before you start a rebuild is what will show after the rebuild.

 

if you would prefer then it is perfectly acceptable to run the file system check/repair on the emulated drive before doing a rebuild.   If that does not fix the ‘unmountable’ file system then the rebuild will not help as the rebuild simply transfers what is on the emulated drive to the physical drive.

Thanks, I don't think there was any mention of an issue with the fs before shutting down. I have array status and alerts email notifications enabled and don't have an email about the fs, not sure if I would expect to have seen one.  If there is a chance the rebuild will fix everything, or that after the rebuild, fixing the fs will solve everything, ie no 2nd rebuild is needed, then I think letting it run makes sense.  If I would need to do a 2nd rebuild then I tihnk stopping, fixing the fs, then doing a rebuild makes more sense.  Also I guess my preference is safer vs faster. 

Link to comment

And as you have exchanged the failed drive, do not format or discard the old drive before your array is up and running again.

Just in case anything goes south from here, you still have a copy of the data on that old disk! With a file system corruption of some kind, but still a viable source for data recovery just in case.

Link to comment
2 hours ago, bobobeastie said:

Thanks, I don't think there was any mention of an issue with the fs before shutting down. I have array status and alerts email notifications enabled and don't have an email about the fs, not sure if I would expect to have seen one.  If there is a chance the rebuild will fix everything, or that after the rebuild, fixing the fs will solve everything, ie no 2nd rebuild is needed, then I think letting it run makes sense.  If I would need to do a 2nd rebuild then I tihnk stopping, fixing the fs, then doing a rebuild makes more sense.  Also I guess my preference is safer vs faster. 

At this point I would let the rebuild finish and then attempt the fs repair.

 

the point I was making is that (if this happens again) you can first try the repair on the emulated disk before starting the rebuild.   If this fails then the result of the rebuild will fail as well so no point in even attempting it, but if it succeeds you can switch back to normal mode and then rebuild knowing that what is now showing on the emulated drive is what will end up on the physical disk after the rebuild.

Link to comment
2 hours ago, Kevek79 said:

And as you have exchanged the failed drive, do not format or discard the old drive before your array is up and running again.

Just in case anything goes south from here, you still have a copy of the data on that old disk! With a file system corruption of some kind, but still a viable source for data recovery just in case.

Thanks, that's my plan.  After I get things working and I don't need the backup, I'm going to pleclear the old drive 3 times and if it passes replace the smallest drive in my array.  I really wish I had money and that the price of NORCO RPC-4224's hadn't nearly doubled, or I'd buy a 2nd one.  The fist one I have has been relatively rock solid.

Link to comment

It's back up and running.  The  first xfs_repair resulted in this message:

Quote

Phase 1 - find and verify superblock...
bad primary superblock - bad CRC in superblock !!!

attempting to find secondary superblock...
.found candidate secondary superblock...
verified secondary superblock...
writing modified primary superblock
Phase 2 - using internal log
        - zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.  If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

So I stopped the array, and started it not in maintenance mode, still showed unmountable, I stopped the array, and started it in maintenance mode, ran xfs_repair, after that I could start the array and the issue went away.  The resulting trash folder is basically empty, and all I had were torrents on the disk, so looks like I was lucky.  Thank you @itimpi and @Kevek79 .

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.