Unmountable: Not Mounted - Help with Rebuild Options

[stratigary]

Returned home today and noticed that my Unraid server had shut down for some reason. Upon rebooting I noticed that Disk 1 (20 days old) was listed as "unmountable: not mounted". When I ran the xfx_repair it said "bad primary superblock - bad primary number!!" and then searched but never found a secondary superblock.

 

Should I even bother to try and rebuild this drive and, if so, how do I go about this. When I choose the "format" option in the GUI it makes it sound like it will clear out the drive and then update the parity to reflect that cleared drive and I lose all the data.

 

Or should I just RMA the drive and rebuild on a new one?

[trurl]

Rebuild will not fix filesystem, and format will make it an empty filesystem and parity would agree it is empty.

1 hour ago, stratigary said:

ran the xfx_repair

How exactly did you run it?

 

Attach diagnostics to your NEXT post in this thread

[stratigary]

1 hour ago, trurl said:

Rebuild will not fix filesystem, and format will make it an empty filesystem and parity would agree it is empty.

How exactly did you run it?

 

Attach diagnostics to your NEXT post in this thread

I did what was suggested on this website.  I started in maintenance mode and then clicked on the disk on the main screen.  In the Check Filesystem Status I ran it as -n.  That is when it said there was a bad primary superblock and searched forever and never found a secondary superblock.

 

What is the best way to go about rebuilding this drive to recover the data?  I can get a hold of a new drive to put in its place tomorrow, would that be the best option to do and then just rebuild it from parity?

 

server-diagnostics-20220312-2323.zip

[JorgeB]

Did you run xfs_repair without -n? With -n nothing will be done. Rebuilding a disk won't help with filesystem corruption, if xfs_repair can't fix it there's not much else you can do, except restoring the data from backups if available.

[stratigary]

Yes, I did run xfs_repair without the -n and I get the same result, bad primary superblock and searches forever and never finds a secondary superblock just ........ forever.  I have some slightly out of date backups that I can work with, but it will be a pain.

 

I do have one spare drive on hand that's the same size as the corrupted one.  Are there any hail mary options I can try before I have to reformat?

[stratigary]

7 hours ago, JorgeB said:

Did you run xfs_repair without -n? With -n nothing will be done. Rebuilding a disk won't help with filesystem corruption, if xfs_repair can't fix it there's not much else you can do, except restoring the data from backups if available.

Ok, might have some good news but I could really use your advice on how to proceed.  I just realized that this disk that is corrupted (6tb) is one that I just installed 20 days ago to replace/upgrade a previous drive (3tb) and I still have this drive that hasn't been touched.  Using UFS explorer on my Windows machine I can see that all the data is still on the old drive.

 

What are the steps I should take to bring the new 6tb drive into good standing and then how to move the files from the older 3tb drive onto the fixed 6tb drive.  I do have another 6tb drive that is brand new if using that in some way would be easier.

 

I really appreciate any help you can give me!

[trurl]

1 hour ago, stratigary said:

installed 20 days ago to replace/upgrade a previous drive (3tb) and I still have this drive that hasn't been touched

You can configure a new array with whichever disks you want and rebuild parity using New Config.

https://wiki.unraid.net/Manual/Tools#New_Config

[stratigary]

3 hours ago, trurl said:

You can configure a new array with whichever disks you want and rebuild parity using New Config.

 Ok, this seems like the way to go but (please bear with me) could you please help me with a step-by-step.  I've been using this system for years without issue and I'm afraid of doing something in the wrong order and making it worse.  

 

Currently, my other 4 drives are fine and contain all the shares and data minus what I lost on the corrupted disk.  I have a copy of the data that was lost on the corrupted disk on a separate, but smaller, disk that I just pulled.  I also have another disk the same size as the corrupted one I could use as well.

 

What order of steps should I take using New Config to get this corrupted disk back in working order and then copy over the files to it that were lost?

 

Thank you for any help you can provide!

[trurl]

What I had in mind was just making a New Config with the original disk and all other disks, but without the replacement disk. Then after parity rebuilds, you can do the replacement again. No copying needed.

[trurl]

Or you could just go from where you are, format the corrupt drive while its in the array to get rid of its corruption, mount the original disk as an Unassigned Device and copy its files to the newly formatted array disk.

 

Do you know how to work with files on the server? I usually use Midnight Commander (builtin, mc from the command line to launch), but others use Krusader docker, and there is even a new file manager plugin available for Unraid 6.10.0-rc3

[stratigary]

23 minutes ago, trurl said:

What I had in mind was just making a New Config with the original disk and all other disks, but without the replacement disk. Then after parity rebuilds, you can do the replacement again. No copying needed.

Ah, that makes sense!!  To answer your other question, I am somewhat familiar with working with files on the server using MC and/or Krusader but not confident enough I think.  Your first suggestion sounds like a workable plan for me.  To clarify, is this how I should do it:

1. Remove corrupted disk assignment in the GUI to no device
2. Shut down server and replace disk with original disk
3. Start server and make New Config (should I preserve all current assignments?)
4. Let parity rebuild and check for any errors on disks
5. Remove original disk assignment in the GUI to no device
6. Shut down server and replace original disk with new larger disk
7. Start server and rebuild new larger disk from parity.

I really do appreciate all of your help. I've been searching everywhere for this specific issue but I can't seem to find much.  If there is anything I need to change here I would be happy to hear it.

[trurl]

@stratigary

1. Disable Autostart in Disk Settings
2. Shutdown
3. Remove corrupt disk and install original disk. Double-check all connections
4. Boot up
5. New Config - Preserve All - Apply
6. On the Main page, the only change you want to make is assigning the original disk
7. Start array and parity will be rebuilt based on the contents of all the assigned array disks

While parity rebuilds, on Main you should see lots of reads from all data disks, lots of writes to parity, and all zeros for the Errors column on Main. When parity rebuild completes, check that no data disks show as Unmountable.

 

If that all looks good

1. Shutdown
2. Replace disk with larger disk. Double-check all connections
3. Boot up
4. Assign larger disk
5. Start array to begin rebuild

While the larger disk rebuilds, on Main you should see lots of reads from parity and all other data disks, lots of writes to the larger disk, and all zeros for the Errors column on Main.

 

Ask if you have any questions or problems.

[stratigary]

@trurl

 

Thank you so very much for your help, everything has worked well so far and it seems like I was able to recover from this event.  Quick question though if you don' t mind.

 

After reinstalling the old 3TB disk, I planned on zeroing out the 6TB disk that had the filesystem error so that I could possibly reuse it in the future.  So, I popped it into a USB dock to use with unassigned devices and preclear it again.   Oddly, the disk showed up as being half full with all the files intact and the filesystem listed was reiserfs when it was originally xfs. 

 

Is there something odd going on here or did I do something wrong when I originally upgraded the 3TB disk (reisefs) to this 6TB disk (xfs)?

 

[trurl]

If you rebuilt a 3TB reiserfs disk to a 6TB disk, the rebuild can only result in a reiserfs disk.

[stratigary]

1 hour ago, trurl said:

If you rebuilt a 3TB reiserfs disk to a 6TB disk, the rebuild can only result in a reiserfs disk.

This is quite odd as it definitely showed as xfs on the main screen for the 2 weeks it was in operation.  I take it this where the issues came from.

 

I know reiserfs is old and not recommended for new drives, but is there a reason I should switch right now?  Could I add the new drive to my array as xfs (or suggested file system) and then shrink the array to fill that disk and remove the reiserfs disk?