unmountable XFS data disk


ShaneH

Recommended Posts

Hi
I have an 8tb data disk that stopped responding yesterday. Unraid would not mount the disk. I have removed the disk from the system to see if I could mount the 8tb disk on another Ubuntu system. No luck. "failed: Structure needs cleaning"

I have been reading about xfs_repair. No luck with the minor stuff.
I was thinking about xfs_repair -L /dev/disk

I read https://wiki.unraid.net/Replacing_a_Data_Drive for info on installing a new replacement drive.

How can I re-install the existing 8tb drive and have it rebuilt?
Thank you

Link to comment

Rebuilding will not fix filesystem corruption, and you shouldn't have tried to put the disk in another system, but maybe it doesn't matter. You should have asked us for help before doing anything.

 

Can you start the array without the disk? Possibly we can fix the emulated filesystem and then rebuild.

 

Go to Tools - Diagnostics and attach the complete diagnostics zip file to your next post.

 

Link to comment

Hello,     Sorry for the delay, crazy busy week.

I was able to get data off the 8tb drive by executing a XFS repair in another system.    
  I should have come to the forum first. I will work with you moving forward.

The 8tb disk is back into the unRaid server. Since the "repair" (I am sure this was not 100%)  the 8tb disk is now recognized by unRaid.  I don't think the 8tb disk returned 100%.   I was expecting unRaid to see the disk as unrecognized, then format and rebuild.

I am unsure of where to proceed next.
Yes, I can start the array.  

diagnostics file is attached.

Thank you for the help.

 

tower-diagnostics-20190120-1151.zip

Link to comment
48 minutes ago, ShaneH said:

What about removing the 8tb disk to another computer.   Erase the 8tb disk.   Then bring the 8tb disk back to unRaid and rebuild?

The notion of "erasing" is very vague so I don't really know what you even mean. Every bit on the disk has some value. And format certainly doesn't erase a disk. We often talk of clearing a disk around here, which means filling it with zeros.

 

If you want to rebuild to that same disk there is no need to put it into another computer and "erase" it. Rebuilding it will completely overwrite it with the rebuild anyway.

 

But, since you have already repaired it in another computer it might be just as well to

50 minutes ago, itimpi said:

completely rebuild parity or else run a correcting parity check.

I would probably opt for the correcting parity check, since the repairs may not have changed a lot of the disk.

 

I don't fully understand this part though:

49 minutes ago, ShaneH said:

When using my desktop OS to view the unRaid shares some files show up with a lock on the icon. 

What OS is the desktop? Can you give us a screenshot? Can you copy one of the locked files to your desktop?

Link to comment

The OS Ubuntu  16.04.5 LTS (Xenial Xerus) 64-bit.    Screenshot attached.

Yes, I can copy a "file with a lock icon" to my local machine and view it.

 

I am ready to move forward with the disk repair.  

Please add / edit / adjust the steps:

1) move 8tb to another PC

2) erase 8tb disk

3) return 8tb back to unRaid

4) allow unRaid to format and rebuild 8tb disk

 

Comments?

 

Screenshot at 2019-01-20 13:00:51.png

Link to comment
25 minutes ago, ShaneH said:

1) move 8tb to another PC

2) erase 8tb disk

3) return 8tb back to unRaid

Why have you completely ignored most of my post just above this?

 

All 3 of these steps are completely unnecessary. What does "erase" even mean to you?

 

Please go back and reread my post and ask if you don't understand something about it.

 

Link to comment
40 minutes ago, ShaneH said:

4) allow unRaid to format and rebuild 8tb disk

Also, this isn't what Unraid would do since formatting is not part of rebuilding. There is some danger that it would offer to let you format an unmountable filesystem though.

 

You MUST NOT EVER agree to format a disk you want Unraid to rebuild. There is a prominent warning in the webUI against this.

 

Format means "write an empty filesytem to this disk". That is what it has always meant in every operating system you have ever used. Possibly this is what you meant above when you used the word "erase". I don't know.

 

Unraid parity is realtime. Any time any write operation is done to a disk in the parity array, Unraid updates parity. If you format a disk in the array (format is a write operation), Unraid updates parity. So, after formatting a disk in the parity array, parity agrees the disk has an empty filesystem, and then rebuilding will result in an empty filesystem.

 

Link to comment

Another option would be to rebuild to a new disk and keep that already repaired disk as a backup in case it produced different results.

 

But, rather than explaining that, let me just ask this question.

 

Do you have another copy of anything important and irreplaceable? If not, that should be your first priority.

Link to comment

Sorry, I was not trying to ignore your post.
When I said 'erase' I mean delete the partition table (when the disk was in another PC).   When this issue started unRaid gave me a message about an "unmountable file system" and the unRaid GUI seemed to suggest 'format' before the disk could be used again.

 

During my first read, I did not realize this was one sentence:

2 hours ago, trurl said:

But, since you have already repaired it in another computer it might be just as well to

3 hours ago, itimpi said:

completely rebuild parity or else run a correcting parity check. 

I am 98% sure all of the pictures are backed up.   Some other files which I consider important are not.   I had a good backup before but it got damaged in a recent house move.  I did not fix that situation yet.

I may select the "install a new disk option".    I am going to check on prices and money.   Then I will return here to discuss with you.    Thank you

Link to comment
3 hours ago, ShaneH said:

I may select the "install a new disk option". 

If you want to rebuild to a new disk then you have to assign a replacement to the same slot. Probably that is what you mean.

 

 If however you want to add a disk to a new slot then that is a completely different process.

 

3 hours ago, ShaneH said:

I will return here to discuss with you.

👍

 

Link to comment

I was going to comment on that screenshot, but instead let me ask for a different version of that screenshot.

 

That screenshot is showing us the speed in the Reads and Writes column. It would be more relevant to know whether or not you (or anything else) have written to the array at all since putting the disk back.

 

Just above the "View" label in the column with the folder icons is another icon. Click on it and it will show us the total number of reads and writes instead of the speed as you have it now. Give us that screenshot.

 

Link to comment

Hi,  I have been keeping the array stopped as much as possible.   I have copied data off the array but I don't think I added anything major (I can't think of anything I added....  but never say never).  I would assume there have been some writes.  The array was stopped.  I started the array to provide the screenshot requested by Jonathanm.  I then stopped the array.   I started the array to obtain another screenshot for Trurl.

 

unRaid 2019-01-21 14:59:19.png

Link to comment

OK that doesn't look too bad as far as writes. Of course, those numbers are reset on reboot so I don't know if those are all that has happened in the array since the disk was modified (repaired) in another system.

 

But lets go back in the thread a bit for why we are considering going to a lot of extra time and effort (and expense of a new disk) to try a rebuild from parity instead of just getting parity back in sync with the already repaired disk.

On 1/20/2019 at 11:01 AM, ShaneH said:

I am 95% sure some files are missing.

What evidence do you have for your 95% certainty.

Link to comment

There are a lot of photos so it is difficult to find missing files without a directory tree compare tool.  I did not run a compare of photo against the most recent back up. 

  However, there we some movie files in groups of three.   I believe two groups of 3, are missing at least 1 file.   I did not do a lot of digging.

I am concerned other files (more difficult to notice) are missing also.   (IE:  photos or documents.  I have a few years of work files backed up to unRaid.  Not the end of the world if I lost some but I do like to refer to old projects some days.  Photos are important.)

  1)  You mentioned rebuild to a new disk and keep the old 8tb offline just in case.   Good idea.

  2)  I also needed to give more attention (money?) to backups.    I was thinking about buying a new disk for unRaid and using the 8tb for backups.

 

Link to comment

When you did the filesystem repair, did it create a lost+found folder?

 

Just to simplify the discussion, let's disregard any writes to the array that may have happened since the disk was repaired in another system.

 

If you remove the disk (or just unassign it) and start the array, the disk will be emulated from the parity calculation. Parity plus all other disks will be read to calculate the data of the missing disk. Of course, in your situation, that may be different than the actual contents of the physical disk at this point. The emulated disk is what would be rebuilt if you replaced the disk.

 

Since parity wasn't involved in the repair of the filesystem, presumably the emulated disk still has filesystem corruption. You could rebuild the disk, then repair it's filesystem (in the array, not like you did before), or, you could even repair the emulated filesystem, then rebuild the disk.

 

I think there is a very good chance that the end result would be the same as far as the disk contents are concerned, but maybe a slim chance it would turn out slightly differently.

 

You would still have to figure out what the differences between the two results were.

 

1 hour ago, ShaneH said:

I also needed to give more attention (money?) to backups.    I was thinking about buying a new disk for unRaid and using the 8tb for backups.

Looks like this is the first mention of backups anyone has made in the thread. I usually say something about it before this point. Do you have another copy of anything important and irreplaceable? That should be your first priority.

 

 

Link to comment

Hi.  I did not see a lost+found folder.
I removed the 8tb disk from the array and started unRaid.  The "Pictures" share is not present.
This file exists on the flash drive  /boot/config/shares/Pictures.cfg

I wanted to check if "Pictures" was present with the 8tb disk. I stopped the array and added the 8tb disk back.
I clicked on "start array".  

Warning [TOWER] - Disk 2, drive not ready, content being reconstructed

Notice [TOWER] - Parity sync / Data rebuild started

The 8tb disk returned to "unmountable".  Not sure what else happened but unRaid is now rebuilding the 8tb disk.   Rebuild in progress.

 

1 hour ago, trurl said:

You could rebuild the disk, then repair it's filesystem (in the array, not like you did before)

Looks like this is now the option.

 

1 hour ago, trurl said:

Since parity wasn't involved in the repair of the filesystem, presumably the emulated disk still has filesystem corruption.

We should complete this after the rebuild.

I am 98% sure all of the pictures are backed up.   Some other files which I consider important are not.   I had a good backup before but it got damaged in a recent house move.  I did not fix that situation yet.

 

Link to comment
1 hour ago, ShaneH said:

I removed the 8tb disk from the array and started unRaid...

... I stopped the array and added the 8tb disk back.
I clicked on "start array".  

This is exactly the procedure used to get Unraid to do the rebuild to the same disk. I don't remember anybody telling you to actually do this though.

1 hour ago, ShaneH said:

The 8tb disk returned to "unmountable".  Not sure what else happened but unRaid is now rebuilding the 8tb disk.   Rebuild in progress.

1 hour ago, ShaneH said:

Looks like this is now the ONLY option.

FTFY

 

And when its done you will be back to where you were before you repaired the filesystem in the other computer. The reason it returned to "unmountable" is because that is what it is rebuilding.

Link to comment

Reading my post just before yours though

3 hours ago, trurl said:

If you remove the disk (or just unassign it) and start the array, the disk will be emulated from the parity calculation. Parity plus all other disks will be read to calculate the data of the missing disk. Of course, in your situation, that may be different than the actual contents of the physical disk at this point. The emulated disk is what would be rebuilt if you replaced the disk.

I also didn't tell you not to do this. And the thing you added to this idea

1 hour ago, ShaneH said:

I stopped the array and added the 8tb disk back.
I clicked on "start array".  

was the last step in the procedure. If you had left that last bit off you would have been able to see what you would rebuild (to another disk), but without actually starting the rebuild on the same disk.

 

So I guess we will see what you have after the rebuild and (re)repair of the filesystem.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.