Data and Parity Fail?


JP

Recommended Posts

I'm not sure what happened here because I have replaced failed drives before without any problem, but something odd happened here.  I had an old 2 GB data that was either failing or completely failed.  I still do have it.  I purchased a new 4 GB drive to replace it and had to wait for it to come.  unRAID was just shutdown while I waited.

 

Once the new drive arrived I took out the old drive and put the new one in, started up the server, did a pre-clear (passed just fine) and for some reason after the pre-clear the mounting of the disks was very slow and then I couldn't even reach it.  It appears to have just frozen.  I had to force a reboot.  Once I rebooted I think I started the array expecting the drive to be rebuilt, but instead it did a parity check and since the new drive didn't have any data on it, it looks to have resulted in a huge number of errors.  

 

So I'm left with an invalid parity drive and a new data disk that doesn't have any data rebuilt on it.  I'm not exactly sure what the safest thing is to do from here.  Should I try re-inserting the old data drive and see if I can possibly rebuild the parity drive and start over?  Any guidance is appreciated. 

 

 

unraid issue2.jpg

unraid issue3.jpg

 

tower-diagnostics-20190927-1554.zip

Edited by JP
Adding diagnostics
Link to comment

Something is missing in your description:

 

16 minutes ago, JP said:

but instead it did a parity check and since the new drive didn't have any data on it

- the diags are after this, ideally we'd want those

- what disk failed and doesn't have any data? all your disks show some data, though disk1 has fs corruption.

- if there was a parity check after the unclean shutdown it would be non correct, i.e., parity wouldn't be touched.

- parity is now disabled, did this happened before or after that?

 

Link to comment
1 hour ago, johnnie.black said:

Something is missing in your description:

 

- the diags are after this, ideally we'd want those

- what disk failed and doesn't have any data? all your disks show some data, though disk1 has fs corruption.

- if there was a parity check after the unclean shutdown it would be non correct, i.e., parity wouldn't be touched.

- parity is now disabled, did this happened before or after that?

 

Thank you and very sorry.  I don't always know exactly what information is needed.

  • I attached the diags right after posting so you should see them now.
  • DISK1 failed or at least was showing signs of failure (can't remember exactly).  I have no idea why DISK1 is showing that there is data on it because there is clearly nothing there.  I go look at it from FILE EXPLORER and it shows the MOVIES folder in it, but no files.  Every other data disk shows files I would expect.  Something tells me that maybe the parity tried to rebuild is, but failed in doing so. 
  • The parity becoming disabled seemed to take place after all this.  This is pretty much when I stopped things and sought advice.

 

Link to comment

I sincerely appreciate johnnie.black's help here, but I'm not exactly sure where he was heading with his guidance here and I don't think he has been online in a few days.  If anyone has an idea of where he might have been heading with these steps and can provide additional guidance, I would appreciate it.

Link to comment
8 hours ago, JP said:

I sincerely appreciate johnnie.black's help here, but I'm not exactly sure where he was heading with his guidance here and I don't think he has been online in a few days.  If anyone has an idea of where he might have been heading with these steps and can provide additional guidance, I would appreciate it.

If the reiserfsck with —rebuild-tree completed successfully the disk in question should now mount when the array starts and its contents be visible.   The results would have been visible in the window that was running the reiserfsck - as far as I. Know they are not saved elsewhere.

Link to comment

Thanks.  The rebuild appears to have worked fine, but since there was never any data on this drive (it is new) there wasn't anything there to rebuild.  It appears to simply reflect the space being used accurately now.  Before it reflected something like a terabyte being used, now it only reflects 33MB being used, which is only one Folder and no files.  

 

I don't know why, but this drive mounts really poorly.  It takes about 5 minutes to mount whereas all the other drives mount quickly.  I'm not sure what is going on there.

 

Ultimately, since the parity seems to be invalid, I continue to think I need to put the old drive back in the array and keep my fingers crossed that the data is still there and then do a parity check.  Does that seem reasonable?

 

I'm still very confused about what happened here since I've rebuilt drives before, but in this case it seems to have gone from one drive failing and then the parity failing and becoming disabled.

Link to comment

Well, I tried the old data disk that was failing and I'm hesitant to force it because it is reflected as WRONG in the GUI.  I didn't start the array with it reflected as WRONG, but I'm wondering if I should?  If I can at least get to the contents on the drive then I can move them to another data disk, leverage the new blank 4 TB data disk in its place and then rebuild parity.  Does this seem reasonable or should I be going down a better path?

Link to comment
7 minutes ago, itimpi said:

I would not be trying to get the old data disk into the array, but instead try to mount it as an unassigned device.     If it mounts that way you will be able to copy data off it back to the array.

Thanks.  The parity drive still reflects as having parity disabled, but the drive is healthy and I'm still not sure why that happened.  Is it best to rebuild parity first so that all my other drives are protected and then attempt to bring the data over from the old data drive?

 

Link to comment
22 minutes ago, JP said:

Thanks.  The parity drive still reflects as having parity disabled, but the drive is healthy and I'm still not sure why that happened.  Is it best to rebuild parity first so that all my other drives are protected and then attempt to bring the data over from the old data drive?

 

If parity is showing as disabled then a write to it failed for some reason!    


Have you confirmed that you can mount the old drive as an Unassigned device?    If so then, yes, I would recommend getting the array back into an error free protected state before copying the data back from the UD mounted drive.   .

Link to comment
18 minutes ago, itimpi said:

If parity is showing as disabled then a write to it failed for some reason!    


Have you confirmed that you can mount the old drive as an Unassigned device?    If so then, yes, I would recommend getting the array back into an error free protected state before copying the data back from the UD mounted drive.   .

The drive does show as unassigned, but isn't healthy.  Is it possible to make an attempt to access the data on an unassigned drive outside of the array?  I'm not sure how you would access data on an unassigned drive outside of the array. 

Edited by JP
update
Link to comment
1 minute ago, JP said:

The drive does show as unassigned, but isn't healthy.  Is it possible to make an attempt to access the data on an unassigned drive outside of the array?  I'm not sure how you would access data on an unassigned drive outside of the array. 

Do you have the Unassigned Devices plugin installed?     If so does the ‘mount’ option work for the drive?

Link to comment
9 minutes ago, JP said:

I just now installed the plugin.  I see MOUNT reflected next to the drive, but hovering over it does nothing.  I can't click on it.  Does that mean the data on it is unrecoverable?

Quite possibly unfortunately.  Mount not being offered means that the plugin is not recognising the disk format.

 

The other possibility is that the disk is not faulty but that there is file system level corruption that is stopping the disk being mounted.   In that case running the appropriate file system recovery tool may make the disk usable.   What format is the drive meant to be in?

Link to comment
1 minute ago, itimpi said:

Quite possibly unfortunately.  Mount not being offered means that the plugin is not recognising the disk format.

 

The other possibility is that the disk is not faulty but that there is file system level corruption that is stopping the disk being mounted.   In that case running the appropriate file system recovery tool may make the disk usable.   What format is the drive meant to be in?

reiserfs I believe.  Is that what you are asking for?

Link to comment
Just now, JP said:

reiserfs I believe.  Is that what you are asking for?

If that is the case then you want to try from the Unraid command line running 

reiserfsck /dev/sdX1

where 'X' is the value for the drive in question.   If reiserfsck now suggests to use --rebuild-tree rerun reiserfsck with the --rebuild-tree option added to the command go ahead and do that.   Note that the reiserfack command can take some time to run so do not close the console window while it is running.   If reiserfack completes without a fatal error the drive should then be mountable.

 

If you get some other response from reiserfsck then post what happens for further ideas.

Link to comment

Thanks.  I thought that might be the case.  Oh well.  At the very least I have my most important files backed up on an external hard drive so I should be ok or at least I hope so. :)  I'll know more once I get all the dockers up and running, but first I need to ensure the new disk is operating properly and parity is up and running again. 

 

First, it would probably be better to change the FS from reiserfs to XFS on the replacement data drive since I think XFS is a more current standard and there isn't any data on the drive.

 

Once I do that, I noticed that an option for a Parity Check no longer exists for me.  Instead it wants to do a read-check on all the array disks.  I guess after I do that, the option to rebuild Parity will appear...correct?

2019_10_04__12_19_15.jpg

Link to comment
8 minutes ago, JP said:

Thanks.  I thought that might be the case.  Oh well.  At the very least I have my most important files backed up on an external hard drive so I should be ok or at least I hope so. :)  I'll know more once I get all the dockers up and running, but first I need to ensure the new disk is operating properly and parity is up and running again. 

 

First, it would probably be better to change the FS from reiserfs to XFS on the replacement data drive since I think XFS is a more current standard and there isn't any data on the drive.

 

Once I do that, I noticed that an option for a Parity Check no longer exists for me.  Instead it wants to do a read-check on all the array disks.  I guess after I do that, the option to rebuild Parity will appear...correct?

2019_10_04__12_19_15.jpg

That is because you do not have a parity to check at the moment!   It comes back when you have rebuilt valid parity.   To do this you stop the array; unassign the parity disk; start the array to ‘forget’ the current parity assignment; stop the array; assign the parity disk; start the array to build new parity based on the current assigned data disks.
 

it certainly makes sense to change the new disk to XFS before you put any data on it.    To do this with the array stopped click on the disk and you can change the file system to be XFS.    When you start the array you will see the disk is unmountable and there is a checkbox to format unmountable disks.  Check it is the correct disk and then go ahead with the format (it only takes a minute or so).

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.