(Solved) - Data Rebuild - unmountable disk present - is everything as it should be?

[FrozenGamer]

I replaced a disk that showed as faulty with a disk which had not been precleared and i had shucked from a seagate enclosure.  I added it to the array in place of faulty disk and started array with parity sync/data rebuild.  

 

see attached screenshot. it appears to be rebuilding the disk, but there is also an option to format (unmountable disk present) - Is it or is it not rebuilding properly?

 

I thought it was possible to insert a disk to array and not preclear it first?  Did preclear plugin ever get fixed? I am unclear on that and in this case would prefer not to preclear the disk.

 

I know that as tempting as it is to hit that format button - it is not the thing to do - been there done that!!! -   I still think this trap should be fixed.

 

EDIT - replaced screenshot with proper one.  sorry.

 

 

Edited April 14, 2018 by FrozenGamer

[FrozenGamer]

based on what i quoted below, did i bring the new disk into the array wrong? i am going to stop the rebuild and try it this way , see if it works.  Not even sure if the data rebuild was really going to work out in the end the way it was going in above screenshot?

 

 

 

This from this thread (sorry i don't know how to link exactly to the specific post in the thread - (its the 2nd to last on first page)

 

 

"A few thoughts after reading this thread ...

 

The fundamental problem was that you used the wrong steps to initiate the rebuild.

 

Looking at the steps you outlined ...

 

  On 2/18/2015 at 3:28 PM, momoz said:

... this is the detail of what happened:

1) rebooted unraid

2) Disk1 was redballed (ResierFS)

3) shut down array powered off and replaced disk 1 (2tb) with new 3tb precleared disk

4) booted unraid

5) selected new disk as disk 1 placement

6) started array figuring it will trigger the rebuild and it showed unformatted disk

7) I selected the checkbox to format and unraid started formatting disk1 as XFS

8 ) Array showed drive rebuilding

 

 

After step 2, what you SHOULD have done was stopped the array; unassigned disk1; restarted the array so it showed disk2 missing; and THEN shut down the array and replaced disk1 with the new disk.

 

Then when you rebooted; assign the new disk as disk1, and then the rebuild would have proceeded just fine.

 

For whatever reason, when you missed that step, you were presented with a Format option in Step 6 that should not have been there -- and, as already noted, when you selected Format you reformatted the emulated disk ... and that's what was then rebuilt.

 

 

There WAS a way to recover and start the rebuild over as long as you had NOT written any new data to the array; but since you've already been able to recover your data by mounting the old disk it seems you're essentially okay.    Any data you can't recover from the failed disk can simply be copied from your backups."

[FrozenGamer]

I tried this  - canceled rebuild, removed disk, started array with out disk 9 ( i didn't do any reboots though as both disks are in the machine already)

 

At this point it still gave me the option to format disk 9 even though it wasn't there - see screenshot

stopped array, started again with new disk in disk 9 slot.  

 

started array with rebuild - option to format is still there.

 

Something seems very wrong.  No i  am not an advanced user, perhaps i am good for dummy testing things.. It just seems really dangerous.

Edited April 12, 2018 by FrozenGamer

[remotevisitor]

I have to question your very first statement that you replaced a faulty disk,

 

In what way was the disk faulty which made you replace it?

 

Was the disk completely dead and not seen by the system after a system boot?

Was the disk showing SMART errors?

Were you getting sector read/write errors reported in the syslog?

 

Most disk problems are due to issues with the cabling, controllers, power supply with actual disk failures probably the least likely problem.

 

Then there is the problem that there is no actual problem with the hardware but for some reason the file system on the disk has some corruption (this sort of thing can occur after something like an unclean shutdown) and just needs the file system check/recovery to be run.   Replacing a disk and rebuilding it will not fix this problem, it will just exactly replicate the corrupted file system on the new disk.

 

The format option is given as a option if the system is unable to mount the drive, which can be due to there being no file system that Unraid considers valid (which includes a corrupt file system).  As you have been offered this option I suspect there was nothing wrong with your original disk but the file system had some corruption on it.

 

My advice at this stage is to let the rebuild complete and then seek advice here on how to fix file system corruption.   Keep the ‘failed’ disk in case it is needed to help recover file.

 

What ever you do, do not do any further actions without advice .... otherwise you are likely to do something which will prevent recovery of the disk .   And certainly do not select the format option.

[FrozenGamer]

Thanks remote visitor - answered questions below in quote. Unfortunately i have some smart errors in the parity disk as well - see attached labeled parity disk. - at this point i am doing as advised.   

7 minutes ago, remotevisitor said:

I have to question your very first statement that you replaced a faulty disk,

 

In what way was the disk faulty which made you replace it? - there was a brownout and disk showed up as faulty later when the the mover was trying to move from cache to disk 9. It was marked faulty.

 

Was the disk completely dead and not seen by the system after a system boot? - the disk is not dead, i believe the data is still good on the disk, but it has smart errors.

Was the disk showing SMART errors? - yes, see attached pipe-smart-20180411-1920.zip

Were you getting sector read/write errors reported in the syslog? - i don't now but there were a lot of read errors if i remember correctly.  I have syslog setup to store last 10 versions now from tips and tweaks so i should be able to find it with some help. 

 

Most disk problems are due to issues with the cabling, controllers, power supply with actual disk failures probably the least likely problem.

 

Then there is the problem that there is no actual problem with the hardware but for some reason the file system on the disk has some corruption (this sort of thing can occur after something like an unclean shutdown) and just needs the file system check/recovery to be run.   Replacing a disk and rebuilding it will not fix this problem, it will just exactly replicate the corrupted file system on the new disk. 

 

The format option is given as a option if the system is unable to mount the drive, which can be due to there being no file system that Unraid considers valid (which includes a corrupt file system).  As you have been offered this option I suspect there was nothing wrong with your original disk but the file system had some corruption on it.

 

My advice at this stage is to let the rebuild complete and then seek advice here on how to fix file system corruption.   Keep the ‘failed’ disk in case it is needed to help recover file.

 

What ever you do, do not do any further actions without advice .... otherwise you are likely to do something which will prevent recovery of the disk .   And certainly do not select the format option.

 

pipe-smart-20180411-2140-Parity Disk.zip

[JorgeB]

Looks like your parity disk is failing, which means you're in trouble and will likely lose data, but please post your diagnostics: Tools -> Diagnostics

[FrozenGamer]

pipe-diagnostics-20180411-1921.zip

[JorgeB]

Both parity and disk9 are failing, IMO best way forward would be to replace both with new disks, resync parity and then try to copy everything you can from old disk9.

 

Consider adding a second parity disk to the array.

[FrozenGamer]

I am about 1/3 the way through rebuilding a new disk 9 with 24 hours left.  should i stop this and do that or let it finish?

 

Can you give a little explanation of what is happening with the data in that case?  I think i understand but am not sure.  It sounds like we get a new array that has all of the data but disk 9 and new safe parity, followed by copying the the data off old disk 9, ending with most of the data, if all goes well.

[JorgeB]

9 minutes ago, FrozenGamer said:

I am about 1/3 the way through rebuilding a new disk 9 with 24 hours left.  should i stop this and do that or let it finish?

You can let it finish, but the rebuild disk will be partially (or completely) corrupt.

 

9 minutes ago, FrozenGamer said:

Can you give a little explanation of what is happening with the data in that case?

Since two disks are bad and you only have one parity disk unRAID is incapable of correctly rebuilding the failing data disk.

[itimpi]

1 hour ago, FrozenGamer said:

Can you give a little explanation of what is happening with the data in that case?  I think i understand but am not sure.  It sounds like we get a new array that has all of the data but disk 9 and new safe parity, followed by copying the the data off old disk 9, ending with most of the data, if all goes well.

The problem in this case is that for every sector that cannot be read correctly from the parity disk the corresponding sector on the rebuilt disk will be corrupt.   What the overall effect will be is difficult to determine as it depends on exactly which these sectors were located and how badly the sector was corrupted.    If they were in the middle of a file then only that file is corrupt,   If they were part of a directory control structure then that may mean that many files on the rebuilt disk cannot be located.

 

If you have a comprehensive backup strategy then you would have checksums for every file on your system.   This would allow you to identify any files that are missing, and also which ones (if any) are corrupted and need to be copied back from your backups.   The File Integrity plugin can help create such checksums for your XFS format disks.

[FrozenGamer]

Johnnie - I now have 2 8tb drives available, plus the one that is 1/3 through the rebuild - Can i put them all in and do the resync without preclearing?

Correct me if wrong.

Stop rebuild

Stop array 

ensure both bad disks are out of array

Change disk 9 that is partially through rebuild, but not failing into a parity disk

set another as parity

set another as disk 9

start array and let it resync (this step im not sure on because i haven't seen a situation where i don't have enough disks for proper rebuild resync) maybe it will offer the option to format?

after resync mount old disk 9 and start copying over to array.

 

Thanks for help.

[JorgeB]

You need to do a new config:

 

-Tools -> New Config -> Retain current configuration: All -> Apply

 

Then change any assignments you want, when done start the array to begin the parity sync, assuming the new disk9 is new it will be unmountable, you'll need to format it during or after the parity sync, during will take a few minutes.

[FrozenGamer]

thanks.

[FrozenGamer]

I am ready to press start but it says too many wrong and or missing disks on parity 1 - see screenshot. it also doesn't give me an option to start on bottom- says too many wrong and or missing disks.

I did Tools -> New Config -> Retain current configuration: All -> Apply

then i made the following changes - changed old parity 1 drive to a blank new drive

removed drive i was trying to rebuild as disk 9 and put unassigned

added the drive which i was trying to rebuild as disk 9 as parity 2

added blank new drive to slot 9

 

 

[FrozenGamer]

3 hours ago, johnnie.black said:

You need to do a new config:

 

-Tools -> New Config -> Retain current configuration: All -> Apply

 

Then change any assignments you want, when done start the array to begin the parity sync, assuming the new disk9 is new it will be unmountable, you'll need to format it during or after the parity sync, during will take a few minutes.

Just to confirm (once i get to that part) it is ok to format during sync, this is what erased data in the past for me.  - i assume this is ok because we are going to try and pull the data off the old disk 9 which i have saved? 

Edited April 12, 2018 by FrozenGamer

[FrozenGamer]

Standing by, until i hear what to do here,  Either i did something wrong on new config or new config method will not work?  My assumption was that new config allows you to build an array with more removed disks than necessary for parity recovery?  see screenshot 2 posts above for error.

[JorgeB]

You didn't do a new config, or all disks would be blue (new), repeat the porcedure.

[itimpi]

4 hours ago, johnnie.black said:

You didn't do a new config, or all disks would be blue (new), repeat the porcedure.

I wonder if he forgot to hit the Apply button in New Config and simply hit the Done button?    I have done that myself in the past!

[BobPhoenix]

10 hours ago, itimpi said:

I wonder if he forgot to hit the Apply button in New Config and simply hit the Done button?    I have done that myself in the past!

Me too.

[FrozenGamer]

3 hours ago, itimpi said:

I wonder if he forgot to hit the Apply button in New Config and simply hit the Done button?    I have done that myself in the past!

I believe that must be what happened - its all blue now.  Thanks to both for your help. I really appreciate it..  Really really really , its a bad feeling not knowing what is gonna happen to your data.

The parity rebuild is going, disk 9 is formatting

 

I have 2 disks with the following smart reports - could these errors have been caused by brownout and power outage(2 separate incidents, first an outage, then a brownout during rebuild a few days later, no UPS was there.)  Errors weren't present that i know of before.  once done with rebuild what should/can i do with these 2 8tb seagate archive disks?  preclear them a few times and see if they continue to fail?   i have attached smart reports.

 

pipe-smart-20180413-1620.zip

pipe-smart-20180413-1619.zip

[FrozenGamer]

I am going to mark this solved, might need a little help copying stuff back tomorrow when it finishes, but i think assuming that the power doesn't go out again during rebuild it should be ok

 

  - if i get an answer on last post on the smart reports and using disks here, great, if not i will ask later with a separate post.  Thanks so much for all who helped me.  Also i am trying to get better about being more concise and easy to understand with posts.  Thats a work in progress but constructive advice is appreciated.

Edited April 14, 2018 by FrozenGamer

[JorgeB]

6 hours ago, FrozenGamer said:

could these errors have been caused by brownout and power outage(2 separate incidents, first an outage, then a brownout during rebuild a few days later, no UPS was there.)

Possible but not very likely.

 

6 hours ago, FrozenGamer said:

what should/can i do with these 2 8tb seagate archive disks?

Paper weights?

[FrozenGamer]

thats what i thought!   thanks johnnie