Greetings.  Its been awhile since I dropped by but I have a question.

 

I've had unRAID for years.  My box is running ver 4.7.  Its a tower I bought directly from limetech.    Has 14 drives in it ranging from 320GB to 2TB.  I've lost a drive a couple years or so, replaced it and rebuilt no problem.  A couple days ago I noticed a drive had given errors and died.  It was a 500GB drive and I replaced it with a 2TB drive yesterday.  It was drive number 6.  I booted, the array started and began doing a data rebuild.  All drives were green other then drive 6 that was being rebuilt.  Meanwhile I had access to the array and things seemed fine.  It told me it would be 3-4 days till it was done.

 

Now its been a bit less then a day since I  put the new drive in.  I lost all contact with the server.  After checking that it was not my computer I finally hooked a monitor upto the unRaid box and got no signal.  So I hit the reset switch.  It was slow to come back up but it did.  All drives other then number 6 are green and it says a rebuild is in progress.  It looks like it started over.  If that was all I saw I'd be happy.

 

However.... It also says drive 8 is unformatted,  it did not say this yesterday.  Yesterday it was fine.  But, the reads are going up on all drives including 8 while the writes on 6 are counting up.  So it looks like a data rebuild might be in progress.  But how is that possible if there was error with drive 8 and its unformatted.  Drive 8 was fine this morning.  I had checked the status a couple hours before I lost contact with the server and all was good.  My shares seem to work.  But if I goto \\tower I can't see the default \disk8 share.  Its missing.  Ironicly I do see \disk6 and its the one I replaced that I hope is rebuilding.  It does report 31 errors on drive 8.  this number is not changing.  Just the read count is changing.

 

What are your thoughts?  Have I just lost a bunch of data?  Do you think it will straiten itself out?  Should I do anything.  I'm leaving it alone for now.

 

See the attachment for a screen shot of what I'm seeing.

 

Cal.

 

 

 

 

 

 

 

Hm.  Not good.  Went for supper and came back..  Server was dead again, no responce.  Rebooted it.  Its restarted the refresh.  Same disk8 still says unformatted.

Zip and attach a syslog. Drive 8 is having read errors. This may impact some files on drive 6 but the rebuild should eventually complete. Don't worry about disk 8 until the rebuild is complete.

Thanks for the reply dgaschk.

 

Got up this morning and its still running so thats good.  Only another 33 hours to go. 

Here is the syslog.  And thanks to your footer I actually saw how to get it.

 

Hmm.  I just noticed I'm posting in the 5.x section while I"m running 4.7.  Guess its time to update after all is back to normal.  I should check  back here more often but unRaid has been so rock solid I often forget about it.  Just let it do its thing.

syslog_-_calvin_Apr_27_2014.zip

The syslog is showing read errors for disk3 as well. Attach another screenshot.

Yes.  I noticed the errors.  It has not been 3 days but its down to 44 min till its done.  These errors did not show the first time.

 

And its done.  Tried a few random movies from Disk6 and they play.  But they still could be corrupt.  Nor do I know if anything is missing.  But I suppose that's a good sign. 

 

Checking data shows about 378 GB of used space.  It was originally a 500Gb drive so that's likely correct.  Now I guess I need to get drive8 back and see why drive3 had errors.

 

So I take is that its having problems mounting drive 8 but was able to read it in order to rebuild drive6.  From reading, am I to understand that "Unformatted" is more of a general error if it had issues mounting?  So I may have a filesystem issue at this point but no lost data?  At least not for sure yet?

 

Its 1am.  I'm off to bed, I'll check back tomorrow after work.

Paste SMART reports for disk 8 and disk 3. Attach a new syslog.

OK.  Took a couple min to figure out what that meant.

 

I ran the following lines in tellnet as those are drives 3 and 8.

 

smartctl -a -A /dev/sdg | todos >/boot/smart.txt

smartctl -a -A /dev/sdn | todos >/boot/smart.txt

 

I also see mention of short vs long tests.  I can run those if this is not what is being asked for.

smart_reports_and_syslog_-_calvin_apr_28_2014.zip

Do you have 2 spare drives to replace disk 3 and disk 8? Those drives will likely be usable but it's best the keep them untouched, until the array is fixed. If not, do you have 1 spare drive? pre-clear any spare drives.

2 spare drives?  hmm.  I do but I doubt they will work for this purpose.  I have 5  320GB drives not in use.  They were from an old raid 5 box back when I ran raid in my desktops.  I  doubt this helps however as both 3 and 8 are 500GB drives.  As old as they are they are, they are newer then the 320s.  I assume I can not replace and rebuild with a smaller drive for this purpose.  Actually I don't think its even possible to go backwards. 

 

If its just a matter of clearing data off the unRaid server,  then  I have 2 machines and each has a nearly empty 2TB drive in them.  I could clear data from any unRaid drive, then use it in whatever manner you have in mind.  Not sure if that helps or not.  I assume however that I need the drives in the unRaid box for whatever you have in mind.

 

I actually just rebuilt these boxes with win8.1 and all the data from before the rebuild currently sits on the unRaid server.  Thats what I was doing when I ran into this issue with my unRaid server and why they are nearly empty.  One is my Main computer and the other runs my TV and Home Automation stuff.  Both are still good hardware.  One was win7 but the other was end of life with XP and the next ver of XBMC won't even install on it.  So I figured I'd redo both boxes with win8.1.

 

Now that I think about it.  I could easily use the one 2TB drive from my main computer.  Its just a data drive that is nearly empty at the moment.  The OS and programs run from a SSD drive.

 

If I truly need two spare drives to recover from this problem.  Then I will just go buy another 2TB drive after work tomorrow and use that as well.  It would end up in the unRaid box anyhow after all is done and get used so its not like it would be a waste.

 

What did you have in mind?

 

 

I'm sure I've seen the preclear being talked about, but have never done it.  I've always just either replaced a drive and let it rebuild or added a drive and let it expand.  Never had any issues.

 

Is this what your refering to?

http://lime-technology.com/wiki/index.php/Configuration_Tutorial#Preclear_Hard_Drives

 

I just checked.  I don't  have "screen".  I know I don't have unmenu.  Is preclearing needed?  Does that not just write zeros to it?  Does the formatting not clear them?  Is there another way to preclear them?  How about just removing any partitions so the entire thing is unallocated before I stick it in the unRaid server?  I'll give this preclearing thing a go if thats what is needed to recover from this kind of situation without data loss.

 

 

This looks daunting.  But I'm sure its easier then it looks.  On the plus side.... I found my WD TV Live media player thats been missing for a few weeks when I went hunting for spare hard drives around the house. 

 

 

Was looking for some other stuff.  Came across a spare 500GB drive sitting with some tax papers.  Seems to work ok when connected to my main windows box.  Looks like it was a test install of windows I did a couple years ago.  I have no idea why this was in a box unused and not part of the unRaid server.

 

So I have one spare drive.  And another I can take out of my main box if needed.

The pre-clear will test the drive surface. If the replacement drive starts having issues you will be chasing your tail. Preclear the drive and then paste a SMART report.

I see.  Very well then.  After work, I will look into all the pre-clear instructions and do that with the spare 500GB drive I found and post the results.

 

I just wanted to say Thanks for your help regardless of how all this turns out.  I do appreciate it.

 

Cal.

 

OK.  I've looked into how this works.  I got screen, unmenu, and a couple other things it said to do.  I was not able to get the email portion working.  But that might just be a setting.  I've put the spare in my last free slot and booted.

 

According to the instructions the preclear list command was supposed to show a unallocated drive.  It never did.  It also says I can run preclear while the array is up.  But it the server wants to allocate the new drive.  So I assume that I just needed to deallocate the new drive.  I did so and the array stated just fine.  The preclear list command now shows the spare drive like expected.  I've now got a telnet section going, ran screen and under it I have a preclear in progress.  I see that even with a 500GB this may take awhile.  The notes say to run it 3 times so thats whats in progress.

 

Funny thing however.  The array is now fully up.  disk3 now shows NO errors this time and disk8 now shows only 2 this time and it no longer shows as unformatted.  The filesystem must have mounted this time after not mounting the previous 2 boots.  Sure enough the default share disk8 does show.  I can copy stuff from it and play the files.

 

Still.  I want this fully fixed so I will post the results of the preclear on this spare disk when its done as requested.  Obviously both disk 3 and 8 are suspect and need to be properly tested.  But this is good news.

 

Cal.

 

 

seems it failed.  I have no idea what this means.

In the zip is a screenshot of the preclear window when I got home.  And bits of the syslog.  The log was 182MB.  Yes.  lots of errors.  Either the spare is no good.  Or I just did something wrong.  This drive was readable when I stuck it in my windows box to see what was on it before I put in in the unRaid box and tried to pre-clear it.

 

I can't run smart on the spare drive.  I get the following.

 

 

 

 

smartctl 5.39.1 2010-01-28 r3054 [i486-slackware-linux-gnu] (local build)

Copyright © 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

 

Device: /8:0:0:0  Version:

scsiModePageOffset: response length too short, resp_len=47 offset=50 bd_len=46

>> Terminate command early due to bad response to IEC mode page

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

 

 

 

 

Did I just do something wrong?  Or is this spare drive bad.  I did the same smart command as I did earlier.

... And I see I picked the wrong smart parameters a couple posts ago.  "sdg" is disk3 but "sdn" is not 8,  its 7.  that drive was fine.  disk 8 is "sdo"

 

OK.  the smart.txt in this zip is drive8 done just now.

 

 

So I'm not quite sure what to do next.  If I just did it wrong.  Let me know and I'll try again with whaever advice is offered.    My array seems to be up ok right now.  All drives are mounted and seems to be working.  But with the past errors I"m not sure where I stand.  I've never seen errors before on any drive since I got this a few years ago.

 

 

Cal.

 

syslog_calvin_-_may_1.zip

It looks like the drive is bad but check the cabling to the drive and try again. Disk 3 and disk 8 require work. The /dev/sdX labels can change with every boot. Always make sure that you are designating the correct disk by checking unRAID main after each reboot.

I had no idea the letters could change every boot.  OK.  I will make sure to check each time, thanks.

 

OK.  In trying to get ready for the "work" you mention.  I will try the following tonight after work.

- open it up and check the cables.  I've not had anything in that spot before. Its my last free spot.

- I may take it out and try this drive again in another computer.  See if I can format it and read/write to it.  Its very possible I never had it in the array because it was bad.  But I see no writting on the drive that I would normally do to mark such a drive.

- try to pre-clear it again and see what happens.

- If it fails again I'll take my 2TB data drive out of my main box and try to pre-clear that.

- if the 2TB pre-clears then at least I'll have a working drive to play with when we get to the testing of drives 3 and 8.

    - I know the 2TB is a good drive.  I've been using it for a couple years in my main computer.

    - I may go buy another 2TB drive tomorrow instead of robbing my main box.  I'll see how the 500GB spare does overnight tonight.

 

 

I want to clarify something with you.  You mention checking "unRAID main".  In my ver4.7 clicking this in unmenu gives me the original page that I normally see with out unmenu.  Or perhaps you meant the normal main page.  In either case its the same page at least for 4.7 it is.  This page shows the /dev/sdX letters for assigned drives only.  It does not show unassigned drives. 

 

The pre-clear list command did show the letters for an unassigned drive.  That is what I used in the pre-clear command when I started it off.  I just want to confirm I'm doing this correctly.

 

I thought from reading that the array can be online while pre-clearing a drive.  And the only way I can start the array without it wanting to add the new spare is to unassign it.  Is this correct?

 

 

Last thing.  Would it help if I updated to ver 5?  Last time I was around here, 5 was just a beta.  Or is it better to continue as is until all the testing is done and update after.  I seem ok, but the drives are still suspect and who knows what will happen on the next boot.

 

 

 

Cables check good.  I took the spare out and win8 was unable to do a full format on it.  Speedfan can check smart as well and it say "error in an unknown location" and does not return anything meaningful.  My other drives report lots and are all good as a comparison.

 

I've stuck it back in the unRaid box and I've started another pre-clear.  I expect it to fail by the time I get up in the morning and I'll declare the spare drive as dead.

 

I'm thinking I'll just go buy a new 2TB drive in the morning.  Keep in mind I just replaced the failed disk 6 with a new 2TB drive a couple days ago.  I supposed I should run a test on that old drive as well.  I'll plug it into my win box in a moment and look at it.  I expect it to be very much dead.

 

I've booted twice tonight and both times the array came up as good.  No errors on either drive 3 or 8 tonight.

 

Here is a smart of 3 and 8 as well as a current syslog.

 

I'm thinking 3 and 8 had issues.  smart remapped the bad sectors and they are no longer in use.  But thats just me guessing based on not seeing any issues.  How do I properly determine if I have an issue?  What plans did you have for me once I have a new drive to play with?

smart_and_syslog.zip

Both disk 3 and disk 8 are showing pending re-allocated sectors which is not good (albeit not fatal).    If pre-clear does not set these back to zero then the drives are not usable in unRAID.

 

Both the drives also have re-allocated sectors.  This in itself is not an issue as that is what the spare sectors are there for, but it becomes an issue if the number keeps rising rather than remaining stable as that can be a sign of pending failure.

Had to run to work this morning for a few min.  But forgot to pickup a drive on the way back.

 

I've been reading up on what "current pending sector" means and a few of the other smart stats.  My take on it (and if I'm misreading this let me know), is that both drives are OK right now, but are in danger and could fail.  The pending thing means those sectors had issues but did not remap.  If they are written to and it fails they will get remapped.

 

Drive 3 has 8 pending and 34 already remapped sectors.  threshold is only 36.  So even if these 8 get marked, I'm over the threashold and in the danger zone.  This drive is running out of remappable space.  This drive should be replaced.

 

Drive 8 has 2 pending and 7 already remapped.  If the pre-clear remaps them I'm still only at 9 total and well below the threshold.  This drive is likely OK, but its some thing I should start tracking to see what happens to it.  If this number starts to go higher, it should be replaced.

 

I am a bit confused by the threshold for remappable sectors.  Is there a limit?  Can't a bad sector simply be remapped to any good one.  I'm aware that once they start to go then the chances are more likely others are in a similar problem state since the drive is likely just starting to show its age so its something to consider.

 

I'm also a bit confused by the fact that it still has the "pending count" when running under unRaid.  I see how it works on a normal system as if it can't read it. It does not know what should be there.  therefor it can't overright it to check if its ok or not and dertermine if it should be remapped or not.

 

But this should not be the case for unRaid.  I read that that it can and does rewrite to a sector that it had problems reading since if it can't read the drive it knows from the parity exactly what should be there and can therefor attempt to write the correct value to the sector in question.  The should have the effect of always getting rid of any pending counts and forcing the remap to happen.  Unless all the smart stuff is at a lower level and unRaid does not know whats happening and I misread or misunderstood this part as it seems to be at conflict with what is happening.

 

 

OK.  Screw it.  Current plan.  I'm going to go buy 2 more 2TB drives right now.  I'll replace both drives 3 and 8 one at a time letting it rebuild each time.  After Its all done I'll run pre-clear on the old drives.  Assuming things get marked as bad properly.  I'll keep drive 8 as a hot spare in case I need it for a temp patch job but I won't trust it long term.  Drive 3 I'm trashing just like I did to drive 6 a couple days ago as it will be well over the threshold.  (assuming it means what I think it does)

 

 

 

If it does not already have it.  unRaid needs (and so does windows) a utility that reads/writes and verifies all sectors to check for this kind of thing.  The scan disk in windows definitely does not do this despite telling me it does.  Both these drives (the old drive 6 and the spare I found in a box) pass the windows scan.  Both failed a full format that does write to each sector.  It would be nice if there was a way to scan while the system is still in use.

 

 

 

Arg...  Went and bought 2 more drives.  They had a sale on 3TB drives so I got 2 of those instead.  Since its larger then my 2TB parity drive, I replaced that only to see that unRaid can't see the 3TB drive.  I see now that I have update to ver 5x first. 

 

I guess I should not have let it start without the parity drive as now it wants to rebuild.  So now its rebuilding parity with the orig 2TB drive back in.  Once its done I will update to 5.0.5 and then replace the parity again.  Guess its wait time again.  I had wondered about the 3T... but thought... nah.... it will work.    :'(

 

Here is where I'm at.  Not sure if I've hurt things or not.  Not sure what my next step is.

 

Last time I wrote.  It was doing a parity sync after putting the original 2TB drive back in.  It finished and I updated to 5.0.5.  This was a clean upgrade.  I saved my original flash contents.  Deleted everything from the flash.  Copied everyting with the new structure from the download.  Replaced the downed config dir with my original.  Replaced my "go" file with the default one from the download.  Finally ran the make_bootable as admin just in case.  This gave me my config but put back the clean "go" file and let me have no plugins till I'm ready.

 

It booted and started another parity sync.  I was unaware it would do that as it was not mentioned in the upgrade notes.  While this was running I ran the permissions reset script as per the upgrade notes.  Both drives 3 and 8 gave errors as I expected.  When the reset script got to drive 8 it  took a lot longer then it did for the other drives but it did finally finish.  The parity sync finally finished a bit ago.

 

I was able to access all my shares but one.  But I was able to go into the drives directly and see stuff in this share.  Thinking I just needed to reboot I did so.  I clicked start and it got stuck.  I could not access the interface nor was anything mounted.  I could see all the physical lights on the hardware lit up green other then drive 8 that is orange.  Those are light on my hardware.  Not the status balls on the web interface.  The status ball have all been green.

 

 

 

Now here is where I may have made things worse.  I think its ok but I'm not sure.  I gave it 30 min and it still had not booted.  So I forced a reset on the hardware.  It rebooted and I got back to the main page.  This time it said it had detected an "unclean shutdown".  This of course is due to my forced restart.  I had thought I could start the array in maintenance mode and it would not restart the parity sync yet again as It would not mount the drives.  I was wrong.  Its currently doing yet another parity sync.

 

So now what?  I have two 3TB drives I can use.  Thats why I did the upgrade.  But I've not gotten to the point where I can put one in and see if unRaid will now see it.  My parity drive is a 2TB drive so it needs to be replaced first.  But now I'm not sure I can even mount the array until I replace drive8.  But I can't replace it with a 3TB drive because thats bigger then my parity.

 

I was going to replace the parity drive with the bigger 3TB disk and then replace both drive 3 and 8 with the other 3TB and the pulled 2TB original parity drive.  But seeing as I can't start the array due to disc 8 I may have to rethink that.  I wonder if I had left it long enough it would have mounted and the array would have started.

 

So.  I'm thinking I now have to

- wait till this current parity sync is done. 

- Pull my 2TB drive out of my main computer. 

- Put it in place of drive 8. 

- Restart and let it rebuild drive 8. and hope that some data has survived.

---- at this point I should be able to start the array normally and see if the rebuilding drive 8 has data. 

---------(virtual data? whats the correct term.  Its still being built at this point)

 

- Then pull my 2TB parity drive and put a 3TB drive in.     

---- I could put this pulled 2TB drive back in my main computer but its a green drive. 

---- My mainbox is a faster black one.  I want it back in the main computer at the end. and this one back in the unRaid box.

- restart and now rebuild the new parity drive.

- then pull disk 3 and replace it with the other 3TB drive.

- restart and now rebuild disk 3.

- pull disk 8 (my black 2TB from my main box) and put in the orig 2TB green drive that was the original parity drive.

- restart and let disk 8 rebuild.

 

Thoughts?  Its a lot of time but I don't see anything else I can do.

 

 

Crap.  Another wrinkle.  Its only been about an hour since I started the above parity.  I just now hit refresh and its saying its done.  There is no way it can be done.  Drive 8 is red balled.  Here is a screen shot and another syslog.  I'm not touching anything till I hear from you guys.

 

syslog_may_6.zip

bump.

Any thoughts?  I have not yet touched it.

 

With 2 brand new 3TB drives I can't yet put in and drive 8 now gone red.  (and only a mater of time till drive 3 dies)

 

Should I swap out disk8 with the 2TB drive from my main computer and see what happens hoping that parity built last time and therefor would rebuild disk 8 this time,  It was good last time before the hard boot.  But wanted to rebuild, see notes above.

 

Is there some sort of command or something I should do first.

 

 

Worst case senario I suppose is that I lose the contents of 8 and rebuild the parity again.  Then I could continue and get drive 3 replaced before it dies.  I do have a full listing of most of the tower but its by share not by drive.  Its just a file list generated and put dropbox, there is no checksum data to know whats corrupt.  It was just a list for checking from work or my phone if I had a certain movie or show without having to remotely access my network.  So I don't know exactly what was on 8.  I could do a new listing after I get it all fixed and compare the two to find out what was originally on 8.  Doing that, I should at least be able to figure out what I lost.  Hopefully it won't come to that.  In the future I'll have to somehow put a system in place to better track this.