On Oct 1st the monthly check kicked in.  Drive 12 failed.  (2TB drive  serial 2V2)

 

I took it out and the system while unprotected, seemed normal.  I checked warranty and its still covered so I ran the seagate tools to get the error codes so I could send it back.  However, its passed every test the tool does.  I know unraid does not disable a disk without good reason so I put it back in to let unraid start to recover it.  Unraid had issues booting.  Seems to be extremely slow and I saw a bunch of errors in the syslog about something being frozen.  After a few hours it only told me it would need another couple months to rebuild.  So something is obviously not right.  It should be a couple days.

 

I figured the drive must be bad after all and told it to cancel and shutdown.  It took forever to do so.  I bought a new drive  (3TB serial 610) and put it in.  This drive however is performing the same way.  very slow.  unraid wants the next 3 months to rebuild it.  Obviously this is not right.  I wonder if I do not have a drive issue but some other issue.

 

The syslog does say frozen.  I don't know if is the same error as before.  Its just a word that jumped out at me.  The log is full of errors.  here is an attachment.

 

I'm wondering what my options are.  If this is a problem with this slot itself.  Can I not use this slot?  Disk 13 is occupied but slot 14 is free.  Thats my last slot.  I could move all the data from the virtual disk12 to other drives and then shrink the array so I'm protected again.  I've never done that before but it seems logical.

 

What would happen if I stopped the array right now and put the new drive in slot 14 instead of 12 where it is now?  Would it try and rebuild the data from the disk12 using disk14?  Or do I need to get rid of 12 first.

 

seeing as slot 14 is my last open spot, perhaps I might be safest if I shrink the array and get protected.  Then run preclear on both these drives using the last open spot to check them before I try expanding again.

 

I guess this all depends on what you guys think the issue is.  Hardware in slot 12 or a bad drive and then an unlucky bad brand new drive.  It seems unlikely its the drives but I  have no idea what the syslog is telling me.

 

Seems the syslog is to big even compressed.  here is the last third.  The start of it already did not have the bootup info.  I assume it must only get so big before trimming itself.

 

syslog-2014-10-05_-_part.zip

First thing.  Do NOT panic!!! 

 

From what you have written, I would guess that you are are on ver 5--- Hopefully, ver5.0.5 as ver 5 identifies disks by their serial numbers-- Not anything else.  So you can move the disk between slots and controller ports without any problem. 

 

First question (that any expert is going to want to know), did you preclear this replacement disk?  This provides some assurance that the replacement disk is actually good.

 

Second, are you doing the rebuild with the GUI running full time?  (The GUI will steal CPU cycles and make things run slower and make that time to completion much longer than it should be.) 

 

Remember, don't do any unnecessary writing to your array until you get this issue resolved. 

 

I am not enough of an expert to give a clear set of procedural steps to get you out of trouble but I recall reading threads about problems very similar to yours and the expert walked the OP through the problem and got them back on line again. 

 

 

ver 5.0.5 pro.  yes.

 

I'm fairly stock other then a couple basic things.

running unmenu and email notification package.

there is a monthly checkdisk that kicks in and a command that keeps most of the dir structure in memory so the drives don't spin up when I'm just looking.  But those are all minor tweaks.

 

Precleared.  Actually no.  I just stuck it in as I've done many times in the past.  I know its a good idea and I likely should have precleared it.  I know what the preclear thing is as I used it last time I had issues to confirm a couple dead HDs.

 

No.  I don't  leave the page up.  I don't think that would cause much of an issue.  Certainly not the levels I"m seeing.  But for what its worth... no.  Its habit,  I close stuff I'm not actively using.

 

Back when I first got the system I'm pretty sure you could not move drives around but I thought I had read that you now can.  Was that a ver 5 thing?  my original rig (still running) was bought in 2007.  I've updated a couple times since then and went to 5.0.5 a couple months or so ago.

 

No panic,  not yet.  I"ve recovered from worst.  But it was always HD related or my own stupid move.  This feels like its not the HD.  Im just not sure how to verify that.  I think I need to move the new drive to another slot.  But I would like confirmation that this is what I think it is.  And the proper method to do this.    Is this what my syslog is showing? How do I test or confirm?

 

The most I've done to the main unit is replace a fan buried deep in the enclosures.  If this is a slot issue, then I'll just not use that one. (since that seems to be an option now)  Cables seem tight.  Since I'm nearly out of slots and I've been using this for the past 7 years.  I may be due for whatever the current build from limetech is.  But I do have some old drives in this one so Its easy to replace smaller drives If I run out of slots.

 

Unneeded writes....  two boxes running genie timeline that backup to the unraid box.  Currently paused.  thanks.  Forgot about those running.  Not that this is causing the slowness.  These are set to run early morning so have not actually run yet.  But I've turned them off for now.

 

The syslog shows repeated errors on whatever drive is connected as 'ata11'.  Without the boot-up part of the syslog it is difficult to identify exactly which drive that might be.

 

The symptoms in the syslog look rather like a power/SATA cabling issue.  Another option is an inadequate power related issue.  However it could be the disk itself and a SMART report might help identify whether it is that.

I'll shutdown when I get home after work.  I assume ata11 is referencing slot 12 but I'll have to see if its obvious when I get home.  If not, then I'll reboot it and get a syslog that includes the boot process.  I'm pretty sure the syslog was clean without disk 12 connected.  I'll boot once without it plugged in and then boot with it connected to get the logs.

 

I know I can do this much without hurting things so I'll get those logs.  I'll wait till someone can confirm the proper method to move to another slot before trying that.

 

Considering that I'm running out of slots here.  Is it possible to run preclear from a windows 8.1 box.  I know its part of the unraid but I'm wondering if there is a utility that does the same job.  Or was this put in place specifically for unraid and it must be run from a telnet/screen session.

 

And now the panic hits.  I can no longer get to my admin page.  But I can get the unmenu page.  I'm not sure if its fully functional.  This syslog is a boot without drive 12 connected.  At the moment the array is up and running (slowly).  I've started a copy from the virtual drive 12 to an external drive.  The syslog still shows all sorts of errors.  I thought it was from drive 12 but that drive is not physically connected at the moment.  Hopefully this copy is done by the time I come back from work tomorrow.

 

How can I tell status from a telnet session?

syslog-2014-10-07.zip

While I'm thinking about it.  What is a good windows app that supports massive copying/moving files around a network that uses a queue and does not have issues with unRaid?  As soon as you add more stuff to copy windows starts getting in its own way.  While it does complete it could be a lot faster.

Disk4 has a bad or loose SATA cable.

Copying data from the array was to slow.  I mounted the old disk 12 on my win8.1 machine and was able to pull the entire 1.5 TB of data it had on it to another machine.  So It looks like my old disk may have been fine.  But I'll test it before it gets used again.

 

Thanks.  I'll have a look at the cables for disk4.  Could this be why it originally disabled disk12 on me?

 

At some point over the last couple days the web interface came back on its own.  After I check the cables on Disk 4.  What is my next step?  Keep in mind we can shrink the array if needed now that I have all the data from disk12.  Once I check the cables on disk 4 I'll report back.

 

Should I pull disk 4 and try to copy data from it first before doing anything?  I just want to make sure messing with the cables and booting a couple times won't corrupt anything on it.

 

 

Back near a computer again....  I was thinking.  Currently disk 12 is disabled.  I've been told to check disk 4 cables.  Once I'm home tonight.  Can I simply pull disk 4 and put it in another slot in order to see if the cables in disk 4 was the issue?  Would it boot up, find the new location for disk 4 and boot with disk 12 still disabled.  Is that possible?  If it is and the array is not super slow like it is now, is that proof that disk 4 cables are an issue?

 

Or does  the fact that disk 12 is disabled complicate this?

 

dgaschk says my cables for disk 4 are a problem.  Sounds like a very good possibly.

 

So I get home.  Do a bit of reading that says I should be able to pull a data drive and move it to another slot as long as I don't move the parity disk.

 

I figured I'd give it a try as a test of the disk4 cables.  I shutdown, then pulled disk 4 and moved it to slot 14 that is a previously unused slot.  Now remember disk 12 is still currently missing.  Then I booted.  It booted normally.  The array came online right away as it used to do.

 

I did a screenshot of my hard drives before I shutdown and again after.  At first I did not see any difference at all.  The serial for disk 4 is the same.  But why.  I moved it to slot 14.  Should not disk 4 be blank and disk 14 have the serial that was in disk4?  I must have misunderstood something.

 

Now I see the difference.  The letters in the brackets are different on 4 of the drives.  What do these represent?  Here is the new syslog on bootup after I moved disk 4 to 14 as well as the before and after screenshots.  I don't see any errors. Can you confirm?  Speed wise, I can pull data from any drive at a normal rate across the network.  Even the virtual disk 12 copies data fine.  Slower since its reading all drives to do so but performing as expected.

 

So what happened here?  I thought disk number directly related to slot number.  I guess not.  Should I label all my disks right now so I don't mess them up in the future.

 

I've always thought of them as the following in my tower.  Parity is bottom left and we just count up from there.  I have 3 rows and 5 drives per row.

 

All slots other then 4 and 12 have drives in them.

 

10 11 12 13 14

5 6 7 8 9

P 1 2 3 4

 

As you can See to move disk 4 I pulled the drive from the bottom right and put in in at the top right.  I'm assuming now that ALL my drives are accessible and speeds are normal that the original issue was the cables in slot 4.  Do you guys think this is the case?

 

So what is the slot/disk relationship?  Where can I read about that?  I guess I need to keep track of what HD(via serial) is in what slot and what its now called now that its possible that slot number does not match drive number.

 

Do you think the bad cables in slot 4 caused my original disk 12 to fail.  Or do I likely have bad cables in slot 12 as well.  Remember I mounted disk12 on my windows machine as was able to copy all the data to an external usb drive.

 

So whats my next step.  I don't  have any empty known good slots.  I have a virtual disk12 due to the original fail.  It looks like the drive is good.  Its possible slot 12 also has bad cables but I'm not sure how to test.  I don't have a free slot to use for a preclear.  Should I stick the drive back in slot 12 and see if it will rebuild.  If its extremely slow after a few hours saying it will take months then I'll know the slot is bad as well.  If it rebuilds in a couple days then all is good.  I'm assuming that cancelling a rebuild in progress wont hurt anything as I've already done that once.

 

 

 

attached is the syslog.

 

and here is the before and after shots since I can't attach them with only 192K.

 

before.

 

after.

 

 

Not sure why they look so small in the preview post.  The pics are fully readable on photobucket.  Does this forum shrink posted images?

 

 

 

syslog-2014-10-09.zip

Starting with ver 5, (as I seem to recall) when you first install unRAID on a new machine, you have to assign one of the harddisks that is physically in the machine as 'disk 1'.  unRAID then stores the serial number of that harddisk and permanently associates it as 'disk 1', regardless of what the actual hardware configuration is.  If you physically remove that harddisk (and then put a new hardddisk in system), unRAID will then insist that you select that new disk to become 'disk 1'.  At that point, a changed configuration will be stored for future reference.

The syslog is too short to see any ATA issues. Rebuild disk12 and then post a syslog. You don't need to wait for it to finish.

I started the rebuild of disk 12 at 2059  the last line is and email status update at 0047.  So its been rebuilding for almost 4 hours now.  I don't see any errors in the log other then my one cache dir command but I know what that is.

 

The only thing I removed was my email address in the last few lines of the log.  Its looks like it will take about 2 days to rebuild.  This is normal.  Its  currently got the new 3TB drive that I bought as a replacement in as the one being rebuilt.  The syslog from the last 4 hours is attached.

 

I believe the original HD that was in  disk 12 is ok.  I would like to preclear it as a test but I have no free slots other then the faulty slot 4.  Is there a windows app that I can run to do the preclear job or is it strictly part of unraid?  I'd like to confirm the original drive that was in disk 12 is ok before trying it in the likely bad slot 4.  If there is no windows app that can preclear it.  is there one that can at least test the drive as fully as the preclear would?

 

 

If the rebuild runs to completion I'll mark this as solved.

 

Once this is protected again I may open it up and check the cables on slot 4 to see if anything is loose.

 

I just want to say thanks guys.  I would have never guessed that the cables on disk 4 had caused disk 12 to fail.

 

 

I think its time to look at another Raid server.  Use one for daily use and another as backup.    I'd love to look at online backup but that will never happen due to monthly limits.  I may do it for just the critical stuff.  But thats another topic.  I think I'll go read though that big backup thread I see.

 

syslog-2014-10-11.zip

The log looks ok. It's unlikely that disk4 cabling caused the disk12 error. After disk12 failed the problem with disk4 caused the slowdown and freezes.

disk 12 finished rebuilding this morning without issues.  Now I'll have to check the old drive and the cabling on slot 4.  But the array is back in a protected state and working normally so I've marked this as solved.

 

Thankyou.