Hi everyone,

 

So last night I noticed that reads from my unRAID box were incredibly slow.

 

So slow that Windows would freeze when trying to get a directory listing much of the time.  Often it would error out.

 

I've checked the cables, and everything appears okay.  I've also power cycled several times.  It seems a little bit better now in that I can still browse the directory tree and open files, but it's still extremely slow.  Video is unwatchable, and a 2kb text file can take 20secs to open.

 

This was accompanied with a large number of errors on two of my disks in the web UI.  Well, one had about 80, but the other around 1000.

 

I did a reiserfsck --check on my three data disks, but it didn't find any errors.  After running the reiserfsck on my disk3, the error count in the web UI was over 5000.  Disk3 actually took over 12 hours, whereas the other two disks only took about an hour each.

 

Sadly I've power cycled the box several times since then, and I didn't get a copy of the syslog back then.

 

The web UI is not reporting any failed disks however, which would indicate there hasn't been a write failure?

 

Given this, and looking at the SMART information, as well as the syslog, it looks to me like there's a problem with at least disk4.

 

I've attached copies of the smartctl output, as well as my syslog and a screenshot of the web UI.

 

Do I definitely have a problem with disk3?  Why is disk1 showing the exact same count of Current_Pending_Sector as disk3?  This seems quite odd to me...

 

Please tell me I don't have two failed drives! :'(

 

Any help would be greatly appreciated.

 

 

Cheers,

 

Nick

logs.zip

No one has any ideas?

 

Sorry, I'm pulling my hair out thinking I may have lost data!  I just ordered a new 2TB drive which I should have tomorrow, so I can replace a drive if necessary...

 

Thanks!

sdb and sdd both look like dogs to me from the SMART reports.

 

sda and sdc look fine.

 

If you swap sdb, then sdd may die during the rebuild.... and vice versa.

 

I'd order 2 drives, add them both as new, and COPY the contents of each dog to a new drive, then throw the dogs out or get them replaced under warranty..

 

But if you have to work with just 1 replacement, here is what I would do.

 

I would put the replacement 2TB in, preclear it, and add it to the array.  Then COPY the contents of sdb to it.

 

Now that the contents of sdb are safe on the new drive, I'd then zero out sdb leaving it in the Array so you keep parity intact.  Zeroing it out should allow any bad sectors to reallocate.

 

Then I'd do another pass with zeros and see if any more sectors were reallocated.  If not, then I'd COPY the contents of sdd to sdb, and wipe sdd with zeros, and see how much reallocation takes place.

 

You're in a tough spot mate.

 

 

 

Ahhh!  How scary!

 

Thanks Bubba 

 

I might try to get another 2TB drive tomorrow to play it safe (so a total of two new drives).  Hey, hopefully I can get the other two replaced under warranty and then I'll be able to extend the array a bit!

 

I got some new SATA cables today and replaced sdb and sdd.  sdb, I can't get to come up with a single error, in either Raw_Read_Error_Rate or the unRAID web UI.

 

sdd on the other hand is still having more read errors.

 

I'm not convinced that I will be able to copy all the data off sdd, as even doing a recursive directory listing ends up failing (but at different points each time).

 

How does this sound:

 

Copy the data from sdb to a new disk.  Then try to rebuild sdd based on parity?

 

I haven't done this before though, so the big question is, what is the absolute correct way to do this?

 

Sorry for such a newb question!

 

 

Cheers,

 

Nick

Copy the data from sdb to a new disk.  Then try to rebuild sdd based on parity?

 

As long as both sdb and sdd are both in the system, I fear that neither will last through a a full parity check or parity rebuild of the other.

 

If you copy sdb to a new disk, then try to replace sdd and let it rebuild, sdb may crap out during the rebuild, so sdd will NOT be rebuilt, so you could lose data on it.

 

As long as I have any drive (other than the one being rebuilt) that is flaky in the system, I never attempt to rebuild a drive from parity.... I always add another drive, and COPY the dying drive rather than rebuilding it.

You definitely have 2 dying hard drives based on SMART.

 

You might want to make sure you are handling your server/disks with care.. or you got extremely unlucky. Both need to be replaced, I would get data off them ASAP.

Thanks for your help guys!

 

I've now replaced both disks.  sdd was clearly the most fubared, so I replaced that first and rebuilt from parity as I couldn't even copy data off this disk.  Luckily, sdb held up, and I was then able to replace this disk.

 

It looks like all worked out well!

 

I'm hoping it was just a case of me being unlucky, I do try to take care of my disks, I have them scattered throughout the tower so no two disks are next to each other, and I have HDD fans installed in the front of the case to ensure airflow.

 

I do however use BitTorrent on the drives pretty much 24x7, so they never really get a break.  I'm guessing this might contribute, considering they're consumer grade disk and probably not designed for 24x7 disk access?

I'm hoping it was just a case of me being unlucky, I do try to take care of my disks, I have them scattered throughout the tower so no two disks are next to each other, and I have HDD fans installed in the front of the case to ensure airflow.

 

I do however use BitTorrent on the drives pretty much 24x7, so they never really get a break.  I'm guessing this might contribute, considering they're consumer grade disk and probably not designed for 24x7 disk access?

 

I have consumer disks and keep my server on 24x7.  Google's study on HDDs and data centers pretty much showed that failure rates between consumer and enterprise disks are the similar; it's factors like heat that cause failures.  In your case, it's not just luck; you could have had two drives fail over time (say, one in March and one just recently) but just happened to detect them now.

 

I monitor my array regularly (especially in summer) and keep track of drive life cycles in a spreadsheet. Also, I work in an IT department and am the manager with ownership of the storage, and, without giving away proprietary details, I wouldn't say that our failure rates on enterprise drives are much better.

 

Bottom line: glad to hear you got out relatively unscathed.  If you value your data (and time) unRAID just paid for itself, I'd say. 

 

One more data point for consideration: your problematic drives (sdb and sdd) are all the same models with relatively close serial numbers (ie within 1000 units of manufacture if the serial numbers are truly serial).  It is conceivable that you got 2 drives from a less than good lot.  Notice sdc's serial number is considerably different.  I believe unRAID uncovers these problems because initial and parity check i/o activity is so high, especially when adding drives or doing frequent parity calculations.

 

My newest rule of thumb is to NEVER use a parity and data drive from the same lot, for what its worth.

 

Glad you hear got your data onto a new drive in time.