I have no problem accepting any number of errors from any number of causes happening, but how can they pass that last CRC test correctly?

 

One way this happens is that the data doesn't necessarily pass the CRC... the CRC fails, the drive returns the data it got, and the ERROR the drive returned got lost, so the data is accepted.

 

I was wondering if the drive had produced errors that were somehow lost.  I don't recall ever having seen a non-zero value in the errors column on the main web page - if there were read errors reported by the drive shouldn't they be flagged there?

 

As you might have noticed when I was adding the results of a parity check to my notes file (parity.txt) I was doing a copy/paste from the syslog, and I don't remember spotting any odd error messages between the messages that announce the start and end of the parity check.  This makes me think that if the drive had reported errors they somehow got lost.

 

Regards,

 

Stephen

 

 

This drive was RETURNING DIFFERENT DATA even though nothing was being written to it. 

 

I replace the drive with a new one (keeping the same SATA cable, port and power connector) and rebuilt.  After that I have not had a single problem in over 20 parity checks.

 

I put the bad drive on another SATA port but did not make it part of the array, and I ran the md5sum on it and found it still had the same problems.  So then I did a long SMART on it and did another round of md5sums and the problems still repeated.  Then I ran preclear on the drive and all the problems stopped.  I even copied (using dd) the data from the array back on to the suspect drive and repeated the md5 tests and still the drive behaved fine.

 

All this time there was almost no indication of any problem from that drive using the SMART reports.  At the most I got

 

 

If a drive was bit-flipping but well-behaved after a preclear, it seems to me it was a simple case of analogue meets digital.  We're assuming drives spit out bits, but bits are NOT what is stored on the surface of the platters.

 

Drive heads are analogue devices.  Data are written to disc via the heads having a voltage presented across their gaps.  Said voltage is one of two relatively-tightly-regulated values (i.e. digital) so there are two "choices" for every space that might contain a bit.  But what changes at the platter's surface for each bit is the density of the magnetic field.

 

So you may well imagine (or at least, I do) a scenario where there was some small amount of magnetic contamination, either physically or electrically.  What if there was some bit-sized amount of mud the head ran across while trying to store bits in that sector?  What if there was a brown-out of sorts in your computer system at the exact moment that hard drive was writing that bit?  In either scenario there is a possibility that the magnetic flux density is now neither representing a one nor a zero... it is in between to such a degree that no amount of error-correcting algorithmic magic is going to get that bit to be truly represented.

 

Data are written once.  We must assume they are written correctly every time that one time - or at least to the degree that the error-checking and the bit patterns recorded on the surfaces will reliably present the same bit time and again after only one single shot at the bit being written in the first place.  See where I am going?

 

It makes absolute sense (again, to me at least) that after a preclear, and subsequent re-copying of the data to the same drive, that all would be well with that drive.  This time, everything went smoothly and happened correctly.  There are no bits having ambiguous magnetic flux densities, or at least none that big brother isn't able to correct with 100% certainty.

 

This is a special case, to be sure.  But consider how many bits we're talking about routinely now (with 2TB drives being almost par for the course), and how closely-packed they are, and how many hard drives are manufactured every year, and how many underpaid people are at the controls during their production... Is it really that inconceivable that these errors can and do happen?  I think not.

 

So, to the suggestions of a revamped and improved parity checker, I am in agreement.  I have been having issues, too, and with 22TB online I grow weary trying to find where the problems might exist.

 

It is time, with 3TB drives around the corner, 4TB drives under the tree for Xmas 2012, for tools that are keeping pace.  Just today I was dreading doing a parity check with 15 3TB drives in my array a couple years from now.  It is not something I want to do 70 times in sequence to attempt to find errors.  Not in its current configuration, at least.

 

 

This is an old ZDnet article talking about this issue of "silent errors": http://www.zdnet.com/blog/storage/data-corruption-is-worse-than-you-know/191

 

For this reason the most recent filesystems, like ZFS and Btrfs, have some form of data integrity feature, like CRC or HASH, to allow to detect these errors at block level. 

 

The ZFS DataIntegrity section in Wikipedia is a nice read about it, with also other interesting links: http://en.wikipedia.org/wiki/ZFS#Data_Integrity

 

Ciao!

I have just read all this (VCA's troubleshooting), and my observations are as follows:

1) the only plausible explanation for the errors being concentrated to 5% of the disk is surface related issues.

2) such surface related issues should have been flagged by the firmware as a pending sector

3) in support of 1), the only plausible explanation for the fact that the errors disappeared after being precleared is, again, surface errors

 

This leads me to think of a firmware problem with the drive. Unfortunately I can't think of a way to verify such a firmware bug.

 

Moving the drive to another computer, running something other than unraid and performing the same MD5 check before the faulty drive was precleared would have helped even more certainly locating the error to the drive (but I'm not going to claim that would have directly confirmed my firmware bug theory).

 

In any case, thanks a lot to VCA for following through this far with this, and for pointing me to this thread. Some men would have given up long before.

 

I'll use the MD5 check tip to check for this in my server which has it's own very strange problem - parity errors after a disk rebuild, in same two blocks persistently - and after rebuilding again to another disk, 5 errors which are also in same locations each time with several parity checks. See my thread for more details.

http://lime-technology.com/forum/index.php?topic=12884.0

 

I'm not sure these problems are directly related, but it will be interesting to find out.

 

*snip*

 

I'm afraid that I don't agree with you here

 

There is, always, the possibility that a bit or block of bits has decided to do something they shouldn't.

 

P+Q parity would solve this problem, would it not? As you have two equations that disk can only have one answer so identification is easier?

 

Personally I'm trying to find an app that will either:

 

A. Compare the md5 checksums of the data on my primary drives vs that on the backup drives to ensure consistency.

B. If no originals are found, ensure the file can be read without error and generate a md5 checksum for any files over xxMb.

 

That way every month I can check my backups should still be what they should be!

 

 

That's OK, You are entitled to any opinion you like.

 

That a bit can flip is a given - but remember that the discs use their own checksum blocks per sector (I believe they have more than just one parity bit per sector, as I recall they have a small checksum), and the chance that a bit on both the storage area and the checksum (or even if it's only one parity bit) is remote, at best.

 

I like MD5's too as a extra check and would also like a system like the one You described, but that is kind of out of the scope of this discussion, it is something that works on a higher level and will guard you against errors occurring in transfers over the network or on computers with faulty RAM etc.

 

I stand by my observations that firmware should have reallocated bad blocks and my suggestion that if that did not happen in this situation there may in fact be a firmware problem on the discs in question.

That's OK, You are entitled to any opinion you like.

 

That a bit can flip is a given - but remember that the discs use their own checksum blocks per sector (I believe they have more than just one parity bit per sector, as I recall they have a small checksum), and the chance that a bit on both the storage area and the checksum (or even if it's only one parity bit) is remote, at best.

 

I like MD5's too as a extra check and would also like a system like the one You described, but that is kind of out of the scope of this discussion, it is something that works on a higher level and will guard you against errors occurring in transfers over the network or on computers with faulty RAM etc.

 

I stand by my observations that firmware should have reallocated bad blocks and my suggestion that if that did not happen in this situation there may in fact be a firmware problem on the discs in question.

 

I've been in the situation where blocks haven't been marked as bad and I've lost entire video files because they wouldn't read correctly anymore.

 

However that was one drive in many and even that drive seems to be working fine now it's been pre-cleared twice.

 

But still, P+Q will allow for the dodgy drive to be found.