It was a dark and stormy night when I unpacked the 400 GB ST3400620A IDE hard drive that I had just received from Seagate as an RMA replacement.  I decided to use the drive to replace a smaller one in my array.  When the array had finished its rebuild I decided to do a quick manual parity check and noticed that multiple sync errors were flagged.  It turned out that I was getting between five and 10 sync errors per gigabyte but none of the discs were showing any errors on the web interface.

 

I checked all 14 of the data disks in the array using smartctl and reiserfsck with no errors.  I then wrote a 2 GB test file to each drive in turn and did a bit by bit compare with the original from Windows XP.  Sure enough the new 400 GB drive was producing five to 10 errors per gigabyte but all of these were silent.  It seemed incredible to me that this kind of error rate would not be picked up by the drive electronics or the IDE cable CRC protection. 

 

The SMART data for this drive showed PASSED and was unremarkable except for an unusual number of seek errors and relative high raw Raw_Read_Error_Rate and Hardware_ECC_Recovered count.

 

ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE

  1 Raw_Read_Error_Rate    0x000f  114  100  006    Pre-fail  Always      -      144563809

  7 Seek_Error_Rate        0x000f  067  060  030    Pre-fail  Always      -      5433102

195 Hardware_ECC_Recovered  0x001a  068  065  000    Old_age  Always      -      208854657

 

 

This problem persisted and followed the drive  when I swapped out the IDE cable, changed the offending drive from slave to master, and used another IDE channel.

 

I then moved the offending hard drive to another computer running Windows XP, did a quick NTFS format, and transferred and compared 50 GB worth of test files to it with no problems.

 

I moved it back to the unRAID array, installed it as a new drive and repeated the 2GB write and compare test with the same result as before 5 - 10 errors per GB.

 

So I conclude that this drive has a problem which is producing random single bit errors which are silent in the unRAID version: 4.3-beta6 environment but which are masked and corrected in the XP environment.  Any ideas?

That is indeed troubling.

 

You've analyzed it well.  Did you happen to check the Windows Event Viewers (System, I believe) for Disk errors?

 

The drive is obviously unusable in an unRAID server.  I would recommend using a tool like SpinRite for deep testing of the surface, to try to force the SMART system to retire all of the weak areas.  I suspect you were sent a well-used, but refurbished drive, with weaker magnetic areas.  They could have reset the firmware, and sent it back out, to give it a little more life.

 

...Did you happen to check the Windows Event Viewers (System, I believe) for Disk errors?

 

The drive is obviously unusable in an unRAID server.  I would recommend using a tool like SpinRite for deep testing of the surface, to try to force the SMART system to retire all of the weak areas.  I suspect you were sent a well-used, but refurbished drive, with weaker magnetic areas.  They could have reset the firmware, and sent it back out, to give it a little more life.

 

 

Nothing in any of the XP logs, including Events, related to this device. I'll check it with Spinrite shortly and don't plan to use it in my array.

 

However my biggest concern is that nothing was flagged in unRAID in spite of a ~ 1 in 10^9 BER for this drive. Had I not ran the manual parity check I would not have known that the drive was bad. I use this part of my array for ISO movie images and I guess that neither my virtual DVD drive (Daemon Tools) nor my player would have flagged these random bit errors. But if I'd had another drive fail then restoration; using parity derived from my silent peril,  would have created bit errors in the restored image that may have created problems depending on what kind of data that image contained.

 

 

 

 

Yuck...

 

That sounds similar to what I might be experiencing... although in my other thread I was leaning toward a memory glitch. I also did not have any issues with smartctl and reiserfsck.

 

Can I ask what method you use to compare bit for bit?

 

I use Vice Versa Pro to check/compare files, but I think it only checks the filesize and timestamp.

 

It is very easy to compare files in windows.  The command is built in...

 

Open up a command prompt and then

 

comp file1 file2

 

Joe L.

 

[pre]

C:\Users\User>comp /?

Compares the contents of two files or sets of files.

 

COMP [data1] [data2] [/D] [/A] [/L] [/N=number] [/C] [/OFF[liNE]]

 

  data1      Specifies location and name(s) of first file(s) to compare.

  data2      Specifies location and name(s) of second files to compare.

  /D        Displays differences in decimal format.

  /A        Displays differences in ASCII characters.

  /L        Displays line numbers for differences.

  /N=number  Compares only the first specified number of lines in each file.

  /C        Disregards case of ASCII letters when comparing files.

  /OFF[liNE] Do not skip files with offline attribute set.

 

To compare sets of files, use wildcards in data1 and data2 parameters.

[/pre]

However my biggest concern is that nothing was flagged in unRAID

 

Just a technicality, but it is the Linux disk controller driver that is failing to handle the data errors more visibly or effectively, not the unRAID driver.  If Linux doesn't report it, unRAID can't 'flag' it.

 

Thanks Joe L,

 

No matter how many years I put into doing this (repair, maintenance, support, etc)... It amazes me just how many little things I ::don't:: know that seem like no-brainers.

@ jakev - I read your post before I posted and had the same thought that it looked familiar. BTW I used XP command line fc - just like comp - in a batch file.

 

@robj - I take your point about linux. I'm a total linux dummy but I wonder if there's anything that can be done to address better error reporting, like a better device driver.

 

 

BTW I just found out that the peril is not silent when it comes to FLAC encoded music files. I had some on the 'peril' along with the movies and just played a few that I'd rescued . Foobar complained a few times about CRC mismatches then crashed on one track (Rinaldo Overture). Obviously Handel can't tolerate single bit errors  . Fortunately I can re-rip most of the impacted music.

 

I just mounted a 27GB '300' HD-DVD image also rescued from the peril - looks good so far.

 

more later - paul

 

 

 

 

paulvincit-

 

Please try this: on the 'Main' page, click 'Clear statistics'.  Then write your 2GB test file to the bad drive & verify that only it and the parity drive did any I/O.  Now 'Clear statistics' again and then read the test file back, and again verify: does only the 'bad' drive do any I/O?

 

Also during either the write or the read of your test file, are any entries made in the syslog?

paulvincit-

 

Please try this: on the 'Main' page, click 'Clear statistics'.  Then write your 2GB test file to the bad drive & verify that only it and the parity drive did any I/O.  Now 'Clear statistics' again and then read the test file back, and again verify: does only the 'bad' drive do any I/O?

 

Also during either the write or the read of your test file, are any entries made in the syslog?

 

I'll add the bad drive back into the array and try that later today. - paul

Update - the silent peril disk just got through a 20 hour spinrite level 4 test on another machine (not on the raid array - I wanted an independent baseline). All sectors showed good with zero read or write errors. That's excellent for a 400 GB drive. The test did show more than 2.5 million seek errors with a fairly consistent error rate. I've transfered the drive to the array and its now zeroizing. I'll try Tom's test when its done.

 

BTW - I watched most of the '300' HD-DVD image while running this morning. I did see playback hesitations and one big video glitch. So you can tell indirectly when this is happening to HD-DVD ISO images. Oh well I was going to replace it with the BLU-RAY version anyway.

 

Here's my action plan:

 

Once the 'peril' is part of the array take care of Tom's test and publish the results here.

Re-run spinrite on the drive in the array

 

paulvincit-

 

Please try this: on the 'Main' page, click 'Clear statistics'.  Then write your 2GB test file to the bad drive & verify that only it and the parity drive did any I/O.  Now 'Clear statistics' again and then read the test file back, and again verify: does only the 'bad' drive do any I/O?

 

Also during either the write or the read of your test file, are any entries made in the syslog?

 

Only the 'bad' and parity drives were active on write and only the 'bad' drive on read. Lots of single bit errors flagged by XP fc /b , nothing flagged in unRAID. Syslog attached for whole sequence including bad drive addition, clearing, formatting , and W / R test.

 

paul

 

 

 

Have you ever run a smartctl test on the drive?

 

smartctl -d ata -tshort /dev/sd? where ? = drive 

This one takes 2 mnutes

and also

smartctl -d ata -tlong /dev/sd?

this one takes 2 hours or so

 

Thereafter do

 

smartctl -d ata -a /dev/sd? > smartctl.txt

 

Please post the full smartctl.txt.

 

It will be interesting to see what the smart statistics say about all of this.

I would be surprised the smart status says the drive is healthy when you have all of those seek errors.

 

smartctl -d ata -H /dev/sda

 

Update - I ran spinrite level 4 on the unRAID platform for one hour, more than 20 GB tested no R/W errors same seek error rate as baseline. So I have to conclude the silent errors are related to a linux / Nforce 410 southbridge issue based on other comments on this forum and elsewhere. The current mobo is a Biostar Tforce 6100.

 

@weebo - I'll do the smartctl tests that you suggested then change the mobo. I'm very concerned about system integrity with the Biostar mobo at this point.

 

more later

Have you ever run a smartctl test on the drive?

 

smartctl -d ata -tshort /dev/sd? where ? = drive 

This one takes 2 mnutes

and also

smartctl -d ata -tlong /dev/sd?

this one takes 2 hours or so

 

Thereafter do

 

smartctl -d ata -a /dev/sd? > smartctl.txt

 

Please post the full smartctl.txt.

 

It will be interesting to see what the smart statistics say about all of this.

I would be surprised the smart status says the drive is healthy when you have all of those seek errors.

 

smartctl -d ata -H /dev/sda

 

 

smartctl.txt attached. I'm now switching mobos.

 

paul

 

Swapped out Biostar mobo and installed MSI K9MM-V with VIA VT3237R southbridge. 'Bad' disk still in the array as disk 12.

 

2.5 GB write / read and compare on 'bad' disk - no errors silent or otherwise.

 

I plan to remove the 'bad' disk from the array - I don't like the seek errors  - , restore parity, then recheck parity tomorrow.

 

Conclusions -

 

1. Nforce 410 / Linux don't play well together and can silently break your unRAID array.

2. Verify that important files are correctly written to the array (post cache move)

3. Do periodic array parity checks.

 

Good news - my wife says that I should get a new hard drive 

 

 

 

 

- paul

 

 

 

Good news - my wife says that I should get a new hard drive 

 

I agree!

 

Here's s'more food for thought.

http://en.wikipedia.org/wiki/S.M.A.R.T.#Known_S.M.A.R.T._attributes

 

There were quite a few drives in the array, yes? Why did you then experience the problem with this single drive only I wonder ?

Was it the only IDE drive? (the others being SATA ? ) Or was it the only one connected to the onboard/chipset IDE-controller?

 

taken from a mailing-list  regarding data corruption issues with linux and nforce4, it would have been interesting if you also could try the following. But I guess you already have switched the motherboard?:

Quote from a mailing list discussion:

 

 

 

"...Some days ago,.. an engineer of Tyan suggested me to boot the kernel

with mem=3072M.

When doing this,.. the issue did not occur.

Then he suggested me to disable the memory hole mapping in the BIOS,...

When doing so,.. the error doesn't occur, too....."

 

Mailing-list discussion available here: http://lkml.org/lkml/2006/12/1/352

 

 

I am thinking about building an unRAID-server using the parts from my current computer,  using a Epox9npa+ Ultra motherboard (nforce4 ultra)

 

But your experiences are getting me worried about the prospect of doing this. My plan was to only connect SATA-drives to the nforce4-controller, no IDE drives, if that would make any difference?

 

 

 

 

@vsonrud

 

The problem was most obvious on the 'bad' drive which was IDE. The other three IDE drives appeared to be OK in the W/R?compare tests as did all the SATA drives. However I did a parity check after I rebuilt the old system without the bad drive and got 6 errors at 20% completion so there was evidently another bit error problem source unknown - could be IDE or SATA . Bottom line I'd advise you not to use any Nforce 410 mobo's like mine, unless you have disposable data and limitless time. I've got a long list or re-rips to get through in addition to the time spent investigating the problem.

 

As I type my UnRAID system running on the MSI mobo is doing a pariity check and is at 7% with no errors.

 

To paraphrase the commercial - New mobo $60 , being able to trust unRAID priceless

 

cheers - paul

@vsonrud

 

Missed this first time  -  all the IDE drives were connected to the onboard/chipset IDE-controller, two of the SATA drives were connected to the onboard/chipset SATA controller

 

-paul

 

 

 

Are you re-using the questionable drive?

If so I have you compared the smartlogs to see if the numbers grew?

Following this thread with great interest, obviously.  Thanks for all your effort on this.

 

An interesting thing to try, if your h/w makes this easy to do, would be to write a series of files to the suspect drive with your server using the NForce-based motherboard.  Verify read back/compare against original reveals data corruptions.  Next, replace the motherboard, and then read back those same files and again (without re-writing them first) and check if compare against original still reveals data corruption (or if read errors start appearing in syslog).  This may confirm if it were a read problem or a write problem.

 

 

Following this thread with great interest, obviously.  Thanks for all your effort on this.

 

An interesting thing to try, if your h/w makes this easy to do, would be to write a series of files to the suspect drive with your server using the NForce-based motherboard.  Verify read back/compare against original reveals data corruptions.  Next, replace the motherboard, and then read back those same files and again (without re-writing them first) and check if compare against original still reveals data corruption (or if read errors start appearing in syslog).  This may confirm if it were a read problem or a write problem.

 

 

 

I kind of did that before and after I did the swap. I bit checked a file that had been written to the 'bad' drive while it was plugged into the Biostar mobo when the drive was later connected to MSI mobo - lots of errors. rewrote and re-verified - no errors. So it looks like a write problem. BTW earlier I eliminated the LAN by writing files to the share (cache) verifying them - no errors, initiating a move to the 'bad' drive and then reverifying them - multiple errors. So this can happen during mover operation also.

 

Parity check still going strong 14% no errors.

 

@WeeboTech  I'm not using the bad drive right now but based on my observations during the spinrite test it was accumulating seek errors at a rate of about 1 in 10^3 accesses. I get the impression that they might have messed with the platters when this unit was refurbished and that is creating the problem. I'll probably use this drive for the torrents I'm seeding in one of my XP PCs and keep an eye on it. I'll let you know if anything interesting happens.

 

- paul

 

 

paul - again, great analysis.  How much main memory is in your server?