March 5, 201115 yr The parity check feature has become an invaluable tool for users to maintain the health of their drives and to find and correct drive problems. However, there are times when a parity check can do more harm than good. For example, if a data drive starts acting up, it can return bad data that will pollute parity and make drive construction imperfect. This happened to me and was a PITA. Memory errors can also cause apparent sync errors and pollute parity. The read-only parity check is helpful in these cases. But knowing in advance that you are going to have such problems is not always possible. And if errors are indeed found, figuring out if they are transient or real, and getting them fixed, is difficult and extremely time consuming. Users that have issues with sync errors have the fewest tools to figure out what is going on, and are forced to run many many parity checks to try to diagnose and correct. A simple question like which drive is causing the problem is unanswerable. This can be the most frustrating experience! Another issue with the parity check is there is no way to put it on hold (to watch a movie, for example). Although not a big deal to schedule a parity check during an overnight period, as drives get bigger and especially if there is some problem and you are running checks one after another, the parity check durations can become extended and interfere with prime media times. Also, sometimes a reboot is needed for some reason, so an ability to resume a parity check across reboots would be helpful. Here is a suggestion to revamp the venerable parity check … 1. Have the parity check NOT update parity. Instead, have unRAID log parity errors to a sync error file (on flash or cache or array disk) when they occur. 2. When each sync error is detected, have unRAID calculate a CRC (or other calculation) that would make it possible to discern what value changed if that block were checked again later and the values from one disk changed. Store this in the sync error file as well. (More on the reason for this later). 3. If some maximum number of parity errors occurs (e.g., 2000), parity check can stop and prompt the user (on a browser refresh) to complete the parity check in real-time correction mode (current functionality), or stop the parity check (and go to the options in #5 below). 4. Implement pause functionality, which would write a file to flash with resume information. Later, user would have an option to resume or cancel the partially completed parity check (even after a reboot). 5. When a parity check finishes, is canceled, or is stopped, and there were sync errors found, provide a series of options. These options should exist across reboots (would allow a user to fix a hardware issue, for example, and quickly see if they fixed a problem): a. Recheck – Recheck the blocks where sync errors were found. With the CRC value, if a recheck results in a different value, the parity check could identify which value changed pointing the finger towards a problematic drive. b. Correct – Cause unRAID to correct the sync errors found in the parity check. After the correction, a recheck would be performed to confirm that it was corrected. c. Ignore – Cause unRAID to leave the sync errors in place. (This would be selected if the user determined that parity was more accurate than one of the data disks, and that the data disk should be rebuilt using current parity.) Thoughts?
March 5, 201115 yr Brilliant, I support all of these changes. At very least make the parity check NOCORRECT the default option.
March 5, 201115 yr That sounds like a good plan, I would also like an option to enter a range of blocks to be checked, this would have been useful when I was trying to track down a problem (which turned out to be a single drive) where most of the errors were occurring in about a 100GB band, then I could have cut my testing time (once I understood this) by only having to test 100GB x 8 drives instead of 2TB x 8 drives. The other tool I would really like to see is one that can take the block number where an error occurred and give me a list of all the files that might be affected. Regards, Stephen
March 6, 201115 yr I never ever want to disagree with Brian, (or Rajahal,) and I do apologize to him, but I am very uncomfortable with the basic premise underlying his suggestions, that a drive can return differing data from the same sector. We went through this issue quite awhile back, and I was very unhappy at the time with some of the conjecture as to what was really happening, but I never got around to fully analyzing it, and did not feel confident enough in my thoughts to present them. For the same reason, I'm not a fan of the NOCORRECT option, but I still have not thought it through sufficiently to be sure I haven't missed something important, and I'm certainly not an expert here. I probably *have* missed some of the logic development behind the current thinking of others here. So I'll try to present my incomplete thoughts, and hope that more knowledgeable users will correct me. In particular, I would really like to have BubbaQ, if he has time, weigh in on this issue, as he is a drive expert, and this is obviously an issue of some importance, since it relates to data integrity. For me, the truism is that IF a drive returns data, then that data IS correct, always, consistently. It cannot return different data at different times, without an intervening write. Sector data is associated with a CRC, and if it is not correct for the data read, then none of the data of that sector is returned. It is an all or nothing proposition, either all of the good data is returned, or none of it and an error code is returned instead. The underlying process is more complicated, but not different from that. As I understand it, if the saved CRC does not agree with the CRC of the data, then the error correction info is used to reconstruct the correct data, then checked again against the saved CRC. If still wrong, then a series of retries occurs, until it is either correctly read or it gives up and reports an error. The drive never tries to 'help' you by returning some of the data, or a best estimate of the data, or even just the part it is confident in. It returns the entire sector data if it agrees with the CRC calculation, or it returns none of it. If you accept that this is true, and apply it to the parity check process, then if all drives return data, and the calculated parity indicates an error, then the parity disk is wrong, and should be rewritten. Which is why the NOCORRECT option has not seemed useful to me. Here, I do admit probable ignorance, I am probably not aware of situations that other users have had, where delaying the correction of parity was reasonable, or useful. I won't be surprised to be corrected on that. Now Brian's suggestions and premise are based, quite reasonably, on real situations that do make it easy to believe that drives are, in certain circumstances, returning inconsistent data. Brian has had one (or more) drives, and several other users have had inconsistent parity checks, with multiple and repeating and differing parity errors, and recently one obviously knowledgeable user ran a series of MD5 tests to isolate a bad drive. I really don't understand how these are possible, and I wish I did. But I cannot throw out what I believe to be true, in order to explain them. I know I have not been through all Brian has been through though. I really hope other knowledgeable users can provide answers or ideas. I suspect each of these cases needed more exhaustive analysis, might have helped us better understand what is going on. Perhaps the disk controllers may have been to blame? Or the associated drivers/modules? I don't think the bad memory angle is at all likely, it seems to me to be such an extremely rare possibility, to corrupt buffer data without causing other obvious problems with system operation. There is one thing that troubles me, and I hope Tom will consider a change in the parity process, to return an additional type of parity error. Currently, if the parity calculation returns an incorrect parity, then that is reported. But what about the case when one (or more) drives could not return their data for a given block? If the above is true, and either good data is returned or no data is returned, then parity CANNOT be calculated for that block, and an appropriate error should really be reported, something like "read error, parity can not be calculated", not "parity incorrect". The parity drive should not be written for this case. But perhaps I'm forgetting something fundamental... I am in favor of some of the other ideas, such as being able to pause and resume a parity check (I could have used that several times), and having the option to run a parity check on a specified block range.
March 6, 201115 yr Author ... RobJ - you and I see most issues from the same side, and I respect your position on this one immensely. And I agree that drives are designed to NOT return bad data. But saying that a drive can't fail in a way that would return bad data is a tough thing for me to accept, especially given the firmware issues we've seen in recent years. I can say with certainty that symptoms DO make it look like they do, at times. But even accepting the premise, that drives won't return bad data, there are still the possibiliteis of noise altering the signals, bad cables, lose ports, drives dropping offline due to power or other issues, controllers going bad, bad memory, unRAID bugs caused by unforseen events, and other things in the general "shit happens" category that cause sync errors to occur. We've seen users that have exactly the same sync errors occur as the values flip flop back and forth between two values on subsequent checks. With no insight into what the parity error cause, and no ability to easily determine if they repeat or are transient, and no ability to isolate the problem to a certain disk, we really have no tools to evaluate your hypothesis. So even if drives are never returning bad data, the symptoms can certainly make it appear that they are. And my recommendations are all about helping a user isolate the problem to a drive. It could be that a user would find sync errors are occuring only on drives that are on one contoller, and that retries validate that parity is, in fact, right. That would help point a user towards a bad controller. Or it could be that sync errors are transient on only one drive, but when that drive is switched with another drive on a different port, the transient errors move to the switched disk, it might mean a bad cable or lose port. Right now we have no ways to narrow things down and do any real problem determination. It is pure guess and check at a macro level. Is it possible for a drive return bad data when failing? These tools may actually help answer that question!
March 6, 201115 yr Over the past few years I've helped at least three or four people who tracked repeated parity errors to a disk drive that returned data with an occasional bit flipped. No error were logged. No CRC errors were generated. The only indication of anything wrong was just found in repeated parity checks. Those servers would occasionally show parity errors. It was finally isolated by running multiple MD5 checksums on the same files over and over on various disks. Occasionally, one disk would return a different checksum, but NO OTHER error. I agree, it should never happen, but those disks cause hair loss. (You will pull your hair out trying to figure it out)
March 6, 201115 yr Over the past few years I've helped at least three or four people who tracked repeated parity errors to a disk drive that returned data with an occasional bit flipped. No error were logged. No CRC errors were generated. The only indication of anything wrong was just found in repeated parity checks. Those servers would occasionally show parity errors. It was finally isolated by running multiple MD5 checksums on the same files over and over on various disks. Occasionally, one disk would return a different checksum, but NO OTHER error. I agree, it should never happen, but those disks cause hair loss. (You will pull your hair out trying to figure it out) This happened to me last year. In order to find the cause of these errors I: 1. replaced all the SATA cables (twice) and replaced the power splitters 2. replaced two out of my 8 drives (these were the only drives that showed any errors on the SMART tests) 3. ran short SMART tests many times 4. ran long SMART tests at least twice on each drive 5. ran MANY parity checks (roughly 70 over about 6 months) 6. tested the memory on a few occasions 7. replaced the SIL 2 port SATA cards with a Supermicro 8 port card (at this point got SATA breakout cables, so another cable swap) 8. moved all the drives off the motherboard to the 8 port card 9. replaced the CPU/RAM and motherboard (switched from Intel to AMD) 10. because of the addition to unRAID 4.5.5 where it finally reported the parity block location I was able to observe that the checks were picking up some of the same blocks 11. noticed that the "bad blocks" were in a range of roughly 100G out of the 2TB 12. started running md5sums on the 100G range 13. the md5sums quickly (within a day of testing) identified the drive that was causing the problem This drive was RETURNING DIFFERENT DATA even though nothing was being written to it. I replace the drive with a new one (keeping the same SATA cable, port and power connector) and rebuilt. After that I have not had a single problem in over 20 parity checks. I put the bad drive on another SATA port but did not make it part of the array, and I ran the md5sum on it and found it still had the same problems. So then I did a long SMART on it and did another round of md5sums and the problems still repeated. Then I ran preclear on the drive and all the problems stopped. I even copied (using dd) the data from the array back on to the suspect drive and repeated the md5 tests and still the drive behaved fine. All this time there was almost no indication of any problem from that drive using the SMART reports. At the most I got an occasional non-zero value in the raw count of: Multi_Zone_Error_Rate, UDMA_CRC_Error_Count or Raw_Read_Error_Rate but these were not well correlated with actual parity check errors (they would sometimes show up when there had been a parity check error and sometimes they would show after a successful parity check). Through this time I was saving the SMART reports of all the drives after each parity check and comparing them (kdiff3) to the previous (or earlier) reports to see if I could see any issues. This is why I replace two other drives along the way. I now know that it is possible for a drive to return bad data without any error indication, perhaps this is not a general issue, perhaps it is a firmware bug on one particular model of drive. But I doubt it will be the last time this ever happens. I still have all the SMART reports and notes on what I tried and when if someone is really interested in this stuff. I have attached two files: parity.txt contains my notes on this (from Jul 19, 2010 - which is about the time I updated to 4.5.5 and noticed that the parity error block number had been added to the syslog) to the present time. parity.zip contains parity.xls which is a small spreadsheet I put together, to try and look for a pattern in the errors that were occuring, I had noticed that some blocks were getting repeated errors in later parity checks and wanted to see if there was some correlation I could spot. The main thing that came out of this was the realization that most parity errors were occurring in about a 5% band within the full array, so I might be able to concentrate on testing that area of each drive. This is what I did using the md5sum tool, starting at Nov 30 in the parity.txt file. Regards, Stephen parity.txt parity.zip
March 7, 201115 yr It cannot return different data at different times, without an intervening write. I have personal experience that this is not true. This type of error has also been documented repeatedly in forensic environments, which is why forensic imaging applications like FTK, EnCase, and dc3dd, zero out the entire sector when it contains any error, because weak reads will make subsequent hash verification fail. It can and does happen.
March 7, 201115 yr It cannot return different data at different times, without an intervening write. I have personal experience that this is not true. This type of error has also been documented repeatedly in forensic environments, which is why forensic imaging applications like FTK, EnCase, and dc3dd, zero out the entire sector when it contains any error, because weak reads will make subsequent hash verification fail. It can and does happen. Wow. My faith in technology has completely crashed. The implications here are ... wow! Ah ... but wait (he thinks he has seen the flaw in that statement!) ... In special environments like yours, could your tools perhaps be making only low level raw reads, which bypass the higher level error-correcting mechanisms? I fully expect that low level raw reads can return inconsistent data from weak or marginal media, that is why there are CRC's and ECC info. Our reads are only at the CRC-checked and error-corrected level. How can it return inconsistent data without an error detected?
March 7, 201115 yr Sounds like I'm taking the part of the professorial type, arguing the theory, and saying "Nope it cannot happen", and Joe and Brian and vca and BubbaQ and others are the people in the field saying "screw the theory, it's happening!". How to solve this? We can agree that it sure looks like it is, but I really wish someone could come up with a proposed mechanism that could explain *how* it can happen, a way to modify the theory to better explain the reality. In science, this is a great moment, when anomalies are found that don't fit the current theories, and lead to great breakthroughs. We are not looking for something too great, but an explanation that leads to a practical workaround for this issue. Brian's ideas could really help us here. VCA though has really helped already, with his notes. VCA, that was really great work! I haven't had much time yet, so there is probably more to be found in it, but already 2 conclusions from your work have jumped out. The first step was to convert a sampling of your sector LBA's into hex, and look for patterns, and look what is immediately obvious! Jul 19 00:50:35 saturn kernel: md: parity incorrect: 1485297976 (first) 5887 D938 Jul 19 00:57:47 saturn kernel: md: parity incorrect: 1536676720 (only) 5B97 D370 Jul 19 01:12:25 saturn kernel: md: parity incorrect: 1636175680 (first) 6186 0F40 Jul 23 00:34:04 saturn kernel: md: parity incorrect: 1472497824 (first) 57C4 88A0 Jul 23 00:39:36 saturn kernel: md: parity incorrect: 1512072552 (first) 5A20 6568 Jul 25 00:56:25 saturn kernel: md: parity incorrect: 1477474488 (first) 5810 78B8 Jul 25 01:00:59 saturn kernel: md: parity incorrect: 1510045800 (first) 5A01 7868 Jul 25 01:01:04 saturn kernel: md: parity incorrect: 1510600792 (first/4) 5A09 F058 Aug 4 00:21:29 saturn kernel: md: parity incorrect: 1473687856 (first) 57D6 B130 Aug 4 00:25:11 saturn kernel: md: parity incorrect: 1500156176 (only) 596A 9110 Aug 4 09:43:12 saturn kernel: md: parity incorrect: 1538520440 (first) 5BB3 F578 Aug 5 00:21:20 saturn kernel: md: parity incorrect: 1510461320 (first) 5A07 CF88 Aug 6 00:54:53 saturn kernel: md: parity incorrect: 1509379344 (only) 59F7 4D10 Aug 7 00:18:13 saturn kernel: md: parity incorrect: 1568758360 (only) 5D81 5A58 Aug 10 21:54:41 saturn kernel: md: parity incorrect: 1510171176 (first) 5A03 6228 Aug 12 00:17:14 saturn kernel: md: parity incorrect: 1510045792 (first) 5A01 7860 Aug 20 23:45:37 saturn kernel: md: parity incorrect: 1471919504 (only) 57BB B590 Aug 20 23:59:14 saturn kernel: md: parity incorrect: 1567937656 (first) 5D74 D478 Aug 28 00:25:11 saturn kernel: md: parity incorrect: 1669071816 (only) 637C 03C8 Sep 5 20:02:07 saturn kernel: md: parity incorrect: 1568501552 (only) 5D7D 6F30 The first conclusion is that the faulty sector is always a multiple of 8, in a pattern that begins with zero. In other words, the unit of transfer appears to be 4K blocks (8 sectors), and the problem always occurs in the first sector of that block. That leads to the second obvious but useful conclusion, that the problem is NOT random! That rules out any cause like noise that is non-aligned or is random in nature. That is useful, helps narrow the culprits. I would have wanted to disallow noise and cabling anyway, because any modern data communications path is generally CRC or parity checked. That is why you see the BadCRC or ICRC flags raised in drive exceptions. A faulty transmission caused by noise or a bad cable causes a CRC error, resulting in a retry. We still need Brian's tool to tell us exactly which byte of that faulty sector, and specifically which bit of that byte. I strongly suspect that in vca's case, they won't be random either. You know, someone could write a tool that would do much of what the parity check is doing, with Brian's suggested improvements. Specify an LBA range to be tested and logged, and it could relatively quickly isolate which block is inconsistent, then report exactly which byte and which bit of that byte is flipping (as well as which drive!). Determining which file is involved is a different issue, but would be a nice enhancement. Now to see what else we can discover ...
March 7, 201115 yr We can agree that it sure looks like it is, but I really wish someone could come up with a proposed mechanism that could explain *how* it can happen, a way to modify the theory to better explain the reality. Here's my theory: Computers are complicated and prone to technical difficulties. They are made by humans and thus prone to errors. For a system reading from a hard drive it can involve all sorts of various issues such as the drive's firmware being buggy (Samsung), the electrical supply system meaning differences in reference level voltages for what is a 0 vs what is a 1 (some drives pickier about PSUs than others), media warping due to typical wear and tear, or electrical substrate going bad from excessive temp cycles (Nvidia GPU's BumpGate and MS's XBox360 RROD), bad implementation of base functions (Nvidia nForce systems), electron migration at the lowest level (Intel SATA Chipset for SandyBridge), bad cables producing erratic data, leaky capacitors from bad electrolytic formula or from excessive temps (bad motherboards, bad PSUs), incorrect memory setups or mistuned memory settings causing data corruption on the client side (subtle memtest errors), and the list can go on and on.
March 7, 201115 yr We can agree that it sure looks like it is, but I really wish someone could come up with a proposed mechanism that could explain *how* it can happen, a way to modify the theory to better explain the reality. Here's my theory: Computers are complicated and prone to technical difficulties. They are made by humans and thus prone to errors. For a system reading from a hard drive it can involve all sorts of various issues such as the drive's firmware being buggy (Samsung), the electrical supply system meaning differences in reference level voltages for what is a 0 vs what is a 1 (some drives pickier about PSUs than others), media warping due to typical wear and tear, or electrical substrate going bad from excessive temp cycles (Nvidia GPU's BumpGate and MS's XBox360 RROD), bad implementation of base functions (Nvidia nForce systems), electron migration at the lowest level (Intel SATA Chipset for SandyBridge), bad cables producing erratic data, leaky capacitors from bad electrolytic formula or from excessive temps (bad motherboards, bad PSUs), incorrect memory setups or mistuned memory settings causing data corruption on the client side (subtle memtest errors), and the list can go on and on. You are correct, and I can completely agree that there can be faults and failures and off-nominal states at every level. BUT (oh boy here we go), the problem here is not that there can be many sources of error (and you gave some great examples), but that some of them could pass through uncaught. Devices and systems are first engineered and extensively tested for their tolerances, so that their specs are known quantities, then the appropriate checks are added, tests and sensors and traps and CRC values, etc, to trap each out-of-nominal result and act accordingly. In our case, I have no problem accepting any number of errors from any number of causes happening, but how can they pass that last CRC test correctly? Certainly if a lightning strike hit a drive, it is possible that the data in a sector could be rearranged so correctly that it still matched its CRC value, but that would be a once-in-a-lifetime event. What we are seeing here, are cases where the CRC test passes very regularly, and yet the data was changed! How can that be possible? I suspect that some are losing interest in this discussion, would rather think about solutions, so I'll try not to press it any further. I'm unfortunately one of those that likes order in the universe, reasons for everything, but I'll quickly be distracted by something else, and move on. I'm not happy though at simply leaving it at "that's just the way it is, computers are fallible".
March 7, 201115 yr Sounds like I'm taking the part of the professorial type, arguing the theory, and saying "Nope it cannot happen", and Joe and Brian and vca and BubbaQ and others are the people in the field saying "screw the theory, it's happening!". How to solve this? ... The first conclusion is that the faulty sector is always a multiple of 8, in a pattern that begins with zero. In other words, the unit of transfer appears to be 4K blocks (8 sectors), and the problem always occurs in the first sector of that block. ... An interesting observation, but might the 4K block transfers be caused by the way Linux is doing its file IO? When I was watching this happen and I started to notice that the errors were within a particular range of blocks I started to wonder if a drive was having problems with one of its heads. The faulty drive was a WesternDigital WD15EADS From the label it was manufactured 21 Oct 2009, so I wouldn't be surprised if I actually bought it in Nov or Dec 2009. The first SMART report that I have for this drive is here: Statistics for /dev/sdf WDC_WD15EADS-00P8B0_WD-WCAVU0359932 smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: WDC WD15EADS-00P8B0 Serial Number: WD-WCAVU0359932 Firmware Version: 01.00A01 User Capacity: 1,500,301,910,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Mon Mar 15 08:23:21 2010 MDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 249) Self-test routine in progress... 90% of test remaining. Total time to complete Offline data collection: (33000) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3037) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 183 183 021 Pre-fail Always - 5833 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 309 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 097 097 000 Old_age Always - 2345 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 13 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 7 193 Load_Cycle_Count 0x0032 199 199 000 Old_age Always - 5667 194 Temperature_Celsius 0x0022 124 116 000 Old_age Always - 26 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. The last SMART report I have for the drive was taken after I had done the long SMART test and then precleared the drive, this is it: SMART status Info for /dev/sdj smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: WDC WD15EADS-00P8B0 Serial Number: WD-WCAVU0359932 Firmware Version: 01.00A01 User Capacity: 1,500,301,910,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Tue Dec 7 17:59:53 2010 MST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (33000) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3037) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 182 181 021 Pre-fail Always - 5891 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 438 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 089 089 000 Old_age Always - 8596 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 34 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 8 193 Load_Cycle_Count 0x0032 186 186 000 Old_age Always - 43755 194 Temperature_Celsius 0x0022 122 115 000 Old_age Always - 28 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 8559 - # 2 Short offline Completed without error 00% 6661 - # 3 Short offline Completed without error 00% 6137 - # 4 Short offline Completed without error 00% 5674 - # 5 Short offline Completed without error 00% 3418 - # 6 Extended offline Completed without error 00% 3041 - # 7 Short offline Completed without error 00% 3011 - # 8 Short offline Completed without error 00% 2989 - # 9 Short offline Completed without error 00% 2773 - #10 Short offline Completed without error 00% 2761 - #11 Short offline Completed without error 00% 2664 - #12 Short offline Completed without error 00% 2629 - #13 Short offline Completed without error 00% 2607 - #14 Short offline Completed without error 00% 2558 - #15 Short offline Completed without error 00% 2510 - #16 Short offline Completed without error 00% 2487 - #17 Short offline Completed without error 00% 2463 - #18 Short offline Completed without error 00% 2440 - #19 Short offline Completed without error 00% 2415 - #20 Short offline Completed without error 00% 2368 - #21 Short offline Completed without error 00% 2358 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Regards Stephen
March 7, 201115 yr Author You are correct, and I can completely agree that there can be faults and failures and off-nominal states at every level. BUT (oh boy here we go), the problem here is not that there can be many sources of error (and you gave some great examples), but that some of them could pass through uncaught. Devices and systems are first engineered and extensively tested for their tolerances, so that their specs are known quantities, then the appropriate checks are added, tests and sensors and traps and CRC values, etc, to trap each out-of-nominal result and act accordingly. In our case, I have no problem accepting any number of errors from any number of causes happening, but how can they pass that last CRC test correctly? Certainly if a lightning strike hit a drive, it is possible that the data in a sector could be rearranged so correctly that it still matched its CRC value, but that would be a once-in-a-lifetime event. What we are seeing here, are cases where the CRC test passes very regularly, and yet the data was changed! How can that be possible? I suspect that some are losing interest in this discussion, would rather think about solutions, so I'll try not to press it any further. I'm unfortunately one of those that likes order in the universe, reasons for everything, but I'll quickly be distracted by something else, and move on. I'm not happy though at simply leaving it at "that's just the way it is, computers are fallible". Never say never. Murphy is just too damn smart.
March 7, 201115 yr We can agree that it sure looks like it is, but I really wish someone could come up with a proposed mechanism that could explain *how* it can happen, a way to modify the theory to better explain the reality. Here's my theory: Computers are complicated and prone to technical difficulties. They are made by humans and thus prone to errors. ... ... In our case, I have no problem accepting any number of errors from any number of causes happening, but how can they pass that last CRC test correctly? Certainly if a lightning strike hit a drive, it is possible that the data in a sector could be rearranged so correctly that it still matched its CRC value, but that would be a once-in-a-lifetime event. What we are seeing here, are cases where the CRC test passes very regularly, and yet the data was changed! How can that be possible? ... In my case it appeared that the data was correctly stored on the drive, but 7 out of 10 reads returned different values, you can see this because when I did the md5sum checks there was one particular MD5 digest that kept coming back ( 73c0b467088be422c9999f415e81adab ): fe1804307062eeb93261b18bb63036bf *- 740bc5c5ef4eab169f627d5c2ae45dfa *- 73c0b467088be422c9999f415e81adab *- 73c0b467088be422c9999f415e81adab *- 73c0b467088be422c9999f415e81adab *- 600cb1e4adc284ca913c5ef4d83b9d20 *- 8978f191ba809134207e5f385997aac1 *- c7a4ed6c9c4dd1b71794ce5d94644e72 *- 3cb91ce1e320d7de8e0e302a9d70d2f9 *- 93c82b4ff2b98172ffe9a48266a43361 *- and it looks like the other values that got computed on different runs were never repeated - this is why I concluded that in the 73c0b467088be422c9999f415e81adab case the data was probably being correctly read. Even if it was not the correct value it was the "most likely" value. Of course with an MD5 hash we still have no way of knowing if the hash changed because of a single bit error or many bits were in error. About all we can say is that: 1. at least one bit changed 2. probably no more than one block was incorrect (though there were parity checks where I observed several blocks in error, so this is not a solid conclusion) and if only one bit changed, then it was not the same bit each time, because if it was then there would only be two values for the hash. Regards, Stephen
March 7, 201115 yr An interesting observation, but might the 4K block transfers be caused by the way Linux is doing its file IO? That is what is great about finding a pattern, it indicated a specific number that helps to greatly narrow the list of suspects down, to only those associated with this number. We know now that we can concentrate on only those devices and processes that would involve 4K, such as the one you mentioned above. That particular generic transfer process is probably off the hook, because you would have to come up with a way to explain why these 4K transfers are not faulty for other drives and devices too. When I was watching this happen and I started to notice that the errors were within a particular range of blocks I started to wonder if a drive was having problems with one of its heads. The faulty drive was a WesternDigital WD15EADS Those SMART reports look very good, and I don't see anything even slightly concerning. This drive is in excellent condition, at least according to the SMART reporting. The fact that your errors occur within a narrow band is very interesting, but I can't see a connection with a single head, since all heads are used across the entire drive. I can't think of any reason at all why the errors should be confined to only that range, at about a third of the way into the drive. The only thing I can think of is a firmware bug, perhaps a buffer overrun, or bad pointer. You can possibly eliminate general Linux kernel processes, by testing with very different kernels, and see if the error behavior changes at all.
March 7, 201115 yr In my case it appeared that the data was correctly stored on the drive, but 7 out of 10 reads returned different values, you can see this because when I did the md5sum checks there was one particular MD5 digest that kept coming back ( 73c0b467088be422c9999f415e81adab ): fe1804307062eeb93261b18bb63036bf *- 740bc5c5ef4eab169f627d5c2ae45dfa *- 73c0b467088be422c9999f415e81adab *- 73c0b467088be422c9999f415e81adab *- 73c0b467088be422c9999f415e81adab *- 600cb1e4adc284ca913c5ef4d83b9d20 *- 8978f191ba809134207e5f385997aac1 *- c7a4ed6c9c4dd1b71794ce5d94644e72 *- 3cb91ce1e320d7de8e0e302a9d70d2f9 *- 93c82b4ff2b98172ffe9a48266a43361 *- and it looks like the other values that got computed on different runs were never repeated - this is why I concluded that in the 73c0b467088be422c9999f415e81adab case the data was probably being correctly read. Even if it was not the correct value it was the "most likely" value. Of course with an MD5 hash we still have no way of knowing if the hash changed because of a single bit error or many bits were in error. About all we can say is that: 1. at least one bit changed 2. probably no more than one block was incorrect (though there were parity checks where I observed several blocks in error, so this is not a solid conclusion) and if only one bit changed, then it was not the same bit each time, because if it was then there would only be two values for the hash. I'm getting tired, and I think I work tomorrow, so I'll be out of touch for awhile, but your reasoning seems very sound to me. I'll get back to this when I can. You've provided a lot of good info to think about.
March 7, 201115 yr Sounds like I'm taking the part of the professorial type, arguing the theory, and saying "Nope it cannot happen", and Joe and Brian and vca and BubbaQ and others are the people in the field saying "screw the theory, it's happening!". How to solve this? ... The first conclusion is that the faulty sector is always a multiple of 8, in a pattern that begins with zero. In other words, the unit of transfer appears to be 4K blocks (8 sectors), and the problem always occurs in the first sector of that block. ... An interesting observation, but might the 4K block transfers be caused by the way Linux is doing its file IO? When I was watching this happen and I started to notice that the errors were within a particular range of blocks I started to wonder if a drive was having problems with one of its heads. The problem could easily be in the "cache" ram in the disk itself, or the circuitry reading it and sending its contents down the SATA cable to the disk controller. When the disk contents are read from the platters and checked against the affiliated CRC check for the sector all would be fine. Those contents are probably THEN stored in the internal cache memory on the disk. Later, when sent across the SATA cable I'll bet a new checksum is calculated for the actual transfer across the cable. (it is highly unlikely to use the same size blocks as the reads from the physical disk sectors, so the CRC checksum needed across the SATA link would be different) In between all you need is some flaky internal RAM, or RAM that is sensitive to a noisy power supply rail, or, sensitive to operations in certain address ranges. That could explain why the errors fail with specific patterns. If they blindly trust the data written to the ram is identical to that read back out of the ram, the errors could creep in. These types of errors would be rare, and very difficult to detect. Joe L.
March 7, 201115 yr Author When the disk contents are read from the platters and checked against the affiliated CRC check for the sector all would be fine. Those contents are probably THEN stored in the internal cache memory on the disk. Later, when sent across the SATA cable I'll bet a new checksum is calculated for the actual transfer across the cable. (it is highly unlikely to use the same size blocks as the reads from the physical disk sectors, so the CRC checksum needed across the SATA link would be different) In between all you need is some flaky internal RAM, or RAM that is sensitive to a noisy power supply rail, or, sensitive to operations in certain address ranges. That could explain why the errors fail with specific patterns. If they blindly trust the data written to the ram is identical to that read back out of the ram, the errors could creep in. These types of errors would be rare, and very difficult to detect. Joe L. But if the controller RAM was bad, it would tend to speckle errors over the parity blocks - and not yield the same parity blocks being flagged over and over and the 100G band that vca found to be causing all of the problems for him. Remarkably, we have not seen that kind of issue in actual user incidents. If we started to see them, such incidents could be anything - from disk to cable to controller to ram to motherboard. The kind of almost repeatable parity error results we've seen are much more likely to be confined to the disk itself.
March 7, 201115 yr I have no problem accepting any number of errors from any number of causes happening, but how can they pass that last CRC test correctly? One way this happens is that the data doesn't necessarily pass the CRC... the CRC fails, the drive returns the data it got, and the ERROR the drive returned got lost, so the data is accepted.
March 7, 201115 yr When the disk contents are read from the platters and checked against the affiliated CRC check for the sector all would be fine. Those contents are probably THEN stored in the internal cache memory on the disk. Later, when sent across the SATA cable I'll bet a new checksum is calculated for the actual transfer across the cable. (it is highly unlikely to use the same size blocks as the reads from the physical disk sectors, so the CRC checksum needed across the SATA link would be different) In between all you need is some flaky internal RAM, or RAM that is sensitive to a noisy power supply rail, or, sensitive to operations in certain address ranges. That could explain why the errors fail with specific patterns. If they blindly trust the data written to the ram is identical to that read back out of the ram, the errors could creep in. These types of errors would be rare, and very difficult to detect. Joe L. But if the controller RAM was bad, it would tend to speckle errors over the parity blocks - and not yield the same parity blocks being flagged over and over and the 100G band that vca found to be causing all of the problems for him. Remarkably, we have not seen that kind of issue in actual user incidents. If we started to see them, such incidents could be anything - from disk to cable to controller to ram to motherboard. The kind of almost repeatable parity error results we've seen are much more likely to be confined to the disk itself. I was referring to the small amount of cache RAM internal to the disk, not memory on the disk controller card.
March 7, 201115 yr Author I was referring to the small amount of cache RAM internal to the disk, not memory on the disk controller card. I meant drive, not controller, cache - but either way it doesn't matter. If the RAM is bad, the same parity blocks wouldn't repeat as vca has seen.
March 7, 201115 yr I was referring to the small amount of cache RAM internal to the disk, not memory on the disk controller card. I meant drive, not controller, cache - but either way it doesn't matter. If the RAM is bad, the same parity blocks wouldn't repeat as vca has seen. I understand, but a bit of silicon that reacts to a range of addresses, or when certain bits are set can still cause all kinds of havoc if they act up under whatever conditions the drive is subjected to. It might be power supply related, or decoupling on the circuit board, or, who knows what else... I only know it happens, and if it seems to happen with specific bit patterns, or ranges of address, or stored values only lets us guess the actual cause. The fix is easy. You must first test it as a wheel-chock. Place it behind the wheel of your car and test its ability to impede the forward and reverse motions of your car. Once you find it has flattened enough to not effectively stop the car, you can then see if it passes a SMART test. Usually, the intermittent failures will have stopped. (replaced by permanent failures across broad address ranges) ;) ;)
March 7, 201115 yr I was referring to the small amount of cache RAM internal to the disk, not memory on the disk controller card. I meant drive, not controller, cache - but either way it doesn't matter. If the RAM is bad, the same parity blocks wouldn't repeat as vca has seen. Yes, that's what I was thinking, also the errors should have been more frequent (and more widely distributed) as the same bad RAM (wherever it is, drive, controller or main computer) would tend to get reused many times in one parity check. The thing that points more to the actual disk surface is that once I had zeroed the bad drive and used dd to copy the original data back onto the drive, the issue stopped. This suggests that some blocks had not been written properly and that the drive was having problems reading them sometimes, it was correcting (correctly) what it had read some of the time and other times the correction failed and if it reported an error that was lost. Stephen
Archived
This topic is now archived and is closed to further replies.