March 15, 201016 yr These are all Seagate 1.5tb (ST31500341AS) drives and all are showing the same types of readings. I know the Relocated Sector count is not good, but it has not changed for any of these drives in the last few weeks I've been checking. I'm concerned over some of these other numbers and don't know if they mean I've got a problem: Raw read error rate, Seek error rate, Hardware ECC recovered, Hight fly writes, Head flying hours. The first drive is already part of my array but he second and third are not yet. Any advice? Thank you First drive: SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 118 099 006 Pre-fail Always - 190583669 3 Spin_Up_Time 0x0003 092 092 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 191 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 6 7 Seek_Error_Rate 0x000f 075 060 030 Pre-fail Always - 36361751 9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 3593 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 160 184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 095 000 Old_age Always - 47245426701 189 High_Fly_Writes 0x003a 083 083 000 Old_age Always - 17 190 Airflow_Temperature_Cel 0x0022 075 041 045 Old_age Always In_the_past 25 (15 15 34 23) 194 Temperature_Celsius 0x0022 025 059 000 Old_age Always - 25 (0 20 0 0) 195 Hardware_ECC_Recovered 0x001a 042 026 000 Old_age Always - 190583669 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 73718818672093 241 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 3011643887 242 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 608984173 Second drive: SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 114 100 006 Pre-fail Always - 63697031 3 Spin_Up_Time 0x0003 100 100 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 139 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 9 7 Seek_Error_Rate 0x000f 069 060 030 Pre-fail Always - 9306042 9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 2126 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 139 184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 089 089 000 Old_age Always - 11 190 Airflow_Temperature_Cel 0x0022 068 050 045 Old_age Always - 32 (Lifetime Min/Max 29/34) 194 Temperature_Celsius 0x0022 032 050 000 Old_age Always - 32 (0 18 0 0) 195 Hardware_ECC_Recovered 0x001a 039 018 000 Old_age Always - 63697031 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 21758304323661 241 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 2472658178 242 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 3125586931 Third drive: SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 117 099 006 Pre-fail Always - 145907528 3 Spin_Up_Time 0x0003 100 100 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 150 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 3 7 Seek_Error_Rate 0x000f 072 060 030 Pre-fail Always - 17674425 9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 3817 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 150 184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 099 000 Old_age Always - 4295032833 189 High_Fly_Writes 0x003a 048 048 000 Old_age Always - 52 190 Airflow_Temperature_Cel 0x0022 067 043 045 Old_age Always In_the_past 33 (0 200 34 29) 194 Temperature_Celsius 0x0022 033 057 000 Old_age Always - 33 (0 19 0 0) 195 Hardware_ECC_Recovered 0x001a 054 024 000 Old_age Always - 145907528 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 48348446854887 241 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 439874905 242 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 3119734451
March 15, 201016 yr IIRC, Seagates return bogus numbers for some SMART parameters, like Raw_Read_Error_Rate.
March 15, 201016 yr I would say that it is unusual to have 3 drives each with reallocated sectors. It is not enough to monitor these numbers - you need to run parity checks and then monitor the numbers after each parity check. Below is a cut-and-paste of something I wrote in another thread about a user reporting 2 reallocated sectors that might be of interest ... A few or even a few hundred reallocated sectors is not really a problem. But experience here has shown that even 1 or 2 reallocated sectors is a predictor of increased reallocated sectors on neary every parity check. It's almost like a string on a garmet and once you start pulling on it, the garmet falls apart. I'd suggest starting to run very frequent parity checks - every day or every other day. If after 4-5 parity checks the number has held steady at 2, I'd feel comfortable that the drive is stable and the the 2 reallocated sectors are isolated and not a problem. But if the number increases you need to keep running parity checks until you get at least 4 consecutive parity checks with NO increase in reallocated sectors. If they start to creep upward and you find you've run 10-20 parity checks and the reallocated sectors keep going up, the drive will never by reliable IMO. If the count starts incrementing, the tricky part may be getting it to report enough reallocated sectors to qualify the drive for RMA replacement. BTW, I noticed that on older generation drives the number of reallocated sectors would frequently stabilize and not increase, but in drives >= 500G, I have seen very few that have a stable number of reallocated sectors.
March 15, 201016 yr Author Since two of the drives are not in the array yet would the preclear script be a good alternative? All of these drives are probably a year old or more and it's only been recently that I've started looking into the SMART reports. These reallocated sectors could have happened the day I got the drives or last month, I simply don't know.
March 15, 201016 yr Since two of the drives are not in the array yet would the preclear script be a good alternative?Yes. It will work to exercise the drives.
March 15, 201016 yr That brings up a question about parity checks. Say one of your data disks is dieing and keeps getting reallocated errors. Parity check will correct errors right? Then if that data disk starts having errors that are corrupting the data will the parity check "correct" the parity so that when that drive is replaced the new drive will be rebuilt with corrupted data? I hope it is smarter than this, it's just an issue I have been wondering about. When parity is corrected how does unraid know which one is correct? Parity or data?
March 15, 201016 yr That brings up a question about parity checks. Say one of your data disks is dieing and keeps getting reallocated errors. Parity check will correct errors right? Then if that data disk starts having errors that are corrupting the data will the parity check "correct" the parity so that when that drive is replaced the new drive will be rebuilt with corrupted data? I hope it is smarter than this, it's just an issue I have been wondering about. When parity is corrected how does unraid know which one is correct? Parity or data? it always assumes parity is wrong. You can perform a NOCORRECT parity check. It will not change anything, but report if differences exist.
March 15, 201016 yr That brings up a question about parity checks. Say one of your data disks is dieing and keeps getting reallocated errors. Parity check will correct errors right? Then if that data disk starts having errors that are corrupting the data will the parity check "correct" the parity so that when that drive is replaced the new drive will be rebuilt with corrupted data? I hope it is smarter than this, it's just an issue I have been wondering about. When parity is corrected how does unraid know which one is correct? Parity or data? If unRAID is able to read data from a data disk, it will "trust" that data to be correct - no parity checking occurs "on the fly" as data is read. Therefore, if the drives internal mechanisms are reallocating sectors and not returning errors, unRAID will never know or care. If unRAID receives a true READ ERROR when reading a sector from a data disk, it will spring into action. It will read the corresponding sectors from each of the disks in the array (including parity), to determine what the unreadable sector data should have been. It will then WRITE that data back to the original drive. If the drive has implemented SMART correctly, the read error followed by the write should cause a sector reallocation - in effect correcting the error on the disk. The error counter on the drive should increment when this occurs. During a parity check, if unRAID detects that the numbers don't add up (i.e., parity is not consistent with the data) - it always assumes that parity is wrong and updates it to make parity consistent. (I make a distinction between making parity consistent and correcting the problem, because there are rare cases where one of the drives is flakey and updating parity interferes with the ability to recover). As Joe L. points out, there is a way to run a read-only parity check, which I highly recommend if you are running parity checks with a failing disk! The problem with all of the parity checks is that unRAID does not report where the mismatches occurs. You can thank NAS () for that! unRAID used to report these in the syslog, but in some rare situations it could cause the syslog to grow uncontrollably large and ultimately crash the server. NAS pushed hard for this to be fixed, and as part of the solution, Tom removed these logging entries.
March 15, 201016 yr The problem with all of the parity checks is that unRAID does not report where the mismatches occurs. You can thank NAS () for that! unRAID used to report these in the syslog, but in some rare situations it could cause the syslog to grow uncontrollably large and ultimately crash the server. NAS pushed hard for this to be fixed, and as part of the solution, Tom removed these logging entries. A solution was proposed to put all the log files in a tempfs file system. This would prevent any log from growing to where it is using all the available RAM. Tom never implemented this. I think it should be re-visited. If it is, then logging the parity check addresses is not an issue. If there are so many parity errors that the log-file fills then we really don't care about the specific individual addresses... there is a much bigger issue. The bigger issue, at least in my mind is what we would do if we knew that the "parity" at byte address 100,000,000 has bad parity. (an address I picked at random) So... here is the question... Given an address where parity is bad. and let's say 10 data disks, one bit from one of the drives was read differently than when parity was last calculated. Which of the 10 drives is it? We can't tell... nothing will tell us, not with a single parity disk and not other checks available. Let's say we had "par" checksums... on all the files... Which file (if any) is at address 100,000,000?? It could be part of a file, or completely un-used spare on the disk, or part of the file-system hierarchy pointing to files. I don't know of a utility which when given a byte address will tell us what is on the disk on a reiserfs at that address. In any case, we would need to run that utility on 10 disks, find potentially the names of up to 10 files, then determine which is being read incorrectly. (An almost impossible task, as a single bit error in many media files may be noticed... or not. ) Let's assume they were all movies... We can play all 10 and see if any of the video or sound looks/or sounds odd... I don't think that is realistic.... you must pay attention, since the single bit might result in a click in the sound, or a blip in the video rather than a complete failure to play. If it was a program instead, the bit might be in a feature of the product you've never used, and never will. Again, no easy way to detect the problem without using every feature of every program involved. If you have md5 checksums of all your files, you can use them to determine if a given file is changed. That is about the only way I can think of to verify which file is corrupt, if any. (The bad parity might be at random addresses based on heat, power supply noise, the phase of the moon, high-tide, etc.) With all this in mind, I'd still like to know the addresses of the failures... but put into place the tempfs for logs so the system will not crash if they fill. Joe L.
March 16, 201016 yr A solution that I use is to generate PAR2 files as I fill each volume. It allows me to not only detect, but correct, minor file corruptions (should they occur). Although you are right that knowing the exact disk address is not all that helpful in identifying an actual potentially corrupted file (although it is theoretically possible to narrow the list down to a single file on each disk), but even without this knowledge I used to find it helpful in several ways: 1 - If the sync errors are early in the disk you know that it is in the disk housekeeping / journaling area. This is frequently where hard power downs will generate parity sync errors and I never saw this pattern lead to data loss or corruption; 2 - If the sync errors are sequential, it is indicative of an issue related to a single file and a single disk. Perhaps the owner knows a file they were copying to the array and can simply recopy that file and have some confidence that was the only impacted file; and 3 - If the sync errors are occuring at seemingly random locations all over the place, you know you've got bigger problems.
March 16, 201016 yr A solution that I use is to generate PAR2 files as I fill each volume. It allows me to not only detect, but correct, minor file corruptions (should they occur). Could you elaborate on this please?
March 17, 201016 yr http://lime-technology.com/forum/index.php?topic=3124.msg26018#msg26018 That started a very interesting discussion there, which lasted for a day. Have you devised any automation for your server? Do you know whatever happened to Tom's idea that bubbaQ was referring to? Tom's idea of a second parity disk would work dynamically and accomplish the same thing.
March 18, 201016 yr Another PAR related thread: http://lime-technology.com/forum/index.php?topic=4327.msg38424#msg38424 Also, if you do an advanced search for par2cmdline you can find severl links to download a slackware version of par2. Running par2 is pretty easy. As I said, there is a limitation that you must run PAR2 on a single directory. It will not traverse and store directory info. So if you have a bunch of files stored in a single directory it is easy to do. If you have a bunch of files stored in folder structures - it is more difficult. My grand plan was to create links to all of the files on a drive in a single directory. The link names could be the fully qualified file names, the "/"s converted to periods or some other character. An automated script could be run to do this. You could then run PAR2 against that directory and have the same effect. I did an experiment and found that PAR2 worked with the links. But I never wrote the script. Most of my files are in a single directory and work file with PAR2 without this technique. I tried searching for a "PAR3" as NAS suggests. Found nothing. I am not sure that Bubbaq is correct in his belief that Tom's Q-Parity would provide a similar level of protection. I guess we'd have to wait and see. But for now, PAR2 works and provides a level of protection that is comforting for me ot have. If ever I get a bunch of parity errors (which hasn't happened since I got my UPS), I will run par checks on my drives and see if there is any data corruption.
Archived
This topic is now archived and is closed to further replies.