May 1, 201214 yr First time I've seen a parity correction so I'm a tad nervous as to whether it was supposed to be corrected or was parity drive actually correct and my data was corrupted ( I guess now it means that data is unrecoverable - I don't even know what file(s) were corrected ). Is there a log saying which file(s) triggered a parity correction? I checked the syslog, nothing much in there. There were several thousand lines referring to duplicate files. Should I be nervous? edit: solved. No SMART errors, no filesystem errors. Repeated DD/checksum tests on suspected blocks went fine. Assume parity data was not correct and am now doing a parity sync.
May 1, 201214 yr The default monthly parity check is in a NOCORRECT mode. Unless you changed this, the parity disk was not updated, even though the messages male it appear as if it was. The messages will always say "corrected" since the web-interface was never coded to change the wording on a NOCORRECT check. You are faced with determining the reason for the differences. It could be anything from a single disk, to bad memory, to a bad disk controller. There are threads describing repeated "dd" commands of specific disks in the regions where parity block differences were detected. repeated checksums on disks should always return the same value as long as the disk is not being written to. Good luck in finding the cause. (and I hope you installed the NOCORRECT option on the monthly check) Joe L.
May 1, 201214 yr Author The default monthly parity check is in a NOCORRECT mode. Unless you changed this, the parity disk was not updated, even though the messages male it appear as if it was. Phew, small sigh of relief. Default was left unchanged so I'm good there. Now, where are my errors coming from lol. I redo the hardware ( new cpu, board, psu, ram, drive cages ) and added 1 extra drive. I also manually moved data between the disks to balance them out a bit since I had a few drives that were on the brink of being completely full. There are threads describing repeated "dd" commands of specific disks in the regions where parity block differences were detected. repeated checksums on disks should always return the same value as long as the disk is not being written to. How would I know where parity block differences occured? Thanks for the help
May 1, 201214 yr How would I know where parity block differences occurred? The block numbers are printed in the syslog. (I seem to remember only the first 100 or so were printed, but I might be mistaken, as I cannot find it in the code where the print limit is enforced) Unfortunately, there is no way to determine which file/files a given block represents, or even if it is part of a file or directory hierarchy. Joe L.
May 1, 201214 yr Author It seems like only the first 20 errors are printed May 2 00:00:23 Glutton kernel: md: recovery thread checking parity... May 2 00:00:23 Glutton kernel: md: parity incorrect: 128 May 2 00:00:23 Glutton kernel: md: parity incorrect: 136 May 2 00:00:24 Glutton kernel: md: parity incorrect: 65680 May 2 00:00:24 Glutton kernel: md: parity incorrect: 65688 May 2 00:00:26 Glutton kernel: md: parity incorrect: 262144 May 2 00:00:28 Glutton kernel: md: parity incorrect: 524288 May 2 00:00:30 Glutton kernel: md: parity incorrect: 786432 May 2 00:00:31 Glutton kernel: md: parity incorrect: 1048576 May 2 00:00:33 Glutton kernel: md: parity incorrect: 1310720 May 2 00:00:34 Glutton kernel: md: parity incorrect: 1572864 May 2 00:00:35 Glutton kernel: md: parity incorrect: 1835008 May 2 00:00:37 Glutton kernel: md: parity incorrect: 2097152 May 2 00:00:38 Glutton kernel: md: parity incorrect: 2359296 May 2 00:00:39 Glutton kernel: md: parity incorrect: 2621440 May 2 00:00:41 Glutton kernel: md: parity incorrect: 2883584 May 2 00:00:42 Glutton kernel: md: parity incorrect: 3145728 May 2 00:00:43 Glutton kernel: md: parity incorrect: 3407872 May 2 00:00:45 Glutton kernel: md: parity incorrect: 3670016 May 2 00:00:46 Glutton kernel: md: parity incorrect: 3932160 May 2 00:00:47 Glutton kernel: md: parity incorrect: 4194304 I've ran the check twice and the same blocks come up in the same order. I guess that points to an actual filesystem error rather than a random data error to do with power, controller, memory or drives?
May 1, 201214 yr I've ran the check twice and the same blocks come up in the same order. I guess that points to an actual filesystem error rather than a random data error to do with power, controller, memory or drives? Correct. I'd check the file systems as described here, and once all check out correct, run a correcting parity sync. http://lime-technology.com/wiki/index.php/Check_Disk_Filesystems Joe L.
May 1, 201214 yr here are a couple scripts you can use to get the md5 checksums of all your drives md5_sd #!/bin/bash LOG_DIR=/boot/LOGS/hashes mkdir -p ${LOG_DIR} DeviceAddr=$1 BadParity=$2 StartHere=`expr ${BadParity} - 2000000 ` RunTime=`date +%F_%H%M` for i in {1..5} do echo "Begin ${DeviceAddr} for the $i time. Bad spot = ${BadParity} Start at ${StartHere} `date` " dd if=/dev/${DeviceAddr} skip=${StartHere} count=10000000 | md5sum -b >> ${LOG_DIR}/${DeviceAddr}_${RunTime}.log done exit md5_all #!/bin/bash LOG_DIR=/boot/LOGS/hashes mkdir -p ${LOG_DIR} BadParity=$1 RunTime=`date +%F_%H%M` echo "${RunTime} BadParity=${BadParity}" >> ${LOG_DIR}/BadParity.log md5_sd sdb $1 & md5_sd sdc $1 & md5_sd sdd $1 & md5_sd sde $1 & md5_sd sdf $1 & md5_sd sdg $1 & md5_sd sdh $1 & md5_sd sdi $1 & md5_sd sdk $1 & md5_sd sdl $1 & md5_sd sdm $1 & md5_sd sdn $1 & md5_sd sdo $1 & md5_sd sdp $1 & exit In md5_all, modify the list of drives. should be a md5_sd sdX $1 & for each of your drives. they should all be on the same line. Then to run it, type md5_all <bad block> so for bad block 1048576 you would type: md5_all 1048576 It will read 10,000,000 blocks, 2,000,000 before the inputted number, 7,999,999 after. You do not want ANY writes occurring during this, else it may not give you true results. For each drive, the md5 value for that drive should be the same.
May 1, 201214 yr The SMART reports should be inspected before running the dd commands. If only a single drive shows pending sectors then that is the problem drive.
May 1, 201214 yr The SMART reports should be inspected before running the dd commands. If only a single drive shows pending sectors then that is the problem drive. We're starting to sound like broken records. Post the SMART reports.
May 1, 201214 yr The SMART reports should be inspected before running the dd commands. If only a single drive shows pending sectors then that is the problem drive. exactly. And that drive can then be re-constructed from parity (assuming you do not clobber parity in the interim)
May 2, 201214 yr Author due to popular demand, I will now post SMART reports for my 7 drives Device Model: SAMSUNG HD204UI Serial Number: S2H7J1EB200189 Firmware Version: 1AQ10001 ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 100 100 051 Pre-fail Always - 2 2 Throughput_Performance 0x0026 252 252 000 Old_age Always - 0 3 Spin_Up_Time 0x0023 067 067 025 Pre-fail Always - 10187 4 Start_Stop_Count 0x0032 098 098 000 Old_age Always - 2819 5 Reallocated_Sector_Ct 0x0033 252 252 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 252 252 051 Old_age Always - 0 8 Seek_Time_Performance 0x0024 252 252 015 Old_age Offline - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 2961 10 Spin_Retry_Count 0x0032 252 252 051 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 252 252 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 22 181 Unknown_Attribute 0x0022 099 099 000 Old_age Always - 39218590 191 G-Sense_Error_Rate 0x0022 100 100 000 Old_age Always - 36 192 Power-Off_Retract_Count 0x0022 252 252 000 Old_age Always - 0 194 Temperature_Celsius 0x0002 064 064 000 Old_age Always - 20 (Lifetime Min/Max 14/35) 195 Hardware_ECC_Recovered 0x003a 100 100 000 Old_age Always - 0 196 Reallocated_Event_Count 0x0032 252 252 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 252 252 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 252 252 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0036 100 100 000 Old_age Always - 4 200 Multi_Zone_Error_Rate 0x002a 100 100 000 Old_age Always - 0 223 Load_Retry_Count 0x0032 252 252 000 Old_age Always - 0 225 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 2826 Device Model: SAMSUNG HD154UI Serial Number: S1XWJ1BZ129144 Firmware Version: 1AG01118 ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0007 063 063 011 Pre-fail Always - 11740 4 Start_Stop_Count 0x0032 098 098 000 Old_age Always - 2462 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0 8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 0 9 Power_On_Hours 0x0032 097 097 000 Old_age Always - 17343 10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0 11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 53 13 Read_Soft_Error_Rate 0x000e 100 100 000 Old_age Always - 0 183 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 184 Unknown_Attribute 0x0033 100 100 000 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 077 068 000 Old_age Always - 23 (Lifetime Min/Max 18/24) 194 Temperature_Celsius 0x0022 083 067 000 Old_age Always - 17 (Lifetime Min/Max 15/25) 195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 580631073 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 0 201 Soft_Read_Error_Rate 0x000a 100 100 000 Old_age Always - 0 Device Model: SAMSUNG HD154UI Serial Number: S1XWJ1BZ129472 Firmware Version: 1AG01118 ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0007 064 064 011 Pre-fail Always - 11690 4 Start_Stop_Count 0x0032 097 097 000 Old_age Always - 3422 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0 8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 0 9 Power_On_Hours 0x0032 097 097 000 Old_age Always - 17372 10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0 11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 53 13 Read_Soft_Error_Rate 0x000e 100 100 000 Old_age Always - 0 183 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 184 Unknown_Attribute 0x0033 100 100 000 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 078 067 000 Old_age Always - 22 (Lifetime Min/Max 20/25) 194 Temperature_Celsius 0x0022 081 066 000 Old_age Always - 19 (Lifetime Min/Max 17/28) 195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 546109966 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 0 201 Soft_Read_Error_Rate 0x000a 100 100 000 Old_age Always - 0 Device Model: SAMSUNG HD154UI Serial Number: S1XWJ1BZ129387 Firmware Version: 1AG01118 ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0007 063 063 011 Pre-fail Always - 11740 4 Start_Stop_Count 0x0032 098 098 000 Old_age Always - 2393 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 253 253 051 Pre-fail Always - 0 8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 0 9 Power_On_Hours 0x0032 097 097 000 Old_age Always - 17278 10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0 11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 36 13 Read_Soft_Error_Rate 0x000e 100 100 000 Old_age Always - 0 183 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 184 Unknown_Attribute 0x0033 100 100 000 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 079 067 000 Old_age Always - 21 (Lifetime Min/Max 19/25) 194 Temperature_Celsius 0x0022 082 066 000 Old_age Always - 18 (Lifetime Min/Max 16/26) 195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 696469776 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 0 201 Soft_Read_Error_Rate 0x000a 100 100 000 Old_age Always - 0 Device Model: SAMSUNG HD204UI Serial Number: S2H7J1EB200187 Firmware Version: 1AQ10001 ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 100 100 051 Pre-fail Always - 39 2 Throughput_Performance 0x0026 252 252 000 Old_age Always - 0 3 Spin_Up_Time 0x0023 067 066 025 Pre-fail Always - 10247 4 Start_Stop_Count 0x0032 098 098 000 Old_age Always - 2530 5 Reallocated_Sector_Ct 0x0033 252 252 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 252 252 051 Old_age Always - 0 8 Seek_Time_Performance 0x0024 252 252 015 Old_age Offline - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 3896 10 Spin_Retry_Count 0x0032 252 252 051 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 252 252 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 23 181 Unknown_Attribute 0x0022 100 100 000 Old_age Always - 8976269 191 G-Sense_Error_Rate 0x0022 100 100 000 Old_age Always - 113 192 Power-Off_Retract_Count 0x0022 252 252 000 Old_age Always - 0 194 Temperature_Celsius 0x0002 064 063 000 Old_age Always - 21 (Lifetime Min/Max 15/37) 195 Hardware_ECC_Recovered 0x003a 100 100 000 Old_age Always - 0 196 Reallocated_Event_Count 0x0032 252 252 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 252 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 252 252 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0036 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x002a 100 100 000 Old_age Always - 0 223 Load_Retry_Count 0x0032 252 252 000 Old_age Always - 0 225 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 2534 Device Model: SAMSUNG HD154UI Serial Number: S1XWJ1BZ129381 Firmware Version: 1AG01118 1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0007 063 063 011 Pre-fail Always - 11790 4 Start_Stop_Count 0x0032 096 096 000 Old_age Always - 3554 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0 8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 12275 9 Power_On_Hours 0x0032 097 097 000 Old_age Always - 17354 10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0 11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 54 13 Read_Soft_Error_Rate 0x000e 100 100 000 Old_age Always - 0 183 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 9 184 Unknown_Attribute 0x0033 100 100 000 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 078 066 000 Old_age Always - 22 (Lifetime Min/Max 20/25) 194 Temperature_Celsius 0x0022 082 064 000 Old_age Always - 18 (Lifetime Min/Max 16/26) 195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 579839372 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 0 201 Soft_Read_Error_Rate 0x000a 100 100 000 Old_age Always - 0 Device Model: WDC WD20EARX-00PASB0 Serial Number: WD-WMAZA7426175 Firmware Version: 51.0AB51 ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 170 169 021 Pre-fail Always - 6458 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 78 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 382 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 8 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 7 193 Load_Cycle_Count 0x0032 199 199 000 Old_age Always - 3494 194 Temperature_Celsius 0x0022 127 116 000 Old_age Always - 23 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 The WD drive was the new drive added just a week ago.
May 2, 201214 yr Since the smart reports show no re-allocated sectors, and no sectors pending re-allocation, perform the file-system checks next. If the file-systems all check out OK, then you can try the repeated "dd/checksum" tests. If they show nothing, then I guess the next step would be to correct the parity.
May 2, 201214 yr Since you have a number of Samsung drives is there a chance any of them are F4's and suffering from the firmware issue discussed here: http://lime-technology.com/forum/index.php?topic=9339.0 Regards, Stephen
May 3, 201214 yr Author Since you have a number of Samsung drives is there a chance any of them are F4's and suffering from the firmware issue discussed here: http://lime-technology.com/forum/index.php?topic=9339.0 Regards, Stephen Yeh I thought that before but the 204's were purchased after the firmware had been released and supposedly included in new drives. And I've been using the system for a while without issues.
Archived
This topic is now archived and is closed to further replies.