April 2, 201016 yr Author Thanks again for you help Joe. Unfortunately, your response is a bit cryptic. I am guessing that you are being sarcastic and it is normal to see an incremental disk write from 480 to 481 during preclear. Or are you saying that the heads loading is a problem, but not a major problem??? I am completely new to all of this, so I am not really qualified to judge the significance of smart or syslog statements. I currently have assigned a different parity disk and 1 drive and am showing no errors so far during the parity check which is at 87%. I am hoping that I can add drives back one at a time until I start seeing parity errors again. This should identify a bad drive, cable or sata port for that device.
April 2, 201016 yr Thanks again for you help Joe. Unfortunately, your response is a bit cryptic. I am guessing that you are being sarcastic and it is normal to see an incremental disk write from 480 to 481 during preclear. Yes, I was being a tiny bit rude. Sorry. Every time the disk heads move from their "parked" position to one over the disk patters the count will increment. It is very very normal. Or are you saying that the heads loading is a problem, but not a major problem??? It is not a problem at all. If the disk heads did not move to the spinning platter, that WOULD be a problem. (because it could then not read or write any data) I am completely new to all of this, so I am not really qualified to judge the significance of smart or syslog statements. I currently have assigned a different parity disk and 1 drive and am showing no errors so far during the parity check which is at 87%. I am hoping that I can add drives back one at a time until I start seeing parity errors again. This should identify a bad drive, cable or sata port for that device. Let's hope you can isolate it.
April 2, 201016 yr Author parity finished with zero errors, however the server is now unresponsive to the stop button...preclear is still running on 2nd drive. The old parity drives smart showed 12 crc errors, which apparently would support something being wrong with the drive, cabling or port.
April 2, 201016 yr Author Just read your previous post. I am struggling to get up to speed so I apologize for my newbieness. I was hoping that unraid would not be such a technical endeavor. Turning out to be a bit more difficult that building a regular computer system. I do appreciate your help.
April 5, 201016 yr Author Well after days of testing, it looks like the problem was two fold... Firstly the usb drive was returning errors and unrepairable. Secondly there is one drive that causes errors when added to the array. I am going to try different cable today and a different port. The strange thing is there is nothing in the smart report or syslog to indicate a problem.
April 5, 201016 yr Well after days of testing, it looks like the problem was two fold... Firstly the usb drive was returning errors and unrepairable. That is fairly easy to resolve.Secondly there is one drive that causes errors when added to the array. I am going to try different cable today and a different port. The strange thing is there is nothing in the smart report or syslog to indicate a problem. That is exactly the same as the other person. The "smart" report is unable to show the errors... probably because they occur so seldom... or, perhaps it is the cable or the port... I'll be curious to learn more. at least you are in the process of narrowing it down.
April 5, 201016 yr Author Well I ran preclear on the suspect drive and have now added it back to the array with a new cable. So far no parity errors are showing up. Could be as simple as a bad cable. I am leaving town for a week, so I won't be able to test any further until next weekend. Thanks again for your help Joe.
April 5, 201016 yr Well I ran preclear on the suspect drive and have now added it back to the array with a new cable. So far no parity errors are showing up. Could be as simple as a bad cable. I am leaving town for a week, so I won't be able to test any further until next weekend. Thanks again for your help Joe. The parity errors will not likely show up until you perform a parity test or two. Press the "Check" button, let it have something to do while you get ready for your trip. (You don't need to have any files on the drive, in fact, it is easier as you won't have to move them if the drive is bad. Yes, it can easily be a bad cable. And so much aggravation, and hair loss. Let's hope that's all it was. Joe L.
April 12, 201016 yr Author Well, it looks like the server is running smoothly now with no errors and 6 disks in the array. I am a little worried about how difficult it was to isolate this problem. Nothing in the syslog or smart reports indicated a problem. Fortunately I was willing to start over and add one disk at a time to find the culprit. How would one isolate this problem expediently while maintaining data? What could I have done differently? It would be great to get feedback on my troubleshooting process to help streamline it in the future. Thanks again for your help guys!
June 2, 201016 yr Author Well, after 2 months of error free operation I am getting parity errors again. Not sure where to start. Syslog is not showing any errors.
June 2, 201016 yr Author Here's the smart statistics for the parity drive. It looks like it is returning a ton of errors. Should I replace this drive?? ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 104 099 006 Pre-fail Always - 7203032 3 Spin_Up_Time 0x0003 093 092 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 110 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 1 7 Seek_Error_Rate 0x000f 071 060 030 Pre-fail Always - 14485766 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1279 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 10 184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 074 074 000 Old_age Always - 26 190 Airflow_Temperature_Cel 0x0022 058 055 045 Old_age Always - 42 (Lifetime Min/Max 42/42) 194 Temperature_Celsius 0x0022 042 045 000 Old_age Always - 42 (0 18 0 0) 195 Hardware_ECC_Recovered 0x001a 044 036 000 Old_age Always - 7203032 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 255090992612136 241 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 1055420659 242 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 2885366066
June 2, 201016 yr Here's the smart statistics for the parity drive. It looks like it is returning a ton of errors. Should I replace this drive?? ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 104 099 006 Pre-fail Always - 7203032 3 Spin_Up_Time 0x0003 093 092 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 110 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 1 7 Seek_Error_Rate 0x000f 071 060 030 Pre-fail Always - 14485766 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1279 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 10 184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 074 074 000 Old_age Always - 26 190 Airflow_Temperature_Cel 0x0022 058 055 045 Old_age Always - 42 (Lifetime Min/Max 42/42) 194 Temperature_Celsius 0x0022 042 045 000 Old_age Always - 42 (0 18 0 0) 195 Hardware_ECC_Recovered 0x001a 044 036 000 Old_age Always - 7203032 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 255090992612136 241 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 1055420659 242 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 2885366066 I see one re-allocated sector and nothing else... What errors are you talking about? No "VALUE" is anywhere near its affiliated failure THRESH (threshold) value. No, you should not replace the drive. You should instead read more about the SMART reporting. http://en.wikipedia.org/wiki/S.M.A.R.T.
June 2, 201016 yr Author Thanks Joe. Wish I wasn't back here with errors. I was looking at the raw read error rate and seek error rates that say pre fail in front of them. The other drives don't show these high values. I will read up on smart checking. I am trying to run the short test using unmenu, but when i click the button there is no response dialogue. Is there normally an indication that it is running?
June 2, 201016 yr Thanks Joe. Wish I wasn't back here with errors. I was looking at the raw read error rate and seek error rates that say pre fail in front of them. The other drives don't show these high values. I will read up on smart checking. I am trying to run the short test using unmenu, but when i click the button there is no response dialogue. Is there normally an indication that it is running? Old_Age is a category for that attribute. The values shown depend on the drive, the model, the firmware version, and the manufacturer. There are two types of attribute categories. Those that will increment over time and indicate Old_Age. Power-cycles for example, if you power cycle the drive enough times, it will be considered to be "Old" and prone to failure. The other type are not age related, but performance related. Every attribute belongs to one of those two categories. The "short" and "Long" reports are requests for those tests to be started. You must then wait an appropriate time and get a status report to see the results. you must turn off the spin-down timer for the long test to run to completion. Joe L.
June 5, 201016 yr Author OK So I read up on smart reports and think I more or less understand them. I just ran long smart tests on all drive and the only unusual finding is this: 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 12 I understand that crc errors are usually cables or port related, so I will recheck the connections for that drive and run parity check again and report back.
June 5, 201016 yr OK So I read up on smart reports and think I more or less understand them. I just ran long smart tests on all drive and the only unusual finding is this: 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 12 I understand that crc errors are usually cables or port related, so I will recheck the connections for that drive and run parity check again and report back. You're correct, and so is the Raw_Read_Error_Rate linked to bad port, cable or connection, and 7203032 is a very high value to this attribute. The best tool to test the array stability is run a parity check with "NOCORRECT", because if your system is ok you should not receive any sync errors; otherwise, if your hardware is somehow faulty, it will not result in a bad parity data.
June 5, 201016 yr OK So I read up on smart reports and think I more or less understand them. I just ran long smart tests on all drive and the only unusual finding is this: 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 12 I understand that crc errors are usually cables or port related, so I will recheck the connections for that drive and run parity check again and report back. You're correct, and so is the Raw_Read_Error_Rate linked to bad port, cable or connection, and 7203032 is a very high value to this attribute. It has meaning ONLY to the manufacturer. ALL disk drives have raw-read errors. Some manufacturers show the raw value, some do not. It is meaningless to us if they do show it regardless. You can only look at the normalized value and compare it to the threshold. It is not something you can control or affect in any way by changing a cable or connections. The Normalized value of 200 is not even close to the failure-threshold of 0 for that parameter. The best tool to test the array stability is run a parity check with "NOCORRECT", because if your system is ok you should not receive any sync errors; otherwise, if your hardware is somehow faulty, it will not result in a bad parity data. Good advice.
June 5, 201016 yr Author Apparently Seagates display very high raw read error rates and that is not an indication of a problem, as Joe stated. The 12 crc errors seem unusual since they are only occurring on one of 5 identical wd drives. I will run a nocorrect parity and post the log. The problem in the past has been that I get repeat parity errors, but the syslog does not report any problems.
June 5, 201016 yr Author One problem...I don't know how to run a no correct parity check. Am running 1.51
June 5, 201016 yr One problem...I don't know how to run a no correct parity check. Am running 1.51 If you have unMENU installed, there is a button for it on the Array-Management page. If not, you can just issue the command on the Linux command line /root/mdcmd check NOCORRECT Described here: http://lime-technology.com/forum/index.php?topic=5729.msg54081#msg54081 and introduced here: http://lime-technology.com/forum/index.php?topic=3430.msg29619#msg29619
June 6, 201016 yr Author Well, I ran the nocorrect check and it found 8 parity errors. Here are the parity check results. Not sure if tsc unstable means anything. I will attach the full syslog. Maybe there is something there that I can't see. Jun 5 15:56:43 Tower kernel: md: recovery thread checking parity... Jun 5 15:56:43 Tower kernel: md: using 1152k window, over a total of 1465138552 blocks. Jun 5 19:08:34 Tower kernel: Clocksource tsc unstable (delta = 62502548 ns) Jun 5 21:38:12 Tower kernel: md: sync done. time=20489sec rate=71508K/sec Jun 5 21:38:12 Tower kernel: md: recovery thread sync completion status: 0 syslog_6_5_10_norepair.txt.zip
June 8, 201016 yr Author I have been doing some more searching while waiting to hear back. I came across some threads regarding ears drives. I have 2 drives that are unjumpered ears drives. Lime-tech say to jumper the drives. I am really not sure what to do at this point. I am getting 8 parity errors and the wiki says don't swap out the ears drive if you have parity errors. I also see no solution to my parity errors and the problem could be the ears drives...I suppose. What should my course of action be? I have some new drives that I could use to replace the ear drives one at a time, but this does not address current parity.
June 8, 201016 yr Can you confirm that the 8 errors are you seeing are sync errors (found during a parity check) and not drive errors (shown in the error column of the web gui)? I am assuming these are sync errors. unRAID used to log the locations of parity sync errors, but there were complaints about an issue where the log file could consume all memory in some unusual situations and this logging was removed. Pitty cause it would be helpful to see where the 8 parity errors are located. All a parity error means is that, somewhere on the disk, you have a bit that is out of whack with the corresponding bits on the other drives. It is difficult to know for sure which drive is wrong (unless you find some evidence in a smart report). Parity errors are commonly caused by unclean shutdowns. They have also been seen in incompatible motherboards, and with bad memory. It could mean a bad drive but that is very unusual. Drives have special error checking to avoid returning bad data. 1. The first thing I would do is run a memory test. Bad memory can cause subtle corruption. 2 - If the memory test runs overnight without finding any errors, run the read-only parity check again (maybe several times) and see if you get the same results. Post the results and we can help you with next steps.
June 8, 201016 yr Author Thanks for the reply. Yes they are sync errors, not drive errors. I will run the memory test over night and then re-do the read only parity.
Archived
This topic is now archived and is closed to further replies.