March 4, 200917 yr ... Also, I still see ~140 errors on drive 4 (see screenshot). I will try running a SMART check tonight. This drive was a replacement for a DOA Samsung 1TB drive--I'm assuming it's refurbished. Are errors common on replacements for warrantied drives? In my experience, yes. For example, my refurb 1.5TB Seagate SMART view shows: » reallocated_sector_ct=28 » spin_retry_count=2 » high_fly_writes=70 » head_flying_hours=1.1864e+14 » attribute_241=2.69961e+09 » attribute_242=1133063340 In fact, my brand new Seagate 1.5TB (which I use as the parity drive) shows that high_fly_writes are slowly increasing. Started with none, now I have 3... but I'm not too worried about it: » high_fly_writes=3 » head_flying_hours=2.27925e+14 » attribute_241=3.67952e+09 » attribute_242=148917170 What board/chipset are you on? The qd tool didn't work for me. I'm on an Asus P5N-E with the Nvidia nForce 650i chipset. Glad you got yours working as well! I have the eVGA nForce 680i SLI 122-CK-NF68-AR. It's a bummer that the linux 2.6.27.7 (4.4.2 release) kernel support for Nvidia is finicky for us. Come to think of it, there was a period of a few weeks where the 4.4-beta1 release with kernel 2.6.26.5 was working just fine for me in conjunction with the swncq=0 boot code. I definitely wouldn't be where I am without the great advice from RobJ, bjp999 and the other members of this forum. I'll let you guys know if I can get the qd tool to work with 4.4.2 .
March 4, 200917 yr Author Are you running Bubba with 4.3.3? Is it even possible? I noticed that my Samsung 1TBs don't powerdown properly in 4.3.3 which is kind of a bummer.
March 4, 200917 yr Reallocated sectors and pending reallocatiions, even a very few, are something to be concerned about. The reason is that once you have some, there is a frequent tendency to develop more and more until the drive finally fails. If the reallocated sectors holds steady at some number, even 100 or more, it is okay. But if every time you run a parity check, the number is going up, that is bad. Your parity check found 140 READ errors on that one drive. That is not good, because it likely means that you have 140 reallocated sectors. Again, if these really were little bad spots on the disk and no new ones develop, everything is fine. But if the next time you get more of them, the prognosis is not good. You should understand that when your unRAID array is protected, as your is, and a read error occurs, unRAID has all the info it needs to reconstruct the sector it couldn't read. unRAID does that and then WRITES that sector back to the bad sector. This causes the SMART system to REMAP the bad sector for a good sector. This is exactly what you want it to do. There should be no corruption on your disk as a result. (That's what you paid for). But now that they're remapped, they shouldn't show up again. Next time should be totally clean. If you run a smart report and see pending reallocations, that means that unRAID attempt to force the remap didn't work. I see these sometimes and not sure if its a bug in unRAID or that, for some reason, the drive didn't remap when it should have. After another parity check, however, they should get found and remapped.
March 5, 200917 yr No such luck, system messed up over night again. Log here: http://www.2shared.com/file/5001132/58d2f3d/syslog.html I decided to just go ahead and downgrade to 4.3.3 via the method you described. Parity check running now. Fingers crossed. A belated analysis, this syslog gives a mixed report, one I'm not sure I fully understand. You indicate later that qd did not work for you, and if you don't mind, I would like to hear what steps and commands you took, and the results you saw. Until 4 minutes after the parity check had finished, everything looked so much better, that I thought that we might have fixed it. Ignoring all of the Disk 4 issues (on ata13), there were only 2 error sequences, one before the parity check was started and one near the end of the check, and BOTH were of a different nature, probably cable related, to Disk 1. I would replace that SATA cable. The parity check finished with decent speed, considering the problems with Disk 4. Four minutes later however! Is there any possibility you might have turned NCQ back on, with the qd on command? This was about 3:40 AM, if your clock is correct, so seems unlikely you waited for the parity check to finish and then began file operations. I would have thought you were sleeping, and there would not have been any disk I/O at all. So I don't know how to account for the sudden disintegration of your system. All of the same type of errors began spewing out, and worse, the other 2 drives on the motherboard nForce SATA ports began issuing the same error sequences too. I thought it strange in your earlier syslog, that only the first 2 nForce SATA ports were causing so much trouble, and the other 2 behaving so well. But this time, all 4 of those ports are having the same problems, the parity drive, Disk 1, Disk 2, and Disk 3 (ata2, ata3, ata4, ata5). I'm happy that v4.3.3 is working for you. The alternative is to disconnect all drives from the motherboard ports and connect them to your 8 port card. You would have slower parity checks, but should be able to run v4.4.2 without issue. Does this mean that I won't be able to upgrade? It just means you will need to wait for a kernel release that has fixed the problems with the sata_nv driver, or whatever drive support modules that seem currently incompatible with some nForce SATA ports. Are you running Bubba with 4.3.3? Is it even possible? According to the BubbaRAID addon info, v0.0.7 works with v4.3.3. I don't think it is available any more though.
March 5, 200917 yr Author Here's the SMART output for disk4. I'm guessing I just need to watch to make sure the reallocated_event_count doesn't go up? root@unRAID:/mnt/disk1/apps# smartctl -a -d ata /dev/sdg smartctl version 5.36 [i486-slackware-linux-gnu] Copyright (C) 2002-6 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: SAMSUNG HD103UJ Serial Number: S13PJDWQA04076 Firmware Version: 1AA01113 User Capacity: 1,000,204,886,016 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 7 ATA Standard is: Not recognized. Minor revision code: 0x52 Local Time is: Wed Mar 4 17:45:56 2009 GMT ==> WARNING: May need -F samsung or -F samsung2 enabled; see manual for details. SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (11700) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 196) minutes. Conveyance self-test routine recommended polling time: ( 21) minutes. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 100 099 051 Pre-fail Always - 114 3 Spin_Up_Time 0x0007 075 075 011 Pre-fail Always - 8200 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 53 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 253 253 051 Pre-fail Always - 0 8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 1242 10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0 11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 49 13 Read_Soft_Error_Rate 0x000e 100 099 000 Old_age Always - 58 183 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 184 Unknown_Attribute 0x0033 100 100 000 Pre-fail Always - 0 187 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 88 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 190 Unknown_Attribute 0x0022 065 061 000 Old_age Always - 639762467 194 Temperature_Celsius 0x0022 065 060 000 Old_age Always - 35 (Lifetime Min/Max 2/10274) 195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 175781407 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 20 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 5 201 Soft_Read_Error_Rate 0x000a 100 099 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. root@unRAID:/mnt/disk1/apps#
March 5, 200917 yr Now I'm baffled! That SMART report is essentially completely clean, with no bad sectors, no UNCorrectable sectors, not even any questionable sectors now or in the past! I had to double and triple check the serial number to make sure we are looking at the right drive. So why is this very clean drive producing many many sequences of this: Mar 3 23:44:29 unRAID kernel: ata13.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 Mar 3 23:44:29 unRAID kernel: ata13.00: edma_err_cause=00000104 pp_flags=00000001, dev error, EDMA self-disable Mar 3 23:44:29 unRAID kernel: ata13.00: cmd 25/00:10:ef:04:e8/00:02:29:00:00/e0 tag 0 dma 270336 in Mar 3 23:44:29 unRAID kernel: res 51/40:00:68:05:e8/40:00:29:00:00/e0 Emask 0x9 (media error) Mar 3 23:44:29 unRAID kernel: ata13.00: status: { DRDY ERR } Mar 3 23:44:29 unRAID kernel: ata13.00: error: { UNC } Mar 3 23:44:29 unRAID kernel: ata13: hard resetting link Mar 3 23:44:29 unRAID kernel: ata13: SATA link up 1.5 Gbps (SStatus 113 SControl 300) Mar 3 23:44:29 unRAID kernel: ata13.00: configured for UDMA/133 Mar 3 23:44:29 unRAID kernel: ata13: EH complete followed by this: Mar 3 23:44:37 unRAID kernel: end_request: I/O error, dev sdg, sector 703071592 Mar 3 23:44:37 unRAID kernel: ata13: EH complete Mar 3 23:44:37 unRAID kernel: md: disk4 read error Mar 3 23:44:37 unRAID kernel: handle_stripe read error: 703071528/4, count: 1 Mar 3 23:44:37 unRAID kernel: md: disk4 read error Mar 3 23:44:37 unRAID kernel: handle_stripe read error: 703071536/4, count: 1 Mar 3 23:44:37 unRAID kernel: md: disk4 read error Mar 3 23:44:37 unRAID kernel: handle_stripe read error: 703071544/4, count: 1 Mar 3 23:44:37 unRAID kernel: md: disk4 read error ... and many more ... The (media error) and { UNC } are typical of a bad sector, UNCorrectable, and shows up as such in the next SMART report. Yours did not! This part (edma_err_cause=00000104 pp_flags=00000001, dev error, EDMA self-disable) is very unusual, something I have never seen before, and is possibly confusing the error handler, looks possibly like something is broken internally. I'm afraid this is beyond my knowledge level here. I welcome others comments and expertise. All I can advise is, according to the SMART report, the drive is fine, but because of the way it is responding in your unRAID server, it is unusable as a drive in your array. I don't see an alternative to removing it from your server.
March 6, 200917 yr Author RobJ, again, thanks for helping me get everything fixed! Still running strong: 08:26:53 up 2 days, 1:45, 1 user, load average: 0.50, 0.59, 0.56 I've copied over 1TB+ now. Guess I'm going to have to deal with disk4 this weekend :<
Archived
This topic is now archived and is closed to further replies.