smakovits Posted October 20, 2011 Share Posted October 20, 2011 Hey guys. Been running solid for a good bit ever since I replaced the on-board NIC with a add-in Intel card, until today. All of a sudden access became slow and sometimes unresponsive. I thought it was a switch that I added to the network in the office to work on another computer and removing that I thought the issue was resolved, but I was wrong and the trouble returned. As a result I thought it could only be one of 2 things, maybe a bad NIC cable in the office or trouble on the server. I checked the logs and didnt like what I saw, but it was from several days ago, so i am not sure if it is associated with today's issues or not. as for the cable, it is yet to be replaced. At any rate, I wanted to start by uploading the logs from the system to see if there is anything useful in them. This was on the smart report page: Disk 1: *ERROR* - Current_Pending_Sector has increased from 0 to 1461 since 2011-03-20 Disk 1: *ERROR* - Current_Pending_Sector it is now 1461 (error threshold is 5) Disk 1: OK - Power_On_Hours is 14003 Disk 1: *ERROR* - Offline_Uncorrectable has increased from 0 to 7 since 2011-03-20 Disk 1: *ERROR* - Offline_Uncorrectable it is now 7 (error threshold is 5) As always, thanks a ton. Hopefully it is nothing super terrible, but we will see. In the mean time I will also try replacing the cable. I had to compress the log due to size syslog-2011-10-19.zip Link to comment
mbryanr Posted October 20, 2011 Share Posted October 20, 2011 That drive is failing. You probably have errors next to that drive as well. 1461 pending sectors since the "trouble" started. Oct 1 00:05:04 Tower kernel: ata2.00: failed command: READ DMA EXT Oct 1 00:05:04 Tower kernel: ata2.00: cmd 25/00:00:2f:a0:1a/00:04:01:00:00/e0 tag 0 dma 524288 in Oct 1 00:05:04 Tower kernel: res 51/40:5f:c8:a3:1a/00:00:01:00:00/e0 Emask 0x9 (media error) Post the complete smart test report as well from unMENU or command line smartctl -a -d ata /dev/sdX where X is the drive in question. http://lime-technology.com/wiki/index.php?title=Troubleshooting#Obtaining_a_SMART_report Link to comment
smakovits Posted October 20, 2011 Author Share Posted October 20, 2011 Is there an easy way to tell which drive that is? I thought it was sdb, but the smart report said passed, same for sda, therefore, I either have the wrong drive or I am missing something. Link to comment
mbryanr Posted October 20, 2011 Share Posted October 20, 2011 The smart report is overly optimistic for the manufacturers.. There are probably >1000 spare sectors, the problem is you continue to get read errors as well. No there is not an easy way to determine the the disk assignment with the disk#. One of the reasons it is important to keep a screen shot of your configuration (sda=Serial# etc) In your case it is easier... Oct 1 00:05:42 Tower kernel: sd 2:0:0:0: [sdb] Unhandled sense code Oct 1 00:05:42 Tower kernel: md: disk1 read error Oct 1 00:05:42 Tower kernel: handle_stripe read error: 18538040/1, count: 1 Oct 1 00:06:47 Tower kernel: end_request: I/O error, dev sdb, sector 18545645 Oct 1 00:06:47 Tower kernel: ata2: EH complete Oct 1 00:06:47 Tower kernel: md: disk1 read error Oct 1 00:06:47 Tower kernel: handle_stripe read error: 18545576/1, count: 1 Also there is a way to get rid of the ntp synchronization log entries... Oct 9 07:00:06 Tower ntpd[2703]: synchronized to 64.90.182.55, stratum 1 Oct 9 07:13:02 Tower ntpd[2703]: time reset +1.951675 s Oct 9 07:14:16 Tower ntpd[2703]: synchronized to 64.90.182.55, stratum 1 Oct 9 07:29:04 Tower ntpd[2703]: time reset +1.953068 s Oct 9 07:29:35 Tower ntpd[2703]: synchronized to 64.90.182.55, stratum 1 I just forget what it was. It could be the CMOS battery. Link to comment
smakovits Posted October 20, 2011 Author Share Posted October 20, 2011 OK, I will look into a cmos battery. otherwise, here is the SMART output. smart.txt Link to comment
mbryanr Posted October 20, 2011 Share Posted October 20, 2011 197 Current_Pending_Sector 0x0032 196 196 000 Old_age Always - 1461 198 Offline_Uncorrectable 0x0030 200 198 000 Old_age Offline - 7 Time to replace it. Normally, if that number is much lower you may be able to save it - if the number does not continue to increase. The next time a write occurs it will reallocate the sector if possible....and go into the "offline uncorrectable" category. A good practice is to run and review the smart tests almost as periodically as you do a parity check ~1x/month; or more frequently if you notice errors. You caught it before it any real damage. Now you can just preclear (recommended) a new drive, install, and rebuild the contents of the old drive to the new one from parity. http://lime-technology.com/wiki/index.php?title=Replacing_a_Data_Drive as far as the ntpd log entries...it is just reporting what it should be doing..but your MB is losing time for some reason. If they become bothersome you could follow this to prevent it from logging: http://lime-technology.com/forum/index.php?topic=5681.msg53196#msg53196 Link to comment
smakovits Posted October 20, 2011 Author Share Posted October 20, 2011 Is it OK to just leave the drive in the array for the time being? RMA submitted, so I assume advance turn around is next week +1 day for preclear. Link to comment
mbryanr Posted October 20, 2011 Share Posted October 20, 2011 as long as you don't get another failure. I had to do the same thing. Link to comment
smakovits Posted October 21, 2011 Author Share Posted October 21, 2011 Drive shipped... Crazy thing, I can still access unraid via putty and as a share to write to and read/watch content. However, the web interface is now down completely and inaccessible. I was going to reboot, but thought it might only make issues. So I will wait on the drive to arrive first. Link to comment
dgaschk Posted October 21, 2011 Share Posted October 21, 2011 Enter this to restart the web interface: killall emhttp nohup /usr/local/sbin/emhttp & Link to comment
smakovits Posted October 24, 2011 Author Share Posted October 24, 2011 Thanks, that worked. Now, while on topic of failing drives, I want to ensure that I only have the one bad drive and that some of my failed writes to another disk are simply a result of the server being unstable. Therefore, I got the smart report from the other drive that was being written at the time of the write failure, just to check its health. smart.txt Link to comment
dgaschk Posted October 24, 2011 Share Posted October 24, 2011 That drive looks fine. You might want to look into reducing the rate of the Load_Cycle_Count. But Load_Cycle_Count is not causing any operational problems. Link to comment
smakovits Posted October 24, 2011 Author Share Posted October 24, 2011 excuse my ignorance but how does one do that? Link to comment
BobPhoenix Posted October 25, 2011 Share Posted October 25, 2011 excuse my ignorance but how does one do that? Use this utility from WD. Link to comment
smakovits Posted October 25, 2011 Author Share Posted October 25, 2011 If I were to run this do I have to remove the drives from the Array? what is the overall affect on the disk and its content and the way it is seen by unRAID? I have a few drives showing 300k and even 400k + load cycles. If a typical lifetime is supposed to average 300-600k should I be worried? Link to comment
smakovits Posted October 26, 2011 Author Share Posted October 26, 2011 funny thing. so before stopping my array to add the drive to replace the failing one, MyMain was showing the failing drive as needing sectors moved or what not, now after the reboot it does not. What gives? here is the latest report smartctl -a -d ata /dev/sdb (disk1) smartctl version 5.38 [i486-slackware-linux-gnu] Copyright © 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: WDC WD20EADS-65R6B0 Serial Number: WD-WCAVY2136899 Firmware Version: 01.00A01 User Capacity: 2,000,398,934,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Tue Oct 25 23:31:37 2011 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (42360) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x303f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 155 149 021 Pre-fail Always - 9225 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 702 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 081 081 000 Old_age Always - 14147 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 56 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 25 193 Load_Cycle_Count 0x0032 069 069 000 Old_age Always - 393110 194 Temperature_Celsius 0x0022 121 109 000 Old_age Always - 31 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 196 196 000 Old_age Always - 1461 198 Offline_Uncorrectable 0x0030 200 198 000 Old_age Offline - 7 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 196 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Link to comment
dgaschk Posted October 26, 2011 Share Posted October 26, 2011 That disk has 1461 sectors pending reallocation. However, none have actually been reallocated. The safest the to do would be to unassign this drive and assign a spare pre-cleared disk as its replacement and then rebuild the contents. Then you can pre-clear this disk and see if the pending sectors disappear or become reallocated. If they become reallocated then you should RMA this drive. If you don't have a spare, unassign this drive and pre-clear it. If the pending sectors drops to zero then reassign and rebuild. If the pending sectors become reallocated then you should RMA this drive. Then you can rebuild. Link to comment
smakovits Posted October 26, 2011 Author Share Posted October 26, 2011 Ah, I see it now. The bad drive has been unassigned and the new precleared drive is rebuilding. This drive will be RMA-ed. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.