October 4, 201114 yr We had a small power outage this morning, when the array came back up disk10(sdd) came up red-balled. Its a Hitachi 2TB. Being a complete novice in the Linux/UnRAID world I thought I'd ask here before doing anything. What's the best way to proceed? The disk shows up fine in the browser, and luckily there is only a very small amount of non-critical data on it, but I sure would like to get it up and running. edit: it seems like these are the correct steps to follow (just want to be sure that I don't screw things up): * Stop the array, go to the Devices page and un-assign the disk. * Reboot (use the button on the Main page) * Stop the array, go to the Devices page and re-assign the disk * Go to the Main page - system should detect a "new" drive for the disabled one, click Start to start a parity-reconstruct of the disk. Attached are a SMART report for the drive and the most current syslog, I had to restart the array after the power went out so I'm not sure if the syslog will be the most helpful. There seem to be some scary bits in the SMART report about the disk failing within 24 hrs, and such. Thanks. p.s. I am on 4.7. p.p.s. reiserfsck reported no errors SHSMART_disk_10.txt syslog-2011-10-04.txt.zip
October 4, 201114 yr SMART overall-health self-assessment test result: FAILED! Drive failure expected in less than 24 hours. SAVE ALL DATA 2 Throughput_Performance 0x0005 047 047 054 Pre-fail Offline FAILING_NOW 3326 I will defer to someone more experienced, but the above could be the specific port, cable, or the drive failing...take a look at your previous smart tests on the drive to compare. Redball probably a result of 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 1
October 4, 201114 yr Author Not to be a complete idiot (but if the shoe fits...): Where would I go to see the old SMART reports, or would I have had to save them myself somewhere?
October 4, 201114 yr check power/data cables to drive perform a smartctl report. smartctl -a -d ata /dev/sd? where ? is the associated letter for that drive stop arrary unassign disk start array (NO NEED TO REBOOT) stop array reassign disk BACK start array (checkbox are you sure) data rebuild in progress http://lime-technology.com/wiki/index.php?title=Troubleshooting#What_do_I_do_if_I_get_a_red_ball_next_to_a_hard_disk.3F
October 4, 201114 yr Not to be a complete idiot (but if the shoe fits...): Where would I go to see the old SMART reports, or would I have had to save them myself somewhere? Check thru your old syslogs. Powerdown does the smartctl, so you should be able to check the end of your last syslog.
October 4, 201114 yr Author Check thru your old syslogs. Powerdown does the smartctl, so you should be able to check the end of your last syslog. Thanks. Now to firmly entrench myself in the newb hall-of-fame, where are the old syslogs? Or would it be the one from the OP? Here is the output of the SMART command you recommended (seems the same as the one in the OP). Anyone know what the throughput_performance error means? Cables/power all reseated and seem fine. Other drives on that SAS (SuperMicro SAS-MV8) card are working, too. root@Tower:/boot# smartctl -a -d ata /dev/sdd smartctl 5.39.1 2010-01-28 r3054 [i486-slackware-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Device Model: Hitachi HDS5C3020ALA632 Serial Number: ML0220F30HSLRD Firmware Version: ML6OA580 User Capacity: 2,000,398,934,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is: Tue Oct 4 11:05:01 2011 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: FAILED! Drive failure expected in less than 24 hours. SAVE ALL DATA. See vendor-specific Attribute list for failed Attributes. General SMART Values: Offline data collection status: (0x80) Offline data collection activity was never started. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (21608) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 255) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 095 095 016 Pre-fail Always - 11 2 Throughput_Performance 0x0005 047 047 054 Pre-fail Offline FAILING_NOW 3326 3 Spin_Up_Time 0x0007 134 134 024 Pre-fail Always - 409 (Average 406) 4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 83 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0 7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0 8 Seek_Time_Performance 0x0005 146 146 020 Pre-fail Offline - 29 9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 1339 10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 9 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 83 193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 83 194 Temperature_Celsius 0x0002 193 193 000 Old_age Always - 31 (Lifetime Min/Max 22/39) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 1 198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0 SMART Error Log Version: 1 ATA Error Count: 1 CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 1 occurred at disk power-on lifetime: 1337 hours (55 days + 17 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 0f 21 50 00 00 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 30 4e 00 40 00 00:01:33.288 READ FPDMA QUEUED 60 00 00 30 4c 00 40 00 00:01:33.287 READ FPDMA QUEUED 60 08 00 18 54 00 40 00 00:01:33.264 READ FPDMA QUEUED 60 20 00 10 4b 00 40 00 00:01:33.262 READ FPDMA QUEUED 60 00 00 10 49 00 40 00 00:01:33.261 READ FPDMA QUEUED SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 1337 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
October 4, 201114 yr Author mine get copied to the /boot/LOGS directory (\\tower\flash\LOGS via windows) Hmmm...can't find that directory anywhere. I may have to wait until I get home as I'm trying this through VNC and it's painfully slow and awkward. On another note the disk is rebuilding now, so well see what happens when I get home. Thanks for all the suggestions/ tips.
October 4, 201114 yr mine get copied to the /boot/LOGS directory (\\tower\flash\LOGS via windows) Hmmm...can't find that directory anywhere. I may have to wait until I get home as I'm trying this through VNC and it's painfully slow and awkward. On another note the disk is rebuilding now, so well see what happens when I get home. Thanks for all the suggestions/ tips. I dont have that directory either, is there something that has to be set to archive logs?
October 4, 201114 yr Hmmm....maybe this was a mod/pkg (powerdown-1.02-noarch-unRAID.tgz ??) that I installed. Do not recall. All I know is that when I powerdown (maybe part of clean powerdown pkg?), it performs a smartctl of all the drives and copies the syslog to the flash drive.
October 4, 201114 yr Hmmm....maybe this was a mod/pkg (powerdown-1.02-noarch-unRAID.tgz ??) that I installed. Do not recall. All I know is that when I powerdown (maybe part of clean powerdown pkg?), it performs a smartctl of all the drives and copies the syslog to the flash drive. yes, the above is correct.
October 4, 201114 yr Author Hmmm....maybe this was a mod/pkg (powerdown-1.02-noarch-unRAID.tgz ??) that I installed. Do not recall. All I know is that when I powerdown (maybe part of clean powerdown pkg?), it performs a smartctl of all the drives and copies the syslog to the flash drive. yes, the above is correct. Thanks. Once things are up and running I'll look into adding that package.
October 5, 201114 yr Author After rebuilding the drive. All is well. The repeat SMART report on the drive is normal, the pending sector is gone, reallocation events are back to zero, and all is ring with the world. I did install the clean powerdown package so hopefully that will help in the future to save some sys logs for analysis. Thank you all for your help and patience, it's quite disconcerting to get your first red ball (that doesn't sound good, does it? ).
October 5, 201114 yr If you have the opportunity, you could always do a powerdown now, then bring it up and verify that the syslog gets copied to the flash drive and it contains the smartctl on all your drives. Better now then later when you might need it.
October 5, 201114 yr Author If you have the opportunity, you could always do a powerdown now, then bring it up and verify that the syslog gets copied to the flash drive and it contains the smartctl on all your drives. Better now then later when you might need it. Agreed. It was late last night when the rebuild finished and I just wanted to pass out. My plan is to shutdown as soon as i get home from work today to verify the location and presence of the log files. Thanks again!
Archived
This topic is now archived and is closed to further replies.