aaron330i Posted October 5, 2011 Share Posted October 5, 2011 I was running the preclear script on a new 2TB disk when after a while one of my data disks shows up as being disabled with SMART unable to show any kind of information for the disk. The disk log for the disk before I rebooted was this: cat /var/log/syslog|egrep "(sdi|scsi17 :|scsi 17:0:0:0| ata17:| ata17.00:|disk5[^0-9]| md5:)" (Filtered syslog for disk5) Oct 5 09:32:39 Tower kernel: scsi17 : ahci Oct 5 09:32:39 Tower kernel: ata17: SATA max UDMA/133 abar m2048@0x92100000 port 0x92100100 irq 33 Oct 5 09:32:39 Tower kernel: ata17: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Oct 5 09:32:39 Tower kernel: ata17.00: ATA-8: ST31500541AS, CC34, max UDMA/133 Oct 5 09:32:39 Tower kernel: ata17.00: 2930277168 sectors, multi 0: LBA48 NCQ (depth 31/32) Oct 5 09:32:39 Tower kernel: ata17.00: configured for UDMA/133 Oct 5 09:32:39 Tower kernel: scsi 17:0:0:0: Direct-Access ATA ST31500541AS CC34 PQ: 0 ANSI: 5 Oct 5 09:32:39 Tower kernel: sd 17:0:0:0: [sdi] 2930277168 512-byte logical blocks: (1.50 TB/1.36 TiB) Oct 5 09:32:39 Tower kernel: sd 17:0:0:0: [sdi] Write Protect is off Oct 5 09:32:39 Tower kernel: sd 17:0:0:0: [sdi] Mode Sense: 00 3a 00 00 Oct 5 09:32:39 Tower kernel: sd 17:0:0:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Oct 5 09:32:39 Tower kernel: sdi: Oct 5 09:32:39 Tower kernel: sdi1 Oct 5 09:32:39 Tower kernel: sd 17:0:0:0: [sdi] Attached SCSI disk Oct 5 09:32:39 Tower emhttp: pci-0000:06:00.0-scsi-0:0:0:0 host17 (sdi) ST31500541AS_6XW15LJX Oct 5 09:32:39 Tower kernel: md: import disk5: [8,128] (sdi) ST31500541AS 6XW15LJX offset: 63 size: 1465138552 Oct 5 09:32:40 Tower emhttp: shcmd (9): /usr/local/sbin/set_ncq sdi 1 >/dev/null Oct 5 09:32:40 Tower kernel: md5: running, size: 1465138552 blocks Oct 5 09:32:41 Tower emhttp: shcmd (28): mount -t reiserfs -o noacl,nouser_xattr,noatime,nodiratime /dev/md5 /mnt/disk5 >/dev/null 2>&1 Oct 5 09:32:43 Tower emhttp: shcmd (63): echo \"/mnt/disk5\" '-async,no_subtree_check,anongid=0,anonuid=0,all_squash' '*(rw)' >>/etc/exports Oct 5 10:14:07 Tower emhttp: shcmd (97): umount /mnt/disk5 >/dev/null 2>&1 Oct 5 10:14:07 Tower emhttp: shcmd (98): rmdir /mnt/disk5 >/dev/null 2>&1 Oct 5 10:14:11 Tower kernel: md5: stopping Oct 5 10:14:29 Tower emhttp: shcmd (130): /usr/local/sbin/set_ncq sdi 1 >/dev/null Oct 5 10:14:29 Tower kernel: md5: running, size: 1465138552 blocks Oct 5 10:14:30 Tower emhttp: shcmd (143): mount -t reiserfs -o noacl,nouser_xattr,noatime,nodiratime /dev/md5 /mnt/disk5 >/dev/null 2>&1 Oct 5 10:14:32 Tower emhttp: shcmd (180): echo \"/mnt/disk5\" '-async,no_subtree_check,anongid=0,anonuid=0,all_squash' '*(rw)' >>/etc/exports Oct 5 10:57:33 Tower kernel: ata17: exception Emask 0x10 SAct 0x0 SErr 0x4010000 action 0xe frozen Oct 5 10:57:33 Tower kernel: ata17: irq_stat 0x80400040, connection status changed Oct 5 10:57:33 Tower kernel: ata17: SError: { PHYRdyChg DevExch } Oct 5 10:57:33 Tower kernel: ata17: hard resetting link Oct 5 10:57:34 Tower kernel: ata17: SATA link down (SStatus 0 SControl 300) Oct 5 10:57:39 Tower kernel: ata17: hard resetting link Oct 5 10:57:39 Tower kernel: ata17: SATA link down (SStatus 0 SControl 300) Oct 5 10:57:39 Tower kernel: ata17: limiting SATA link speed to 1.5 Gbps Oct 5 10:57:44 Tower kernel: ata17: hard resetting link Oct 5 10:57:44 Tower kernel: ata17: SATA link down (SStatus 0 SControl 310) Oct 5 10:57:44 Tower kernel: ata17.00: disabled Oct 5 10:57:44 Tower kernel: ata17: EH complete Oct 5 10:57:44 Tower kernel: ata17.00: detaching (SCSI 17:0:0:0) Oct 5 10:57:44 Tower kernel: sd 17:0:0:0: [sdi] Synchronizing SCSI cache Oct 5 10:57:44 Tower kernel: sd 17:0:0:0: [sdi] Result: hostbyte=0x04 driverbyte=0x00 Oct 5 10:57:44 Tower kernel: sd 17:0:0:0: [sdi] Stopping disk Oct 5 10:57:44 Tower kernel: sd 17:0:0:0: [sdi] START_STOP FAILED Oct 5 10:57:44 Tower kernel: sd 17:0:0:0: [sdi] Result: hostbyte=0x04 driverbyte=0x00 Oct 5 10:58:01 Tower kernel: ata17: exception Emask 0x10 SAct 0x0 SErr 0x4010000 action 0xe frozen Oct 5 10:58:01 Tower kernel: ata17: irq_stat 0x80400040, connection status changed Oct 5 10:58:01 Tower kernel: ata17: SError: { PHYRdyChg DevExch } Oct 5 10:58:01 Tower kernel: ata17: hard resetting link Oct 5 10:58:01 Tower kernel: ata17: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Oct 5 10:58:01 Tower kernel: ata17.00: native sectors (1) is smaller than sectors (2930277168) Oct 5 10:58:01 Tower kernel: ata17.00: ATA-8: ST31500541AS, CC34, max UDMA/133 Oct 5 10:58:01 Tower kernel: ata17.00: 2930277168 sectors, multi 0: LBA48 NCQ (depth 31/32) Oct 5 10:58:01 Tower kernel: ata17.00: configured for UDMA/133 Oct 5 10:58:01 Tower kernel: ata17: EH complete Oct 5 10:58:01 Tower kernel: scsi 17:0:0:0: Direct-Access ATA ST31500541AS CC34 PQ: 0 ANSI: 5 Oct 5 10:58:01 Tower kernel: ata17.00: exception Emask 0x10 SAct 0x1 SErr 0x4010000 action 0xe frozen Oct 5 10:58:01 Tower kernel: ata17.00: irq_stat 0x80400040, connection status changed Oct 5 10:58:01 Tower kernel: ata17: SError: { PHYRdyChg DevExch } Oct 5 10:58:01 Tower kernel: ata17.00: failed command: READ FPDMA QUEUED Oct 5 10:58:01 Tower kernel: ata17.00: cmd 60/20:00:00:00:00/00:00:00:00:00/40 tag 0 ncq 16384 in Oct 5 10:58:01 Tower kernel: ata17.00: status: { Busy } Oct 5 10:58:01 Tower kernel: ata17: hard resetting link Oct 5 10:58:02 Tower kernel: ata17: SATA link down (SStatus 0 SControl 300) Oct 5 10:58:06 Tower kernel: ata17: hard resetting link Oct 5 10:58:07 Tower kernel: ata17: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Oct 5 10:58:07 Tower kernel: ata17.00: configured for UDMA/133 Oct 5 10:58:07 Tower kernel: ata17: EH complete Oct 5 10:58:07 Tower kernel: ata17.00: exception Emask 0x10 SAct 0x1 SErr 0x4010000 action 0xe frozen Oct 5 10:58:07 Tower kernel: ata17.00: irq_stat 0x80400040, connection status changed Oct 5 10:58:07 Tower kernel: ata17: SError: { PHYRdyChg DevExch } Oct 5 10:58:07 Tower kernel: ata17.00: failed command: READ FPDMA QUEUED Oct 5 10:58:07 Tower kernel: ata17.00: cmd 60/08:00:80:7a:a8/00:00:ae:00:00/40 tag 0 ncq 4096 in Oct 5 10:58:07 Tower kernel: ata17.00: status: { Busy } Oct 5 10:58:07 Tower kernel: ata17: hard resetting link Oct 5 10:58:07 Tower kernel: ata17: SATA link down (SStatus 0 SControl 300) Oct 5 10:58:12 Tower kernel: ata17: hard resetting link Oct 5 10:58:13 Tower kernel: ata17: SATA link down (SStatus 0 SControl 300) Oct 5 10:58:13 Tower kernel: ata17: limiting SATA link speed to 1.5 Gbps Oct 5 10:58:18 Tower kernel: ata17: hard resetting link Oct 5 10:58:18 Tower kernel: ata17: SATA link down (SStatus 0 SControl 310) Oct 5 10:58:18 Tower kernel: ata17.00: disabled Oct 5 10:58:18 Tower kernel: ata17: EH complete Oct 5 10:58:18 Tower kernel: ata17.00: detaching (SCSI 17:0:0:0) Oct 5 11:00:54 Tower kernel: ata17: exception Emask 0x10 SAct 0x0 SErr 0x4010000 action 0xe frozen Oct 5 11:00:54 Tower kernel: ata17: irq_stat 0x80400040, connection status changed Oct 5 11:00:54 Tower kernel: ata17: SError: { PHYRdyChg DevExch } Oct 5 11:00:54 Tower kernel: ata17: hard resetting link Oct 5 11:00:55 Tower kernel: ata17: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Oct 5 11:00:55 Tower kernel: ata17.00: native sectors (1) is smaller than sectors (2930277168) Oct 5 11:00:55 Tower kernel: ata17.00: ATA-8: ST31500541AS, CC34, max UDMA/133 Oct 5 11:00:55 Tower kernel: ata17.00: 2930277168 sectors, multi 0: LBA48 NCQ (depth 31/32) Oct 5 11:00:55 Tower kernel: ata17.00: configured for UDMA/133 Oct 5 11:00:55 Tower kernel: ata17: EH complete Oct 5 11:00:55 Tower kernel: scsi 17:0:0:0: Direct-Access ATA ST31500541AS CC34 PQ: 0 ANSI: 5 Oct 5 11:40:07 Tower kernel: md: disk5 read error Oct 5 11:40:07 Tower kernel: md: disk5 write error Oct 5 11:40:07 Tower kernel: md: disk5 read error Oct 5 11:40:15 Tower kernel: md: disk5 write error Oct 5 11:43:13 Tower kernel: md: disk5: open_by_devnum error: -6 Oct 5 11:43:17 Tower emhttp: shcmd (214): umount /mnt/disk5 >/dev/null 2>&1 Oct 5 11:43:18 Tower emhttp: shcmd (215): rmdir /mnt/disk5 >/dev/null 2>&1 Oct 5 11:43:21 Tower kernel: md5: stopping Done After rebooting, I was able to run smartctl to get information about the disk and for the life of me, I can't really see anything wrong with it.: smartctl -a -d ata /dev/sdi (disk5) smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: ST31500541AS Serial Number: 6XW15LJX Firmware Version: CC34 User Capacity: 1,500,301,910,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is: Wed Oct 5 12:13:44 2011 GMT+6 SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 653) seconds. Offline data collection capabilities: (0x73) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x103f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 118 099 006 Pre-fail Always - 195402452 3 Spin_Up_Time 0x0003 100 100 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1963 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 067 060 030 Pre-fail Always - 5610545 9 Power_On_Hours 0x0032 087 087 000 Old_age Always - 11923 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 97 183 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 099 000 Old_age Always - 4295032833 189 High_Fly_Writes 0x003a 099 099 000 Old_age Always - 1 190 Airflow_Temperature_Cel 0x0022 070 063 045 Old_age Always - 30 (Lifetime Min/Max 29/30) 194 Temperature_Celsius 0x0022 030 040 000 Old_age Always - 30 (0 17 0 0) 195 Hardware_ECC_Recovered 0x001a 061 049 000 Old_age Always - 195402452 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 226306121796863 241 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 1779109296 242 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 829014263 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 11781 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. I'm sorta lost as to whether or not the disk is truly bad, or if preclearing the new disk may have caused some sort of controller error, or something else entirely. I don't really know what to do to get my server into a protected state asap. Quote Link to comment
aaron330i Posted October 5, 2011 Author Share Posted October 5, 2011 Question: Since I have that extra unassigned 2TB disk in the system, would it be better to : 1: Remove the current failed disk5 from the array then assign the new 2TB disk to disk5 and let the parity rebuild? 2: Rebuild the current disk5 from parity. My personal opinion is to go ahead and assign the new disk5 to parity leaving the old disk5 around "just in case" parity should fail to rebuild. Question: If I assign the new disk to disk5, will the server preclear the disk prior to rebuild or will it rebuild immediately? Minimizing server unavailability is the chief concern. Quote Link to comment
aaron330i Posted October 5, 2011 Author Share Posted October 5, 2011 I would also like to mention that the new 2TB disk was supposed to be used to replace a disk showing signs of impending terminal peril. I'm crossing my fingers on the rebuild. If it's successful, then the old disk 5 may become the new disk10. ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 39 smartctl -a -d ata /dev/sdh (disk10) smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.11 Device Model: ST31000340AS Serial Number: 9QJ1TWGL Firmware Version: SD15 User Capacity: 1,000,204,886,016 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is: Wed Oct 5 12:54:00 2011 GMT+6 SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 650) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 233) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x103b) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 107 099 006 Pre-fail Always - 42641272 3 Spin_Up_Time 0x0003 091 084 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 097 097 020 Old_age Always - 3192 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 39 7 Seek_Error_Rate 0x000f 077 060 030 Pre-fail Always - 55119825 9 Power_On_Hours 0x0032 070 070 000 Old_age Always - 26835 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 10 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 126 184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 087 000 Old_age Always - 137441050657 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 066 057 045 Old_age Always - 34 (Lifetime Min/Max 33/34) 194 Temperature_Celsius 0x0022 034 043 000 Old_age Always - 34 (0 17 0 0) 195 Hardware_ECC_Recovered 0x001a 040 019 000 Old_age Always - 42641272 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 26691 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Quote Link to comment
dgaschk Posted October 5, 2011 Share Posted October 5, 2011 A replacement disk does not need to be cleared (although it's safest to pre-clear). The entirety of the disk is written during a rebuild. Quote Link to comment
aaron330i Posted October 5, 2011 Author Share Posted October 5, 2011 I've decided to rebuild using the new disk. My 16 disk array will take almost 20 hours to rebuild. When it gets past the 1TB mark of (the size of disk10), I'll breath a bit easier. Then it'll be time to preclear the old disk on a different SATA channel to verify the disk is truely good before using it as a replacement for disk10. Sigh, I hate babysitting. Heh. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.