Jump to content

2 Drives Failed?


Recommended Posts

I just saw that I had 2 drives that had several errors (284, 41) and the one with 41 errors was red balled. I stopped the array and started it back up to see if it was something that could be rebuilt itself with a parity check. When it started back up, it showed both of the drives that had errors are now showing "Unformatted". I shut the server down.

 

I just went and bought 2 replacement drives and was hoping someone could give me an idea on the best way to go about this and keep as much data intact as possible. I'm running 4.7, btw.

 

Thanks

Link to comment

I was going to post a syslog, but it's massive because of the errors (max message length). Is there a specific chunk of data I could look for to post?

 

By hardware configuration, do you mean drives, or also motherboard, controllers, etc also? If it 's just hard drives, I have 10 drives (1 as a parity of course) 3-2TB Drives and 7-1.5TB Drives. The red balled one is 2TB and the other a 1.5TB. The parity drive is still showing as good.

Link to comment

I was going to post a syslog, but it's massive because of the errors (max message length). Is there a specific chunk of data I could look for to post?

 

By hardware configuration, do you mean drives, or also motherboard, controllers, etc also? If it 's just hard drives, I have 10 drives (1 as a parity of course) 3-2TB Drives and 7-1.5TB Drives. The red balled one is 2TB and the other a 1.5TB. The parity drive is still showing as good.

zip up the syslog, they compress really well. Then attach.
Link to comment

I read the syslog earlier, and didn't have time to reply..

Redballed means a "write" to the drive failed. They appeared unformatted after the restart because it could not read the partition

Your first clue was back on Sept 18th:

Sep 18 07:43:50 MediaServer kernel: md: parity incorrect: 128  (which you correctly ran a parity check)

then

Sep 24 16:28:51 MediaServer kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen

Sep 24 16:28:51 MediaServer kernel: ata1.00: failed command: CHECK POWER MODE

then

 

24 16:28:51 MediaServer kernel: ata1.00: failed command: CHECK POWER MODE

Oct 18 22:18:19 MediaServer kernel: ata6.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen

Oct 18 22:18:19 MediaServer kernel: ata6.00: failed command: CHECK POWER MODE

Oct 18 22:18:19 MediaServer kernel: ata6.00: cmd e5/00:00:00:00:00/00:00:00:00:00/00 tag 0

Oct 18 22:18:19 MediaServer kernel:          res 40/00:ff:00:00:00/00:00:00:00:00/50 Emask 0x4 (timeout)

Oct 18 22:18:19 MediaServer kernel: ata6.00: status: { DRDY }

Oct 18 22:18:19 MediaServer kernel: ata6.00: hard resetting link

Oct 18 22:18:19 MediaServer kernel: ata6.01: hard resetting link

Oct 18 22:18:25 MediaServer kernel: ata6.00: link is slow to respond, please be patient (ready=0)

Oct 18 22:18:29 MediaServer kernel: ata6.00: SRST failed (errno=-16)

Oct 18 22:18:29 MediaServer kernel: ata6.00: hard resetting link

Oct 18 22:18:29 MediaServer kernel: ata6.01: hard resetting link

Oct 18 22:18:35 MediaServer kernel: ata6.00: link is slow to respond, please be patient (ready=0)

Oct 18 22:18:39 MediaServer kernel: ata6.00: SRST failed (errno=-16)

Oct 18 22:18:39 MediaServer kernel: ata6.00: hard resetting link

Oct 18 22:18:39 MediaServer kernel: ata6.01: hard resetting link

Oct 18 22:18:45 MediaServer kernel: ata6.00: link is slow to respond, please be patient (ready=0)

Oct 18 22:19:14 MediaServer kernel: ata6.00: SRST failed (errno=-16)

Oct 18 22:19:14 MediaServer kernel: ata6.00: limiting SATA link speed to 1.5 Gbps

Oct 18 22:19:14 MediaServer kernel: ata6.01: limiting SATA link speed to 1.5 Gbps

Oct 18 22:19:14 MediaServer kernel: ata6.00: hard resetting link

Oct 18 22:19:14 MediaServer kernel: ata6.01: hard resetting link

Oct 18 22:19:19 MediaServer kernel: ata6.00: SRST failed (errno=-16)

Oct 18 22:19:19 MediaServer kernel: ata6.00: reset failed, giving up

Oct 18 22:19:19 MediaServer kernel: ata6.00: disabled

Oct 18 22:19:19 MediaServer kernel: ata6.01: disabled

 

afterwards you get a bunch of I/O and read errors on sdg and sdh...

 

1.  Check all connections and cables first.

2.  sdg and sdh look dead...but try to run on a smart test on both of those.

 

Either from unMenu, or telnet/putty

smartctl  -a  -d  ata  /dev/sdX

where "X" is the drive assignment.

(sdg and sdh)

Post the results here.

 

Likely the system can not find the partition (either due to drive failure or it is looking in the wrong place)

IF the smart test looks ok, type..

3.  for sdg

 

fdisk -lu /dev/sdg

 

and for sdh

fdisk -lu /dev/sdh

http://lime-technology.com/forum/index.php?topic=15807.15

 

Link to comment

Thanks for looking into it. I finally got around to running this test. Here are the results of the Smart tests:

 

Linux 2.6.32.9-unRAID.
root@MediaServer:~# smartctl -a -d ata /dev/sdg
smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Alle
n
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     ST_M13FQBL
Serial Number:    QNR_BFW
Firmware Version: 04240957
User Capacity:    4,142,054,400 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   6
ATA Standard is:  ATA/ATAPI-6 T13 1410D revision 2
Local Time is:    Thu Oct 27 22:27:48 2011 Local time zone must be set--see zic
m
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

SMART Disabled. Use option -s with argument 'on' to enable it.
root@MediaServer:~# smartctl -s on /dev/sdg
smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Alle
n
Home page is http://smartmontools.sourceforge.net/

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
Error SMART Enable failed
Smartctl: SMART Enable Failed.

A mandatory SMART command failed: exiting. To continue, add one or more '-T perm
issive' options.
root@MediaServer:~# smartctl -a -d ata /dev/sdh
smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Alle
n
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     ST32000542AS
Serial Number:    6XW21MWS
Firmware Version: CC95
User Capacity:    2,000,398,934,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Thu Oct 27 22:30:48 2011 Local time zone must be set--see zic
m
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                 ( 633) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off supp
ort.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 255) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x103f) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_
FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   114   100   006    Pre-fail  Always       -
       60598630
  3 Spin_Up_Time            0x0003   100   100   000    Pre-fail  Always       -
       0
  4 Start_Stop_Count        0x0032   037   037   020    Old_age   Always       -
       65535
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -
       0
  7 Seek_Error_Rate         0x000f   069   060   030    Pre-fail  Always       -
       8923044
  9 Power_On_Hours          0x0032   094   094   000    Old_age   Always       -
       5625
10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -
       0
12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -
       24
183 Unknown_Attribute       0x0032   099   099   000    Old_age   Always       -
       1
184 Unknown_Attribute       0x0032   100   100   099    Old_age   Always       -
       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -
       0
188 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -
       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -
       0
190 Airflow_Temperature_Cel 0x0022   076   070   045    Old_age   Always       -
       24 (Lifetime Min/Max 23/24)
194 Temperature_Celsius     0x0022   024   040   000    Old_age   Always       -
       24 (0 17 0 0)
195 Hardware_ECC_Recovered  0x001a   042   033   000    Old_age   Always       -
       60598630
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -
       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -
       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -
       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -
       72675141617283
241 Unknown_Attribute       0x0000   100   253   000    Old_age   Offline      -
       51833439
242 Unknown_Attribute       0x0000   100   253   000    Old_age   Offline      -
       3573286947

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA
_of_first_error
# 1  Short offline       Completed without error       00%      5625         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@MediaServer:~# fdisk -lu /dev/sdh

Disk /dev/sdh: 2000.3 GB, 2000398934016 bytes
1 heads, 63 sectors/track, 62016336 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdh1              63  3907029167  1953514552+  83  Linux
Partition 1 does not end on cylinder boundary.
root@MediaServer:~#

 

It looks like sdh may be ok, so I ran the fdisk command and you see at the end what I got back. It's still red-balled.

 

Looks a little better, I suppose. Does this help?

 

Thanks!

Link to comment

For some reason my BIOS was jacked up, not allowing the machine to boot, so I finally tracked down the issue after a couple hours.

 

The new drive is installed, but I bring up the control panel and it doesn't really give me any options. I have "Too many wrong or missing disks!"

 

I've attached the syslog and a screenshot of the control panel. Any ideas?

 

Thanks

drive_status.jpg.240d4f1f4ec8c8c17f6a60edf724da97.jpg

syslog-20111105-222639.zip

Link to comment

Yes, that is referring to the old drive, but it has been replaced. The line above it is the new drive size, I'm assuming.

 

This has been my issue from the start. I'm not sure what I'm supposed to do if they are both red. So, if I cannot replace a failed drive when two are red, then does that mean my entire raid is toast? I thought I read somewhere that if two drives fail, you may lose the data on those two drives. It does appear that one of the red ones is functional, but was taken offline. Is there a way I can "repair" and bring the functional one back online, then rebuild the failed drive?

 

As a last resort, what if I removed the failed drive from the array altogether? I would obviously lose the data on that drive, but maybe it would rebuild the functional one?

Link to comment

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...