Posted October 19, 201113 yr I just saw that I had 2 drives that had several errors (284, 41) and the one with 41 errors was red balled. I stopped the array and started it back up to see if it was something that could be rebuilt itself with a parity check. When it started back up, it showed both of the drives that had errors are now showing "Unformatted". I shut the server down. I just went and bought 2 replacement drives and was hoping someone could give me an idea on the best way to go about this and keep as much data intact as possible. I'm running 4.7, btw. Thanks
October 19, 201113 yr Post a syslog, hardware configuration and don't try to format or reset configuration. Most can only guess what happened.
October 19, 201113 yr Author I was going to post a syslog, but it's massive because of the errors (max message length). Is there a specific chunk of data I could look for to post? By hardware configuration, do you mean drives, or also motherboard, controllers, etc also? If it 's just hard drives, I have 10 drives (1 as a parity of course) 3-2TB Drives and 7-1.5TB Drives. The red balled one is 2TB and the other a 1.5TB. The parity drive is still showing as good.
October 20, 201113 yr I was going to post a syslog, but it's massive because of the errors (max message length). Is there a specific chunk of data I could look for to post? By hardware configuration, do you mean drives, or also motherboard, controllers, etc also? If it 's just hard drives, I have 10 drives (1 as a parity of course) 3-2TB Drives and 7-1.5TB Drives. The red balled one is 2TB and the other a 1.5TB. The parity drive is still showing as good. zip up the syslog, they compress really well. Then attach.
October 20, 201113 yr By hardware configuration......or also motherboard, controllers, etc also? Motherboard, add-on cards etc. Be able to figure out most of it from the syslog, but nice to have.
October 20, 201113 yr Author Ok, here is what I have: Motherboard: Gigabyte GA-EP45-UD3L Video: Galaxy nVidia GeForce 8400GS SATA Controller: Highpoint RocketRaid 2300 RAM: 4GB Corsair (2x2GB) Power Supply: Corsair 750W Syslog attached. Thanks! syslog-20111019-201239.zip
October 23, 201113 yr I read the syslog earlier, and didn't have time to reply.. Redballed means a "write" to the drive failed. They appeared unformatted after the restart because it could not read the partition Your first clue was back on Sept 18th: Sep 18 07:43:50 MediaServer kernel: md: parity incorrect: 128 (which you correctly ran a parity check) then Sep 24 16:28:51 MediaServer kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Sep 24 16:28:51 MediaServer kernel: ata1.00: failed command: CHECK POWER MODE then 24 16:28:51 MediaServer kernel: ata1.00: failed command: CHECK POWER MODE Oct 18 22:18:19 MediaServer kernel: ata6.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Oct 18 22:18:19 MediaServer kernel: ata6.00: failed command: CHECK POWER MODE Oct 18 22:18:19 MediaServer kernel: ata6.00: cmd e5/00:00:00:00:00/00:00:00:00:00/00 tag 0 Oct 18 22:18:19 MediaServer kernel: res 40/00:ff:00:00:00/00:00:00:00:00/50 Emask 0x4 (timeout) Oct 18 22:18:19 MediaServer kernel: ata6.00: status: { DRDY } Oct 18 22:18:19 MediaServer kernel: ata6.00: hard resetting link Oct 18 22:18:19 MediaServer kernel: ata6.01: hard resetting link Oct 18 22:18:25 MediaServer kernel: ata6.00: link is slow to respond, please be patient (ready=0) Oct 18 22:18:29 MediaServer kernel: ata6.00: SRST failed (errno=-16) Oct 18 22:18:29 MediaServer kernel: ata6.00: hard resetting link Oct 18 22:18:29 MediaServer kernel: ata6.01: hard resetting link Oct 18 22:18:35 MediaServer kernel: ata6.00: link is slow to respond, please be patient (ready=0) Oct 18 22:18:39 MediaServer kernel: ata6.00: SRST failed (errno=-16) Oct 18 22:18:39 MediaServer kernel: ata6.00: hard resetting link Oct 18 22:18:39 MediaServer kernel: ata6.01: hard resetting link Oct 18 22:18:45 MediaServer kernel: ata6.00: link is slow to respond, please be patient (ready=0) Oct 18 22:19:14 MediaServer kernel: ata6.00: SRST failed (errno=-16) Oct 18 22:19:14 MediaServer kernel: ata6.00: limiting SATA link speed to 1.5 Gbps Oct 18 22:19:14 MediaServer kernel: ata6.01: limiting SATA link speed to 1.5 Gbps Oct 18 22:19:14 MediaServer kernel: ata6.00: hard resetting link Oct 18 22:19:14 MediaServer kernel: ata6.01: hard resetting link Oct 18 22:19:19 MediaServer kernel: ata6.00: SRST failed (errno=-16) Oct 18 22:19:19 MediaServer kernel: ata6.00: reset failed, giving up Oct 18 22:19:19 MediaServer kernel: ata6.00: disabled Oct 18 22:19:19 MediaServer kernel: ata6.01: disabled afterwards you get a bunch of I/O and read errors on sdg and sdh... 1. Check all connections and cables first. 2. sdg and sdh look dead...but try to run on a smart test on both of those. Either from unMenu, or telnet/putty smartctl -a -d ata /dev/sdX where "X" is the drive assignment. (sdg and sdh) Post the results here. Likely the system can not find the partition (either due to drive failure or it is looking in the wrong place) IF the smart test looks ok, type.. 3. for sdg fdisk -lu /dev/sdg and for sdh fdisk -lu /dev/sdh http://lime-technology.com/forum/index.php?topic=15807.15
October 27, 201113 yr Author Thanks for looking into it. I finally got around to running this test. Here are the results of the Smart tests: Linux 2.6.32.9-unRAID. root@MediaServer:~# smartctl -a -d ata /dev/sdg smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Alle n Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: ST_M13FQBL Serial Number: QNR_BFW Firmware Version: 04240957 User Capacity: 4,142,054,400 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 6 ATA Standard is: ATA/ATAPI-6 T13 1410D revision 2 Local Time is: Thu Oct 27 22:27:48 2011 Local time zone must be set--see zic m SMART support is: Available - device has SMART capability. SMART support is: Enabled SMART Disabled. Use option -s with argument 'on' to enable it. root@MediaServer:~# smartctl -s on /dev/sdg smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Alle n Home page is http://smartmontools.sourceforge.net/ === START OF ENABLE/DISABLE COMMANDS SECTION === Error SMART Enable failed Smartctl: SMART Enable Failed. A mandatory SMART command failed: exiting. To continue, add one or more '-T perm issive' options. root@MediaServer:~# smartctl -a -d ata /dev/sdh smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Alle n Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: ST32000542AS Serial Number: 6XW21MWS Firmware Version: CC95 User Capacity: 2,000,398,934,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is: Thu Oct 27 22:30:48 2011 Local time zone must be set--see zic m SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 633) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off supp ort. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x103f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_ FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 114 100 006 Pre-fail Always - 60598630 3 Spin_Up_Time 0x0003 100 100 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 037 037 020 Old_age Always - 65535 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 069 060 030 Pre-fail Always - 8923044 9 Power_On_Hours 0x0032 094 094 000 Old_age Always - 5625 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 24 183 Unknown_Attribute 0x0032 099 099 000 Old_age Always - 1 184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 076 070 045 Old_age Always - 24 (Lifetime Min/Max 23/24) 194 Temperature_Celsius 0x0022 024 040 000 Old_age Always - 24 (0 17 0 0) 195 Hardware_ECC_Recovered 0x001a 042 033 000 Old_age Always - 60598630 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 72675141617283 241 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 51833439 242 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 3573286947 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA _of_first_error # 1 Short offline Completed without error 00% 5625 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. root@MediaServer:~# fdisk -lu /dev/sdh Disk /dev/sdh: 2000.3 GB, 2000398934016 bytes 1 heads, 63 sectors/track, 62016336 cylinders, total 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdh1 63 3907029167 1953514552+ 83 Linux Partition 1 does not end on cylinder boundary. root@MediaServer:~# It looks like sdh may be ok, so I ran the fdisk command and you see at the end what I got back. It's still red-balled. Looks a little better, I suppose. Does this help? Thanks!
October 28, 201113 yr Author Neither one shows as unformatted now, but sdg shows an odd size display. 4,044,940 1,465,138,552 Both are still redballed. Screenshot attached.
October 29, 201113 yr I've never seen that before. Maybe someone else can chime in with thoughts. Post another syslog, maybe that will provide a clue; and yes, the latest smart test looked ok.
October 31, 201113 yr Author I have attached the latest syslog. If the sdh drive is ok, I'm assuming I'll be able to repair it somehow, then replace the sdg drive and have the array rebuild itself, correct? I sure hope so... Thanks! syslog-20111028-123121.zip
November 2, 201113 yr Author So, should I just try to replace the sdg drive and hope for the best? Is there anything special I can try to do to repair/bring online/greenball the sdh drive before replacing sdg? Thanks
November 4, 201113 yr Author Ok, I was going to try to replace the sdg drive, then boot up and then use the Trust My Array procedure to hopefully bring back the second drive to rebuild what it can. Do you think this is a good plan, or am I going to screw something up by doing this?
November 4, 201113 yr The likelihood of file corruption is high. The problem will be determining which files are corrupt. Run rieserfsck --check on the rebuilt disk.
November 4, 201113 yr Author Great! Thanks for the response. I just didn't want to do something incredibly stupid and hose everything. I'll give it a shot tomorrow. Wish me luck! *crosses fingers*
November 5, 201113 yr Author For some reason my BIOS was jacked up, not allowing the machine to boot, so I finally tracked down the issue after a couple hours. The new drive is installed, but I bring up the control panel and it doesn't really give me any options. I have "Too many wrong or missing disks!" I've attached the syslog and a screenshot of the control panel. Any ideas? Thanks syslog-20111105-222639.zip
November 6, 201113 yr The new drive is smaller than the old drive. You must use at least the same size drive as a replacement.
November 6, 201113 yr Author Actually, it's bigger. The new drive is 2TB, the old one was 1.5TB. If you look at one of my previous screenshots, you will see the same odd reading on the old drive.
November 6, 201113 yr So ST31500541AS_6XW0GRH1 is the old drive? You cannot replace a failed drive when two are red.
November 6, 201113 yr Author Yes, that is referring to the old drive, but it has been replaced. The line above it is the new drive size, I'm assuming. This has been my issue from the start. I'm not sure what I'm supposed to do if they are both red. So, if I cannot replace a failed drive when two are red, then does that mean my entire raid is toast? I thought I read somewhere that if two drives fail, you may lose the data on those two drives. It does appear that one of the red ones is functional, but was taken offline. Is there a way I can "repair" and bring the functional one back online, then rebuild the failed drive? As a last resort, what if I removed the failed drive from the array altogether? I would obviously lose the data on that drive, but maybe it would rebuild the functional one?
Archived
This topic is now archived and is closed to further replies.