guyonphone Posted June 4, 2015 Share Posted June 4, 2015 My server started giving me errors on disk 13, and it looks like my raid card rebooted. Attached is my log. syslog_2.zip Link to comment
dgaschk Posted June 4, 2015 Share Posted June 4, 2015 Post a SMART report for disk 13. Link to comment
tyrindor Posted June 4, 2015 Share Posted June 4, 2015 I have 2 servers with a total of 6 SAS2LP cards, connected to almost 48 drives and I never had any issues on RC3 or RC4. Doesn't help your problem, but I don't think that's it. My guess is either Disk13 is dying, or your SAS card is. Link to comment
guyonphone Posted June 5, 2015 Author Share Posted June 5, 2015 Hello dgaschk, Here is my smart report smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.0.4-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: /12:0:1: Product: 0 User Capacity: 600,332,565,813,390,450 bytes [600 PB] Logical block size: 774843950 bytes Physical block size: 3166222336 bytes Lowest aligned LBA: 12602 >> Terminate command early due to bad response to IEC mode page A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. I had this issue just happen, and then resolved it. There is a thread I marked as resolved here https://lime-technology.com/forum/index.php?topic=40210.0 In that thread RobJ says the following: That looked very familiar, and then I realized you are still running 6.0-beta14b. We used to see a fair number of cases of this, where a whole SAS card seems to drop, and whatever drive is accessed next is often dropped from the array, because all or most of the drives on that card can no longer respond. PLEASE upgrade, at least to 6.0-beta15, or the latest -rc. Since beta15 was released, I don't think I've seen a case of this since. SAS card support does not even yet seem to be completely stable, and if I had one, I would constantly check for newer firmware for the card, and newer unRAID releases, even if they are beta, in order to always have the latest SAS support modules and drivers and firmware. Support for them does seem to be improving, with fewer problems now. I thought upgrading to 6 Beta-RC3 would resolve this, but it looks like im still having an issue. I have two different cards Sata Controller 1: SUPERMICRO AOC-SASLP-MV8 Sata Controller 2: SUPERMICRO AOC-SAS2LP-MV8 The SAS2LP I think has a newer firmware I can upgrade, so I will try that, but the older SASLP does not have any firmware update software available to download from the manufacturer. What would you recommend I do next? And thanks in advance for your support :-) Link to comment
dgaschk Posted June 5, 2015 Share Posted June 5, 2015 Power cycle and retry SMART report. Link to comment
guyonphone Posted June 5, 2015 Author Share Posted June 5, 2015 Hello dgaschk, Smart Report is here: smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.0.4-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST3000DM001-1CH166 Serial Number: Z1F2HEZY LU WWN Device Id: 5 000c50 05002695c Firmware Version: CC24 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Thu Jun 4 19:51:08 2015 PDT ==> WARNING: A firmware update for this drive may be available, see the following Seagate web pages: http://knowledge.seagate.com/articles/en_US/FAQ/207931en http://knowledge.seagate.com/articles/en_US/FAQ/223651en SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 584) seconds. Offline data collection capabilities: (0x73) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 340) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x3085) SCT Status supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 112 099 006 Pre-fail Always - 42516568 3 Spin_Up_Time 0x0003 094 094 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1898 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 066 055 030 Pre-fail Always - 30099042447 9 Power_On_Hours 0x0032 081 081 000 Old_age Always - 16833 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 179 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 099 000 Old_age Always - 13 13 13 189 High_Fly_Writes 0x003a 097 097 000 Old_age Always - 3 190 Airflow_Temperature_Cel 0x0022 061 052 045 Old_age Always - 39 (Min/Max 37/39) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 125 193 Load_Cycle_Count 0x0032 061 061 000 Old_age Always - 79277 194 Temperature_Celsius 0x0022 039 048 000 Old_age Always - 39 (0 18 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 4222h+04m+20.742s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 18157344808 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 265931052632 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 16629 - # 2 Short offline Completed without error 00% 5023 - # 3 Short offline Completed without error 00% 5019 - # 4 Short offline Completed without error 00% 168 - # 5 Short offline Interrupted (host reset) 00% 168 - # 6 Short offline Completed without error 00% 167 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Also I am starting to check the firmware of my cards, they are as follows: Sata Controller 1: SUPERMICRO AOC-SASLP-MV8 - Firmware Version: 3.1.0.21 (Latest Version) Sata Controller 2: SUPERMICRO AOC-SAS2LP-MV8- Firmware Version: 4.0.0.1808 (Newer Version Avaliable 4.0.0.1812) I am working on updating the firmware on the AOC-SAS2LP-MV8 right now. Link to comment
guyonphone Posted June 5, 2015 Author Share Posted June 5, 2015 OK, I have updated my SAS card firmware: Sata Controller 1: SUPERMICRO AOC-SASLP-MV8 - Firmware Version: 3.1.0.21 (Latest Version as of 6/4/2015) Sata Controller 2: SUPERMICRO AOC-SAS2LP-MV8- Firmware Version: 4.0.0.1812 (Latest Version as of 6/4/2015) I have also updated to 6 Beta-RC4. However my disk 13 is still disabled. What is the safest way to recover it? Thank You Link to comment
dgaschk Posted June 5, 2015 Share Posted June 5, 2015 http://lime-technology.com/wiki/index.php/Troubleshooting#What_do_I_do_if_I_get_a_red_ball_next_to_a_hard_disk.3F Link to comment
tdallen Posted June 5, 2015 Share Posted June 5, 2015 I consistently have had these problems on 12, 14b, and 15 (I have a single SAS2LP). It only seems to affect some hardware configurations, but it's common enough to worry. I'll upgrade to RC4 this weekend and see what happens. FYI, I also have 1812 and power cycling brings back the missing drive with a healthy SMART report. Link to comment
guyonphone Posted June 5, 2015 Author Share Posted June 5, 2015 My Parity is rebuilding. I think that my failure occurred this time during heavy writes to the server. I think I was copying too and from the server on different machines with 3 separate copies going at one time, and the mover may have been running as well. Would it be beneficial for me to move my cache drive to a sata port on my motherboard? I suspect that I may have just overloaded the SAS card. Link to comment
dgaschk Posted June 5, 2015 Share Posted June 5, 2015 My Parity is rebuilding. I think that my failure occurred this time during heavy writes to the server. I think I was copying too and from the server on different machines with 3 separate copies going at one time, and the mover may have been running as well. Would it be beneficial for me to move my cache drive to a sata port on my motherboard? I suspect that I may have just overloaded the SAS card. If this is true then you need a new SAS card. Working cards do not fail when loaded. There is no way to overload a SATA card. It will just slow down. Link to comment
guyonphone Posted June 5, 2015 Author Share Posted June 5, 2015 Hello dgaschk, I don't know specifically if the card is bad, it was just speculation on my part. I know as much as I see in the logs. If this continues to happen at times of high load, then I will assume a bad card, but as of now its not enough information for a definitive analysis. Looking at the log of the failue, does it provide any hints as to what went wrong? Or are you as much in the dark as I am? Thanks for your help! Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.