December 8, 201411 yr 5.0-rc5 Array has been running 100% for months. Have a few brand new (less than 6mo) Seagate NAS 4TB disks in there. Yesterday I shut down the array to add a couple of WD Reds I got on sale. When I brought the array back up, one of the Seagate NAS disks was red-balled. Took the array down, reseated the cables, booted up to a red ball. Reseated cables again, everything was green. Went to work, came home this morning, same disk is red balled. I am going to bed, but tomorrow I will swap the cable on this disk and hope that it fixes it. I ordered a new disk anyway just in case as I realized I don't have any spares. Parity check completed on the 1st with no issues (disk ran fine). Don't think its a power issue, even though adding two disks would increase the draw, I have a Corsair in there and I think its 700w+ (I'll confirm in the morning when I open it to replace the cable). 9 disks and one cache, an AMD CPU that sips power, and a Supermicro SATA card. PS should easily handle the load. Drives are all in iStarUSA 5-in-3 cages. I know its not unreasonable for a disk to fail randomly, but since it happened quite coincidentally with a> adding new disks and b> a reboot after months of uptime, I'm curious what else I should be considering. Hopefully swapping cables works. If not I guess I'll wait til the new drive arrives and run checks on the old drive in the meantime. Thanks.
December 8, 201411 yr Author Just popped it open (couldn't wait) and I have a 500w Corsair. I wouldn't think 500w would be too little for a system with 9 3.5" and one 2.5" drive, all drives are new(er) NAS, Greens, Reds, and the cache is a 2.5" Black. Maybe I have a power problem after all?
December 8, 201411 yr Just popped it open (couldn't wait) and I have a 500w Corsair. I wouldn't think 500w would be too little for a system with 9 3.5" and one 2.5" drive, all drives are new(er) NAS, Greens, Reds, and the cache is a 2.5" Black. Maybe I have a power problem after all? Is it a single rail 12v supply? (You can tell dual rail supplies because they have two 12v current ratings on the spec sheet.)
December 8, 201411 yr Author its a corsair buildermaster, cx series, so i believe it does have a single rail. i will check in the morning. it was on the recommended list back when i initially built the server. i have a 600cx i can try just for testing, but i see that the cx series is not recommended.
December 8, 201411 yr ... Took the array down, reseated the cables, booted up to a red ball. Reseated cables again, everything was green. Went to work, came home this morning, same disk is red balled. ... Once a disk has red-balled, fixing a cabling issue will NOT turn it green. You would have to rebuild the failed disk either on top of itself (a bit risky) or on a new disk.
December 8, 201411 yr Author OK so I will have to wait until the new disk arrives. meantime, should I swap power supplies, or just assume its a bad disk? Swap the cable now or wait and see what the new drive does?
December 8, 201411 yr Pull a SMART report on the drive and post it. It should tell us if the drive is failing.
December 8, 201411 yr Author i tried running a smart report and got error messages. I'll try again and post the error.
December 8, 201411 yr Author I pulled the drive upstairs to my Windows machine; here is the output: C:\Program Files\smartmontools\bin>smartctl /dev/sdc -a smartctl 6.3 2014-07-26 r3976 [x86_64-w64-mingw32-win7-sp1] (sf-6.3-1) Copyright © 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate NAS HDD Device Model: ST4000VN000-1H4168 Serial Number: S30063KN LU WWN Device Id: 5 000c50 06d132a41 Firmware Version: SC43 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5900 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 1.5 Gb/s) Local Time is: Mon Dec 08 14:37:43 2014 USMST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 107) seconds. Offline data collection capabilities: (0x73) SMART execute Offline immediate. Auto Offline data collection on/off supp ort. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 530) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x10bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_ FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 108 099 006 Pre-fail Always - 18466096 3 Spin_Up_Time 0x0003 093 093 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1781 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 071 060 030 Pre-fail Always - 14028387 9 Power_On_Hours 0x0032 093 093 000 Old_age Always - 6722 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 31 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 001 001 000 Old_age Always - 147 190 Airflow_Temperature_Cel 0x0022 076 066 045 Old_age Always - 24 (Min/Max 21/24) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 27 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 1781 194 Temperature_Celsius 0x0022 024 040 000 Old_age Always - 24 (0 20 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 197 000 Old_age Always - 9 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. C:\Program Files\smartmontools\bin>
December 8, 201411 yr Author I ran the "short" test and here are the results: C:\Program Files\smartmontools\bin>smartctl -a /dev/sdc smartctl 6.3 2014-07-26 r3976 [x86_64-w64-mingw32-win7-sp1] (sf-6.3-1) Copyright © 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate NAS HDD Device Model: ST4000VN000-1H4168 Serial Number: S30063KN LU WWN Device Id: 5 000c50 06d132a41 Firmware Version: SC43 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5900 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 1.5 Gb/s) Local Time is: Mon Dec 08 15:11:04 2014 USMST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 107) seconds. Offline data collection capabilities: (0x73) SMART execute Offline immediate. Auto Offline data collection on/off supp ort. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 530) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x10bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_ FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 108 099 006 Pre-fail Always - 18476536 3 Spin_Up_Time 0x0003 093 093 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1782 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 071 060 030 Pre-fail Always - 14028511 9 Power_On_Hours 0x0032 093 093 000 Old_age Always - 6722 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 32 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 001 001 000 Old_age Always - 147 190 Airflow_Temperature_Cel 0x0022 074 066 045 Old_age Always - 26 (Min/Max 23/26) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 28 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 1782 194 Temperature_Celsius 0x0022 026 040 000 Old_age Always - 26 (0 20 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 197 000 Old_age Always - 9 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA _of_first_error # 1 Short offline Completed without error 00% 6722 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. C:\Program Files\smartmontools\bin>
December 8, 201411 yr The drive looks fine. The UDMA_CRC_Error_Count, which had a value of 9, is highly suggestive of a bad or loose cable. If it were me, I would replace the data cable and reconstruct the disk on top of itself. This is pretty easy to do, but takes a while. If you have questions about how to do it, post back. Rebuilding onto a new drive is equally effective and your disk stays untouched and a source of recovery if things go sideways. But while in this state your array is not protected and therefore at risk for a drive to fail with no way to recover. That's why I recommend the other option in your case.
December 9, 201411 yr Author i appreciate the help and probable good news! i will replace the cable tonight. not sure how to reconstruct the disk back onto itself tho.
December 10, 201411 yr Author All green balls. I have a cable on order for the second set of ports on my Supermicro card, will be shutting down the server again in a day or two to add another disk. Hoping I don't run into this problem again!
December 11, 201411 yr Author Roger that. My second 3Ware cable arrives today so I can get another disk installed, then yep, parity check time again.
December 11, 201411 yr Roger that. My second 3Ware cable arrives today so I can get another disk installed, then yep, parity check time again. One thing to do with your new cable to check the drives ends for the condition shown in this post. http://lime-technology.com/forum/index.php?topic=36065.0 The actual pictures are in the second post following this one.
December 12, 201411 yr Author Interesting read and thanks. So let's start a new discussion. I am using iStarUSA 5-in-3 bays, what is the *best* most reliable cable I can get? After this little cable fiasco (and me ordering a new drive in a panic) I am interested in eliminating variables. $50 cables not withstanding, what is the best SATA cable money can buy? Frankly, I have six SATA ports on the MB and another 8 on a Supermicro card. Half tempted to order a second SM card and run 3Ware cables to all my drives and completely eliminate the onboard SATA ports and one-for-one SATA cables.
December 12, 201411 yr Problem is that SATA cables have virtually become a commodity item. They are very cheap (both to manufacture and cost to the customer) and the profit margins are low. They are all manufactured in China and purchased by 'Distributors' in the USA. These distributors have few incentives to do much more than to give a cursory inspection inspection (probably to verify count more than anything) of the product that they are buying. Even if you find a distributor whose cables have the rib in the connector, who is to say that the distributor doesn't change manufacturers for his next order to save a few pennies per cable? Who is to say that if the molding die breaks and the manufacturer decides to save some money on the new die by eliminating the rib? I, personally, feel that the whole SATA connector design is marginal. Both the power connector and the data connector should either have a locking mechanism designed into the connectors or the disconnection force should be much more than is provided in the current design. The principal problem with the current design is that is much to easy to accidentally loosen (or completely uncouple) the connector when you move the cables while installing or removing another drive. (I know this to be a fact as I HAVE DONE IT!!!) I also suspect that vibration could loosen the connector if the cable is constricted in a manner that results in a force on the connection. I am not sure that I know what the propose of the rib is. Is it to assure that the force on the two sets of contacts is sufficient to guarantee a reliable electrical connection? Or is it to increase the force required to remove the cable? Or is a combination of these two? Apparently, what WD did was to eliminate a a portion of the shroud on some of their drives that the locking SATA connectors used for locking their mechanical latch to the drive. Doing this definitely reduces the force to remove a locking cable and, if the rib is not there, there is nothing to retain the cable in the drive connector! It could also reduce the force between the contacts to the point that the connection opens under vibration! Obviously, the best cable from a reliable connection standpoint would be a locking cable with a rib. I looked at the cables I had here. I had six non-locking cable from two different manufacturers. All of them had a rib. I had one locking cable and it did not have a rib. (I suspect that if I had more locking cables they are all inside of my two working servers.) I am not sure where I got these cables from. Most of the motherboards that I have ordered always had SATA data cables packed in the box. In my opinion, the (imperfect) solution is to inspect the cables you have and the new ones as you get them to verify that they have the rib in them. If you are going to use locking cables, verify that they have the rib and the connecting point on the other end has the shroud to actually lock it into position.
December 13, 201411 yr I recommend the locking cables and have had very good luck with them remaining secured for long periods of time. I also recommend drive cages/backplanes (e.g., 5in3s), because once you have them installed and burned in, you do not have to open the server box to swap out drives. This proves very useful in maintaining a reliable server over time. In my early days swapping a drive was often an unpleasant adventure as I had to troubleshoot red balls and weird behavior that occurred from inadvertently knocking something loose. With the drive cages it is a piece of cake! YMMV but I would not set up a server without them.
December 13, 201411 yr I recommend the locking cables and have had very good luck with them remaining secured for long periods of time. While not the case here (since he's using 5 in 3's, locking cables do NOT properly secure to all WD drives, and the non locking cables should be used instead
Archived
This topic is now archived and is closed to further replies.