neal_is_king Posted January 29 Share Posted January 29 (edited) Edit: After posting it occurs to me my post is quite long so here is a summary: Summary Array is 4 12TB Toshiba N300 drives, just under a year old Can't run short or extended SMART test on any of them - Completed: unknown failure 90% 3 of them logged a Error: ABRT trying to run the test and now my BIOS is telling me they're dead All plagued by a Seek_Error_Rate issue that is apparently quite common for these drives I think they're OK but maybe not Hi all, This likely is not an Unraid issue exactly, but the drives in question are the totality of my parity and data drives. I'm a little bit worried but also wonder if it's a big nothingburger. My current build has 4 Toshiba N300 drives, 2 data and 2 parity, and 2 970 EVO Plus cache drives in a RAID1 btrfs pool. I bought the mechanical drives in the same month about a year ago from 3 different stores, not sure how I did w/r/t different batches. The way my present issue came about is, last week I started playing with new DVR software and was also downloading a lot of Linux ISOs lets say and my cache pool was under a bit of stress. I think. I'm not sure exactly what happened, but I woke up the next day and my box was unreachable. Apparently it rebooted and was stuck in the BIOS which wouldn't boot until I acknowledged that one of the NVME drives was dying or dead. Turns out to be parity2, that's kind of a drag and I should probably RMA it because it's not that old but OK, order a new one, rebuild the pool, done. Somewhere throughout this process it occurs to me that I haven't run any SMART tests since I finished preclearing these 12TB drives 6808 LifeTime hours ago. I kick off a short test for each, but none of them finish! I just get Completed: unknown failure 90% in the self-test history, and now two of them have an Error: ABRT and one has Error: ICRC, ABRT in the error log as well. I powered off when I received a new heat sync for the replacement cache drive from above, and now the machine won't boot because the BIOS wants to tell me about these 3 SMART errors. After seeing very scary sounding warnings about replacing the drives immediately to prevent data loss, with some extra steps I am still able to boot. There is a BIOS option to not check SMART during post, which I have not changed yet. So back in Unraid, I try and try to get one of these drives to run a short SMART test, but it's the same thing every time, Completed: unknown failure 90%. I logged maybe 6 or 7 failures like that for each drive. I'm currently running an early parity check just to see if I get any errors or if the stress does anything to the SMART reporting. It's 70% done and so far so good. Also if I run the short SMART *during* the parity sync, it gets further along - Completed: unknown failure 10% - not sure what to make of that either. I tried an extended test one one of them too, also fails with 90% remaining. All of these drives are (quite annoyingly) plagued by the Seek_Error_Rate issue that seems to be just par for the course with them. My strategy there is just to ignore that metric completely. And all 4 of them report SMART overall-health self-assessment test result: PASSED. I don't know what SMART does outside of user initiated self tests, I assume it's reporting metrics without running a self test because I almost never do that. As for troubleshooting, I have tried re-seating the SATA cables, replacing the SATA cables, and moving all the drives from the motherboard to an HBA I have in a box from my previous build. Nothing has made a difference in getting these self tests to finish successfully. The only thing I can think of to try next is run SMART with the drives in an external enclosure plugged into another machine, but I don't think that would make a difference as the test is running "on" the drive itself? I also have no idea if the Error: ABRT would be fixed if I did get one to finish, which would let me keep that check on in the BIOS. I have a sense that these drives are fine to be honest. Reallocated_Sector_Ct, Current_Pending_Sector, Offline_Uncorrectable all fine. I'm also not really experienced enough in this to be confident in that assessment. I am curious what you guys would do in my situation, or if anyone has any advice or thoughts to offer. I am out of the return window at all 3 stores, and everything I've read about this Seek_Error_Rate issue says Toshiba just sends refurb drives that quickly have the same issue too. I haven't found anything about SMART tests failing to complete on these drives however. I attached diagnostics if anyone would like to see the SMART history. I have redundant, offsite backups of important data but there's plenty of non-critcal data I would strongly prefer to keep. Thanks! Edited January 30 by neal_is_king Quote Link to comment
Solution JorgeB Posted January 30 Solution Share Posted January 30 If they are all assigned to the array run a parity check, for drive testing purposes it's basically the same as running an extend SMART test. Quote Link to comment
neal_is_king Posted January 30 Author Share Posted January 30 Thanks for your reply JorgeB. See attached screenshots; parity check ran successfully. Would you not be worried in my situation or would you be looking to RMA and/or replace drives? I will probably try yanking one of the drives and running SMART from an external enclosure attached to another machine. I read somewhere in my various searching over the last few days that these drives can abort if SMART status is polled too often while the test is running. I see the unraid UI updating live when I run a test, 10%, 20%, etc... wonder if that's killing the SMART run. Quote Link to comment
JorgeB Posted January 30 Share Posted January 30 13 minutes ago, neal_is_king said: Would you not be worried in my situation Nope, parity check completed without errors and SMART looks good and, except the seek error, but like you've mentioned it's a know issue with those drives, I had it myself. Quote Link to comment
neal_is_king Posted January 30 Author Share Posted January 30 1 hour ago, JorgeB said: I had it myself. Indeed, I saw that in the many threads I read through before I posted - I was actually hoping you would reply to this! I did end up yanking one of the drives and running a SMART short test with it plugged into another machine, failed "Unknown failure". I guess I just can't run self tests on these 4 drives for whatever reason. Probably won't be buying any more because that's quite odd, but as long as the rest of the attributes stay healthy I'll just be hoping for the best. Thank you for your help JorgeB! 1 Quote Link to comment
neal_is_king Posted January 30 Author Share Posted January 30 In case it might help someone with a similar issue in the future, here is the SMART log of one of the disks mentioned above. # smartctl -a /dev/sdc smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.19.17-Unraid] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Toshiba N300/MN NAS HDD Device Model: TOSHIBA HDWG21C Serial Number: redacted LU WWN Device Id: 5 000039 b28cb91ec Firmware Version: 0601 User Capacity: 12,000,138,625,024 bytes [12.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database 7.3/5417 ATA Version is: ACS-3 T13/2161-D revision 5 SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Mon Jan 30 14:08:35 2023 PST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED See vendor-specific Attribute list for marginal Attributes. General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 65) The previous self-test completed having a test element that failed and the test element that failed is not known. Total time to complete Offline data collection: ( 120) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: (1177) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0 2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0 3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 7089 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 650 5 Reallocated_Sector_Ct 0x0033 100 100 050 Pre-fail Always - 0 7 Seek_Error_Rate 0x000b 086 001 050 Pre-fail Always In_the_past 0 8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0 9 Power_On_Hours 0x0032 083 083 000 Old_age Always - 7026 10 Spin_Retry_Count 0x0033 100 100 030 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 91 23 Helium_Condition_Lower 0x0023 100 100 075 Pre-fail Always - 0 24 Helium_Condition_Upper 0x0023 100 100 075 Pre-fail Always - 0 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 114 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 76 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 658 194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 37 (Min/Max 18/51) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 220 Disk_Shift 0x0002 100 100 000 Old_age Always - 17825794 222 Loaded_Hours 0x0032 083 083 000 Old_age Always - 7006 223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0 224 Load_Friction 0x0022 100 100 000 Old_age Always - 0 226 Load-in_Time 0x0026 100 100 000 Old_age Always - 591 240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0 SMART Error Log Version: 1 ATA Error Count: 2 CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 2 occurred at disk power-on lifetime: 6986 hours (291 days + 2 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 04 53 00 91 f4 2c 02 Error: ABRT Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- b0 d4 00 82 4f c2 00 00 00:59:05.943 SMART EXECUTE OFF-LINE IMMEDIATE b0 d0 01 00 4f c2 00 00 00:59:05.941 SMART READ DATA ec 00 01 00 00 00 00 00 00:59:05.935 IDENTIFY DEVICE ec 00 01 00 00 00 00 00 00:59:05.934 IDENTIFY DEVICE 60 20 50 08 00 00 40 00 00:58:53.442 READ FPDMA QUEUED Error 1 occurred at disk power-on lifetime: 6986 hours (291 days + 2 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 04 53 00 90 f4 2c 02 Error: ABRT Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- b0 d4 00 81 4f c2 00 00 00:51:51.844 SMART EXECUTE OFF-LINE IMMEDIATE b0 d0 01 00 4f c2 00 00 00:51:51.842 SMART READ DATA ec 00 01 00 00 00 00 00 00:51:51.836 IDENTIFY DEVICE ec 00 01 00 00 00 00 00 00:51:51.835 IDENTIFY DEVICE b0 d0 01 00 4f c2 00 00 00:51:29.383 SMART READ DATA SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed: unknown failure 10% 6988 0 # 2 Short offline Completed: unknown failure 60% 6987 0 # 3 Short offline Completed: unknown failure 90% 6987 0 # 4 Short offline Completed: unknown failure 90% 6986 0 # 5 Short offline Completed: unknown failure 90% 6986 0 # 6 Extended offline Completed: unknown failure 90% 6986 0 # 7 Extended captive Completed: unknown failure 90% 6986 0 # 8 Short captive Completed: unknown failure 90% 6986 0 # 9 Short offline Completed: unknown failure 90% 6986 0 #10 Short offline Completed without error 00% 0 - #11 Short offline Aborted by host 60% 0 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Quote Link to comment
itimpi Posted January 30 Share Posted January 30 That SMART report suggest an unhealthy drive as it could not even complete the short SMART test let alone the extended version. Quote Link to comment
neal_is_king Posted January 31 Author Share Posted January 31 Yeah... it's unnerving, that's for sure. I think I'll order an IronWolf 12TB and preclear it so I have a standby if anything starts failing. And maybe I'll replace a drive every few months for a year. Can't return... don't see much point in an RMA when this happened with all 4 drives, I'm not going to trust the replacements. I only went with N300 because the mount holes were the right spacing for my Node 804 drive cage. It took a few months to get mounting adapters from Fractal Design and I didn't want to wait. So the $1200 lesson here is patience. And don't buy Tosihba drives I guess. Quote Link to comment
neal_is_king Posted February 1 Author Share Posted February 1 Man, I just can't get this out of my head. I want to believe these drives are alright but I just can't. Maybe if it was one but all four of them can't run SMART self tests? What is that... I ordered 2 16TB drives from different manufacturers that are not Toshiba, and I'm going to move my whole setup to 1 parity, 1 data for a time. I read in a few places I might be able to get a refund when I RMA these 4 N300s. Fingers crossed. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.