October 26, 201312 yr Nice to see the 2nd cycle was faster than the first by some 5 hours... I'm assuming that's normal. Yes, that's normal. The pre-read cycle is only done for the first cycle; so subsequent cycles save that time. Can I expect even faster for the 3rd? No
October 26, 201312 yr Joe can confirm, but I believe you understand it correctly. The pre-read simply reads the entire drive to confirm all is readable; the subsequent pre-clear pass writes all zeroes and the pre-clear signature. These take approximately the same time, as they're simply linear passes through the disk. Then the post-read does a complete read, but NOT simply linear -- it's designed to exercise the seek mechanism enough to have high confidence that all is working well. That's why the final phase takes so much longer than the initial pre-read [in fact, the post-read takes longer than the first two phases together]. Joe doesn't like the characterization "random" -- and I'm sure he's correct, it's not actually random; but it's certainly not linear either [since he wrote it, I'm sure he could outline the algorithm ... but it's not important => the key is that it does indeed read every sector -- just not in order]. Wrong. The pre-read and post-read are EXACTLY the same in how they read the data. BOTH read linearly AND ALSO intersperse random blocks in addition to the first and last sectors. pre-read throws away what is read by sending it to /dev/null. The post-read verifies what was read checking it for all zeros. That verification is what adds the extra time. Joe L.
October 26, 201312 yr Author Considering how long this thing takes... and how I know I personally can attribute adjectives such as "annoying", "tedious" and "omgwtf" to the process... I can imagine, Joe L, as the author (right?), that you may not get as much appreciation for your work on it as you probably deserve. Somewhat like the tow truck driver that's doing a job that everyone loathes but, for the good of everyone, must get done... I want to say "thank you Joe L". So thanks
October 26, 201312 yr Author Day 3: Ok, one of the drives is finished! Yay. A whole bunch of gobbledegook came to my email. I guess this is all good news, but what do I know: ========================================================================1.13 == invoked as: ./preclear_disk.sh -r 65536 -w 65536 -b 2000 -A -c 3 -M 4 /dev/sdc == ST2000VN000-1H3164 W1H24DNG == Disk /dev/sdc has been successfully precleared == with a starting sector of 64 == Ran 3 cycles == == Using :Read block size = 65536 Bytes == Last Cycle's Pre Read Time : 4:54:28 (113 MB/s) == Last Cycle's Zeroing time : 4:07:36 (134 MB/s) == Last Cycle's Post Read Time : 9:04:27 (61 MB/s) == Last Cycle's Total Time : 13:13:02 == == Total Elapsed Time 44:29:20 == == Disk Start Temperature: 34C == == Current Disk Temperature: 39C, == ============================================================================ ** Changed attributes in files: /tmp/smart_start_sdc /tmp/smart_finish_sdc ATTRIBUTE NEW_VAL OLD_VAL FAILURE_THRESHOLD STATUS RAW_VALUE Raw_Read_Error_Rate = 111 100 6 ok 39639936 Spin_Retry_Count = 100 100 97 near_thresh 0 End-to-End_Error = 100 100 99 near_thresh 0 High_Fly_Writes = 71 100 0 ok 29 Airflow_Temperature_Cel = 62 66 45 near_thresh 38 Temperature_Celsius = 38 34 0 ok 38 No SMART attributes are FAILING_NOW 0 sectors were pending re-allocation before the start of the preclear. 0 sectors were pending re-allocation after pre-read in cycle 1 of 3. 0 sectors were pending re-allocation after zero of disk in cycle 1 of 3. 0 sectors were pending re-allocation after post-read in cycle 1 of 3. 0 sectors were pending re-allocation after zero of disk in cycle 2 of 3. 0 sectors were pending re-allocation after post-read in cycle 2 of 3. 0 sectors were pending re-allocation after zero of disk in cycle 3 of 3. 0 sectors are pending re-allocation at the end of the preclear, the number of sectors pending re-allocation did not change. 0 sectors had been re-allocated before the start of the preclear. 0 sectors are re-allocated at the end of the preclear, the number of sectors re-allocated did not change. ============================================================================ ============================================================================ == == S.M.A.R.T Initial Report for /dev/sdc == Disk: /dev/sdc smartctl 5.40 2010-10-16 r3189 [i486-slackware-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Device Model: ST2000VN000-1H3164 Serial Number: W1H24DNG Firmware Version: SC42 User Capacity: 2,000,398,934,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 9 ATA Standard is: Not recognized. Minor revision code: 0x001f Local Time is: Thu Oct 24 11:23:16 2013 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 97) seconds. Offline data collection capabilities: (0x73) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x10bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 100 100 006 Pre-fail Always - 50640 3 Spin_Up_Time 0x0003 099 099 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 2 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 100 253 030 Pre-fail Always - 13 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 1 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 2 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 253 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 066 065 045 Old_age Always - 34 (Min/Max 26/35) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 2 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 2 194 Temperature_Celsius 0x0022 034 040 000 Old_age Always - 34 (0 18 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 253 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. == ============================================================================ ============================================================================ == == S.M.A.R.T Final Report for /dev/sdc == Disk: /dev/sdc smartctl 5.40 2010-10-16 r3189 [i486-slackware-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Device Model: ST2000VN000-1H3164 Serial Number: W1H24DNG Firmware Version: SC42 User Capacity: 2,000,398,934,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 9 ATA Standard is: Not recognized. Minor revision code: 0x001f Local Time is: Sat Oct 26 07:52:36 2013 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 97) seconds. Offline data collection capabilities: (0x73) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x10bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 111 099 006 Pre-fail Always - 39639936 3 Spin_Up_Time 0x0003 099 099 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 2 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 100 253 030 Pre-fail Always - 439500 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 46 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 2 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 071 071 000 Old_age Always - 29 190 Airflow_Temperature_Cel 0x0022 062 057 045 Old_age Always - 38 (Min/Max 26/43) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 2 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 2 194 Temperature_Celsius 0x0022 038 043 000 Old_age Always - 38 (0 18 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. == ============================================================================
October 26, 201312 yr Wrong. The pre-read and post-read are EXACTLY the same in how they read the data. BOTH read linearly AND ALSO intersperse random blocks in addition to the first and last sectors. pre-read throws away what is read by sending it to /dev/null. The post-read verifies what was read checking it for all zeros. That verification is what adds the extra time. Joe L. Oh, that's interesting! Thanks for the clarification. It's really impressive how fast modern drives handle that pre-read process in that case.
October 26, 201312 yr Day 3: Ok, one of the drives is finished! Yay. A whole bunch of gobbledegook came to my email. I guess this is all good news, but what do I know: Well, you know the pre-clear was successful and there are no sectors pending re-allocation or re-allocated. In other words, a perfectly healthy drive. Joe L.
October 26, 201312 yr Wrong. The pre-read and post-read are EXACTLY the same in how they read the data. BOTH read linearly AND ALSO intersperse random blocks in addition to the first and last sectors. pre-read throws away what is read by sending it to /dev/null. The post-read verifies what was read checking it for all zeros. That verification is what adds the extra time. Joe L. Oh, that's interesting! Thanks for the clarification. It's really impressive how fast modern drives handle that pre-read process in that case. Most disks tend to average about 100MB/s. Yours are slightly faster at about 115MB/s.
October 26, 201312 yr Joe can confirm, but I believe you understand it correctly. The pre-read simply reads the entire drive to confirm all is readable; the subsequent pre-clear pass writes all zeroes and the pre-clear signature. These take approximately the same time, as they're simply linear passes through the disk. Then the post-read does a complete read, but NOT simply linear -- it's designed to exercise the seek mechanism enough to have high confidence that all is working well. That's why the final phase takes so much longer than the initial pre-read [in fact, the post-read takes longer than the first two phases together]. Joe doesn't like the characterization "random" -- and I'm sure he's correct, it's not actually random; but it's certainly not linear either [since he wrote it, I'm sure he could outline the algorithm ... but it's not important => the key is that it does indeed read every sector -- just not in order]. Wrong. The pre-read and post-read are EXACTLY the same in how they read the data. BOTH read linearly AND ALSO intersperse random blocks in addition to the first and last sectors. pre-read throws away what is read by sending it to /dev/null. The post-read verifies what was read checking it for all zeros. That verification is what adds the extra time. Joe L. You wrote the utility, so I'll defer to your comments ... but the timings don't seem to make sense. Consider: a 4TB drive typically takes ~ 9 hrs to do a pre-read. This is an average of over 123MB/s for the entire 9 hours. This is certainly consistent with a sequential sweep of the disk, with not much additional seek time. Seeks take a LONG time relative to other disk activity ... typically 15ms or so for a seek/settle/latency cycle to actually start reading data. It IS true that the subsequent pre-clear cycle is slightly faster (~ 40 minutes quicker), so a few interspersed random seeks explain this timing difference [i always wondered why there was this small difference -- now I know ] I had always assumed that the MUCH slower post-read was due to a lot of interspersed seeks, which would easily slow things down a LOT (and since the post-read takes longer than the pre-read and pre-clear cycles combined, that makes sense). You said the extra time is purely due to verification?? Realistically, there's NO difference in read times with a destination of "nowhere" (e.g. /dev/null) vs. a memory buffer. And the CPU could certainly validate a buffer full of zeroes faster than the next read request would fill a different buffer => so it would seem this should effectively add NO time to the process. Does the utility do these sequentially? [i.e. read - validate - THEN do another read - validate - etc.] instead of in parallel? I had always thought the much longer times for this phase were easily understood due to a lot of seek time; but apparently that's not the case. If the utility is waiting for an I/O completion; then validating the data; and only then initiating another I/O, that also explains it ... but that effectively doubles the time needed to do this !!
Archived
This topic is now archived and is closed to further replies.