Preclear plugin


Recommended Posts

I don't see the "mail" option when I start a pre-clear?  I see it as an option in video... but not in my options?

 

I get the mail notifications for other stuff like array health and what not..

 

Jim

 

If you're using Joe L. version, you don't. The reason is that the "mail" command used before unRAID 6 could interfere with the built-in Notification system. I've adapted the new system into preclear_disk and submit it here, but there's no response from Joe L.

 

In the mean time, you can use bjp999 version of the script that's stored on my github:

 

 

There's a version hosted here: http://bit.ly/1G44UhZ . This version have fast post-read and notifications enabled.

 

You can add this with this command:

wget http://bit.ly/1G44UhZ -O /boot/config/plugins/preclear.disk/preclear_disk.sh

 

readvz is already downloaded by the plugin.

 

I uninstalled Preclear plugin and reinstalled using PLG link from first page, then uploaded the preclear_disk.sh you referenced in quoted text. Now it is offering to Preclear members of my array & cache disk? Something is not right.

Screenshot_2.png.188277d2fabfa56cf3779b95b168ce51.png

Link to comment

I don't see the "mail" option when I start a pre-clear?  I see it as an option in video... but not in my options?

 

I get the mail notifications for other stuff like array health and what not..

 

Jim

 

If you're using Joe L. version, you don't. The reason is that the "mail" command used before unRAID 6 could interfere with the built-in Notification system. I've adapted the new system into preclear_disk and submit it here, but there's no response from Joe L.

 

In the mean time, you can use bjp999 version of the script that's stored on my github:

 

 

There's a version hosted here: http://bit.ly/1G44UhZ . This version have fast post-read and notifications enabled.

 

You can add this with this command:

wget http://bit.ly/1G44UhZ -O /boot/config/plugins/preclear.disk/preclear_disk.sh

 

readvz is already downloaded by the plugin.

 

I uninstalled Preclear plugin and reinstalled using PLG link from first page, then uploaded the preclear_disk.sh you referenced in quoted text. Now it is offering to Preclear members of my array & cache disk? Something is not right.

Are you using the 2015.09.28 version? If so, I just messed with some detection code, so it's possible that it broke something. I'll revert it soon.

Link to comment

Thanks.. I just loaded this on my unRAID1a setup. It seems to be working fine so far. I picked up six of the Seagate 3TB drives for $75 each from Newegg last week. In the past I typically just loaded my drives without testing them and didn't have any problems. But it seems like the drives today are less reliable than in the past. So I wanted to start using the pre-clear script. And this plugin certainly makes things much easier. Thanks!

Link to comment

The script seems to be removing some linefeeds from the  analyze_for_errors output and the cats of the smart reports..  It didn't do that before.

 

What the e-mail looks like now:

 

Event: Preclear ST4000VN000-1H4168 ZXXXXXX
Subject: Preclear: PASS! Preclearing Disk sdm Finished!!!
Description: Preclear: PASS! Preclearing Disk sdm Finished!!! Cycle 1 of 1
Importance: normal

========================================================================1.15
== invoked as: /boot/config/plugins/preclear.disk/preclear_disk.sh -M 4 -o 2 -c 1 -f -J /dev/sdm
== ST4000VN000-1H4168 ZXXXXXXX
== Disk /dev/sdm has been successfully precleared
== with a starting sector of 1 
== Ran 1 cycle
==
== Using :Read block size = 1000448 Bytes
== Last Cycle`s Pre Read Time : 10:05:41 (110 MB/s)
== Last Cycle`s Zeroing time : 8:01:11 (138 MB/s)
== Last Cycle`s Post Read Time : 10:16:20 (108 MB/s)
== Last Cycle`s Total Time : 28:24:38
==
== Total Elapsed Time 28:24:38
==
== Disk Start Temperature: 32C
==
== Current Disk Temperature: 35C, 
==
============================================================================
** Changed attributes in files: /tmp/smart_start_sdm /tmp/smart_finish_sdm ATTRIBUTE NEW_VAL OLD_VAL FAILURE_THRESHOLD STATUS RAW_VALUE Raw_Read_Error_Rate = 117 100 6 ok 133493080 Spin_Retry_Count = 100 100 97 near_thresh 0 End-to-End_Error = 100 100 99 near_thresh 0 Airflow_Temperature_Cel = 65 68 45 near_thresh 35 Temperature_Celsius = 35 32 0 ok 35 No SMART attributes are FAILING_NOW 0 sectors were pending re-allocation before the start of the preclear. 0 sectors were pending re-allocation after pre-read in cycle 1 of 1. 0 sectors were pending re-allocation after zero of disk in cycle 1 of 1. 0 sectors are pending re-allocation at the end of the preclear, the number of sectors pending re-allocation did not change. 0 sectors had been re-allocated before the start of the preclear. 0 sectors are re-allocated at the end of the preclear, the number of sectors re-allocated did not change. 
============================================================================
============================================================================
==
== S.M.A.R.T Initial Report for /dev/sdm 
==
Disk: /dev/sdm smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.1.7-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: ST4000VN000-1H4168 Serial Number: ZXXXXXXX LU WWN Device Id: 5 000c50 07ba4cf8c Firmware Version: SC46 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5900 rpm Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Tue Sep 29 21:24:04 2015 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 128) seconds. Offline data collection capabilities: (0x73) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 505) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x10bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Str
==
============================================================================

 

What is used to look like in the e-mail (using the original Joe script and V5 of unraid:

 

========================================================================1.15
== invoked as: ./preclear_disk.sh -M 4 -m [email protected] /dev/sdl
== WDCWD40EZRX-00SPEB0   WD-WXXXXXXXX
== Disk /dev/sdl has been successfully precleared
== with a starting sector of 1
== Ran 1 cycle
==
== Using :Read block size = 8388608 Bytes
== Last Cycle's Pre Read Time  : 10:28:36 (106 MB/s)
== Last Cycle's Zeroing time   : 9:58:54 (111 MB/s)
== Last Cycle's Post Read Time : 23:29:46 (47 MB/s)
== Last Cycle's Total Time     : 43:58:25
==
== Total Elapsed Time 43:58:25
==
== Disk Start Temperature: 23C
==
== Current Disk Temperature: 28C,
==
============================================================================
** Changed attributes in files: /tmp/smart_start_sdl  /tmp/smart_finish_sdl
ATTRIBUTE   NEW_VAL OLD_VAL FAILURE_THRESHOLD STATUS      RAW_VALUE
Temperature_Celsius =   124     129            0        ok          28
No SMART attributes are FAILING_NOW

0 sectors were pending re-allocation before the start of the preclear.
0 sectors were pending re-allocation after pre-read in cycle 1 of 1.
0 sectors were pending re-allocation after zero of disk in cycle 1 of 1.
0 sectors are pending re-allocation at the end of the preclear,
the number of sectors pending re-allocation did not change.
0 sectors had been re-allocated before the start of the preclear.
0 sectors are re-allocated at the end of the preclear,
the number of sectors re-allocated did not change.
============================================================================
============================================================================
==
== S.M.A.R.T Initial Report for /dev/sdl
==
Disk: /dev/sdl
smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD40EZRX-00SPEB0
Serial Number:    WD-XXXXXXXXX
Firmware Version: 80.00A80
User Capacity:    4,000,787,030,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   9
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Sat Apr 25 18:13:13 2015 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80)	Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: 		 (52380) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 255) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x7035)	SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate     0x002f   100   253   051    Pre-fail  Always       -       0
3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       4
5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       0
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       4
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       2
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       6
194 Temperature_Celsius     0x0022   129   120   000    Old_age   Always       -       23
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   253   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
1        0        0  Not_testing
2        0        0  Not_testing
3        0        0  Not_testing
4        0        0  Not_testing
5        0        0  Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
==
============================================================================

 

Any thoughts to why?  It would  be nice if this was more readable...

 

But great job, otherwise, on this plugin!  It's nice to have a gui for it!  I always had to look back and see what command line parameters I used in the past for any new pre-clears

 

Jim

 

Link to comment

You might need some quotes around the echo statement in the send_mail subroutine.

# gfjardim - add notification system capability without breaking legacy mail.

send_mail() {

  subject=$(echo ${1} | tr "'" '`' )

  description=$(echo ${2} | tr "'" '`' )

  message=$(echo ${3} | tr "'" '`' )

  recipient=${4}

  if [ -f "$notify_script" ]; then

    $notify_script -e "Preclear ${model} ${serial}" -s """${subject}""" -d """${description}""" -m """${message}""" -i "normal ${notify_channels}"

  else

    echo -e "${message}" | mail -s "${subject}" "${recipient}"

  fi

}

 

Changed to:

# gfjardim - add notification system capability without breaking legacy mail.

send_mail() {

  subject=$(echo ${1} | tr "'" '`' )

  description=$(echo ${2} | tr "'" '`' )

  message=$(echo "${3}" | tr "'" '`' )

  recipient=${4}

  if [ -f "$notify_script" ]; then

    $notify_script -e "Preclear ${model} ${serial}" -s """${subject}""" -d """${description}""" -m """${message}""" -i "normal ${notify_channels}"

  else

    echo -e "${message}" | mail -s "${subject}" "${recipient}"

  fi

}

Someone else can probably explain why it's needed for the cat output and not for the "\n" inserted into the variable!

 

but if I do this:

my_message="`cat /tmp/smart_start_sdl` \n";m2=$(echo ${my_message} | tr "'" '`' ) ; echo "$m2"

I get this:

Disk: /dev/sdl smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.1.7-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: ST4000VN000-1H4168 Serial Number: ZZZZZZZ LU WWN Device Id: 5 000c50 07b9b59ce Firmware Version: SC46 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5900 rpm Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Tue Sep 29 21:23:31 2015 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 107) seconds. Offline data collection capabilities: (0x73) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 502) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x10bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 100 100 006 Pre-fail Always - 12496 3 Spin_Up_Time 0x0003 096 096 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 4 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 100 253 030 Pre-fail Always - 11 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 0 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 4 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 253 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 070 070 045 Old_age Always - 30 (Min/Max 22/30) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 4 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 4 194 Temperature_Celsius 0x0022 030 040 000 Old_age Always - 30 (0 22 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 253 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. \n

 

but if I do this:

my_message="`cat /tmp/smart_start_sdl` \n";m2=$(echo "${my_message}" | tr "'" '`' ) ; echo "$m2"

I get the correctly formatted output.

 

Disk: /dev/sdl
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.1.7-unRAID] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     ST4000VN000-1H4168
Serial Number:    ZXXXXXX
LU WWN Device Id: 5 000c50 07b9b59ce
Firmware Version: SC46
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5900 rpm
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Sep 29 21:23:31 2015 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  107) seconds.
Offline data collection

 

Now I haven't tried this in the script.  If I had a small disk to try this on I would..  I got rid of all mine.

Link to comment

 

Is there a fix for this small problem

I am using Chrome as my browser

I have a difficult time getting the Plugin to run after it starts

It takes 10 or 20 or more clicks of the Start button to start the plugin and just as many to the X button to stop the plugin.

 

I see this same behavior as well, regardless of which browser I use.  However, it might be related to the array disks being spun down.  If the majority of the disks in my system are spun down, it's sometimes impossible to get the preclear plugin popup to appear.  If the disks are spun up, then it's usually only one or two clicks to get the popup.

Link to comment

Question: If you are maxed out on drives (according to your unraid license) but you have SATA Ports available, can you still preclear a drive to test/check it? I guess I am asking if preclear is considered to be doing something to a volume outside the array?

 

Depends.    If you're running v5, there's no limit on how many physical devices can be attached to the system.  If you're running v6, the license limits the number of physical devices you can have in the system UNLESS you have a Pro license.

 

Link to comment

Question: If you are maxed out on drives (according to your unraid license) but you have SATA Ports available, can you still preclear a drive to test/check it? I guess I am asking if preclear is considered to be doing something to a volume outside the array?

 

Depends.    If you're running v5, there's no limit on how many physical devices can be attached to the system.  If you're running v6, the license limits the number of physical devices you can have in the system UNLESS you have a Pro license.

Really??  what does it do?  disable them?

How can it? 

Link to comment

You'll get a "too many attached devices" message and the array will refuse to start.

 

This limit was imposed starting with v6 => it's 3 devices for the trial; 6 for Basic, and 12 for Plus.    The 25 devices supported by Pro WAS initially also a limit for attached devices, but that's been removed.

 

 

Link to comment

I did not know that..  That seems stupid? Why limit the devices attached? Clearly limit the devices in the array..  but physical devices???

I don't remember the whole story, but there was considerable discussion about many ways to handle this, and this is what was decided.  I believe it had something to do with the fact that the number of array drives used to be the primary quantity, but now you can have multiple cache drives, other managed drives, and shortly multiple parity drives, all due to LimeTech development.  It's not a limitation, because Tom increased the numbers.  Before you could have 6 array drives, now you can have 12 of any kind of drive.  It's a change that recognizes the advances in unRAID.

Link to comment

There was indeed a lot of discussion about limiting the attached devices ... and as Rob noted, the way it ended up is with the 3 (trial), 6 (Basic), and 12 (Plus) device limits.    Limetech did eliminate the 25 device limit for Pro ... so folks with maxed out systems who have a few spare ports for other uses (e.g. pre-clears and/or occasional attached drives that are "outside" of the array) won't encounter any limits.

 

It can be an occasional nuisance, but it's easily avoided by just upgrading to a Pro key  :)

 

 

Link to comment
  • Squid featured, unfeatured and pinned this topic
  • Squid unpinned this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.