Preclear plugin


Recommended Posts

unRaid 6.1.4

Fresh preclear and plugin as of last week.

 

Dug a bit further into my earlier experience with the plugin not working.  To recap, when I load the plugin, I see the drive, but the "Start Preclear" link does not work (Red Arrow).  It appears there is an error in the javascript (Blue Arrow) but that is for the SEMRush extension which is not loading because the URL is local and not a website (as far as I can tell.  I disabled the extension, but made no difference).

 

20151209-buxv-203kb.jpg

 

Also appears that the HTML contains "nothing" (Yellow Arrow).  This prompted me to take a look at the HTML source.  I see the javascript there for the start preclear function.  But its not loading thus the link has no value.

 

Tried to load in IE9, nothing displays:

20151209-iovx-52kb.jpg

 

Tried to load in Waterfox, same thing:

20151209-5flo-77kb.jpg

 

And finally in Safari, same thing:

20151209-ikbm-74kb.jpg

 

Any further thoughts on what could be wrong?  When I do a traditional preclear via telnet, then I can see the update from the manual preclear that I initiated...

20151209-hoth-145kb.jpg

 

Thanks in advance!

Happy to do any diagnostics that would make sense...

 

Link to comment

Thanks for the ideas...

 

Not sure if the card is defective or what is going on... I did update to the lastest firmware. The preclear script will not run (minor issue). Cache rebuilding is running at about 10-15MB/s with 3 disks (Green 2TB, 1TB and blue 2.5" 640GB). This seems really slow.. Maybe I am expecting to much... Further no temperature or smart status is being reported

 

Anyways I am looking to replace the controller right now unless someone wants to help troubleshoot it :-)

 

Thanks,

 

Mike

 

Ok, Preclear plugin is working on the onboard SATA controller, but it is failing on the 9650SE-24M8. It gets to the prompt Yes to continue and stops.

 

A few other people have also had this when running disks from something other than the onboard SATA controller.

One user fixed his by updating the firmware on his SAS card.

 

What I think is actually happening, is that the preclear script is detecting a different situation(or non-standard, or something it is not sure about), and asks for an 'extra' prompt before continuing.

It's very hard(maybe impossible) for gfjardim's wonderful GUI front end to catch this, and hence gets stuck at that point.

 

You have a couple ways around this:

  • Move the disk(s) to the onboard SATA to do the preclear.(and move it back when finished)
  • Run the script directly within a terminal session.(I believe you can still monitor progress from the GUI)
  • possibly, try any available firmware updates for your card

Link to comment

I was running unraid 6.0.1 previously with preclear.disk-2015.07.23

I update the plugin first and successfuly update to preclear.disk-2015.11.18, upgraded unraid via gui to 6.1.6 and rebooted. All seemed normal.

 

I went to preclear a new 5tb drive and selected email and popup notifications every 25% of progress. I received neither (in notifications clicking test works, UPS alerts are being received as i have a bad battery and haven't got around to replacing). So I let the preclear finished and saw the following:

 

================================================================== 1.15c

=                unRAID server Pre-Clear disk /dev/sdc

=              cycle 2 of 2, partition start on sector 1

=

= Step 1 of 10 - Copying zeros to first 2048k bytes            DONE

= Step 2 of 10 - Copying zeros to remainder of disk to clear it DONE

= Step 3 of 10 - Disk is now cleared from MBR onward.          DONE

= Step 4 of 10 - Clearing MBR bytes for partition 2,3 & 4      DONE

= Step 5 of 10 - Clearing MBR code area                        DONE

= Step 6 of 10 - Setting MBR signature bytes                    DONE

= Step 7 of 10 - Setting partition 1 to precleared state        DONE

= Step 8 of 10 - Notifying kernel we changed the partitioning  DONE

= Step 9 of 10 - Creating the /dev/disk/by* entries            DONE

= Step 10 of 10 - Verifying if the MBR is cleared.              DONE

= Disk Post-Clear-Read completed                                DONE

Disk Temperature: 35C, Elapsed Time:  65:05:02

========================================================================1.15c

== TOSHIBAMD04ACA500  75TAK3QBFS9A

== Disk /dev/sdc has been successfully precleared

== with a starting sector of 1

============================================================================

No SMART attributes are FAILING_NOW

0 sectors were pending re-allocation before the start of the preclear.

0 sectors were pending re-allocation after pre-read in cycle 1 of 2.

0 sectors were pending re-allocation after zero of disk in cycle 1 of 2.

0 sectors were pending re-allocation after post-read in cycle 1 of 2.

0 sectors were pending re-allocation after zero of disk in cycle 2 of 2.

0 sectors are pending re-allocation at the end of the preclear,

    the number of sectors pending re-allocation did not change.

0 sectors had been re-allocated before the start of the preclear.

0 sectors are re-allocated at the end of the preclear,

    the number of sectors re-allocated did not change.

/boot/config/plugins/preclear.disk/preclear_disk.sh: line 316: mail: command not found

root@Tower:/usr/local/emhttp#

 

I ssh'ed in and typed "mail" received "-bash: mail: command not found" this is about as far as i can troubleshoot in linux. Outside of updating this plugin and unraid, nothing else changed.

 

I checked the "preclear_reports" directory on the flash drive, all 3 are there (start, finish and rpt)

here is what rpt is showing the plugin used for a commandline:

 

========================================================================1.15c

== invoked as: /boot/config/plugins/preclear.disk/preclear_disk.sh -M 4 -o 3 -c 2 -f -J /dev/sdc

== TOSHIBAMD04ACA500  75TAK3QBFS9A

== Disk /dev/sdc has been successfully precleared

== with a starting sector of 1

== Ran 2 cycles

==

== Using :Read block size = 1000448 Bytes

== Last Cycle's Pre Read Time  : 13:22:46 (103 MB/s)

== Last Cycle's Zeroing time  : 12:07:06 (114 MB/s)

== Last Cycle's Post Read Time : 13:43:38 (101 MB/s)

== Last Cycle's Total Time    : 25:51:46

==

== Total Elapsed Time 65:05:02

==

== Disk Start Temperature: 31C

==

== Current Disk Temperature: 35C,

==

============================================================================

No SMART attributes are FAILING_NOW

 

0 sectors were pending re-allocation before the start of the preclear.

0 sectors were pending re-allocation after pre-read in cycle 1 of 2.

0 sectors were pending re-allocation after zero of disk in cycle 1 of 2.

0 sectors were pending re-allocation after post-read in cycle 1 of 2.

0 sectors were pending re-allocation after zero of disk in cycle 2 of 2.

0 sectors are pending re-allocation at the end of the preclear,

    the number of sectors pending re-allocation did not change.

0 sectors had been re-allocated before the start of the preclear.

0 sectors are re-allocated at the end of the preclear,

    the number of sectors re-allocated did not change.

============================================================================

 

Did something change with mail in 6.1.6 and this plugin is no longer compatible?

Link to comment

I was running unraid 6.0.1 previously with preclear.disk-2015.07.23

I update the plugin first and successfuly update to preclear.disk-2015.11.18, upgraded unraid via gui to 6.1.6 and rebooted. All seemed normal.

 

I went to preclear a new 5tb drive and selected email and popup notifications every 25% of progress. I received neither (in notifications clicking test works, UPS alerts are being received as i have a bad battery and haven't got around to replacing). So I let the preclear finished and saw the following:

...

Did something change with mail in 6.1.6 and this plugin is no longer compatible?

See Reply #200 on this thread.
Link to comment

I was running unraid 6.0.1 previously with preclear.disk-2015.07.23

I update the plugin first and successfuly update to preclear.disk-2015.11.18, upgraded unraid via gui to 6.1.6 and rebooted. All seemed normal.

 

I went to preclear a new 5tb drive and selected email and popup notifications every 25% of progress. I received neither (in notifications clicking test works, UPS alerts are being received as i have a bad battery and haven't got around to replacing). So I let the preclear finished and saw the following:

...

Did something change with mail in 6.1.6 and this plugin is no longer compatible?

See Reply #200 on this thread.

 

Thanks, i read it now. I'm confused a bit as I am not using Joe L. script, I am using this plugin which brings down version 1.15c of the script. And it was working before i upgraded the plugin and unraid itself.

 

Looking at /flash/config/plugins/preclear.disk/preclear_disk.sh

 

# Version 1.15  - a) Added PID to preclear_stat_sdX files and support for notifications - gfjardim

#                b) Add notification channel choice

#                c) Added support for fast post read (bjp999)

ver="1.15c"

 

 

...

 

 

Line 307 # gfjardim - add notification system capability without breaking legacy mail.

Line 308 send_mail() {

Line 309  subject=$(echo ${1} | tr "'" '`' )

Line 310  description=$(echo ${2} | tr "'" '`' )

Line 311  message=$(echo ${3} | tr "'" '`' )

Line 312  recipient=${4}

Line 313  if [ -f "/usr/local/sbin/notify" ]; then

Line 314    /usr/local/sbin/notify -e "Preclear ${model} ${serial}" -s """${subject}""" -d """${description}""" -m Line 315 """${message}""" -i "normal ${notify_channels}"

Line 316  else

Line 317    echo -e "${message}" | mail -s "${subject}" "${recipient}"

Line 318  fi

Line 319 }

 

So why is it failing to detect and use notify and goes to use mail? that my understanding from this code snippet.

Looking for /usr/local/sbin/notify I don't see a file 'notify' on my unraid server.

 

 

Using SCP and running a search it did find a 'notify' in /usr/local/emhttp/plugins/dynamix/scripts/notify

 

P.S. Even the pop-up notifications are not working, which has nothing to do with email right?

 

Link to comment

Changing the follwoing lines and running a test enabled pop-ups and email to function once again.

It mostly will break this functionality for prior version, whenever the notify change was made.

 

  # if [ -f "/usr/local/sbin/notify" ]; then

    if [ -f "/usr/local/emhttp/plugins/dynamix/scripts/notify" ]; then

  #/usr/local/sbin/notify -e "Preclear ${model} ${serial}" -s """${subject}""" -d """${description}""" -m """${message}""" -i "normal ${notify_channels}"

    /usr/local/emhttp/plugins/dynamix/scripts/notify -e "Preclear ${model} ${serial}" -s """${subject}""" -d """${description}""" -m """${message}""" -i "normal ${notify_channels}"

 

    #if [ "$no_mail" = "no" && ! -f "/usr/local/sbin/notify" ]

  if [ "$no_mail" = "no" && ! -f "/usr/local/emhttp/plugins/dynamix/scripts/notify" ]

 

Was 'notify' moved in v6.1.6?

 

Link to comment

Changing the follwoing lines and running a test enabled pop-ups and email to function once again.

It mostly will break this functionality for prior version, whenever the notify change was made.

 

  # if [ -f "/usr/local/sbin/notify" ]; then

    if [ -f "/usr/local/emhttp/plugins/dynamix/scripts/notify" ]; then

  #/usr/local/sbin/notify -e "Preclear ${model} ${serial}" -s """${subject}""" -d """${description}""" -m """${message}""" -i "normal ${notify_channels}"

    /usr/local/emhttp/plugins/dynamix/scripts/notify -e "Preclear ${model} ${serial}" -s """${subject}""" -d """${description}""" -m """${message}""" -i "normal ${notify_channels}"

 

    #if [ "$no_mail" = "no" && ! -f "/usr/local/sbin/notify" ]

  if [ "$no_mail" = "no" && ! -f "/usr/local/emhttp/plugins/dynamix/scripts/notify" ]

 

Was 'notify' moved in v6.1.6?

Notify was moved in 6.1.0 as part of some security fixes. That's basically why we have 6.1 compatible plugins separated out. The version of the script that is linked in that post I sent you to includes the fixes needed for compatibility so you don't need to make those changes unless you are determined to not use that version of the script. Even if you do use that version of the script the use of faster postread is optional so there is not really any benefit from not using it.
Link to comment

Wow, i finally understand and looking into the two scripts i see the change from i want to say bjp999 1.15c version per that link you posted. But talk about total confusion.

 

I am new to all this. I never used Joe L. script, I just install the plugin under unraid 6.0.1 and with your explanation I now see why it worked under 6.0.1 with the preclear 1.15c file i had downloaded somewhere in this thread if i am not mistaken dont know who coded it except for the comments in it (a) b) c))

 

I did as recommended and update all my plugins and then upgrade unraid to 6.1.6. popup AND email notifications no longer work even though I have the latest plugin installed?

 

From "preclear_disk.sh" i downloaded on 07/24/2015 with latest version of plugin:

# Version 1.15  - a) Added PID to preclear_stat_sdX files and support for notifications - gfjardim

#                b) Add notification channel choice

#                c) Added support for fast post read (bjp999)

ver="1.15c"

 

From the post you linked to, file is named "preclear_disk_15c.sh" (should be exactly the same no, they are both 1.15c):

# Version 1.15  - Added export of LC_CTYPE=C to prevent unicode use on 64 bit printf.

#                Added PID to preclear_stat_sdX files and support for notifications - gfjardim

#                Add notification channel choice - gfjardim

#                Added support for fast post read - bjp999

#                Remove /root/mdcmd dependency - gfjardim

ver="1.15"

File name is 15c but not in the comments nor "ver ="

Wouldn't that make you think that the file is first iteration of 1.15, and the other the latest 1.15c?

 

 

Lastestly, why would the OP be not update to say hey you have a choice:

 

go here -> for Joe L. original script, Note: *** popups and email notifications will not work as of 6.1 with it though and no Fast post-read option available ***

 

go here -> to get a modified version of Joe L.'s preclear script that works with all unraid versions notification systems and has a Fast post-read option available

 

Without your help I would have never found that buried post and that exact script (dont know what version to call it 1.15 with comments?)

 

Still confused how both were 1.15c versions but are different internally. The one i placed with the plugin under unraid version 6.0.1 had a download date of 07-24-2015 and also named preclear_disk_15c.sh. So someone made changes and still named it 1.15c. OR gfjardim created one called 1.15c and bjp999 created one called 1.15c and coded differently? Im starting to think that is the original link i used and that file was modified and switched out: because I used this post to grab mine back on 07/24/2015 http://lime-technology.com/forum/index.php?topic=39985.msg387071#msg387071

Which is the same as you quoted for me just in a later post. I think gfjardim knows what happened there.

 

I am good now trurl and appreciate your help getting me to the preclear script version that supports everything as far as i can see.

 

Link to comment

This is in response to fantax's post above. I have been helping him with his new unRAID system. He had a drive die. The array was stopped, the drive was removed from the array, and the array was started again. This seemed to work just fine, and the data on the missing drive was still accessible. He then replaced the drive. The new drive took the assignment of /dev/sdc . This was not shown under the unassigned devices plugin, so the option to preclear it wasn't there either. Stopping the array gives him the option of being able to assign it to the array and begin a parity rebuild of the data, but nothing from the preclear script, or plugin was reported. From the CLI, doing a "preclear_disk.sh -l" produced no results, it did not show the new drive. However, doing a "preclear_disk.sh /dev/sdc" began the preclear process just fine.

He's running unRAID 6.1.3 . He has 5 drives attached to his motherboard, and 3 drives attached to an IBM M1015 flashed to IT mode. The drive in question is one of the ones connected to the controller card.

I wasn't exactly sure where to post this, but it seems as though both this plugin, and the preclear script, were unable to see this new drive for whatever reason. Any help would be greatly appreciated.

Link to comment

Just followed the instructions to install fast preclear, but I'm not seeing the options show up.  Here's the layout of files.  Pretty clear the files are in the right places (?), but I'm not seeing the options as advertised.  What did I miss?

 

20151220-xjqc-187kb.jpg

 

If you read the first post in this thread you will see they are not in the right place. Also, the plugin will use the script named preclear_disk.sh, so if you want it to use the faster preclear_bjp.sh, you must rename it. And if you want notifications read my reply #334 just a few posts above this one for a link to reply #200.

 

Chasing you all around the forum on this. I should go to bed. ;D

Link to comment

*sorry* for the chase!  But thanks for chasing!  Yep, I needed to copy the file... I should be in bed too, I guess I misread putting the file in the root...  Regardless, its working now!

 

One quick follow-up.  There's an option to skip the pre-read.  When would you want to skip and not to skip the pre-read?

 

Again, thanks for chasing me around while I found the right place.  Love this forum!

Link to comment

*sorry* for the chase!  But thanks for chasing!  Yep, I needed to copy the file... I should be in bed too, I guess I misread putting the file in the root...  Regardless, its working now!

 

One quick follow-up.  There's an option to skip the pre-read.  When would you want to skip and not to skip the pre-read?

 

Again, thanks for chasing me around while I found the right place.  Love this forum!

Skipping pre-read makes sense if you are re-clearing a disk you already have confidence in.
Link to comment

So is that correct? If there is a failed drive in the array, unRAID "claims it" and you are not able to preclear the drive normally? This doesn't seem like what would be expected behavior unless I'm really missing something. Since unRAID places a fairly high value on pre-clearing drives before they're added to the array (which is one of the nicest features imho) you'd think that it would want you to pre-clear any drive added. Is this a bug in the preclear script or is that actually expected behavior?

 

 

Link to comment

So is that correct? If there is a failed drive in the array, unRAID "claims it" and you are not able to preclear the drive normally? This doesn't seem like what would be expected behavior unless I'm really missing something. Since unRAID places a fairly high value on pre-clearing drives before they're added to the array (which is one of the nicest features imho)

Actually it does not (at the moment at least) as pre-clear is an add-on and not built-in functionality.

 

Note that it is not necessary for a drive that is replacing a failed drive (or when replacing a drive by a larger one) to be pre-cleared.  In such a scenario a pre-clear is just an initial stress-test of a drive .

you'd think that it would want you to pre-clear any drive added. Is this a bug in the preclear script or is that actually expected behavior?

The pre-clear will not let you preclear a drive that is assigned to the array to avoid the chance of accidentally pre-clearing a drive that ahs data on it (and thus losing data).  In the case of adding a new drive then obviously initially it is not assigned to the array and is thus available to pre-clear.  If you have a drive fail and want to pre-clear the drive before replacing it in the array, then I think if you unassign the failed drive and start the array with the drive marked as 'missing' so it is emulated unRAID may let you pre-clear the new drive without grabbing it. However I have not actually tried this so I am not sure.

Link to comment

That's what I would expect as well, but it's not what happened. The drive failed, the array was stopped and the drive was removed, and the array restarted. All the data was visible. The new drive was added and the machine was rebooted. Listing the drives available for preclear showed no available drives, via the CLI or the unRAID UI. If the array was stopped, I was able to assign the new drive to the array. However, the only way I was able to clear this drive was

preclear_disk.sh /dev/sdc

So the script did not show the drive as available for a preclear, but it started anyway via the command line. The reason I was reporting it here, was because with the addition of this plugin, this was the last reason I had to regularly use the CLI and I thought that's a huge leap forward for less than experienced users who were possibly unfamiliar with the command line.

 

So either this is expected behavior and unRAID does somehow claim the drive when there is a failed drive in the array, or there is a bug in the preclear script or plugin, that says a drive is not unavailable for preclear when indeed it is.

Link to comment

That's what I would expect as well, but it's not what happened. The drive failed, the array was stopped and the drive was removed, and the array restarted. All the data was visible. The new drive was added and the machine was rebooted. Listing the drives available for preclear showed no available drives, via the CLI or the unRAID UI. If the array was stopped, I was able to assign the new drive to the array. However, the only way I was able to clear this drive was

preclear_disk.sh /dev/sdc

So the script did not show the drive as available for a preclear, but it started anyway via the command line. The reason I was reporting it here, was because with the addition of this plugin, this was the last reason I had to regularly use the CLI and I thought that's a huge leap forward for less than experienced users who were possibly unfamiliar with the command line.

 

So either this is expected behavior and unRAID does somehow claim the drive when there is a failed drive in the array, or there is a bug in the preclear script or plugin, that says a drive is not unavailable for preclear when indeed it is.

 

As I recall, the problem resulted from your removal of the 'bad' drive from the array.  If you have simply inserted the drive into the case, it would have been available for 'preclearing'.  But once the bad drive goes physically missing, any drive that is installed in the case will be automatically added to the array and the rebuilt process started (if the drive is equal to or larger in size). 

 

If you think about it this is the way most of us want it to work as it makes replacing a defective drive a simple process.

 

Remember the preclear script is NOT a part of the Limetech's unRAID distribution package but a 'user developed' utility. 

 

 

Link to comment

No that's what I was after. Thanks very much! I guess I just misread or botched the replacement process. I keep a precleared warm spare for myself and this was for someone else, so I honestly haven't been through this actual process in a while. I was just curious as to expected behavior in each scenario. That makes it perfectly clear. So in the future, do not remove the bad drive from the array, add the new one and preclear, and reassign.

 

The only question I would still have is, when a drive is failing and you try to watch video stored on it, it tends to buffer and skip. Which would be expected. Removing the failing drive first at least corrects that error while waiting for another drive. I guess the only way to achieve that would be doing it how I did, and just forcing a preclear via the command line?

 

Thanks very much again for the quick response.

Link to comment
  • Squid featured, unfeatured and pinned this topic
  • Squid unpinned this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.