Preclear plugin


3042 posts in this topic Last Reply

Recommended Posts

  • Replies 3k
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Hi guys, this is a simple plugin that allows users to clear their disks before add them to the array.   The main characteristics of this plugin are: Modularity: can be used standalon

I think Thanks should be directed at all those members that have been dealing with the mess the last few months. @dlandon and @Squid have been working behind the scenes to try and clean up, and o

Dontlookoverthere over on the unRAID subreddit figured out how to get the plugin running again with a simple edit to the plg:   https://www.reddit.com/r/unRAID/comments/7wjpus/preclear_worki

Posted Images

Hi gfjardim,

 

First thank you for the script/plugin, been using them for a while now since unRAID 6.0, and using Joe L.'s pre-clear script long before.  You've made the process such a breeze now.

 

Quick question, and I'm sure it's been asked before, but I just don't have the time to go through the thread. (Perhaps edit(ing) the first post with updates, and a FAQ?, something I think needs to be done a lot more on these forums, or move to a better forum software...  but I digress)

 

My issue is I'm not getting any S.M.A.R.T. reporting in neither the status view while pre-clear is running, or at the end in my reports.  The box is just blank.  I'm sure it's a quick fix, and other than that everything is working well.

 

Thanks again!

 

S.M.A.R.T. status becomes unavailable if the script can't detect your controller properly. Do you mind sharing your server configuration and which controller is hosting your disk?

Link to post

Why do clearing times vary so much by drive? I have 4 3TB drives all being precleared at once. Three are Seagate Barracudas and one is WD Red. The red is taking probably 30% slower than the rest to finish 1 cycle. Could it be a sign of a bad drive?

 

Also, do SSD drives need the same number of cycles as a SATA drive? Thank you.

Link to post

Why do clearing times vary so much by drive? I have 4 3TB drives all being precleared at once. Three are Seagate Barracudas and one is WD Red. The red is taking probably 30% slower than the rest to finish 1 cycle. Could it be a sign of a bad drive?

 

Also, do SSD drives need the same number of cycles as a SATA drive? Thank you.

 

Um, I was indthink r the impression SSD di GWS don't need preclearing, but in saying that I have never tried as I've never thrown one into the array.

Link to post

Hi gfjardim,

 

First thank you for the script/plugin, been using them for a while now since unRAID 6.0, and using Joe L.'s pre-clear script long before.  You've made the process such a breeze now.

 

Quick question, and I'm sure it's been asked before, but I just don't have the time to go through the thread. (Perhaps edit(ing) the first post with updates, and a FAQ?, something I think needs to be done a lot more on these forums, or move to a better forum software...  but I digress)

 

My issue is I'm not getting any S.M.A.R.T. reporting in neither the status view while pre-clear is running, or at the end in my reports.  The box is just blank.  I'm sure it's a quick fix, and other than that everything is working well.

 

Thanks again!

 

S.M.A.R.T. status becomes unavailable if the script can't detect your controller properly. Do you mind sharing your server configuration and which controller is hosting your disk?

 

But of course,

 

It's happened on two different controllers now, first the on board Intel C602 chipset controller. 

(The board I use also comes with 4 on board Marvell SE923, Though I've not tried on those SATA headers as there's already some known issues in unRAID with the Marvell chipsets. )

 

The second, and now currently running pre-clear, an Intel RS2WC080 RAID Controller SAS/SATA MD2 6GB/s PCIe 2.0x8 flashed to the  LSI SAS2008 IR/IT firmware.

 

Fun part is if I look at the drives through the unRAID web GUI, all S.M.A.R.T. status is reported, and correct.

Link to post

Hey gfjardim,

 

In another thread Tom wrote:

 

You should not run pre-clear on an SSD.  If you insist maybe use 'blkdiscard' command to tell the SSD to discard all the data you wrote, and then let unRAID format it and add to cache pool.

 

Perhaps the preclear script could detect SSDs and do this rather than the standard preclear?

Link to post

Hey gfjardim,

 

In another thread Tom wrote:

 

You should not run pre-clear on an SSD.  If you insist maybe use 'blkdiscard' command to tell the SSD to discard all the data you wrote, and then let unRAID format it and add to cache pool.

 

Perhaps the preclear script could detect SSDs and do this rather than the standard preclear?

 

+1

 

I tried it yesterday and it's much faster than normal clear while causing less wear on the SSD.

 

Like with trim, some LSI controllers don't work with this command, so don't completely remove the option to normal preclear a SSD as a last resort.

Link to post

Hi gfjardim,

 

First thank you for the script/plugin, been using them for a while now since unRAID 6.0, and using Joe L.'s pre-clear script long before.  You've made the process such a breeze now.

 

Quick question, and I'm sure it's been asked before, but I just don't have the time to go through the thread. (Perhaps edit(ing) the first post with updates, and a FAQ?, something I think needs to be done a lot more on these forums, or move to a better forum software...  but I digress)

 

My issue is I'm not getting any S.M.A.R.T. reporting in neither the status view while pre-clear is running, or at the end in my reports.  The box is just blank.  I'm sure it's a quick fix, and other than that everything is working well.

 

Thanks again!

 

S.M.A.R.T. status becomes unavailable if the script can't detect your controller properly. Do you mind sharing your server configuration and which controller is hosting your disk?

 

I think this has been mentioned a couple of times before in this thread, that some new drives doesn't have SMART enabled by default and therefor need it enabled before starting the script. After adding the drive to the array, SMART is available.

I have had this on a few new drives when using your plugin and script. It might have been johnnie.black that mentioned this.

Link to post

Hi gfjardim,

 

First thank you for the script/plugin, been using them for a while now since unRAID 6.0, and using Joe L.'s pre-clear script long before.  You've made the process such a breeze now.

 

Quick question, and I'm sure it's been asked before, but I just don't have the time to go through the thread. (Perhaps edit(ing) the first post with updates, and a FAQ?, something I think needs to be done a lot more on these forums, or move to a better forum software...  but I digress)

 

My issue is I'm not getting any S.M.A.R.T. reporting in neither the status view while pre-clear is running, or at the end in my reports.  The box is just blank.  I'm sure it's a quick fix, and other than that everything is working well.

 

Thanks again!

 

S.M.A.R.T. status becomes unavailable if the script can't detect your controller properly. Do you mind sharing your server configuration and which controller is hosting your disk?

 

I think this has been mentioned a couple of times before in this thread, that some new drives doesn't have SMART enabled by default and therefor need it enabled before starting the script. After adding the drive to the array, SMART is available.

I have had this on a few new drives when using your plugin and script. It might have been johnnie.black that mentioned this.

 

I question that this is my issue, because I can run full SMART status requests from the unRAID web GUI, and I get full SMART status reporting through the unRAID

GUI as well.

Link to post

I've got a preclear running on a new drive, and all is going wonderfully, except...

 

I've been having issues getting email notifications since I changed my gmail password (separate issue raised for that), so I  set up a Pushover account. I am getting test messages sent via Pushover from the Notification Settings menu in unRAID.

 

When I launched the preclear, I turned on all notifications - email, GUI & Notifications.

 

I'm not getting email - as I expected

I am getting GUI pop ups (see attachment)

I'm not getting Pushover - I'm surprised.

 

Is there any logging I can look at for the preclear to see if it's returning an error to see what the issue is? I do know that I can get a test message via unRAID. I don't remember if I've yet received a regular status update via Pushover, so I don't know if unRAID is working in general beyond just the test messages.

 

And thank you so much for picking up where JoeL left off on this! It's greatly appreciated!!

unRAID_preclear_notifications.PNG.7bfaae65e7ee9f0e36fcc25731301900.PNG

Link to post

@ gfjardim - after reading some of the recent post - I'm confused about the what options to use, can you please confirm your recommendations on how best to use the Preclear plugin with spinners, in terms of how many cycles, options to select, etc?

 

For me, I have HGST Deskstar NAS drives and I run the Preclear from the gui, I change no options, take all your defaults, and go.  The default is 1 cycle.  When that completes with no issues, I'm done and move on to the next drive.  So far, this has worked good for me.

 

Some say you have to run more than 1 cycle.  Others say that running preclear takes life away from the drive, so you need to limit the number of cycles. 

 

What do you recommend?

 

 

Link to post

@ gfjardim - after reading some of the recent post - I'm confused about the what options to use, can you please confirm your recommendations on how best to use the Preclear plugin with spinners, in terms of how many cycles, options to select, etc?

 

For me, I have HGST Deskstar NAS drives and I run the Preclear from the gui, I change no options, take all your defaults, and go.  The default is 1 cycle.  When that completes with no issues, I'm done and move on to the next drive.  So far, this has worked good for me.

 

Some say you have to run more than 1 cycle.  Others say that running preclear takes life away from the drive, so you need to limit the number of cycles. 

 

What do you recommend?

 

I know you specifically asked gfjardim, but... you've hit the nail on the head. It's been a debate that's run as long as the preclear has been available.

 

Some folks have had their drives fail on the 3rd preclear, so they always run 3+, some run 1 just to get it into the system.

 

With version 6.x (maybe 6.1?) you don't actually have to preclear a drive anymore. unRAID will do it for you without taking the array off-line to do it. (It used to take the array offline while zeroing out the drive, thus you were down for hours to days, thus the need to preclear the drive.)

 

I don't think there's a definitive answer to your question, though others are welcome to pipe in . :)

Link to post

Never run Pre-Clear on a SSD as you'll consume some of its limited life.  On HDD the I run 3 Pre-Clear cycles as I want the drive over the first bathtub failure hump.  I have had a few drives fail on the first Pre-Clear,  1 (maybe 2) on the second.  I have yet to experience a drive failure on anything that has passed 3 cycles.  We even use 3 Pre-Clear to prep non Unraid drives at work! 

Link to post

@ gfjardim - after reading some of the recent post - I'm confused about the what options to use, can you please confirm your recommendations on how best to use the Preclear plugin with spinners, in terms of how many cycles, options to select, etc?

 

For me, I have HGST Deskstar NAS drives and I run the Preclear from the gui, I change no options, take all your defaults, and go.  The default is 1 cycle.  When that completes with no issues, I'm done and move on to the next drive.  So far, this has worked good for me.

 

Some say you have to run more than 1 cycle.  Others say that running preclear takes life away from the drive, so you need to limit the number of cycles. 

 

What do you recommend?

 

I know you specifically asked gfjardim, but... you've hit the nail on the head. It's been a debate that's run as long as the preclear has been available.

 

Some folks have had their drives fail on the 3rd preclear, so they always run 3+, some run 1 just to get it into the system.

 

With version 6.x (maybe 6.1?) you don't actually have to preclear a drive anymore. unRAID will do it for you without taking the array off-line to do it. (It used to take the array offline while zeroing out the drive, thus you were down for hours to days, thus the need to preclear the drive.)

 

I don't think there's a definitive answer to your question, though others are welcome to pipe in . :)

 

I agree with this, and I think you'll get as many opinions as people you ask, good opinions too from knowledgeable users.

 

I only Preclear once, and have never had a drive fail afterwards, but then again, I've never had a drive fail during a Preclear either!  I'm guessing I've done about 6 to 10 drives.  I know of another user, very respected, who Preclears 4 or 5 times, because he has had drives pass after 3 Preclears but fail shortly afterwards.

 

My feeling is that you have to draw the line somewhere, and it's up to the user's personal preference.  And besides, if you are adding the drive to your unRAID array, a fault tolerant system designed for drives to fail without data loss, then it doesn't really matter as much if it fails soon, just a little more work to replace than if it hadn't yet entered the array.  Drives don't fail very often, and I suspect there are a few that feel the chance of failure is low enough, they don't bother to Preclear at all, just let normal operation expose the lemons.

Link to post

My nickel's worth on this ...

 

(1) "PreClear" is no longer needed for its original function => i.e.clearing a drive BEFORE it was added to the array, so the array wouldn't be down for the time it took to zero a new drive (which can be MANY hours).  That's why JoeL wrote the PreClear utility -- with Tom's blessing and cooperation by providing the details for a "pre-clear signature" which allowed UnRAID to "know" a drive had been pre-cleared.    However, as noted above, the latest version of UnRAID does not take the array off line for this -- it simply clears the drive and THEN adds it to the array -- so the "clearing" function is no longer needed (doesn't hurt to do it -- it just doesn't save any array downtime).

 

(2)  An ancilliary function of the Pre-Clear utility was that it effectively tested a new drive, since it did a complete read pass (pre-clear) BEFORE it wrote the zeroes; and then did a complete verification of the zeroes (post-read), and Joe included a bunch of random seeks to further test the drive being cleared.    It became the defacto way to "test" a new drive for any infant mortality issues.

 

So ... the real question isn't whether or not you need to "PreClear" a new drive -- but whether you should TEST it before putting it to use.    As Rob noted, you can get a lot of opinions on that ... but my take is you SHOULD test a drive before putting it in service, but there's no reason to go overboard on this, as most infant mortality issues will surface in an initial thorough test.

 

What I do with new drives -- regardless of the intended destination (UnRAID or a PC) -- is connect them to a spare SATA port on my main desktop; run WD's Data Lifeguard; and do a Quick Test; then an Extended Test; then a full Write Zeroes; and then repeat the Quick & Extended tests.  ANY error => I return the drive (doesn't happen often).  Error free => put the drive in service.    I used to also run a full pass with Spinrite, which is a VERY thorough test of a drive; but this takes a VERY long time (over a day with modern drives) ... and I simply decided it wasn't necessary [i found a few bad drives with Data Lifeguard; but never had a drive fail with Spinrite after it had passed the Data Lifeguard testing].    Until the most recent version of UnRAID, I would then run a "clear only" pass of Pre-Clear ( "-n" option) on any drives that were destined for one of my servers ... but I no longer bother with that.

 

Some folks don't bother with any testing -- and as Rob noted, since this IS a fault tolerant server, it is true that the impact of an early failure shouldn't be data loss ... just a bit of inconvenience.    Note that the mere fact of adding a drive to the array effectively "tests" a drive fairly well anyway, as long as you do a parity check after you've added it (which you should always do).    Just adding a drive to the array will cause the entire drive to be written (cleared if it's a new drive; written with data if you're rebuilding a failed drive or upgrading the size of a drive); and then if you do a parity check the entire drive is read.    Personally, I prefer to do an initial round of testing to weed out any infant mortality drives [i've had a couple of those over the years -- but have NEVER had a drive fail early once it passed my initial round of testing] ... if you discovered this while adding it to your array you may wonder whether it was the drive or some other issue with your system.

 

Link to post

I started a preclear, but it seems to be hung. When I click the little blue eyeball for the preview, I see:

 

/usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh: fork: cannot allocate memory

 

My pre-clear hasn't progressed beyond 53% overnight.

 

Clicking the X to cancel the preclear doesn't seem to be working, either.

 

Troubleshooting tips appreciated.

unRAID_preclear_memory_error.PNG.0429cc891890670deaf0b27eb2671807.PNG

Link to post

Guys, I wish I could give proper support right now, but I can't.

 

1) SSD wear out: if you want to add a SSD to the array, you should run blkdiscard /dev/sdX an then run /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --preclear-only /dev/sdX on the command line. I'll make this accessible by the GUI when I have time;

 

2) Killed session of preclear: if the X icon isn't working, you can kill it by typing the following code: tmux kill-session -t preclear_disk_sdX

 

3) Number of passes: the more the merrier, IMHO. If you have time to let it preclear for a week, then you should do it. Mechanical failures are extremely rare these days, so mechanical wear out is debatable.

Link to post

Guys, I wish I could give proper support right now, but I can't.

 

Appreciate you stopping by!

 

2) Killed session of preclear: if the X icon isn't working, you can kill it by typing the following code: tmux kill-session -t preclear_disk_sdX

 

I did that, but it's still showing results, and the X says it will clear stats, but doesn't. I guess I could restart it by hand, but... ewww!  ;)

Any other tips?

Link to post

Take a cue from Office Space  :P

 

(edit....  Sorry,  I've been really tired with a teething tot and this seemed cute in the moment)

 

Is the drive In a hot swap bay?  Does the sata connection have hot swap enabled?  I am curious if it can be popped out?

Link to post

Guys, I wish I could give proper support right now, but I can't.

 

Appreciate you stopping by!

 

2) Killed session of preclear: if the X icon isn't working, you can kill it by typing the following code: tmux kill-session -t preclear_disk_sdX

 

I did that, but it's still showing results, and the X says it will clear stats, but doesn't. I guess I could restart it by hand, but... ewww!  ;)

Any other tips?

 

I rebooted the server, and it let me restart the preclear. Now it's telling me that the preclear is done. gfjardim, you've done some amazing work, brother! That's a 4TB drive cleared in... 12 hours!!

 

OK, probably not. :(

 

 preclear_disk.sh -t /dev/sdh

Yields all the instructions on how to call preclear_disk.sh followed by:

Sorry: Device /dev/sdh is busy.: 1

 

All that's in /boot/preclear_reports is a report for the drive that died last week. It's dated Sept 17th, and is full of:

tput: unknown terminal "screen"

(6536 bytes worth).

 

I redownloaded the original preclear_disk.zip, extracted preclear_disk.sh from it and compared SHA256 fingerprints from the fresh copy v the copy I'm actually using, and they match.


I know gfjardim is busy, so I probably won't get a response from him for a couple of days (I get it - real life. Glad he's got one!). Anyone else have any suggestions?

 

No, the drive's not hot swappable, unfortunately. (I guess I need to watch Office Space again, 'cause I totally didn't get the reference...)

 

I'm attaching screenshots of the "preview" (clicking the blue eye), since it won't let me copy/pasta the text.

unraid_preclear_message_1.PNG.1e1502fab6e24e7b0a848f0ae029da3e.PNG

unraid_preclear_message_2.PNG.b66f4958a6c7025b6947a7f5fbb49ac7.PNG

Link to post

I redownloaded the original preclear_disk.zip, extracted preclear_disk.sh from it and compared SHA256 fingerprints from the fresh copy v the copy I'm actually using, and they match.

 

The original preclear_disk.sh has to be patched in order to work with unRAID 6.2

 

Link to post

Guys, I wish I could give proper support right now, but I can't.

 

Appreciate you stopping by!

 

2) Killed session of preclear: if the X icon isn't working, you can kill it by typing the following code: tmux kill-session -t preclear_disk_sdX

 

I did that, but it's still showing results, and the X says it will clear stats, but doesn't. I guess I could restart it by hand, but... ewww!  ;)

Any other tips?

 

I rebooted the server, and it let me restart the preclear. Now it's telling me that the preclear is done. gfjardim, you've done some amazing work, brother! That's a 4TB drive cleared in... 12 hours!!

 

OK, probably not. :(

 

 preclear_disk.sh -t /dev/sdh

Yields all the instructions on how to call preclear_disk.sh followed by:

Sorry: Device /dev/sdh is busy.: 1

 

All that's in /boot/preclear_reports is a report for the drive that died last week. It's dated Sept 17th, and is full of:

tput: unknown terminal "screen"

(6536 bytes worth).

 

I redownloaded the original preclear_disk.zip, extracted preclear_disk.sh from it and compared SHA256 fingerprints from the fresh copy v the copy I'm actually using, and they match.


I know gfjardim is busy, so I probably won't get a response from him for a couple of days (I get it - real life. Glad he's got one!). Anyone else have any suggestions?

 

No, the drive's not hot swappable, unfortunately. (I guess I need to watch Office Space again, 'cause I totally didn't get the reference...)

 

I'm attaching screenshots of the "preview" (clicking the blue eye), since it won't let me copy/pasta the text.

 

Remove the plugin and reinstall it again.

Link to post
  • trurl featured, unfeatured and pinned this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.