Preclear plugin


Recommended Posts

...only wish this board had a search within the thread...

It does. See search tips in my sig.

 

huh.. never noticed that chevron, and the search worked great unfortunately the search only came up with it stalling for the 6.2 beta.

 

nothing for what i'm seeing :/

 

 

Link to comment

With the array running, I then started a single pre-clear while watching "top".  Memory utilization jumped from 8% to 15%.  Interestingly, I see multiple "preclear_+" processes.  There seems to be a main one, consuming 1.023GB within a few seconds of starting.

 

I was comparing your findings on one of my servers and it appears preclear beta uses much less RAM than on yours, I have the array started doing a dual disk upgrade and preclearing two disks at the same time, this server has 8GB of RAM.

 

EDIT: Just though of a reason that can explain the difference, since these are previously used disks I'm skipping the pre and post reads, so the pre read can have higher memory utilization than the zeroing.

Screenshot_2016-08-14_10_25_38.png.128be9e7b84d7b61772ceb8cbc31415d.png

Screenshot_2016-08-14_10_30_19.png.bae6ad78e3b61d4d0272aa6a991cfbc0.png

Link to comment

That's very odd, how little memory the beta preclear is using on your system, and how much it is using on mine.  I'll run a zero only, skipping the pre/post reads, and see if that has an impact.

 

I would like to bring to light some additional concerns.  Last year I suffered a dual-drive failure.  Luckily I was able to fully recover (part of the brilliance of unRAID).  Since I still had them laying around, I decided to give both of these bad drives a workout in the new beta pre-clear, and I was not pleased with the results.

 

Both drives (WD Red 3TB) have the same problem, they randomly go offline.  Through multiple efforts, I've never been able to even complete a single pass, start to finish, on either drive.  This is both inside the array and outside the server on my desktop with my various rebuild attempts.

 

On the beta pre-clear, I ran a 3 pass with pre/post reads. 

 

One drive (my old parity drive) went offline spectacularly during the first pre-read (my server couldn't see it anymore, but in its place a new drive showed up with 6PB, yes 6 PetaBytes, how I wish!!!).  Oddly, the pre-clear didn't report any issues, and when I clicked on the "preview" icon the output window was blank.

 

The other drive (an old data drive) somehow miraculously made it through the first pre-read and the zeroing, before going offline at exactly the beginning of the first post-read - at least that's is what the pre-clear would have me believe.  Oddly, I let the first post-read run for several hours, the progress stayed at exactly 0%, but the beta pre-clear script was happily continuing as if nothing was wrong.

 

In reality, I don't think that drive even made it through the zeroing, I think the pre-clear just kept marching on ignoring how the drive was responding.  The chances that this drive made it through two complete back to back passes (a read and a zero), when I could even get 1 complete pass in over a dozen tries, only to fail exactly at the beginning of the 3rd pass... I think I have better odds playing the lottery.

 

I plan to run Joe L.'s script and see how it behaves on the same drives.  I don't recall if I've ever used his script on a known bad drive, but I'm thinking that it correctly reports when a drive has issues.

 

-Paul

Link to comment
  • 2 weeks later...

2016.08.26

Remove this plugin

 

Ehhhmm? What is this? Why?

 

You should remove Preclear Beta from your system.

 

The new script has been integrated in the normal Preclear Script plugin.

So I don't have to install the seperate preclear script with the normal preclear plugin anymore?

 

If so, you should update the description on the xml (for CA).

Link to comment

2016.08.26

Remove this plugin

 

Ehhhmm? What is this? Why?

 

You should remove Preclear Beta from your system.

 

The new script has been integrated in the normal Preclear Script plugin.

 

I am a bit confused. I updated beta preclear plugin and it appears to have removed itself. Came here and read your note about them merging. Installed Preclear from CA (btw, beta is still listed in there). Opened plugin and noted the version it sees is "Present, version 0.6-beta".

 

Am I on the latest version or not?

Link to comment

2016.08.26

Remove this plugin

 

Ehhhmm? What is this? Why?

 

You should remove Preclear Beta from your system.

 

The new script has been integrated in the normal Preclear Script plugin.

So I don't have to install the seperate preclear script with the normal preclear plugin anymore?

 

If so, you should update the description on the xml (for CA).

And remove the xml for the beta version.  Don't suppose there's much reason to have a plg for CA that does absolutely nothing.  (Additionally, update the image within the .xml since it doesn't work anymore either)
Link to comment

is this broken in the latest version for RC4? I kicked off a preclear last night for a new drive, get up this morning and it's here...

 

Pre-Read (1 of 3): 2% @ 165 MB/s (10:09:39)

 

tried to stop it and it won't stop, well it gives the cancel dialog but keeps going.. finally had to reboot the server to get it to stop

 

version shows as - 2016.08.26b

set up to use the internal script

 

Script: Present, version 0.6-beta

 

and getting

 

find: '/tmp/.preclear/' : No such file or directory

 

on the console...

 

er... this is not good?

 

Link to comment

ADDENDUM:

 

Restarted it this after the above was posted as of right now -

 

Pre-Read: 1% @ 152 MB/s (7:51:46)

 

pulled up the status page and get...

\

unRAID Server Pre-Clear of disk /dev/sdb

Cycle 1 of 1, partition start on sector 64.

Step 1 of 5 - Pre-read in progress: (1% Done)

** Time elapsed: 7:52:29 | Current speed: 159 MB/s | Average speed: 2 MB/s

 

and a constant stream of -

 

find: '/tmp/.preclear/' : No such file or directory

 

putty in and there IS '/tmp/.preclear/' directory which has 'sdb' folder which has

 

ls -l

total 32

-rw-rw-rw- 1 root root  97 Aug 27 18:41 dd_output

-rw-rw-rw- 1 root root 3984 Aug 27 18:41 display_out

-rw-rw-rw- 1 root root    6 Aug 27 10:44 pid

-rw-rw-rw- 1 root root  187 Aug 27 10:44 smart_cycle_1_start

-rw-rw-rw- 1 root root  187 Aug 27 10:44 smart_cycle_initial_start

-rw-rw-rw- 1 root root  205 Aug 27 10:44 smart_final

-rw-rw-rw- 1 root root  226 Aug 27 10:44 smart_final_540484710986852980

-rw-rw-rw- 1 root root  414 Aug 27 10:44 smart_out

 

seeing no (obvious) errors in the log that would be repeating every 30 seconds oe ao (ie. find: '/tmp/.preclear/' : No such file or directory)

 

diagnostics attached...

 

this is definitely not good

pinewood-diagnostics-20160827-1845.zip

Link to comment

well the good new is the find error has gone away, the bad new is it's still runing like cold molasses

 

been running about 15 minutes, peek at the preclear report says...

                                                                                             

#                                        unRAID Server Pre-Clear of disk /dev/sdb

#                                      Cycle 1 of 3, partition start on sector 64

 

#  Step 1 of 5 - Pre-read in progress: (0% Done) 

#  ** Time elapsed: 0:16:28 | Current speed: 158 MB/s | Average speed: 2 MB/s 

 

same conditions as before, I dtopped the last one at 14 hours and 4% done of the 1st pass .... got a problem here somewhere

Link to comment

Let's clear some things:

 

1) this plugin currently works with two different preclear scripts: the original one from Joe L., which I can't redistribute (see OP for more detailed info) and the one I wrote (previously called "Beta script"), which is included in the plugin by default. If you download the preclear disk script (Joe L. or bjp999) and put it on "/boot/config/plugins/preclear.disk/" you will be able to select between them when starting a preclear;

 

2) Performance hits: the new preclear script is, by design, faster than the original script wrote by Joe L. All performance hits are result of me decreasing the default read chunk size to 32MB. You can easily increase it by setting a higher number on the "Read Size" select box. I'll improve the code soon.

Link to comment

h, I understand, was just just such a glarring hit in performance, the way it's running right now (your script, default settings) it would be 500 hours per pass :(  Also it still won't stop on it's own, when trying to cancel it. Have to reboot the server to get it clear to start again.

 

Yes this has been with your script, not Joe or bjp999, do those STILL require modification to work under 6.2?

What do you recommend the read size be set at to get performance back?

 

Might I suggest a new OP bringing things up to date? Gather together all the info on one spot since this has gone thru so much changes and such, spread over 50+ pages it's confusing and things get missed when people are trying to set things up.

 

 

Link to comment

Thanks for the info.

 

When switching to the Joe L. script(actually bjp999) I have the option for fast post read.

 

But, I have a different issue now. I just ran 2 simultaneous 5 cycle preclears in 15 minutes. The reason it was so fast is that it is only looking at 1gb of the 2tb. The only thing I did is choose the script and activate the fast post read. Tried it without the fast post read and it did the same thing. Normally I would think it was the drive but to have 2 drives with the exact same problem. Any help anyone can provide would be greatly appreciated.

 

Here is a screenshot of the log on a freshly started preclear.

 

f3SoGdn.jpg

Link to comment

Thanks for the info.

 

When switching to the Joe L. script(actually bjp999) I have the option for fast post read.

 

But, I have a different issue now. I just ran 2 simultaneous 5 cycle preclears in 15 minutes. The reason it was so fast is that it is only looking at 1gb of the 2tb. The only thing I did is choose the script and activate the fast post read. Tried it without the fast post read and it did the same thing. Normally I would think it was the drive but to have 2 drives with the exact same problem. Any help anyone can provide would be greatly appreciated.

 

Here is a screenshot of the log on a freshly started preclear.

 

f3SoGdn.jpg

 

Please update to 2016.08.29 and try again.

Link to comment

ok frustration level rising sorry...

 

started with your script.. 

kicked off but still slow, was showing 2mb/s read on the log,  oh forgot to raise it.

Click the red "X" to stop... wait .. wair.. wait...

won't stop so I can restart it after changing values. reboot server

start again, changed it to 2mb read size, kick it off...

showing 28mb/s read speevs (ve what I guess an effective read speed of 165mb/s.

kill it to try Joe L. version... wait wait wait... reboot server

 

see the note about 08.29 version, load it

get Joe.L script, first time hangs on "starting" got the 'Drive Busy' message, ok guess it still needs patching

kill it.. wait.. wait... wait.. reboot

ok patch the script after finally finding the thread... kick it off with Joe L version,

 

/boot/config/plugins/preclear.disk/preclear_disk.sh  -M 4 -o 2 -c 3 -J /dev/sdb
sfdisk: invalid option -- 'R'

Usage:
sfdisk [options]  [[-N] ]
sfdisk [options] 

Display or manipulate a disk partition table.

Commands:
-A, --activate  [ ...] list or set bootable MBR partitions
-d, --dump                   dump partition table (usable for later input)
-J, --json                   dump partition table in JSON format
-g, --show-geometry [ ...]   list geometry of all or specified devices
-l, --list [ ...]            list partitions of each device
-F, --list-free [ ...]       list unpartitions free areas of each device
-s, --show-size [ ...]       list sizes of all or specified devices
-T, --list-types                  print the recognized types (see -X)
-V, --verify [ ...]          test whether partitions seem correct

--part-label   [] print or change partition label
--part-type   [] print or change partition type
--part-uuid   [] print or change partition uuid
--part-attrs   [] print or change partition attributes

                      device (usually disk) path
                     partition number
                     partition type, GUID for GPT, hex for MBR

Options:
-a, --append              append partitions to existing partition table
-b, --backup              backup partition table sectors (see -O)
     --bytes               print SIZE in bytes rather than in human readable format
-f, --force               disable all consistency checking
     --color[=]      colorize output (auto, always or never)
                             colors are enabled by default
-N, --partno         specify partition number
-n, --no-act              do everything except write to device
     --no-reread           do not check whether the device is in use
-O, --backup-file   override default backup file name
-o, --output        output columns
-q, --quiet               suppress extra info messages
-X, --label         specify label type (dos, gpt, ...)
-Y, --label-nested  specify nested label type (dos, bsd)

-L, --Linux               deprecated, only for backward compatibility
-u, --unit S              deprecated, only sector unit is supported

-h, --help     display this help and exit
-v, --version  output version information and exit

Available columns (for -o):
gpt: Device Start End Sectors Size Type Type-UUID Attrs Name UUID
dos: Device Start End Sectors Cylinders Size Type Id Attrs Boot End-C/H/S Start-C/H/S
bsd: Slice Start End Sectors Cylinders Size Type Bsize Cpg Fsize
sgi: Device Start End Sectors Cylinders Size Type Id Attrs
sun: Device Start End Sectors Cylinders Size Type Id Flags

For more details see sfdisk(.
Sorry: Device /dev/sdb is busy.: 1

 

what am I doing wrong? I've precleared a dozen drives on this machine pre-6.2

 

 

 

 

Link to comment
  • trurl featured, unfeatured and pinned this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.