Preclear plugin


3040 posts in this topic Last Reply

Recommended Posts

My script formats in 4k alignment by default; didn't implemented "start on sector 63" because I thought it wasn't relevant anymore. If you need, you can use Joe L. script and convert it.

 

Enviado de meu SM-G930F usando Tapatalk

 

 

Link to post
  • Replies 3k
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Hi guys, this is a simple plugin that allows users to clear their disks before add them to the array.   The main characteristics of this plugin are: Modularity: can be used standalon

I think Thanks should be directed at all those members that have been dealing with the mess the last few months. @dlandon and @Squid have been working behind the scenes to try and clean up, and o

Dontlookoverthere over on the unRAID subreddit figured out how to get the plugin running again with a simple edit to the plg:   https://www.reddit.com/r/unRAID/comments/7wjpus/preclear_worki

Posted Images

On 6.2.4 I have 2 disks that are suddenly failing to post read at all.  I am guessing it is a configuration problem.

 

I wanted to try and run it via command line.  Is this likely to result in a different result?  Or is something else wrong?

 

See the log

 

preclear_disk_sdo: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --cycles 1 --no-prompt /dev/sdo
preclear_disk_sdo: Pre-Read: dd if=/dev/sdo of=/dev/null bs=2097152  iflag=direct
preclear_disk_sdn: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --cycles 1 --no-prompt /dev/sdn
preclear_disk_sdn: Pre-Read: dd if=/dev/sdn of=/dev/null bs=2097152  iflag=direct
preclear_disk_sdo: Zeroing: dd if=/dev/zero of=/dev/sdo bs=2097152 seek=1 conv=fdatasync,noerror oflag=direct
preclear_disk_sdo: Zeroing: dd pid [15956]
preclear_disk_sdn: Zeroing: dd if=/dev/zero of=/dev/sdn bs=2097152 seek=1 conv=fdatasync,noerror oflag=direct
preclear_disk_sdn: Zeroing: dd pid [26846]
preclear_disk_sdo: Post-Read: dd if=/dev/sdo bs=512 count=4096 skip=1 iflag=direct
preclear_disk_sdo: Post-Read: dd pid [3189]
preclear_disk_sdo: Post-Read: dd if=/dev/sdo bs=2097152  skip=1 iflag=direct 2>/tmp/.preclear/sdo/dd_output | cmp - /dev/zero &>/tmp/.preclear/sdo/cmp_out 
preclear_disk_sdo: Post-Read: dd pid [3232]
preclear_disk_sdo: Post-Read: fail - disk not zeroed
preclear_disk_sdo: 
preclear_disk_sdo: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --cycles 1 --no-prompt /dev/sdo
preclear_disk_sdo: Pre-Read: dd if=/dev/sdo of=/dev/null bs=2097152  iflag=direct
preclear_disk_sdn: Post-Read: dd if=/dev/sdn bs=512 count=4096 skip=1 iflag=direct
preclear_disk_sdn: Post-Read: dd pid [13590]
preclear_disk_sdn: Post-Read: dd if=/dev/sdn bs=2097152  skip=1 iflag=direct 2>/tmp/.preclear/sdn/dd_output | cmp - /dev/zero &>/tmp/.preclear/sdn/cmp_out 
preclear_disk_sdn: Post-Read: dd pid [13627]
preclear_disk_sdn: Post-Read: fail - disk not zeroed
preclear_disk_sdn: 
preclear_disk_sdo: Zeroing: dd if=/dev/zero of=/dev/sdo bs=2097152 seek=1 conv=fdatasync,noerror oflag=direct
preclear_disk_sdo: Zeroing: dd pid [13096]
preclear_disk_sdo: Post-Read: dd if=/dev/sdo bs=512 count=4096 skip=1 iflag=direct
preclear_disk_sdo: Post-Read: dd pid [29811]
preclear_disk_sdo: Post-Read: dd if=/dev/sdo bs=2097152  skip=1 iflag=direct 2>/tmp/.preclear/sdo/dd_output | cmp - /dev/zero &>/tmp/.preclear/sdo/cmp_out 
preclear_disk_sdo: Post-Read: dd pid [29881]
preclear_disk_sdo: Post-Read: fail - disk not zeroed
preclear_disk_sdo: 
preclear_disk_sdo: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --cycles 1 --no-prompt /dev/sdo
preclear_disk_sdo: Pre-Read: dd if=/dev/sdo of=/dev/null bs=2097152  iflag=direct

Link to post

Running 6.2.4.  Put a new 4TB Seagate in, but I'm getting the following when trying to preclear using the plugin:

 

/boot/config/plugins/preclear.disk/preclear_disk.sh  -M 3 -o 4 -c 3 -f -J /dev/sdl
root@Tower:/usr/local/emhttp# /boot/config/plugins/preclear.disk/preclear_disk.sh  -M 3 -o 4 -c 3 -f -J /dev/sdl
sfdisk: invalid option -- 'R'

Usage:
sfdisk [options]  [[-N] ]
sfdisk [options] 

Display or manipulate a disk partition table.

Commands:
-A, --activate  [ ...] list or set bootable MBR partitions
-d, --dump                   dump partition table (usable for later input)
-J, --json                   dump partition table in JSON format
-g, --show-geometry [ ...]   list geometry of all or specified devices
-l, --list [ ...]            list partitions of each device
-F, --list-free [ ...]       list unpartitions free areas of each device
-s, --show-size [ ...]       list sizes of all or specified devices
-T, --list-types                  print the recognized types (see -X)
-V, --verify [ ...]          test whether partitions seem correct

--part-label   [] print or change partition label
--part-type   [] print or change partition type
--part-uuid   [] print or change partition uuid
--part-attrs   [] print or change partition attributes

                      device (usually disk) path
                     partition number
                     partition type, GUID for GPT, hex for MBR

Options:
-a, --append              append partitions to existing partition table
-b, --backup              backup partition table sectors (see -O)
     --bytes               print SIZE in bytes rather than in human readable format
-f, --force               disable all consistency checking
     --color[=]      colorize output (auto, always or never)
                             colors are enabled by default
-N, --partno         specify partition number
-n, --no-act              do everything except write to device
     --no-reread           do not check whether the device is in use
-O, --backup-file   override default backup file name
-o, --output        output columns
-q, --quiet               suppress extra info messages
-X, --label         specify label type (dos, gpt, ...)
-Y, --label-nested  specify nested label type (dos, bsd)

-L, --Linux               deprecated, only for backward compatibility
-u, --unit S              deprecated, only sector unit is supported

-h, --help     display this help and exit
-v, --version  output version information and exit

Available columns (for -o):
gpt: Device Start End Sectors Size Type Type-UUID Attrs Name UUID
dos: Device Start End Sectors Cylinders Size Type Id Attrs Boot End-C/H/S Start-C/H/S
bsd: Slice Start End Sectors Cylinders Size Type Bsize Cpg Fsize
sgi: Device Start End Sectors Cylinders Size Type Id Attrs
sun: Device Start End Sectors Cylinders Size Type Id Flags

For more details see sfdisk(.
Sorry: Device /dev/sdl is busy.: 1
root@Tower:/usr/local/emhttp#

 

Unassigned devices shows the preclear as "Starting..."  What can I do?  The "sfdisk: invalid option -- 'R'" sure sounds a lot like the errors reported back in post #220, is the beta plugin still necessary?

Link to post

IIRC, Remove the beta plugin and install the nonbeta plugin.

 

Oh, what I meant is, should I try the beta plugin?  I've never used it before but I saw that it was developed to work with 6.2-beta.  Now that 6.2 is final, is it still needed?

Have you tried using the built in script rather than the bjp999 script? Worked great for me and should be faster.

Link to post

Post-read has determined that the disk was not zeroed. I had a similar problem once that was caused by a bad RAM stick.

 

You are quite a fortune teller.  The disks both precleared fine in another server, and when I shut this server down to run memtest, it immediately failed.  This was my only server without ECC memory, and I am sure I ran memtest on it when it was set up a year ago. 

 

I hope I don't have corrupted files.... 

Link to post

Have you tried using the built in script rather than the bjp999 script? Worked great for me and should be faster.

 

Doh!  You're right.

 

I ended up completely uninstalling the Preclear Plugin as well as Unassigned Devices, and removing all traces of them from my flash drive.  Upon reinstall I only had bjp999's script so I tried that.  Working like a charm ;)

Link to post

Guys I need help:

 

Yesterday I built a unRaid server with 4 WD Red 6TB drives. I downloaded Preclear app from the community store, and currently clearing all 4 drives simultaneously with 16GB of ram, is it wise?  For more than an hour the statuses still show "Starting..." and clicking on preview I get this:

 

/usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh  --notify 2 --frequency 4 --cycles 2 --no-prompt /dev/sdb

 

As I understand it: I ran the script, report every 25%, and run 2 cycles. I read the last 10 pages trying to understand and the FAQ to see if I did something wrong but this is unnerving

 

Edit: I found out that I need to hit New Config button in the Tools tab since all of my drives are new. Preclear is running now

Link to post
  • 2 weeks later...
  • 2 weeks later...

Hi, Guys. Here's a video tutorial about preclearing your hard drives. It gives a little information about how it works and how to use it.

I hope some people find it useful :)

How and why to PreClear your hard drives before adding to your array

 

Awesome video, gridrunner, added to OP if you don't mind.

Link to post

Hi, Guys. Here's a video tutorial about preclearing your hard drives. It gives a little information about how it works and how to use it.

I hope some people find it useful :)

How and why to PreClear your hard drives before adding to your array

 

Awesome video, gridrunner, added to OP if you don't mind.

Thanks gfjardim i'm glad you like it! I'm more than happy for it to be in the OP. Thankyou :)

 

Excellent job gridrunner!

 

Small remark, starting with unRAID 6.2 regular clearing of a new disk is done in the background and the array stays available as before.

 

Thanks bonienl. I didnt know that about the clearing in 6.2, its great the array stays available.  :)

Link to post

Guys, I'm considering add some sort of statistics report inside the plugin script. This will allow me to track potential bugs and performance data. Initially I'm thinking in gathering model, size, smart device type, total elapsed time, log and if it was successful or not.

 

Any concerns and/or suggestions?

 

Link to post

Guys, I'm considering add some sort of statistics report inside the plugin script. This will allow me to track potential bugs and performance data. Initially I'm thinking in gathering model, size, smart device type, total elapsed time, log and if it was successful or not.

 

Any concerns and/or suggestions?

 

Where is this data going?  If it is to be transmitted back to you, will there be an opt-out setting?

Link to post

Guys, I'm considering add some sort of statistics report inside the plugin script. This will allow me to track potential bugs and performance data. Initially I'm thinking in gathering model, size, smart device type, total elapsed time, log and if it was successful or not.

 

Any concerns and/or suggestions?

 

Where is this data going?  If it is to be transmitted back to you, will there be an opt-out setting?

 

To Google Forms. Since all data is anonymous (no serial exposed, for example), is there a real need for an opt-out setting?

Link to post

Guys, I'm considering add some sort of statistics report inside the plugin script. This will allow me to track potential bugs and performance data. Initially I'm thinking in gathering model, size, smart device type, total elapsed time, log and if it was successful or not.

 

Any concerns and/or suggestions?

 

Where is this data going?  If it is to be transmitted back to you, will there be an opt-out setting?

 

To Google Forms. Since all data is anonymous (no serial exposed, for example), is there a real need for an opt-out setting?

 

To be brutally  honest, I am very suspicious of any type of application that phones-home.  Almost everyone of them promises that all Data is be anonymized but it has often been found later that was not the complete case.  I would prefer to have an opt-out option but at least in this case, I have the option to doing the preclear by using the old-old shell script.  (I admit that I am paranoid and that I have disable both Flash and Java on all of my household computers.  I refuse to link e-mail servers together and/or any of the social media sites together.  I avoid clicking on 98% of the links that I am sent by 'Friends' until I have verified them from a second source.  But then I have never been hit by a virus or malware to date...)

 

EDIT----  And I didn't go to Windows 10 because I want to avoid MS snooping...

Link to post

Guys, I'm considering add some sort of statistics report inside the plugin script. This will allow me to track potential bugs and performance data. Initially I'm thinking in gathering model, size, smart device type, total elapsed time, log and if it was successful or not.

 

Any concerns and/or suggestions?

 

Where is this data going?  If it is to be transmitted back to you, will there be an opt-out setting?

 

To Google Forms. Since all data is anonymous (no serial exposed, for example), is there a real need for an opt-out setting?

 

To be brutally  honest, I am very suspicious of any type of application that phones-home.  Almost everyone of them promises that all Data is be anonymized but it has often been found later that was not the complete case.  I would prefer to have an opt-out option but at least in this case, I have the option to doing the preclear by using the old-old shell script.  (I admit that I am paranoid and that I have disable both Flash and Java on all of my household computers.  I refuse to link e-mail servers together and/or any of the social media sites together.  I avoid clicking on 98% of the links that I am sent by 'Friends' until I have verified them from a second source.  But then I have never been hit by a virus or malware to date...)

 

Since you fell strong about it can add a toggle to disable the report. These are the statistics I expect to collect:

 

https://docs.google.com/spreadsheets/d/1zERJQlFWGgMGKnKgGUNKnguGUdCQ2g0XT9Q5yMJio6o/edit?usp=sharing

Link to post
Since all data is anonymous (no serial exposed, for example), is there a real need for an opt-out setting?

Yes. And not just for the obvious reason of distrust of data collection. Adding a data upload means internet connectivity is expected, and data will be traveling. I prefer to be able to isolate a system if necessary without sacrificing functionality that has no need of internet. A purely local function that breaks because of lack of internet is broken, period. Internet activity and data collection needs to be opt in, not opt out. I understand that forces you to beg people and ask for them to please allow you to collect data, but that's the way it should be. You need to justify to the end user WHY the data you are asking them to send on their dime benefits them as well as you.
Link to post

 

Since you fell strong about it can add a toggle to disable the report. These are the statistics I expect to collect:

 

https://docs.google.com/spreadsheets/d/1zERJQlFWGgMGKnKgGUNKnguGUdCQ2g0XT9Q5yMJio6o/edit?usp=sharing

 

I hope that you will do so.  I looked at the data and I was wondering exactly what important information you expect to be able to extract from what you propose to collect. 

 

Since all data is anonymous (no serial exposed, for example), is there a real need for an opt-out setting?

Yes. And not just for the obvious reason of distrust of data collection. Adding a data upload means internet connectivity is expected, and data will be traveling. I prefer to be able to isolate a system if necessary without sacrificing functionality that has no need of internet. A purely local function that breaks because of lack of internet is broken, period. Internet activity and data collection needs to be opt in, not opt out. I understand that forces you to beg people and ask for them to please allow you to collect data, but that's the way it should be. You need to justify to the end user WHY the data you are asking them to send on their dime benefits them as well as you.

 

+1

Link to post

What I expect to collect:

 

1) number of disks that are precleared (dying to know  ;D);

2) any silent bugs that gets reported on the logs;

3) average size of disks that gets precleared, their model and the average speed and time we should expect to that model;

4) success rate;

5) rate of disks with SMART problems;

 

 

Link to post

What I expect to collect:

 

1) number of disks that are precleared (dying to know  ;D);

2) any silent bugs that gets reported on the logs;

3) average size of disks that gets precleared, their model and the average speed and time we should expect to that model;

4) success rate;

5) rate of disks with SMART problems;

 

As far as I am concerned, #2 is the only one that I would consider is not an invasion of my privacy.  While I realize that the other information is nice to have, it is not essential to working of this product.  While LimeTech 'knows' how many time its software is download (which I assume you also know) and how many licenses it has issued, it has no real knowledge of how its product is being used, actual numbers of disks and with what plugins, Dockers, or VM's.  (There are a fair number of us with Pro licenses that aren't using anywhere near the numbers of disks to require it!)

 

I realize that I have exposed some of this data in my 'signature' line of this post.  However, the collection of that data would be very difficult and expensive process and even more difficult to tie directly to me as an individual.  (The upload of the data you propose to Google will have my IP address as a part of that data which would serve Google's purposes for THEIR data collection!  Read their terms of use...)

Link to post

I must admit that if it is properly anomonized i do  cannot see how any of the suggested items invades priority.  However I guess if the decision as to whether such stats can be uploaded is an end-user choice everyone can make their own decision.

Link to post
  • trurl featured, unfeatured and pinned this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.