Preclear plugin

[gfjardim]

Hi guys, this is a simple plugin that allows users to clear their disks before add them to the array.

 

The main characteristics of this plugin are:

• Modularity: can be used standalone or in conjunction with Joe L. or bjp999 scripts;
• Ease of use: with a few clicks you can start a clear session on your disk;
• Integration: you can always access the plugin under Tools > Preclear Disk menu. If you have Unassigned Devices installed, you can start/stop/view preclear sessions directly from Main > Unassigned Devices.
• All dependencies included: you don't need SCREEN to run a preclear session in the background; all jobs are executed in the background by default, so you can close your browser while the preclear runs.

 

You can install it directly or via Community Apps.

 

Q & A:

 

Q) Why Joe L. or bjp999 scripts are not included?

A) I'm not authorized by Joe L. to redistribute his script, so you need to download a copy from the topic above and put it under /boot/config/plugins/preclear.disk/ renaming it to preclear_disk.sh if necessary. bjp999 modifications are unofficial, so I decided not to include it by default.

A) the bjp999 script is now included by default.

 

Q) By default, I see a "gfjardim" script always available. Why?

A) Since I'm not authorized to redistribute Joe L. script and the recent slow support by the author, I decided a major code rewrite in the script was needed. The new script is being actively supported, compatible with unRAID notifications, is faster than bjp999 script and has a cleaner output so users can easily visualizes what's going on with their disks.

 

Q) I want to use one of the older scripts(Joe L. or bjp999) in conjunction with notifications. Is that possible?

A) Yes. I've made some adjustments on both scripts so they become compatible with unRAID notifications; Joe L. version can be found here and bjp999 can be found here.

A) the bjp999 script is now included by default; it includes support for Unraid notifications.

 

Q) Is there any howtos available?

A) gridrunner made a awesome video explaining why preclearig a hard disk is a good idea, and how you can accomplish that:

 

 

Q) The plugin asked me to send some statistics information. How does the statistics report system work? Is it safe? Is it anonymous?

A) To better track the usage of the plugin, a statistics report system was put in place. The main goals I intend to archive are:

• know number of disks that gets precleared;
• fix any silent bugs that gets reported on the logs;
• know average size of disks, their model, their average speed and elapsed time we should expect from that model;
• success rate;
• rate of disks with SMART problems;

 

This system is totally optional and users will get prompted if they want to send each report. It is also safe and totally anonymous, since all data is sent to Google Forms and no identifying data is exported, like disks serial numbers. Detailed info can be found here. The statistics are public and can be found here.

 

Q) How can I download a copy of the plugin log?

A) Please go to Tools, then Pleclear Disk, and click on the Download icon:

 

Q) Which are the differences between Erase and Clear?

A) The Clear option uses zeroes to fill the drive; at the end, the drive can be added to the array the array immediately.

The Erase All the Disk option uses random data to wipe out the drive; the resulting drive can't be quickly added to the array. If you want to add if after erase, you must select Erase and Clear the Disk.

 

Troubleshooting: 

 

Q) After Zeroing the disk, the Post-Read operation fails saying my drive isn't zeroed.

A) When zeroing the disk, the script uses a zero filled data stream produced by the pseudo-device /dev/zero. If a Post-Read fails just after a Zeroing operation, chances are that you have bad RAM memory, or less frequently bad PSU, bad cables or bad SAS/SATA card. Please run some rounds of MEMTEST on your machine to test your RAM modules.

 

Q) A Pre-Read operation failed and I see Pending Sectors on the SMART report.

A) Pending Sectors will lead to read errors, and the Pre-Read operation will fail. To force the hard drive firmware to remap those sectors, you have to run a Preclear session with the Skip Pre-Read option checked. 

 

Q) I've lost communication with the webgui, can I manage preclear sessions from the terminal?

A) Yes, you can. If you lost communication with the webgui or want to use the command line interface to manage your preclear sessions, you just need to type preclear on your terminal to start/stop or observe a preclear session.

Edited May 6, 2020 by gfjardim
Add cli interface

[WeeboTech]

Nice job!

[interwebtech]

way cool. no disks to test on . Can we get "faster" preclear as an option?

[gfjardim]

way cool. no disks to test on . Can we get "faster" preclear as an option?

 

Will try to add it, since I based this script on Joe L. version, not bjp999 one. Will run a diff and see what I need to change.

[garycase]

VERY nice -- I run VERY few add-ons, but this will definitely be one of them 

[gfjardim]

way cool. no disks to test on . Can we get "faster" preclear as an option?

 

Will try to add it, since I based this script on Joe L. version, not bjp999 one. Will run a diff and see what I need to change.

 

Done. It was easier than I think. A good diff tool is capable of doing magical things...

 

VERY nice -- I run VERY few add-ons, but this will definitely be one of them 

 

Thanks garycase! Hope you enjoy it.

[interwebtech]

Thanks for doing this. Now I just need an excuse for a new disk

[archedraft]

Wow that looks great! unRAID 6 is really going to be amazing with all of these new features.

[CHBMB]

Hi guys, I made a very simple plugin to act as a frontend to Joe L. preclear_disk.sh script.

 

It's not thoroughly tested so be careful. If possible, use a test server.

 

Plugin: https://raw.githubusercontent.com/gfjardim/unRAID-plugins/master/plugins/preclear.disk.plg

 

It's available under Settings/Preclear.

 

Best regards.

 

Assuming Jo is cool about this, I remember he got a bit annoyed a fair while back when someone, albeit with the best of intentions, took his work and implemented it in a plugin or something, although if I'm correct it was a little more complicated than that and to be honest I could see his point.

[gfjardim]

 

Assuming Jo is cool about this, I remember he got a bit annoyed a fair while back when someone, albeit with the best of intentions, took his work and implemented it in a plugin or something, although if I'm correct it was a little more complicated than that and to be honest I could see his point.

 

I contacted Joe, but no word yet. If he complains about this, I will withdraw the plugin. But honest, I don't see why he would do that, since I'm not stealing code or trying to take credit for his work. I never claimed this plugin do preclear disks, only that it is a frontend to his script.

 

I'm hoping he will enjoy it. 

[PeterB]

I suspect that JoeL will be fine with this.

 

The problem with the 'other' plugin was that, rather than acting as a front end, it reproduced the whole of Joe's script - creating a maintenance nightmare.  Joe had no control over the contents of the plugin.

[thestewman]

Installed as per the URL above and the installation went without any errors.

The plugin was installed and noted on the Plugin page as installed but it was not listed on the Settings/utilities page.

I rebooted the system and still no listing on the Settings/Utilities page.

I did the latest update that added the fast preclear and still no listing in Settings/utilities

syslog-2015-05-18.txt

[gfjardim]

Installed as per the URL above and the installation went without any errors.

The plugin was installed and noted on the Plugin page as installed but it was not listed on the Settings/utilities page.

I rebooted the system and still no listing on the Settings/Utilities page.

I did the latest update that added the fast preclear and still no listing in Settings/utilities

 

Can you post a screenshot of the Settings page?

[CHBMB]

Installed as per the URL above and the installation went without any errors.

The plugin was installed and noted on the Plugin page as installed but it was not listed on the Settings/utilities page.

I rebooted the system and still no listing on the Settings/Utilities page.

I did the latest update that added the fast preclear and still no listing in Settings/utilities

 

I've just noticed this as well.

 

[trurl]

Haven't used it yet, but I can confirm that it does appear on my Settings page in User Utilities, where the Dynamix System Stats and System Temp are.

[interwebtech]

down here

[gfjardim]

My bad. It wouldn't appear if it's the only plugin there.

 

Fixed in 2015.05.18a

[CHBMB]

My bad. It wouldn't appear if it's the only plugin there.

 

Fixed in 2015.05.18a

 

Yep, that's done the trick.

[thestewman]

My bad. It wouldn't appear if it's the only plugin there.

 

Fixed in 2015.05.18a

 

Thank You

 

Updated to 2015.05.18b and now it appears properly

 

[morbidpete]

I have 3 drives preclearing right now. Used screen on all 3 sessions. installed the plugin. Actually picked up all 3 preclears and is reporting status. Awesome! I do wish I could enable the email notification after the fact. but not a big deal. Awesome job man!

[gfjardim]

I have 3 drives preclearing right now. Used screen on all 3 sessions. installed the plugin. Actually picked up all 3 preclears and is reporting status. Awesome! I do wish I could enable the email notification after the fact. but not a big deal. Awesome job man!

 

Thank you.

 

Please note that as you started your preclears with screen sessions, you can't control them from the plugin, just see the progress. This plugin uses another program, called tmux, to run preclear script on background sessions. Tmux is a lot better than screen, mainly when we talk about scripting - we can easily start/stop processes and get a session's output. Therefore you can't stop your current preclear sessions, nor see it's output.

 

The preclear_disk.sh script version I include in this plugin already has support for the new unRAID notification system. Right now, you will receive notifications through all configured channels, but in a future version I'll add the ability to set which channels to use (email, browser popup and agents).

[kamhighway]

Very nice!

 

Thank you for making this available to the community.

 

 

[JonathanM]

The preclear_disk.sh script version I include in this plugin already has support for the new unRAID notification system. Right now, you will receive notifications through all configured channels, but in a future version I'll add the ability to set which channels to use (email, browser popup and agents).

I STRONGLY recommend not modifying Joe. L's script without his direct permission and control. It got ugly the last time someone did. If you wish to rewrite the preclear script from scratch so it is your own work and name it something else, there will be no conflict.

[gfjardim]

The preclear_disk.sh script version I include in this plugin already has support for the new unRAID notification system. Right now, you will receive notifications through all configured channels, but in a future version I'll add the ability to set which channels to use (email, browser popup and agents).

I STRONGLY recommend not modifying Joe. L's script without his direct permission and control. It got ugly the last time someone did. If you wish to rewrite the preclear script from scratch so it is your own work and name it something else, there will be no conflict.

 

I'm not doing it by choice. I had to skip some of those prompts, and we can't use the unmenu's mail package with unRAID 6.

 

He has being notified about this; I already told that if he has any problems, I'll shut this down. I'm not appropriating his code, only updating it to v6 notification system and make it proper to automation (which by the way is bjp999's work). The core functionality is untouched.

 

bjp999 has actively agreed to let me use his version of the code with the fast post-read verification.

 

EDIT: You were right, jonathanm. I didn't know about the previous attempt of making a plugin without his permission. Just removed the script from the plugin before any problems.

[gfjardim]

Guys, I've just being alerted of this: http://lime-technology.com/forum/index.php?topic=26121.0

 

So I removed the plugin link until I get a response from Joe L.