Preclear plugin


Recommended Posts

Hello All,

 

I tried going through the previous post listings, but was not able to see (or maybe I gleamed over it?), but is there a limit on how many drives can be pre-cleared at one time? eg, is the pre-clear based on cpu/mem or is it isolated to the hdd itself? I have a 1230v3/32GB and 20 drives, am i able to start the pre-clear on all 20 (well 18 now, 1 in array and 1 is currently running 49% @ 14:48:32) so that they will all be done roughly around the same time. Or will i have to break it up to say 5x4HDD's per pre-clear?

 

Sorry if this type of question was asked before.

 

 

Thanks.

Link to comment

Hello All,

 

I tried going through the previous post listings, but was not able to see (or maybe I gleamed over it?), but is there a limit on how many drives can be pre-cleared at one time? eg, is the pre-clear based on cpu/mem or is it isolated to the hdd itself? I have a 1230v3/32GB and 20 drives, am i able to start the pre-clear on all 20 (well 18 now, 1 in array and 1 is currently running 49% @ 14:48:32) so that they will all be done roughly around the same time. Or will i have to break it up to say 5x4HDD's per pre-clear?

 

Sorry if this type of question was asked before.

 

 

Thanks.

With unRAID 32bit v5 it was definitely possible to run out of memory preclearing. Not sure if it still an issue with 64bit v6. I'm not aware of anyone trying as many as you are suggesting so you may just have to try it and see. Can't imagine you really need all of them at once though so you might go with fewer at one time. Unless you break it by running too many you should be able to use your server for other things while preclearing, such as loading data to the drives already precleared so maybe not that important to do it all at once anyway.
Link to comment

Hello All,

 

I tried going through the previous post listings, but was not able to see (or maybe I gleamed over it?), but is there a limit on how many drives can be pre-cleared at one time? eg, is the pre-clear based on cpu/mem or is it isolated to the hdd itself? I have a 1230v3/32GB and 20 drives, am i able to start the pre-clear on all 20 (well 18 now, 1 in array and 1 is currently running 49% @ 14:48:32) so that they will all be done roughly around the same time. Or will i have to break it up to say 5x4HDD's per pre-clear?

 

Sorry if this type of question was asked before.

 

 

Thanks.

With unRAID 32bit v5 it was definitely possible to run out of memory preclearing. Not sure if it still an issue with 64bit v6. I'm not aware of anyone trying as many as you are suggesting so you may just have to try it and see. Can't imagine you really need all of them at once though so you might go with fewer at one time. Unless you break it by running too many you should be able to use your server for other things while preclearing, such as loading data to the drives already precleared so maybe not that important to do it all at once anyway.

 

Thanks for the reply,

 

I have Unraid V6.1.3? ( I think). The issue was i had a failure of my unraid HBA cards. I had 3 Supermicro HBA cards and for some reason only 2 were showing during post. So when 1 drive failed (amusingly) when it tried to rebuild, it could not. only once i shut the array down, did i notice that drives 1-8 had "disconnected device", causing all kinds of issues. I'm now on a M1015 flashed to 2008IT & SAS expander and all seems well. So now i want to wipe clean all the dirty data and start fresh. Which is why i was looking to try as many drives as possible as the current 4TB is taking 16hrs at 70%... So if i want to start my uploading this weekend... i need them all cleaned.

 

On a side note, i do notice that my cpu usage is 24-29%, so i'm going to try to load 3 more and see what happens.

Link to comment

Hello All,

 

I tried going through the previous post listings, but was not able to see (or maybe I gleamed over it?), but is there a limit on how many drives can be pre-cleared at one time? eg, is the pre-clear based on cpu/mem or is it isolated to the hdd itself? I have a 1230v3/32GB and 20 drives, am i able to start the pre-clear on all 20 (well 18 now, 1 in array and 1 is currently running 49% @ 14:48:32) so that they will all be done roughly around the same time. Or will i have to break it up to say 5x4HDD's per pre-clear?

 

Sorry if this type of question was asked before.

 

 

Thanks.

With unRAID 32bit v5 it was definitely possible to run out of memory preclearing. Not sure if it still an issue with 64bit v6. I'm not aware of anyone trying as many as you are suggesting so you may just have to try it and see. Can't imagine you really need all of them at once though so you might go with fewer at one time. Unless you break it by running too many you should be able to use your server for other things while preclearing, such as loading data to the drives already precleared so maybe not that important to do it all at once anyway.

 

Thanks for the reply,

 

I have Unraid V6.1.3? ( I think).

 

.......

 

On a side note, i do notice that my cpu usage is 24-29%, so i'm going to try to load 3 more and see what happens.

 

I seem to recall that the issue is actually the memory required for each disk being precleared, not the CPU usage. 

Link to comment

On a side note, i do notice that my cpu usage is 24-29%, so i'm going to try to load 3 more and see what happens.

 

I seem to recall that the issue is actually the memory required for each disk being precleared, not the CPU usage.

 

Hmm, from what i see, the mem usage is 2% but the cpu is now 45-53% (with 4 pre-clear drives running). In this instance it seems to be more cpu then memory?

 

I think i'm going to throw 2 more disk's on and call it. I don't want to stress the disk's/system/plugin with too much. I'll do another 6 after these finish.

 

Link to comment

I'm having an issue where my system is saying the script isn't present.  This is on a system that I started building in the spring, but it got put on the back burner until now.  At the time, I precleared 4 drives, with the script, and I'm still using the same flash drive.

When I try running the plugin, I get script not present.

Any way to diagnose why it's not seeing the script?

Link to comment

I'm having an issue where my system is saying the script isn't present.  This is on a system that I started building in the spring, but it got put on the back burner until now.  At the time, I precleared 4 drives, with the script, and I'm still using the same flash drive.

When I try running the plugin, I get script not present.

Any way to diagnose why it's not seeing the script?

 

Is it in the right folder?

 

It doesn't come with the preclear script itself, which can be found here. Please put a copy of the script to /boot/config/plugins/preclear.disk/ directory.

Link to comment

Love this plugin, it's wonderful.

 

Today I almost got a heart attack, though. I had preclear expected to be finished today, and it was, but the result was written in alarmingly red color, making me think for a split second that something dramatically bad had happened. But actually everything was perfectly alright.

 

My suggestion would be to use a green font for successful preclear. And maybe a red font if something went wrong. In the end I don't care much about the color, but a red font for a successful operation is not a good choice, IMHO. Obviously this is an extremely minor complaint.

Link to comment

is this something to be worried about during a preclear? never seen these in the logs when preclearing a drive. sdg is the drive that i'm currently preclearing, and viewing the status it seems to be still preclearing OK.

 

Dec  2 00:12:50 Tower udevd[3424]: timeout: killing 'ata_id --export /dev/sdg' [3425]
Dec  2 00:12:51 Tower udevd[3424]: timeout: killing 'ata_id --export /dev/sdg' [3425]
Dec  2 00:12:52 Tower udevd[3424]: timeout: killing 'ata_id --export /dev/sdg' [3425]
Dec  2 00:12:53 Tower udevd[3424]: 'ata_id --export /dev/sdg' [3425] terminated by signal 9 (Killed)
Dec  2 00:12:53 Tower udevd[3424]: timeout 'scsi_id --export --whitelisted -d /dev/sdg'
Dec  2 00:13:35 Tower kernel: sdg: sdg1

 

 

EDIT: okay just to add a little more info..

 

root@Tower:~# cat /var/log/syslog |grep udevd
Dec  2 00:12:49 Tower udevd[3424]: timeout 'ata_id --export /dev/sdg'
Dec  2 00:12:50 Tower udevd[3424]: timeout: killing 'ata_id --export /dev/sdg' [3425]
Dec  2 00:12:51 Tower udevd[3424]: timeout: killing 'ata_id --export /dev/sdg' [3425]
Dec  2 00:12:52 Tower udevd[3424]: timeout: killing 'ata_id --export /dev/sdg' [3425]
Dec  2 00:12:53 Tower udevd[3424]: 'ata_id --export /dev/sdg' [3425] terminated by signal 9 (Killed)
Dec  2 00:12:53 Tower udevd[3424]: timeout 'scsi_id --export --whitelisted -d /dev/sdg'
Dec  2 14:06:25 Tower udevd[30612]: timeout 'ata_id --export /dev/sdg'
Dec  2 14:06:26 Tower udevd[30612]: timeout: killing 'ata_id --export /dev/sdg' [30613]
Dec  2 14:06:27 Tower udevd[30612]: timeout: killing 'ata_id --export /dev/sdg' [30613]
Dec  2 14:06:28 Tower udevd[30612]: 'ata_id --export /dev/sdg' [30613] terminated by signal 9 (Killed)
Dec  2 14:06:28 Tower udevd[30612]: timeout 'scsi_id --export --whitelisted -d /dev/sdg'
Dec  3 03:59:05 Tower udevd[18972]: timeout 'ata_id --export /dev/sdg'
Dec  3 03:59:06 Tower udevd[18972]: timeout: killing 'ata_id --export /dev/sdg' [18973]
Dec  3 03:59:07 Tower udevd[18972]: 'ata_id --export /dev/sdg' [18973] terminated by signal 9 (Killed)
Dec  3 03:59:07 Tower udevd[18972]: timeout 'scsi_id --export --whitelisted -d /dev/sdg'

 

i started the preclear on dec 1 at around 1-2pm, the times of these errors is about dec 2 00:12, 14:06, and dec 3 03:59. i'm thinking this aligns with some part of the preclear process, as i've run 3 cycles, and it's about 11-14 hours apart which based on my estimation, this is about how long it takes for 1 cycle??

 

and lastly, my preclear report ended up fine, no issues.

 

root@Tower:/boot/preclear_reports# cat preclear_rpt_WD-WCC4M4LCZXJY_2015-12-03
========================================================================1.15
== invoked as: /boot/config/plugins/preclear.disk/preclear_disk.sh -c 3 /dev/sdg
== WDCWD20EFRX-68EUZN0   WD-WCC4M4LCZXJY
== Disk /dev/sdg has been successfully precleared
== with a starting sector of 64
== Ran 3 cycles
==
== Using :Read block size = 1000448 Bytes
== Last Cycle's Pre Read Time  : 5:36:55 (98 MB/s)
== Last Cycle's Zeroing time   : 4:37:50 (119 MB/s)
== Last Cycle's Post Read Time : 9:15:10 (60 MB/s)
== Last Cycle's Total Time     : 13:53:59
==
== Total Elapsed Time 47:17:09
==
== Disk Start Temperature: 26C
==
== Current Disk Temperature: 27C,
==
============================================================================
** Changed attributes in files: /tmp/smart_start_sdg  /tmp/smart_finish_sdg
                ATTRIBUTE   NEW_VAL OLD_VAL FAILURE_THRESHOLD STATUS      RAW_VALUE
      Temperature_Celsius =   120     121            0        ok          27
No SMART attributes are FAILING_NOW

0 sectors were pending re-allocation before the start of the preclear.
0 sectors were pending re-allocation after pre-read in cycle 1 of 3.
0 sectors were pending re-allocation after zero of disk in cycle 1 of 3.
0 sectors were pending re-allocation after post-read in cycle 1 of 3.
0 sectors were pending re-allocation after zero of disk in cycle 2 of 3.
0 sectors were pending re-allocation after post-read in cycle 2 of 3.
0 sectors were pending re-allocation after zero of disk in cycle 3 of 3.
0 sectors are pending re-allocation at the end of the preclear,
    the number of sectors pending re-allocation did not change.
0 sectors had been re-allocated before the start of the preclear.
0 sectors are re-allocated at the end of the preclear,
    the number of sectors re-allocated did not change.
============================================================================

 

anyone preclear any drives with the plugin recently and see the same lines in their logs? is this normal? a bug?

 

 

Link to comment

Just found your plugin and installed it as directed. Moved the preclear*sh script as directed and did reboot. I can see the plugin and start it. But the Link "Start Preclear" does nothing. I tested with Chrome and IE. Please see attachments.

 

It's like "Start Preclear" is simply text and no link.

 

Thanks in advance.

preclear_gui.png.7d95601eff5a7c2becb025cf9b56caed.png

preclear_shell.png.492817d7c444e5a1365e5f92da8dc6bf.png

Link to comment

Probably feel like an idiot after I hear the answer but I cannot figure it out. I am new to unraid so be gentle ;-)

 

I have a 'production' server up and running. I thought I would fire up a test machine for some raid/sata cards I want to test out and to preclear disks. Using the onboard SATA ports with a demo lic of UnRaid I was able to get the plugin working no problems.

 

Now I want to test out a RAID card I have. Same disks, same Lic of UnRaid. Now when I click the start preclair button I get a window pop up with the following

 

Pre-Clear unRAID Disk /dev/sdc

################################################################## 1.15

smartctl may not be able to run on /dev/sdc with the -d ata option.

however this should not affect the clearing of a disk.

smartctl exit status = 2

smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.1.13-unRAID] (local build)

Copyright © 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

Read Device Identity failed: Invalid argument

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.$

Do you wish to continue?

(Answer Yes to continue. Capital 'Y', lower case 'es'): Yes

Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes

1 heads, 63 sectors/track, 62016336 cylinders, total 3907029168 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00000000

  Device Boot      Start        End      Blocks  Id  System

/dev/sdc1              64  3907029167  1953514552  83  Linux

########################################################################

invoked as  /boot/config/plugins/preclear.disk/preclear_disk.sh -c 1 /dev/sdc

########################################################################

(MBR 4k-aligned set. Partition will start on sector 64 for disks <= 2.2TB and sector 1 for disks > 2.2TB)

Are you absolutely sure you want to clear this drive?

(Answer Yes to continue. Capital 'Y', lower case 'es'):

 

I have no way of typing the Yes it is looking for as the window will not accept input. Prior to the raid card it would just start on its own. Or so that what I remember. I was testing it out a couple days ago.

 

What am I missing?

 

Many thanks in advance!

 

Mike

Link to comment

Ok, Preclear plugin is working on the onboard SATA controller, but it is failing on the 9650SE-24M8. It gets to the prompt Yes to continue and stops.

 

Thoughts?

 

I was going to use speeds to see how the Raid card performs to see if it works well enough or I should buy something else. The Raid card does work I can create an Array and write out data. Or I could.... Maybe I will try again.

 

Mike

Link to comment

If you installed the plugin you should be running the PreClear from the Plugin or Settings page in the UnRaid GUI not from the command line.

 

That's what I tried to do. As I wrote. The text "Start Preclear" in your plugin simply is plain text and no link. If I click on it nothing happens.

 

So I issued preclear*sh the traditional way from command line - and your plugin shows progress.

 

Regards.

 

Link to comment

 

There's a version hosted here: http://bit.ly/1G44UhZ . This version have fast post-read and notifications enabled.

 

You can add this with this command:

wget http://bit.ly/1G44UhZ -O /boot/config/plugins/preclear.disk/preclear_disk.sh

 

readvz is already downloaded by the plugin.

 

Used the plugin with the fast post-read and notifications enabled version from host site listed. Shows as ver. 1.115c, plugin worked great as has been the case, and pre clear of three new 4TB drives were successful also. 10 hours quicker for each drive than ones done in the past, however I never received any notifications. I had set for both the web GUI pop ups and email. i tried the 25%, end of each step, and end of cycle and never got anything.

My notifications are working fine for unraid. Is something broke or did i not do something correctly in setting it up?

Link to comment

There's a version hosted here: http://bit.ly/1G44UhZ . This version have fast post-read and notifications enabled.

 

You can add this with this command:

wget http://bit.ly/1G44UhZ -O /boot/config/plugins/preclear.disk/preclear_disk.sh

 

readvz is already downloaded by the plugin.

 

Used the plugin with the fast post-read and notifications enabled version from host site listed. Shows as ver. 1.115c, plugin worked great as has been the case, and pre clear of three new 4TB drives were successful also. 10 hours quicker for each drive than ones done in the past, however I never received any notifications. I had set for both the web GUI pop ups and email. i tried the 25%, end of each step, and end of cycle and never got anything.

My notifications are working fine for unraid. Is something broke or did i not do something correctly in setting it up?

According to your sig you are not running 6.1.x. This is likely the problem, and you should upgrade anyway. Nobody can support 6.0rc6
Link to comment

There's a version hosted here: http://bit.ly/1G44UhZ . This version have fast post-read and notifications enabled.

 

You can add this with this command:

wget http://bit.ly/1G44UhZ -O /boot/config/plugins/preclear.disk/preclear_disk.sh

 

readvz is already downloaded by the plugin.

 

Used the plugin with the fast post-read and notifications enabled version from host site listed. Shows as ver. 1.115c, plugin worked great as has been the case, and pre clear of three new 4TB drives were successful also. 10 hours quicker for each drive than ones done in the past, however I never received any notifications. I had set for both the web GUI pop ups and email. i tried the 25%, end of each step, and end of cycle and never got anything.

My notifications are working fine for unraid. Is something broke or did i not do something correctly in setting it up?

According to your sig you are not running 6.1.x. This is likely the problem, and you should upgrade anyway. Nobody can support 6.0rc6

Sorry, hadn't changed that yet. Was on 6.1.4 while preclearing. Now updated to 6.1.6

Link to comment

There's a version hosted here: http://bit.ly/1G44UhZ . This version have fast post-read and notifications enabled.

 

You can add this with this command:

wget http://bit.ly/1G44UhZ -O /boot/config/plugins/preclear.disk/preclear_disk.sh

 

readvz is already downloaded by the plugin.

 

Used the plugin with the fast post-read and notifications enabled version from host site listed. Shows as ver. 1.115c, plugin worked great as has been the case, and pre clear of three new 4TB drives were successful also. 10 hours quicker for each drive than ones done in the past, however I never received any notifications. I had set for both the web GUI pop ups and email. i tried the 25%, end of each step, and end of cycle and never got anything.

My notifications are working fine for unraid. Is something broke or did i not do something correctly in setting it up?

According to your sig you are not running 6.1.x. This is likely the problem, and you should upgrade anyway. Nobody can support 6.0rc6

Sorry, hadn't changed that yet. Was on 6.1.4 while preclearing. Now updated to 6.1.6

I also realize now that updating which required a restart wiped any logs that may of been useful correct? It appears to be working for those that use it so it must be unique to my setup. I have a small drive to test with but can't get to it for a week or so. When I do I'll run a single pass with the fast option and the 25% setting and post back results then.

The difference would be unraid version, and a single drive. I was preclearing all three of the new 4TB drives at the same time, if that makes a difference.

Link to comment

Ok, Preclear plugin is working on the onboard SATA controller, but it is failing on the 9650SE-24M8. It gets to the prompt Yes to continue and stops.

 

A few other people have also had this when running disks from something other than the onboard SATA controller.

One user fixed his by updating the firmware on his SAS card.

 

What I think is actually happening, is that the preclear script is detecting a different situation(or non-standard, or something it is not sure about), and asks for an 'extra' prompt before continuing.

It's very hard(maybe impossible) for gfjardim's wonderful GUI front end to catch this, and hence gets stuck at that point.

 

You have a couple ways around this:

  • Move the disk(s) to the onboard SATA to do the preclear.(and move it back when finished)
  • Run the script directly within a terminal session.(I believe you can still monitor progress from the GUI)
  • possibly, try any available firmware updates for your card

Link to comment
  • Squid featured, unfeatured and pinned this topic
  • Squid unpinned this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.