Jump to content
Biggy2872

Re: preclear_disk.sh - a new utility to burn-in and pre-clear disks for quick add

1379 posts in this topic Last Reply

Recommended Posts

My bad. Not paying enough attention to which thread I'm reading/posting in. Probably the major cause of my confusion. :D

Thanks.

Share this post


Link to post

The preclear_disk.sh script is failing on v6.2 betas because the sfdisk -R is no longer supported.  Preclear_disk.sh is reporting the disk as busy and will not preclear it.  It looks like 'blockdev --rereadpt' is the replacement according to the sfdisk man-pages here http://man7.org/linux/man-pages/man8/sfdisk.8.html.

 

"      Since version 2.26 sfdisk no longer provides the -R or --re-read

      option to force the kernel to reread the partition table.  Use

      blockdev --rereadpt instead."

 

EDIT: There is also an issue with reads failing.  I changed the following:

 

read_entire_disk( ) {
  # Get the disk geometry (cylinders, heads, sectors)
  fgeometry=`fdisk -l $1 2>/dev/null`
  units=`echo "$fgeometry" | grep Units | awk '{ print $9 }'`

 

to

 

read_entire_disk( ) {
  # Get the disk geometry (cylinders, heads, sectors)
  fgeometry=`fdisk -l $1 2>/dev/null`
  units=`echo "$fgeometry" | grep Units | awk '{ print $8 }'`

 

and the reads will work.

 

Joe L. - Can we get an official fix and an update from you?

I only (very) recently put 6.2 beta on my server.

 

I did not have any issue pre-clearing the second parity disk I have just added to my array.

 

The fix will need to wait until I add/replace one of the existing disks with a larger one.

(Otherwise, I have no way to test the process. )

 

Whatever the fix might be, it must be backwards compatible with the older releases of unRAID.

 

In the interim, you can type this command to "patch" the preclear_disk.sh command

First change directory to the directory holding the preclear_disk.sh command.  For most, it will be

cd /boot

then type (or copy from here and paste) the following:

sed -i -e "s/print \$9 /print \$8 /" -e "s/sfdisk -R /blockdev --rereadpt /" preclear_disk.sh

 

Your preclear disk script will be edited and should work with the two changes you mentioned.  (actually, each occurs in two places, so there are a total of 4 lines changed)

 

Joe L.

 

There's one more fix to be made. Because of dd output change, pre-read and post-read status are showing time instead of speed. I propose a more elaborate command:

 

 

sed -i -e "s/print \$9 /print \$8 /" -e "s/sfdisk -R /blockdev --rereadpt /" -e "s/  sed -n 3p [^\}]*}'/awk -F',' 'END{print \$NF}'/g" preclear_disk.sh

 

 

Share this post


Link to post

Is it quicker to preclear from a motherboard or doing it through a Supermicro AOC-SASLP-MV8 wont make a difference?

Share this post


Link to post

Is it quicker to preclear from a motherboard or doing it through a Supermicro AOC-SASLP-MV8 wont make a difference?

 

It depends. The AOC-SASLP-MV8 has a 600MB/s cap, so if you plan to do a parity check with more than 5 disks connected to it you could have a bottleneck.

Share this post


Link to post

Is it quicker to preclear from a motherboard or doing it through a Supermicro AOC-SASLP-MV8 wont make a difference?

 

It depends. The AOC-SASLP-MV8 has a 600MB/s cap, so if you plan to do a parity check with more than 5 disks connected to it you could have a bottleneck.

 

I just got 3 6TB drives that need clearing so think I will move all of my now smaller drives, except for the parity, over to the SASLP and preclear through the MB.

 

Thanks

Share this post


Link to post

I'm currently running unRaid 6.1.7 with the latest preclear plugin version. I have the following problem:

 

The preclear.disc.log is constantly growing. At some point it's so large that \var\log is full.

 

This seems like a bug? Is there any way to change preclear configuration so that the log doesn't exceed a certain file size?

Share this post


Link to post
3 hours ago, madshi said:

I'm currently running unRaid 6.1.7 with the latest preclear plugin version. I have the following problem:

 

The preclear.disc.log is constantly growing. At some point it's so large that \var\log is full.

 

This seems like a bug? Is there any way to change preclear configuration so that the log doesn't exceed a certain file size?

This is not the thread for the plugin.

Share this post


Link to post

Quick question, is there an option to auto-confirm / auto answer the "Yes" prompt in JoeL's 1.15 preclear script, or will it need to be edited to comment out that section?

 

 

For reference I'm trying to string together the following 4pass drive preparation and would like to launch the script in an unattended manor...

  • preclear
  • badblocks (destructive) 0xaa
  • badblocks (destructive) 0x55
  • preclear -W

Manual confirmation of the 1st pass preclear is sensible, however I'd like the final pass of preclear to kick off automatically rather than waiting for user input.  Note my *nix skills are non-existent and so far my googling has only found results for auto-answering a command, not a script.

Share this post


Link to post

Started a pre-clear cycle on my 4TB drive.  Round one is at 24hr+.  I guess I didn't do my math correctly on the time.  I have it set to 3 pre-clear cycles. 

 

Is there a way to stop it before it start another cycle?

 

Edited by sgt_spike

Share this post


Link to post
6 minutes ago, sgt_spike said:

Started a pre-clear cycle on my 4TB drive.  Round one is at 24hr+.  I guess I didn't do my math correctly on the time.  I have it set to 3 pre-clear cycles. 

 

Is there a way to stop it before it start another cycle?

 

 

If you are really using this utility and not the plugin then I think you can just close the screen session or ssh connection you are using.

 

If you're using the plugin then probably you should ask in that support thread. If Im remembering correctly there will be a red x displayed next to the drive for this.

Share this post


Link to post

Can anybody on here comment as to whether or not the preclear add-on (I.e. the script, NOT plugin) is still working OK after the jump to unRAID 6.4.0-Stable? Or if any new updates that are required for it have come to light??

Edited by magic144

Share this post


Link to post
4 hours ago, magic144 said:

Can anybody on here comment as to whether or not the preclear add-on (I.e. the script, NOT plugin) is still working OK after the jump to unRAID 6.4.0-Stable? Or if any new updates that are required for it have come to light??

The script still works as long as it's the v6.2+ patched version.

Share this post


Link to post

Yup. I've been using it successfully on 6.3.5.

 

It's a shame it's not officially maintained or incorporated into unRAID itself. Still I've always had issues with the plugin (and it sounds like they might even be exacerbated by 6.4.0) but the script just keeps on working (when keeping up with the patches!)

 

Thanks for the feedback. Good to be sure about these things before taking the plunge :)

Edited by magic144

Share this post


Link to post
On 4/15/2016 at 8:51 PM, Joe L. said:

I only (very) recently put 6.2 beta on my server.

 

I did not have any issue pre-clearing the second parity disk I have just added to my array.

 

The fix will need to wait until I add/replace one of the existing disks with a larger one.

(Otherwise, I have no way to test the process. )

 

Whatever the fix might be, it must be backwards compatible with the older releases of unRAID.

 

In the interim, you can type this command to "patch" the preclear_disk.sh command

First change directory to the directory holding the preclear_disk.sh command.  For most, it will be

cd /boot

then type (or copy from here and paste) the following:

sed -i -e "s/print \$9 /print \$8 /" -e "s/sfdisk -R /blockdev --rereadpt /" preclear_disk.sh

 

Your preclear disk script will be edited and should work with the two changes you mentioned.  (actually, each occurs in two places, so there are a total of 4 lines changed)

 

Joe L.

 

On 10/26/2016 at 10:06 PM, gfjardim said:

 

There's one more fix to be made. Because of dd output change, pre-read and post-read status are showing time instead of speed. I propose a more elaborate command:

 

 

sed -i -e "s/print \$9 /print \$8 /" -e "s/sfdisk -R /blockdev --rereadpt /" -e "s/  sed -n 3p [^\}]*}'/awk -F',' 'END{print \$NF}'/g" preclear_disk.sh

 

 

Which patch is everyone using?

Share this post


Link to post

I used @Joe L. command to fix it to run  post 6.2.  That fixed the problem with the script not working at all.  As far as the issue the second part of the second one 'fixes', I don't really care which parameter the report contains, time or speed, either is equivalent of the other.   (As a note of disclosure, I have not run the script for more than a year.  The last new disks that I precleared, I used the plugin.  But that was before the release of 6.4.X.)  

Share this post


Link to post

... and back to the script from the plugin.   thanks for leaving this up joe.  (though i miss the plugin greatly already!)

Share this post


Link to post

So Disk2 which has been in use for a few years now was kicked offline in the array... but the disk has no negative smart report items.

 

I moved the Disk2 to another slot on the backplane, set the array to use NONE in disk2, used MC to move the emulated Disk2 data to disk3 and disk4, then began a preclear on Disk2 (which was no longer physically assigned to the array) 

 

Running a preclear on this disk has given me the confidence that the physical port of my backplane has faulted and not the physical disk.

 

To the 6.4.1 naysayeers... Preclear is a critical component of Unraid.  It is great to have as a plugin, but will use screen and terminal if the Joe L script is all we are allotted. 

 

I personally do not feel comfortable just tossing a bare drive into the array without a few cycles of passes... and have had a goodly portion of drives fail a preclear before ever touching my array.

 

In addition, any data drive that is taken OUT of commission has a preclear run on it to destroy data prior to recycling or sale. --- so long as it is physically able to run. 

Edited by landS

Share this post


Link to post

Moving files from the emulated drive to other drives in the array isn't really the best approach, since that involves a lot of writing to an unprotected array. Rebuilding to another disk, then trying to figure out if the original has a problem is the preferred method. Assuming the disk has good SMART, the next best method would be to just rebuild to the same drive. I don't see any reason to copy or worse yet move files from the emulated disk to other disks in the array. It would make more sense to just copy them somewhere off the array if you're that worried about losing them.

 

Moving files from the emulated disk to other disks in the array requires reading all disks so the file can be copied to the destination and parity updated, and reading all disks so the file can be deleted from the (emulated) source and parity updated.

 

I don't know why anyone would want to move files around on an unprotected array in order to save them, when just rebuilding the disk will save them and get you back to a protected state much more quickly.

Share this post


Link to post

 Thanks Trurl.  This machine is in our work environment and the replacement disk has to go through sourcing.  The other disks had enough capacity to hold the data & disk2 actually had little data on it, so I thought moving the disk2 data to the remaining disks, then running a new config would be better in the interim as then the entire array would be protected from another disk failure.  Leaving the data emulated while we await sourcing to deliver the replacement would not allow the data to be protected from any further failures.

 

Given that the disk was passing all smart tests, and that the data was fully accessible, and no physical changes had occurred I was very interested into WHAT caused the disk to get knocked out of the array.   Given that the disk has passed preclear on another backplane slot I am fairly confident that the backplane slot is to blame.  Tomorrow I am going to run a preclear on the suspect backplane slot. 

Edited by landS

Share this post


Link to post

Looks like unRAID 6.4.1 (not sure about 6.4.0) no longer has strings as a command, but does have strings-BSD which appears to output the same thing.  There is also a call to /root/mdcmd which now resides in /usr/local/sbin/mdcmd  - The following command will resolve both these issues:

 

sed -i -e  "s|strings|strings-BSD|" -e "s|\/root\/mdcmd|\/usr\/local\/sbin\/mdcmd|" preclear_disk.sh

 

  • Upvote 1

Share this post


Link to post
8 minutes ago, sureguy said:

Looks like unRAID 6.4.1 (not sure about 6.4.0) no longer has strings as a command, but does have strings-BSD which appears to output the same thing.  There is also a call to /root/mdcmd which now resides in /usr/local/sbin/mdcmd  - The following command will resolve both these issues:

 


sed -i -e  "s|strings|strings-BSD|" -e "s|\/root\/mdcmd|\/usr\/local\/sbin\/mdcmd|" preclear_disk.sh

 

 

Where (or at what point) in the preclear operation is this causing a problem?  What are the symptoms?   I seem to recall that someone had actually used Joe L's script successfully with 6..4.1.

Share this post


Link to post
56 minutes ago, Frank1940 said:

 

Where (or at what point) in the preclear operation is this causing a problem?  What are the symptoms?   I seem to recall that someone had actually used Joe L's script successfully with 6..4.1.

 

If you run

 

preclear_disk.sh -l

The following error is generated as part of the output:  ./preclear_disk.sh: line 236: strings: command not found

 

In regards to the /root/mdcmd issue, I'm not sure when that's called, I only noticed it because the same line also calls strings.  Apparently the script still works with these issues - it will preclear the disk.  The first instance of the strings command is reading super.dat.  Off the top of my head that could be used to discern what drives shouldn't be precleared as they're part of the array.

 

 

Edited by sureguy

Share this post


Link to post

i ran Joe L.s script on the work machine with Joe L.'s patch - on 6.4.1

It was successful, the 3 preclear reports looked good, the screen session looked fine

 

I can confirm SureGuys finding - however immediately after the line 236 message follows a list of those disks that can be precleared.

 

root@LDB1:~# cd /boot
root@LDB1:/boot# preclear_disk.sh -l
====================================1.15
 Disks not assigned to the unRAID array
  (potential candidates for clearing)
========================================
./preclear_disk.sh: line 236: strings: command not found
     /dev/sde = ata-ST2000NM0033-9ZM175_Z1X3JTYJ
root@LDB1:/boot#

 

 

 

root@Tower:~# cd /boot
root@Tower:/boot# preclear_disk.sh -l
====================================1.15
 Disks not assigned to the unRAID array
  (potential candidates for clearing)
========================================
./preclear_disk.sh: line 236: strings: command not found
     /dev/sdi = ata-ST2000VN004-2E4164_Z523T4NK
root@Tower:/boot#

 

 

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now