Re: preclear_disk.sh - a new utility to burn-in and pre-clear disks for quick add


Recommended Posts

The preclear_disk.sh script is failing on v6.2 betas because the sfdisk -R is no longer supported.  Preclear_disk.sh is reporting the disk as busy and will not preclear it.  It looks like 'blockdev --rereadpt' is the replacement according to the sfdisk man-pages here http://man7.org/linux/man-pages/man8/sfdisk.8.html.

 

"      Since version 2.26 sfdisk no longer provides the -R or --re-read

      option to force the kernel to reread the partition table.  Use

      blockdev --rereadpt instead."

 

EDIT: There is also an issue with reads failing.  I changed the following:

 

read_entire_disk( ) {
  # Get the disk geometry (cylinders, heads, sectors)
  fgeometry=`fdisk -l $1 2>/dev/null`
  units=`echo "$fgeometry" | grep Units | awk '{ print $9 }'`

 

to

 

read_entire_disk( ) {
  # Get the disk geometry (cylinders, heads, sectors)
  fgeometry=`fdisk -l $1 2>/dev/null`
  units=`echo "$fgeometry" | grep Units | awk '{ print $8 }'`

 

and the reads will work.

 

Joe L. - Can we get an official fix and an update from you?

I only (very) recently put 6.2 beta on my server.

 

I did not have any issue pre-clearing the second parity disk I have just added to my array.

 

The fix will need to wait until I add/replace one of the existing disks with a larger one.

(Otherwise, I have no way to test the process. )

 

Whatever the fix might be, it must be backwards compatible with the older releases of unRAID.

 

In the interim, you can type this command to "patch" the preclear_disk.sh command

First change directory to the directory holding the preclear_disk.sh command.  For most, it will be

cd /boot

then type (or copy from here and paste) the following:

sed -i -e "s/print \$9 /print \$8 /" -e "s/sfdisk -R /blockdev --rereadpt /" preclear_disk.sh

 

Your preclear disk script will be edited and should work with the two changes you mentioned.  (actually, each occurs in two places, so there are a total of 4 lines changed)

 

Joe L.

 

There's one more fix to be made. Because of dd output change, pre-read and post-read status are showing time instead of speed. I propose a more elaborate command:

 

 

sed -i -e "s/print \$9 /print \$8 /" -e "s/sfdisk -R /blockdev --rereadpt /" -e "s/  sed -n 3p [^\}]*}'/awk -F',' 'END{print \$NF}'/g" preclear_disk.sh

 

 

Link to comment

Is it quicker to preclear from a motherboard or doing it through a Supermicro AOC-SASLP-MV8 wont make a difference?

 

It depends. The AOC-SASLP-MV8 has a 600MB/s cap, so if you plan to do a parity check with more than 5 disks connected to it you could have a bottleneck.

 

I just got 3 6TB drives that need clearing so think I will move all of my now smaller drives, except for the parity, over to the SASLP and preclear through the MB.

 

Thanks

Link to comment
  • 8 months later...

I'm currently running unRaid 6.1.7 with the latest preclear plugin version. I have the following problem:

 

The preclear.disc.log is constantly growing. At some point it's so large that \var\log is full.

 

This seems like a bug? Is there any way to change preclear configuration so that the log doesn't exceed a certain file size?

Link to comment
3 hours ago, madshi said:

I'm currently running unRaid 6.1.7 with the latest preclear plugin version. I have the following problem:

 

The preclear.disc.log is constantly growing. At some point it's so large that \var\log is full.

 

This seems like a bug? Is there any way to change preclear configuration so that the log doesn't exceed a certain file size?

This is not the thread for the plugin.

Link to comment
  • 3 months later...

Quick question, is there an option to auto-confirm / auto answer the "Yes" prompt in JoeL's 1.15 preclear script, or will it need to be edited to comment out that section?

 

 

For reference I'm trying to string together the following 4pass drive preparation and would like to launch the script in an unattended manor...

  • preclear
  • badblocks (destructive) 0xaa
  • badblocks (destructive) 0x55
  • preclear -W

Manual confirmation of the 1st pass preclear is sensible, however I'd like the final pass of preclear to kick off automatically rather than waiting for user input.  Note my *nix skills are non-existent and so far my googling has only found results for auto-answering a command, not a script.

Link to comment
  • 5 weeks later...
  • 3 weeks later...
6 minutes ago, sgt_spike said:

Started a pre-clear cycle on my 4TB drive.  Round one is at 24hr+.  I guess I didn't do my math correctly on the time.  I have it set to 3 pre-clear cycles. 

 

Is there a way to stop it before it start another cycle?

 

 

If you are really using this utility and not the plugin then I think you can just close the screen session or ssh connection you are using.

 

If you're using the plugin then probably you should ask in that support thread. If Im remembering correctly there will be a red x displayed next to the drive for this.

Link to comment
  • 1 month later...
4 hours ago, magic144 said:

Can anybody on here comment as to whether or not the preclear add-on (I.e. the script, NOT plugin) is still working OK after the jump to unRAID 6.4.0-Stable? Or if any new updates that are required for it have come to light??

The script still works as long as it's the v6.2+ patched version.

Link to comment

Yup. I've been using it successfully on 6.3.5.

 

It's a shame it's not officially maintained or incorporated into unRAID itself. Still I've always had issues with the plugin (and it sounds like they might even be exacerbated by 6.4.0) but the script just keeps on working (when keeping up with the patches!)

 

Thanks for the feedback. Good to be sure about these things before taking the plunge :)

Edited by magic144
Link to comment
On 4/15/2016 at 8:51 PM, Joe L. said:

I only (very) recently put 6.2 beta on my server.

 

I did not have any issue pre-clearing the second parity disk I have just added to my array.

 

The fix will need to wait until I add/replace one of the existing disks with a larger one.

(Otherwise, I have no way to test the process. )

 

Whatever the fix might be, it must be backwards compatible with the older releases of unRAID.

 

In the interim, you can type this command to "patch" the preclear_disk.sh command

First change directory to the directory holding the preclear_disk.sh command.  For most, it will be

cd /boot

then type (or copy from here and paste) the following:

sed -i -e "s/print \$9 /print \$8 /" -e "s/sfdisk -R /blockdev --rereadpt /" preclear_disk.sh

 

Your preclear disk script will be edited and should work with the two changes you mentioned.  (actually, each occurs in two places, so there are a total of 4 lines changed)

 

Joe L.

 

On 10/26/2016 at 10:06 PM, gfjardim said:

 

There's one more fix to be made. Because of dd output change, pre-read and post-read status are showing time instead of speed. I propose a more elaborate command:

 

 

sed -i -e "s/print \$9 /print \$8 /" -e "s/sfdisk -R /blockdev --rereadpt /" -e "s/  sed -n 3p [^\}]*}'/awk -F',' 'END{print \$NF}'/g" preclear_disk.sh

 

 

Which patch is everyone using?

Link to comment

I used @Joe L. command to fix it to run  post 6.2.  That fixed the problem with the script not working at all.  As far as the issue the second part of the second one 'fixes', I don't really care which parameter the report contains, time or speed, either is equivalent of the other.   (As a note of disclosure, I have not run the script for more than a year.  The last new disks that I precleared, I used the plugin.  But that was before the release of 6.4.X.)  

Link to comment
  • 3 weeks later...

So Disk2 which has been in use for a few years now was kicked offline in the array... but the disk has no negative smart report items.

 

I moved the Disk2 to another slot on the backplane, set the array to use NONE in disk2, used MC to move the emulated Disk2 data to disk3 and disk4, then began a preclear on Disk2 (which was no longer physically assigned to the array) 

 

Running a preclear on this disk has given me the confidence that the physical port of my backplane has faulted and not the physical disk.

 

To the 6.4.1 naysayeers... Preclear is a critical component of Unraid.  It is great to have as a plugin, but will use screen and terminal if the Joe L script is all we are allotted. 

 

I personally do not feel comfortable just tossing a bare drive into the array without a few cycles of passes... and have had a goodly portion of drives fail a preclear before ever touching my array.

 

In addition, any data drive that is taken OUT of commission has a preclear run on it to destroy data prior to recycling or sale. --- so long as it is physically able to run. 

Edited by landS
Link to comment

Moving files from the emulated drive to other drives in the array isn't really the best approach, since that involves a lot of writing to an unprotected array. Rebuilding to another disk, then trying to figure out if the original has a problem is the preferred method. Assuming the disk has good SMART, the next best method would be to just rebuild to the same drive. I don't see any reason to copy or worse yet move files from the emulated disk to other disks in the array. It would make more sense to just copy them somewhere off the array if you're that worried about losing them.

 

Moving files from the emulated disk to other disks in the array requires reading all disks so the file can be copied to the destination and parity updated, and reading all disks so the file can be deleted from the (emulated) source and parity updated.

 

I don't know why anyone would want to move files around on an unprotected array in order to save them, when just rebuilding the disk will save them and get you back to a protected state much more quickly.

Link to comment

 Thanks Trurl.  This machine is in our work environment and the replacement disk has to go through sourcing.  The other disks had enough capacity to hold the data & disk2 actually had little data on it, so I thought moving the disk2 data to the remaining disks, then running a new config would be better in the interim as then the entire array would be protected from another disk failure.  Leaving the data emulated while we await sourcing to deliver the replacement would not allow the data to be protected from any further failures.

 

Given that the disk was passing all smart tests, and that the data was fully accessible, and no physical changes had occurred I was very interested into WHAT caused the disk to get knocked out of the array.   Given that the disk has passed preclear on another backplane slot I am fairly confident that the backplane slot is to blame.  Tomorrow I am going to run a preclear on the suspect backplane slot. 

Edited by landS
Link to comment

Looks like unRAID 6.4.1 (not sure about 6.4.0) no longer has strings as a command, but does have strings-BSD which appears to output the same thing.  There is also a call to /root/mdcmd which now resides in /usr/local/sbin/mdcmd  - The following command will resolve both these issues:

 

sed -i -e  "s|strings|strings-BSD|" -e "s|\/root\/mdcmd|\/usr\/local\/sbin\/mdcmd|" preclear_disk.sh

 

Link to comment
8 minutes ago, sureguy said:

Looks like unRAID 6.4.1 (not sure about 6.4.0) no longer has strings as a command, but does have strings-BSD which appears to output the same thing.  There is also a call to /root/mdcmd which now resides in /usr/local/sbin/mdcmd  - The following command will resolve both these issues:

 


sed -i -e  "s|strings|strings-BSD|" -e "s|\/root\/mdcmd|\/usr\/local\/sbin\/mdcmd|" preclear_disk.sh

 

 

Where (or at what point) in the preclear operation is this causing a problem?  What are the symptoms?   I seem to recall that someone had actually used Joe L's script successfully with 6..4.1.

Link to comment
56 minutes ago, Frank1940 said:

 

Where (or at what point) in the preclear operation is this causing a problem?  What are the symptoms?   I seem to recall that someone had actually used Joe L's script successfully with 6..4.1.

 

If you run

 

preclear_disk.sh -l

The following error is generated as part of the output:  ./preclear_disk.sh: line 236: strings: command not found

 

In regards to the /root/mdcmd issue, I'm not sure when that's called, I only noticed it because the same line also calls strings.  Apparently the script still works with these issues - it will preclear the disk.  The first instance of the strings command is reading super.dat.  Off the top of my head that could be used to discern what drives shouldn't be precleared as they're part of the array.

 

 

Edited by sureguy
Link to comment

i ran Joe L.s script on the work machine with Joe L.'s patch - on 6.4.1

It was successful, the 3 preclear reports looked good, the screen session looked fine

 

I can confirm SureGuys finding - however immediately after the line 236 message follows a list of those disks that can be precleared.

 

root@LDB1:~# cd /boot
root@LDB1:/boot# preclear_disk.sh -l
====================================1.15
 Disks not assigned to the unRAID array
  (potential candidates for clearing)
========================================
./preclear_disk.sh: line 236: strings: command not found
     /dev/sde = ata-ST2000NM0033-9ZM175_Z1X3JTYJ
root@LDB1:/boot#

 

 

 

root@Tower:~# cd /boot
root@Tower:/boot# preclear_disk.sh -l
====================================1.15
 Disks not assigned to the unRAID array
  (potential candidates for clearing)
========================================
./preclear_disk.sh: line 236: strings: command not found
     /dev/sdi = ata-ST2000VN004-2E4164_Z523T4NK
root@Tower:/boot#

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.