Re: preclear_disk.sh - a new utility to burn-in and pre-clear disks for quick add


Recommended Posts

As promised, there is now a updated version of the preclear_disk.sh script attached to the first post in this thread.

 

It now is version 0.9.6:

 

5th edit: August 31, 2009.  New version 0.9.6  ( 0.9.4 and 0.9.5 were internal versions as jbuszkie and I tested)

# Version .9.4 - Enable SMART monitoring, just in case it is disabled on a drive.

# Version .9.5 - Added disk temperature display and disk read speed display.

# Version .9.6 - Enhanced the mail reporting to include some statistics (time, MB/s temp .. ect)

#              - Fixed a bug with using zero.txt and concurrent tests. Each test will use it's own file.

#              - Changed read block size to be bigger than 1,000,000 if smaller, to improve read speed

 

There is nothing wrong with the previous version, but the newer one provides more statistics as it runs,

especially if have configured mail on your server and you use the -m mailaddress@mailhost option to mail progress reports to yourself. 

 

This time, a lot of the improvements are credited to jbuszkie.  He took my original script and made it even more informative.

I just prettied things up once he made his improvements and sent them onward to me for review.

 

Oh yes, the "diff" of the smart reports is now logged to the syslog...  Makes it easier to see the changes to the results all in one spot.

 

As always, if you spot anything we missed, or have a suggestion for an improvement, let me know.

 

Joe L.

Link to comment

This time I am attaching my syslog, in hope that it could help resolve this problem.

 

One of your disks is running in ATA/33 mode

 

Aug 28 12:58:35 Tower kernel: Probing IDE interface ide0...

Aug 28 12:58:35 Tower kernel: hda: WDC WD3200BEVE-00A0HT0, ATA DISK drive

Aug 28 12:58:35 Tower kernel: hda: host max PIO4 wanted PIO255(auto-tune) selected PIO4

Aug 28 12:58:35 Tower kernel: hda: host side 80-wire cable detection failed, limiting max speed to UDMA33

Aug 28 12:58:35 Tower kernel: hda: UDMA/33 mode selected

Aug 28 12:58:35 Tower kernel: ide0 at 0x170-0x177,0x376 on irq 15

Aug 28 12:58:35 Tower kernel: hda: max request size: 512KiB

Aug 28 12:58:35 Tower kernel: hda: 625142448 sectors (320072 MB) w/8192KiB Cache, CHS=38913/255/63

Aug 28 12:58:35 Tower kernel: hda: cache flushes supported

 

Now this is one thing that drives me crazy:

 

The system is trying to detect a 80-wire cable, and I can find no way to override that!

 

See, the disk in question is a brand new 2.5" mini-IDE disk from Western Digital. These disks don't have 40-pin or 80-pin cables. Their cables are 44-pin! I expected that Linux should be smart enough by now to know how to work with 2.5" PATA disks.

 

As documented in:

http://www.kernel.org/doc/Documentation/kernel-parameters.txt

I tried adding libata.force=80c in syslinux.cfg, but the system replies with "unknown command: libata.force=80c, ignoring."

 

Any ideas how I can make the system ignore the cable test, and just ask the disk what itis capable of?

 

Yours,

Purko

 

 

 

 

Link to comment
  • 4 weeks later...

 

Hi Joe!

 

A quick question:

 

On the second read, doesn't it make sense to verify that you are actually reading zeros from the disk instead of dumping it straight to /dev/null ?

 

If it makes sense, then maybe it can be a configurable command-line option, like -v for 'verify' or something?

 

Yours,

Purko

 

 

Link to comment

 

Hi Joe!

 

A quick question:

 

On the second read, doesn't it make sense to verify that you are actually reading zeros from the disk instead of dumping it straight to /dev/null ?

 

If it makes sense, then maybe it can be a configurable command-line option, like -v for 'verify' or something?

 

Yours,

Purko

 

 

 

Funny you should mention that, as this past week I made a improvement that does exactly what you suggested.   I did not post it, as I had just finished posting a new version a day or so before and wanted to test it a bit myself...

 

The new version does the verification of zeros as you described as its default behavior in the post-read phase.  It will take about 10-15% more time, as it has to evaluate what it is reading.  It only does it in the post-read phase, as it is there where the zeros are expected.

 

You can elect to NOT do the post-read verification of all zeros with a new -N option...  (if you are in a hurry)

 

I'll attach the new version to the first post in this thread. 

Edit: Version .9.7 is now attached to the first post in this thread... Enjoy.

 

Joe L.

Link to comment

Hi Joe,

 

Just a quick query. If I use the pre-clear script to exercise a few old HDDs, can I still use the disks in a Windows-based system after that? As in, I will be able to just partition as per normal using Windows Disk Management or some other third party tool right?

 

Sure, Windows will probably recognize the partition it creates.  I see no reason why it would not... In fact, it is specifically partitioned to be compatible with the BIOS on most MB. 

 

The partition the pre-clear script creates starts on the second cylinder (usually at sector 63, depending on the "reported" disk geometry) and goes to the end of the disk.  The practice of leaving the first full cylinder un-used except for the MBr in the first 512 bytes is a holdover from DOS days. 

 

You need to create a file-system in the partition preclear_disk.sh creates, or delete the one the pre_clear script creates and re-create the partition if you want to make it smaller, or to create multiple partitions and use it for windows.

 

Joe L.

Link to comment

 

Hi Joe!

 

How about a new command line option to specify the block sizes dd will use?

 

The reason I'am asking this is that the preclear script is still capable of totally rendering a weak machine unusable: processes get stuck at 'disk wait' state for looooong time, samba transfers time-out and abort with errors, etc., etc..

 

I think that experimenting with smaller dd block sizes at a time will solve all these problems, albeit at the expense of taking much longer time to complete the preclear script.

 

I contemplated making some experiments myself, but I was stopped short when I saw how many actual numbers (rather than variables) are scattered all over the script. It makes any experimentation by anyone other than yourself quite risky.

 

So it occured to me that an option to specify the bs on the command line would be really nice. (Or at least a nicely defined variable with the default bs value in the first few lines of the script)

 

Seriously, I would rather have the preclear script run three times longer than now, rather than having it put so much strain on a weak box.

 

Yours,

Purko

 

 

Link to comment

 

Hi Joe!

 

How about a new command line option to specify the block sizes dd will use?

<snip>

Seriously, I would rather have the preclear script run three times longer than now, rather than having it put so much strain on a weak box.

 

Yours,

Purko

Good idea.  I'll put it in the next version.
Link to comment

How about a new command line option to specify the block sizes dd will use?

 

Better yet, just a new command line option like -s (for "slow mode")

 

When started with this option, the script will use a bs number much smaller than the default.

 

Also, the number of blocks it does at a time must be greatly reduced, not just the block size.

 

Link to comment

 

Hi Joe!

 

How about a new command line option to specify the block sizes dd will use?

 

The reason I'am asking this is that the preclear script is still capable of totally rendering a weak machine unusable: processes get stuck at 'disk wait' state for looooong time, samba transfers time-out and abort with errors, etc., etc..

 

I think that experimenting with smaller dd block sizes at a time will solve all these problems, albeit at the expense of taking much longer time to complete the preclear script.

 

I contemplated making some experiments myself, but I was stopped short when I saw how many actual numbers (rather than variables) are scattered all over the script. It makes any experimentation by anyone other than yourself quite risky.

 

So it occured to me that an option to specify the bs on the command line would be really nice. (Or at least a nicely defined variable with the default bs value in the first few lines of the script)

 

Seriously, I would rather have the preclear script run three times longer than now, rather than having it put so much strain on a weak box.

 

Yours,

Purko

Version .9.8 of preclear_disk.sh is now attached to the first post in this thread.   It now has three new options that can be specified on the command line to accomplish exactly what you requested.  You can now specify the block size for reading, for writing, and the number of blocks to read at a time.

 

The sizes are all in bytes.  So if you want 1 thousand byte blocks, and to read 100 blocks at a time, specify

preclear_disk.sh -r 1000 -w 1000 -b 100 /dev/sdX

if you want 32k byte blocks, use:

preclear_disk.sh -r 32k -w 32k -b 100 /dev/sdX

or

preclear_disk.sh -r 512 -w 32768 -b 1k /dev/sdX

 

For the most part, using smaller block sizes will lengthen the time it takes to clear a drive.  It just will use the sizes you supply and continue until done.  When specifying the sizes, you can use 1k for 1000, or 1M for 1000000.  The "K" or "M"suffix  will be expanded by multiplying the supplied value by 1000 or 1000000.  I did not put an upper limit on size, so if you specify a size bigger than your available ram, you could run out and crash your server.  You've been warned... Don't use -r 1000000000 -b 10000000000.  You will run out of ram... :'(

 

If you do not use the new -b, -w, or -r options the preclear_disk will use the default values used in all the prior versions of the script. 

        -w size  = write block size in bytes.  If not specified, default is 2048k.

 

       -r size  = read block size in bytes.  If not specified, default is one cylinder at a time ( heads * sectors * 512))

 

       -b count = number of blocks to read at a time.  If not specified, default is 200 blocks at a time.

 

 

 As requested, these options are for running the script on a server with a limited amount of free ram, and limited processor power, where taking longer is acceptable.   

 

Joe L.

Link to comment

I am currently running Preclear...but an error message keeps on flashing on the screen below the elapsed time.

 

"smartctl: error while loading shared libraries: libstc++.so.6: cannot open shared object file: no such file or directory"

 

Can you please tell me what I have done wrong.

Link to comment

I am currently running Preclear...but an error message keeps on flashing on the screen below the elapsed time.

 

"smartctl: error while loading shared libraries: libstc++.so.6: cannot open shared object file: no such file or directory"

 

Can you please tell me what I have done wrong.

Yes, re-read the first post in this thread.  It describes how the support library needed for SMART reports is missing on some unRAID releases and points to instructions on where to download it and how to install it.  That can all be done even as the pre-clear continues to run.

 

Or look here: http://lime-technology.com/forum/index.php?topic=4407.msg39409#msg39409

 

Joe L.

 

Link to comment

When I run the command: ls -l /dev/disk/by-id to locate the drive names ( so I can pre-clear the right one ), I notice some follow one naming pattern and the others a different one (ie. hdx and sdx)

 

I thought I distinguished SATA from IDE as I read this elsewhere:

The actual drive would depend on your hardware, since "hdX" drives are IDE, and "sdX" drives are SATA, I'm pretty sure it will not be /dev/hdk.  Grin   for an SATA drive, it will be /dev/sdX  (where X will depend on your hardware)

 

Here is my output.  Although it has 21 entries, I actually have 6 physical drives + Flash Drive ( 1.5TB X 2 Seagates, 1.5TB WD , 1TB Seagate, 750GB Seagate and 20GB Seagate )

 

 

lrwxrwxrwx 1 root root  9 Oct 14 18:12 ata-ST31000528AS_9VP0H3TD -> ../../sdd

lrwxrwxrwx 1 root root 10 Oct 14 18:12 ata-ST31000528AS_9VP0H3TD-part1 -> ../../sdd1

lrwxrwxrwx 1 root root  9 Oct 14 18:12 ata-ST31500341AS_9VS2L1L4 -> ../../hdc

lrwxrwxrwx 1 root root  9 Oct 14 18:12 ata-ST31500341AS_9VS2LXAF -> ../../sda

lrwxrwxrwx 1 root root 10 Oct 14 18:12 ata-ST31500341AS_9VS2LXAF-part1 -> ../../sda1

lrwxrwxrwx 1 root root  9 Oct 14 18:12 ata-ST320011A_3HT0GTF1 -> ../../hda

lrwxrwxrwx 1 root root 10 Oct 14 18:12 ata-ST320011A_3HT0GTF1-part1 -> ../../hda1

lrwxrwxrwx 1 root root  9 Oct 14 18:12 ata-ST3750330AS_5QK00GT5 -> ../../sdc

lrwxrwxrwx 1 root root 10 Oct 14 18:12 ata-ST3750330AS_5QK00GT5-part1 -> ../../sdc1

lrwxrwxrwx 1 root root  9 Oct 14 18:12 ata-WDC_WD15EADS-00P8B0_WD-WMAVU0072618 -> ../../sdb

lrwxrwxrwx 1 root root 10 Oct 14 18:12 ata-WDC_WD15EADS-00P8B0_WD-WMAVU0072618-part1 -> ../../sdb1

lrwxrwxrwx 1 root root  9 Oct 14 18:12 scsi-SATA_ST31000528AS_9VP0H3TD -> ../../sdd

lrwxrwxrwx 1 root root 10 Oct 14 18:12 scsi-SATA_ST31000528AS_9VP0H3TD-part1 -> ../../sdd1

lrwxrwxrwx 1 root root  9 Oct 14 18:12 scsi-SATA_ST31500341AS_9VS2LXAF -> ../../sda

lrwxrwxrwx 1 root root 10 Oct 14 18:12 scsi-SATA_ST31500341AS_9VS2LXAF-part1 -> ../../sda1

lrwxrwxrwx 1 root root  9 Oct 14 18:12 scsi-SATA_ST3750330AS_5QK00GT5 -> ../../sdc

lrwxrwxrwx 1 root root 10 Oct 14 18:12 scsi-SATA_ST3750330AS_5QK00GT5-part1 -> ../../sdc1

lrwxrwxrwx 1 root root  9 Oct 14 18:12 scsi-SATA_WDC_WD15EADS-00_WD-WMAVU0072618 -> ../../sdb

lrwxrwxrwx 1 root root 10 Oct 14 18:12 scsi-SATA_WDC_WD15EADS-00_WD-WMAVU0072618-part1 -> ../../sdb1

lrwxrwxrwx 1 root root  9 Oct 14 18:12 usb-Lexar_JD_FireFly_AA04011000035322-0:0 -> ../../sde

lrwxrwxrwx 1 root root 10 Oct 14 18:12 usb-Lexar_JD_FireFly_AA04011000035322-0:0-part1 -> ../../sde1

 

I've just installed the one recognized as "hdc".   I still have to pre-clear it and it will become my Parity drive.

 

The Question:  Why are all my other SATA drives labeled "sda, sdb, sdc, sdd"  and this new one not called "sdf" ?  You can see that "sda" is the exact same make of drive as "hdc"

(I've noticed the Flash drive is sde/sde1)

 

I can say for sure, it is SATA as I had to search for an hour to find a darn SATA cable for it  :)

 

Link to comment

When I run the command: ls -l /dev/disk/by-id to locate the drive names ( so I can pre-clear the right one ), I notice some follow one naming pattern and the others a different one (ie. hdx and sdx)

 

I thought I distinguished SATA from IDE as I read this elsewhere:

The actual drive would depend on your hardware, since "hdX" drives are IDE, and "sdX" drives are SATA, I'm pretty sure it will not be /dev/hdk.  Grin   for an SATA drive, it will be /dev/sdX  (where X will depend on your hardware)

I thought the naming would be consistent too.

Here is my output.  Although it has 21 entries, I actually have 6 physical drives + Flash Drive ( 1.5TB X 2 Seagates, 1.5TB WD , 1TB Seagate, 750GB Seagate and 20GB Seagate )

<snip>

I've just installed the one recognized as "hdc".   I still have to pre-clear it and it will become my Parity drive.

 

The Question:  Why are all my other SATA drives labeled "sda, sdb, sdc, sdd"  and this new one not called "sdf" ?  You can see that "sda" is the exact same make of drive as "hdc"

(I've noticed the Flash drive is sde/sde1)

 

I can say for sure, it is SATA as I had to search for an hour to find a darn SATA cable for it  :)

The make/model of drive is not what determines hdX vs sdX, but how the disk controller chipset on the MB or controller card identifies itself to Linux.

 

There might be a BIOS option to get a disk to identify itself as one or the other, and it really does not matter unless the disk is accessed in PIO mode (really slow) instead of a DMA mode of some kind.  Do not be surprised if the actual assigned devices change as you add or remove drives, or update to different versions of Linux, as the devices might be scanned in a slightly different order with a new Linux version.

 

Post a copy of your syslog so anything odd might be identified in your configuration.

 

Joe L.

Link to comment

The new drive is added on my motherboard directly.  It in the 5th of 6 ports.

 

Syslog is attached.

 

I'll hold off on starting the preclear until I hear back  :)

 

Thanks

The port is being found on the MB as follows:

Oct 14 18:12:22 Tower kernel: hdc: ST31500341AS, ATA DISK drive

Oct 14 18:12:22 Tower kernel: hdc: host max PIO4 wanted PIO255(auto-tune) selected PIO4

Oct 14 18:12:22 Tower kernel: hdc: UDMA/100 mode selected

 

Can you do a

hdparm -i /dev/hdc

 

so we can confirm the mode being used to access it.  It looks like it will be using UDMA/100.

 

Go ahead and start the pre-clear.  It too will give you an idea of the speed the disk can be accessed.

 

Joe L.

Link to comment

root@Tower:~# hdparm -i /dev/hdc

 

/dev/hdc:

 

Model=ST31500341AS, FwRev=CC1H, SerialNo=9VS2L1L4

Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% }

RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4

BuffType=unknown, BuffSize=0kB, MaxMultSect=16, MultSect=16

CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=18446744072344861488

IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}

PIO modes:  pio0 pio1 pio2 pio3 pio4

DMA modes:  mdma0 mdma1 mdma2

UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5 udma6

AdvancedPM=no WriteCache=enabled

Drive conforms to: unknown:  ATA/ATAPI-4,5,6,7

 

* signifies the current active mode

 

 

I started preclear to get an idea of speeds and everything looks normal ( i think )

 

unRAID server Pre-Clear disk /dev/hdc

=                      cycle 1 of 1

= Disk Pre-Read in progress: 0% complete

= ( 1,645,056,000  bytes of  1,500,301,910,016  read ) 116 MB/s

=

=

=

=

=

=

=

=

=

=

Disk Temperature: 30C, Elapsed Time:  0:00:15

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.