Unofficial Faster Preclear


Recommended Posts

 

Good to know...thanks guys.

 

So I set my WD Red 6tb to run 2 preclears. Is there a way to get it to safely quit after the first successful preclear?

I want to add my ssd to the array and then run the second pre-clear

 

You can abort the pre-clear with Ctrl-C

 

It's always "safe" to do ... it just may (depending on where in the cycle you stopped it) not have a valid pre-clear signature on the drive, so you'll either need to do it again or just let UnRAID clear the disk when  you add it.

 

As for adding your SSD to the array ... you are, I presume, aware that you'll not likely see any major performance gains with this.    Reason is simple:  (a)  For writes, the speed is still going to be limited by the speed of the parity drive; and (b) for reads, you're likely already limited by the speed of your network.  Reads will, of course, start much quicker, due to the near-instantaneous "seek" times;  but the network will quickly be the bottleneck.

 

Link to comment
As for adding your SSD to the array ... you are, I presume, aware that you'll not likely see any major performance gains with this.    Reason is simple:  (a)  For writes, the speed is still going to be limited by the speed of the parity drive; and (b) for reads, you're likely already limited by the speed of your network.  Reads will, of course, start much quicker, due to the near-instantaneous "seek" times;  but the network will quickly be the bottleneck

 

Are you saying the speeds are relation to the ssd being added to the array like another "storage" drive or that it applies to a cache drive also?

 

Sorry, I was not clear with what I said. I want to add the ssd as a cache only drive, probably just for dockers/vm's etc. I have never had a cache drive before and never had any issues with speed so I'll stick with writing to parity directly.

 

 

 

 

 

Link to comment

As for adding your SSD to the array ... you are, I presume, aware that you'll not likely see any major performance gains with this.    Reason is simple:  (a)  For writes, the speed is still going to be limited by the speed of the parity drive; and (b) for reads, you're likely already limited by the speed of your network.  Reads will, of course, start much quicker, due to the near-instantaneous "seek" times;  but the network will quickly be the bottleneck

 

Are you saying the speeds are relation to the ssd being added to the array like another "storage" drive or that it applies to a cache drive also?

 

Sorry, I was not clear with what I said. I want to add the ssd as a cache only drive, probably just for dockers/vm's etc. I have never had a cache drive before and never had any issues with speed so I'll stick with writing to parity directly.

 

I misunderstood your intent -- when you said you wanted to "add my SSD to the array" I assumed you meant add it to the array.    A cache drive is not part of the array ... it's "outside" of the storage array ... so my comments don't apply.  It will in fact provide some nice performance gains as a cache drive.

 

No impact on reads;  but a nice impact on writes, since they'll now be at the full speed of the cache drive, without being limited by parity.    And for any local usage within the UnRAID box (i.e. Dockers, VM's, etc.) you'll get the full benefit of the SSD.

 

Link to comment

The extra day matters when you are in an emergency situation and have to preclear a drive for a rebuild.

 

In my case, I usually do a 5-7 pass badblocks test before even doing a preclear.

Then I stage the drive in the emergency repair zone.

That can take almost a week.

 

I'm sure many people will appreciate saving a day!

Thanks bjp999!

Would you care to share how you do this step by step for a new user ?

 

Explain this also if you will.

"Then I stage the drive in the emergency repair zone"

 

As far as staging to an emergency zone. Well that's me just being a bit silly. I prepare the drive then sit it on top of the server.(or near the server).

I'm cross posting the other text from another post about 4TB preclear times.

 

I've done my own form of preclear on 2 4T drives.

This consists of a smart long test (Which I do on every drive right away).

I suppose this is equivalent to the pre-read.

 

A 4 Pass pattern using badblocks 0xaa, 0x55, 0xff and finally 0x00.

Then another final smart long test

I suppose this is equivalent to the post-read.

 

I did these both at the same time via screen on unRAID 5.0.6 under ESXI 5 on an HP Microserver.

 

Times are below.

Model Pre-read Badblocks Post-Read Total
Seagate ST4000DX001 - 4TB 5400 RPM Hybrid 9:03 67:34 9:03 85
Hitachi HDN724040ALE640 - 4TB 7200 RPM 9:37 65:56 9:37 83

 

I've included the logs in case anyone is curious about using smartctl long test and the badblocks pattern test for burn in/clear

Seagate ST4000DX001 - 4TB 5400 RPM Hybrid

root@unRAID2:/boot/logs# date;badblocks -sv -w -c 512 -t0xaa -t0x55 -t0xff -t0x00 -o /boot/logs/ST4000DX001-1CE168_Z3014JCE.201410221000.badblocks  /dev/sdd
Wed Oct 22 10:15:47 EDT 2014
Checking for bad blocks in read-write mode
From block 0 to 3907018583
Testing with pattern 0xaa: done
Reading and comparing: done
Testing with pattern 0x55: done
Reading and comparing: done
Testing with pattern 0xff: done
Reading and comparing: done
Testing with pattern 0x00: done
Reading and comparing: 59:11:20 elapsed
86.75% done, 65:59:48 elapsed
86.75% done, 65:59:50 elapsed
done
Pass completed, 0 bad blocks found.
root@unRAID2:/boot/logs# date
Sat Oct 25 05:50:13 EDT 2014

Hitachi HDN724040ALE640 - 4TB 7200 RPM

root@unRAID2:/boot/logs# date;badblocks -sv -w -c 512 -o /boot/logs/HGST_HDN724040ALE640_PK2334PCGTEKJB.201410221000.badblocks -t0xaa -t0x55 -t0xff -t0x00 /dev/sdc
Wed Oct 22 10:13:56 EDT 2014
Checking for bad blocks in read-write mode
From block 0 to 3907018583
Testing with pattern 0xaa: done
Reading and comparing: done
Testing with pattern 0x55: done
Reading and comparing: done
Testing with pattern 0xff: done
Reading and comparing: done
Testing with pattern 0x00: done
Reading and comparing: 94% done, 59:13:05 elapsed
done                                
Pass completed, 0 bad blocks found.
root@unRAID2:/boot/logs# date
Sat Oct 25 04:09:51 EDT 2014

root@unRAID2:/boot/logs# smartctl -t long /dev/sdc   
smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 577 minutes for test to complete.
Test will complete after Sat Oct 25 13:51:50 2014

 

At the end of this all,. I'll use JoeL's preclear to put the signature on the drive.

I prefer to use smartctl long test for the pre/post reads as it logs it into the drive itself.

I prefer to use badblocks as I can do multiple passes with patterns.

I'm not fond of a blind dd.  I like the ability to use patterns walking one way or another then clear with the -t0x00

However, as JoeL mentions, badblocks alone cannot determine the drive's health.

You need to look at the syslog for retries and the smart logs for changes.

Link to comment

As far as staging to an emergency zone. Well that's me just being a bit silly. I prepare the drive then sit it on top of the server.(or near the server).

That's fine as long as you are the only living thing larger than a couple pounds with access to the room. Those of us with pets / kids / etc can't just leave drives laying around.
Link to comment

Sorry about that. The skip_pct was something that I was working on to resume a preclear that was partially started. I accidentally posted the version I was testing with.

 

EVERYONE, please download the version below. The original post has also been updated to download this corrected version.

 

Here is the link to the download file Fast Preclear

 

Can you put this link in the first post?  It could get tricky to find in the future.

 

Thanks!

Link to comment

Sorry about that. The skip_pct was something that I was working on to resume a preclear that was partially started. I accidentally posted the version I was testing with.

 

EVERYONE, please download the version below. The original post has also been updated to download this corrected version.

 

Here is the link to the download file Fast Preclear

 

Can you put this link in the first post?  It could get tricky to find in the future.

 

Thanks!

 

It was already done. No need to duplicate.

Link to comment
  • 4 weeks later...

Just as a point of reference - Using an old Dell laptop with WD 2G green drives, i had the following results - thats a pretty substantial time savings overall -for me on a 2tb drive its the difference between starting a preclear late at night, and being up and running the next evening vs the day after - so thanks!

 

========================================================================1.15

== invoked as: ./preclear_disk.sh -A /dev/sdc

== WDCWD20EZRX-00D8PB0  WD-WMC4M3155598

== Disk /dev/sdc has been successfully precleared

== with a starting sector of 64

== Ran 1 cycle

==

== Using :Read block size = 8388608 Bytes

== Last Cycle's Pre Read Time  : 5:33:15 (100 MB/s)

== Last Cycle's Zeroing time  : 5:11:48 (106 MB/s)

== Last Cycle's Post Read Time : 11:57:17 (46 MB/s)

== Last Cycle's Total Time    : 22:43:21

==

== Total Elapsed Time 22:43:21

 

========================================================================1.15b

== invoked as: ./preclear_bjp.sh -f -A /dev/sdc

== WDCWD20EZRX-00D8PB0  WD-WMC4M3155598

== Disk /dev/sdc has been successfully precleared

== with a starting sector of 64

== Ran 1 cycle

==

== Using :Read block size = 8388608 Bytes

== Last Cycle's Pre Read Time  : 5:33:08 (100 MB/s)

== Last Cycle's Zeroing time  : 5:04:06 (109 MB/s)

== Last Cycle's Post Read Time : 6:20:01 (87 MB/s)

== Last Cycle's Total Time    : 16:58:15

==

== Total Elapsed Time 16:58:15

Link to comment
  • 4 weeks later...

I use a HP N36L Microserver with 1gb memory for preclearing using 32bit unraid. Here are my results while preclearing 4 - 6tb WD Red drives at the same time. These results were of the first pass preclear using Joe's standard preclear and the second pass preclear using the faster preclear.

 

========================================================================1.15

== invoked as: ./preclear_disk.sh -r 65536 -w 65536 -b 2000 -A /dev/sdd

== WDCWD60EFRX-68MYMN1  WD-WX11D9435LPF

== Disk /dev/sdd has been successfully precleared

== with a starting sector of 1

== Ran 1 cycle

==

== Using :Read block size = 65536 Bytes

== Last Cycle's Pre Read Time  : 17:16:54 (96 MB/s)

== Last Cycle's Zeroing time  : 13:55:03 (119 MB/s)

== Last Cycle's Post Read Time : 40:40:12 (40 MB/s)

== Last Cycle's Total Time    : 71:53:11

==

== Total Elapsed Time 71:53:12

==

== Disk Start Temperature: 35C

==

== Current Disk Temperature: 36C,

==

============================================================================

 

========================================================================1.15b

== invoked as: ./preclear_bjp.sh -f -r 65536 -w 65536 -b 2000 -A /dev/sdd

== WDCWD60EFRX-68MYMN1  WD-WX11D9435LPF

== Disk /dev/sdd has been successfully precleared

== with a starting sector of 1

== Ran 1 cycle

==

== Using :Read block size = 65536 Bytes

== Last Cycle's Pre Read Time  : 17:19:37 (96 MB/s)

== Last Cycle's Zeroing time  : 13:55:42 (119 MB/s)

== Last Cycle's Post Read Time : 22:31:47 (73 MB/s)

== Last Cycle's Total Time    : 53:48:07

==

== Total Elapsed Time 53:48:07

==

== Disk Start Temperature: 32C

==

== Current Disk Temperature: 36C,

==

============================================================================

 

As you can see it cut the Post Read time by 18 hours or almost 50%.

Link to comment

I got an error on line 1182:

      read_speed=`/boot/readvz if=$1 bs=$units count=$bcount skip=$skip memcnt=50 2>>/tmp/postread_errors$disk_basename | awk '{ print $8,$9 }'`

because it's hard coded to /boot

 

Yes, I read the readme.txt indicating they both needed to be in /boot, but I didn't really think you meant it...  :(

 

Could you change the "`/boot/readvz" to "`./readvz" to allow the preclear_bjp.sh and readvz to be in the same directory, whatever it may be? I'd give it a shot, but it already took 32 hours to get through the first 9 steps, and I don't want to go through that again just to find that it won't work. I've moved readvz to /boot for now...

Link to comment

If this were put at the top of the shell

 

[ ${DEBUG:=0} -gt 0 ] && set -x -v

 

B=${0##*/}              # basename of program

D=${0%%/$B}            # dirname of program

E=${B#*.}              # Extension

P=${B%.*}              # strip off after last . character

 

echo "D=${D} B=${B} P=${P} E=${E}"

echo "readz=${D}/readz"

 

Then you can determine how the shell was started and use ${D} to get the root directory of the shell's execution.

Example:

 

root@unRAID:/boot/bin# /boot/bin/rootofshelltest.sh   

D=/boot/bin B=rootofshelltest.sh P=rootofshelltest E=sh

readz=/boot/bin/readz

 

root@unRAID:/boot/bin# ./rootofshelltest.sh

D=. B=rootofshelltest.sh P=rootofshelltest E=sh

readz=./readz

 

The requirement would then be to have readz installed in the same directory as the calling program. i.e. preclear.

Having the requirement that it exists in the root of /boot is a bit strict.

Link to comment

Great tip, Weebo!

 

And the results of my preclear:

Preclear Successful

... Total time 81:00:40

... Pre-Read time 14:59:36 (74 MB/s)

... Zeroing time 17:25:10 (63 MB/s)

... Post-Read time 15:32:50 (71 MB/s)

 

That was a 2-pass on a 4TB HGST NAS drive. That's ~48 hours for the 2nd pass, but only about 33 hours for the first pass. Lots of varying use on the server over the last 3 days, so I guess that could be enough to account for the difference. Also, it's plugged into a slow card - I'll rearrange all my SATA connections when I rearrange the drives here in the next few days.

 

Since that's the 2nd & 3rd run (the first failed just before the pre-read), and there have been 0 pending sectors the whole time, I think I'll call this one good!

 

Link to comment

I'm preclearing 2 drives and just checked the progress.

At the moment the post-read of one drive is running and it says:

./preclear_bjp.sh: line 1186: post_read_err: command not found

Obviously it's running pretty fast as the screen updates approximately every .3s

It seems there is something wrong here.

Probably related to the elderly hardware?

Motherboard:

      CPU Typ                                          DualCore Intel Core 2 Duo E4300, 1800 MHz (9 x 200)

      Motherboard Name                            MSI MS-7293VP (FSC OEM)  (2 PCI, 1 PCI-E x1, 1 PCI-E x16, 2 DDR2 DIMM, Audio, LAN, IEEE-1394)

      Motherboard Chipset                        VIA PT890

      RAM                                                2 GB

 

The run is over soon. I will post some news along with the preclear report next year ^^. ...6 minutes to go...

Link to comment

Your drive, which should have been all zeroes after the writing phase, is not.

 

Line 1186 says:

        post_read_err = "Y"

 

it should say:

        post_read_err="Y"

 

(Note there should not be spaces before and after the "=". Did I every say that I had shell scripts because the syntax is so incredibly unforgiving!)

 

I will post an update, but in the mean time it would be an easy edit for you to make.

 

This is not a good thing. It seems to imply that your system is not able to reliably write to the disk. Could be a disk issue (very unlikely, a memory issue, or a controller issue). A post read mismatch is very rare.

Link to comment

Happy New Year!

 

Here is the screen after the preclear is finished.

Unfortunately I switched consoles and now I'm not able to scroll upwards.

 

Also attached the preclear logs.

 

Did I every say that I had shell scripts because the syntax is so incredibly unforgiving!

Oh, how well I know that feeling...!

This is not a good thing. It seems to imply that your system is not able to reliably write to the disk. Could be a disk issue (very unlikely, a memory issue, or a controller issue). A post read mismatch is very rare.

I can't remember if I ever precleared drives with this PC.

Maybe the onboard controller has issues with big drives? But wouldn't there be write errors also (first phase of the script)?

preclear_bjp.jpg.ba31595ed99b27822a722a2dcdfea7e7.jpg

preclear_reports.zip

Link to comment

The screen shot looks like it is having problems running readvz. This is the heart of the post-read speed improvemnt. Based on this I take back what I said about there being a write issue. Problem is probably just that your post read verify didn't run.

 

Ensure you have the correct version (32 bit for version 5, 64 bit for version 6) of readvz in the boot of the flash drive (/boot).

Link to comment

You're probably right with the write error issue.

After thinking further before I went to bed, I remember having cleared a drive but when I plugged it

into my PC, windows still reported 2 partitions on that drive.

I was in a hurry and did not investigate further but now it makes sense.

There are probably no writes at all on the drives when clearing with this rig.

 

I attached the syslog - perhaps you can find something?

 

I don't know where the issue with readvz could be. I downloaded the zip in the OP 2 days ago.

 

Edit:

Probably found the issue.

It's unraid 6b6 and I used the 32bit readvz.

Will this solve the write issue also?

Is it still necessary to use the LC_CTYPE? There is no info on this in the readme.

syslog.txt

Link to comment

You need to use the 64 bit version of readvz with unRAID version 6.

 

I do think the disk read and disk zeroing steps completed. But the final step is the post read / verify.  And although it was reading, it was not verifying.

 

I would fix the line 1186 issue, put the 64-bit version of readvz in /boot, and re-preclear the disk (use -W option).

 

You can confirm that the version runs by entering the command

 

/boot/readvz

 

using putty or on the unRAID console.

 

It should return some usage instructions if properly installed.

Link to comment

I've just precleared a WD 6TB using the script and it's giving me an 'HPA?' entry against the drive when viewed via unmenu. A bit out of the loop here but is this anything to get concerned about? Drive has been allocated as parity for the array and has been precleared 3 cycles and parity checked against existing data drives which are still at 4TB. Am i likely to see any problems as i migrate drives to 6TB if there actually IS an hpa present or is this a known issue?

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.