Jump to content
SSD

Unofficial Faster Preclear

250 posts in this topic Last Reply

Recommended Posts

Definitely worthwhile to read and verify within one program.

 

Perhaps change the parameter in the open and see if O_DIRECT changes things.

Either that or drop the buffer in the readvz program with some command line option.

Share this post


Link to post

bkastner -

 

Can you run the following command:

 

/boot/readvz if=/dev/sd? bs=8388608 count=200 skip=25000 memcnt=50

 

Change the sd? to one of your newly precleared drives (e.g., sdh).

 

Post the results.

 

Thanks!

Share this post


Link to post

My core question is,  Is the unraid server (virtualized or not) capable of holding 4TB in ram?

 

You are, of course, kidding !!  :)

... unless you've acquired some RAM modules from the future  8)

(and a motherboard that supports them)

 

Share this post


Link to post

bkastner -

 

Can you run the following command:

 

/boot/readvz if=/dev/sd? bs=8388608 count=200 skip=25000 memcnt=50

 

Change the sd? to one of your newly precleared drives (e.g., sdh).

 

Post the results.

 

Thanks!

 

Here you go:

 

root@CydStorage:/boot# /boot/readvz if=/dev/sdo bs=8388608 counto200 skip=25000 memcnt=50

Segmentation fault (core dumped)

root@CydStorage:/boot#

 

Share this post


Link to post

bkastner -

 

Can you run the following command:

 

/boot/readvz if=/dev/sd? bs=8388608 count=200 skip=25000 memcnt=50

 

Change the sd? to one of your newly precleared drives (e.g., sdh).

 

Post the results.

 

Thanks!

 

Here you go:

 

root@CydStorage:/boot# /boot/readvz if=/dev/sdo bs=8388608 counto200 skip=25000 memcnt=50

Segmentation fault (core dumped)

root@CydStorage:/boot#

 

Looks like a typo on the command (see blue text above).

 

Try again please.

 

Thanks!

Share this post


Link to post

bkastner -

 

Can you run the following command:

 

/boot/readvz if=/dev/sd? bs=8388608 count=200 skip=25000 memcnt=50

 

Change the sd? to one of your newly precleared drives (e.g., sdh).

 

Post the results.

 

Thanks!

 

Here you go:

 

root@CydStorage:/boot# /boot/readvz if=/dev/sdo bs=8388608 counto200 skip=25000 memcnt=50

Segmentation fault (core dumped)

root@CydStorage:/boot#

 

Looks like a typo on the command (see blue text above).

 

Try again please.

 

Thanks!

 

It took a lot longer, but same result:

 

root@CydStorage:/boot# /boot/readvz if=/dev/sdo bs=8388608 count=200 skip=25000 memcnt=50

Segmentation fault (core dumped)

root@CydStorage:/boot#

 

Share this post


Link to post

My core question is,  Is the unraid server (virtualized or not) capable of holding 4TB in ram?

 

You are, of course, kidding !!  :)

... unless you've acquired some RAM modules from the future  8)

(and a motherboard that supports them)

LMAO, I just woke up and must not have been thinking or adding correctly!!!

Share this post


Link to post

My standard preclear just finished on my other drive:

 

========================================================================1.13

== invoked as: ./preclear_disk.sh -A /dev/sdc

==  WDC WD40EZRX-00SPEB0    WD-WCC4E0503635

== Disk /dev/sdc has been successfully precleared

== with a starting sector of 1

== Ran 1 cycle

==

== Using :Read block size = 1000448 Bytes

== Last Cycle's Pre Read Time  : 12:00:21 (92 MB/s)

== Last Cycle's Zeroing time  : 10:39:12 (104 MB/s)

== Last Cycle's Post Read Time : 19:35:00 (56 MB/s)

== Last Cycle's Total Time    : 42:15:32

==

== Total Elapsed Time 42:15:32

==

== Disk Start Temperature: 27C

==

== Current Disk Temperature: 29C,

 

I am going to load up 6.0-beta4 and need to mount this one drive, but can still play around with the other that finished in 23 hours if there is any further testing to do.

Share this post


Link to post

My standard preclear just finished on my other drive:

 

========================================================================1.13

== invoked as: ./preclear_disk.sh -A /dev/sdc

==  WDC WD40EZRX-00SPEB0    WD-WCC4E0503635

== Disk /dev/sdc has been successfully precleared

== with a starting sector of 1

== Ran 1 cycle

==

== Using :Read block size = 1000448 Bytes

== Last Cycle's Pre Read Time  : 12:00:21 (92 MB/s)

== Last Cycle's Zeroing time  : 10:39:12 (104 MB/s)

== Last Cycle's Post Read Time : 19:35:00 (56 MB/s)

== Last Cycle's Total Time    : 42:15:32

==

== Total Elapsed Time 42:15:32

==

== Disk Start Temperature: 27C

==

== Current Disk Temperature: 29C,

 

I am going to load up 6.0-beta4 and need to mount this one drive, but can still play around with the other that finished in 23 hours if there is any further testing to do.

 

Thanks. I just sent Weebo an updated version to test under 64 bit. Hopefully this fixed the problem you were seeing. It also adds an option to display status messages of what it is doing which will help me figure out where it blows up if this happens again. Stay tuned.

Share this post


Link to post

Do you want me to try and preclear again under 6.0 as well? Or do you want to leave it with Weebo? I have one 4TB drive I don't need immediately I can play with.

Share this post


Link to post

Do you want me to try and preclear again under 6.0 as well? Or do you want to leave it with Weebo? I have one 4TB drive I don't need immediately I can play with.

 

I can update and build / test under 32 bit, but I do not have the ability to recompile with 64 bit. Tom did it for me last time, but Weebo was running some tests this time. But I am not sure the Linux he is using would produce an executable compatible for Slackware 64 bit or not. Maybe I'll try sending it to Tom to compile again.

 

But I definitely would like you to retest on your drive!

 

Thanks.

Share this post


Link to post

Do you want me to try and preclear again under 6.0 as well? Or do you want to leave it with Weebo? I have one 4TB drive I don't need immediately I can play with.

 

I can update and build / test under 32 bit, but I do not have the ability to recompile with 64 bit. Tom did it for me last time, but Weebo was running some tests this time. But I am not sure the Linux he is using would produce an executable compatible for Slackware 64 bit or not. Maybe I'll try sending it to Tom to compile again.

 

But I definitely would like you to retest on your drive!

 

Thanks.

 

Okay. Just let me know when you are ready for me to retest. I bought 2 4TB drives as they were on sale, but I am in no hurry to use the 2nd drive, so am happy to keep testing things out.

Share this post


Link to post

Sorry to say I'm slow at getting started on testing of this.

 

 

I do have the Regular test on this:

==
== Using :Read block size = 8225280 Bytes
== Last Cycle's Pre Read Time  : 8:08:04 (102 MB/s)
== Last Cycle's Zeroing time   : 9:06:15 (91 MB/s)
== Last Cycle's Post Read Time : 16:34:25 (50 MB/s)
== Last Cycle's Total Time     : 25:41:38
==

== Total Elapsed Time 85:16:41

I have just started the fast timings on this drive.  I will not try the 64bit version for now.

Share this post


Link to post

Do you want me to try and preclear again under 6.0 as well? Or do you want to leave it with Weebo? I have one 4TB drive I don't need immediately I can play with.

 

I can update and build / test under 32 bit, but I do not have the ability to recompile with 64 bit. Tom did it for me last time, but Weebo was running some tests this time. But I am not sure the Linux he is using would produce an executable compatible for Slackware 64 bit or not. Maybe I'll try sending it to Tom to compile again.

 

But I definitely would like you to retest on your drive!

 

Thanks.

 

 

What version of slackware is 6.0 compatible with? I'll see about making a virtual machine.

Share this post


Link to post

What version of slackware is 6.0 compatible with? I'll see about making a virtual machine.

 

Looks like Slackware 14.1.

 

I run vmware workstation 10.0. Would be lovely to have a 64-bit VM that I could test with. I have created a vmware 32 bit unRaid image, and used the unmenu package manager (a thing of beauty!) to install the C compiler and supporting libraries. (My virtual machine is kind of kludgy - it boots from a virtual drive and then switches to the USB stick as unRAID boots, but hey, it works). I assume I could do the same thing with x64, but without a package manager or instructions to load a 64-bit version of the c-compiler, I would be lost.

 

Thanks Weebo!

 

BTW, Tom did a compile which I sent to bkastner and asked him to test it out. We'll report back once we got it fixed on 64-bit. Anyone run any tests with 32 bit (besides me).

Share this post


Link to post

Bkastner is rerunning his 4T test. Looks like the 64 bit issue is solved! Once I get the thumbs up from him will redistribute the beta to the beta testers.

Share this post


Link to post

Here's my first result from the 32 bit version of the first beta, this is on a pair of old WD 2TB green drives:

 

== invoked as: ./pc15b.sh -f /dev/sdc
== WDCWD20EARS-00J2GB0   WD-WCAYY0100121
== Disk /dev/sdc has been successfully precleared
== with a starting sector of 63 
== Ran 1 cycle
==
== Using :Read block size = 8388608 Bytes
== Last Cycle's Pre Read Time  : 8:08:34 (68 MB/s)
== Last Cycle's Zeroing time   : 10:56:21 (50 MB/s)
== Last Cycle's Post Read Time : 8:00:06 (69 MB/s)
== Last Cycle's Total Time     : 27:06:01

 

== invoked as: ./pc15b.sh -f /dev/sdd
== WDCWD20EARS-00MVWB0   WD-WCAZA6293604
== Disk /dev/sdd has been successfully precleared
== with a starting sector of 63 
== Ran 1 cycle
==
== Using :Read block size = 8388608 Bytes
== Last Cycle's Pre Read Time  : 8:09:39 (68 MB/s)
== Last Cycle's Zeroing time   : 10:57:49 (50 MB/s)
== Last Cycle's Post Read Time : 7:59:14 (69 MB/s)
== Last Cycle's Total Time     : 27:07:44

 

I'll rerun these drives through an old preclear next.  And here's the results with the old preclear:

 

== invoked as: ./preclear_disk.sh /dev/sdc
== WDCWD20EARS-00J2GB0   WD-WCAYY0100121
== Disk /dev/sdc has been successfully precleared
== with a starting sector of 63 
== Ran 1 cycle
==
== Using :Read block size = 8388608 Bytes
== Last Cycle's Pre Read Time  : 7:37:40 (72 MB/s)
== Last Cycle's Zeroing time   : 11:13:52 (49 MB/s)
== Last Cycle's Post Read Time : 15:02:26 (36 MB/s)
== Last Cycle's Total Time     : 33:54:57
==
== Total Elapsed Time 33:54:57

 

== invoked as: ./preclear_disk.sh /dev/sdd
== WDCWD20EARS-00MVWB0   WD-WCAZA6293604
== Disk /dev/sdd has been successfully precleared
== with a starting sector of 63 
== Ran 1 cycle
==
== Using :Read block size = 8388608 Bytes
== Last Cycle's Pre Read Time  : 7:37:22 (72 MB/s)
== Last Cycle's Zeroing time   : 11:13:41 (49 MB/s)
== Last Cycle's Post Read Time : 14:51:50 (37 MB/s)
== Last Cycle's Total Time     : 33:43:53
==
== Total Elapsed Time 33:43:53

 

So the new preclear cut the second pass time from 15 hours to 8 hours, which is great.

 

One odd thing is the preread time is about 30 minutes longer with the new code.

 

Stephen

Share this post


Link to post

Times look very good.  The original pre-clear script typically takes about as long for the post-read as the SUM of the first two processes (pre-read and zeroing) -- so this has saved a significant amount of time (likely 8-10 hours).

 

While the pre-clear cycle still takes a long time, it's certainly nice to trim off those extra hours!!

 

Share this post


Link to post

Here's my first result from the 32 bit version of the first beta, this is on a pair of old WD 2TB green drives:

 

== invoked as: ./pc15b.sh -f /dev/sdc
== WDCWD20EARS-00J2GB0   WD-WCAYY0100121
== Disk /dev/sdc has been successfully precleared
== with a starting sector of 63 
== Ran 1 cycle
==
== Using :Read block size = 8388608 Bytes
== Last Cycle's Pre Read Time  : 8:08:34 (68 MB/s)
== Last Cycle's Zeroing time   : 10:56:21 (50 MB/s)
== Last Cycle's Post Read Time : 8:00:06 (69 MB/s)
== Last Cycle's Total Time     : 27:06:01

 

== invoked as: ./pc15b.sh -f /dev/sdd
== WDCWD20EARS-00MVWB0   WD-WCAZA6293604
== Disk /dev/sdd has been successfully precleared
== with a starting sector of 63 
== Ran 1 cycle
==
== Using :Read block size = 8388608 Bytes
== Last Cycle's Pre Read Time  : 8:09:39 (68 MB/s)
== Last Cycle's Zeroing time   : 10:57:49 (50 MB/s)
== Last Cycle's Post Read Time : 7:59:14 (69 MB/s)
== Last Cycle's Total Time     : 27:07:44

 

I'll rerun these drives through an old preclear next.

 

Stephen

 

Looks good! Thanks vca!

Share this post


Link to post

Sorry to say I'm slow at getting started on testing of this.

 

 

I do have the Regular test on this:

==
== Using :Read block size = 8225280 Bytes
== Last Cycle's Pre Read Time  : 8:08:04 (102 MB/s)
== Last Cycle's Zeroing time   : 9:06:15 (91 MB/s)
== Last Cycle's Post Read Time : 16:34:25 (50 MB/s)
== Last Cycle's Total Time     : 25:41:38
==

== Total Elapsed Time 85:16:41

I have just started the fast timings on this drive.  I will not try the 64bit version for now.

Here are my fast times for 3 cycles on 3TB Red (same drive):

== Using :Read block size = 8388608 Bytes
== Last Cycle's Pre Read Time  : 8:09:22 (102 MB/s)
== Last Cycle's Zeroing time   : 8:34:41 (97 MB/s)
== Last Cycle's Post Read Time : 9:19:26 (89 MB/s)
== Last Cycle's Total Time     : 17:55:07
==
== Total Elapsed Time 61:53:09

quite a bit of time saved when you look at the Total Elapsed Time - saved a whole day!  This was 32 bit version on 5.0 beta12a that is my clearing station.  Never bothered to upgrade it since 12a came out.  Must do that sometime.

Share this post


Link to post

I've completed the preclear of my 4TB drive under unRAID 6-beta4.

 

Here were the previous results using Joe's script:

 

== Using :Read block size = 1000448 Bytes

== Last Cycle's Pre Read Time  : 12:00:21 (92 MB/s)

== Last Cycle's Zeroing time  : 10:39:12 (104 MB/s)

== Last Cycle's Post Read Time : 19:35:00 (56 MB/s)

== Last Cycle's Total Time    : 42:15:32

==

== Total Elapsed Time 42:15:32

 

Here are the results using the new script:

 

== Using :Read block size = 1000448 Bytes

== Last Cycle's Pre Read Time  : 11:43:59 (94 MB/s)

== Last Cycle's Zeroing time  : 9:56:12 (111 MB/s)

== Last Cycle's Post Read Time : 11:23:44 (97 MB/s)

== Last Cycle's Total Time    : 33:04:55

==

== Total Elapsed Time 33:04:55

 

Everything looks good (other than the unRAID 6 issue of the drive not actually being marked precleared).

Share this post


Link to post

I'm going to start the preclear of 3x 4TB Hitachi now.

I will start with 5.0.5 and the pc15b2.zip

Stop me if I'm wrong.

 

Share this post


Link to post

I'm going to start the preclear of 3x 4TB Hitachi now.

I will start with 5.0.5 and the pc15b2.zip

Stop me if I'm wrong.

 

Use pc15b.zip, not 15b2. 15b2 has a significant bug and will always cause the drive to appear failing post read verify.

Share this post


Link to post

OK, switching to pc15b

Is this also valid for x64 or for x86 only?

 

Edit:

Also noticed, that without the -A it reports that the drive won't be aligned although it is a 4TB drive.

Is this correct?

Share this post


Link to post

OK, switching to pc15b

Is this also valid for x64 or for x86 only?

 

Edit:

Also noticed, that without the -A it reports that the drive won't be aligned although it is a 4TB drive.

Is this correct?

 

The 15b version works only with 32 bit. I will post an update with both 32 bit and 64 bit support on Friday.

 

Use the -A option. I don't think it is required but I always do.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.