Unofficial Faster Preclear


Recommended Posts

If you run myMain within unmenu, it will allow you to monitor progress. When complete it will show you the preclear, zeroing, and post clear times. Not sure it handles multi-pass - been a while since I created it and I don't run multiple cycles.

If all you mean is report which pass it is on then you can see that from my graphic in previous post as that is from the myMain menu.  If you meant something else then ignore this.
Link to comment

I just ran a successful test on my slow 20G drive.

 

Pre Read: 5:14

Zeroing: 4:54

Post Read: 5:38 (new method)

................. 5:17 (new method omitting randomized I/O)

................. 9:51 (original method)

 

Beta testers - look for a PM with the link to the new version and further instructions.

 

PC20G.PNG.57a799d4adbb658bc2d5fe7b2397032d.PNG

Link to comment

You guys do know the preclear script creates a report on the flash drive giving the 3 times and speeds?

 

Here's examples from 2 different Toshiba 3T and a Hitachi 3T

 

== Last Cycle's Pre Read Time  : 6:43:49 (123 MB/s)

== Last Cycle's Zeroing time  : 5:48:03 (143 MB/s)

== Last Cycle's Post Read Time : 13:52:32 (60 MB/s)

== Last Cycle's Total Time    : 26:25:32

 

== Last Cycle's Pre Read Time  : 6:33:50 (126 MB/s)

== Last Cycle's Zeroing time  : 5:38:29 (147 MB/s)

== Last Cycle's Post Read Time : 13:35:50 (61 MB/s)

== Last Cycle's Total Time    : 25:49:19

 

== Last Cycle's Pre Read Time  : 6:38:14 (125 MB/s)

== Last Cycle's Zeroing time  : 5:36:23 (148 MB/s)

== Last Cycle's Post Read Time : 13:49:19 (60 MB/s)

== Last Cycle's Total Time    : 26:05:05

 

 

Link to comment

You guys do know the preclear script creates a report on the flash drive giving the 3 times and speeds?

 

Here's examples from 2 different Toshiba 3T and a Hitachi 3T

 

== Last Cycle's Pre Read Time  : 6:43:49 (123 MB/s)

== Last Cycle's Zeroing time  : 5:48:03 (143 MB/s)

== Last Cycle's Post Read Time : 13:52:32 (60 MB/s)

== Last Cycle's Total Time    : 26:25:32

 

== Last Cycle's Pre Read Time  : 6:33:50 (126 MB/s)

== Last Cycle's Zeroing time  : 5:38:29 (147 MB/s)

== Last Cycle's Post Read Time : 13:35:50 (61 MB/s)

== Last Cycle's Total Time    : 25:49:19

 

== Last Cycle's Pre Read Time  : 6:38:14 (125 MB/s)

== Last Cycle's Zeroing time  : 5:36:23 (148 MB/s)

== Last Cycle's Post Read Time : 13:49:19 (60 MB/s)

== Last Cycle's Total Time    : 26:05:05

 

Thanks! Didn't remember it did that. I sent you the beta - there are instructions included to just rerun the post-read and verify on an already precleared disk. If you haven't added these to your array you could give it a try.

Link to comment

Thanks for the add but those were all at the end of last year and they are in service now and I won't need another one for a while. I sold my old ones off except one I put in a USB case so I got no drives left to test with. If I had a drive to test I've give it a run and help you out. I'm just following along because the preclear is so heavily used that any improvement is most excellent to see happening.

Link to comment

Wow...this new preclear is hauling ass.

 

I have 2 4TB drives I am preclearing. One with the new script, one with the legacy. The legacy one is 24 hours in and 9% of post read. The new script one is 23 hours in, and 90% post read. It's jumped from 81 ->90% in the last 5-10 minutes I've been sitting here. While the legacy one will likely still take 12+ hours, the new script should be done in 10-15 minutes.

 

This is great. :)

 

I will post final results from both once they are finished.

 

UPDATE:

 

The new preclear just finished:

 

========================================================================1.15b

== invoked as: ./preclear_disk15b.sh -f /dev/sdo

== WDCWD40EZRX-00SPEB0  WD-WMC4E0043050

== Disk /dev/sdo has been successfully precleared

== with a starting sector of 1

== Ran 1 cycle

==

== Using :Read block size = 1000448 Bytes

== Last Cycle's Pre Read Time  : 11:41:19 (95 MB/s)

== Last Cycle's Zeroing time  : 10:14:10 (108 MB/s)

== Last Cycle's Post Read Time : 1:23:07 (802 MB/s)

== Last Cycle's Total Time    : 23:19:37

==

== Total Elapsed Time 23:19:37

==

== Disk Start Temperature: 30C

==

== Current Disk Temperature: 29C,

 

Wow!!! Preclearing a 4TB drive in under 24 hours is amazing! I will update this once the original preclear finishes at some point tomorrow.

 

Link to comment

Wow...this new preclear is hauling ass.

 

I suppose it's a matter of perspective.  While I certainly agree ~ 24 hours is better than ~ 36 hours, I'm not sure it makes much effective difference.    Either is a LONG wait ... but 24 hours does have one very nice advantage -- you can do one pass/day ... so 3 passes will take 3 days vice the 4 it would take with the original script.  Probably not "hauling ass" by most measures ... but still a very nice improvement  8)

Link to comment

something doesn't see right about this.

 

== Last Cycle's Pre Read Time  : 11:41:19 (95 MB/s)
== Last Cycle's Zeroing time   : 10:14:10 (108 MB/s)
== Last Cycle's Post Read Time : 1:23:07 (802 MB/s)

 

Given today's current technology, how does an application read a 5400 RPM 4tb drive at 800MB/s ?

Maybe something is cached somewhere.

In all my tests the fastest I could get out of my drives was 195MB/s and approx 300-400MB/s on an SSD.

 

While it seems feasible to read through a 4TB drive in a couple hours.

The speed difference is so drastic that I question it.

Link to comment

something doesn't see right about this.

... how does an application read a 5400 RPM 4tb drive at 800MB/s ?

 

Good catch.  I posted my note before the edit that added the final details ... but you're right, the post-read could NOT have been done correctly in that amount of time.  At best, I'd expect it to take ~ the same amount of time as the zeroing time, since it has to make the same full pass on the drive (and an average of 108MB/s is reasonable).

 

But it certainly CANNOT do it at 800MB/s !! -- that's faster than the SATA-III interface  8) 8)

Link to comment

something doesn't see right about this.

 

== Last Cycle's Pre Read Time  : 11:41:19 (95 MB/s)
== Last Cycle's Zeroing time   : 10:14:10 (108 MB/s)
== Last Cycle's Post Read Time : 1:23:07 (802 MB/s)

 

Given today's current technology, how does an application read a 5400 RPM 4tb drive at 800MB/s ?

Maybe something is cached somewhere.

In all my tests the fastest I could get out of my drives was 195MB/s and approx 300-400MB/s on an SSD.

 

While it seems feasible to read through a 4TB drive in a couple hours.

The speed difference is so drastic that I question it.

 

Something definitely wrong.

 

My 32-bit preclear of a 4T drive is still running. I just checked on it and it is running at 90MB/sec - which is what I expect. It is 51% done.

 

bkastner - are you sure that you copied the 64 bit version of readvz onto the root of your flash disk?

 

Go to a command line and enter the command "/boot/readvz" and press enter. What do you see?

 

Also run the command "ls -la /boot/readvz" and post results.

 

You may try the command "chmod 777 /boot/readvz" if you see that the execute attributes are not on the file.

 

Link to comment

something doesn't see right about this.

 

== Last Cycle's Pre Read Time  : 11:41:19 (95 MB/s)
== Last Cycle's Zeroing time   : 10:14:10 (108 MB/s)
== Last Cycle's Post Read Time : 1:23:07 (802 MB/s)

 

Given today's current technology, how does an application read a 5400 RPM 4tb drive at 800MB/s ?

Maybe something is cached somewhere.

In all my tests the fastest I could get out of my drives was 195MB/s and approx 300-400MB/s on an SSD.

 

While it seems feasible to read through a 4TB drive in a couple hours.

The speed difference is so drastic that I question it.

 

Something definitely wrong.

 

My 32-bit preclear of a 4T drive is still running. I just checked on it and it is running at 90MB/sec - which is what I expect. It is 51% done.

 

bkastner - are you sure that you copied the 64 bit version of readvz onto the root of your flash disk?

 

Go to a command line and enter the command "/boot/readvz" and press enter. What do you see?

 

Also run the command "ls -la /boot/readvz" and post results.

 

You may try the command "chmod 777 /boot/readvz" if you see that the execute attributes are not on the file.

 

Yes, I am sure I copied the 64bit version.

 

If I type /boot/readvz I get garbage:

 

root@CydStorage:/boot# /boot/readvz

��_�� not found

root@CydStorage:/boot#

 

Here is the LS output:

root@CydStorage:/boot# ls -la /boot/readvz

-rwxrwxrwx 1 root root 13666 Mar 28 00:10 /boot/readvz

root@CydStorage:/boot#

 

Obviously no need for the chmod.

 

As I mentioned, during the post read it was flying. Rather than an update every 1-2 seconds, the screen would just flash, flash, flash. It was updating 1% roughly every minute.

 

I figured it was just the new algorithm.

 

My old school preclear is still running. Post read is 70% after 34 hours.

 

 

Link to comment

Wow...this new preclear is hauling ass.

 

I suppose it's a matter of perspective.  While I certainly agree ~ 24 hours is better than ~ 36 hours, I'm not sure it makes much effective difference.    Either is a LONG wait ... but 24 hours does have one very nice advantage -- you can do one pass/day ... so 3 passes will take 3 days vice the 4 it would take with the original script.  Probably not "hauling ass" by most measures ... but still a very nice improvement  8)

 

It sucks if this isn't accurate. Given a 4TB usually takes me closer to 48 hours, 24 seemed awesome. :)

Link to comment

something doesn't see right about this.

 

== Last Cycle's Pre Read Time  : 11:41:19 (95 MB/s)
== Last Cycle's Zeroing time   : 10:14:10 (108 MB/s)
== Last Cycle's Post Read Time : 1:23:07 (802 MB/s)

 

Given today's current technology, how does an application read a 5400 RPM 4tb drive at 800MB/s ?

Maybe something is cached somewhere.

In all my tests the fastest I could get out of my drives was 195MB/s and approx 300-400MB/s on an SSD.

 

While it seems feasible to read through a 4TB drive in a couple hours.

The speed difference is so drastic that I question it.

 

I agree it seems odd. I just assumed it was the result of bjp999's new algorithm.

Link to comment

Based in this issue I will add a handshake with the readvz program to make sure it is functional rather than letting it run like this if readvz is failing.

 

If there are any C literate folks here running 64 bit unRaid let me know. Maybe you can help me with this. Probably something small. Otherwise I will need to set up a 64 bit test capability which won't be this weekend.

 

I agree it seems odd. I just assumed it was the result of bjp999's new algorithm.

 

Thanks bkastner for the vote of confidence. Wish I could make magnetic disks faster than SSDs! ;)

 

It is interesting that almost 1.5 hours is spent without the actual read and verify. Maybe there is a way to speed things up further!

Link to comment

My initial code review reveals that there is nothing out of the ordinary other then an open, read, verify buffer.

At this point I can only surmise there is data in the buffer cache allowing the program to read at a higher rate then is physically possible.

 

bkastner, how much ram is in your system?

 

Link to comment

At this point I can only surmise there is data in the buffer cache allowing the program to read at a higher rate then is physically possible.

 

Unless there's also an addressing error that's causing it to reread the same data (from the local buffer cache), this wouldn't explain it either, since it would have to read at least some new data from the disk.  And even if some addressing error was causing it to always read the same spot on the disk (thus the data would be in the disk's buffer),  it wouldn't be this fast, since the indicated speed is appreciably faster than the SATA-III interface !

 

Link to comment

My initial code review reveals that there is nothing out of the ordinary other then an open, read, verify buffer.

At this point I can only surmise there is data in the buffer cache allowing the program to read at a higher rate then is physically possible.

 

bkastner, how much ram is in your system?

 

16GB.

Link to comment

At this point I can only surmise there is data in the buffer cache allowing the program to read at a higher rate then is physically possible.

 

Unless there's also an addressing error that's causing it to reread the same data (from the local buffer cache), this wouldn't explain it either, since it would have to read at least some new data from the disk.  And even if some addressing error was causing it to always read the same spot on the disk (thus the data would be in the disk's buffer),  it wouldn't be this fast, since the indicated speed is appreciably faster than the SATA-III interface !

 

Maybe I had the drive tilted so it was running downhill?  ;D

Link to comment

My initial code review reveals that there is nothing out of the ordinary other then an open, read, verify buffer.

At this point I can only surmise there is data in the buffer cache allowing the program to read at a higher rate then is physically possible.

 

bkastner, how much ram is in your system?

 

16GB.

 

 

Is that 16GB just for the unRAID machine or is the unraid machine virtualized somehow.

do a free and post it for us please.

 

My core question is,  Is the unraid server (virtualized or not) capable of holding 4TB in ram?

if so, that might explain the speed. In that case, there should probably be a directive to drop the buffer cache or open the device in direct mode.

 

 

 

 

Link to comment
O_DIRECT (since Linux 2.4.10)
              Try to minimize cache effects of the I/O to and from this
              file.  In general this will degrade performance, but it is
              useful in special situations, such as when applications do
              their own caching.  File I/O is done directly to/from user-
              space buffers.  The O_DIRECT flag on its own makes an effort
              to transfer data synchronously, but does not give the
              guarantees of the O_SYNC flag that data and necessary metadata
              are transferred.  To guarantee synchronous I/O, O_SYNC must be
              used in addition to O_DIRECT.  See NOTES below for further
              discussion.


              A semantically similar (but deprecated) interface for block
              devices is described in raw(.

Link to comment

My initial code review reveals that there is nothing out of the ordinary other then an open, read, verify buffer.

At this point I can only surmise there is data in the buffer cache allowing the program to read at a higher rate then is physically possible.

 

bkastner, how much ram is in your system?

 

16GB.

 

 

Is that 16GB just for the unRAID machine or is the unraid machine virtualized somehow.

do a free and post it for us please.

 

My core question is,  Is the unraid server (virtualized or not) capable of holding 4TB in ram?

if so, that might explain the speed. In that case, there should probably be a directive to drop the buffer cache or open the device in direct mode.

 

I wasn't thinking when I posted 16GB because I am running in the Xen mode:

 

root@CydStorage:/boot# free

            total      used      free    shared    buffers    cached

Mem:      1701248    1663268      37980          0    231080    802948

-/+ buffers/cache:    629240    1072008

Swap:            0          0          0

 

I haven't changed the defaults on UnRAID, so I still have the 2GB Tom had reserved for the base OS.

Link to comment

At this point I can only surmise there is data in the buffer cache allowing the program to read at a higher rate then is physically possible.

 

Unless there's also an addressing error that's causing it to reread the same data (from the local buffer cache), this wouldn't explain it either, since it would have to read at least some new data from the disk.  And even if some addressing error was causing it to always read the same spot on the disk (thus the data would be in the disk's buffer),  it wouldn't be this fast, since the indicated speed is appreciably faster than the SATA-III interface !

 

 

Years ago when we used to run bonnie tests on our web servers, we had to account for the buffer cache effect. We would have to select a test that was twice as much as ram or it would skew the results.  It has some effect and it could be the reason for the extremely fast results.

Link to comment

Here are the results of running my preclear script on a 4T drive (Seagate ST4000DM000):

 

== Last Cycle's Pre Read Time  : 9:37:34 (115 MB/s)

== Last Cycle's Zeroing time  : 8:20:07 (133 MB/s)

== Last Cycle's Post Read Time : 10:33:36 (107 MB/s)

== Last Cycle's Total Time    : 28:31:19

 

Here is the time to run post-read and verify with old preclear

== Last Cycle's Post Read Time : 20:35:13 (53 MB/s)

 

Total savings: 10:01:37 (about 25% reduction)

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.