I made some additions to the pre clear script which will really "burn" it in. Basically, on every write step, it will perform 4 writes: 01010101, 10101010, 11111111, and finally 00000000, so the last thing written is zeros. The idea was that I wanted to test bit flips with maximum entropy on the bit and byte level. I've tested it on 5 drives, all of which initially precleared properly with the original script, and it found 1 drive that no longer precleared successfully with this extended script.

 

I originally also added a fifth step of writing /dev/urandom in between the first two and second two byte writes (between the 10101010 and 11111111), but it was taking an extremely long time to clear a 1.5TB drive, and I have to do some 3TB drives after. Therefore, I commented this line out, but feel free to uncomment it if you want to use it.

 

I'm a n00b to all of this, but love these forums, and decided to contribute some code. I've attached the modified preclear extended script. Any comments or additions are welcome. I hope it helps others as it has helped me.

 

 

** Do not download this version. Please download v2 of this script, provided in a post below.

preclear_disk_ext.sh.zip

I made some additions to the pre clear script which will really "burn" it in. Basically, on every write step, it will perform 4 writes: 01010101, 10101010, 11111111, and finally 00000000, so the last thing written is zeros. The idea was that I wanted to test bit flips with maximum entropy on the bit and byte level. I've tested it on 5 drives, all of which initially precleared properly with the original script, and it found 1 drive that no longer precleared successfully with this extended script.

 

I originally also added a fifth step of writing /dev/urandom in between the first two and second two byte writes (between the 10101010 and 11111111), but it was taking an extremely long time to clear a 1.5TB drive, and I have to do some 3TB drives after. Therefore, I commented this line out, but feel free to uncomment it if you want to use it.

 

I'm a n00b to all of this, but love these forums, and decided to contribute some code. I've attached the modified preclear extended script. Any comments or additions are welcome. I hope it helps others as it has helped me.

Nice method of "generating" the alternate bit patterns.  You could improve efficiency by writing three temp files once outside of the loop in /tmp (one for each "bit" pattern) and then using them inside of the loop instead of invoking "tr" three times per write. 

 

There is a different issue though you've probably not considered.  The odds are that those bit patterns are not actually being written to the disk, they are probably being written to the disk driver buffer cache.  Linux will simply queue up the physical writes to the disk and only write the final zeros to the disk.  The rest of the bits you are flipping are likely all going to be in RAM.  (If the block being written is still in the disk buffer cache, then the cache memory will be written, not the physical disk) 

 

You probably need to add the "oflag=direct" to the "dd" commands you've added for them to be effective.  (and expect the process to take substantially longer, as it will be performing a lot more physical I/O.)

 

To prevent confusion, I'd ask you also change the version number, or output in a way to be able to identify which program is being used when looking at the output. It certainly is no longer version 1.13.  (perhaps you can use 1.13E )

 

Joe L.

I made some additions to the pre clear script which will really "burn" it in. Basically, on every write step, it will perform 4 writes: 01010101, 10101010, 11111111, and finally 00000000, so the last thing written is zeros. The idea was that I wanted to test bit flips with maximum entropy on the bit and byte level. I've tested it on 5 drives, all of which initially precleared properly with the original script, and it found 1 drive that no longer precleared successfully with this extended script.

 

I originally also added a fifth step of writing /dev/urandom in between the first two and second two byte writes (between the 10101010 and 11111111), but it was taking an extremely long time to clear a 1.5TB drive, and I have to do some 3TB drives after. Therefore, I commented this line out, but feel free to uncomment it if you want to use it.

 

I'm a n00b to all of this, but love these forums, and decided to contribute some code. I've attached the modified preclear extended script. Any comments or additions are welcome. I hope it helps others as it has helped me.

Nice method of "generating" the alternate bit patterns.  You could improve efficiency by writing three temp files once outside of the loop in /tmp (one for each "bit" pattern) and then using them inside of the loop instead of invoking "tr" three times per write. 

 

There is a different issue though you've probably not considered.  The odds are that those bit patterns are not actually being written to the disk, they are probably being written to the disk driver buffer cache.  Linux will simply queue up the physical writes to the disk and only write the final zeros to the disk.  The rest of the bits you are flipping are likely all going to be in RAM.  (If the block being written is still in the disk buffer cache, then the cache memory will be written, not the physical disk) 

 

You probably need to add the "oflag=direct" to the "dd" commands you've added for them to be effective.  (and expect the process to take substantially longer, as it will be performing a lot more physical I/O.)

 

To prevent confusion, I'd ask you also change the version number, or output in a way to be able to identify which program is being used when looking at the output. It certainly is no longer version 1.13.  (perhaps you can use 1.13E )

 

Joe L.

 

Thanks for your suggestions. I will make those changes. I did run this script on drives that I had pre cleared before, and it took substantially longer to pre clear with this script (from 20 hours to 36hrs), which makes me believe that it was writing it to the disk. However, just to be sure, i'll add that statement to the dd commands, so thanks!

I have no doubt it was taking longer, but I suspect a percentage of what was being written was only going to the buffer cache.    Again, clever method of generating the alternate bit patterns.

 

I would expect the "write" phase to increase in time proportionally if all the blocks were going to the physical disk.  If my script took 5 hours for the "write" phase, then I'd expect 6 additional hours for each of your added commands.  The extra hour on each "write" added for the overhead of the "tr" processing of /dev/zero. 

original = 5 (pre-read) + 5 (write) + 10 (post-read verify) = 20 hours.

 

yours  = 5 (pre-read) + 6 (write 01010101) + 6 (write 10101010) + 6 (write 11111111) + 5 (write 00000000) + 10 (post-read verify) = 38 hours.  Your actual timing seems a bit shorter at 36 hours.  Either the added "tr" commands takes no time at all (unlikely) or some bytes are not being physically written.

 

Joe L.

How come the cache is not an issue with the original pre-clear script, which does not use the oflag=direct? Thanks.

How come the cache is not an issue with the original pre-clear script, which does not use the oflag=direct? Thanks.

Easy, it writes the entire disk, and does not re-write the bytes possibly still in the buffer cache that have not yet been physically written. 

 

If you modified the script so it does 4 complete passes, one with each of the bit patterns, so each bit pattern is written to the entire disk before continuing with the next pattern, then you would not need to worry about the buffer cache either.

 

My logic was:

while not at end of disk (starting at block 0)

      write 4096 blocks with zeros

The logic as you have it now is:

while not at end of disk (starting at block 0)

  write 4096 blocks with first bit pattern

  write 4096 blocks with second bit pattern

  write 4096 blocks with third bit pattern

  write 4096 blocks with zeros

The better alternative would be:

while not at end of disk (starting at block 0)

  write 4096 blocks with first bit pattern

while not at end of disk (starting at block 0)

  write 4096 blocks with second bit pattern

while not at end of disk (starting at block 0)

  write 4096 blocks with third bit pattern

while not at end of disk (starting at block 0)

  write 4096 blocks with zeros

There is no way to buffer an entire physical disk (nobody has hundreds or thousands of gigabytes of RAM) so writing an entire disk at a time guarantees that when writing the next bit pattern, the prior one is already written to the disk for those same blocks.

 

Joe L.

How come the cache is not an issue with the original pre-clear script, which does not use the oflag=direct? Thanks.

Easy, it writes the entire disk, and does not re-write the bytes possibly still in the buffer cache that have not yet been physically written. 

 

If you modified the script so it does 4 complete passes, one with each of the bit patterns, so each bit pattern is written to the entire disk before continuing with the next pattern, then you would not need to worry about the buffer cache either.

 

My logic was:

while not at end of disk (starting at block 0)

      write 4096 blocks with zeros

The logic as you have it now is:

while not at end of disk (starting at block 0)

  write 4096 blocks with first bit pattern

  write 4096 blocks with second bit pattern

  write 4096 blocks with third bit pattern

  write 4096 blocks with zeros

The better alternative would be:

while not at end of disk (starting at block 0)

  write 4096 blocks with first bit pattern

while not at end of disk (starting at block 0)

  write 4096 blocks with second bit pattern

while not at end of disk (starting at block 0)

  write 4096 blocks with third bit pattern

while not at end of disk (starting at block 0)

  write 4096 blocks with zeros

There is no way to buffer an entire physical disk (nobody has hundreds or thousands of gigabytes of RAM) so writing an entire disk at a time guarantees that when writing the next bit pattern, the prior one is already written to the disk for those same blocks.

 

Joe L.

 

I spent a bit more time trying to understand that execution block and came to the understanding that you just explained. I've made a bunch of changes to that original script and have attached it here. I put the version number as 1.13E2, for the fact that it is the second version of this extended script that I've uploaded.

 

Now it does a full disk write of each of the sequences before moving on, just as you describe. There is output to tell the user which write pass it is on and what it is writing that gets displayed with each progress update, e.g. "Write Pass 1 of 4: 01".

 

I wasn't sure exactly how to create an endless temporary prewritten structure instead of using tr, so I've kept the tr command as is for right now.

 

Thanks for all your help thus far, and please let me know if you have any other suggestions!

preclear_disk_ext_v2.zip

I tested this new script on a few new Seagate 3TB drives and it works beautifully. This is with the script that writes each bit pattern to the ENTIRE disk before moving to the next bit pattern, v1.13E2 or version 2. Since this is writing to the entire disk first, we do not need to worry about the buffer cache as Joe pointed out.

 

I know there was some concern about whether to pre-buffer the alternate bit patterns in RAM and copy this pre-buffer to disk during write phases instead of computing them on the fly. However, in testing, on-the-fly 'tr' has extremely negligible performance hit. This is seen by comparing the elapsed times of the write phase of the three non-zero bit patterns with the elapsed time of the zero bit pattern.

 

In fact, the last step of writing all zeros always took slightly longer on all drives, but this is more likely a statistical anomaly than anything. This was not expected as writing all zeros should not have any CPU performance hit since the write command sources directly from /dev/zero and there is no additional 'tr' piping.

 

Benchmarking Results for Preclear Extended script on Seagate 3TB drives (n=3) - I have noted the approximate time each major step took along with the read/write speeds I was experiencing for these drives during that step, presented as (low-high).

 

Total Time to Completion: 44h30m

 

Pre-Read: ~6h

 

Write Step 1 - Writing 01: 6h15m-6h20m (135-155 MB/s)

Write Step 2 - Writing 10: 6h15m-6h20m (135-155 MB/s)

Write Step 3 - Writing 11: 6h20m-6h25m (135-155 MB/s)

Write Step 4 - Writing 00: 6h20m-6h30m (135-155 MB/s)

 

Post-Read: ~13h

 

* All other steps, while they do perform more writes than the original preclear script (4 bit patterns + a random bit pattern), took less than 10 seconds each to perform.

 

Given that this script took significantly longer than my first script, Joe's comment about writing to the cache and not actually to the physical disk was likely true for the first script.

 

In conclusion, expect a significant increase in the amount of time necessary to preclear your drives with this script, but it puts my mind more at ease about the potential longevity of the drives. Only time will tell for sure though.