Re: preclear_disk.sh - a new utility to burn-in and pre-clear disks for quick add


Recommended Posts

 

preclear_disk.sh -C 63 /dev/sdX

or

preclear_disk.sh -C 64 /dev/sdX

options .   

 

 

Joe L.

 

Please help.. I just precleared 2 drives and have set one up as data disk and loaded content on it. Can it still be changed or do i need to transfer data out and preclear again? Also can the I have drives all mixed and matched?

 

 

Link to comment

 

preclear_disk.sh -C 63 /dev/sdX

or

preclear_disk.sh -C 64 /dev/sdX

options .   

 

 

Joe L.

 

Please help.. I just precleared 2 drives and have set one up as data disk and loaded content on it. Can it still be changed or do i need to transfer data out and preclear again? Also can the I have drives all mixed and matched?

 

 

Once the disk has been assigned to the array it cannot be changed.

 

Disks can easily be mixed and matched.  Any disk can be used with any partition start.  I cannot stress that enough.  There are many who have used an AF drive with a partition starting on sector 63 and it did not affect their use of the drive for playing movies/music.

 

If you are obsessed with absolute max performance of an AF drive, then yes, you'll need to transfer the files elsewhere, un-assign it from the array, and run the pre-clear again.

Link to comment

 

preclear_disk.sh -C 63 /dev/sdX

or

preclear_disk.sh -C 64 /dev/sdX

options .  

 

 

Joe L.

 

Please help.. I just precleared 2 drives and have set one up as data disk and loaded content on it. Can it still be changed or do i need to transfer data out and preclear again? Also can the I have drives all mixed and matched?

 

 

Once the disk has been assigned to the array it cannot be changed.

 

Disks can easily be mixed and matched.  Any disk can be used with any partition start.   I cannot stress that enough.   There are many who have used an AF drive with a partition starting on sector 63 and it did not affect their use of the drive for playing movies/music.

 

If you are obsessed with absolute max performance of an AF drive, then yes, you'll need to transfer the files elsewhere, un-assign it from the array, and run the pre-clear again.

 

i'm not obsessed lol this is my first time.. i just don't want to screw it up.

 

i'll just make sure the other ones are 4k and maybe do a data swap next time my drives are full..thanks very much!

 

so once my drive is done preclearing... i can just run this command and it will change it to 4k correct?

 

./preclear_disk.sh -C 64 -m [email protected] /dev/sdX 

 

Link to comment

 

preclear_disk.sh -C 63 /dev/sdX

or

preclear_disk.sh -C 64 /dev/sdX

options .  

 

 

Joe L.

 

Please help.. I just precleared 2 drives and have set one up as data disk and loaded content on it. Can it still be changed or do i need to transfer data out and preclear again? Also can the I have drives all mixed and matched?

 

 

Once the disk has been assigned to the array it cannot be changed.

 

Disks can easily be mixed and matched.  Any disk can be used with any partition start.   I cannot stress that enough.   There are many who have used an AF drive with a partition starting on sector 63 and it did not affect their use of the drive for playing movies/music.

 

If you are obsessed with absolute max performance of an AF drive, then yes, you'll need to transfer the files elsewhere, un-assign it from the array, and run the pre-clear again.

 

i'm not obsessed lol this is my first time.. i just don't want to screw it up.

 

i'll just make sure the other ones are 4k and maybe do a data swap next time my drives are full..thanks very much!

 

so once my drive is done preclearing... i can just run this command and it will change it to 4k correct?

 

./preclear_disk.sh -C 64 -m [email protected] /dev/sdX  

 

Correct, but the "-m" option is not implemented when converting, since converting takes only a few seconds.

 

./preclear_disk.sh -C 64 /dev/sdX

is more correct.

 

Joe L.

Link to comment

Version 1.7 of the preclear script is now posted.  It has been updated work properly with unRAID 5.0beta5 onward.

 

(The -l option on 1.6 and earlier versions of the preclear script will not work. The format of the /boot/config/disk.cfg file has changed once more in unRAID 5.0beta5 )

 

Joe L.

Link to comment

I just updated to unRAID 4.7 and also dumped the new preclear script (v1.7) onto my USB dive as well.  I launched a preclear cycle on my new WD20EARS drive using the -A option and NO jumper installed.  The preclear in now hung at 1% complete (time=4:17).  The drive responds to smart status requests using the unMenu Disk Management page.  I do not see any errors in my syslog either but there is a message about an unknown partition table on the drive in question (sdd).  What should I do?

syslog-2011-02-28.txt

Link to comment

I just updated to unRAID 4.7 and also dumped the new preclear script (v1.7) onto my USB dive as well.  I launched a preclear cycle on my new WD20EARS drive using the -A option and NO jumper installed.  The preclear ins not hung at 1% complete (time=4:17).  The drive responds to smart status requests using the unMenu Disk Management page.  I do not see any errors in my syslog either but there is a message about an unknown partition table on the drive in question (sdd).  What should I do?

Is the pre-clear continuing?  (Your sentence is not clear?)

 

If it is a brand new disk there is no partition table yet... Ignore the message stating there is none.

Link to comment

Sorry, you read the post before I corrected my typos.

 

The preclear was hung at 1% and not progressing.  The time and bytes read counters were not incrementing either.  I stopped the array and rebooted.  Switched back to preclear v1.6 and launched another preclear.  This time I'm getting essentially the same result.  Now hung at 4:09 & 1% complete.

 

I've never had a drive get hung in the preclear process.  Is this a sign of a bad drive or something else?

Link to comment

Sorry, you read the post before I corrected my typos.

 

The preclear was hung at 1% and not progressing.  The time and bytes read counters were not incrementing either.  I stopped the array and rebooted.  Switched back to preclear v1.6 and launched another preclear.  This time I'm getting essentially the same result.  Now hung at 4:09 & 1% complete.

 

I've never had a drive get hung in the preclear process.  Is this a sign of a bad drive or something else?

I have no way of knowing.  Most times when a drive locks up it is the fault of the drive.  Some-times EARS drives just like to lock up.  It is why I probably will never have one in my array.  Many time it will take a power cycle to get it to respond again.

 

Can you get any output from

fdisk -lu /dev/sdX

??

 

Link to comment

I checked my power connection and replaced the sata cable and started another preclear.  It froze at basically the same point again.  This drive is now 3 for 3.  I'm starting to see a pattern here.

 

It does respond to the fdisk command.  I attached a screen shot of the response.

 

I also got a few read errors from the second preclear process...

Feb 28 18:34:59 Tower kernel: sd 3:0:0:0: [sdd] Unhandled error code
Feb 28 18:34:59 Tower kernel: sd 3:0:0:0: [sdd] Result: hostbyte=0x00 driverbyte=0x06
Feb 28 18:34:59 Tower kernel: sd 3:0:0:0: [sdd] CDB: cdb[0]=0x28: 28 00 03 66 19 48 00 01 00 00
Feb 28 18:34:59 Tower kernel: end_request: I/O error, dev sdd, sector 57022792
Feb 28 18:34:59 Tower kernel: Buffer I/O error on device sdd, logical block 7127849
Feb 28 18:34:59 Tower kernel: Buffer I/O error on device sdd, logical block 7127850
Feb 28 18:34:59 Tower kernel: Buffer I/O error on device sdd, logical block 7127851
Feb 28 18:34:59 Tower kernel: Buffer I/O error on device sdd, logical block 7127852
Feb 28 18:34:59 Tower kernel: Buffer I/O error on device sdd, logical block 7127853
Feb 28 18:34:59 Tower kernel: Buffer I/O error on device sdd, logical block 7127854
Feb 28 18:34:59 Tower kernel: Buffer I/O error on device sdd, logical block 7127855
Feb 28 18:34:59 Tower kernel: Buffer I/O error on device sdd, logical block 7127856
Feb 28 18:34:59 Tower kernel: Buffer I/O error on device sdd, logical block 7127857
Feb 28 18:34:59 Tower kernel: Buffer I/O error on device sdd, logical block 7127858

 

I've gotten similar errors from the thrid cycle so far as well ...

Feb 28 19:07:18 Tower kernel: sd 3:0:0:0: [sdd] Unhandled error code
Feb 28 19:07:18 Tower kernel: sd 3:0:0:0: [sdd] Result: hostbyte=0x00 driverbyte=0x06
Feb 28 19:07:18 Tower kernel: sd 3:0:0:0: [sdd] CDB: cdb[0]=0x28: 28 00 03 67 a0 48 00 01 00 00
Feb 28 19:07:18 Tower kernel: end_request: I/O error, dev sdd, sector 57122888
Feb 28 19:07:18 Tower kernel: Buffer I/O error on device sdd, logical block 7140361
Feb 28 19:07:18 Tower kernel: Buffer I/O error on device sdd, logical block 7140362
Feb 28 19:07:18 Tower kernel: Buffer I/O error on device sdd, logical block 7140363
Feb 28 19:07:18 Tower kernel: Buffer I/O error on device sdd, logical block 7140364
Feb 28 19:07:18 Tower kernel: Buffer I/O error on device sdd, logical block 7140365
Feb 28 19:07:18 Tower kernel: Buffer I/O error on device sdd, logical block 7140366
Feb 28 19:07:18 Tower kernel: Buffer I/O error on device sdd, logical block 7140367
Feb 28 19:07:18 Tower kernel: Buffer I/O error on device sdd, logical block 7140368
Feb 28 19:07:18 Tower kernel: Buffer I/O error on device sdd, logical block 7140369
Feb 28 19:07:18 Tower kernel: Buffer I/O error on device sdd, logical block 7140370

 

I'm going to just let this preclear run for a while, maybe overnight, and see what happens.  I'm thinking that I'll probably RMA this drive based on what I see so far.  Is there something else I should check before doing that?

 

fdisk.JPG.f8de283968ae8eba63b7b75a9d3a54cc.JPG

Link to comment

Well I'm RMAing the drive.  I let the 3rd preclear attempt run for two hours and I'm only at ~37GB out of 2TB read.  That's an average read rate of ~5 MB/sec.  At that rate it would take almost 5 days just to complete the pre-read of the drive!  Not to mention the fact that I have thousands of lines worth of read errors being reported in the syslog.

 

But my real question is I'm wondering how the preclear script determines what frequency to update the display?  That is what really threw me off at the beginning.  I'm accustomed - based on what I've seen from previous preclear cycles on other drives - to seeing the display refreshed with new data (bytes processed, drive speed (MB/sec), drive temperature, and elapsed time) at a regular frequency.  However when running my first preclear I initially thought that something was wrong with the preclear script because the data did not update for more than 10 minutes before I eventually killed the process.  It seems like when it encountered the "bad" area on my new drive the frequency of screen refreshes drastically changed.  During the first four minutes when the drive appeared to be behaving normally the data would be refreshed about every 30-40 seconds until about 4:15 (~27GB read) then the process appeared to be hung because no updates were being displayed.  Even if there was little to no progess in the number of bytes processed due to the read errors I would still expect to see the elapsed time increment at a regular frequency.  It wasn't until the third preclear cycle that I decided to just let the process run for a while that the display was eventually refreshed with new data at something like 25 minutes elapsed time and then again about every 20 minutes thereafter.  This was not a situation I had ever encountered before although I have read about drives being "hung" during preclear.  So perhaps a little ignorance combined with some impatience lead me to the wrong conclusion - i.e. thinking something was wrong with the preclear script.  But it certainly would have been more helpful if at least the elapsed time counter would have kept refreshing at a more reasonable frequency (i.e. every 1-2 minutes).  That way I would have at least known that the process was still operating and not "hung".

 

PS - Sometimes its hard to be patient when your wife is asking "I thought this was only going to take 10 minutes."

Link to comment

I've noticed that variability in refresh time too.  Joe, maybe you could add some sort of indication that preclear is still running and not frozen?  Maybe an asterisks that appears and disappears every second, or something like that?

No... I cannot with the current program.

 

The reading of the disk works in a loop like this:

 

X=some_number_of_blocks_to_be_read

while not-yet-at-the-end-of-the-disk

do

    read X blocks.

    display status

done

 

So, the time it takes to update the display is determined by the time it takes to read a set of blocks of data.  I don't explicitly time the display, it just naturally occurs in between reading each set of blocks.

 

When the disk is failing, the OS tries, and tries, and tries again to reset the disk to read the disk.  It can take seemingly forever to "read X blocks" and the preclear script is just sitting there waiting for the read to complete.  It has no concept of the time it takes.

 

I myself had a disk fail in the same way.  After about 18 hours it finally failed the SMART test and died.  It was barely into 1% of the pre-read phase.

Joe L.

Link to comment

So, the time it takes to update the display is determined by the time it takes to read a set of blocks of data.  I don't explicitly time the display, it just naturally occurs in between reading each set of blocks.

Thanks for the explanation.  That is exactly how I imagined that it worked.  That explains why the update frequency slows as the reading of the disk progresses and the read rate slows.

Link to comment

I've got a drive that came from a second unraid server, is there a way to have unraid just clear the drive and write the signature, but not the whole reads/writes?  The drive has been in service for a while, I trust it already, just need it moved between rigs.

Link to comment

I've got a drive that came from a second unraid server, is there a way to have unraid just clear the drive and write the signature, but not the whole reads/writes?  The drive has been in service for a while, I trust it already, just need it moved between rigs.

 

:sigh:  I meant is there a way to have PRECLEAR do this...

 

And yea, as I mentioned, this drive just came out of another server that had heavy usage, the 2TB is nearly full, I don't need the data as it was moved to the unraid system I'm moving the drive to via teracopy.  It has had preclear run on it in the not too distant past.  If it was a new drive, I'd preclear no questions.

Link to comment

Edit: Feb 1, 2011   1.3   - Added logic to read desired "default" Partition Type from /boot/config.

                                - Added logic to save dated copies of the final preclear and SMART reports to a "preclear_reports" subdirectory on the flash drive.

                                - Added "-R" option to suppress the copy of final reports to a "preclear_reports" directory on the flash drive. (they are always in /tmp until you reboot)

Does the /boot/preclear_reports directory automatically get created when preclear script is invoked?  I'm just about to start pre-clearing a new drive and I thought I ran a pre_clear with a version later than 1.3 a few weeks ago but I don't see a directory with a report inside so I wanted to confirm.  Thanks.

 

EDIT:  Just figured out the pre_clear I did was on 1/31!

Link to comment

forgive me if this is a naive question, but can I run a couple of preclears on a different system, using unraid basic, and then add them to my main unraid plus system?

 

i realize this would take the stress testing of cables/controllers out of the equation, but the drives shouldn't care what system they're precleared on, right?

Link to comment

forgive me if this is a naive question, but can I run a couple of preclears on a different system, using unraid basic, and then add them to my main unraid plus system?

 

i realize this would take the stress testing of cables/controllers out of the equation, but the drives shouldn't care what system they're precleared on, right?

 

yes, just don't format them after they are done being precleared

Link to comment

forgive me if this is a naive question, but can I run a couple of preclears on a different system, using unraid basic, and then add them to my main unraid plus system?

 

i realize this would take the stress testing of cables/controllers out of the equation, but the drives shouldn't care what system they're precleared on, right?

 

yes, just don't format them after they are done being precleared

 

thanks

 

the only way i would accidentally format them is by first assigning them to a slot on the unraid basic 'devices' page, then hitting format on the main page, right? preclear without any additional switches won't do any formatting, as i understand it

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.