Re: preclear_disk.sh - a new utility to burn-in and pre-clear disks for quick add


Recommended Posts

I moved a one of the EARS drives from the onboard to the supermicro card and it works on the supermicro card.  Leads me to believe preclearing these drives without the jumper and adding them after the fact really screwed something up big time.  A friend suggested formatting the drives in windows and then trying it again.  Otherwise I am at a total loss?

Link to comment

Interesting piece of information.  Here is the following command on the supermicro card with a previously cleared drive without the jump and now with the jumper:

 

root@Storage:~# dd if=/dev/zero of=/dev/sdb count=2 bs=512

dd: writing `/dev/sdb': Input/output error

1+0 records in

0+0 records out

0 bytes (0 B) copied, 0.00157492 s, 0.0 kB/s

root@Storage:~#

 

 

 

Now here is the same command with the same type of drive now moved to the onboard:

 

root@Storage:/boot# dd if=/dev/zero of=/dev/sdh count=2 bs=512

2+0 records in

2+0 records out

1024 bytes (1.0 kB) copied, 0.000786006 s, 1.3 MB/s

 

 

sdH is now being pre-cleared so go figure!

 

Neil

 

 

Link to comment

Interesting piece of information.  Here is the following command on the supermicro card with a previously cleared drive without the jump and now with the jumper:

 

root@Storage:~# dd if=/dev/zero of=/dev/sdb count=2 bs=512

dd: writing `/dev/sdb': Input/output error

1+0 records in

0+0 records out

0 bytes (0 B) copied, 0.00157492 s, 0.0 kB/s

root@Storage:~#

 

 

 

Now here is the same command with the same type of drive now moved to the onboard:

 

root@Storage:/boot# dd if=/dev/zero of=/dev/sdh count=2 bs=512

2+0 records in

2+0 records out

1024 bytes (1.0 kB) copied, 0.000786006 s, 1.3 MB/s

 

 

sdH is now being pre-cleared so go figure!

 

Neil

 

 

We've seen before where a power cycle would get a drive that was un-responsive to subsequently be responsive.  (perhaps a side effect of adding the jumper lets you now access the drive, since you powered down in between. It might have nothing to do directly with the jumper.)  The first "dd" failed to write anything to the drive.
Link to comment

That is true when it was on the supermicro card.  I can't seem to get the drive with the jump to take the DD command.  After I preclear this drive sometime tomorrow night I will move it back to the supermicro card and take the one drive off the supermicro card and put it on the onboard.  At that time I believe I will be able to run the DD command properly!?

 

Quick question on the side: I am running the pre-clear from a terminal window on my laptop.  Anyway to preserve this session and turn my laptop off?

 

Thanks,

 

Neil

 

Link to comment

That is true when it was on the supermicro card.  I can't seem to get the drive with the jump to take the DD command.  After I preclear this drive sometime tomorrow night I will move it back to the supermicro card and take the one drive off the supermicro card and put it on the onboard.  At that time I believe I will be able to run the DD command properly!?

 

Quick question on the side: I am running the pre-clear from a terminal window on my laptop.  Anyway to preserve this session and turn my laptop off?

 

Thanks,

 

Neil

 

Yes, but it is too late now.  You need to install and use "screen" and then start the preclear, and then detach.
Link to comment

Joe, If you remember I had a problem with a disk not preclearing (with the jumper). I tried formatting it in windows with the jumper but it also failed. I am now formatting it in windows without the jumper. If the formatting works, should i use the jumper again and preclear it or there is no point to it?

 

I can try it anyway but, since I need to RMA it within some time limits, I do not have the time...

Link to comment

Ok, need some advice guys  ;D

 

Just bought a new 2TB Samsung F4.  Naturally I want to make use of the preclear script to make sure all is ok.  Here's the problem...  I currently have a 750 (P), 750 (Data), 160GB (Data) array configuration.  I don't have any spare sata cables.  I thought I could use the DVD drive's cable, but thats SCSI.  So I pulled the 160GB drive out and popped in the new drive.  Won't work since preclear says it's part of the array... and of course the array is offline since Unraid is complaining the parity drive is no longer the biggest.

 

So the question is, is there anyway I can preclear the drive without buying more cables?  I'm not too fussed about the array being up and running during the preclear process - if it is, that's a bonus.

 

Any ideas anyone (Joe!)?  Can I remove the 160GB drive from the array, swap out the 2TB, preclear, then re-add the 160GB drive to array, swap out the parity drive for the 2TB, then re-arrange the data drives once parity has been re-built?  Or something...

 

Thanks, Matt.

Link to comment

Joe, If you remember I had a problem with a disk not preclearing (with the jumper). I tried formatting it in windows with the jumper but it also failed. I am now formatting it in windows without the jumper. If the formatting works, should i use the jumper again and preclear it or there is no point to it?

 

I can try it anyway but, since I need to RMA it within some time limits, I do not have the time...

I would RMA the drive.  It is not worth your time and aggravation.

 

I have no experience to draw upon with those drives, other than reading about them in these forums.  I've elected to keep clear of them, as there are alternatives.  (besides, all my drives in my newer server are 7200 RPM drives)

Link to comment

Ok, need some advice guys  ;D

 

Just bought a new 2TB Samsung F4.  Naturally I want to make use of the preclear script to make sure all is ok.  Here's the problem...  I currently have a 750 (P), 750 (Data), 160GB (Data) array configuration.  I don't have any spare sata cables.  I thought I could use the DVD drive's cable, but thats SCSI.  So I pulled the 160GB drive out and popped in the new drive.  Won't work since preclear says it's part of the array... and of course the array is offline since Unraid is complaining the parity drive is no longer the biggest.

 

So the question is, is there anyway I can preclear the drive without buying more cables?  I'm not too fussed about the array being up and running during the preclear process - if it is, that's a bonus.

 

Any ideas anyone (Joe!)?  Can I remove the 160GB drive from the array, swap out the 2TB, preclear, then re-add the 160GB drive to array, swap out the parity drive for the 2TB, then re-arrange the data drives once parity has been re-built?  Or something...

 

Thanks, Matt.

 

When the array is stopped, you can remove a disk from it from the devices tab in the unraid webgui. then you wont have any problem with precelear...

Link to comment

Ok, need some advice guys  ;D

 

Just bought a new 2TB Samsung F4.  Naturally I want to make use of the preclear script to make sure all is ok.  Here's the problem...  I currently have a 750 (P), 750 (Data), 160GB (Data) array configuration.  I don't have any spare sata cables.  I thought I could use the DVD drive's cable, but thats SCSI.  So I pulled the 160GB drive out and popped in the new drive.  Won't work since preclear says it's part of the array... and of course the array is offline since Unraid is complaining the parity drive is no longer the biggest.

 

So the question is, is there anyway I can preclear the drive without buying more cables?  I'm not too fussed about the array being up and running during the preclear process - if it is, that's a bonus.

 

Any ideas anyone (Joe!)?  Can I remove the 160GB drive from the array, swap out the 2TB, preclear, then re-add the 160GB drive to array, swap out the parity drive for the 2TB, then re-arrange the data drives once parity has been re-built?  Or something...

 

Thanks, Matt.

 

When the array is stopped, you can remove a disk from it from the devices tab in the unraid webgui. then you wont have any problem with precelear...

 

If I remove the disk from the devices tab, that removes it from the array yeah?  Can I then re-add the original drive to the array once preclear has finished - will there be any issues with the parity drive?

 

Thanks.

Link to comment

as long as you dont start the array you will have no problem.

Bear in mind that since you have a new 2tb drive and it is the largest, you have to use this for parity...

also, i have heard that there are some problems with samsung HDs, so better check the forums for extra info, especially this: http://lime-technology.com/wiki/index.php?title=Hardware_Compatibility#Samsung

 

Thanks for the heads up. 

 

Brilliant, the F4204UI isn't supported yet.  So what to do, send it back to Amazon, or wait and see if there is a solution... which could be a fair few months.  From reading around it looks like it may well work, but isn't advised as a parity drive.  And I need the space now.

:(

Link to comment

as long as you dont start the array you will have no problem.

Bear in mind that since you have a new 2tb drive and it is the largest, you have to use this for parity...

also, i have heard that there are some problems with samsung HDs, so better check the forums for extra info, especially this: http://lime-technology.com/wiki/index.php?title=Hardware_Compatibility#Samsung

 

Thanks for the heads up. 

 

Brilliant, the F4204UI isn't supported yet.  So what to do, send it back to Amazon, or wait and see if there is a solution... which could be a fair few months.  From reading around it looks like it may well work, but isn't advised as a parity drive.  And I need the space now.

:(

 

I would send it back since you can only use it as parity. Maybe get a refund from amazon and order a new drive right now...

Link to comment
  • 2 weeks later...

I've just connected up my new WD20EARS (with jumper installed), and started running the preclear script on it.  I had understood that the raison d'etre for preclear was that the system should still be usable while the new disk was being prepared.

 

However, I'm finding that there is insufficient resource for an HD movie file to be played while the preclear is in progress ... at least I presume that is what the problem is.  The movie will start playing, but then fail after a few seconds - this is happening while attempting to play on my ubuntu desktop machine and on my Popcorn Hour.  Is this expected?

 

I'm running the preclear on a telnet session from ubuntu.  After 40 minutes, it is 10% into the Pre-Read, reporting transfer rates of up to 110MB/s. 

 

Now, I'm sure that there is minimal network traffic being generatd by the preclear/telnet, so I guess that the restriction is on the disk interface/controller, or the data bus.  Is there a way to ensure that normal array activity has precedence over the preclear, so that I can play a movie

Link to comment

I just received my motherboard back from rma. Here's the thing: i've precleared 4 disks succesfully. Now when i add the fifth it's not sda, sdb etc. but hde, it's the same drive as the other ones...could it be that it's because port 5 and 6 are on a different chipset? Just asking because i don't want to start off the wrong foot.

Link to comment

I just received my motherboard back from rma. Here's the thing: i've precleared 4 disks succesfully. Now when i add the fifth it's not sda, sdb etc. but hde, it's the same drive as the other ones...could it be that it's because port 5 and 6 are on a different chipset? Just asking because i don't want to start off the wrong foot.

It indicates the second chipset is configured in your BIOS in Legacy mode where it is emulating an old IDE controller  (IDE devices show as "hda, hdb,etc").

 

You should change the option in your BIOS to AHCI mode and get it out of legacy mode.  Many BIOS have a setting for each of their disk controllers.  The Legacy mode is enabled by default on many motherboards to allow them to boot older microsoft operating systems.  Win-XP and prior have no SATA drivers, and therefore could not boot windows if the chipset was in SATA mode.  This is not an issue with Linux or unRAID.

 

All that said, either setting will work, and you can change the mode later with no effect or impact on your stored files.  It is just that the emulated legacy mode is frequently slower, as it does not take advantage of the features available in modern drives. 

 

Joe L.

Link to comment

However, I'm finding that there is insufficient resource for an HD movie file to be played while the preclear is in progress ... at least I presume that is what the problem is.  The movie will start playing, but then fail after a few seconds - this is happening while attempting to play on my ubuntu desktop machine and on my Popcorn Hour.  Is this expected?

 

I'm a little concerned that, during the pre-read, with no other device accessing unRAID (as far as I am aware), disks 1 and 2 are not being allowed to spin down, and are experiencing something in the order of 4000 reads a minute.  All of the array disks are showing status green, and the parity has now spun up for no apparent reason.

 

If the preclear is reading from more than one disk, is it going to start writing to more than one disk????

Link to comment

I just received my motherboard back from rma. Here's the thing: i've precleared 4 disks succesfully. Now when i add the fifth it's not sda, sdb etc. but hde, it's the same drive as the other ones...could it be that it's because port 5 and 6 are on a different chipset? Just asking because i don't want to start off the wrong foot.

It indicates the second chipset is configured in your BIOS in Legacy mode where it is emulating an old IDE controller   (IDE devices show as "hda, hdb,etc").

 

You should change the option in your BIOS to AHCI mode and get it out of legacy mode.   Many BIOS have a setting for each of their disk controllers.   The Legacy mode is enabled by default on many motherboards to allow them to boot older microsoft operating systems.  Win-XP and prior have no SATA drivers, and therefore could not boot windows if the chipset was in SATA mode.   This is not an issue with Linux or unRAID.

 

All that said, either setting will work, and you can change the mode later with no effect or impact on your stored files.  It is just that the emulated legacy mode is frequently slower, as it does not take advantage of the features available in modern drives.   

 

Joe L.

 

I understand, i was already preclearing the disk but i'll set it to ahci after preclearing. Thanks :)

Link to comment

However, I'm finding that there is insufficient resource for an HD movie file to be played while the preclear is in progress ... at least I presume that is what the problem is.  The movie will start playing, but then fail after a few seconds - this is happening while attempting to play on my ubuntu desktop machine and on my Popcorn Hour.  Is this expected?

 

I'm a little concerned that, during the pre-read, with no other device accessing unRAID (as far as I am aware), disks 1 and 2 are not being allowed to spin down, and are experiencing something in the order of 4000 reads a minute.  All of the array disks are showing status green, and the parity has now spun up for no apparent reason.

 

If the preclear is reading from more than one disk, is it going to start writing to more than one disk????

The pre-clear only reads/writes one disk.

 

It does however read the entire disk, and therefore uses all possible disk buffer cache, displacing any older cached files and directory entries.  It sure sounds as if something on your LAN is accessing the other drives perhaps a windows box doing an index scan, or creating thumbnail images...

 

Joe L.

Link to comment
The pre-clear only reads/writes one disk.

 

It does however read the entire disk, and therefore uses all possible disk buffer cache, displacing any older cached files and directory entries.   It sure sounds as if something on your LAN is accessing the other drives perhaps a windows box doing an index scan, or creating thumbnail images...

 

... in theory!  But the only devices currently active on the network are two Popcornhours, and my ubuntu desktop.  If I kill the preclear, the reads on disk2 cease instantly, and the drive spins down after 15 minutes.  If I then try playing a movie on the Popcornhour, it plays perfectly.  If I restart the preclear, the movie stops playing within 30 seconds.  Even if I then turn the Popcornhour off, the reads on disk2 continue as before.  When I then kill the preclear again, the reads on disk2 cease, and the drive spins down again after 15 minutes!

 

I am convinced that the evidence all points to preclear reading disk2 (sdc) (which is mounted, and part of the array), while it is also reading the new drive (sdf).

 

I'm not sure whether there is a connection, but it might be worth pointing out that I also have the problem that nfs activity on the cache drive actually performs the operations on disk2!

Link to comment

The pre-clear only reads/writes one disk.

 

It does however read the entire disk, and therefore uses all possible disk buffer cache, displacing any older cached files and directory entries.   It sure sounds as if something on your LAN is accessing the other drives perhaps a windows box doing an index scan, or creating thumbnail images...

 

... in theory!  But the only devices currently active on the network are two Popcornhours, and my ubuntu desktop.  If I kill the preclear, the reads on disk2 cease instantly, and the drive spins down after 15 minutes.  If I then try playing a movie on the Popcornhour, it plays perfectly.  If I restart the preclear, the movie stops playing within 30 seconds.  Even if I then turn the Popcornhour off, the reads on disk2 continue as before.  When I then kill the preclear again, the reads on disk2 cease, and the drive spins down again after 15 minutes!

 

I am convinced that the evidence all points to preclear reading disk2 (sdc) (which is mounted, and part of the array), while it is also reading the new drive (sdf).

 

I'm not sure whether there is a connection, but it might be worth pointing out that I also have the problem that nfs activity on the cache drive actually performs the operations on disk2!

You probably have a lot of memory being used by other processes on your server.    As I pinup groups might be involved if they are on the same disk controller. 

 

I'd suggest you set the cache pressure to 200, to allow the cache to be re-used, and also use the options on the pre-clear command to have it use smaller buffers.

 

It sounds as if the cache drive, disk2, and the drive being cleared all share the same spinup group.

 

Joe L.

Link to comment

The pre-clear only reads/writes one disk.

 

... in theory!  But the only devices currently active on the network are two Popcornhours, and my ubuntu desktop.  If I kill the preclear, the reads on disk2 cease instantly, and the drive spins down after 15 minutes.

 

[snip]

 

I am convinced that the evidence all points to preclear reading disk2 (sdc) (which is mounted, and part of the array), while it is also reading the new drive (sdf).

 

Of course, I have no hard proof that the new drive is being read at all - there are no read/write counts reported in unMenu for the new drive, while it is unmounted/unassigned.  However, the new drive does not spin down but, perhaps this means that there is no spin down command sent to an unassigned drive, rather than being an indication that the drive is being accessed?

 

I have now rebooted the unRAID server.  The new drive has now become sde but there are no other changes.  The behaviour when running preclear is exactly the same.

 

You probably have a lot of memory being used by other processes on your server.

 

Memory Info

(from /usr/bin/free)

             total       used       free     shared    buffers     cached
Mem:       3943140     583048    3360092          0      64204     311224
-/+ buffers/cache:     207620    3735520
Swap:            0          0          0

 

As I pinup groups might be involved if they are on the same disk controller.

 

Spin up groups are turned on (I will try turning off), but each of my 5 drives is reporting a different host.

 

 

I'd suggest you set the cache pressure to 200, to allow the cache to be re-used, and also use the options on the pre-clear command to have it use smaller buffers.

 

It sounds as if the cache drive, disk2, and the drive being cleared all share the same spinup group.

 

Okay, I will try some experiments in those areas.

 

As things stand, I'm frightened to add the new drive to the array in case formatting/preparing it will write to the wrong drive.  Also, I'm not comfortable about playing with drive assignments, in case I lose data.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.