I added a bunch of 640G drives and started the clear operation.  3 hours later it is at 17% finished.  

 

Why so long?  And why does the array have to be offline for this to happen.  

 

This seems a natural for threads.  Run the clear code for each drive in its own thread and allow the array to be online while this is going on.  I mean it is just writing zeros to every sector right?  The same with the format (though that seems to be lightning fast).  Once all the prep work is done then the drives could be brought into the array.

 

jwcolby

I'm no expert (see my other threads!) but I would assume its the disk I/O that is the bottle neck, not the processor.

 

My proposed solution (awaiting forum comment and appraisal) and raised on my thread above is to zero the drives in a separate temporary (booted from a basic unRaid USB key) machine. Then physically move the drives to the main unRaid machine and just have the very short wait while they format.

 

Make sense?

There appears to be a utility that will run under UnMenu that will clear the drives outside of the array.  This just seems to be something that the UnRaid should do natively.  What I really don't understand is why it prevents the raid system from coming up.  Unless it is caclulating the parity as it does the clear.

There appears to be a utility that will run under UnMenu that will clear the drives outside of the array.  This just seems to be something that the UnRaid should do natively.  What I really don't understand is why it prevents the raid system from coming up.  Unless it is caclulating the parity as it does the clear.

There is NO utility under unMENU.  There is a shell script I wrote that runs on the command line.

The reason it runs on the command line is simple, it takes many hours to run and unMENU is a "awk" script acting as a web-server.  You would not want to wait 30 hours for a web-page to return. 

 

As far as why the drive is zeroed...  It is to MAINTAIN parity that already exists.  Since the parity calculation is of EVEN PARITY  (an even number of bits set to a "1" for a given bit position) Adding a new disk that is zeroed allows for no change to the parity bit.  Adding a new disk with a "zero" bit in every bit position will not change parity at all.

 

5 years ago, with 500Gig drives, they could be cleared in a few hours.  Today, with 2TB drives common, 6 to 8 hours is more normal. 

 

Even 5 years ago Lime-technology recognized that a few hours of down-time could be eliminated if a disk was pre-cleared and recognized by unRAID so it could skip the clearing step.  It would just need to format the disk, a process that only takes a minute or two.

 

So... either pre-clear the disks like most of us, or deal with the design decision of lime-tech to keep valid parity to protect your data by keeping the array off-line until a newly added drive is completely zeroed by unRAID.

 

The preclear_disk.sh script does far more than just zero the drive.  It exercises the disk in a way to attempt to identify those disks prone to an early failure.  It fully reads the disk to allow the SMART firmware on the disk to identify sectors it cannot read, then zeros the disk, then reads it once more to ensure the written zeros can be read back.  On a 2TB drive that process takes about 30 to 35 hours.  Think of it as a bun-in period for a new drive.

 

Joe L.

Joe - can you confirm that the optimal approach / best practice would be to clear the new disc on a separate machine, to avoid load / delays on the live machine?

... best practice would be to clear the new disc on a separate machine, to avoid load / delays on the live machine?

 

A disk can be precleared on the live machine before assigning the disk to the array, i.e. it doesn't really load or introduce delays on a live machine.

 

Steps (off the top of my head are):

0 - stop the array

1 - stop the machine

2 - install the new disk in the machine

3 - start the machine, which will also start unRAID. Noting that unRAID does not have any knowledge of the new disk

4 - preclear the new disk through the console, screen, etc. While the new disk is being cleared the array will still be available and useable as per normal.

5 - when preclear has completed, stop the array

6 - assign the new disk to the array

7 - start the array

 

 

Strange that preclearing / parity checking my brand new array seemed to bring the machine to its knees, but running the preclear via a script will be OK - maybe its that I had NO disks in a cleared and ready state when I was trying to use it? unRaid didnt seem to make it clear (via the webgui) whether it was zeroing or parity checking when it say there for 20+ hours (it said parity check, but the posts elsewhere seem to suggest as this was simply a new disk addition it was actually zeroing, not calculating parity).

 

Bottom line though, now have working array with reasonable performance!

Bottom line, me too!

 

I now have 11.5 TB usable.  Now I have to figure out how to use it.

 

I have a medium size video collection, around 500 videos.  On my WHS I had two 1.5 TB and one 1 TB drives dedicated to the video.  For obvious reasons one of the 1.5 TB drives is now a parity drive.  The videos are now on three drives.

 

One other 1 TB drive is being used for general storage such as photos / music and my software source disk

 

After that I have an amalgamation of disks left over from old servers.  Mostly 600G drives.  I don't really have a use for these directly, IOW I am not planning to put X on disk Y.  So what I though I might do is use one disk for backing up each of my computers. 

 

I used to use WHS, which I built specifically because of its backup prowess.  In the end the backup prowess had some serious shortcomings and just wasn't a reliable solution.  So I intend to (probably today) purchase Acronis and use it to backup every machine.  Usually this means a snapshot plus storage, and my 600g drives are far larger than any of my home system's boot disks so just pointing the backup on each PC to its own disk out on UnRaid seems like a way to use these smallish old drives.

 

jwcolby