preclear.sh

[starcat]

Yes, I had to snapshot that window first :-) typed Yes afterwards. Any word about those GNU/MBR types for the partition table itself?

[starcat]

Hmm... looks pretty slow. The disk does something but elapsed time updates infrequently and I am still at 0% complete... 31.256.064.000 from 1.500.301.910.016 in 16mins, means it will need almost 13 hours to complete the pre-read process. And there are the write and the post-read processes, so at least 1.5 days to complete, right. This is probably the slow USB I've got.

[Joe L.]

Hmm... looks pretty slow. The disk does something but elapsed time updates infrequently and I am still at 0% complete.

For a 1.5TB disk, connected via normal SATA, it would  probably be over 12 hours to clear the disk... be patient... it could take longer over USB.

 

It should display a periodic estimate of the speed if I remember correctly.  At what speed are you reading the disk?

 

Joe L.

[starcat]

It says about 32.7 MB/s. 49.3GB finished in 25min.

 

I have another PC laying around with hotswap SATA carriers, however I have not managed it to boot from the unRAID USB stick. It gives me "Boot error" on the black screen and wont boot at all. Nothing more. The same stick that boots on another Windows laptop and also within Parallels on a Mac.

[prostuff1]

It says about 32.7 MB/s. 49.3GB in 25min.

 

I have another PC laying around with hotswap SATA carriers, however I have not managed it to boot from the unRAID USB stick. It gives me "Boot error" on the black screen and wont boot at all. Nothing more. The same stick that boots on another Windows laptop and also within Parallels on a Mac.

 

You may be able to get it to boot if you use Plop Boot Manager.  You should be able to find it in the Topical Index.

[starcat]

Looking at the preclear script it seems that the partition scheme on the disk must be MBR.

[starcat]

It says about 32.7 MB/s. 49.3GB in 25min.

 

I have another PC laying around with hotswap SATA carriers, however I have not managed it to boot from the unRAID USB stick. It gives me "Boot error" on the black screen and wont boot at all. Nothing more. The same stick that boots on another Windows laptop and also within Parallels on a Mac.

 

Got the other PC to understand to boot from the unRAID stick and now I preclear read with 119MB/s. This is 3 times more. Never liked slow performance...

 

One question to the experts. Can someone post the format command (in unRAID latest beta) with details to have the drive formatted after preclearing finishes? Thanks!

[limetech]

One question to the experts. Can someone post the format command (in unRAID latest beta) with details to have the drive formatted after preclearing finishes? Thanks!

 

Formatting after pre-clear defeats the purpose of pre-clear.  When you plug a 'pre-cleared' drive into an existing array with correct parity, you are telling the array that the new drive has all-zeros over it's entire surface (except for the partition table).  In this case, we can add the drive to the array without affecting parity (since all-zeros will not change parity).  Once the drive has been added to the array, then you can 'format' it with a file system (reiserfs).

[starcat]

Thanks much, now I understand!

 

I thought I would be able to pre-format the drive, fillup with data, then later insert it into an array and have the data intact. I thought I run then the parity calculation only. My initial idea was to preclear and format all drives (still waiting for the unRAID server hardware to appear), create the folder structure, populate with data and then insert into the array, calculate the parity and go online. It says everywhere that single disks from an unRAID array maybe read on other boxes after failure for instance. Guess, this make me think it goes also the other way round. Sorry, for wasting bandwidth and thanks much for the clarification.

[Joe L.]

Thanks much, now I understand!

 

I thought I would be able to pre-format the drive, fillup with data, then later insert it into an array and have the data intact. I thought I run then the parity calculation only. My initial idea was to preclear and format all drives (still waiting for the unRAID server hardware to appear), create the folder structure, populate with data and then insert into the array, calculate the parity and go online. It says everywhere that single disks from an unRAID array maybe read on other boxes after failure for instance. Guess, this make me think it goes also the other way round. Sorry, for wasting bandwidth and thanks much for the clarification.

If you do not have a parity drive assigned to your array you might be able to do exactly as you described.  you would want to use unRAID to format your disks.

 

Just boot unRAID, do not assign a parity disk, then assign the pre-cleared data disk, then start the array and a "Format" button will be present.  PRess it and it will create a file-system on the un-formatted disks.

 

Then, you can populate the disks as you desire.  Later, when in the actual hardware, and assign a parity drive, parity will be calculated on all the data drives. 

[starcat]

Yeah, without a parity drive, can I populate them drive by drive in my other PC (booted unRAID from the stick with network connection and mounting them individually) and then "combine" them again in the real unRAID server. This looks like to be possible. I stoped pre-clearing the other drive, but I will start it again. Thanks for the clarification!

 

I will put one pre-cleared drive in unRAID, create an array, format, mount and populate with data. When it fills up, stop the array, stop unRAID, pul out the first drive, insert the second pre-cleared drive, put in an array, format, mount, populate with data over the network, when it fills up, dismount, stop array, stop unRAID, take drive out... Then when the real server is up and running I will put those drives in there, put some more empty and pre-cleared drives, create an array with them and having the parity rebuild at once for all drives. Does this make sense or is this something I should avoid doing?

 

 

[limetech]

Yeah, without a parity drive, can I populate them drive by drive in my other PC (booted unRAID from the stick with network connection and mounting them individually) and then "combine" them again in the real unRAID server. This looks like to be possible. I stoped pre-clearing the other drive, but I will start it again. Thanks for the clarification!

 

I will put one pre-cleared drive in unRAID, create an array, format, mount and populate with data. When it fills up, stop the array, stop unRAID, pul out the first drive, insert the second pre-cleared drive, put in an array, format, mount, populate with data over the network, when it fills up, dismount, stop array, stop unRAID, take drive out... Then when the real server is up and running I will put those drives in there, put some more empty and pre-cleared drives, create an array with them and having the parity rebuild at once for all drives. Does this make sense or is this something I should avoid doing?

 

You do not need to pre-clear in this case.

[Joe L.]

Yeah, without a parity drive, can I populate them drive by drive in my other PC (booted unRAID from the stick with network connection and mounting them individually) and then "combine" them again in the real unRAID server. This looks like to be possible. I stoped pre-clearing the other drive, but I will start it again. Thanks for the clarification!

 

I will put one pre-cleared drive in unRAID, create an array, format, mount and populate with data. When it fills up, stop the array, stop unRAID, pul out the first drive, insert the second pre-cleared drive, put in an array, format, mount, populate with data over the network, when it fills up, dismount, stop array, stop unRAID, take drive out... Then when the real server is up and running I will put those drives in there, put some more empty and pre-cleared drives, create an array with them and having the parity rebuild at once for all drives. Does this make sense or is this something I should avoid doing?

 

You do not need to pre-clear in this case.

Tom is correct, if you have not assigned a parity drive you do not need to pre-clear to prevent unRAID from performing a clear on it own,

but ... if you have the time, I would highly recommend you perform the pre-clear process anyway.  It will weed out any drive prone to an early failure and alert you to any major problem before you put your data onto it.   For that reason alone I think the consensus is to always run at least one cycle of the pre-clear script and better yet, two or three cycles on every drive before you add it to the array.

 

Tom himself sells that are "burned in for 48 hours or more"  In his "burn-in" and pre-clear process he is able to weed out any drives with errors in their first few days of operation.

 

So, you do not need to use the pre-clear process if you have not yet assigned a parity drive, but it sure does not hurt to exercise your drives while it is still easy to swap then out and RMA it if needed because they are NOT part of your array holding your data.

 

One last comment, before you start re-using the old drive that had your data before transferring it to unRAID's disks, add a parity drive to unRAID and get the protection it offers.   We've seen a disk fail in the first week or so...  It would not be good if it held the only copy of your data, and parity was not available to re-construct it.

 

Joe L.

[limetech]

What Joe said 

[starcat]

Guys, thank you very much! I am preclearing now two new drives from within separate telnet sessions and will add them to the array and format them. As you have suggested.

 

I have also found this thread, for anyone interested, describing a similar procedure. Thanks Joe, thanks Tom!

http://lime-technology.com/forum/index.php?topic=4017

[starcat]

Stupid, by I accidentally closed one of the telnet windows running preclear on the one of the two disks... did I killed the preclear process and if not, is there a way to see when the process finishes without having the telnet window anymore?

 

Don't see the

/bin/bash ./preclear_disk.sh /dev/sdb

process anymore in the process tree... looks bad I think and have to rerun it?

[Joe L.]

Stupid, by I accidentally closed one of the telnet windows running preclear on the one of the two disks... did I killed the preclear process and if not, is there a way to see when the process finishes without having the telnet window open?

It terminated when you closed the window.   You'll need to re-start it from the beginning...

 

Instead of using telnet, you can start them using the system console... Use Alt-F1 through Alt-F6 to switch between 6 virtual consoles. You can get up to 6 running at the same time that way.

 

Or, you can download and install a program named "screen" that allows you to detach and re-attach to a running process through telnet

 

See this post: http://lime-technology.com/forum/index.php?topic=72.msg27984#msg27984

The links in that post are no longer valid, use these instead:

http://slackware.cs.utah.edu/pub/slackware/slackware-12.0/slackware/ap/screen-4.0.3-i486-1.tgz

http://slackware.cs.utah.edu/pub/slackware/slackware-12.0/slackware/a/utempter-1.1.4-i486-1.tgz

 

Download both to your flash drive to a folder named "packages"

Install with

cd /boot/packages

installpkg utempter-1.1.4-i486-1.tgz

installpkg screen-4.0.3-i486-1.tgz

 

You'll need to re-install each time you reboot, but it is not too difficult if they are already downloaded in your /boot/packages folder.

Or, you can just not log off or close the telnet window...

 

Joe L

[starcat]

Thanks, Joe! Isn't it possible to specify an alternate root for installation when I run installpkg, i.e. install them to the mounted flash drive instead of the running OS in RAM?

[Joe L.]

Thanks, Joe! Isn't it possibel to specify an alternate root for installation when I run installpkg, i.e. install them to the mounted flash drive instead of the running OS in RAM?

installpkg does allow you to specify an alternate "root" but I've never tried it... installing to RAM is automated on my server, so I really have never experimented with the alternate-root-install feature.

[starcat]

Will try it. I really like to have all packages installed to the USB drive, then reboot. On the other hand it is not needed to reboot if you install to RAM. Where do you put the installpkg commands in your bootup scripts in order to "automate" installing (end of the "go" script?), and what packages do you typically auto-install on boot? Thanks!

[Joe L.]

Will try it. I really like to have all packages installed to the USB drive, then reboot. On the other hand it is not needed to reboot if you install to RAM. Where do you put the installpkg commands in your bootup scripts in order to "automate" installing (end of the "go" script?), and what packages do you typically auto-install on boot? Thanks!

I use the unMENU package manager to auto-reinstall on re-boot.

 

Basically, it can be as simple as one line in your "go" script.    It will then loop and install the packages.  (I use a more complicated scheme that implements a /boot/custom/etc/rc.d structure similar to the one used by Slackware when it boots)

 

In its simplest form, it is this one line added to the end of the "go" script. 

cd /boot/packages && find . -name '*.auto_install' -type f -print | sort | xargs -n1 sh -c

 

It will invoke each file named

*.auto_install

that exists in the /boot/packages directory. 

 

In my case, as of this moment, I have these:

root@Tower:/boot/custom/bin# ls -l /boot/packages/*.auto_install

-rwxrwxrwx 1 root root 569 Sep 28 22:35 /boot/packages/apcupsd-3.14.3-i486-1kjz.tgz.auto_install*

-rwxrwxrwx 1 root root  72 Sep 22 21:12 /boot/packages/bwm-ng-0.6-i486-2bj.tgz.auto_install*

-rwxrwxrwx 1 root root  32 Sep 22 10:06 /boot/packages/file-4.21-i486-1.tgz.auto_install*

-rwxrwxrwx 1 root root  83 Nov 17 19:47 /boot/packages/htop-0.8.3-i486-1sl.tgz.auto_install*

-rwxrwxrwx 1 root root  84 Nov 16 17:45 /boot/packages/iftop-0.17-i486-1kan.tgz.auto_install*

-rwxrwxrwx 1 root root  72 Sep 22 18:18 /boot/packages/infozip-5.52-i486-2.tgz.auto_install*

-rwxrwxrwx 1 root root  70 Nov 18 22:19 /boot/packages/libxml2-2.6.31-i486-2.tgz.auto_install*

-rwxrwxrwx 1 root root  69 Sep 22 18:16 /boot/packages/lsof-4.78-i486-1.tgz.auto_install*

-rwxrwxrwx 1 root root 283 Sep 22 18:16 /boot/packages/ntfs-3g-2009.3.8-i486-1.tgz.auto_install*

-rwxrwxrwx 1 root root  75 Sep 22 18:17 /boot/packages/pciutils-2.2.10-i486-2.tgz.auto_install*

-rwxrwxrwx 1 root root 562 Nov 18 22:20 /boot/packages/php-5.2.8-i486-1.tgz.auto_install*

-rwxrwxrwx 1 root root 111 Sep 22 18:17 /boot/packages/powerdown-1.02-noarch-unRAID.tgz.auto_install*

-rwxrwxrwx 1 root root 160 Oct 11 00:46 /boot/packages/ssmtp-2.61-i486-1suk.tgz.auto_install*

Each .auto_install has the few commands needed for that package.

For example, the contents of  /boot/packages/ssmtp-2.61-i486-1suk.tgz.auto_install is:

root@Tower:/boot/custom/bin# cat /boot/packages/ssmtp-2.61-i486-1suk.tgz.auto_install

installpkg ssmtp-2.61-i486-1suk.tgz

installpkg mailx-12.3-i486-1.tgz

ln -s /usr/sbin/ssmtp /usr/sbin/sendmail

cp /boot/custom/ssmtp.conf /etc/ssmtp/ssmtp.conf

 

In that one .auto_install file I install the two packages needed for mail, link the installed command to where "sendmail" is expected to live, and then copy its configuration file from the flash drive to where it is expected to reside under /etc/ssmtp folder.  The .auto_install is just a shell script with a common suffix that can be found/executed in turn by the one line I added to the "go" script.

 

For the "screen" command, a "/boot/packages/screen.auto_install" file would just contain:

installpkg utempter-1.1.4-i486-1.tgz

installpkg screen-4.0.3-i486-1.tgz

 

and would expect the two package files to be in the packages folder located at the root of your flash drive.

 

Joe L.

 

[starcat]

Joe, thanks, much appreciated! Just implementing the *.auto_install. Also, looking for the other packages like mail, apc, etc. seems that I would need them too.

[prostuff1]

Joe, thanks, much appreciated! Just implementing the *.auto_install. Also, looking for the other packages like mail, apc, etc. seems that I would need them too.

 

Take a look at the Third party Boot Flash Plugin Architecture entry in the wiki.  A lot of fourm members that write these scripts and addons will and do take advantage of it, if it exists.  Instead of cluttering my go script I use this structure along with a line in my go script that reads:

fromdos < /boot/custom/etc/rc.d/rc.local_startup | sh

 

the rc.local_startup script looks like this:

#!/bin/bash
logger -trc.local_startup -plocal7.info -is "Initiating Local Custom Startup."
scripts=`ls /boot/custom/etc/rc.d/ | egrep -v "rc.local_startup|rc.local_shutdown"`
cd /boot/custom/etc/rc.d
for script in $scripts
do scriptbase=${script##*/}      # Strip pathname
   ( echo "Processing $script" 
fromdos < $script | sh -xv
   ) 2>&1 | logger -t$scriptbase -plocal7.info -is
done

 

this basically goes through and runs all the scripts that are in that folder and installed the required software that way.  I made a post in another thread (can't find it right this second) but it had attached all of my /boot/custom related stuff.

[Joe L.]

Joe, thanks, much appreciated! Just implementing the *.auto_install. Also, looking for the other packages like mail, apc, etc. seems that I would need them too.

Download and install the unMENU add-on.  Its package manager is the one that actually creates the .auto_install files and it has the ability to download the packages and initially install them.  (You can also create an .auto_install file by hand, as I did with the ssmtp package I installed for mail.)

 

You will want to install the powerdown package at the very least, and possibly one or more of the others.  The APC UPS package makes it trivial to get an APC UPS running.  Basically, just plut it into the USB port and it will use it.  The whole scheme is one I devised using unmenu package.conf files.  They have all the data needed to create .auto_install files and to download the files themselves.

[starcat]

Thanks guys, will investigate the addons. The files can be nicely downloaded directly on the unRAID box in /boot/packages as there is wget installaed by default.

 

In the meantime I have switched from smb to nfs mouting (the clients directly on unRAID) and got a performance boost from 27MB/s to about 70MB/s. Not bad at all!

 

PS: will a serial based APC UPS like the SmartUPS 700 or 1000 work with the apcupsd package?