New Setup Questions

[richgnoe]

I am getting ready to setup my unraid server with 20 3TB Sata DISKS. I have a couple of Questions

 

Do I have to run preclear on each of the disks before setup of array ? If so how long does it take to do a 3tb disk?

Once I get the array setup I have about 12 TB of data to migrate over to the new server, most of it movies. Is there a best way to move the data? I heard it was best not to set up the parity disk until all of the data is transferred.

[boof]

Do I have to run preclear on each of the disks before setup of array ? If so how long does it take to do a 3tb disk?

 

You don't *have* to but it avoids unraid having to zero them for you at the point in time you add them to the array (it will do this when you add them via the web interface). One by one.

 

Preclear will let you do it in the background, disks in parallel and give you stats and information on any problems discovered with your drives.

 

So whilst you don't have to - I would.

 

Presuming you can do ~100 megs a second from one of your disks (your system depending, and if you're going to be doing disks in parallel) it will take around 9 hours for a single pass. You probably want to do multiple passes.

 

Once I get the array setup I have about 12 TB of data to migrate over to the new server, most of it movies. Is there a best way to move the data? I heard it was best not to set up the parity disk until all of the data is transferred.

 

Where are you moving it from? And what do you categorise as 'best'? Most secure? Or quickest?

 

Most secure - use a program that will checksum the data at either end to ensure the copy is ok. rsync, teracopy - something like that.

 

Speed - if you can put the disks with the data you already have in unraid and copy direct onto your new array that will be the fastest.

 

Leaving the parity disk until the data is transferred will increase the speed of the copy (as it doesn't need to compute the parity in real time) but will leave the data unprotected. I would suggest if you want to do this that you don't move the data - just copy it. Once the copy is complete then set the parity building.

 

All the while you still have the original source data on your current platform in case of mishap.

 

Hope this helps.

[vl1969]

I am getting ready to setup my unraid server with 20 3TB Sata DISKS. I have a couple of Questions

 

Do I have to run preclear on each of the disks before setup of array ? If so how long does it take to do a 3tb disk?

Once I get the array setup I have about 12 TB of data to migrate over to the new server, most of it movies. Is there a best way to move the data? I heard it was best not to set up the parity disk until all of the data is transferred.

 

let me first say wow  60TB I am impressed :-)

 

#1.  yes you need to preclear all of it, but you do not have to do it all at the same time.

you can use Screen Plug-in  to run preclear on several disks at once.

not sure if anything changed for RC16 but on 5.0Beta13 the easier way to setup screen was to install unMenu and use that to install plugins.

 

here is the wiki link "http://lime-technology.com/wiki/index.php/Configuration_Tutorial#Preclearing_With_Screen"

 

SREEN helps as it allows you to create multipler sessions and run  preclear in them without keeping the main terminal open, so if you do this remotely via telnet/SSH 

you can start the process, close the terminal and check on it later by login in and view the screen session at will.

 

 

it is recomended to do at least 3 passes  to ensure disks are good.

not sure exactly how long it will take for 3TB discs but it can be at least 3 days per disk

so doing multiple disks at once is a good idea.

when I started it took me almost 2 days per disk for 2TB.

and I have cleared 2 of them one at a time as i did not know about screen at first.

 

I say do the first 4 , this will give you either 16TB array OR 12TB array + parity. to start with.

than you can start the array and begin copying your data as you preclear the next batch.

 

#2 it is obviously your choice if you want to start without the parity, but

I personally do not see the benefits of it.

it might be faster to transfer data, sure, but than your data is sitting there with no protection until the parity is enabled and built.

and the time you save on data transfer is than lost while the parity is building.

 

Personally I went with full array + parity from the get go. speed was never a concern or an issue. even though I did my data move over 10/100 network, using  windows explorer.  kept my main PC running for several days as I sift through files and put them in proper folders/shares.

 

Is your data properly  configured? I mean are your folder structure in good form

if not take your time and think what exactly you want to end up with.

I did this mistake at first, just dump the data onto unraid and than had to redo things the hard way.

is all of your data in good working format.

I have recently re-encode  2TB worth of movies into MKV format right on the array.

 

if all you want is straight data copy.

install S.N.A.P plug-in , than you can hook-up your HDD directly to unraid ,mount it with SNAP and copy data drive to drive directly(if you are good with linux CLI  that is.)

is would be fast enough.

 

if you are in Windows camp,

I can also recommend  going a bit back in time and getting Total Commander ("http://www.ghisler.com/download.htm") to use instead windows Explorer . this is a port of popular DOS era file manager that is still very popular and very powerful.

it is some what free, as  you can use it indefinitely but with nag screen at start up.

but it has a copy que feature. thus you can select files you want to move/copy

added to the que and it will tell you if there is any issue, and it will restart the copy/move if you tell it too. or if something happens.

 

where is your data now?

 

 

[garycase]

Pre-clearing is a good test of the drives, but it is NOT required.    Nor does UnRAID need to clear the disks IF you add them all initially.  A new disk being added to a parity protected array requires clearing;  but the initial disks (BEFORE parity is assigned) do not ... they only need to be formatted (a fairly quick process -- a few minutes/disk as opposed to ~ 30 hours or so per disk to run the pre-clear script for 3TB drives.

 

It's not "best" to set up parity after all your data is transferred;  but it's certainly FASTER to transfer your data before you assign parity.  The downside, of course, is that the array is not fault-tolerant (i.e. protected) until you assign parity ... so if you choose to transfer all your data before you assign parity, any failure during that process will result in data loss.

 

[JonathanM]

I am getting ready to setup my unraid server with 20 3TB Sata DISKS.

Once I get the array setup I have about 12 TB of data to migrate over to the new server,

I have a suggestion. Instead of fully populating the array with all 20 disks, why not incrementally add disks as the need arises? Unraid isn't like a traditional array where you really need to have things thought out in advance capacity wise, it's very friendly to adding new members to the array as you go.

 

If I were setting up an unraid server given what you've told us here, I'd start with 7-10 disks, depending on how the directories were organized, and set the rest of the disks to spin down manually when you start the server. All 20 disks would be physically installed and accessible, but only the needed ones would be members of the unraid array, the rest would be warm spares. They won't automatically spin down if they aren't members of unraid, but it's pretty trivial to manually spin them down after the machine is booted.

 

The chances of simultaneous multiple disk failure is pretty high if all the drives were purchased from the same place at the same time, and used equally. I'd hedge my bets and keep some drives unused for use as replacements.

[mobias1313]

You can run as many preclear passes of the disks as you want. I personally run 2, some people run 1 or 3. It is solely up to you. Every failure I have had while doing a preclear has been in the first pass. For me to run 2 passes on a single 3tb WD Red it takes a solid two days. I think the total time is actually more than 48 hours. I am not in a hurry to add disks so I just set it and check it periodically. I totally agree with jonathanm, if you have 20 disks and you will not be using all of them right away, just add in enough to cover your parity and all of your current data with maybe one or two extra drives. Then preclear the other drives at your leisure and leave them as warm spares to be added in if a drive fails or if you need additional space.

 

As for transferring your files over, depending on the method you want to use and where you data is currently located you could use SNAP or Mount Disks to install the drives outside the array and copy the data over locally, this is the fastest way I have found, or you could do it over the network which may take longer, depending on your network configuration. Personally I have used both. When transferring over the network I use TeraCopy which lets me pause and resume as well as not fail the entire transfer for one problem file.

[boof]

Nor does UnRAID need to clear the disks IF you add them all initially.  A new disk being added to a parity protected array requires clearing;  but the initial disks (BEFORE parity is assigned) do not ... they only need to be formatted

 

I *always* forget this! So long since I've had a completely empty array. Thanks for correcting me.