hi gang, have been doing a ton of reading and i'm almost ready for my first build

 

for a bit of background, i have an existing pc with about 10 hard drives most internal, but some usb external.  all of these hard drives are completely full of data.  i'd like to build a 20 or 24 bay beast (i realize that more than 20 is currently not supported).  i'd like to migrate these drives into the beast!

 

i'm thinking of building the server, with two brand new drives, then copy my data over, then add the old drives to array, etc, and i guess keep copying old drives data from my pc into the array, then add the old now-copied drives over, etc.

 

i need to ensure i set aside the proper amount of time for this project!  i'm guessing with all the pre-clearing, formatting, data copying, physically adding drives, pre-clearing more copying etc as well as building parity that this could take weeks?  maybe a month or more to copy over data from 10 drives which are approx 20tb?

 

maybe i'm missing something obvious but i assume i need to build the rig, then copy the data from each drive one at a time, install, preclear etc. 

 

should i wait till the very end ie when all drives are installed before building a parity drive, or should i do that after each drive is added to the array?

 

sorry to sound a bit confused but i'm a bit nervous about disrupting my existing setup of movies etc for a long time!  not trying to complain, because i think the benefits of having an unraid setup are worth it, however i want to make sure my approach is sound, and that i'm not missing anything too obvious.  lastly, i'm looking for a bit of an estimate of the time involved from starting from scratch, to having a running server with the 20tb or so of data copied over

 

hi gang, have been doing a ton of reading and i'm almost ready for my first build

 

for a bit of background, i have an existing pc with about 10 hard drives most internal, but some usb external.  all of these hard drives are completely full of data.  i'd like to build a 20 or 24 bay beast (i realize that more than 20 is currently not supported).  i'd like to migrate these drives into the beast!

 

i'm thinking of building the server, with two brand new drives, then copy my data over, then add the old drives to array, etc, and i guess keep copying old drives data from my pc into the array, then add the old now-copied drives over, etc.

 

i need to ensure i set aside the proper amount of time for this project!  i'm guessing with all the pre-clearing, formatting, data copying, physically adding drives, pre-clearing more copying etc as well as building parity that this could take weeks?  maybe a month or more to copy over data from 10 drives which are approx 20tb?

 

maybe i'm missing something obvious but i assume i need to build the rig, then copy the data from each drive one at a time, install, preclear etc. 

 

should i wait till the very end ie when all drives are installed before building a parity drive, or should i do that after each drive is added to the array?

 

sorry to sound a bit confused but i'm a bit nervous about disrupting my existing setup of movies etc for a long time!  not trying to complain, because i think the benefits of having an unraid setup are worth it, however i want to make sure my approach is sound, and that i'm not missing anything too obvious.  lastly, i'm looking for a bit of an estimate of the time involved from starting from scratch, to having a running server with the 20tb or so of data copied over

I recommend installing parity from the beginning since it sounds as if you will be re-using the existing drives.  (in other words, you have no other back up if something were to go wrong)

 

Joe L.

ok thanks.  i guess i was thinking that i've been living without backup for a long time, that the extra time while building up the server won't matter much.  but i think you're right

 

so as for time estimate- how long would it take to build a parity drive for say a single 2 tb drive?  (then i can estimate how long it will take for my total of 10 drives)

 

also, should i be copying each drive from my pc, to the server over ethernet?  or is there a better way ie installing the drive directly into the server, then copying data from the drive into the array, then formatting?

 

ok thanks.  i guess i was thinking that i've been living without backup for a long time, that the extra time while building up the server won't matter much.  but i think you're right

 

so as for time estimate- how long would it take to build a parity drive for say a single 2 tb drive?  (then i can estimate how long it will take for my total of 10 drives)

 

also, should i be copying each drive from my pc, to the server over ethernet?  or is there a better way ie installing the drive directly into the server, then copying data from the drive into the array, then formatting?

Least risky is to just do it over the LAN.

 

Parity calcs on modern hardware are around 85-90MB/s on outer cylinders of disks, and 60-75MB/s on inner cylinders.  If you were able to get 100MB/s it would take 10 seconds per Gigabyte, or 6 Gigabyte per minute.

 

You have 2000Gb in a 2TB drive.  I figure you can do the math. 2000 / 6 = number of minutes, at best.    If you have a bottleneck on the motherboard's bus, it will be slower.

 

You only need to calc parity once though.  Foe all the remaining transfers of files, parity is calculated while you are writing the files.

 

I'd get a copy of teracopy, install it on your PC, schedule a disk to be copied, go to sleep, wake in the morning, shave, shower, rinse, lather, repeat for the next disk.

I would start the server with a 2T or 3T parity and a single 2T or 3T data drive. Set the array and build the parity. Then, copy data off one of your existing drives to the server, run the preclear on that data drive and then add it to the server. You can run the preclear without the pre and post reads to be faster. The reason for the preclear is that the server will otherwise clear the drive itself and the array is offline while this clearing occurs. It should take something like 15 hours to transfer 2T to the protected array assuming decent hardware which gets around 35MB/s to 40MB/s writing to the array.

 

so as for time estimate- how long would it take to build a parity drive for say a single 2 tb drive?  (then i can estimate how long it will take for my total of 10 drives)

 

Probably about 8 to 10 hours. The number of drives won't much matter if they're all on good fast SATA ports since you always write 2T of data when building parity on a 2T drive.

 

Peter

ok thanks to both for your replies... i guess building the parity drive won't be the longest part, sounds like the copying will be...

 

how long (roughly) does the preclear process take?  i thought i've read that people run a preclear two or three times on a single drive... can this be done automatically?

I would go ahead and try and buy a few extra disks up front.

not to mention it sounds like you will need the space real soon.

start maybe with 3-4 drives. that way you can be a little more set and forget and more of a safety net to fall back on.

 

either way, i would recommend you have parity from the very start!!! i have seen to many stories of something going wrong and  lot of lost data..

 

a preclear on a 3TB drive is about 30-36 hours if i remember correctly... this also varies with your build. CPU/RAM etc.

 

if i had to guess on total time..

if you only have 2 drives to start with.. i would guess 3-5 days per drive with copy.. verify.. preclear of old drive... (here is where a spare drive or 2 would help. you could preclear and copy at the time)

If you have been using the drive for a while and you have a history with it and feel it's good then I would just check the SMART data and it looks OK I would run the preclear without the pre-read and post read. It will take about 8 hours. You could also run it with the pre-read and the clearing without the post read which would take about 16 hours. Check the SMART data again after. If you want to exercise the disk then run one full preclear pass which is about 24 hours on a 2T disk. I personally would not do multiple passes on a disk I have a good usage history with.

 

So, 15 hours to move 2T of data plus either 8 to 24 hours to preclear tells you the time it will take to migrate each disk.

 

If you take John's advice and buy 3 or 4 disks then you can be preclearing the first disk you copied over while copying over the next disk of data. So, you might do a disk each day.

 

Peter

 

thanks for all the helpful replies... i think i'm confused about something (maybe many things!!!)

 

does the preclear run on the drive while its in the array, or while its in the pc?

 

like when you say i can copy one drive while preclearing another, am i preclearing a drive on the pc, or one in the server?

thanks for all the helpful replies... i think i'm confused about something (maybe many things!!!)

 

does the preclear run on the drive while its in the array, or while its in the pc?

 

like when you say i can copy one drive while preclearing another, am i preclearing a drive on the pc, or one in the server?

preclear runs on a drive AFTER you physically install it in your unRAID server, but BEFORE you assign it to your array.

You can be copying to drives already assigned in the array while pre-clearing other drives not yet assigned to the array.

While lionelhutz advice is sound,

 

personally, I would do a single pass on you're existing drives with the pre and  post reads just to help alert you of any pending defects.

 

I say this because you are deleting your backup on the fly. if you run into disk trouble, there is no going back.

 

in my opinion, the extra 16 hours is worth it in this case. especially if you are clearing 1 drive and copying to a second drive at the same time. there will be little down time and you can afford the extra preclear time.

 

plus, we don't know how much abuse those drives have seen.. they might be close to end of life already.

 

if you get errors on your existing drives after a single pass, you might consider more passes or removing them from service. I'd post here for advice in that situation.

thanks again for all the replies.  despite taking longer, i think i better do the thorough check as my drives all range in age from 6 months, up to a few being almost 3 years.  i guess since these are 'long running' jobs, its not like i need to be seated at the computer... i assume for the copying, and the preclearing, you just set it and let it do its thing?  i assume there's some kind of notification of when the jobs are complete (or if there are errors)?

 

also, i intend for my rig to be headless but i assume i need to connect a monitor and keyboard at least for the setup, does that sound correct?

 

PS- great to know this is such a friendly and helpful community!  we need a 'thanks' button!!!

 

Yes. Setup requires a keyboard and monitor to configure BIOS and possibly to determine the IP address. I suggest that you configure the router with a permanent IP address reservation for the server.

cool thanks.  i have second keyboard so that shouldn't be a problem.  also my main pc has a 3 monitor setup so i can just disable one and then use for the server!

 

i'm don't mean to stretch my initial question, but since everyone's being so helpful i hope it doesn't hurt to ask:

 

-i do have a dhcp environment so i would like to learn how to give a device a fixed address...as you say i'll need this for my server, as well my xbox gives me a real pain as everytime it boots it gets a different address so i'm constantly having to ftp to a different address (note i'm ftping because its jtagged).  not end of the world but a pain.  any guides on how to do this?

 

-lastly, where's the best place to find a parts list for a 24 bay server?  i see some builds on greenleaf but i'm not above copying someone's else's build if its working great!  also i'll be ordering from canada so any good suggestions for building in canada would be greatly appreciated!!!  i have used newegg.ca, canada computers, and ncix.com in the past

Connect to the router/DHCP server and set permanent reservations for the server and xbox.

 

Build options: http://lime-technology.com/forum/index.php?topic=7998.0

 

Shopping: http://lime-technology.com/forum/index.php?topic=12397.0

I connected a monitor and keyboard when I first built my server. I booted into the BIOS and set it to boot from USB and disabled the extras. That was the last time it's had a monitor and keyobard and it's likely been close to 3 years with the server now. I just use Putty on the PC to connect to it.

 

My router allows fixed DHCP address assignments based on the MAC address. It's pretty easy to do, just figure out what IP address the server has then in the router settings select the MAC from MAC vs IP list of connected devices and give it a fixed IP address. That way, the router does it my connected devices just use DHCP.

 

I used to like NCIX because they had an easy price match system and I'd save the 8% Ontario tax since they didn't have a storefront here. Now, I just like their easy price match they have. Use shopbot.ca to search for parts and at least you can price match at another place that has everything you need.

 

Peter