co-lo an unraid server

[Jim Beam]

Would it make sense to co-lo an unraid server?

 

I am considering how to move large amounts of data from download point in a Data Center to a remote home location with extremely poor Internet speeds. It's just impossible to move TB's of data from the DC to home with my Internet connection and there is no way to improve the connection.

 

The normal way to go about this for people who live in the 1st world with REAL connections (100+Mbps)  is to download files to a server in the DC then ftp the files from the DC to home location. But because of my very very low BW speeds (think dial up) this does not work for me.

 

So given that particular scenario - i am wondering if it would make sense to have an unraid server built up for the purpose of downloading my files to the unraid server in a co-lo then having the HDD's pulled from the server when full and sent to me for install to an unraid server at home? Is it possible to move HDD's from one unraid to another like that? If not that, then move files on an unraid server to a particular drive/s on the server and have that/those HDD('s) sent to me to put into an unraid server at home.

 

I dont know how practical it would be to try to use unraid this way - i've not used unraid before and reading and reading at this time trying to figure it all out. So given my basic understanding of unraid at this time, is what i am considering practical?

 

Thought i would throw the question out there for those who know what they are doing with unraid while i keep chugging through the reading trying to understand all the in's and out's of unraid.

 

Thanks for any thoughts on this.

 

 

 

[trurl]

You can add drives with data on them by setting a New Config and rebuilding parity.

[garycase]

(a)  How much data are you talking about moving at one time?

 

(b)  Do you need fault tolerance for the data while it's being moved ... or only after you've completed the move?

 

©  How much of a hassle is the moving process?  [e.g. do you live conveniently enough to the data center that you'll be simply moving the disk(s) in a car?  ... do they need to be shipped?  ... and, if so, is this an "easy" shipment (think UPS/FedEx domestically) or an international shipment with customs and other regulatory issues?

 

[Jim Beam]

Volume is multi TB's of data. Would only want to ship once a quarter or so. So as many TB's as 4 -6 drives or so can hold. Using 8TB drives with around 7.5TB of usable space that would be somewhere around 45TB's or so over 6 drives. So if it did become a shipment once every 3 months - then its about 15TB's a month, about 3 to 4 TB's a week of collected data.

 

But no hard or fast rule - however it works out and i guess it depends on the server setup.

 

Its an international shipment - so DHL etc to me. There is customs and duty on the incoming HDD's but thats fine. Every shipment gets hit with around $300 of fees, so only want to do a shipment when enough data has built up that i could take 4, 5 or 6 drives out and have them shipped altogether as one incoming shipment.

 

Drives would be well packed for shipment - don't need any fault tolerance while in transit. If i lost any data it could be downloaded again worst case scenario. It's a pain to do have to do it all again if it came to that, but if i had too it's no train smash.

 

One of the issues that got me thinking along these lines is that the Seagate Archive drive is ideal for this application - you guys have me convinced of this after reading the threads here on them and the answers i got to some questions i asked about them.

 

However they do seem to have high failure rates in some circumstances. So it makes sense to buy them in the same country as the DC is in and fit them to a server and run them through the testing sequence you guys put your drives through. If any drive fails its testing it can be returned to the seller in the same country before any shipping and fees etc are incurred for that drive. So having an unraid server to mount them in and test on seems the absolute ideal way to make sure all is well with them in an unraid environment.

 

Other thoughts are that on ver 6 of unraid, virtual servers and different OS's can be run in various ways - so a control VM to remotely run and control all the functions of the server should be doable. Not really worried about all the fancy stuff like Plex in the cloud etc as i dont have the BW to enjoy all that from here behind the banana curtain.

[garycase]

You can't add a disk from an UnRAID server that already has data on it without doing a New Config -- which will unprotect (no fault tolerance) the array until you do a new parity sync.    So adding the data to an UnRAID array at the DC and then shipping the disks to your may not be the best way to attack this.

 

I would, however, use the 8TB Seagate Archive drives, as you're already planning for.

 

I'd probably set up something like this ...

 

(a)  Buy the drives at the DC (Data Center); and test them using one of the manufacturer's diagnostics -- either Seagate's SeaTools or WD's Data Lifeguard.  I prefer Data Lifeguard -- use it for all drives, regardless of manufacturer.    What I do is a Quick Test; then an Extended Test; then a full write zeroes;  and then repeat the first two tests.  If ANY errors, I get the drive replaced;  otherwise it's fine (MOST drives are fine).  You can get a SATA Dock [e.g. http://www.newegg.com/Product/Product.aspx?Item=N82E16817153071 ] to conveniently hold a drive while you're testing it.

 

(b)  Connect a drive to a system that has access to the data you need to transfer, and fill up the drive.  If the PC you use to test the drives has access to the data, this can be conveniently done with the SATA Dock.

 

©  Repeat (a) and (b) until there are enough drives ready for a shipment.

 

(d)  When you receive the drives, what you do depends on how you're going to use this data.    If you need it available on-line, and plan to keep it in an UnRAID server, that's a different process than if you simply need the data available for occasional use and don't need it always online, or in a fault-tolerant server setup.  If the latter, you can simply store the drives in a good storage box, and access them via a SATA Dock like I suggested above.  If that's the case, these are excellent storage boxes (I've used them for years and have a LOT of drives safely "tucked away"):  http://www.amazon.com/DriveBox-Anti-Static-Storage-3-5-inch-10-pack/dp/B004UALLPE

 

If you need the data available online, and are planning to build up a large UnRAID server to hold as much as possible [a maxed out 24-drive UnRAID will hold about a year's data at the rate you've indicated];  then I'd build it up as follows ...

 

(1)  Build the basic server, with 24 drive slots, and populate it with 3 8TB drives ... parity and 2 data.  Do a parity sync so it's ready to use ... and now wait for a shipment of drives.

 

(2)  When you get a drive, put it in a SATA Dock attached to your client PC (or Mac); and copy all of the data from it to your UnRAID server.

 

(3)  Remove the drive from the SATA Dock;  physically add it to the UnRAID server;  do a PreClear with the -n option [This skips the pre-read and post-read phases ... only takes about 1/4th the time of a full cycle];  and then add it to the array (this will be very quick and will preserve parity).  When you Start the array it will format the new drive (also very quick).

 

(4)  Repeat #2 and #3 for each drive in the shipment.    Note this will take ~ 2 days /disk to do the copies, but only a couple minutes of "your time"

 

Done :-)    Note that at the rate you're planning to acquire data, you'll need to build a new 24-drive UnRAID server every year 

 

[BRiT]

It does not make sense to colo an unraid server considering its still touted as not having any security in mind by LimeTech and how they actively state security is not even on their project timeline. Your unraid server would be raped and pillaged in no time at all.

[BRiT]

What you want is a normal non unraid server running some flavor of linux to colo using drives that were prepared and formatted by unraid. Do not even think of putting an unraid server on the internet.

[Helmonder]

This could work fine... Unraid  drives can be transplanted in a new system easily..

 

It will depend on security though.. (added this after reading the other comments), you cannot just put the system on the open internet...

 

If you want to add an extra layer of redundancy you could set a daily job that copies from your movies share to your "movies-to-send" share. You then limit that "movies to send" share to one drive and exclude that same drive from all other user shares.

 

You then setup a notification that sends you a mail when a drive gets full. As soon as your "movies to send" disk is full you will receive a mail, this will trigger you to ask the DC to pull the drive and send it to you. They could add a new drive immediately so that will start to fill up.

 

The server in the DC will continue to hold all files..

 

At home you do the opposite, you add the disk to your unraid system, move all the files to the "movies" share. Then you start binge watching everything untill the unraid system sends you a mail again telling you the new drive is full..

[garycase]

There is an alternative that would let you add the new disks to your server MUCH quicker ... as long as you don't mind losing parity protection on your array for a day or so each time you receive a new shipment.

 

(a)  Build a small UnRAID server at the DC that can hold as many disks as you want to ship (plus one for parity).  For illustrative purposes, I'll assume it's a 6-drive array with 5 data drives and a parity drive.

 

(b)  Configure the array at the DC with the data drives and parity drive assigned.    Now they can copy data to it as it's available -- until the array runs out of space.  When the array is full and they're ready to ship you the drives, they can shut it down; and ship you the 5 data drives.    Meanwhile, they should buy 5 replacement data drives; test them; and then put them in the server and do a New Config -- assigning 5 drives as data and one as parity ... i.e. it will now be an "empty" server again. 

 

©  When you get the 5 new data drives, you can simply shut down your server;  add the new drives;  reboot; do a New Config; and assign all of your data drives again, including the 5 new ones.    Then you just Start the array and let it do a parity sync.    Clearly you have to be CERTAIN you've assigned the correct parity drive;  and of course the array will be "at risk" until the new parity sync has completed.

 

[Jim Beam]

Brit - yes i hear you - had not even thought of security - what about if the server is behind an pfsense fw? the fw can be a 1U server on its own (looks like i have to take a min of 5U of rack space - so have room for such things)

 

garycase - lots of very good info - as always - from your post. Lots of food for thought there for sure.

 

Helmonder - "binge watching"...hahahahaha , ........ yeah i guess it looks like that from how i described it.

[Jim Beam]

OK so looking further into this.......

 

would a Dell R510 12 bay server be OK for this application (co-lo unraid) ? Looking on eBAY for a used server. I want to populate the server with the archive drives and a few SSD's - i have read that one can use Dell servers without using Dell HDD's

 

Any thoughts about that idea?

 

If you dont like the R510 (why?) what would you suggest? (again - why?)