Jump to content
BetaQuasi

Orion - ESXi/unRaid build - now v6 and KVM

60 posts in this topic Last Reply

Recommended Posts

Heyas,

 

Ok this is probably a little premature - I'm still waiting on a few parts to arrive as I've had to import them from the US (I live in Australia), but wanted to share anyways and get some feedback on what people think.  I borrowed a few hardware ideas from this forum (the Supermicro kit, a la Johnm's Atlas and some others) and the rest is fairly standard.  I am particularly looking forward to testing the case/drive bays out.  The case is steel, so it will be bloody heavy when it's filled, but that's ok, not like I need to move it :)

 

This build is ESXi-centric, as I wanted to get rid of all my 'lab' boxes (a lot of my day job work is in virtualisation and storage, and my home kit pretty much consisted of a cheap and nasty Dell tower server and a few random desktops/old gaming rigs, all of which were power hungry and, frankly, a bit crap!)  A list of the parts (prices in AUD unless noted otherwise):

 

Case:  Xigmatek Elysium Super Tower - http://www.pccasegear.com/index.php?main_page=product_info&products_id=18284 $179

Drive bays:  2x Xigmatek 4 in 3 (Case comes with 2, total of 4) - http://www.pccasegear.com/index.php?main_page=product_info&products_id=18507 $17 x2

Case:  Norco 4224 with 120mm fan plate and ball bearing rails

CPU: Intel Xeon E3-1230 - $229

Motherboard: Supermicro X9SCM-F-O - $167 USD @ Superbiiz

Storage Adapter: AOC-SASLP-MV8 - $102 USD @ Superbiiz and 2x LSI SATA breakout cables $26 USD @ Superbiiz (Now a spare)

Storage Adapters: 2x IBM RaidServe M1015 flashed to IT mode

RAM: 2x KVR1333D3E9SK2/8G unbuffered ECC kits (16G total) - $122 USD @ Superbiiz

PSU:Seasonic X660 Gold$159

Flash drives: 2x Lexar Firefly $12 (one unRaid, one ESXi)

Flash drives:  2x Patriot Xporter Rage XT 8Gb

SSD1: 240Gb Corsair Force 3 $309 (Cache Disk)

SSD2: 256Gb Corsair Performance Pro (ESXi Datastore)

SSD Bracket: 2x Startech 2.5" PCI hot swap drive bays

Hard disks: 4x 2Tb Seagate 5900rpm $122 ea plus 4 existing 2Tb Seagate 5900rpm from my old Netgear ReadyNAS NV+, passthrough to unRaid VM off the SASLP

Cables: 6x SFF-8087 to SFF-8087 1m SAS cables

Fans:3x120mm and 2x80mm Arctic F PWM's

NIC: Intel PRO/1000 PT PCIe x4 Quad Port Gigabit NIC, teamed to a HP Procurve 1810G-24 in Etherchannel mode - LAN access for all VM's

Expansion Card:NEC 5 port USB 2.0 PCIe 1x card (1 internal, 4 external ports.)  Internal port for unRAID flash, external port for UPS.  Card passed through to unRAID VM.

 

Total price, minus storage: ~$1000 AUD (few parts in USD, based on current exchange rate)

 

- Parity drive will be one of the above 2Tb and recycling an old 500Gb WD green for cache.  Undecided on cache, may forgo one of the 2Tb as I don't need that much space yet (only have 5Tb to transfer from the old NAS) and put the cache drive on the SASLP, or raw mount it.

- 2 extra 4 in 3's because $17 ea is cheap and they match the case - and I don't want to fall into the trap of being unable to get them down the track when I want to add more drives

- The case is an absolute monster - fits E-ATX, L-ATX, HPATX motherboards, so with the mATX supermicro board in there, there will be a TON of space.  Also has really good cable management (check out the photos in the link above).  I am thinking if/when I hit 16 drives in the unRaid setup, I might look at some sort of custom mount to add another 4 drives in the space usually reserved for a waterblock (at the bottom).

- Case also has a SATA 3.5"/2.5" dock on top, which should be useful.

- Would have preferred 32Gb RAM, but the Kingston 8Gb chips don't make sense at the moment - $420 for 32Gb or $120 for 16Gb.. hmm.. take the 16Gb and upgrade when the prices drop!

 

Anyways, just waiting for my case and for Mr UPS man to deliver my bits from superbiiz and I'm good to go!  They're on their way, but to save on shipping, they're coming on Express Saver (still $85 odd dollars for shipping!  ugh)  Will post photos etc once it's all up and running.

 

EDIT:

 

Couple of changes, hopefully will have all parts late this week or early next.

 

- Ordered 2x M1015's preflashed with latest IT (will sell or keep the MV8 as a backup)

- Changing the USB sticks to 8Gb Patriot Xporter Rage ($14 ea)

- Changing the datastore drive to a Corsair Performance Pro 240Gb for the much better Marvell-based garbage collection ($399)

- Likely use the Force 3 as the cache drive now

- Ordered 2x StarTech PCI hotswap 2.5" mounts for the SSD's.  With 11 card slots in the Elysium and only using a Micro ATX board, should be plenty of room for them.  Unfortunately it's the one downside to the Elysium in that it doesn't have any 2.5" mounting points/cages.. but I kinda like this idea better anyway and it means I keep all my 5.25" bays for 3.5" cages. 

 

http://www.amazon.com/StarTech-com-2-5-Inch-Removable-Expansion-S25SLOTR/dp/B002MWDRD6/ref=sr_1_1?ie=UTF8&qid=1333946172&sr=8-1

 

Getting itchy feet, hopefully everything rocks up soon!

 

EDIT 2:

 

Was advised the Xigmatek wouldn't be available until 8th May and that there is a nationwide shortage of them (tried to get one everywhere with no joy.)  So after some research into Techbuy.com.au (sole Norco reseller in Aus), I've jumped on the Norco bandwagon and ordered:

 

4224

120mm fan wall bracket

1x reverse breakout cable

ball bearing rails

 

..and from Monoprice:

 

6x SFF-8087 to SFF-8087 1m SAS cables

NEC PCIe x1 5 port USB card (1 internal, 4 external) - hopefully I can pass this entire card through to the unraid VM with no issues.  Will have the unraid USB stick mounted on the internal port of the card, and the ESXi stick on the internal port on the motherboard, so all external USB ports will be available too.  Shipping was $37 as they only ship internationally via UPS... racking up the costs, but oh well.  Things we do for our hobbies :)

 

 

Share this post


Link to post

Have most of my bits now - just waiting on the X9SCM-F-O, RAM and MV8 from Superbiiz and the 1015's from the ebay person.  Hurry up! lol

 

3aQTB.jpg

 

Looks like I got one of the newer 4224's too as I ended up with yellow backplanes.  Cross fingers, hopefully no issues!

 

etj8I.jpg

Share this post


Link to post

Don' tease. Now I want to fly to aussieland and help assemble it.. LOL.

 

 

I like those startech PCI slot bays. I had consider those for a while. I think if i had spinners I would have. it is just so easy to let the SSD's lay in the case for now for my build.

Share this post


Link to post

Haha maybe you can pick up my superbiiz gear on the way and get it here faster!

 

On another note, there is a 101 page thread over at overclockers.com.au with a lot of other guys that bought this case and went nuts with it (they're a little quirky over there!.)  One of the mods that I'm considering though is this one:

 

http://www.overclockers.com.au/pix/index.php?page=image&id=y2360

http://www.overclockers.com.au/pix/index.php?page=image&id=vh54x

 

He's used screws to mount them and they are hard up against the back of the backplanes - as far as I can see, even with the new revision of these cases, the reason why drives sometimes require that extra "push" to connect after locking the drawer is that the backplane itself actually moves back a small amount when you're pushing them in.  Apparently a mod like this resolved it and I know my local Bunnings has aluminium strips like this for next to no $$.  Hmmm.... ;)

Share this post


Link to post

Box is built, preclears are finished on the first four disks, they're currently formatting.  It's still in one of the labs at work for the moment, will be bringing it home tomorrow night to start the transfers off my existing NAS's, and then the 2TB drives in my 2x NV+ will be coming out and into the unRAID array too.  Up for a big weekend! lol

 

Also scored a quad port Intel PCIe NIC that's going in there - I'll probably trunk that up to my 24 port Gigabit switch (HP Procurve) and have 4Gb going into the box.  Could be interesting, writes to the SSD cache drive should be as fast as humanly possible!  Thinking of setting up the cache drive as a .vmdk on the 2nd SSD rather than raw mapping one, so I can use some space on it for cache (I don't transfer any more than say 20Gb a day) and the rest for storage/more VM's.

 

 

Share this post


Link to post

re: cache disk - using a .vmdk with a SCSI i/o port in ESXi doesn't work.  Adding it to the array just refreshes the page and the cache disk slot goes back to empty.  Changing the .vmdk to an IDE port worked, but writes were slow... slower than 100Base-T!  (This .vmdk was on my 2nd SSD which was doing nothing other than storing some .ISO images at this stage.)

 

So I ended up moving the .ISO images and just RDM'ing the SSD.  Fastest write so far has been around the 130MB/s level!  (bursting up to 160MB/s).  Fun times! 

 

Updating the first post to reflect the final build - will post some pictures when I can convince myself to stop tinkering! lol

 

VM's currently running:

 

pfSense - WAN port on 82574L off the motherboard running to my ADSL2+ modem, LAN port on the quad port Intel.

unRAID - 12a, with NEC USB and 2xm1015 passed through

Win7 - download box, Sab/Sick/Couch etc

Ubuntu 11.10 - low spec MySQL box

 

Will be migrating my lab boxes over at some stage, they don't get powered up much.  A 2008 R2 Domain controller, a 2008 R2/exchange 2010 server and another 'whatever' 2008 R2 box.  Also a Win8 consumer preview VM and a Server 8 beta VM.

 

A shot of the motherboard etc prior to installing.  I should have got more install shots, but was too damn keen to get it all up and running!

 

i22sP.jpg

Share this post


Link to post

Nice.

 

I love those before and after shots.. it is like porn for those with UCD...

Share this post


Link to post

When I received all my stuff, I wasn't able to think of making pictures... I just start unpacking, drooling and mounting... in a "compulsive" way.

Is it normal to have so much fun in this way??????

Share this post


Link to post

When I received all my stuff, I wasn't able to think of making pictures... I just start unpacking, drooling and mounting... in a "compulsive" way.

Is it normal to have so much fun in this way??????

 

YES... until your wife/GF finds the receipt...

Share this post


Link to post

 

Pre-built .vmdk's - hosted at 4shared.  Click the blue Download box that looks like this:

 

hOPUY.jpg

 

- unRAID 5.0 RC3:  rc3_vmdk.zip

 

Thanks, for all the help and info so far, but........

 

Can't these vmdk's be downloaded somewhere else without signup or nag-timer-screen or having to download some downloadmanager?

 

How do you built such a vmdk? I'd like a 5.0b12a

Share this post


Link to post

I'm not uploading them to any of my own hosting as that has the potential for my monthly traffic to be demolished :)  4shared is the least intrusive of the sharing sites out there.  You don't have to download any download manager (you clicked the wrong link if you did that), just wait for the timer and the download should start.  It's only 20 seconds.

 

Re: building a 5.0b12, I gave instructions above on how to upgrade - downgrade is exactly the same.

Share this post


Link to post

When I received all my stuff, I wasn't able to think of making pictures... I just start unpacking, drooling and mounting... in a "compulsive" way.

Is it normal to have so much fun in this way??????

 

YES... until your wife/GF finds the receipt...

 

Then hell breaks loose?

Share this post


Link to post

I'm not uploading them to any of my own hosting as that has the potential for my monthly traffic to be demolished :)  4shared is the least intrusive of the sharing sites out there.  You don't have to download any download manager (you clicked the wrong link if you did that), just wait for the timer and the download should start.  It's only 20 seconds.

 

Re: building a 5.0b12, I gave instructions above on how to upgrade - downgrade is exactly the same.

 

I understand, sorry for nagging a bit.

 

I downgraded to 5.0b12a and it works a treat! Thanks again for all the help.

Share this post


Link to post

When I received all my stuff, I wasn't able to think of making pictures... I just start unpacking, drooling and mounting... in a "compulsive" way.

Is it normal to have so much fun in this way??????

 

YES... until your wife/GF finds the receipt...

 

Then hell breaks loose?

 

Or, you can avoid all of that spending almost the same amount of money for something more welcomed  ::)

Share this post


Link to post

hi

 

when you said ESXi-centric, where will you install esxi, on a different machine or the same machine as unRAID?

Share this post


Link to post

hi

 

when you said ESXi-centric, where will you install esxi, on a different machine or the same machine as unRAID?

 

VMWare's ESX is a hypervisor, allowing other OSs to be install "on" ESX as guests, sharing the hardware. The hypervisor, ESX, does the sharing. Other guests along with unRAID would be Windows and/or linux for further functionality.

Share this post


Link to post

hi

 

when you said ESXi-centric, where will you install esxi, on a different machine or the same machine as unRAID?

 

VMWare's ESX is a hypervisor, allowing other OSs to be install "on" ESX as guests, sharing the hardware. The hypervisor, ESX, does the sharing. Other guests along with unRAID would be Windows and/or linux for further functionality.

 

hi c3,

 

i'm aware of esxi. my question was, where will you install the hypervisor/ESXi? on the same box i'm thinking, on a flash?  Meaning, will the box be an unRAID box and an ESXi?

Share this post


Link to post

Yes, ESXi is the first thing installed to flash, along with a local SATA disk for an initial datastore.  Then you essentially follow the same procedure as a regular unRaid server (i.e. creating another flash drive with unRaid on it) and pass that through to the unRaid VM.  Alternatively, you can put the base of unRaid into a virtual disk (.vmdk) and boot off that for speed, as noted a few posts up.

 

Check out Johnm's Atlas build post for way more detail (take a few hours off to get through it! lol)

 

 

Share this post


Link to post

Thought I'd throw in a quick update with a couple more changes I've made - have converted one of the unused USB headers on the X9SCM to 2 USB ports with one of these:

 

http://www.amazon.com/StarTech-com-Motherboard-Header-Adapter-USBMBADAPT2/dp/B002GNU2V6/ref=sr_1_4?ie=UTF8&qid=1339320760&sr=8-4&keywords=startech.com+usb+header

 

Curiously, I've ordered a few things like this from Amazon over the last week or two, and every time, shortly after, they can no longer ship whatever it is to Australia.  Weird.  In any case, both the ESXi and unRAID USB sticks are now inside the case.

 

The Ubuntu 11.10 VM now runs Plex (rather than the unRAID plugin I was using previously) as I seem to have less (actually no) issues doing things this way, and don't have to worry about belting the cache drive (the crappy Corsair Force 3) with millions of thumbnails etc.  I have also centralised my XBMC database using MySQL on this VM, and store XBMC thumbnails here as well, with my various HTPC's around the house all pointing towards the database/thumbnail share using path substitution.

 

Plex is being used to transcode video to tablets and phones, as well as MyPlex being used for watching stuff on the go.  Will have 100/40 fibre in the next month or two, so that will make a big difference to watching things when out and about!

 

The Windows 7 VM is still running SabNZBD, CouchPotato and Sickbeard, with a couple of changes:

 

- have upgraded Couch to CouchPotato server v2

- added Headphones for music

- running a copy of XBMC on this VM now as it is the only one that is always on - this way SickBeard/CouchPotato/Headphones can update the shared XBMC database while my actual HTPC's are off.  They also update Plex over on the Ubuntu VM so my libraries are always up to date and ready to go.

- Growl is installed for notifications - connected to Prowl to deliver notifications to my iDevices.

- APCUPSD now installed here rather than ubuntu VM as there is no requirement to pass through an entire USB hub.  Passing through the APC device to the Windows VM just as a USB device is giving full functionality.  APCUPSD is set up to execute a script that first uses plink (from putty) to execute /sbin/powerdown on the unRAID box, and then uses plink to trigger a script in /bootbank on my ESXi box (so it persists through reboots.)  This script first collects a list of all the VM's and then powers them down cleanly.  It is executed using 'nohup' etc, so that the script continues to execute even when the VM with the UPS passed through to it turns off.  It then shuts down the ESXi host as well.

 

The only thing I'm not 100% sure about is whether or not the UPS then shuts down itself - I need to find some time to test that, but I'm guessing it may not as the Win7 VM running APCUPSD isn't the last thing to shut down.

 

I also have a third M1015 in there now (in one of the x4 slots) that is connected to 4x2Tb WD greens.  This is passed through to a FreeNAS VM, which is presenting a couple of iSCSI targets for shared storage.  As I'm looking at doing the VCP5 certification shortly, I picked up a 2nd hand set of IBM HS21 blades in a BladeCenter E series enclosure.  They have 8Gb RAM each and 2x dual Xeon CPU's.  ESXi 5 running on all 14 of them (after replacing the 10k SAS drives with SATA -> SD adapters to run ESXi from). 

 

The BladeCenter setup was a bargain at ~$1800 and the guy that refurbs/resells them here in Melbourne seems to do a roaring trade.  I've also got a 27RU enclosed rack on its way that I intend to put the server kit in - this will likely live out in the garage.  I am thinking of picking up a 2nd Procurve 1810G so I can run a 4 port trunk back into the house (for no other reason than I want to :P).  Either that or try and get the Nortel 6 port giga switches in the blades to do this.  We'll see.

 

Other than that, pretty happy with the whole setup now - it's nice and automated and for the most part, I can forget it's even on.

Share this post


Link to post

Another thing I forgot to mention - I've replaced the Monoprice cables with these:

 

http://www.ebay.com.au/itm/330624847262?ssPageName=STRK:MEWNX:IT&_trksid=p3984.m1497.l2649#ht_3214wt_1396

 

I needed a better solution than the 1m Monoprice as there was too much leftover cable in my case and I'm a bit of a neat freak.  I didn't really have much room to spool up the leftover length in my case, as you'll see from the pic below. 

 

I cut all existing cable ties etc and dropped one of these into place (I ordered one initially to make sure they were ok, @ $10 including postage I was a bit worried about what quality of cable might turn up!).  They are 60cm long, so a perfect length to run from M1015's to backplanes in a Norco (this was actually the first thing I did before the extra drives/M1015/Freenas VM - the pic shows the USB card in slot #3 which I have since removed and replaced with a 3rd M1015, as per previous post.)

 

Anyway, imagine my surprise when the cable that turns up is a Molex.  We use a lot of Molex cables at work and they are typically of a good quality, well made and a good price for quality compromise.  The other good thing about these SAS cables (and one thing that annoyed me about the Monoprice) is that the metal clip is exposed from the top, making it very easy to pop them out of a SAS port on a M1015.  (Those that have the monoprice cables know what I'm talking about - you have to get at the release clip from the side, so it can be a bit fiddly.)  They also clip in much more convincingly (the Monoprice ones seem a little, I dunno.. loose-fitting?)

 

In any case, the first cable worked no problems at all, so I ordered another 7 (couple spare).  They rocked up yesterday, and were all identical Molex cables as well.  Very happy with that, and have just finished replacing the Monoprice cables and cable tying it all up.  Of course, I put the lid on, racked everything up and put it away, forgetting to take a pic of post cable cleanup :)  I'll do that next time I have it open!

 

Inside - you can see the one test Molex cable (perfect length) and one of the old Monoprice cables (curled up over on the right.)  Bit of a mess as I'd pulled off all the cable ties while operating :)

 

QvY0F.jpg

 

 

...and a pic of a Molex SAS cable.  Taken from this angle to illustrate how the clip can be actuated from the top:

 

N6WTk.jpg

 

 

 

 

 

Share this post


Link to post

one question .. are you dual powering each backplane .. i know it's been brought up a couple of times, just curious.

Share this post


Link to post

Nah I haven't bothered - as per the pic above, I'm only running 3 backplanes at the moment (8 drives for unRaid and 4 for FreeNAS) and only powered one of the connectors.  imho, powering both connectors only makes sense if you have two power supplies.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.