DPC YAAC - Yet Another Atlas Copy

[delicatepc]

Hello,

 

Built an Atlas-like clone. Machine is created in an effort to upgrade our NAS capacity as well as consolidate our resources. Running X64 everywhere. VM ESXi 5

 

Hardware:

=====================================================================================================

Case: Norco 4224 - 24 hotswap bays - $400

Case Mid Fan Wall 120MM: 120MM Fan wall for Norco 4224 - $20 (shipped)

Case Middle Fans: Noctua NF-P12-1300 120mm Fan x 3 - $75 ($25 each)

Case Rear Fans: Noctua NF-R8-1800 80mm Fan x 2 - $34 ($17 each)

Power Supply: SeaSonic X-850 (SS-850KM Active PFC F3) 850W 80 PLUS GOLD Modular - $211

 

Mobo: SuperMicro MBD-X9SCM-F-O Intel 1155 C204 MicroATX Intel - IPMI/iKVM+ 2 Gig NICs - $200

CPU: Intel Xeon E3-1270 Sandy Bridge 3.4GHz 4 x 256KB L2 Cache 8MB L3 Cache LGA 1155 80W Quad-Core - $340

RAM: Kingston 16GB (2 of 2x4GB kits) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333 (PC3 10600) - $170

Small and Fast Datatore SSD: Intel 311 Series Larsen Creek SLC SSD 20GB 2.5" SATA II - $115

 

RAID Controller Card: LSI Internal SATA/SAS 9211-8i 6Gb/s PCI-Express 2.0 - $240

Expander Card: Intel RAID Twenty-four port RES2SV240 - $280

Backplane to onboard SATA ports Cable: NORCO C-SFF8087-4S Discrete to SFF-8087 (Reverse breakout) Cable - $15

Extra NIC: Intel Gigabit NIC - $28

 

Accessory - 1to7 Molex Connector: NORCO C-P1T7 4 Pin Molex 1 Male to 7 Female Power Extension Splitter Cable - $8

Accessory - Norco Rails: RL-26 Rails - $40

 

Server Rack: iStarUSA WN228 22U 800mm Depth Rack-mount Server Cabinet - $923

 

2x 2TB Samsung laying around - $0

5x 500GB hdds laying around - $0

 

Total (too much): $2180+ (not counting 1K toward rack or shipping). This is all part of a 6K order for a rack mounted experience.

=====================================================================================================

 

Thoughts:

 

Case:

Fans are too LOUD. Not as loud as a Dell Precision Tower with door off but loud enough to send you looking for a quieter solution.

BARELY FIT in the 800MM deep rack. Any less and i imagine its a no go.

Wiring up backplane is a pain in the ass unless you remove the fan wall.

Used 7 pin molex to connect both of the backplane power slots (per some forum user recommendations).

20GB SSD is floating inside.

The case is not bad. I like the drive caddies - seemed somewhat superior to the SuperMicro caddies from another machine.

However after purchasing PSU ($210), Piece of SHIT Rails ($40), Fans ($110), Fan Wall ($20), and Case itself ($400) it comes to about $800. SuperMicro (which most folks consider a superior brand to Norco) sells a 24 bay chassis with a 900watt redundant PSUs for $950. SUPERMICRO CSE-846TQ-R900B Black 4U Rackmount Server Case 900W.

The cost savings with the Norco can be described as $200 savings which you will repay back with labor. Perhaps getting slightly better tweaked components in the end. Your employer will be frustrated at the end result as the labor costs will be paid out of their pocket.

 

Rails:

Garbage. Get the ball bearing version through a different vendor; price is almost similar - FAIL NEWEGG!

I am living with these current friction rails but disappointing rails.

 

Mobo:

Wish every motherboard was built like this (headless mobos in any case). Best purchase.

Really like dedicated IPMI port

Want - More RAM slots (or buy the 8GB sticks if available), potentially second cpu slot.

Could use addition PCI express slot or two.

 

RAID Controller Card/Intel Expander Card:

I am a fan. So far working well. The Intel Expander came with a set of cables (about $80 worth). 2 went from RAID to expander, and 4 went to backplane. Gives me 16 drives in passthrough to my NAS guest VM.

Recommend. Im not using it in any RAID.

 

Extra NIC:

Dont seem to be able to use it in any of my guests so far (in passthrough). Its pretty standard NIC imagined it would be supported. ??

 

OS:

Went with OpenMediaVault for license free usage, auto update capabilities, virtualization friendly and based upon a stable Debian OS. I said screw it to unRAID manly due to not being virtualization friendly - the online updater of OMV just sealed the deal. Also really not a fan of unRAIDs windows centric delivery framework in the beta's..

 

Fans:

Just installed - day and night difference. Worth it. Buy them and the fan wall.

 

Currently have wired capacity for 16 drives in NAS, 4 drives in datastore/RDM capacity. 1 backplace (4 drives worth) is not wired - suggestions?

Why do only two of my drives show a blue AND a green light? The other caddies show blue only. The only difference i can think of is the other drives are all SATA2 and the two others are SATA3 2TB Drives

 

Once fan wall and fans arrive I will try to throw some pictures up.

http://imgur.com/3o8pY

 

~

dpc

[delicatepc]

At this time running 5 guests offering following network services:

--------------------------------------------------------------

> PXE booting (for OS installs, drive clones, etc)

> NAS (SMB/FTP/NFS/etc)

> FTP host - guest offering external FTP shares

> Internal LAMP stack - for timesheet etc.

> Media Guest - Middleware guest for NAS to serve up our media in variety places/protocols/formats (itunes/airvideo/subsonic/dnla/plex/etc)

 

=== PXE Booting Guest (PXE) ===

2 CPU, 2GB RAM, 8GB Thin Provisioned OS vmdk on SSD

--Ubuntu 11.10 based

- TFTP+PXELinux+Syslinux to serve up local imaging tools, recovery tools, diagnostic utilities etc.

 

=== Network Storage (NAS) ===

4 CPU, 4GB RAM, 8GB Thin Provisioned OS vmdk on SSD

Passthrough of LSI 9211 with two SFF8087 cables going to Intel Expander card. Total of 16 drives available.

--OpenMediaVault based

 

== FTP Host (FTP) ===

2 CPU, 2GB RAM, 8GB Thin Provisioned OS vmdk on SSD

-- FreeNAS 8 based

 

== LAMP Host (LAMP) ===

2 CPU, 2GB RAM, 8GB Thin Provisioned OS vmdk on SSD

-- Ubuntu 11.10 based

 

== Media Middleware Guest (MEDIA) ===

4 CPU, 6GB RAM, 60GB Thin Provisioned OS vmdk on HDD

-- Windows Server 2008 R2 based

- Mapped network drives to NAS

- Multiple software such as AirVideo, Subsonic, Plex Media Server that stream and transcode media

 

[Johnm]

Nice Build..

 

I completely agree with you that the Norco stuff is Craptastic. It is  Hobbiest grade while the supermicro servers are (low end) Enterprise Grade.

The Supermicro case itself is not to loud when used with a Supermicro board, the Supermicro PSU is loud However.

 

For a work environment, the Supermicro Case would be a no brainer. (If you do use a Norco for work, you probably would want to keep the stock fans)

for home.. people want to pinch pennies. making it silent and time spent "tinkering" is time spent as a hobiest...

 

You might be able to send the rails back to IPC direct and get them to swap them for ball bearing versions? send them an email

 

[side note, is it just me or was that Norco 7 connector wire a POS?

I keep meaning to update Atlas to point out they suck. I know My other build's notes point out they are crap.]

 

Kingston is starting to offer the 8GB sticks now.  keep your eye out.

 

I'm glad that expander is working out, I am getting mine in a week or so now, I cant wait.

Remember, you do not need to waste a PCIe slot with that expander if you are  running low on slots.

 

AS far as the second NIC. it is a 2 min hack to get it up and running...

Originally Posted by peetz

...1. Install your machine(s) with the vanilla ESXi 5.0 ISO.

2. Log on to the console (or via ssh) of one of the machines and install the vib file by using the following commands:

esxcli software acceptance set --level=CommunitySupported

esxcli software vib install -v http://files.v-front.de/net-e1001e-1.0.0.x86_64.vib

3. reboot, configure all NICs, and try to enable FDM then

 

Not running unRAID.. shame on you!

I have never actually tried OMV.

 

 

 

 

 

[WeeboTech]

How is OpenMediaVault Virtualization friendly?

 

[prostuff1]

How is OpenMediaVault Virtualization friendly?

 

I was thinking the same thing...

 

My unRAID VM running under ESXi works perfectly fine and has been running great for me since pretty much day one.

[Johnm]

probably the same way unRAID and windows is.. it sees the native hardware?

the only other way i could think of is if it has the vmWARE client built in..

[delicatepc]

Surely you folks jest....??

 

1. (typical) unRAID requires a USB stick to be installed on. Yes a vmdk can be created and its great idea. However not interested in creating a vmdk for each new release. As an additional determent the unRAID requires windows to run the prepackaged betas - the syslinux executable - at home I have only a mac and im ok with that - dont understand why we are stuck to binary blobs for windows for a linux OS.

 

2. unRAID licensing scheme appears to be based off USB stick info. Perhaps it works without the USB stick but I do not feel comfortable to invest into something that is tied to a USB stick or generic host. I dont care if the support backend for the one off situations that could potentially occur was angels themselves - I didnt want to deal with that in a virtualized solution. I also didnt feel like unRAID product was polished enough (speeding ant webgui addon excluded) to be worth the money - I care about the nitty gritty but damn for some paper i want some polish, hobbyist or not. unRAID technology seems sound enough - needs some spit and shine

 

3. Not promoting or denying the stability of unRAID virtualized. It potentially is rock solid but that was not my sticking point.

 

4. No (easy and clean) method to get a sandbox/dev/virtual version of the unRAID up. I did some scripting and automation work for unRAID and it was difficult to test my code because i was having trouble with having a unRAID VM. Too much work for a hobbyist project. I wanted to do more but it was becoming difficult for me to continue fighting getting a clean environment to test with instead of focusing on my script attempts.

 

OMV:

 

1. Simple ISO install. I boot and install the latest version from an ISO. New version comes out i boot up new iso and go. In unRAID newest beta comes out and that means i have to potentially mess with the USB key and/or create new disk image.

2. Built in nice update manager - really nice feature that makes my life a lot easier.

3. Didnt have to create anything or go out of my way to whip up a VM with OMV. 2 minutes and I have a new VM with OMV to test and tinker my code with.

4. Bonus points for the dev who developing it virtualizes the environment as well.

5. Less windows reliant - some of unRAID plugins are flat out made by windows users for windows users (in a bad way). Zips instead of tarballs for example (tar ships with unRAID, infozip (contains unzip) dont not ship with unRAID - why plugins zipped for crying out loud?).

6. Slightly off topic but WAY MORE functionality built into the core offering than the unRAID. To each to his own but i love that i have FTP, NFS, SMB/CIFS, an other services available for the get go and dont have to rely on plugins for what i always considered core functionality in a NAS (built in FTP for the big long transfers where you dont want to mess with SMB/CIFS).

 

Johnm - The Norco 7 connector seemed flimsy but so far i cant complain. I definitely had better molex connectors thats for sure.

BTW went with the Noctua 120mm and 80mm fan replacements and now my drives are cooler and the machine is ALMOST completely SILENT. Day and night difference.

 

Second NIC - I was refering to the extra PCI express NIC in passthrough - cant seem to get it to work. Will try passing it through to my windows host to see if its perhaps something funny driver wise. The second nic onboard is working great for me with that patch.

 

~dpc

[WeeboTech]

Actually you only need one vmdk or usb key that is large enough to hold all the versions you want to use.

 

Edit the syslinux.cfg to point to the new version and reboot. Done.

 

I have one usb key with several versions and other tools too.

 

I just name them as bzroot-4.7 bzimage-4.7, etc, etc and add entries to the syslinux.cfg.

 

unRAID does have FTP, NFS and SMB/CIFS. Not much more then that.. but I like it thin.

I love the fact that it's root in ram and the only thing I need is a USB key or PXE Boot and data disks.

 

 

Interesting points though, Makes me want to check out Open Media Vault, but I'm not sure I want to give up my protected contained JBOD filesystems that spin down when I'm not using them.

[prostuff1]

Surely you folks jest....??

Nope, not in this case

 

1. (typical) unRAID requires a USB stick to be installed on. Yes a vmdk can be created and its great idea. However not interested in creating a vmdk for each new release. As an additional determent the unRAID requires windows to run the prepackaged betas - the syslinux executable - at home I have only a mac and im ok with that - dont understand why we are stuck to binary blobs for windows for a linux OS.

Um... unetbootin runs on OS X and I wrote a tutorial using it.  You can find the tutorial here.

 

2. unRAID licensing scheme appears to be based off USB stick info. Perhaps it works without the USB stick but I do not feel comfortable to invest into something that is tied to a USB stick or generic host. I dont care if the support backend for the one off situations that could potentially occur was angels themselves - I didnt want to deal with that in a virtualized solution. I also didnt feel like unRAID product was polished enough (speeding ant webgui addon excluded) to be worth the money - I care about the nitty gritty but damn for some paper i want some polish, hobbyist or not. unRAID technology seems sound enough - needs some spit and shine

That is correct, the license is tied to the USB GUID.  As for the pretty-ness stuff...  I like as much pretty as the next person, being a mac user and all, but honestly... I rarely look at the unRAID gui and so long as it is functional that is all I really care about.

 

3. Not promoting or denying the stability of unRAID virtualized. It potentially is rock solid but that was not my sticking point.

But this "virtualization friendly and based upon a stable Debian OS. I said screw it to unRAID manly due to not being virtualization friendly" makes it sound exactly like a reason you did not use unRAID.  I have not found any reason to NOT call unRAID virtualization friendly.

 

4. No (easy and clean) method to get a sandbox/dev/virtual version of the unRAID up. I did some scripting and automation work for unRAID and it was difficult to test my code because i was having trouble with having a unRAID VM. Too much work for a hobbyist project. I wanted to do more but it was becoming difficult for me to continue fighting getting a clean environment to test with instead of focusing on my script attempts.

um... I have 2 unRAID VM's up and running on my Mac.  One is for 4.7 and the other is running current beta.  I don't understand the "getting a clean environment" argument either.  Just delete what you installed, reboot the VM and your golden.  Even easier would be to take a snapshot and restore to that snapshot after your done doing your messing around.

 

1. Simple ISO install. I boot and install the latest version from an ISO. New version comes out i boot up new iso and go. In unRAID newest beta comes out and that means i have to potentially mess with the USB key and/or create new disk image.

I boot my ESXi machine using the plop method.  It is slower, but frankly the machine is rebooted so rarely that I could care less.  I just drop the new unRAID version of the USB and reboot and all is good

 

2. Built in nice update manager - really nice feature that makes my life a lot easier.

Again I don't update my production machine enough to care about an auto-update mechanism.  I have only every updated my production unRAID machine if I really wanted a new feature or is something was broken.  I typically hate software that auto-updates (point in case Chrome... love the browser hate the auto-updating).

 

3. Didnt have to create anything or go out of my way to whip up a VM with OMV. 2 minutes and I have a new VM with OMV to test and tinker my code with.

I imported an appliance into Virtual box and was up and running with unMenu install in 2 minutes also.  I used this as a starting point and then upgraded one to 4.7 and the other to current beta.

 

4. Bonus points for the dev who developing it virtualizes the environment as well.

um... k... not sure how this really matters.  I have a full slackware VM set up and am going to be getting unRAID up and running in it soon hopefully.

 

5. Less windows reliant - some of unRAID plugins are flat out made by windows users for windows users (in a bad way). Zips instead of tarballs for example (tar ships with unRAID, infozip (contains unzip) dont not ship with unRAID - why plugins zipped for crying out loud?).

um... and this is no different than on the Mac side where stuff comes packed in .dmg files... really, you are going to complain about the fact that some of those available plugins are in .zip's?  Zip's are pretty much universal and understood on any OS.

 

6. Slightly off topic but WAY MORE functionality built into the core offering than the unRAID. To each to his own but i love that i have FTP, NFS, SMB/CIFS, an other services available for the get go and dont have to rely on plugins for what i always considered core functionality in a NAS (built in FTP for the big long transfers where you dont want to mess with SMB/CIFS).

The ones you list are all included in unRAID, and I have used all of them.  NFS not all that often but I do use FTP and SMB regularly.  I don't understand the "FTP for long transfers" thing as I have moved loads of data over SMB and never had a problem unless it was hardware related.

 

Johnm - The Norco 7 connector seemed flimsy but so far i cant complain. I definitely had better molex connectors thats for sure.

BTW went with the Noctua 120mm and 80mm fan replacements and now my drives are cooler and the machine is ALMOST completely SILENT. Day and night difference.

I have never trusted power spliters, hopefully yours works better than the ones I have had.  I have the 4220 and the fan wall fans were OK, it was the back ones that where loud.  I replaced them with some Antec ones that have the little switch on them for controlling the speed.  If works a treat and I can make them go whatever speed I like.

[delicatepc]

It seems you are more versed in these matters. To each their own.

 

~

dpc

[delicatepc]

My Drive Temps with new fans - temps are about same or lower than with the noisy OEM fans:

Note - The server is *silent* as far as I am concerned.

So to summarize Norco 4224 Chassis:

Case: Norco 4224 - 24 hotswap bays - $400

Case Mid Fan Wall 120MM: 120MM Fan wall for Norco 4224 - $20 (shipped)

Case Middle Fans: Noctua NF-P12-1300 120mm Fan x 3 - $75 ($25 each)

Case Rear Fans: Noctua NF-R8-1800 80mm Fan x 2 - $34 ($17 each)

Power Supply: SeaSonic X-850 (SS-850KM Active PFC F3) 850W 80 PLUS GOLD Modular - $211

Accessory - 1to7 Molex Connector: NORCO C-P1T7 4 Pin Molex 1 Male to 7 Female Power Extension Splitter Cable - $8

Accessory - Norco Rails: RL-26 Rails - $40

Total Cost for Norco 4224 Chassis: About $815 (with shipping) - Approximately $150 dollar savings over the SuperMicro chassis.

 

Would I buy the chassis again? Yes as long as I order all these parts (fans etc) together. I am ok with the Rails (even though its a shame they are such a piece of crap).

 

~

dpc

[c3]

I was going to PM, but pretty sure many would like to hear your take on a few feature differences between unRAID and OpenMediaVault.

 

I am a noob at unRAID, even less skilled at OMV, so don't over estimate these statements.

 

1. Drive spin down/Power Management; This is a key feature for many people using unRAID. OMV seems to have spin down, but with RAID at the block level OMV uses more power. unRAID can serve data from a single drive in a 10+ disk array by powering a single drive.

 

2. RAID6(dual parity, P+Q); not currently available in unRAID, but OMV has RAID6, hot-spare, and auto rebuild from Linux software RAID.

 

3. FlexRAID; There is some collaboration between OMV and FlexRAID. FlexRAID bring several of the features of unRAID to OMV, like using disks with existing data and recovering data from individual drives of a failed RAID stripe (multiple disk failure).

 

PS: I found it trivial to make a vmdk for unRAID, but I am using no license. Running unRAID 4.x on full slackware is documented, so that is pretty virtualization and update friendly.

 

Mods: Delete if mentioning other solutions is a problem.

 

 

 

[WeeboTech]

Mods: Delete if mentioning other solutions is a problem.

 

It's a debate on the finer features of unRAID vs others, as long as it does not become condescending, disparaging and the communication is open and friendly, some points could be learned.

[Johnm]

In my opinion..

 

The OS is just a tool in a toolbox...

 

You just use the tool best suited for the job at hand.

 

I have several hardware raids..

 

I use unraid for the simple fact of mass storage with low power consumption and the ability to mix drive types/sizes/speeds/

 

 

@delicatepc

Those fans are #@$@# expensive and worth every penny.. my servers can sit in a spare bedroom and not bother anyone..

Im glad she is treating you well..

[WeeboTech]

Thanks John, good point. I'll borrow it and add...

 

I use unRAID for simple mass storage, low power consumption with the ability to mix drive types/sizes/speeds and have protected raid volumes confined to single hard drives. Sort of a protected JBOD with a layer to make it look consolidated.

Plus having an OS that boots from flash into a ram drive where the only drives in a system are data drives.