ESX5 Hypervizor - a sea of information, trying to find the sweet spot

[NAS]

So in the spirit of the forlorn hope I am hoping for some advice.

 

I aim to build a new ESXi5 box. On this, amongst other things, will be an unRAID 5 test bed but if i am honest it is mostly for other linux appliances and firewall etc.

 

I pretty much know ESX4 backwards and forwards but until now I have always used HP/Dell hardware. This is not possible this time due to budget and heat/noise i.e. its going in my attic

 

I was holding out for the i7 3820 and X79 since they will allow 64GB of RAM. However this would be non EEC RAM and thats potentially a bad idea.

 

I have a case sorted and datastores will be on JBOD SSD (nothing is critical and backups will be fine).

 

Supermicro seems the obvious choice but the CPU drives the model short list.

 

Other than that I would like:

 

plentyfull PCI-e slots

6+ USB2

USB3 would be nice

VT-d is a must

 

and if I am lucky one of the Supermicro boards will be the onboard quad GigE type.

 

 

So I am looking for advice on motherboard and CPU if anyone had any ideas. I have zero XEON product line knowledge.

 

Kudos to anyone that can assist

[NAS]

After much reading I am going to try and hold out for E5 pricing.

 

The stiicking point with E3 is the price and rarity of 8Gig ECC dimms

[c3]

ESXi5 is limited to 32G.

 

E5 is expected to be more costly than E3 (maybe you're hoping E3 will drop).

[NAS]

ESXi5 is limited to 32G.

 

Free ESXi5 is limited to 32GB of ram. Non free is 2TB from memory.

[c3]

Once you buy a license, it is not called ESXi. You buying Enterprise for 64G?

 

But you are right, everything is available for you with excess wealth. I just don't see the value in spending so much.

[NAS]

Technically i dont think ESXi exists anymore. I think they are all called ESX hypervizor but what its called is not so relevant

[c3]

Technically i dont think ESXi exists anymore. I think they are all called ESX hypervizor but what its called is not so relevant

 

VMWare calls it ESXi.

http://www.vmware.com/products/vsphere-hypervisor/overview.html

 

[WeeboTech]

I find the product line to be confusing at times.

 

ESX/ESXi/Vsphere/workstation/fusion/ace. ugh.

 

Then I saw this from the other link

"Looking for the Free ESXi?

 

The name of the free ESXi product has changed."

 

which went to here.

http://blogs.vmware.com/vsphere/2010/07/introducing-vmware-sphere-hypervisor-41-the-free-edition-of-vmware-vsphere-41.html

 

[NAS]

I find the product line to be confusing at times.

 

I find it confusing all the time because it changes so much and such different products are so similarly named.

 

I tend to use ESXi because everyone knows what I am talking about but even EMC use ESX Hypervizor and no one has used vSphere Hypervisor to me yet. I think thats the actual name but when the people themselves get it wrong what hope is there for us. And if I say vsphere to someone they immediately think of the control server which wasn a program but is now also a virtual machine. arg

 

But in summary the free ESX like product no longer has a shell, it no longer limits cores but limits RAM to 32GB. Once you buy it you get more RAM and access to the advanced commercial features. Thi is in general as you can buy all sorts of lciesning  direct ad via OEMs. Wont be long and it will be as bad as M$ licensing.

[c3]

ESX had the service console. ESXi does not a service console, but it does have a shell, which is BusyBox. The service console was just consuming resources (500-800M) all that functionality remains in the vMA. The only thing I use the ESXi console for is add/remove modules like the X9SCM nic driver, etc. Otherwise vMA does it all.

 

[c3]

But in summary the free ESX like product no longer has a shell, it no longer limits cores but limits RAM to 32GB. Once you buy it you get more RAM and access to the advanced commercial features. Thi is in general as you can buy all sorts of lciesning  direct ad via OEMs. Wont be long and it will be as bad as M$ licensing.

 

Actually, you must buy Enterprise or Enterprise Plus to get more than 32GB RAM. Buying Essentials or Standard still limits you to 32GB. Which in fact was the original point of this discussion, you are wanting to go beyond 32GB which means additional spending on software. So, just buying "it" does not get you more RAM. If you buy "them", two or more licenses, you get past 32GB RAM.

 

VMware licensing is already far worse than Microsoft.

[MrD1234]

attic?  i hope it's cooled or you live in the north somewhere... my attic is 100+ degrees in the summer

 

[NAS]

But in summary the free ESX like product no longer has a shell, it no longer limits cores but limits RAM to 32GB. Once you buy it you get more RAM and access to the advanced commercial features. Thi is in general as you can buy all sorts of lciesning  direct ad via OEMs. Wont be long and it will be as bad as M$ licensing.

 

Actually, you must buy Enterprise or Enterprise Plus to get more than 32GB RAM. Buying Essentials or Standard still limits you to 32GB. Which in fact was the original point of this discussion, you are wanting to go beyond 32GB which means additional spending on software. So, just buying "it" does not get you more RAM. If you buy "them", two or more licenses, you get past 32GB RAM.

 

VMware licensing is already far worse than Microsoft.

 

I have access to a license so 32GB is not a limit i need to worry about artificially. It does seem to coincidentally be the current sweet spot for kit vs cost though.

 

I suppose I am in somewhat of a unique position having a license that is worth several times more than the kit i am buying so for the sake of a wider audience let's just put this fact and the sweet spot together and say 32GB

 

attic?  i hope it's cooled or you live in the north somewhere... my attic is 100+ degrees in the summer

 

yeah its fine i have haad a high low thermometer in for for 3 years as there is already a equipment cabinet in there. fine point though.

[brian89gp]

The free license limites you to 32gb of RAM and 1 processor.  The essentials and standard licenses limit you to 32gb per processor.  Only really important if you are buying a multi socket moterhboard, but an important distiction to make.

 

Anybody that has worked with it for a while including VMware will still call ESXi by that name, sales and marketing people are always a bit confused and call it by both ESX and ESXi.  ESX\i 5.0 is available only in the ESXi version, so whether ESX or ESXi is used it refers to the same thing.  VSphere is the name they use for a suite of products, usually ESXi and VirtualCenter at a minimum.

[NAS]

Good clear definition thanks.

 

Now perhaps ... back on topic since I am still looking for hardware suggestions rather than a decoding the mystery's of vmware naming

[NAS]

An update.

 

I am now seriously looking at X79 workstation motherboard with the new i7 3820 CPU. The downside is i lose IPMI however the big upside is that if i buy a high end MB i get 8 DIMM slots (i.e. cheap 8x4GB = 32GB RAM), SATA3 and USB3 and more importantly a larger count of higher speed PCI-e slots.

 

For instance the P9X79 WS has:

 

4 x PCIe 3.0/2.0 x16 (dual x16, triple x16/8/8, quad x8, black+blue) *1

2 x PCIe 3.0/2.0 x16 (x4 mode, white) *1

 

Since they also support vt-d via the latest BIOS update for C2 i7 CPU's this allows me to consolidate tvheadend and add other things like a CCTV server. I may even be able to do asterix.

 

The research continues.

[Ford Prefect]

I don't like the idea of non-ECC RAM in a server, although 8 DIMMs are an appealing option.

At least I'd double check that ESXi will support the LAN chip(s).

 

 

[Ford Prefect]

...Bios changelog of the ASUS P9X79 WS states, that vt-d is supported for CPUs with C2 stepping....the only i7-3820 with vt-d capability I could find, claims to have M1 stepping from the specs.

 

Don't know if that is an potential issue or if that combo would work as intended...

[NAS]

It is definately a potential probem.

 

From what I could tell there was a bug in a huge suite of intel C1 CPUs that meant vt-d didint work correctly. Alot of it is quite vague when it comes to vt-d so I will be waiting until someone else takes the plunge and posts somewhere on the net a uccess story.

 

I agreed about ECC ram but the costs are too good for me to ignore off hand. Interestingly though this board claims

 

8 x DIMM, Max. 64GB, DDR3 2400(O.C.)/2133(O.C.)/1866/1600/1333/1066 MHz ECC, Non-ECC, Un-buffered Memory

 

Will need to investigate more as that wording is a bit ambiguous for me.

 

Edit:

 

http://ark.intel.com/products/63698/Intel-Core-i7-3820-Processor-%2810M-Cache-3_60-GHz%29

 

Clearly says vt-d support in a completely catagorical way. I would assume that answers that one at least... i hope

 

[Ford Prefect]

I agree that the information available regarding vt-d is not structured well.

When I researched for a server with more headroom, I thought of a Dual-XEON 1366 box.(E56xx), maybe starting

with a single CPU.

None of these CPUs list vt-d capabilities, but some mobos with the E5500 / E5600 chipsets do, like the intel S5500BC

To add to the confusion, one can find this KB-article at intel: http://www.intel.com/support/processors/xeon5k/sb/CS-031637.htm

[...]

Information

VT-d is not a feature of the processor itself but rather a feature of the Chipset. As long as the chipset/BIOS combination support VT-d and the processor support VT, there will be VT-d support.

...should I really trust that statement...is the real world around vt-d that simple?

[NAS]

In theory yes it is that simple. The wording of this is not great as it is a 50:50 split in requirements. i.e. both CPU and chipset/MB support.

 

There is an added gotcha... BIOS support. Even if the chipset supports it some MB either choose not to support it or bury the option in some vague menu.

 

This is why it is nice to see BIOS/UEFI release notes with very clear "vy-d" support statements.

 

I am not saying that in every instance it always works as its quite a new technology but in most instances when the support is there it just works.

 

For me it is the killer feature making ESX far more useful.

[Ford Prefect]

Yes...BIOS.

Whenever I check a mobo for a potential build, I try to download the

manual and check if I really can see that item in the BIOS screenshots given.

I also do not want to miss that feature anymore...just deployed a VM with a 4x DVB-S2 receiver for automated recordings...very nice.

[Ford Prefect]

 

8 x DIMM, Max. 64GB, DDR3 2400(O.C.)/2133(O.C.)/1866/1600/1333/1066 MHz ECC, Non-ECC, Un-buffered Memory

 

Will need to investigate more as that wording is a bit ambiguous for me.

 

Edit:

 

http://ark.intel.com/products/63698/Intel-Core-i7-3820-Processor-%2810M-Cache-3_60-GHz%29

 

Clearly says vt-d support in a completely catagorical way. I would assume that answers that one at least... i hope

 

...also clearly says 32GB max-MEM, although it comes with quad channel RAM support...and no ECC support it says.

 

If you have access to a ESXi license, maybe a DP-Xeon based around an E5xxx IS a valid option...looks as you could get on the safe side with that more easily.

[NAS]

I am not convinced that 32GB limit is a real one although in reality it will be cost wise for me this now.

 

E5 is definitely interesting but I am not seeing much in the way of socket 2011 server boards or support for it in workstation boards. The more i think about it SATA3 and multiple PCI-e 8x + are big requirements for this build.

 

With 3rd gen SSD so cheap I can get a lot of performance for not alot of money and i will use 3 PCIe 4x up right away with two DVD cards and a quad gigE nic

[NAS]

Why oh why do they make this so hard.

 

Intel Ark says max 32GB of RAM for this CPU

Asus says maximum 64GB of RAM for this board

Intel Ark says ECC is not supported but if you open the popup if changes to "may not be supported"

Asus says ECC support in product summary and lists memory in the QVL with ECC e.g. ACTICA 4G DDR3 ECC UNBUFFERED Hynix IC BRAND 1333Mhz Server Memory ACT4GHU72D8H1333H

 

Is it really that hard to make this stuff clearer, I shouldnt have to get a PhD to find a list of components grrr