ASRock C226WS+ (note the plus sign)

[grandprix]

http://www.newegg.com/Product/Product.aspx?Item=N82E16813157410

 

So while reading about ESXi and other Hypervisors, I stumbled across an article that very generally stated that the C224 chipset wouldn't support Vt-d (I have since closed the tab in my browser containing the blog -- but it exists I assure you).  I figured "great now to find a different motherboard" for a VM lab/"production" (use that term loosely as it's not going to be running anything vital, such as at a business or anything, just home use).  I came across this board.  It looks, well, dare I say perfect?  Vt-x, Vt-d, Vpro, etc, etc. and FOUR x8 -3.0- PCIe slots, 10 6Gbps SATA ports (4 of which are on a Marvell controller though, vice the C226 chipset), an I210 Intel Giga NIC, Haswell compatible board, etc, etc.

 

Even though it would have meant a return with restocking fees, I'm finding myself a little disappointed that I don't have to return the SuperMicro board I do have because it does have Vt-d compatibility.  But, it only has three PCIe slots, two of which are 2.0 (x8 and x4, which is more than enough for my needs, but, four x8 3.0 would be better! lol)

 

So in short.  Is anyone using this motherboard?  I see someone is using the AsRock C226WS (note the LACKING plus sign) and did a level 2 on it, but, nadda for this one.  Anyone?

[Whatever]

I'm pretty sure i've seen reports of C224 motherboards supporting Vt-d (namely the mini-itx E3C224D2I), but i digress.

[grandprix]

Right, I mentioned that somewhere up in that mess.

 

However then you have this:

 

http://forums.anandtech.com/showpost.php?p=35253535&postcount=28

 

 

And in case the rabbit hole wasn't deep enough (for me anyway at this stage of the game) as I continued my reading, I'm wondering.  What is the point of having four x8 lanes (32 total PCIe lanes), when the most I can find of any 1150 socket CPU is 16.

 

Does the chipset/CPU "queue" bits until all PCIe lanes are not occupied, or does the PCIe hardware link down of sorts?  <shrug>  Likely beyond the scope of this post (or subforum that it is in), though I'm curious never-the-less.

 

[sgibbers17]

I use a asrock c226 ws but not the ws+ running ESXi 5.5 but the marvell controllers are not supported by ESXi.

 

I suspect the ws+ will also run ESXi just as well.

[garycase]

No need to change motherboards.  Your C224-based SuperMicro MBD-X10SLM-F-O is just fine.

 

The C224 Express specifications clearly say "Intel I/O Virtualization (Vt-d) Support"

 

... so the chipset does provide support for this.    And it's also noted on page 1-13 of Supermicro's manual for the board, so they haven't done anything to disable this support in the BIOS.

 

And of course the E3-1240 supports Vt-d as well.

 

So ... if it's not showing as being enabled, then there MUST be a setting in the BIOS that enables/disables the support.

 

I don't have that specific board, so I can't confirm that a later BIOS version may have changed the specifics as to where to make the changes;  but according to the manual virtualization (Vt-x) is enabled/disabled on the CPU Configuration page under the Advanced tab; and Virtualization support for Direct I/O (Vt-d) is enabled/disabled on the Chipset Configuration page under the Advanced tab.

 

Note that by default (according to the manual -- this could have changed in later BIOS versions) Vt-x is Enabled, and Vt-d is Disabled.

 

I suspect all you need to do is enable Vt-d 

[garycase]

... just to confirm things haven't changed, post back after you look on the Chipset Configuration section of the BIOS for the Vt-d setting.

 

[grandprix]

No need to change motherboards.  Your C224-based SuperMicro MBD-X10SLM-F-O is just fine.

 

The C224 Express specifications clearly say "Intel I/O Virtualization (Vt-d) Support"

 

http://ark.intel.com/products/75519/Intel-DH82C224-PCH?wapkw=c224

 

Is where I looked, as you see, Intel doesn't have Vt-d listed at all, "yes" or "no" (which I find odd not to have the "no").

 

And of course the E3-1240 supports Vt-d as well.

 

Intel was gracious enough to definitively list Vt-d, so yep indeed.

 

http://ark.intel.com/products/75055/Intel-Xeon-Processor-E3-1240-v3-8M-Cache-3_40-GHz?q=e3-1240v3

 

So ... if it's not showing as being enabled, then there MUST be a setting in the BIOS that enables/disables the support.

 

I don't have that specific board, so I can't confirm that a later BIOS version may have changed the specifics as to where to make the changes;  but according to the manual virtualization (Vt-x) is enabled/disabled on the CPU Configuration page under the Advanced tab; and Virtualization support for Direct I/O (Vt-d) is enabled/disabled on the Chipset Configuration page under the Advanced tab.

 

Note that by default (according to the manual -- this could have changed in later BIOS versions) Vt-x is Enabled, and Vt-d is Disabled.

 

I suspect all you need to do is enable Vt-d 

 

It is listed in the BIOS.  It is (clearly - in the link above) supported by the CPU.  It was this response from SuperMicro regarding a chipset that also doesn't list as Vt-d compatible, that had (has?) me still wondering:

 

http://forums.anandtech.com/showpost.php?s=1d7de3f50267cc61b73faebfff391d36&p=35253535&postcount=28

 

http://ark.intel.com/products/75013/Intel-DH82Z87-PCH?q=z87

 

I appreciate the response, I even asked for it in PM, I know bud.  Though, I guess I would just feel so much more at ease if Intel said "yes, its possible for the C224 chipset to support fully compliant Vt-d", SuperMicro wouldn't say "One workaround is to use a VT-d enabled PCIe device and plug it into CPU based PCIe-port on board. Along with a VT-d enabled CPU the above workaround should work per Intel" and if so many people wouldn't seem to suggest that PCIe pass-through was/was not (conflicting posts and I must have read at least 50 such posts) working on C224 chipsets.

 

SuperMicro's response makes it sound like it would be a pseudo compliance of sorts.

 

I just looked at the C222 chipset, it explicitly lists Vt-d as a big fat yes.

 

http://ark.intel.com/products/75516/Intel-DH82C222-PCH

 

Stands to reason despite them all (i.e. C222, C224, C226) sharing the same release yes/quarter, that, they should list the C224 either yes or no for Vt-d, or at least yes if it does.

 

I cannot find anywhere someone that has confirmed proper and full PCIe passthrough using a C224 chipset, regardless if the memory controller was on the CPU or not .. and of course when the CPU and the hardware in the PCIe slot was Vt-d compliant.

 

I do see everywhere (here is no exception) of people having difficulty passing through, say, a PCIe graphics card.  And when I look, sure enough, its usually a C224 based mobo they're using.

 

* Answer to whether the Vt-d shows in BIOS, yes, though, even Supermicro suggests it's a pseudo type of support.

 

[garycase]

I can certainly understand the confusion -- and since I don't have a C224-based board I can't absolutely confirm the support.

 

A few thoughts ...

 

=>  It's indeed strange that Intel omitted this from their Ark page on the C224, but I've seen other cases where the Ark specs were incomplete, so it could very well just be another of those.    Note that it's omitted -- not listed with a "No" ... so Ark doesn't say it's NOT supported;  it's just silent on the issue.  Personally, I think it's VERY unlikely that the C222 has Vt-d support but not the C224.

 

=>  Along those same lines, the Z87 Ark page is also silent on Vt-d support [ http://ark.intel.com/products/75013/Intel-DH82Z87-PCH ]; and one of your references indicates it does not support it;  but Intel specifically notes that their Z87-based boards DO have Vt-d support [ http://www.overclock.net/t/1338063/vt-d-compatible-motherboards ].

 

=>  Bottom line:  I think the omission of Vt-d in the "Advanced Technologies" list is just that -- an omission -- and does not indicate that it's an unsupported feature.

 

=>  I DO have a Z-97 board here; and just checked the BIOS, which DOES list Vt-d as a feature I can enable/disable.    But the Ark site is also silent on Vt-d support for this chipset ... which I suspect is yet-another omission of a feature that's in fact included.  But I have nothing installed that will let me confirm that; and it's a system I built for someone else, so I can't really do any experimentation on it (I have to deliver it tomorrow).

 

=>  I don't think Vt-d would be listed in the BIOS if it was not supported.  Since it IS listed, and I presume you can in fact Enable it without issue, then it's almost certainly supported.    That does, however, beg the question as to whether this is simply enabling it on the CPU without true chipset support.  I do NOT think that's the case; but as I noted, I don't have any way to confirm this.    But based on your research, it seems that it may in fact be true.

 

=>  Unfortunately, Intel's own Processor ID Utility [ http://www.intel.com/support/processors/tools/piu/sb/CS-014921.htm ] only shows "Virtualization Technology" support (yes/no) -- but doesn't differentiate between Vt-x and Vt-d, so it doesn't answer the question definitively either.    And as you've already noted, Intel has extensive details on the support for their CPU's, but very little information on specific chipset support (and key omissions in the Ark details on several of the chipsets).

 

Bottom line:  I agree it's difficult to answer your question definitively without simply experimenting on a C224-based board.    Have you tried doing this in ESXi?  I think that's the best test, as there's a lot more written about exactly how to achieve it than there seems to be with Xen of KVM.    In any event, you've convinced me that a safer path is to get a C226-based board 

 

 

[grandprix]

Bottom line:  I agree it's difficult to answer your question definitively without simply experimenting on a C224-based board.    Have you tried doing this in ESXi?  I think that's the best test, as there's a lot more written about exactly how to achieve it than there seems to be with Xen of KVM.    In any event, you've convinced me that a safer path is to get a C226-based board 

 

I'll be sending the C224 based motherboard back then, I'd rather not open it (haven't opened it yet, since the CPU hasn't been opened yet).  Now, do I take the E5 plunge or stick with the E3, decisions decisions.  The E5 would set me back a good bit, but boy the VM goodness it would run over the E3 (if for nothing else but being able to run more than 32GB of RAM). 

 

 

The C222 is Vt-d according to the BIOS (manual of the SuperMicro I'll be referencing below) but lacks Vpro, however it has IMPI.

 

So the better choices seem to be (from a VM point of view that full support Vt-x / Vt-d):

 

SuperMicro's C222 based board (with great reviews, built in LSI-2308 controller):

http://www.newegg.com/Product/Product.aspx?Item=N82E16813182821

 

SuperMicro C222 based board (decent reviews, seems most of those bad reviews were due to unpatched BIOS, which was patched on 4/24/14):

http://www.newegg.com/Product/Product.aspx?Item=N82E16813182819

 

Tyan C226 based board (I've had nice success with Tyan the times I have used them)

http://www.newegg.com/Product/Product.aspx?Item=N82E16813151297

 

ASRock C226 based board as referenced in the OP:

http://www.newegg.com/Product/Product.aspx?Item=N82E16813157410

 

SuperMicro C226 based board (good reviews, two 3.0 x8 PCIe slots, one 2.0 x4 slot)

http://www.newegg.com/Product/Product.aspx?Item=N82E16813182822

 

Another SuperMicro C226 based board (not sure I care for the Thunderbolt eating PCIe lanes up though, disable perhaps?):

http://www.newegg.com/Product/Product.aspx?Item=N82E1681318283

 

Yet another SuperMicro C226 based board (negative reviews include bent pins, likely person using wrong memory [mentioned trying non-ecc, when the board clearly needs ecc], etc. though good reviews as well, has legacy PCI which would be good for those with tuner cards, etc. UEFI BIOS)

http://www.newegg.com/Product/Product.aspx?Item=N82E1681318283

 

On a side note, it seems this guy totally bypasses the C224 chipset as well.  Says the C222 and C226 is fine to use, despite the C224 chipset and boards (SuperMicro being no exception) existing at the time the C222 and C226 chipset based boards (that he recommends) were produced.

 

http://wahlnetwork.com/2013/12/02/new-haswell-fueled-esxi-5-5-home-lab-build/

 

I may just have to change out my motherboard currently in my unRAID (and I just updated that darn box with it and a E3), seeing that I'm just about convinced at this point that the C224 based SuperMicro board I have in it:  http://www.newegg.com/Product/Product.aspx?Item=N82E16813182820, is not fully Vt-d compatible.  Not the weekend I planned. <grin>

 

Am leaning towards the ASRock for the lab machine.  Since the mobo in the unraid is beyond the reseller return period, I'll wait to see what luck the ASRock has on running the free version of unRAID before I turn it into a lab machine (albeit, someone has recently level 2'd the ASRock C226WS without the plus at the end).

[grandprix]

Seems the only difference between the Asrock C226 WS and the Asrock C226 WS+ are the following:

 

Asrock C226 WS: 2 x PCIe 3.0 x16, 1 x PCIe 2.0 x4, 3 x PCIe 2.0 x1, 1 x PCI

 

Asrock C226 WS+: 4 x PCIe 3.0 x16 (x16 slots but running in x8)

 

[garycase]

If the goal is a superb long-term virtualization setup, going with an E5 Xeon would certainly be attractive, although as you noted it'd have a notably higher up-front cost.    But with buffered RAM support you could grow to prodigious amount of RAM with no concern over bus loading; and of course the E5's are much higher bandwidth systems than their E3 cousins, thanks to the quad-channel memory controllers.  Note that the E3 boards all use unbuffered RAM, so there's some degradation of performance if you install more than 2 modules due to the high bus loading.

 

But between the higher cost board, the higher cost processor, and the higher cost memory you'd add close to $1000 to the cost of the system ... although if you bought a dual CPU capable board you'd have a VERY nice upgrade path, as you could simply add another CPU 

 

For an UnRAID system, I'd just use an E3; but for a serious virtualization platform I'd think seriously about using an E5-based system.

 

[grandprix]

Ok gary, we have an official answer to the question regarding the chipset, Vt-d, etc.

 

http://lime-technology.com/forum/index.php?topic=35820.0

 

In short:  Gen 3 (aka v 3.0) PCIe slots are controlled by the CPU.  Gen 2 (aka v 2.0) PCIe slots are controlled by the PCH (which evidently is NOT in the CPU as I had read).  The "chipset" (i.e. 222, 224, etc.), specifically, the C224 which is what we were discussing, controls only the 2.0 PCIe lanes.

 

See, I thought I had read that, the PCH was in the CPU, leaving only the chipset (or old "south bridge") that was external.  Evidently, I had that all wrong.  So that didn't help matters what-so-ever.

 

As to the C226 chipset, if looking at the ASRock's board is any indication, I believe it's safe to assume that "it" supports up to an additional 16 lanes on top of what, say, the E3-12xx series supports.  So, 32 total lanes.  I personally would still be interested in knowing which lanes the CPU controls directly as it would be one less "layer" the PCIe device has to go through, I suppose.

 

[garycase]

Nice to know that you can pass at least SOME of the slots through via vt-d 

 

This reminds me a bit of the early PCI slots, when some were bus-mastering and some weren't ... so you had to be sure to install certain types of boards in specific slots or they either wouldn't work or wouldn't perform well.    Been quite a few years since that was an issue, but for whatever reason this discussion made me think of that.

 

It's still a very nice board ... I wouldn't "dump" it too low -- you can always use a spare board;  or perhaps a spare system if a nice sale comes along on a nice Xeon 

[grandprix]

Nice to know that you can pass at least SOME of the slots through via vt-d 

 

This reminds me a bit of the early PCI slots, when some were bus-mastering and some weren't ... so you had to be sure to install certain types of boards in specific slots or they either wouldn't work or wouldn't perform well.    Been quite a few years since that was an issue, but for whatever reason this discussion made me think of that.

 

It's still a very nice board ... I wouldn't "dump" it too low -- you can always use a spare board;  or perhaps a spare system if a nice sale comes along on a nice Xeon 

 

Ugg.  Dont tempt me.  The FS thread doesn't seem to be snagging anyone, probably due to all my replies in it.. but I cannot in good conscious sell it without ensuring someone is more informed than I was when I bought it.  It's not a bad board, not at all if all one needs is two slots (one HBA, one NIC/multi-NIC card). <shrug>

 

Anyway, just glad it's all clear now.  I'm waiting on a 9260-8i to arrive and then the "fun" begins on the box with the ASRock.  32 PCIe lanes of goodness.

[garycase]

I suspect the lack of movement on the FS thread is due to the pricing - it's not a "bad" price ... your logic in arriving at it is sound;  but it's also not a compelling deal.  Newegg sells this for $179.99.    Many folks who buy a lot at Newegg (which likely includes a large number of the folks on this forum) get frequent  "... Thanks! Use These Coupons to Save More on Your Next Component Purchase."  e-mails => I had one last month that included a "15% off all server motherboards" offer.    So if I bought it with one of those coupons, it'd cost $152.99  ... and has free shipping.    That's probably LESS than what your $144.49 plus shipping would cost.

 

I'm not trying to negotiate a price -- as I already noted, I don't need the board, and wouldn't buy it at any price.  [Well, maybe $10  ]    I just thought you might want to re-think the price if you really want it to sell ... although as I already noted, it's a very nice board, so I think just keeping it as a spare or for another one-of-these-days systems is a reasonable idea.  Personally, I'd have just "eaten" the restocking fee and returned the board.

 

I agree that your added info r.e. the doubts about the board's features may have also eliminated a few buyers who might have been interested otherwise.  [Anyone who's been following your threads would already know that; but not everyone would have been aware of those.]