Going Virtual - ESXi 5.0 on 990FX

[Dephcon]

I have my VCP and use vSphere extensively at work but never really had a need to use it at home... So I just so happened to find myself at a computer shop on my lunch break and FX-6300s and 990FXA-UD5 mobos were on sale.  I just couldn't resist. 

 

So currently my setup looks like this:

 

CPU: FX-6300 @ 4GHz

Mobo: Gigabyte 990FXA-UD5

RAM: 2x4GB (4x8GB Patriot Viper 3 1833 on order)

VMDK: crappy old seagate laptop hdd for esxi datastore (to be replaced with 7200.11 RAID1/10/5 short term, SSD long term)

unraid storage: 3x1TB 7200.11 via LSI HBA pass-through

OS: ESXi 5.0U2 installed on USB key (to be replaced with Patriot Supersonic XT USB3.0)

 

So far it's working out pretty good.  I have a test unraid VM up with HBA and USB pass-through and and ubuntu VM running sabnzbdplus mounting unraid via NFS.  The sab box downloads to it's local disk then copies completed downloads to the user shares, it's a bit slow to repair/extract on my current VMDK volume, but the NFS transfer speed was pretty good (25MB/s (no cache)).  I also installed the vCenter Server appliance so I can get all the pretty performance graphs for my host/VMs. 

 

I'm hitting about 6GB RAM used on the host so I've ordered more RAM.  My HTPC had 4x4GB so I just pulled half the sticks to get this up right away.  Overclocking to 4GHz was pretty trivial, I need to find another HDD so I can boot bare metal win7 and do EXTENSIVE stress testing at higher clocks.  I have a Nocuta D14 collecting dust that I'll use to keep the CPU temps down.

 

more to come

[Dephcon]

Turns out that the Patriot RAM (PV332G186C0QK) wont run with all 4 DIMMs at 1866.  I'm not shocked as it's not at all certified for AMD, but i was just hoping it wold work.  I've ordered a Kingston 1600 kit with ECC (KVR16E11K4/32) for a little extra money and will give me some piece of mind.

[Ford Prefect]

...really interesting build.

 

Please report on how things work out with using ECC-Memory, is it is not listed as being officially supported.

 

Another thing, I am curious about is idle power draw...do you have measures and did you customize ESXi power settings in order to use the C-States of the CPU?

[Dephcon]

Yeah I'm really curious about the ECC myself.  There's reports of users being successful with the older UD3/UD5 revisions, I don't think any had a newer rev with UEFI.

 

Currently I have the c-states turned off as well as any other power saving features as I was booting into windows for overclock stability tests.  I was actually able to do 4.8 stable with a Noctua D14, but there's no way in hell I'll run ESX with that.  4.0 or 4.5 sound pretty nice though

 

I will look into whether or not using c-states in esxi is worth it, my UPS should give me a good idea of the draw, but currently my power is included in my rent so it's not really an issue.  I do like to idea of being efficient, but not if it negatively impacts the VMs.

[Dephcon]

So it turns out that an old PCI-e SSD card I have works with ESXi for VMDK datastores!  The bad news is that I need an extra PCE-E slot for my final design.

 

I went to the local shop where I bought the CPU/Mobo and I was able to return the mobo and exchange the CPU for a FX-8320 without any restocking fees.   I was able to get the 990FXA-UD7 online for only $30 more than the UD5 and I get an extra PCI-E x8 slot for a total of four plus two PCI-E x4.  Before I left I noticed on their site that the FX-8320 was on sale for $30 more than the FX-6300 so I figured i might as well upgrade as vCenter Server was reporting CPU Ready numbers that I wasn't entirely happy with.  I'm used to working with 12/16 core Xeon 5660/E5-2600 system so I might be a bit too picky.  Also grabbed a pair of Patriot Superspeed XT 32GB USB 3.0 keys for booting ESX and whatever (also on sale).

 

Hopefully I get everything in the mail by next Friday so I can have a weekend to get set back up.

[Dephcon]

Just had an amazing idea - Create a virtual disk on my SSD datastore and mount it as the cache drive in the unraid VM.*

 

I can't wait for the new gear to get here!

 

 

 

*patent pending

[Dephcon]

Finally got the UD7 today!  Boots up no problem with the Kingston ECC RAM that I purchased and it appears to be supporting the ECC features:

 

Physical Memory Array: #44
    Use: 0x03 (System memory)
    Location: 0x03 (Motherboard)
    Slots: 4
    Max. Size: 32 GB
    ECC: 0x06 (Multi-bit)
  Memory Array Mapping: #45
    Memory Array: #44
    Partition Width: 255
    Start Address: 0x0000000000000000
    End Address: 0x0000000800000000
  Memory Device: #46
    Location: "Node0_Dimm0"
    Bank: "Node0_Bank0"
    Manufacturer: "Kingston"
    Serial: "D2306604"
    Asset Tag: "Dimm0_AssetTag"
    Part Number: "9965525-055.A"
    Memory Array: #44
    Form Factor: 0x09 (DIMM)
    Type: 0x18 (Other)
    Type Detail: 0x4080 (Synchronous)
    Data Width: 64 bits (+8 ECC bits)
    Size: 8 GB
    Speed: 800 MHz

[Dephcon]

How many vCPU are people using for unraid?  I have it set for two now, but I think it might be overkill and is probably increasing ready time.

[ZeroK]

Depends on what unraid is being used for.  If it's straight unraid no plugins then one cpu is plenty as it's never going to really need extra cores but assigning an extra one isn't necessarily a bad thing but I doubt it will ever use it to it's full potential.  If you're going to use unraid for mysql, sab, sickbeard, etc., then assigning more cores is a must.  All depends on what you will use it for as to whether you'll need more cores or not.  I would start with two and see how it goes, if it's not using them much then take it down to one.

[Whatever]

Was the FX-8320 worth the jump?

[Dephcon]

Just plan 'ol unraid, plugins a seem useless if you're virtualizing. I'll move it down to one for now.

 

As for the upgrade, I'm currently only running 7 single core VMs. However I think the ready time is a bit lower. Time will tell when I start adding more services, but I think it was well worth the extra $40.

 

 

[Dephcon]

I RMA'd one of my 2TB EARX that's been reporting slow speeds with diskspeed.sh.  The replacement drive might be bad as it caused a kernel panic when I tried to rebuild that array with it in place of the slow drive.  I totally forgot to pre-clear first, so I did that and it choked on step 10 saying something about the MBR.  I have this feeling in my gut that this is a esxi/passthrough issue, I don't know why but I wont be shocked if that's the case.  I'm going to setup a test unraid box with my old mobo/cpu tonight and see what's up.

 

Also, it's quickly become apparent that I need hot swap bays with ESX as I need to unplug 4 drives to replace one and I don't like doing that with the server powered on.  Luckily my birthday is Oct 4 and I've convinced my family to give me cash only to fund a Lian-Li D8000 with the additional BP3SATA and BP1SATA hot swap backplanes.  I'm just trying to find a buyer for my RC590 w/bays to help reduce the overall cost of the Lian-Li.

[Dephcon]

I was right?

 

Booted up a bare metal unraid server and added/formatted the drive no problem.  Then I replaced the bad drive in my prod server and it worked fine this time.

 

Pre-formatting on bare metal might have fixed the issue I was having, but having to do this every time I want to add a new/replacement drive isn't going to fly.  I guess I wont really be able to test further until I need to replace a drive again.  Hopefully this wont be until I can replace my 1.5TBs with 4/5TB Reds this fall/winter.  Also ESXi 5.5 should be out in a few weeks so maybe it is a pass-through issue that is fixed.

 

*EDIT*

with the new drive in, the rebuild process is going SUPER fast (105-108MB/s), which is a huge improvement from before.  I'm glad I decided to RMA that bum disk.

[BobPhoenix]

I would upgrade your HDD controller to latest firmware.  Also update SAS Expander firmware (if you are using one).  I had the same problem and that is what was suggested to me to do.  I haven't actually done that but I did swap to a different SAS Expander (same model) and the problem went away for me.  It could also have been the drive as when I was having problems it was with a WD but when it worked for me it was Hitachi.  I usually preclear my drives from a standalone PC anyway so don't do that in my servers very often anyway.

[Dephcon]

I'm now thinking that one of the eight SAS channels on one M1015's connector/PCB is bad. I swapped the cables but I was still getting errors and a corrupted MBR on one of the ports.  The cards were freebies so I'm not super torn up over it.  For now I'll just avoid that port until I can do further testing, i just want to get this thing back in action.

 

On another note, I got my backplanes installed and got a steal of a deal on six Nexus 120mm fans for the front intakes, 9.88CAD each.  I bought eight but two are damaged and unfortunately they're on clearance at a local retailer so getting an exchange is not possible (I bought their entire stock).  I've tried emailing Nexus about an RMA but haven't heard back yet, but I have another 10 days or so to get a refund.

[KC]

I'm running similar hardware, with the IO/MMU support on the 990fx & 8350, it becomes a nice ESXi server.

 

Few caveats I noticed - first time I fired it up I had horrible performance, I had to disable S*(crap was it S3 states? - Next time I reboot it I'll check... but could be months down the road now that it's stable) in the bios, otherwise it ran at 400 mhz. After I figured that out and enabled cool & quiet, and IO/MMU in the bios, it's worked quite well.

 

Currently it's passing through a M1015 with 8 drives, and hosts 3 drives on the local sata ports.

 

GIGABYTE GA-990FXA-UD3 AM3+

32GB ram

M1015 running in passthrough.

Generic ATI video card (I can't even remember the model #, enough to pass bios, all I care about)

3 drives on onboard SATA (Corsair Force 240gb, 2x500gb Hitachis - cache for Unraid and data drive for my encode box).

 

3 Guests -

Ubuntu 'mess around server'

Windows 7 Encoding server ---- Keep in mind this server can utilize from 50% to 100% of the 7 cores allocated to it. It's uptime is 4 days right (once I got past a couple of config snags (I'm looking at you SASLP) )

Unraid. (I've found a single 8350 core on this can utilize up to 30% total CPU usage, so no need to over-commit the cores - Leaving 7 cores for my encoding guest)

 

I plan on adding a DC to it eventually. One day, when I get the time and energy.

 

... now all I need to figure out is if I can pass through my ceton tv-tuner.

 

I'm now thinking that one of the eight SAS channels on one M1015's connector/PCB is bad. I swapped the cables but I was still getting errors and a corrupted MBR on one of the ports.  The cards were freebies so I'm not super torn up over it.  For now I'll just avoid that port until I can do further testing, i just want to get this thing back in action.

 

On another note, I got my backplanes installed and got a steal of a deal on six Nexus 120mm fans for the front intakes, 9.88CAD each.  I bought eight but two are damaged and unfortunately they're on clearance at a local retailer so getting an exchange is not possible (I bought their entire stock).  I've tried emailing Nexus about an RMA but haven't heard back yet, but I have another 10 days or so to get a refund.

[BobPhoenix]

... now all I need to figure out is if I can pass through my ceton tv-tuner.

I've had good luck so far myself.  As long as the MB has been compatible with the tuner it passes through.  My AverMedia duet doesn't work with Ivy Bridge compatible bios on Tyan and SuperMicro boards.  It does work on Intel board once I upgraded the bios.  Hopefully an AMD based board will not have those dificulties.

[Dephcon]

I have a case open with Gigabyte regarding a triple boot issues I'm having with my UD7.  When I'm using all the x8 and 16x slots, it does a triple post/boot, I'm hoping it's a BIOS thing that can be fixed, which isn't a deal breaker but that's a 2 extra power cycles on my HDDs that I would like to avoid if possible.  Gotta try musical chairs with my cards to see if I can get it to boot properly, didn't have this issue on the UD5.

[Dephcon]

Turns out my BIOS wasn't latest, a quick flash and I'm booting normally.  Also got four 1M SAS fanout cables in the mail today.

 

I'm super close to finishing this build.  Just waiting on wire/connectors for custom power cables and some power distribution PCBs for my fans.  Then it's picture time!