New budget ESXi build

[evlo]

I would like to to build me home server for media storage and torrenting.  So these are main software goals. And also plex media server would be nice.

 

I finished the planing phase, but still want you to check, if you are willing, if i missed something.

 

Well first thing i decided is that i want storage (unraid) separate from other system.

 

My budget is around 800USD.

 

I do live in EU and some things are hard to source or very expensive if i count in shipping and added VAT - ie. Supermicro AOC-SASLP-MV8, norco ss-500, etc. Also i'm sorry if some note from my planning is left untranslated to English.

 

So the software achitecture i imagined like this:

ESXi5 runing from stick

Linux (probably debian) running rTorrent with ruTorrent, flexget and plex media server (not real need for transcoding) with one dedicated HDD for torrents that are moved to unraid storage after some time installed on physical usb stick

unraid installed on physical usb stick

Images of usb stick would be backed up to dropbox

 

So i tough that all hdds should run as ahci and would be mapped to virtual machines as physical drives.

 

Hardware Goals:

- relatively nice looking case (minimalist design)

- minimal maintenance

- energy efficient

- As quiet as possible

- maximal possible reliability

- still keep the budget

 

The actual setup i would order:

RaidSonic Icy Box IB-545SSK - 3pcs - cheapest 5in3 available here, also hotswap without need of tray, disadvantage: dual 6cm fans. If i will be unable to get these i would get Lian Li EX-H34B , which cost 1/4 less here, but are only 4in3.

Cooler Master Centurion 590 - i did not found case with 9 basy without so many holes in it and power button on front and still for a good buck.

Chieftec Smart Series GPS-600A8 600W - should have  1x12V 46A

ST-LAB A-214 - 3 pcs - cheap and based on SiI3114 - support in ESXi 5

dust filtr 120mm - under psu

AMD A6-3500 - dunno how to split cores between VMs

4GB DDR3 1333  - 2 pcs

GA-A55-DS3P - ESXi will probably need to bu customised to support Realtek NIC, 6x sata, 3x PCI

Cables - moles and SATA - already have these

USB sticks - already have these

 

I will start with just 8 drives, but i would like possibility to expand it - so that is why i would go for 3 5in3 cages.

 

Also some drives will be 3TB 7200 seagates as these are only 3 platter and WD greens are 4 platters.

 

Would it be possible for A6 to hande 3VM - 1. unraid, 2. torrent box, 3. PMS?

 

So did i forgot to think about something?

[Ford Prefect]

...I am not on my turf with AMD CPUs, but you would need one with the Intel vt-d equivalent (not just vt-x).

AFAIK this is called AMD-Vi (not only AMD-V) and only available in Opteron family line  of CPU models, isn't it?

[evlo]

Dunno, i'm new to whole ESXi, it just seamed like very nice idea to have it.

[Ford Prefect]

...in your design, as with all ESXi builds, you need to be able to passthrough the real controller(s) to the OS inside the VM.

This feature needs to be supported by CPU, mobo/chipset and BIOS....if you don't have it in one of that three parts, you've lost.

For AMD it is called AMD-Vi or IOMMU.

I personally haven't seen a build based on AMD doing it...most (all?, I that know of) use intel vt-d festured builds.

[evlo]

Thank you, that part was not clear to me at all. So eiter i will run full OS with vmware workstation or chose very different hardware. Until now i tough it will be just like normal vmware on windows where i can assign local physical hdds to running vms.

[Ford Prefect]

---you can, but this is another feature, called RawDeviceMapping.

You can use it in ESXi as well

When you go that route,  you will see a performance penalty and maybe not all features, like SMART reports or spinup/spindown going to be passed through.

The reason for that is, that the VM only "sees" that disk through a software emulated driver.

 

If you want a stock unRAID and other OSes on a single box, go with ESXI and vmdirectpath (for which you need either intel vt-d or AMD Vi/IOMMU support)

[Ford Prefect]

...maybe this will help finding your way: http://lime-technology.com/wiki/index.php/UnRAID_Add_Ons#ESXi

[evlo]

A little, but i'm getting more into it and i'm getting more confused. Now it seams that vt-d is not actually required. Also i think i would use RawDeviceMapping because i would like spare drive for torrent part and if it would be connected to SAS controller than it would be impossible.

[Ford Prefect]

...it'll depend on your requirements if vt-d is required or not...

 

.- do you want a stock UnRaid, no kernel/driver tweaks but all standard features, then you need vt-d in order to virtualize it.

--you depicted that you'll be running your torrent inside a VM...you could pass another controller to that VM as well...or use RDM

and pass a single disk from ESXi to that VM....for torrents, disk performance is not a real  issue.

I maybe would even recommend to add that disk to the ESXi datastore and use virtual disks for your torrent

 

[evlo]

>>> do you want a stock UnRaid, no kernel/driver tweaks but all standard features, then you need vt-d in order to virtualize it.

If it is not too much trouble, can you please explain why? Is it because unRaid don't support some VMWare virtual hardware or just because it needs direct access to HDDs - hence i need to map whole controllers to it?

[evlo]

And one question about PCI speed on normal motherboard(32 bit 33mhz PCI)

 

if i use 2 sil3114, with 8 HDDs connected, then when doing parity check system will need to access all the drives so the real speed would be 15MBps? So then if i would use 3TB drives then parite check would take 64 hours?

[Ford Prefect]

>>> do you want a stock UnRaid, no kernel/driver tweaks but all standard features, then you need vt-d in order to virtualize it.

If it is not too much trouble, can you please explain why? Is it because unRaid don't support some VMWare virtual hardware or just because it needs direct access to HDDs - hence i need to map whole controllers to it?

 

unRAID will support the SCSI virtual hardware...but the connection  will remain virtual...unRAID would not know that the disk is virtual...and that the underlying device is a physical raw device mapped to the virtual disk config.

If you go that route, you will either loose some unRAID features which are amongst the key values of unRAID (S.M.A.R.T. reports and control, like spinup/spindown) or you will need to add the

vmware tools into the unRAID install and run a non-stock unRAID version as a result of that (some users are supporting that by offering pre-build packages for install, but still not a 1,2,click,finish task, I'd say)

I haven't seen many of RDM based builds since the list of hardware working with  vt-d is constantly increasing.

[evlo]

Now i get it, + i can't probably create vmdk bigger then 2TB, so i will need to lurk more and change the build from the basics.

First thing i need to decide is if i would go the adventurous way of AMD-vi, which is about half the price of Intel vt-d for mobo+cpu.

 

Currently i'm thinking more like

Intel Core i5 2400S - 4610 - 65W TDP, intel VT-d, ESXi5 comp.

Intel Crow Point DQ67OWB3 - 2726 - intel nic, intel a VT-d, non ECC, probably ESXi5 comp.

2* 4GB DDR3 1333 - 900

IBM Express ServeRAID M1015 - 3100 - HBA, ESXi5 comp.

Norco RPC-3216 - 9500 - 16HDDs, standard PSU, need to buy 1 sata backplane (found for sale in netherlands)

ST-LAB A-214 - 350 - sil3114, for torrenting HDD

Chieftec Smart Series GPS-600A8 600W - 1238 - 46A 12V, not a total noname builder

 

About $1100 to buy it here, so it seems still quite a budget, but more server like

[Ford Prefect]

The NIC on that intel board will not work with ESXi without tweaks, I am afraid.

...also, for the M1015 you'll better get a PCIe x8 slot (that board has an x16 but it may be working for GPU cards only).

[Johnm]

The NIC on that intel board will not work with ESXi without tweaks, I am afraid.

...also, for the M1015 you'll better get a PCIe x8 slot (that board has an x16 but it may be working for GPU cards only).

 

The true intel boards "tend" to work with raid/hba cards in the 16x slots.

Usually the intel boards tend to be the "reference" boards and all the features work as they should.

 

unfortunately, as Ford pointed out. That NIC does not work without a hack. some people say it will work fine after the hack. some people say it the hack does not work. I do believe i read that it will work after the hack on that board since it is a single nic.

 

[Ford Prefect]

Hmm...wait...here's some stuff regarding ESXi, I/O-passtrough and an AMD based whitebox.

 

http://hardforum.com/showthread.php?t=1376255&page=42

http://www.g33ka.com/2012/08/01/how-to-build-a-esxi-5-0-amd-whitebox-server/

[distracted]

...I am not on my turf with AMD CPUs, but you would need one with the Intel vt-d equivalent (not just vt-x).

AFAIK this is called AMD-Vi (not only AMD-V) and only available in Opteron family line  of CPU models, isn't it?

 

Phenom II + 890FX/990[X,FX] chipsets will work, as long as the MB has a bios that truly supports IOMMU. Finding reliable reports the hard part. I picked up an ASRock Extreme III that supposedly works with directpath but I have another project that I need to complete beforehand. IOMMU is a function of the chipset, with AMD-V a function of the processor. Some Athlon64, Athlon II etc processors support an earlier version of AMD-V.

 

[distracted]

Now i get it, + i can't probably create vmdk bigger then 2TB, so i will need to lurk more and change the build from the basics.

First thing i need to decide is if i would go the adventurous way of AMD-vi, which is about half the price of Intel vt-d for mobo+cpu.

 

Currently i'm thinking more like

Intel Core i5 2400S - 4610 - 65W TDP, intel VT-d, ESXi5 comp.

Intel Crow Point DQ67OWB3 - 2726 - intel nic, intel a VT-d, non ECC, probably ESXi5 comp.

2* 4GB DDR3 1333 - 900

IBM Express ServeRAID M1015 - 3100 - HBA, ESXi5 comp.

Norco RPC-3216 - 9500 - 16HDDs, standard PSU, need to buy 1 sata backplane (found for sale in netherlands)

ST-LAB A-214 - 350 - sil3114, for torrenting HDD

Chieftec Smart Series GPS-600A8 600W - 1238 - 46A 12V, not a total noname builder

 

About $1100 to buy it here, so it seems still quite a budget, but more server like

 

There is no 2TB limit with RDM, just make sure to use '-z' when creating the RDM mapping. Personally I would go with a VT-d/IOMMU setup, though I did use RDM's to test unraid. If you do run torrents in a VM make sure you either preallocate the entire VMDK or use RDM. If not you will be hating life while the VMDK constantly allocates and expands.

[cwesterfield]

Does proxmox handle device passing differently?

 

That might be a better alternative for AMD builds.

[Ford Prefect]

Does proxmox handle device passing differently?

 

That might be a better alternative for AMD builds.

 

IMHO both implementations rely on the same HW feature(s), vt-d or IOMMU respectively.

As a rule, I'd say that if you have a set-up that works with ESXi, it will work with Proxmox, too

XEN might be different AFAIR.

 

Advantange with Proxmox is, that you don't need a seperate Windoze installation for management.

However, passthrough config needs to be added without a GUI, just from CLI into the VM definition file.