ATLAS My Virtualized unRAID server

[redia]

Johnm & all,

 

sometimes the simplest explanation is the good one.

intel core i3 340 does not support vt-d.... I have to change my proc if I want to run passthrough.

 

I have one x3470 ordered as I planned to upgrade my proc (going to build another unraid which does not need to be running esx anyway)

 

I will let you know as soon as I receive it.

 

Cheers,

R

[Johnm]

this is correct.

The I3 series (with the exception of some OEM dell/hp server I3's you cant buy) and unlocked I5/I7 chips do not support VT.

 

Intel has some sort of master plan of taking the same chips and adding/removing or turning on and off features to sell them at different price points.

 

 

 

 

[madburg]

I cant believe I missed this thread, great work Johnm!!! I was going to be posting a Ultimate Virtual Beast build, with alot of what you posted. So that saves me a ton of screenshots and explainations  , I will just forward to your thread, LOL.

 

Great stuff. You guys also ran into what i did with the booting off the vmdk to speed things up and then the usb stick for unRAID license, packages, etc...

 

Johnm, I actually needed shorter SAS cables so I have many long one sitting around. PM me you email address and what size your current cables are tip to tip, I will email you the lenght of mine and how many i have tonight, I give you a good price on them.

[Johnm]

Hey thanks.

 

I still need to do backup tut and some alternative methods others have suggested to Do differently then what I posted.

 

Monoprice got 1m cables back instock for under $10 each.

I ordered 5. I thought a spare would be a good idea.

[madburg]

No problem, I thought you were stuck with getting the cables and I have not had the chance to post them up for sale, so I figured who better than u 

 

Glad you got them. I always buy a spare myself.

[mattsplat]

Brilliant work Johnm, many thanks!  Forgive my extreme n00bishness, but looking to do something similar myself (Win7 for HTPC/WHS 2011/unRAID), and of course, questions arise ...

 

For reference I'm thinking of using this hardware:

i7 920 (Socket 1366), ASUS P6T Deluxe, 12GB Corsair 6x2GB, 60GB Vertex2 for as many OS's as will fit, other drives/controllers TBD

 

#1) Is VT-d required?  As I understand it, you're giving direct hardware access to the storage controllers, and I don't know if VT-d is needed for this.  I ask b/c ASUS "doesn't support Linux" & therefore won't update the piddly BIOS problem needed for VT-d to work properly ... so I'd need a different mobo.  The processor and even the chipset support VT-d *sigh*

 

#2) I see you are using 3TB drives (http://lime-technology.com/forum/index.php?topic=14695.msg139662#msg139662) ... does this work with an 8MB blocksize for the ESXi datastore drives?  Or does this not matter b/c unRAID is seeing the drives directly?

 

#3) Will 12GB of RAM be an issue? (ex: 2GB for WHS2011, 2GB for unRAID, Win7 gets the rest)

 

#4) Might be outside the scope of this thread ... what about using VMware?  I ask b/c I'd like to keep native Win7 performance for media serving, or even games (the server will be in a closet w/ the equipment hooked to my projector ... slapping a Radeon 6950 in this would solve all four needs - gaming/media/backup/file server FTW!)

 

Again, thanks in advance.

[BRiT]

VT-D is required for device pass-through.

[Johnm]

Brilliant work Johnm, many thanks!  Forgive my extreme n00bishness, but looking to do something similar myself (Win7 for HTPC/WHS 2011/unRAID), and of course, questions arise ...

 

For reference I'm thinking of using this hardware:

i7 920 (Socket 1366), ASUS P6T Deluxe, 12GB Corsair 6x2GB, 60GB Vertex2 for as many OS's as will fit, other drives/controllers TBD

 

#1) Is VT-d required?  As I understand it, you're giving direct hardware access to the storage controllers, and I don't know if VT-d is needed for this.  I ask b/c ASUS "doesn't support Linux" & therefore won't update the piddly BIOS problem needed for VT-d to work properly ... so I'd need a different mobo.  The processor and even the chipset support VT-d *sigh*

As I recall,

VT-i is required for 64bit guests to run.

VT-d is required to passthough hardware directly to the guest.

 

I am hearing a lot of what you are saying about desktop boards. the boards are listed as having VT-d, and it is not in the bios or enabled. they built the board to the specs, but don't give it to the customer. I'd stick to serverboards or one thats proven to work.

check the ESXi whitebox forums for help finding a board.

 

 

 

#2) I see you are using 3TB drives (http://lime-technology.com/forum/index.php?topic=14695.msg139662#msg139662) ... does this work with an 8MB blocksize for the ESXi datastore drives?  Or does this not matter b/c unRAID is seeing the drives directly?

ESXi does not see 3TB drives (unless this is changed in 5.0). I am passing my HBA's though to the unRAID at the PCIe bus level so the entire raid card and drives are inside unRAID.

 

#3) Will 12GB of RAM be an issue? (ex: 2GB for WHS2011, 2GB for unRAID, Win7 gets the rest)

 

The more you have the better obviously. but I have have 16Gigs of ram. I have 8 to unRAID, 8 to my usenet client, 4 to WHS, 2 to several other guests.. the ram is shared between guests. that's the max it is allowed to use. You can also set a minimum if you want so a guest always has a dedicated XX amount of ram whether it is in use or not.

 

You'll find in average use, most of your guests are not running at 100% RAM usage so you can have overlap.

 

you run into issues when you have all the RAM in use at once and need even more then you have.

ESX just tells the guest, "i know you have 8 gigs of ram, but 3gigs (for example) are in use by an unknown process, deal with it and use the leftover 5gig". Hence the minimum setting. I believe ESX 5 now has SSD caching to help.

 

My second ESXi box has 12gig also. but is hardly ever turned on. i use it entirely for testing desktop stuff.

 

#4) Might be outside the scope of this thread ... what about using VMware?  I ask b/c I'd like to keep native Win7 performance for media serving, or even games (the server will be in a closet w/ the equipment hooked to my projector ... slapping a Radeon 6950 in this would solve all four needs - gaming/media/backup/file server FTW!)

 

Again, thanks in advance.

 

Running a hypervisor at hardware level (ESXi) is always much faster for all the guest, you can hardly tell it is a VM guest when you use it.

Running a hyperviser on top of an OS is no where close to the same efficiency. any lag in the host is also felt in the guest. also I don't believe you can have full hardware passthough since it is controlled by the underlying OS and not VMware/Hyper-x

 

Windows on ESXi is pretty close to the same as on bare metal. also, you can pass video cards/capture cards through ESXi. you might need to check first before you buy a new card. so you could in theory put a HTPC on ESXi.. or at least a transcoding or PVR recording client...

[mattsplat]

As I expected, thanks for the quick reply.  I'm looking at running 3 VM's max so looks like 12GB is enough.

 

For now I may just run Win7 + WHS2011 with storage on a standard hardware RAID card, not seeing any x58 (VT-d working) motherboards right now, and rather not switch the entire platform yet.

[fade23]

Has anyone attempted an upgrade to ESXi 5.0 yet? In particular, I'm curious to know if the "format all unknown disks" behavior is still present, either on a fresh install of 5.0 or an upgrade from 4.1

[graywolf]

I'm doing a new (first) build so doing 5.0 although I did not have any Drives attached during the install (per the tips earlier in the thread).

 

So I can not answer your question.

[Johnm]

Has anyone attempted an upgrade to ESXi 5.0 yet? In particular, I'm curious to know if the "format all unknown disks" behavior is still present, either on a fresh install of 5.0 or an upgrade from 4.1

 

It works just fine. It did complain about the fact I had OEM drivrs (for my second NIC) and  removed them durring the install.

 

I just mounted the ESXi 5.0 installer ISO in the IPMI virtual CD drive and booted to it.

then followed the prompts.

[jimwhite]

As near as I can tell, 5.0 limits the RAM per CPU to 8gb, whereas 4.1u1 allows 16gb.  I don't think Ill be upgrading as my box has 16gb installed. 

[Johnm]

As near as I can tell, 5.0 limits the RAM per CPU to 8gb, whereas 4.1u1 allows 16gb.  I don't think Ill be upgrading as my box has 16gb installed. 

 

I believe that limit was upgraded to 32Gig in the release version due to consumer backlash.

[Zeron]

As near as I can tell, 5.0 limits the RAM per CPU to 8gb, whereas 4.1u1 allows 16gb.  I don't think Ill be upgrading as my box has 16gb installed. 

 

I believe that limit was upgraded to 32Gig in the release version due to consumer backlash.

 

It was upped to 32G before release

[graywolf]

Trying to create my first VM from an existing vmdk  (ESXi 5.0)

 

I'm unsure of what to put for the SCSI Controller Type and for the Virtual Disk, what does the SCSI (#:#) mean?

 

Keep getting Unable to create Virtual SCSI device for scsi0:0.

[redia]

Johnm & all,

 

sometimes the simplest explanation is the good one.

intel core i3 340 does not support vt-d.... I have to change my proc if I want to run passthrough.

 

I have one x3470 ordered as I planned to upgrade my proc (going to build another unraid which does not need to be running esx anyway)

 

I will let you know as soon as I receive it.

 

Cheers,

R

 

this is correct.

The I3 series (with the exception of some OEM dell/hp server I3's you cant buy) and unlocked I5/I7 chips do not support VT.

 

Intel has some sort of master plan of taking the same chips and adding/removing or turning on and off features to sell them at different price points.

 

 

 

 

 

sorry for the late reaction/response..

 

for the people who doubted it.

I just plugged a xeon in my box.. and as planned passthrough is now working like a charm...

 

I hope this will save some trouble to some of you.

 

Johnm,

maybe you could add a line in the first page of your topic... all up to you..

 

 

Cheers,

R

 

 

[siamsquare]

Yo,

 

Thanks Johnm for all your good advice on hardware...been reading alot lately between different systems and Unraid still can't convince me concerning transferspeeds.

I own several NAS systems which are really fast but they are not easy to upgrade....you need to put equal disks in size in your Raid-5

So I've been looking at ZFS and RAID-Z, a software variant of raid-5 and I decided I want the speeds from my NAS systems or faster combined with good redundancy

The only aim is to store lots of media like BD-iso's and MKV and stream them preferably by NFS to several players in the house.

As I test alot of mediaplayers and NAS for a site, I sometimes need to move around lots of data so I don't want to be confined to Unraid speeds.

I found some good resource like Napp-it.org on ZFS based servers and I was wondering if it wouldn't be better for me to choose this path instead of going Unraid

 

I would like your opinion on this matter as you have tasted both worlds....

 

gr33tz

 

PS: My system would compose of a Norco-4224, Supermicro X9SCM-F , an E-1200 series xeon, amount of ram...not sure yet and an SSD for boot? 

 

 

 

 

 

 

 

[youngm]

I did some research a while ago on ZFS and it appeared to me that ZFS's RAID pools had the same constraints of regular raid.  (e.g. If RAID 5 you need at least 3 drives of the same size, you lose storage for one drive per RAID 5 pool, etc.)  If that is still the case I don't see a big difference between ZFS and your current NAS solutions.

 

Does Napp-it offer something more unraid like with ZFS?

 

Mike

[Johnm]

I run both hardware raid5/6 and unraid.

 

each server has a very different job in my house.

 

I use my unraid for long term storage. think of it as more like a 4TB-60TB CD-RW or WORM drive (Write once, read many. [thx whoever said that before])

 

I use unraid for media shares, archiving and backing up of other servers, storage of ghost images, ISO files ect.  (usually the files on my unraid come from another server first)

 

I also use several smaller 4 to 8 drive RAID5/6 arrays for my day to day file serving, sorting of  files, raring, paring, ripping,  heavy IO traffic.

any files i do not plan to use for a while. then gets moved to unraid off the Raid5/6 boxes.

 

my second unraid is only backups of unraid #1 and my raid 5 arrrays. it is off much of the time.

 

I was thinking about building a 4x 3TB drive unraid just for my newest movies so my main unraid can sleep when not in use. ESXi sort of killed that option because i cant power it off the unraid VM for power savings.

 

The reason i use 2x 20 drive unraid builds is electricity.

Atlas uses about 120w on the average. about 100w when mostly idol.

A 20 drive ZFS or Raid5 array would pull closer to 200+w on the average.

over a year, that adds up fast in power costs.

 

A small 4 drive raid5 server is extremely fast, and i can live with the power usage of such a small box.

I try to keep these severs as empty as possible.

 

Also price is a factor,

ZFS needs massive CPU, RAM and usually large HBA's/Expanders for large arrays.

Hardware Raid needs massive expensive controllers and possibly expanders.

unraid runs on about anything quite well.

A $700-$1000 server beats a $2000-$3000 server anyday when it comes to my wallet.

 

I have come to realize that for files i dont edit or rewrite often, have no reason to be on a mega power hungry expensive speed beast.

 

 

 

Unraid is not for everyone. figured out your personal needs. make a list of of pros and cons for each solution and think about it.

You can always milk much better performance from unRAID by building with 7200 RPM drives.

 

It sounds more like you need a production server and a lab/test server.

 

mixing the 2 is generally not a good idea. that is how data gets lost or hardware broken.

[madburg]

+1, running a HP Server and additional storage array, off loading to the unRAID server as well as backups. And by virtualizing unRAID I get to squeeze out additional secondary and/or non-critical guests out of the chassis.

[znelbok]

Hi Guys

 

Let me start by saying that this is one of the best documented threads I have seen.

 

My hardware list is pretty much idential, excpet for the processor and SM cards.  The processor is just an i3 and the SM cards are the SATA3 updates.  Since I have not built the box yet (waiting on the case to arrive - been 2 months now) I am in a position to change quite easily.

 

Now I run a server at home for the home automation system I have.  It has an 8 port serial card installed and uses a 32 port Digi etherlite (terminal server, 32 serial ports over ethernet).

 

I am interested in combining the two of them together, and I think by looking at what is here its not going to be too hard.

 

I have a small amount of experience with ESXi, did my first install at work last week, and I have been using workstation and server for a little while, so ESXi was not that hard.  I did learn a bit about the raw device mapping, which is good.

 

My question - Do you huys know if I can install the serial card and pass all the serial ports (10 off, 2 on MB and 8 on card) through to the guest OS.  I am assuming that ESXi would need to have the driver for the serial card to work - is this thinking correct?

 

Why use an SSD for the datastore, is this just to increase performance.  What about longevity with windows using it for its swap drive (by cutting down on the RAM to the WinOS, doesn't the swap drive get used a lot more).  Would a normal drive be better in this case?

 

Mick

[Johnm]

Hi znelbok,

 

Unfortunately, the I3 wont work. VT-d is not enabled on those chips.

 

As far as the new SM card and the Serial card, I cant tell you if those are compatible.

IF they do passthrough, the entire card will be inside the guest and work as it would in a raw OS.

the  best way is either google or just try it out on a test box or test drives in you main box.

 

The SSD for the datastore is for performance. the biggest bottleneck of a PC is the hard drive. now imagine 3-5 PC's sharing the same  bottleneck. I do disable the page file on my windows guests. also ESX 5 has SSD optimizations. Yes, I am sure my disks probably wont last as long in an ESX box as they would in a stand alone PC, but by the time they fry, newer, faster, cheaper SSD's will be out so.... ill deal with that when i get there. I would prolly do that upgrade before i even get there anyways.

Also, the savings in cost (hardware & power) of consolidating 8 PC's into one box will give me a few extra dollars in my wallet for more beastly hardware when it comes out.

 

 

Also, most of my windows guests don't really do that much.. The 2 biggest hogs are my usenet client and WHS2011. that WHS has its own hard drive and the usenet box... well it is what it is.. high disk IO's and lots of ram...

 

 

 

 

[znelbok]

Thanks for the reply

 

I was planning to replace the i3 with a xeon should I go the VM way - I'll redeploy the i3 into a media center.

 

I will do a test run before I build it all up.  That will give mre an idea of how the cards will work, but I suspect that if they are working in the previous version the newer one should work fine.

 

Mick

[Madhouse]

Very nice thread..

 

I just tried ESXi with my AOC-SASLP-MV8, didnt work. so now i might need to get one of those IBM m1015

 

however, i do have some concerns about using it with green drives.. i have a 8x1.5tb raid running on a Adaptec 5805 and i know for sure it killed a couple of my green WD15eads.  not entirely sure but i think it has something to do with TLERa sort of self repair that raid card only allow for 7-8 seconds but the green drives can take much longer and are therefore flagged as faulty..

 

do you have any problems running green drives with the ibm..

 

Best regards

 

Mads