ATLAS My Virtualized unRAID server

[redia]

 

I was going to do a writeup on Ghetto. while dong the write up, I corrupted a guests doing the restore.

I am sure it was because i was trying to restore a 4.1u1 guest backup over a 5.0 install.. this was before ghetto supported 5.0

 

They have since upgraded the script for 5.0. I can probably test that out again and let work have that license back.. ( i know they need it back before year end).

 

maybe i can do that this week. i have a lot on my personal to-do list right now.

 

 

 

Johnm,

 

I should have a little bit of available time next week. do you want me to test the export from 4.1 restore to 5.0 ?

I would be more than happy to help if it can save you some work.. this way you can focus on documenting the backup/restore procedure. without worrying about compatibility.

name the type of VM you want to test...

 

I could also look into the inventory restore (which I could document) some people may feel safer to have their config backed up even though I do not feel it is necessary in a 1 server config..

 

Cheers,

R

 

 

[Johnm]

in general, no one should back up on 4.1 then try to restore to 5.0.

i had updated my server the same weekend i was doing the documentation and did not have new backups.

 

my own fault

[redia]

in general, no one should back up on 4.1 then try to restore to 5.0.

i had updated my server the same weekend i was doing the documentation and did not have new backups.

 

my own fault

100% agreed.

unfortunately with the 5.0 being just released I am pretty sure people will forget about that.

anyhow I will simply test on a w2k8r2 just to educate myself.. and I will share the results with you

 

R

[cpshoemake]

Thanks for figuring this out John. Excellent work!

 

I followed your installation instructions and everything seems to work as intended. I don't have any disks installed yet. I'll need a new case to migrate UnRAID to my ESXi server.

 

I do have one question though. Does your unraid vm take forever to boot? I haven't timed it, but it takes something like 5-10 minutes for mine to load bzimage and bzroot.

 

Edit: Nevermind. I found your posts on the topic. UnRAID now boots in 10 seconds, from start button to login prompt.

[Johnm]

yes,

the pass-through usb in ESXi is just a tad better then usb 1.0.

You will have that problem trying to use a usb hard drive or flashdrive with any Guest OS.

 

Personally, i could live with it for unraid since I hardly ever reboot my unraid.

 

with some motherboards, you can pass a complete usb hub though with directpass (just dont pass the usb with your ESXi flashdrive on it or your system is toast)

 

As you found, I actually copied unraid to my sdd for a faster boot (8 seconds?). I prefer this method for speed. The only downside is that you have to make a new boot image every time you upgrade unraid.

[pantner]

yes,

the pass-through usb in ESXi is just a tad better then usb 1.0.

You will have that problem trying to use a usb hard drive or flashdrive with any Guest OS.

 

Personally, i could live with it for unraid since I hardly ever reboot my unraid.

 

with some motherboards, you can pass a complete usb hub though with directpass (just dont pass the usb with your ESXi flashdrive on it or your system is toast)

 

As you found, I actually copied unraid to my sdd for a faster boot (8 seconds?). I prefer this method for speed. The only downside is that you have to make a new boot image every time you upgrade unraid.

 

Will passing through the USB HUB work better that the USB device? I haven't set it up yet, but on my old server i had a USB Dual Tuner which one of the VMs used to record...thats not going to work well on USB1-ish speeds!

[Johnm]

In that case, I would try passing a USB add-on card through. while passing through a motherboard usb hub can be done. it can sometimes have unexpected results, like loosing video.

 

also, some boards may take the entire usb host even if you only select part of it. leaving your esxi unbootable.

 

http://vm-help.com/esx40i/VMDirectPath/USB_Setup.php

[Johnm]

Well,

 

I just got my Supermicro M14TB.. There is always a first time an open box can go wrong. LOL

The SAS cable is missing.. everything is brand new, clearly the box was opened and the cable removed and resealed.

 

Now i need to decide if I want to return it or order the cable ($10). Even if i shell out the added $10, i am still ahead.

[jbrodriguez]

Johnm,

 

Suppose i get a norco 4224 and  a HP SAS Expander.

 

i understand each sas port on the hp expander handles 3 drives (8 ports x 3 drives = 24 total).

 

but the backplane on the 4224 handles 4 driver per sas port, right ?

 

any comments on this "eventual" mismatch .... will i be limited to only 18  drives ?

[Johnm]

lboregard,

 

Each SAS connection is 4 ports. that is part of the SAS standard. The HP SAS Expander can support up to 32 drives off of a single SFF-8087/SFF-8088 cable.

 

Many newer SAS-2 based RAID and HBA cards support link aggregation with two SFF-8087 cables doubling your bandwidth but dropping the maximum drives to 28 drives.

 

If you are truly interested in an HP sas expander, this thread is like the bible for this expander http://hardforum.com/showthread.php?t=1484614

 

It is a very nice expander, overkill for unraid, but nice.

Be aware that there are different flavors and firmware revisions out there.

You need a newer HP Raid card to flash this, making pretty much impossible to re-flash for most people.

 

[Vargas]

Johnm, or anyone:

 

Can you provide link or provide the details on the cost associated w/ ESXi, as well as information which can answer the following:

 

What is the maximum number of cores which can be run for free?

What is the maximum physical Memory i can run for free? I've read that their is limits on vRam but not on physical memory.  Whats the differences?

What is the maximum amount of VM which can be run without incurring additional cost?

Does the free version of ESXI limit the maximum system resources a specific VM is assigned?

 

Thanks in advance

[cpshoemake]

Johnm, or anyone:

 

Can you provide link or provide the details on the cost associated w/ ESXi, as well as information which can answer the following:

 

What is the maximum number of cores which can be run for free?

What is the maximum physical Memory i can run for free? I've read that their is limits on vRam but not on physical memory.  Whats the differences?

What is the maximum amount of VM which can be run without incurring additional cost?

Does the free version of ESXI limit the maximum system resources a specific VM is assigned?

 

Thanks in advance

 

You can run the free version of ESXi on a single CPU with an unlimited number of cores (wish I had one of these ) with up to 32GB of RAM. There is no limit imposed on how many VMs you can run or what resources you can assign to them, other than the 8 virtual CPU limit per VM, which I believe applies to even the enterprise versions of ESXi.

[Vargas]

 

You can run the free version of ESXi on a single CPU with an unlimited number of cores (wish I had one of these ) with up to 32GB of RAM. There is no limit imposed on how many VMs you can run or what resources you can assign to them, other than the 8 virtual CPU limit per VM, which I believe applies to even the enterprise versions of ESXi.

 

so i couldn't run ESXi for free on a dual CPU setup?

[BRiT]

As in 2 PHYSICAL CPUs? And not dual-core CPU?

 

I think it will run but will only use the resources of 1 PHYSICAL CPU.

[Vargas]

As in 2 PHYSICAL CPUs? And not dual-core CPU?

 

I think it will run but will only use the resources of 1 PHYSICAL CPU.

yes as in a dual 1366 MB

[Johnm]

Correct. I believe it will install but only use1 physical CPU.

 

Even if you were to buy ESXi, you need to buy 2 licenses if i am not mistaken. one per CPU.

that will run you about $4k for standard ESXi 5.0 (VMware vSphere Standard).

 

Our new ESXi servers ran us about $45k just for licenses this summer. we wont mention the price of the  servers....

 

you might as well rip the second CPU out and build a new server if you go this route.

[cpshoemake]

Actually you can get the vSphere Essentials Kit for $495, which includes licenses for 3 hosts w/ 2 CPUs each.

 

http://www.vmware.com/products/vsphere/pricing.html

 

This, of course, doesn't include any of the advanced features like HA or vMotion, but it does net you vCenter Server, which is why I've been considering it.

[Johnm]

that's not a terrible price... still more then i'd like to spend on my  home lab.

 

i don't think that was listed when the sales rep came out. also the pricing has changed a bit from 4.1 when we got ours.

 

thanks for the 411..

[znelbok]

Hi Guys

I too am building a system just like this.  I was going to use an i3 but have decided to virtualise instead.  I have a couple of questions

 

Why use the USB stick to boot from and then have a drive for the datastore.  Could you not install ESX on the drive that has the datastore?  Is there a good reason not to install it there?

 

I too need a windows server guest installed.  With using an SSD as the data store, what changes are made to windows to prevent it from killing the driver over time.  Is it just as simple as turning off the swapfile?

 

I am only looking at three guests at the moment, unraid, W2K3/8 and firewall.  I can see me adding a few more in such as security camera NVR type stuff.

 

Thanks

 

Mick

 

 

[AaronD]

I've just completed my build - yeah!!

 

John you were the inspiration for this, thanks for taking the time with this post.

 

I went with the Tyan S5510GM3NR and it has worked flawlessly. I installed straight to ESXi 5, but apart from this just followed the instructions on the first page. I have unraid 5b12a which I upgraded to before starting the build when it was plugged into my old desktop. I also went with the M1015 which I bought from someone on this forum, and they had already reflashed it.

 

So now I have my unraid back up and running fine, it spins up/down perfectly and has been running for a couple of days now. I lost my MySQL database in the process, but perhaps I can move it off unraid and on to another VM.

 

Thanks

Aaron

[cpshoemake]

znelbok: You can. It's just more convenient to have data and boot drives separate.

 

AaronD: Congrats! I was in the same situation as you. I decided on http://www.turnkeylinux.org/mysql for MySQL. It uses very few resources (~50-100Mhz & 50MB) and has an excellent web interface (phpMyAdmin). If you still have the database files from the old installation, you can import them.

[AaronD]

Great! I've never heard of turnkey before, what a great resource. I would love to save my DB from unraid, and I do have the physical database files. Any tips on how I can import them into a new database on this appliance?

 

Also, for anyone who can help - I copied a 5GB file from my unRaid share to a Windows 2008 R2 VM running on the same host. It started out slowly, built up to around 40MB/s, then dropped down to about 24MB/s for the last third. This is much slower than I expected. Any advice?

 

Thanks

Aaron

[Johnm]

@znelbok

I find it much easier if the two are separate. especially if i need to rebuild ESXi or replace a dead datastore drive.

 

the only advantage to an all on one disk solution is your USB's are free for passthrough. I have not run them all in one since ESX ver 3.x

 

A far as the SSD..

I find them to be incredibly faster then spinners. That is what I need for many of my IO intensive guests.

I have one guest that is  running a large database with thousands of writes per minute (or was it per second? i am not awake yet) 24x7. it would not be possible on a spinner without 16+ gigs of ram dedicated to it. The entire system was lagged on spinners, even when on bare metal.

 

It is not necessarily recommended to run a datastore on SSD.  ESXi does not support Trim so that's going to impact the life of the SSD.

I do state I am most likely going to burn out my SSD's. it should take a few years they way I am using them.. If it is sooner, so be it, I just hope not.

you could get the kingston SSD now+ with bios controlled trim and garbage collection.

In the next 2 years SSD's should be larger and cheaper (maybe faster) and I'll just pitch these and get improved models. these should make it until then. just make sure you leave lots of headroom on the drive.

For me, the cost is worth the reward.

 

you could just as easily use individual spinners or a raid array for your solution. It depends on your needs. You can get away with running a few lighter VM's on a single mechanical drive.

If it was in my budget, i would use 4x 500GB 7200 RPM laptop drives in Raid5 on an areca or LSI raid controller for my spinner portion of my build. Perhaps that combined with SSD's

 

---

 

AaronD & cpshoemake

Glad it is running for you. If there were any errors in my  ramblings... err walk though that you caught, let me know and I'll make sure to note and correct that.

 

As far as the lost database; as long as you still have the database files, you can always re-import it. I would make a back up first just in case.

 

I'll agree the tyan boards looks decent. I have had some issues with their quality in the past but these new boards seem to be getting better reviews.

I would defiantly try one out at this point.

there is also an Asus 1U barebones setup that caught our eye at work. It comes in a 1U server already (ASUS RS300-E7/PS4.

We are debating getting 1 instead of a supermicro barebones to replace our pocket sized test box at work. the only drawback is the raid and IPMI are add on cards. It looks like it would be a poor head for storage guests that have DAS boxes due to the Gen1 X4 Link PCIe. If it had Gen2 8X i would get one for home.

 

[Johnm]

Great! I've never heard of turnkey before, what a great resource. I would love to save my DB from unraid, and I do have the physical database files. Any tips on how I can import them into a new database on this appliance?

 

Also, for anyone who can help - I copied a 5GB file from my unRaid share to a Windows 2008 R2 VM running on the same host. It started out slowly, built up to around 40MB/s, then dropped down to about 24MB/s for the last third. This is much slower than I expected. Any advice?

 

Thanks

Aaron

 

I have not played with turnkey yet, but I bet a little poking around it should be quite obvious. I am running an older mySQL on a win box myself. Sorry.

 

As far as the data transfer. that might be  your unraid box? Are your guests on spinners?

I know i am getting 80MB/s read/write between my windows guests and unraid on the same box. that's with an SSD SATAII cache drive on unraid with the 3TB drives in my unraid. This includes my SDD based guests, my RMD WHS2011 and my spinner data store base VDI's.

Of course this can change under varying server loads. I was copying 200GB off a virtual drive and on the same spinner i was installing SP1 and updates on it at the same time and both suffered

 

I don't know your exact setup, would do a little more testing before you panic. there are to many factors to just take a guess.

 

In general, ESXi should perform similar to bare metal guests, the only time that is not true is when you are low on resources, memory, CPU or disk IO. the latter is usually the biggest problem followed by memory when using a modern Xeon based hardware. If you over-saturate the server, all guests will be impacted. this is why load balancing between servers is a nice feature.

 

 

[AaronD]

Glad it is running for you. If there were any errors in my  ramblings... err walk though that you caught, let me know and I'll make sure to note and correct that.

 

I'm sure it wasn't anything to do with your explanation that was at fault. I suspect I hosed by database when I was trying to shutdown unRaid before switching the USB over to the new server. I had problems getting it to shutdown properly, and ended up uninstalling the mySql package in the process.

 

Perhaps one thing that tripped me up which you could mention as a small note. I took my datastore drive from an un-needed Linux build where it had been partitioned using GPT. ESXi complained about it during the Add Datastore wizard... a little googling and I was able to SSH into ESXi and use partedUtil to delete the extra GPT partitions, then proceed with adding the datastore again.

 

Also, the Plop boot manager zip file I downloaded contains two ISOs - the first is in the root of the zip file, the second is in the install folder. The first one is, I guess, a live version of Plop; the second is the installer. In my haste I grabbed the first one... perhaps you could mention this again as a note to save anyone else confusion.

 

But no complaints from me... one of the most painless experiences I've had, given the size of the change!