ATLAS My Virtualized unRAID server

[Johnm]

this will look crazy but I can't get my VM to auto-start  :'(

 

I am interested in this as well. I read that auto-start was broken in the 5.0 Update 1

http://blogs.vmware.com/vsphere/2012/03/free-esxi-hypervisor-auto-start-breaks-with-50-update-1.html

 

THANK YOU, I may not be crazy! I'm effectively running 5.0 Update 1 (VMware-VMvisor-Installer-5.0.0.update01-623860.x86_64.iso)

 

Update 1 was released in March, more than 3 months ago... Why isn't this fixed yet? It's kind of ridiculous!?

 

I can't revert back as explained in the Blog post as I have a 5.0 Update 1 clean install. There's a sketchy workaround in the comments but I would prefer not to mess with that. So I guess that I will have to start all over from scratch with a 5.0 install (VMware-VMvisor-Installer-5.0.0-469512.x86_64.iso)

 

Should I be careful about anything in doing this? (except for unplugging every drives when re-installing ESXi)

 

Looking at the Update 1 release notes, I should be fine downgrading; my OSX VM wasn't working anyway..

 

And then VMware is gonna release Update 2 / a fix in a week haha

 

I just looked i am at 5.0.0 515841.

 

I matched mine to the level of the work servers. That might be why i am still at this level. I know i was at 5u1 at one point then rebuilt..

 

Thanks for the posting the issue.

 

 

This is distrubing.  I use OCZ drives for my datastores attached to a M1015 and I kept having troubles where they would just dissapear and I couldn't get them back without reformatting them.  I finally figured out that somehow the partition table was getting lost and enter the "partedUtil setptbl" ESXi command to reset the partition table (my Agility 3 I have to reformat the partition table every 2-3 days, the Vertex 3 is once a week).  They also were doing this on the onboard LSI 2008 controller too.

 

I recently got a Plextor M3 and for the life of me I couldn't get it to work right on the LSI 2008 controllers (either of them).  I could format it, copy data to it, but I kept getting random IO and/or corruption issues.  For example when trying to install ubuntu I would either get a message saying that my CD was corrupt or it would refuse to boot due to a corrupt install.

 

So, I moved the Plextor and the OCZ to the onboard Intel ICH10 controllers and all problems have dissapeared.

 

I have nothing to back it up on, but from my impression the LSI 2008 HBA doesn't like SSD too much.

 

Keep in mind that SSD was my datastore then moved to my cache drive.

It saw 100's of terabytes go across it. it has almost a year hard of abuse.

It is still in warranty lets see what happens...

First I'll see if i can bring it back to life first. right now, the bios does not see it.

 

strange about SSD's on your m1015. those are usually recommended for SSD raid stripes due to the speed and lack of cache.

 

 

[brian89gp]

strange about SSD's on your m1015. those are usually recommended for SSD raid stripes due to the speed and lack of cache.

 

IR vs IT firmware though.  One the OS see's the actual drive and the other the OS see's the representation of it created by the RAID controller.

[BetaQuasi]

Re: the startup/shutdown issue - I'm not seeing this at all on regular 5.0 U1 (no extra patches.)  I have it set up to boot my Win7 box first/shut down last (it's running APCUPSD, so makes sense), then my pfSense box, then Ubuntu (MySQL for XBMC and everything for Plex), then unRaid.

 

Works 100% every time after a reboot (did some fairly intensive testing to get the UPS stuff all working to my liking, so have probably tested it 10+ times.)

 

I do get that issue where 3 of every VM appears in the list -  I just hit refresh and it fixes itself.

 

Only real difference is that I use a .vmdk rather than plop for boot (from what I can see from those screenies above.)

 

 

[tyrindor]

Johnm, did you ever buy a server rack? I've been looking for days now trying to find something for my 2x Norco 4224s. It seems everything is $350+ after rails or shelves. It looks like you just stack yours on top of each other (but I imagine that's not ideal for heat, plus it scratches the cases?).

 

I don't want to scratch my cases, but I also don't want to spend $350+ on a server rack. Did you ever find a better solution?

[redia]

Johnm, did you ever buy a server rack? I've been looking for days now trying to find something for my 2x Norco 4224s. It seems everything is $350+ after rails or shelves. It looks like you just stack yours on top of each other (but I imagine that's not ideal for heat, plus it scratches the cases?).

 

I don't want to scratch my cases, but I also don't want to spend $350+ on a server rack. Did you ever find a better solution?

 

Tyrindor,

 

if you want to go cheap take a look at

http://wiki.eth-0.nl/index.php/LackRack

 

it is an ikea hack...

I have my doubt about have two 4224.. but it might give you some ideas.

 

R

[pantner]

So, i've been noticing for a while now, my UnRaid VM (running 5 RC5, but this has always happened if i remember correctly) is always using 1000Mhz + (currently more like 2500Mhz)

 

It doesn't matter what the VM is doing, its always using a lot of CPU time.

 

Is that the same with all of yours?

 

This seems to be the process that is doing it

 

root      4163    1 33 Jun29 ?        1-12:52:45 /usr/local/sbin/shfs /mnt/user -disks 2097151 2000000 -o noatime,big_writes,allow_other,default_permissions,use_ino

 

any ideas what could be happening?

[Johnm]

So, i've been noticing for a while now, my UnRaid VM (running 5 RC5, but this has always happened if i remember correctly) is always using 1000Mhz + (currently more like 2500Mhz)

 

It doesn't matter what the VM is doing, its always using a lot of CPU time.

 

Is that the same with all of yours?

 

This seems to be the process that is doing it

 

root      4163    1 33 Jun29 ?        1-12:52:45 /usr/local/sbin/shfs /mnt/user -disks 2097151 2000000 -o noatime,big_writes,allow_other,default_permissions,use_ino

 

any ideas what could be happening?

 

I'm still running RC4. I have not had time to upgrade.

While i do have spikes of around 1375mhz. my guest (1 CPU) averages/idols about 350mhz..  my history says my highest use was 1630 (about 49% of my CPU)

 

i have no major addons or plug-ins other then cache_dirs and the apc software.

i do have several minor unmenu packages also like overheat, server status and few others running.

 

also check that you have the correct VMtools installed.

[hdkhang]

Johnm, did you ever buy a server rack? I've been looking for days now trying to find something for my 2x Norco 4224s. It seems everything is $350+ after rails or shelves. It looks like you just stack yours on top of each other (but I imagine that's not ideal for heat, plus it scratches the cases?).

 

I don't want to scratch my cases, but I also don't want to spend $350+ on a server rack. Did you ever find a better solution?

 

http://www.wegotserved.com/2010/09/24/hands-norco-r415u-19-open-frame-steel-rack/

 

I would rather leave the servers stacked on top of each other on the ground than to trust a cheap rack that is not designed to hold the weight.

 

For enclosed server cabinets you would want 960-1000mm depth. If you can't fit that, stick to some open 4 post racks. Most of them don't hold very much weight at all so choose wisely.

[dheg]

I think this is the largest post on ESXi on the forum so hopefully is the right place to post these questions :

 

Is it possible to set up WOL with ESXi? how would it work to ensure data integrity in open VMs?

 

How would it be the connection with an apc ups to shut down the system on a power outage?

 

Please bear with me, I'm a complete rookie on ESXi virtualization

 

Sent from my GT-I9100 using Tapatalk 2

[brian89gp]

Is it possible to set up WOL with ESXi?

I would imagine if the motherboard supports it, you could WOL an ESXi server.  Most boards that you would use for ESXi though have a lights out management on them or have it as an option and those include power management and a remote console.  Not sure of your intended purpose, but that might work.

 

how would it work to ensure data integrity in open VMs?

There are power on/power off rules for ESXi that can power off and on VM guests on system shutdown and boot.  Just have to make sure VMtools is installed on all VM guests.

 

How would it be the connection with an apc ups to shut down the system on a power outage?

Pass through the USB port to a VM guest and have that kick off a script to run the VMware remote CLI to shut down the ESXi server

 

[dheg]

Is it possible to set up WOL with ESXi?

I would imagine if the motherboard supports it, you could WOL an ESXi server.  Most boards that you would use for ESXi though have a lights out management on them or have it as an option and those include power management and a remote console.  Not sure of your intended purpose, but that might work.

 

Do you know whether the SUPERMICRO MBD-X9SCM-F-O will support it?

 

 

how would it work to ensure data integrity in open VMs?

There are power on/power off rules for ESXi that can power off and on VM guests on system shutdown and boot.  Just have to make sure VMtools is installed on all VM guests.

 

Could I pass the powerdown script through unRaid?

 

 

How would it be the connection with an apc ups to shut down the system on a power outage?

Pass through the USB port to a VM guest and have that kick off a script to run the VMware remote CLI to shut down the ESXi server

 

If I get it right, I have to passthrough the USB port to unRaid, make the APCD addin to do a clean powerdown of the unRaid server and to run the VMware remote CLI to shut down the ESXi server, which through rules will power off VM guests (win+linux). Did I get it right?

[brian89gp]

Do you know whether the SUPERMICRO MBD-X9SCM-F-O will support it?

 

Supermicro calls there lights out IPMI, look in the specs for the mobo to find out.

 

Could I pass the powerdown script through unRaid?

 

Probably.  Leave SSH turned on in ESX then create a script to log in and issue the shutdown command.

 

If I get it right, I have to passthrough the USB port to unRaid, make the APCD addin to do a clean powerdown of the unRaid server and to run the VMware remote CLI to shut down the ESXi server, which through rules will power off VM guests (win+linux). Did I get it right?

 

Pretty much.  Just remember you usually have to pass through the whole USB controller since they are all attached to the same USB hub on the motherboard.

[c3]

  Just remember you usually have to pass through the whole USB controller since they are all attached to the same USB hub on the motherboard.

 

I don't think so. USB does not require using direct-path.

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1022290

[Johnm]

  Just remember you usually have to pass through the whole USB controller since they are all attached to the same USB hub on the motherboard.

 

I don't think so. USB does not require using direct-path.

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1022290

 

No you can just add a single passthough USB port to unraid. ESXi 4.1 and newer support individual usb port passthough to its guests. (at a horrid speed if used for data like a USB hard drive, fine for things like UPS's and printers).

 

Earlier in this thread, I do set up Atlas on a UPS with shutdown of all the guests and host shutdown. and it has come in very handy this summer for sure. I have had at least 5 power failures.

 

 

 

I think this is the largest post on ESXi on the forum so hopefully is the right place to post these questions :

 

Is it possible to set up WOL with ESXi? how would it work to ensure data integrity in open VMs?

 

How would it be the connection with an apc ups to shut down the system on a power outage?

 

Please bear with me, I'm a complete rookie on ESXi virtualization

 

Sent from my GT-I9100 using Tapatalk 2

I assume that the SM supports WOL.. but I dont actually recall seeing it in the Bios... Server boards in general (SM for sure [s1 sleep only]) usually do not support sleep. they are designed for 24x7 activity.  I might try sending a magic packet to my server later and see what it does.

I am pretty sure that ESXi does not support sleep/suspend of the host itself.

 

If you buy an "F" version of the board, you will get IPMI to remote manage (and KVM over IP) it and power it on. in general, you want to leave an ESXi box up. they take forever to boot and esxi

 

For the UPS part. set up the UPS on unRAID. or better yet, any dedicated management guest (win or linux). APCUPSD can launch a script on power fail. you can have the script SSH into ESXi and issue a host shut down command (or individual guests one at a time if you want). If you have VMtools installed on all of the guests, they will shutdown cleanly. it is covered in this thread.

 

 

 

 

[Johnm]

Johnm, did you ever buy a server rack? I've been looking for days now trying to find something for my 2x Norco 4224s. It seems everything is $350+ after rails or shelves. It looks like you just stack yours on top of each other (but I imagine that's not ideal for heat, plus it scratches the cases?).

 

I don't want to scratch my cases, but I also don't want to spend $350+ on a server rack. Did you ever find a better solution?

 

Tyrindor,

 

if you want to go cheap take a look at

http://wiki.eth-0.nl/index.php/LackRack

 

it is an ikea hack...

I have my doubt about have two 4224.. but it might give you some ideas.

 

R

 

Yes and no.. I have had a rack for a while... but it is in my storage unit.

 

 

After my last move, My servers went into the guest bedroom.

I had no room for the rack in my basement due to lower ceiling height in the new place.

My basement is also a bit rough and not wired for network. I have also had some minor water issues down there.

 

On that note. they ended up going into Lack Racks.

You can add a crappy ikea rail system with their towel holders.. or you can just place a dish/hand towel on top of each server (as seen in photo below) to keep them from getting rack rash.

 

My rackmounts when into lack Racks.

my non-rack equipment is on an InterMetro shelving unit (they were not deep enough where they were located for the Norcos).

 

Here is crappy cellphone picture from earlier in the thread.

 

[dheg]

Do you know whether the SUPERMICRO MBD-X9SCM-F-O will support it?

 

Supermicro calls there lights out IPMI, look in the specs for the mobo to find out.

 

Could I pass the powerdown script through unRaid?

 

Probably.  Leave SSH turned on in ESX then create a script to log in and issue the shutdown command.

 

If I get it right, I have to passthrough the USB port to unRaid, make the APCD addin to do a clean powerdown of the unRaid server and to run the VMware remote CLI to shut down the ESXi server, which through rules will power off VM guests (win+linux). Did I get it right?

 

Pretty much.  Just remember you usually have to pass through the whole USB controller since they are all attached to the same USB hub on the motherboard.

thanks Brian

[dheg]

No you can just add a single passthough USB port to unraid. ESXi 4.1 and newer support individual usb port passthough to its guests. (at a horrid speed if used for data like a USB hard drive, fine for things like UPS's and printers).

thanks for that

 

Earlier in this thread, I do set up Atlas on a UPS with shutdown of all the guests and host shutdown. and it has come in very handy this summer for sure. I have had at least 5 power failures.

I'd have to read the whole thread, which is something I was actually going to do since I know close to nothing on virtualization, just have some experience setting up vbox in Linux and Windows

 

Server boards in general (SM for sure [s1 sleep only]) usually do not support sleep. they are designed for 24x7 activity.  I might try sending a magic packet to my server later and see what it does.

I am pretty sure that ESXi does not support sleep/suspend of the host itself.

My concern (not so much a concern as a "green attitude") is consumption. My current server is on 24x7 and I haven't set up WOL either, but I have an Asus P7H55-M LX MB and Intel Core i3 540 CPU, so it's quite "green". I'm guessing having an ESXi server with 2/3 active guests will be quite power-hungry

 

If you buy an "F" version of the board, you will get IPMI to remote manage (and KVM over IP) it and power it on. in general, you want to leave an ESXi box up. they take forever to boot and esxi

That's the one I bought. I'm planning my new server having yours as a reference. Have you measured/estimated the consumption of the X9SCM-F-O + E3-1240 Xeon?

 

For the UPS part. set up the UPS on unRAID. or better yet, any dedicated management guest (win or linux). APCUPSD can launch a script on power fail. you can have the script SSH into ESXi and issue a host shut down command (or individual guests one at a time if you want).

Why is better to set up the UPS in win/linux than in unRAID? My concern is unRAID data integrity, so I'll rather have a clean shut down on unRAID than on any other.

 

If you have VMtools installed on all of the guests, they will shutdown cleanly. it is covered in this thread.

honestly, I plan to read the whole thing  . I'll try to bother you as little as possible 

[c3]

The power consumption comparison would require knowing your drive count and guessimate on activity. The ESX will be comparable to your list i3, plus the drive activities.  If the 3 guest are idle they will not make ESX power hungry. Typically 3 VMs on one host will use less than half the power of three physical machines.

 

If you planning to increase the workload, say run xxyx as a vm which you never did before, then you might see an increase. For example, previously 2TB online, now 60TB and seeding, yup more power consumed. But just moving to ESX and same workload, you'll save power.

[dheg]

The power consumption comparison would require knowing your drive count and guessimate on activity. The ESX will be comparable to your list i3, plus the drive activities.  If the 3 guest are idle they will not make ESX power hungry. Typically 3 VMs on one host will use less than half the power of three physical machines.

 

If you planning to increase the workload, say run xxyx as a vm which you never did before, then you might see an increase. For example, previously 2TB online, now 60TB and seeding, yup more power consumed. But just moving to ESX and same workload, you'll save power.

I'm interested in a pure comparison between the Asus board + i3-540 vs supermicro + xeon-1240 at the same workload. The number of drives (8x 2Tb WD green drives + 1x ssd for apps)  and activity will off course increase the consumption, but it'd increase the same in both systems,  right?

 

Sent from my GT-P7500 using Tapatalk 2

[graywolf]

Is there any advantage between using E1000 or VMXNET3?

 

Would it be as simple as shutting down the VM, removing the old network adapter and adding the new one ?

 

[brian89gp]

Is there any advantage between using E1000 or VMXNET3?

 

Would it be as simple as shutting down the VM, removing the old network adapter and adding the new one ?

 

VMXNET3 is optimized to be run virtually, and its a 10gb adapter (any VM to VM traffic)

 

Yes.

 

[c3]

Is there any advantage between using E1000 or VMXNET3?

 

Would it be as simple as shutting down the VM, removing the old network adapter and adding the new one ?

 

The vmxnet3 is huge advantage if you run a 10Gbe network. If not, you still gain from the potential exchange within your host and lower CPU usage.

 

To change from intel e1000 to vmxnet3 you will want pre-install the latest vmtools, including drivers. Then you can shutdown, make the add/delete for network adapter. You should be ready to handle the first boot without functioning networking, but the pre-install should have the drivers local, or mount the tools iso.

[graywolf]

Went to check to see if I had the latest vmware tools, but it appears that the vmware tools for b11 does not contain vmxnet3.

 

Guess I'll either have to wait some more or review the RCs for which will work with both MV8 & M1015 cards.

 

This is the first time in awhile that I've had some slack time from work to play. Since it's raining, can't do the yard work.

 

 

[Johnm]

I use the vmxnet3 adapter because my cache drive is on a raid5 (formerly was an SATAIII SSD till she died). Since i move a lot of data internally on my ESXi box (guest to guest) between SSD's and guests whose disk images are on raid, The 10GB adapter helps a little for when I do exceed the internal speeds of a 1GB adapter.

 

[JM2005]

 

OK i confuse myself as well sorry about that!  I have the 2 white sata ports open on motherboard.  How do i setup pass thur on those ports? 

 

Thanks!

Ah..

Short answer, you cant

 

long answer, you cant. the cougar SATA controller is a 6 port controller, it is all 6 ports or nothing... (all single controller chip motherboards are like this)

 

If you look at this picture, it is the whole 6 port controller.

 

You are stuck with the instructions in my last post. sorry.

 

 

EDIT.. let me rephrase that... you cant pass only the 4 ports to ESXi for with direct passthough.

However..you can pass the other 4 unraid drives using RDM or raw device mapping.. it is a little more complicated.. and quit bit more work if you need to swap out a drive later.  but you could do it without buying any more hardware (assuming you have a disk for a datastore and a flash drive for ESXi)

 

Ok i decided to add another BR10i card and moved everything off the onboard sata ports and used those for ESXi datastores then i followed your instructions and I now have it all working great! I just finished setting up UPS backup scripts for shutdown per your instructions i just need to test a power outage in the morning to see if its working right and shuts down right.  I also setup a Win 7 Pro vm and have unRaid working fine thru passthru.

 

Thanks Johnm for all your help!!  Your detailed instructions worked great!!