I'm thinking about switching to ESXi but I have some questions I need a bit of clarification on.

 

I currently run an unRAID server with 15 disks and a separate Ubuntu 12.04 server as my front end. I'd like to combine those two servers into one machine, and ESXi is likely the ticket.

 

If I go this route, I'll be picking up an X9SCM-F which comes with 6 SATA ports onboard. I also have an M1015 on order which gives me 8 more ports for a total of 14. However, unRAID needs 15, and I believe I'll go with an SSD I already have as my datastore. That means I'll be short 2 ports and I'll need some sort of additional PCI-E card to get those ports (possibly another M1015).

 

1.) From what I've read, ESXi works best where the controllers are passed through to the VM's. This would mean that if I have 8 drives attached to my M1015, I should have all of those drives passed through to a single VM and not spread between different VM's. Is that the best practice?

 

2.) If I go with an SSD for my datastore, how would I use that? I'll be running unRAID and Ubuntu Server 12.04. Does unRAID need any space on that SSD? Or does it continue to run on the USB stick?

 

3.) If Ubuntu Server is running on the SSD, I would only use that space for the OS. I would then want a second drive attached for things like Sabnzbd downloads right? That way the storage space and I/O on the SSD is reduced, and a standard hard drive can instead step in. But if this is the case, it sounds like I would need to have a controller dedicated just for the Ubuntu Server? I already need 15 drives just for unRAID. That means I'll use MOBO controller and the M1015 controller and still be short by 1 port/controller. I'm a bit confused by how I would handle this. I suppose I could remove 1 drive from unRAID which means that unRAID would run on all the MOBO ports and all the M1015 ports. Then, on a different PCIE controller, I would attach the SSD and my last hard drive and Ubuntu would have access to those drives. Am I understanding this correctly?

 

Sorry for all the questions... I'm just trying to wrap my head around everything and start preparing for how I'm going to go about upgrading my current server.

 

 

It looks like my answer to number 1 is to pick up 2 M1015's and pass both those controllers through to unRAID. That would give me up to 16 drives for unRAID to use.

 

2.) My SSD would store a 2GB virtual drive for unRAID (which points to boot off the USB stick) and another virtual drive for Ubuntu.

 

3.) I'm still not so sure on this one. If I run unRAID off 2 M1015 cards, that leaves me with 6 on board SATA ports. 1 of those would be for my SSD datastore, so that leaves me with 5 additional ports. From what I've gathered, I should be able to use Raw Device Mapping to assign a hard drive to Ubuntu. This involves running some commands in a terminal (and I'd have to re-run them when replacing that disk), but that would work right?

From what I've gathered, I should be able to use Raw Device Mapping to assign a hard drive to Ubuntu. This involves running some commands in a terminal (and I'd have to re-run them when replacing that disk), but that would work right?

That is how I have my WHSv1 boot disk setup (RDM).  The other pool drives are on another controller that is passed through to WHS but I could have set them up as RDM as well.  You could even setup unRAID with all RDM drives but then they wouldn't spin down and no smart reports or temps.  I suspect the same would be true with any other OS.  So keep that in mind when you setup the drives to be RDM drives to the OS - in your case Ubuntu.

 

If you are wondering why I have my WHS boot drive RDM'd instead of on the passed through SASLP-MV8 controller with the other pool drives: The reason is that the VM will not boot from a drive connected to a controller that is passed through.  So far that is true for any OS that I have virtualized (WHS, Win7, WinXP, unRAID).  They would boot from CD or a hard drive either a virtual HDD or an RDM'd one but not from a drive on the passed through controller.  I didn't try tweeking the ESXi VM bios much but with minimal changes it didn't work.  The drives on the controller didn't show up in the bios list of bootable devices anyway.  If I switched to the EFI bios they might have showed up - I never tried that.  I had ports available from MB that I could use with RDM so took the fast approach - didn't want to spend days experimenting.

From what I've gathered, I should be able to use Raw Device Mapping to assign a hard drive to Ubuntu. This involves running some commands in a terminal (and I'd have to re-run them when replacing that disk), but that would work right?

That is how I have my WHSv1 boot disk setup (RDM).  The other pool drives are on another controller that is passed through to WHS but I could have set them up as RDM as well.  You could even setup unRAID with all RDM drives but then they wouldn't spin down and no smart reports or temps.  I suspect the same would be true with any other OS.  So keep that in mind when you setup the drives to be RDM drives to the OS - in your case Ubuntu.

 

If you are wondering why I have my WHS boot drive RDM'd instead of on the passed through SASLP-MV8 controller with the other pool drives: The reason is that the VM will not boot from a drive connected to a controller that is passed through.  So far that is true for any OS that I have virtualized (WHS, Win7, WinXP, unRAID).  They would boot from CD or a hard drive either a virtual HDD or an RDM'd one but not from a drive on the passed through controller.  I didn't try tweeking the ESXi VM bios much but with minimal changes it didn't work.  The drives on the controller didn't show up in the bios list of bootable devices anyway.  If I switched to the EFI bios they might have showed up - I never tried that.  I had ports available from MB that I could use with RDM so took the fast approach - didn't want to spend days experimenting.

 

Got it. So basically, if you pass through any RAID controllers, any drive that exists on that controller can not be used as a boot drive. That makes sense, since ESXi is ignoring that controller and those drives and passing it through directly to the assigned OS so that OS can manage everything.

 

It's also good to know that anything send over RDM won't have the ability to go to sleep or report smart information. With unRAID I'm going to run 2 M1015's and those will cover my 14-15 unRAID drives. I'll run my OS' off an SSD and I'll use another traditional drive as a cache like drive for temporary data the OS's on those drives may need (such as sabnzbd downloads). All of these drives will be attached directly to the MOBO. The SSD will just store the vmdk files for each OS, but my cache drive, for now, will be RDM'd to my Ubuntu Server since I don't plan on having any other OS' that could use it right off the bat.

I don't use ESXi (yet), but have read virtually all the threads/posts on it so when I do decide to build an ESXi server I can minimize the "learning curve."

 

So ... you need Bob to confirm these comments (since he DOES use ESXi), but I think both of the following are true:

 

(a)  With regard to your comment, "... This would mean that if I have 8 drives attached to my M1015, I should have all of those drives passed through to a single VM and not spread between different VM's",  I am fairly sure you don't have a choice about spreading it "between different VM's."      My understanding is that if you pass a controller through to a VM, that VM then has exclusive use of the controller -- so another VM could NOT use any of the drives on it.

 

(b)  You also don't need a virtual drive on your SSD for UnRAID => you can simply boot off the UnRAID USB flash drive within the VM.    Since you need that anyway, there's no real advantage to doing it differently (except perhaps a somewhat faster boot time).    Using just the USB stick also has the advantage that if you need to (for debugging/troubleshooting purposes) boot "bare metal" for UnRAID, all you have to do is select the UnRAID USB from your system's boot menu (instead of the normal ESXi stick).

 

 

 

I don't use ESXi (yet), but have read virtually all the threads/posts on it so when I do decide to build an ESXi server I can minimize the "learning curve."

 

So ... you need Bob to confirm these comments (since he DOES use ESXi), but I think both of the following are true:

 

(a)  With regard to your comment, "... This would mean that if I have 8 drives attached to my M1015, I should have all of those drives passed through to a single VM and not spread between different VM's",  I am fairly sure you don't have a choice about spreading it "between different VM's."      My understanding is that if you pass a controller through to a VM, that VM then has exclusive use of the controller -- so another VM could NOT use any of the drives on it.

Correct.  When a controller is passed to a VM everything on the controller is used by that VM and that VM only.  The only exception is if you setup and share those drives out over a network connection to other VMs - iSCSI or NFS for instance.

 

(b)  You also don't need a virtual drive on your SSD for UnRAID => you can simply boot off the UnRAID USB flash drive within the VM.    Since you need that anyway, there's no real advantage to doing it differently (except perhaps a somewhat faster boot time).    Using just the USB stick also has the advantage that if you need to (for debugging/troubleshooting purposes) boot "bare metal" for UnRAID, all you have to do is select the UnRAID USB from your system's boot menu (instead of the normal ESXi stick).

Actually with ESXi you cannot boot from USB.  You can boot from HDD either RDM'd drive or a virtual drive but not from a controller drive.  You can also boot from CD/DVD drive.  That is why some people have a plop boot setup for the unRAID VM.  You setup a plop boot cd image that then boots from a USB drive.  I started this way but I quickly switched to booting from a virtual VMDK image HDD for unRAID as the boot time is faster and without the plop image in another thread is harder (at least for me) to create.  Matter of fact I tried to create a plop boot CD and it didn't work so I found the image in the other thread and downloaded it.  A month later I switched to VMDK when I figured out how (either read how in Johnm's thread or elsewhere) and created one.  Now you can even download the VMDK that BetaQuasi has created for each release in another thread.

 

Edit.  As of ESXi 5.0 anyway.  I don't believe it has changed with 5.1 but I could be wrong they may have implemented USB boot.

Actually with ESXi you cannot boot from USB.  You can boot from HDD either RDM'd drive

 

Edit.  As of ESXi 5.0 anyway.  I don't believe it has changed with 5.1 but I could be wrong they may have implemented USB boot.

 

Ahh => I knew you couldn't boot from a passed-through controller; but didn't realize you couldn't boot from a USB device.

 

It would be nice if 5.1 changed that, but I doubt it.    THAT explains why BetaQuasi has made all the UnRAID VM's for folks to use 

 

Question:  Does the "UNRAID" labeled USB flash drive have to contain anything other than the key file on it?    ... i.e. where does UnRAID read the config and plugin data from when you're booting from a VM ??

 

Question:  Does the "UNRAID" labeled USB flash drive have to contain anything other than the key file on it?    ... i.e. where does UnRAID read the config and plugin data from when you're booting from a VM ??

Technically the virtual HDD only needs bzroot and bzimage and the other boot files from the root level.  It doesn't need the config folder or any of the plugin code because all of that boots from the USB flash drive like a normal boot from the flash drive.  There is a way to make it boot from the virtual drive only and just use the flash drive to confirm the key but it is complicated and I personally haven't tried it yet.  You basically setup some commands and multiple go scripts which move the flash that gets mounted at /boot to another mount point and the virtual HDD gets mounted to /boot and continues the boot process based on what I read in another thread which if I can find it I will update this post with a link.

 

Edit.  Found it.  This thread details how to use to flash drives to boot unRAID completely from an unlicenced flash you can use a HDD in place of the 2nd flash to boot from the HDD and only use the USB flash for the .key file.

 

Edit2.  Found better single post.  You still need to read in a follow up post for a correction but that post explains the process.  If I was using a lot of plugins I might want to use it but since I don't use many I don't feel the need for it.

Question:  Does the "UNRAID" labeled USB flash drive have to contain anything other than the key file on it?    ... i.e. where does UnRAID read the config and plugin data from when you're booting from a VM ??

Technically the virtual HDD only needs bzroot and bzimage and the other boot files from the root level.  It doesn't need the config folder or any of the plugin code because all of that boots from the USB flash drive like a normal boot from the flash drive.  There is a way to make it boot from the virtual drive only and just use the flash drive to confirm the key but it is complicated and I personally haven't tried it yet.  You basically setup some commands and multiple go scripts which move the flash that gets mounted at /boot to another mount point and the virtual HDD gets mounted to /boot and continues the boot process based on what I read in another thread which if I can find it I will update this post with a link.

 

Actually, that's probably best anyway.  It means that as long as you keep the contents of the root of the flash drive and the VM in sync, you can trivially switch to a "bare metal" boot by simply choosing the UnRAID USB flash drive as the boot device instead of the "normal" ESXi USB device.

 

Hopefully not something you'd need to do -- once it's nicely configured under ESXi with no add-ons or plug-ins it should be VERY reliable.    That's the beauty of the ESXi approach ... all your other "stuff" can be in other VM's, and if they "crash" it doesn't take down UnRAID or ESXi 

 

If I didn't already have two very nicely configured servers I'd probably build an ESXi box ==> in fact I'm getting more & more tempted as time rolls along to just do it anyway !!