Hi,

I am currently using ESXi 6.0 and I find it very easy to configure GPU passthrough, USB devices, assigning host cdrom to any VM (or multiple). From what I've read here, Nvidia cards are able to be passed using Unraid, for instance I have a GT730 I would like to try.

 

I'm looking for the pros & cons for each. I'm very familiar with ESXi so I know the limitations. 

 

ESXi uses 440bx exclusively, you can switch out bios files but I haven't gotten one to work that really gains me anything. I was trying to add a primary GPU setting so I can boot from the passed through adapter. I hear that Unraid is not limited to 440BX and does not have a virtual VGA that the VM is forced to boot from.  - Does this mean that txt mode install runs on the passed through card? I either have to use the VSphere client or VNC into the VM to build a new one.

 

I can pass a Quadro, but no Geforces (I am able to pass old PCI Geforce cards, like a FX5200)

I usually assign the PCI adapter as boot on the host. But I can steal it from the host and run a VM.

 

Unraid is configured by a web interface, which seems more "available" than a > Win7 box with VSphere. BUt you can't steal the GPU from the console. For ESXi I've written an Android APP to manage the system, works on free ESXi since it uses SSH (or serial) rather than the API's. I can build a new vm, assign the ISO or host CD, select passthrough devices, on boot it launches a VNC into the VM. I haven't seen the web interface of Unraid yet, but Ido see ALOT of command line & xml editing in the threads here. I like the Android app, I run an Android VM on the stolen console and auto launch it, so the console provides management. My kid launches his vm from this on his own.....so is the Unraid web interface friendly enough for an 11 year old? Is the command line fiddling only for setting up the system?

 

There is a setting in ESXi to disable ACS checking, but I have only had to set this when using a PCIe switch.  There is another ACS setting added in ESXi 6.0, I haven't needed it yet. I've read that KVM groups similar PCIe devices, so you can't assign them to different VM's. In ESXi, I see this for PCI devices - I have the fx5200 & a PCI USB card I assign to the Android console.

In Unraid do I encounter this with the USB's as well as GPU's?  Has anyone passed child devices connected to a PCIe switch using Unraid?

 

I saw the Linus Tech Tips video, and it worried me the time he spent getting it configured - I think it was 24 hrs straight.  For win10 VM's, which are pretty straightforward. I'm guessing it was the ACS grouping.

 

I have the spare hardware. I have been down this path before and found it not intuitive (to say the least). Also tried HyperV. It was probably 2+ years ago, and Unraid seems to be more "purpose built" for passthrough.

 

Have I missed anything,  either on the plus or minus side?

 

 

 

 

 

 

Also, is there a way to boot from a 2GB sata SSD instead? I'd like to save the USB's for vm's, and it would be faster.

I just hopped on the unRAID train due to Linus' videos as well.

 

Let me tell you he makes it look so simple in the videos, but it's a lot of work and takes a lot of getting used to. I'm still not 100% where I want to be after about a week or so of tinkering, but you may have better results.

Also, is there a way to boot from a 2GB sata SSD instead? I'd like to save the USB's for vm's, and it would be faster.

Must boot from USB. Wouldn't really be faster since boot drive is only accessed to unpack OS into RAM where it runs, and to record settings you make in webUI.

 

(2GB sata SSD, is this a typo? I can't imagine wasting a SATA port for something so small.)

I boot ESXi from a 2GB SSD (the min req'd), if I was to use USB then the USB is unavailable for passthrough.  And I would use PS2 M&K for the host, so it's not mandatory to lose the USB.  My board, and I suspect most, have 6 Sata ports. (they are split with 4 on 1 ctrlr and 2 on the other)

I could pass the 4 sata's to a "storage" VM, and have a cdrom & the 2gb on the other 2.

ESXi boots and decompresses in a similar way, but it takes much longer when installed to USB.

 

 

I just hopped on the unRAID train due to Linus' videos as well.

 

Let me tell you he makes it look so simple in the videos, but it's a lot of work and takes a lot of getting used to. I'm still not 100% where I want to be after about a week or so of tinkering, but you may have better results.

 

I'm guessing that once the VM's are installed & working, the fiddling would no longer be necessary.  It's VERY easy with Vmware, almost trivially, but forget about Geforce. 

I think I make it look easy here: 

I boot ESXi from a 2GB SSD (the min req'd), if I was to use USB then the USB is unavailable for passthrough.  And I would use PS2 M&K for the host, so it's not mandatory to lose the USB.  My board, and I suspect most, have 6 Sata ports. (they are split with 4 on 1 ctrlr and 2 on the other)

I could pass the 4 sata's to a "storage" VM, and have a cdrom & the 2gb on the other 2.

ESXi boots and decompresses in a similar way, but it takes much longer when installed to USB.

 

You can boot Unraid from disk, but you need to have a USB key plugged in for the licensing to work, so there's not much point...

I hear that Unraid is not limited to 440BX and does not have a virtual VGA that the VM is forced to boot from.  Does this mean that txt mode install runs on the passed through card?

 

unRAID provides you the option to use a virtual graphics adapter, but if you intend to pass through a GPU, this isn't always required (in fact, it most often is NOT required).  Typically you can install the entire OS using GPU pass through (no using virtual graphics/VNC required).  Alternatively you can install your OS with virtual graphics as the primary and pass through a GPU as secondary, then once inside the OS, load the graphics drivers for the secondary, shut down the VM, then remove the virtual graphics so you only have the physical, and boot back up. 

 

Another factor on whether or not you can do this is whether you leverage SeaBIOS or OVMF for your VMs.  I wouldn't recommend OVMF on unRAID in 6.1.x because we don't create unique VAR files for each VM yet, but in 6.2 that problem is rectified, and OVMF will even be set as the default BIOS type for Windows VMs going forward in that release as we have full proper support there and it allows us to do GPU pass through without the need for VGA arbitration (huge bonus).

 

I can pass a Quadro, but no Geforces (I am able to pass old PCI Geforce cards, like a FX5200)

I usually assign the PCI adapter as boot on the host. But I can steal it from the host and run a VM.

 

We have implemented workarounds to the GTX graphics card problems.  In the current release, this does require you to disable Hyper-V enlightenments for Windows-based guests, but our web interface does this automatically when it detects that you are assigning an NVIDIA GPU to a VM.  And even that that goes away in our next release of unRAID 6.2 (hyper-v enlightenments can be turned back on in that release).  It's a fairly safe statement that unRAID has broader and simpler GPU pass through support than any other solution available today.

 

Unraid is configured by a web interface, which seems more "available" than a > Win7 box with VSphere. BUt you can't steal the GPU from the console.

 

This is only true for NVIDIA based GPUs.  With AMD GPUs, you can.  Many users have gone down the road of using an AMD GPU for the top PCI slot and then NVIDIA for additional slots, and this works just fine.

 

My kid launches his vm from this on his own.....so is the Unraid web interface friendly enough for an 11 year old? Is the command line fiddling only for setting up the system?

 

You should probably watch some of the videos I've done on VM management to see what the user experience is like and how the web interface looks.  It's pretty darn simple to control the start/stop of VMs.  Videos are found here:  http://lime-technology.com/videos

 

There is a setting in ESXi to disable ACS checking, but I have only had to set this when using a PCIe switch.  There is another ACS setting added in ESXi 6.0, I haven't needed it yet. I've read that KVM groups similar PCIe devices, so you can't assign them to different VM's. In ESXi, I see this for PCI devices - I have the fx5200 & a PCI USB card I assign to the Android console.

In Unraid do I encounter this with the USB's as well as GPU's?  Has anyone passed child devices connected to a PCIe switch using Unraid?

 

For the best possible experience with VMs and GPU pass through, we recommend either to use a Xeon E5 processor (pricing starts at ~$217 MSRP for these) or an Extreme Edition desktop processor.  These have ACS capabilities.  However, if you are trying to repurpose existing hardware that does not, we have an override that breaks up devices into their own IOMMU groups.  Our wiki has links to articles explaining the risk in doing this, but I don't believe that risk is any different from ESXi.

 

Whether you are passing through a graphics, Ethernet, audio, storage, USB, or any other type of controller doesn't matter, if you are passing through a physical PCI device to a VM, IOMMU groups will determine your eligibility.  If your groups aren't isolated enough for your needs, the ACS override is a "use at your own risk" option, though many use it, including myself.

 

I saw the Linus Tech Tips video, and it worried me the time he spent getting it configured - I think it was 24 hrs straight.  For win10 VM's, which are pretty straightforward. I'm guessing it was the ACS grouping.

 

In both the 2 gamers and 7 gamers videos Linus did, he was using hardware that had ACS capabilities, so every device had its own IOMMU group and there was no challenge there.  With the 7 gamers project, the reason it took so long was due to using bleeding-edge AMD GPUs that aren't as friendly with GPU pass through.  They "work" well enough for the purpose of what needed demonstrating, but there were multiple issues we had with getting them configured properly.  None of the issues had to do with software configuration in unRAID at all.  They were purely hardware issues.

 

I have the spare hardware. I have been down this path before and found it not intuitive (to say the least). Also tried HyperV. It was probably 2+ years ago, and Unraid seems to be more "purpose built" for passthrough.

 

unRAID started out as a NAS, but with version 6, we had a desire to address the ever-growing desire our users have expressed to get more out of their hardware.  Systems today are capable of so much more than applications/OSes tend to be able to utilize, but with VMs and Docker Containers, we found a way to make better use of all those cores, threads, and gigabytes of memory.  The traditional market for storage and virtualization solutions has been focused on serving enterprise business and cloud computing.  We are trying to bring the benefits of those technologies down to the consumer market in a way that's far more accessible.  There are still some combinations of hardware that can make things difficult, but things keep getting better every day.

 

Have I missed anything,  either on the plus or minus side?

 

The main thing you need to realize with unRAID is that we didn't start out as a virtualization platform and because we load more on our host OS than just a hypervisor, there are additional considerations we have to think about as compared to alternative virtualization platforms when it comes to device pass through.  A prime example is making sure that if a user tries to pass through a USB controller, they don't accidentally pass through the controller that the boot device is attached to.  These are things we will be adding in the future, but for now, advanced users that want to pass through more devices than just graphics/sound will need to follow the guides here in the forum for doing so.  Also keep in mind that for basic USB input devices like mouse/keyboard, you don't need to pass through an entire USB controller.  Instead, you can just leverage USB assignment, which is built into the web interface itself.

Hey Jon, thanks for the really detailed reply.  Exactly what I wanted to know..

 

I just got home from a long day, so I'll have more to say tomorrow.

 

Eric

OK, I installed Unraid on an Asrock x58 Extreme, with 16 GB & i7 920. Boots from HD6450, and I installed a GT730.

 

Running a Android X86 live cd VM on the passed through GT730. Also booted an xp install. I like how it runs off the passed through GPU even in txt mode.

 

Is there any way to access the bios of a vm by pressing F2 or Delete?

Not that I'm aware of.  What do you want to achieve by accessing the VM bios?

It's just that I was able to in Esxi, to change the boot order. I found where to do it in the xml.

 

I am also having this issue http://lime-technology.com/forum/index.php?topic=41267.0

And it would be easier to boot to the bios to see if the disk shows up, rather than go into an install.

Well, there are definitely some things about Unraid that beat VMWare.

In ESXi, it's impossible to pass a Geforce card, only Quadro.

It's also impossible to boot a new install on the passed through adapter, you can't get rid of the Vmware SVGA. These are the 2 things I have long wished for in ESXi.

 

Booting from the passed GPU can be worked around with VNC. No real workaround for the Nvidia problem. I know you can monkey with the geforce firmware to make it appear to be a quadro, but it's not the right way to fix this. And these are the preferred gaming adapter, so the limitation is glaring...

 

I am passing 3 geforce cards in this box, I haven't installed the Nv drivers yet but I don't see why it wouldn't work. No code 43 on any..

 

But then there's stuff that ESXi does that Unraid does not do.....

 

I ran into this disk device problem, where I have to edit the xml file to make the disk IDE.  This wouldn't be too bad, but it reverts back whenever I edit the vm, such as changing USB's or CD's. So I have to edit the XML every time. There also seems to be no way to change cd's in a running VM, unless I were to use a USB cdrom with real cd's. This is mostly just inconvenient when building VM's,

 

 

I have used PCIe switches under ESXi with various devices assignable to seperate VM's. I had to turn off ACS in ESXi for it to work, but it works. If I insert the switch card in the Unraid box I get "iommu disabled", like I would if VTD was off. It's on, and I even tried more extensive ACS disabling to fix it. (downstream & multifunction). Apart from what I've tried I don't know what else I can do. This switch is a RDK with a GEN 1 chip, and I have a GEN 2 switch coming so maybe that will make a difference. It seems to get the addresses mixed up? This isn't really something most people will run into, but these switches are actually on some system boards (Asus P9D-E/4L). And very useful for a Mini-ITX board with only 1 x16 slot - you can fan it out to 2 GPU's.

 

 

So if anyone can point me in the right direction on how to troubleshoot the switch it would be great.

 

Thanks!