Jump to content
adminmat

10 Questions: unRAID on ESXi

26 posts in this topic Last Reply

Recommended Posts

Reaching out to those with experience for clarification¬†for this¬†new build.¬†¬†Tagging a few of the pros here: @doron, @uldise, @ptmuldoon, @StevenDūüôŹūüôŹ

 

I have a couple other low power servers running services on ESXi. I like the idea of keeping them all under one management application while stretching the resources of this new build. Or is it best for this noob to place unRAID on bare metal and learn it first. (note: i have not run unRAID before)

 

Hardware of new server:

Supermicro X10SRM-TF (bifurcation ‚úĒÔłŹ)

E5-2690 V3

32GB RAM (soon to be 128GB)

2 x 1TB m.2 NVME SSD on PCIe adapter AOC-SLG3-2M2 

3 x 8TB WD HDD via LSI SAS9200-8e-HP in external iStarUSA boxes (room for 8)

EVGA GT710 GPU - not sure why I have thisūü§∑ūüŹĹ‚Äć‚ôāÔłŹ

 

Questions: 

1 - Are there performance impacts doing this, if so what?

 

2 - Is there a trick to setting up shares in ESXi for the unRAID disks?

 

3 - Is it difficult to migrate to a bare metal unRAID instance from unRAID on ESXi? (ya know... after the panic sets in)

 

4 - Are there issues with GPU pass-through when unRAID is on ESXi?

 

5 - Are there other pass-through issues?

 

6 - Is there a version of Open-VMware-Tools for the latest version of unRAID  6.7.2?

 

7 - Are the custom Open-VMware-Tools available in the Community Applications?

 

8 - Do I just use two separate, physical USB drives for boot? One for ESXi and second passed through to unRAID (PlopKExec)? 

 

9 - Does running on ESXi make managing IOMMU groups any different than unRAID on bare metal? (this part scares me)

 

10 - Is viewing and managing VMs on unRAID as easy as using VM Remote Client (VMRC) and VMware vCenter Server Appliance (vCSA)?

 

11 - Anything I'm missing? (bonus questionūüėČ)

 

if this is a go I'll probably purchase a VMUG license. 

 

thank you¬†ūüôŹ

 

Share this post


Link to post

Hi adminmat,

I started using unRAID back in 2008 on bare metal and moved it over to ESXi in 2013 and it has been rock solid.  My ESXi server is a SuperMicro X9SCM-F with a Xeon E3-1230v2 and 32GB RAM.  I have a dedicated LSI 9201-16i that I pass through to unRAID to manage 16 drives just for unRAID, and a separate LSI 9201-16i for drives that ESXi manages for VMs.

 

As for your questions:

1 - Are there performance impacts doing this, if so what?

I am using unRAID just to store files; I am not using it for docker or running other VMs in unRAID.  I have not noticed any performance issues.

 

2 - Is there a trick to setting up shares in ESXi for the unRAID disks?

I pass through a dedicated LSI 9201-16i to the unRAID VM and manage all unRAID drives on it.

 

3 - Is it difficult to migrate to a bare metal unRAID instance from unRAID on ESXi? (ya know... after the panic sets in)

It was very easy.  I pulled the original unRAID drives out of their dedicated server and plugged them into the dedicated LSI-9201-16i for unRAID on the ESXi server and started unRAID on ESXi.

 

4 - Are there issues with GPU pass-through when unRAID is on ESXi?

I just have the GPU on the motherboard and have not run into any issues.  I run the system headless.

 

5 - Are there other pass-through issues?

There have been some performance issues in the past with passing through the unRAID USB drive.  If I remember correctly it had to do with an upgrade moving to 6.0 and some of the config data didn’t get setup properly.  Once fixed I have not had any issues.

 

6 - Is there a version of Open-VMware-Tools for the latest version of unRAID  6.7.2?

I am currently running unRAID 6.7.2 and I have the OpenVMTools plugin installed and it appears to be working properly.

 

7 - Are the custom Open-VMware-Tools available in the Community Applications?

It is available as an unRAID plugin.

 

8 - Do I just use two separate, physical USB drives for boot? One for ESXi and second passed through to unRAID (PlopKExec)? 

Yes, that is how my system is setup and I also use PlopKExec for speed and simplicity.  I originally started with Plop, but it was rather slow to boot.  At some point I moved to the pre-built VMDK unRAID guest drive model that only reads the USB drive for the key; much faster than plop, but it made upgrading unRAID more difficult.  After a couple of years of that I migrated to PlopExec, which boots just as fast and makes upgrading unRAID a breeze.

 

9 - Does running on ESXi make managing IOMMU groups any different than unRAID on bare metal? (this part scares me)

I don’t manage unRAID drives in ESXi.  I pass through the controller and then manage unRAID drives in unRAID; faster, cleaner and easier.

 

10 - Is viewing and managing VMs on unRAID as easy as using VM Remote Client (VMRC) and VMware vCenter Server Appliance (vCSA)?

I don’t run any VMs on unRIAD; I still use ESXi to run all of my VMs.  You could run VMs in unRAID but then you are running through 2 hypervisors, which I would expect to impact performance.

 

11¬†- Anything I'm missing? (bonus questionūüėČ)

If you are still unsure and have not read the ATLAS thread, I suggest that you review it.

 

Share this post


Link to post

1 - Are there performance impacts doing this, if so what?

None

 

2 - Is there a trick to setting up shares in ESXi for the unRAID disks?

You need to pass through a controller or two and access the disks directly. You do NOT want to set up disks on a datastore.

 

3 - Is it difficult to migrate to a bare metal unRAID instance from unRAID on ESXi? (ya know... after the panic sets in)

I can boot my unRAID either baremetal or under ESXi. This is important as you will get zero support from Limetech and others running virtualized.

 

6 - Is there a version of Open-VMware-Tools for the latest version of unRAID  6.7.2?

Yes. However, there are problems with 6.7.  I recommend 6.6.7.

 

7 - Are the custom Open-VMware-Tools available in the Community Applications?

Yes. It was recently added to CA. I try to release an updated version of Open-VM-Tools within a couple of days of a new unRAID release.  Search for "openVMTools_compiled"

 

10 - Is viewing and managing VMs on unRAID as easy as using VM Remote Client (VMRC) and VMware vCenter Server Appliance (vCSA)?

I wouldnt run VMs on a virtualized unRAID.  I use a few Dockers, but no VMs. 

 

Share this post


Link to post

Let me begin by saying this is not considered a Supported Configuration by Limetech and by this community. This means that if you have an issue, practically any issue, when you post here for help you will likely be asked to first reproduce it on bare metal. If you don't (or can't) your issue might be classified as "something to do with virtualization". I just want you to be aware of this.

 

That being said, this is my configuration (running under ESXi), for the most part it is rock solid and I like it.

 

To your questions (some of my answers will be redundant with the good folks above but here goes anyway):

2 hours ago, adminmat said:

 

1 - Are there performance impacts doing this, if so what?

None that I can detect. Your processor and mobo must support VT-x and VT-d but once they do, performance is very close to real iron.

2 hours ago, adminmat said:

 

2 - Is there a trick to setting up shares in ESXi for the unRAID disks?

There are two good ways of doing it:

1. Pass a SAS controller through to the VM, and let it handle all drives. If all the drives on the controller go to Unraid, this would be the way to go. If OTOH you have some drives on this controller that are datastores or are used by other VMs, read on...

2. Use the "vmkfstools -z" method to create RDMs for each of your drives. Then, assign those RDMs to your Unraid VM. Don't believe stuff you read about RDMs not being good for you - they work perfectly well for me...

 

One very bad way of doing it, as others have mentioned, is to create v drives in datastores. So don't do that ūüôā

2 hours ago, adminmat said:

 

3 - Is it difficult to migrate to a bare metal unRAID instance from unRAID on ESXi? (ya know... after the panic sets in)

If you use one of the two ways above to pass your drives to Unraid, then it should be reasonably straightforward and painless to move back and forth. Be very sure to map the drives to locations - you might need to "new config" each time around due to different naming.

To mitigate fear, do a fire drill! Would get you fluent on these things and prevent panic.

2 hours ago, adminmat said:

 

4 - Are there issues with GPU pass-through when unRAID is on ESXi?

Dunno. I use the server headless (not even using its onboard GPU - all via IPMI). Not sure why you'd want to pass a GPU to Unraid (unless I misunderstood the question).

2 hours ago, adminmat said:

 

5 - Are there other pass-through issues?

I'm currently struggling with passing thru a PERC H330 off of eBay. It'll just been a few failed attempts and I'm pretty sure I'll figure it out but moral is - Murphy is still going strong.

2 hours ago, adminmat said:

 

6 - Is there a version of Open-VMware-Tools for the latest version of unRAID  6.7.2?

Use openVMTools_compiled from @StevenD (thank you man!!) off of Community Applications.

2 hours ago, adminmat said:

 

7 - Are the custom Open-VMware-Tools available in the Community Applications?

Yup.

2 hours ago, adminmat said:

 

8 - Do I just use two separate, physical USB drives for boot? One for ESXi and second passed through to unRAID (PlopKExec)? 

That'd be a good way to do it yes.

2 hours ago, adminmat said:

 

9 - Does running on ESXi make managing IOMMU groups any different than unRAID on bare metal? (this part scares me)

Probably. I don't pass through much so I don't really deal with them.

2 hours ago, adminmat said:

 

10 - Is viewing and managing VMs on unRAID as easy as using VM Remote Client (VMRC) and VMware vCenter Server Appliance (vCSA)?

I'm not sure I understand what you're getting at here. You're not planning to run VMs under a virtualized Unraid are you? Dockers yes, VMs no.

2 hours ago, adminmat said:

 

11 - Anything I'm missing? (bonus questionūüėČ)

See my opening statement. It is not a minuscule thing. As I said, I opted for this setup in spite of that, but you need to consider it.

 

Good luck!!

Share this post


Link to post

Thank you @ecuster and @StevenD, this is invaluable help. 

 

Agreed, it does not make sense to run any VMs in unRAID if I can in ESXi. 

 

StevenD, is there a specific method you use to boot from ESXi or unRAID or do you just choose this in the BIOS / IPMI when the time comes? 

 

Amazing news on the VM Tools and thanks for the info on 6.6.7¬†ūüôŹ

 

One more thought: I've noticed that VMware management doesn't accurately show the memory usage for a VM. How do you guys best manage memory usage/allocation? By tracking the memory levels from within each VM and adjusting/reallocating accordingly?

 

 

 

 

Share this post


Link to post
30 minutes ago, adminmat said:

StevenD, is there a specific method you use to boot from ESXi or unRAID or do you just choose this in the BIOS / IPMI when the time comes? 

 

 

 

I think I’m the only one who passes through a USB controller for the license on USB, but actually boots off a vmdk.  Everyone else seems to prefer PLOP. 

Share this post


Link to post

@doron I much appreciate the info. Yes, I was aware of Limetech not supporting this method. It is definitely something to consider. 

 

Regarding #2: passing the SAS controller through, this is my plan:   

 

For unRAID: have 3 to 8 HHD's on the SAS controller, the 2 X m.2 NVME drives (cache) on a PCIe card with bifurcation, and probably 1 SATA SSD download drive on a motherboard SATA port. 

 

For other ESXi VMs: Separate SSDs, m.2 drive and HDD drives using motherboard SATA and m.2 ports. (yes, my Supermicro board does have an m.2 slot although it's just PCIe 2.0. Still faster than SATA!) 

 

I assume all that can be done without creating RMDs, correct? Good to know that I have the ability to share a SAS controller though. 

 

#4: GPU Pass-through: I read somewhere that using a GPU helps with Plex. I have never used Plex so I could be way off here. I plan on running the server headless and my CPU does not have on-board video. I'm not sure if this could be an issue with certain applications. Guess I will find out. 

 

#10: You are correct. I would not be running VMs on unRAID in this setup, just Dockers so no need for this. 

 

 

Share this post


Link to post
10 minutes ago, adminmat said:

this is my plan:   

 

For unRAID: have 3 to 8 HHD's on the SAS controller, the 2 X m.2 NVME drives (cache) on a PCIe card with bifurcation, and probably 1 SATA SSD download drive on a motherboard SATA port. 

(...)

I assume all that can be done without creating RMDs, correct? Good to know that I have the ability to share a SAS controller though. 

This last one will need to be passed as an RDM, no? On the mobo.

 

 

Share this post


Link to post
7 minutes ago, doron said:

This last one will need to be passed as an RDM, no? On the mobo.

 

 

 

Ok, Care to do a brief ELI5 on RDM? Or have a link?¬† I'm quite new to this and it's all self learned within the past couple weeks ūüėĶ¬†Should I assume that you can only pass through PCIe SAS controllers in this instance? But not individual¬†motherboard ports?¬†

Share this post


Link to post
14 minutes ago, adminmat said:

 

Ok, Care to do a brief ELI5 on RDM? Or have a link?¬† I'm quite new to this and it's all self learned within the past couple weeks ūüėĶ¬†Should I assume that you can only pass through PCIe SAS controllers in this instance? But not individual¬†motherboard ports?¬†

Basically it's a Real Device Mapping feature of VMware, essentially creating a dummy vmdk that "describes" the real device, and allocating that vmdk to the VM.

Several guides exist as to how to do that, requires a bit of CLI fiddling but quite straightforward. Look here or here.

 

And yes, you can pass through "devices" (such as PCI bus entities) but not single ports of a device (such as a single port on a SATA controller).

Share this post


Link to post
14 minutes ago, doron said:

Basically it's a Real Device Mapping feature of VMware, essentially creating a dummy vmdk that "describes" the real device, and allocating that vmdk to the VM.

Several guides exist as to how to do that, requires a bit of CLI fiddling but quite straightforward. Look here or here.

 

And yes, you can pass through "devices" (such as PCI bus entities) but not single ports of a device (such as a single port on a SATA controller).

Ok, thanks for the info. I'll dive into it. I actually have 5 open spots on in the hot-swap rack so I can just park the SSD there. 

 

 

Share this post


Link to post

Hi I have a question. I've created a Esxi environment with 1 x 1 TB SSD for Esxi Vm's. Besides that I have 3 x 8 TB + 1 x 6 TB + a IBM 1015 controller card.

I have just bought Unraid and want to make the most of it, so I have assigned the IBM1015 card to Unraid + the 4 drives. I want to add a cache drive also.

So my question is:

 

1. Should I take lets say 500 gb from the ESXi SSD to put in the cache drive in Unraid.

2. Should I buy a new 1 TB SSD to cache drive?

2a. If I buy the new drive. Should I pass it through on the IBM1015 card or would i be just as good to mount it to a SATA port on the motherboard and do the vmtools mount in Esxi?

 

The reason is that I only have 8 ports on the IBM1015 controller, so for future expansion I would like to save it. I had done the vmtools for a drive in Windows 10 VM, but on some files it became write protected, and when I took the drive out and mounted as USB the problem dissapeared, so is a little uncertain about it.

 

The cache drive should be used for dockers, Plex and DL.

Share this post


Link to post

you can use esxi RDM to connect hdd connected to mobo port to unRAID. i'm using this way my cache drive without any issues for years.. 

search a forum, there are several guildes how to do it. one search result from me:

 

Share this post


Link to post

Which commands did you use for that, since the thread you are linking to is a little unclear?

I've been using commands like this, but that gave me the read/write protection in Windows 10:

vmkfstools -z /vmfs/devices/disks/t10.ATA_____SAMSUNG_HD204UI_________________________S2H7J9GB908877______ "/vmfs/volumes/ESXiServer/Windows 10/Samsung_HD204UI_2TB_1.vmdk"

 

But you don't get any SMART info/spindown on this approach right?

And also not a good idea to use the Esxi SSD?

Thanks for the help.

Share this post


Link to post

i think your command is ok, never used such RDM disks on windows.

and yes, no smart info, no spindown, until you RDM a full drive.

in one of my unraid servers, i simply create small datastore on ESXI side(physically on SSD), and attach this disk to unraid, and use it as unussigned disk for Docker image. but you can use that way for cache drive too..

if an SSD have good built-in garbage collection, then i see no problems with such approach until you are using your server very heavy with lot on downloads on cache. in such case SSD will exhaust very fast..

Edited by uldise

Share this post


Link to post

Thanks for the help and the SSDs are Samsung EVO 860.

Share this post


Link to post
On 1/24/2020 at 5:21 PM, Bjur said:

But you don't get any SMART info/spindown on this approach right?

Why would you not get SMART and spin down in RDM? From my experience, you get both.

Share this post


Link to post
14 hours ago, Bjur said:

Are you sure about that? 

Pretty sure.

Is there some common wisdom saying that you don't get SMART and spindown with RDMs?

Share this post


Link to post

Yes, I've read that when you go this way on Esxi. 

Share this post


Link to post

huh, my mistake, i just corrected my previous post - if you RDM a whole drive, then you get SMART and spindown works just fine - just tested it on my offsite backup server.

Share this post


Link to post
10 minutes ago, Bjur said:

Yes, I've read that when you go this way on Esxi. 

Okay this is very interesting.

I just did a brief google study and indeed found references to this claim - e.g. here, here and here.

Honestly, I haven't even heard of that claim until this thread.

 

Conversely, I've been running all my drives as RDMs for quite a few years. SMART was fine, spin down worked.

Ironically, I've recently moved all my mechanical drives to a passed-through controller; the one drive I left as RDM is my cache drive, which is an SSD, which does not spin down.

I just re-tested smartctl again against this drive and it works, as does hdparm.

 

Does anyone have any supporting data to the contrary (i.e. that either of these does not work)? Or could this be some form of "tribal myth" that has been passed on as common wisdom?

Share this post


Link to post

Thanks for the answers. So what Uldise says is that he actually tested this and it works with Unraid and you Doron has mixed results? I would like to passthrough the SSD as RDM but not if it will cost no spindown and no smart.

I would like to use the Samsung as a cache for dockers and for downloads but some says it will trash the SSD quickly with all the download writes and others don't. This Unraid is very nex to me so I want to get it right the first time. 

Share this post


Link to post
Just now, Bjur said:

Thanks for the answers. So what Uldise says is that he actually tested this and it works with Unraid and you Doron has mixed results?

Not exactly. I'm saying I've been working only with RDMs for years and years and both SMART and spin down were working just fine.

 

Just now, Bjur said:

I would like to passthrough the SSD as RDM but not if it will cost no spindown and no smart.

SSDs do not actually spin down. SMART works for them, of course.

 

Just now, Bjur said:

I would like to use the Samsung as a cache for dockers and for downloads but some says it will trash the SSD quickly with all the download writes and others don't. This Unraid is very nex to me so I want to get it right the first time. 

That is a different question that is unrelated to RDM vs. controller p/t.

I'm not authoritative on this one - I do use SSD for cache.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.