Nested Virtualisation Query

[phyzicist]

Hi, I'm learning IT and thought I'd force myself to dive into the deep end.

 

I set up a PC* running with Unraid, and now, I'm figuring out if I can wrangle nested virtualisation for classes.

 

This machine also runs my Plex NAS hence Unraid. 

 

So far, I've managed to get a Windows 11 VM up and running with 8 cores, 8 threads pinned and isolated,  GPU passthrough with vBIOS, NVME passthrough, and more. This setup has been working perfectly.

 

I can boot straight into Windows 11 bare-metal and load VMware, and the nested Windows 11 VM works fine. (Windows 11 > VMware > Windows 11 VM)

 

However, I'm encountering a persistent issue. When I boot Unraid, spin up my Windows 11 VM, then VMware and start the nested Windows 11 VM, I keep getting the DWM.exe has crashed error. I'm not sure what's causing this. (Unraid > Windows 11 VM > VMware > Windows 11 VM).

 

My Kali Linux VM works perfectly in the same manner. I'm hoping I can move past this small Windows error with help.

As I said earlier, I'm a student, so I want to learn, but I've just gotten stuck here, and I was hoping for suggestions to help me pass this minor roadblock.

 

 

*AMD 5950x, B550, 64 GB RAM, RTX 3090, NVME for Windows 11 VM...

[SimonF]

1 hour ago, phyzicist said:

Hi, I'm learning IT and thought I'd force myself to dive into the deep end.

 

I set up a PC* running with Unraid, and now, I'm figuring out if I can wrangle nested virtualisation for classes.

 

This machine also runs my Plex NAS hence Unraid. 

 

So far, I've managed to get a Windows 11 VM up and running with 8 cores, 8 threads pinned and isolated,  GPU passthrough with vBIOS, NVME passthrough, and more. This setup has been working perfectly.

 

I can boot straight into Windows 11 bare-metal and load VMware, and the nested Windows 11 VM works fine. (Windows 11 > VMware > Windows 11 VM)

 

However, I'm encountering a persistent issue. When I boot Unraid, spin up my Windows 11 VM, then VMware and start the nested Windows 11 VM, I keep getting the DWM.exe has crashed error. I'm not sure what's causing this. (Unraid > Windows 11 VM > VMware > Windows 11 VM).

 

My Kali Linux VM works perfectly in the same manner. I'm hoping I can move past this small Windows error with help.

As I said earlier, I'm a student, so I want to learn, but I've just gotten stuck here, and I was hoping for suggestions to help me pass this minor roadblock.

 

 

*AMD 5950x, B550, 64 GB RAM, RTX 3090, NVME for Windows 11 VM...

Why dont you run VMWare as a VM rather than under Win11? i.e. Unraid->VMWare->WIn11

[itimpi]

Is there a reason for using VMWare in the first place?    If not why not run the target OS directly under unRaid?

[phyzicist]

I'm grateful for your responses. I've set up dedicated VMs in Unraid directly, but I was hoping to find a solution for nested virtualization for a few reasons:

 

1. My class requirements specify that the VMs should be in VMware or Virtualbox (unfortunately, I'm experiencing the same issue in both hypervisors) on either Windows or Linux (refer to #2).
2. I am attempting to transition to a Windows 11 VM in Unraid as my primary operating system.
3. It would be more convenient to follow along with the teacher and the class if I'm on the same platform as everyone else (I'm intentionally choosing a more challenging path).

 

I will attempt the Unraid > VMware > Windows 11 setup, just to see how it goes.

 

This is all new to me, and I'm eager to learn as much as possible along the way.