Hi,

i want to ask if it is possible to enable VT-d Support inside a VM. 
I try to use docker inside a VM. 

Is this in general Possible? 

 

I get an error. Screenshot Attached.

Best,
Daniel

Nested virtualisation is what you want to investigate, Unraid supports it, but the VM's OS (in your case a MacOS) needs to support it also.

On windows, Microsoft only supports Intel cpus for nested virtualization, even though AMD hardware can support it.

@tjb_altf4 thank you for the quick reply.

 

 

1. Shutdown all VMs

2. Open the Unraid Terminal

modprobe -r kvm_intel

modprobe kvm_intel nested=1

3. Open VM Settings -> Toggle XML View

Add 

<feature policy='require' name='vmx'/>

to the CPU Section

4. Start the VM

 

Edited April 25, 20206 yr by Daniel Ehrhardt

Hi @Daniel Ehrhardt,

I am facing the same problem as you. Tried to do the same steps you posted but I cant get it up and running. Could you please share your XML?

 

Thanks

 

Got it working with another solution due to I am on Unraid Version: 6.8.3.

modprobe -r kvm_intel

modprobe kvm_intel nested=1

Seem not to work properly for me. So I've done this steps, with help of Spaceinvadors Video (

The problems and solutions of nested vms in kvm on unRAID)  https://www.youtube.com/watch?v=2-saWn6ZbHc

systool -m kvm_intel -v | grep nested

Retuned nested="N" due to on my Unraid Version it is default deactivated. So I've downloaded User Scripts and used the custom script from Spaceinvader One Video and followed the steps.

 

Additional to that I've edited my MacOS XML and changed the ARGS at the bottom. It seems like the CPU which is per default "Penryn" is not supported by nested VMS, so I've swichted to host passthrough. Now my OSX is showing unknown CPU but Docker Client is working fine, due to VTX is passed through now. You can check it witch this:

sysctl -a | grep machdep.cpu.features

Changed this line:

<qemu:arg value='Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check'/>

to that:

<qemu:arg value='host,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check'/>

Let me know if something is not clear.

 

Cheers.

@Skylinar,
I am trying to do the exact same thing, run docker inside a macOS vm, I have a Ryzen 1700x,
this command ` systool -m kvm_amd -v | grep nested` gives me `nested              = "1"`

I have also changed the cpu args to the one you mentioned, I still get the same error.

@vatsalya 

Due to you are using AMD this method could not work because with my mentioned method you change from "Penryn" to "host" which will pass the host CPU. In your case, it will be AMD and not Intel. At the time I posted the solution I was running Dual Xeons and it could be, that it only works with Intel host CPUs.

As I am not running macOS VM anymore I sadly can't help much more here.

@SkylinarThank you for you comment! Using script from Spaceinvaderone and your idea of editing vm`s XML from Penryn to host, I am now able to run x64 W10 on macos catalina and enjoy the advantage of easy access to snapshots.

@mark41 pleased to hear that my research could bring you further!

On 5/9/2021 at 2:01 PM, mark41 said:

@SkylinarThank you for you comment! Using script from Spaceinvaderone and your idea of editing vm`s XML from Penryn to host, I am now able to run x64 W10 on macos catalina and enjoy the advantage of easy access to snapshots.

 

Hey, unraid newbie here - I'm wondering if you had the chance to write your process up? I'm going to wipe my opencore installation and pull the plug on unraid. docker within macOS is a must on my 3900x and wanna figure out all the steps before getting started.

Cheers!

Edited May 27, 20215 yr by Van

58 minutes ago, Van said:

on my 3900x

No way to do it, at least of what I'm aware..

Hi, was this ever resolved to run nested virtualisation on AMD?

I've been trying to get Docker to work within a Windows 10 VM, any advice?

9 hours ago, KptnKMan said:

Hi, was this ever resolved to run nested virtualisation on AMD?

I've been trying to get Docker to work within a Windows 10 VM, any advice?

It works for me my 1950X, has been working for about 12 months now.

Steps were something like:

• unraid: backup vm (if nested doesn't work properly, vm will brick)
• unraid cmd: modprobe -r kvm_amd
• unraid cmd: modprobe kvm_amd nested=1
• unraid: reboot
• win10 vm: install wsl2 (and associated restart)
• win10 vm: install docker desktop

Performance was crap for me in 6.9 when nested virt was enabled, but perf got a lot better in 6.10

(but this may have been my other dockers being ram hogs during 6.9 time)

Edited May 31, 20224 yr by tjb_altf4

5 hours ago, tjb_altf4 said:

It works for me my 1950X, has been working for about 12 months now.

Steps were something like:

• unraid: backup vm (if nested doesn't work properly, vm will brick)
• unraid cmd: modprobe -r kvm_amd
• unraid cmd: modprobe kvm_amd nested=1
• unraid: reboot
• win10 vm: install wsl2 (and associated restart)
• win10 vm: install docker desktop

Performance was crap for me in 6.9 when nested virt was enabled, but perf got a lot better in 6.10

(but this may have been my other dockers being ram hogs during 6.9 time)

Thanks, I'm gonna try this.

Will report results.

So I got it working.

 

I found that KVM already had AMD extensions and nested Virtualisation enabled:

root@primary:~# modprobe -r kvm_amd
modprobe: FATAL: Module kvm_amd is in use.
root@primary:~# systool -m kvm_amd -v | grep nested
    nested              = "1"
root@primary:~#

 

Enabling the nested module did nothing (As expected):

root@primary:~# modprobe kvm_amd nested=1
root@primary:~#

 

I've had issues with this before, because I remember last year I enabled Docker extensions in Visual Studio Code and bricked my VM (Restored from nightly backup so no big deal) but I never tried that again.

So I checked my VMs, and per advice around the forums I added the vmx CPU flag:

  <cpu mode='host-passthrough' check='none' migratable='on'>
    ...
    <feature policy='require' name='vmx'/>
    ...
  </cpu>

 

...and started up my VM, installed Hyper-V and Docker and got it running in WSL2. No errors in Device Manager were seen.

I didn't even need to reboot because the nested extensions were enabled.

 

For good measure, I downloaded SpaceInvaderOne's script and enabled it (And fixed it because there is an error in there on line 20), but it is mostly redundant because the extensions are already enabled. Still, gives me a little more control if I want it in future.

 

I have to say though, the performance took a hit.

I've seen reports that SeaBIOS is more performant, but I'd rather stay with OVMF if I can.

Just looking at Task Manager was a big oof! I'm going to have to assign more resources to this VM!

 

Thanks for all your help!

A quick followup on this...

 

I noticed that the VM is not just more loaded, but was noticeably more "sluggish" and slow generally.

I tried and failed to convert the VM to SeaBIOS, to test comparable performance, and while attempting this I noticed this error would appear each time I tried to update the GUI while the vmx CPU flag was in place (Removing the flag avoids the error):

 

So eventually, I made a backup and removed the vmx CPU flag, expecting my VM to die or brick or something strange.

Nothing happened, Hyper-V works within the VM, Docker works, WSL2 works, and it "seems" to be a little snappier.

 

I'm not entirely sure what is happening here (It could be more related to updates in 6.10.x), but I thought I would post it just so other know, the "vmx" CPU flag might not be required after all.

¯\_(ツ)_/¯

16 hours ago, KptnKMan said:

while the vmx CPU flag was in place (Removing the flag avoids the error):

I think this is because you are using cpu host-passthrough, but since it's an amd it doesn't have such feature (vmx).

It could work if the cpu is intel emulated, like Skylake-Client-IBRS or something else supporting the vmx flag.

Quote

When starting a VM Guest with CPU mode hostl, libvirt will copy its model of the host CPU into the VM Guest definition. The host CPU and features copied to the VM Guest definition can be observed in the output of the virsh capabilities.

 

What changed for amd nested virtualization compatibility is in windows itself, that added support for this, if I remember well starting when the first previews of windows 11 were available, then they were ported to windows 10 too.

 

Anyway since it now works with amd host-passthrough, windows is able to make it work with svm (vmx equivalent for amd).

Edited June 1, 20224 yr by ghost82

49 minutes ago, ghost82 said:

I think this is because you are using cpu host-passthrough, but since it's an amd it doesn't have such feature (vmx).

It should work if the cpu is intel emulated, like Skylake-Client-IBRS or something else.

 

What changed for amd nested virtualization compatibility is in windows itself, that added support for this, if I remember well starting when the first previews of windows 11 were available, then they were ported to windows 10 too.

 

Anyway since it now works with amd host-passthrough, windows is able to make it work with svm (vmx equivalent for amd).

Ah, I thought vmx was used across Intel and AMD, but it was a question of support. Like a common layer on top of VT-X/AMD-V, but alas I am mistaken. As usual, there is a separate name and term for a similar technology between Intel/AMD. ¯\_(ツ)_/¯

 

I also looked into KVM support for VMX/SVM and found there's quite a bit of information, like here and here.

 

As for guest support, and guest awareness of VMX/SVM extensions, that seems to be another layer of issues that (In this case) has had some progress in Windows 10/11.

 

Using Docker and VMs within Windows 10 has given some excitement to test Windows 11 again, now that the new OVMF-TPM BIOS is generally released in unRAID 6.10.x . Time will tell how reliable that is, but I actually found Windows 11 to be faster in VM than Windows 10, before the TPM limitations ended the fun.

Yes, exact.

Basically, Intel:

VT --> intel virtualization technology

This splits into VT-x and Vt-d

VT-x: hardware virtualization assistance

VT-d: allows direct passthrough (enables direct i/o for virtualization)

vmx: virtualization extension flag in cpu

 

You can run vms if hardware supports VT-x, even with VT-d disabled or not supported; you must have VT-d support for passed through devices.

 

AMD:

AMD-V: same as VT-x

AMD-VI: same as VT-d

svm: same as vmx

 

On top of that if you emulate an intel cpu in qemu, that cpu must support also EPT (extended page tables) and UG (unrestricted guest): if you look at the first image of the first post, the user is having that issue because he was emulating a Penryn cpu (intel core 2 duo) that lacks these features.

 

As far as I now enabling nested virtualization in a linux host, whatever it is, is not recent, same for VT-x/AMD-V, VT-d/AMD-VI and vmx/svm.

Nested virtualization for intel was more or less always working for long time, amd was poorly supported by guest oses; linux guests were the firsts to fix this, then came microsoft, apple never did this and will never do for amd, although nested virtualization works for intel (passed through or emulated).

Check the Libvirt storage location:   wahl then other location