Daniel Ehrhardt Posted April 25, 2020 Share Posted April 25, 2020 Hi, i want to ask if it is possible to enable VT-d Support inside a VM. I try to use docker inside a VM. Is this in general Possible? I get an error. Screenshot Attached. Best, Daniel Quote Link to comment
tjb_altf4 Posted April 25, 2020 Share Posted April 25, 2020 Nested virtualisation is what you want to investigate, Unraid supports it, but the VM's OS (in your case a MacOS) needs to support it also. On windows, Microsoft only supports Intel cpus for nested virtualization, even though AMD hardware can support it. Quote Link to comment
Daniel Ehrhardt Posted April 25, 2020 Author Share Posted April 25, 2020 (edited) @tjb_altf4 thank you for the quick reply. 1. Shutdown all VMs 2. Open the Unraid Terminal modprobe -r kvm_intel modprobe kvm_intel nested=1 3. Open VM Settings -> Toggle XML View Add <feature policy='require' name='vmx'/> to the CPU Section 4. Start the VM Edited April 25, 2020 by Daniel Ehrhardt Quote Link to comment
Skylinar Posted September 3, 2020 Share Posted September 3, 2020 Hi @Daniel Ehrhardt, I am facing the same problem as you. Tried to do the same steps you posted but I cant get it up and running. Could you please share your XML? Thanks Quote Link to comment
Skylinar Posted September 4, 2020 Share Posted September 4, 2020 Got it working with another solution due to I am on Unraid Version: 6.8.3. modprobe -r kvm_intel modprobe kvm_intel nested=1 Seem not to work properly for me. So I've done this steps, with help of Spaceinvadors Video ( The problems and solutions of nested vms in kvm on unRAID) https://www.youtube.com/watch?v=2-saWn6ZbHc systool -m kvm_intel -v | grep nested Retuned nested="N" due to on my Unraid Version it is default deactivated. So I've downloaded User Scripts and used the custom script from Spaceinvader One Video and followed the steps. Additional to that I've edited my MacOS XML and changed the ARGS at the bottom. It seems like the CPU which is per default "Penryn" is not supported by nested VMS, so I've swichted to host passthrough. Now my OSX is showing unknown CPU but Docker Client is working fine, due to VTX is passed through now. You can check it witch this: sysctl -a | grep machdep.cpu.features Changed this line: <qemu:arg value='Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check'/> to that: <qemu:arg value='host,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check'/> Let me know if something is not clear. Cheers. Quote Link to comment
vatsalya Posted May 9, 2021 Share Posted May 9, 2021 @Skylinar, I am trying to do the exact same thing, run docker inside a macOS vm, I have a Ryzen 1700x, this command ` systool -m kvm_amd -v | grep nested` gives me `nested = "1"` I have also changed the cpu args to the one you mentioned, I still get the same error. Quote Link to comment
Skylinar Posted May 9, 2021 Share Posted May 9, 2021 @vatsalya Due to you are using AMD this method could not work because with my mentioned method you change from "Penryn" to "host" which will pass the host CPU. In your case, it will be AMD and not Intel. At the time I posted the solution I was running Dual Xeons and it could be, that it only works with Intel host CPUs. As I am not running macOS VM anymore I sadly can't help much more here. Quote Link to comment
mark41 Posted May 9, 2021 Share Posted May 9, 2021 @SkylinarThank you for you comment! Using script from Spaceinvaderone and your idea of editing vm`s XML from Penryn to host, I am now able to run x64 W10 on macos catalina and enjoy the advantage of easy access to snapshots. Quote Link to comment
Skylinar Posted May 9, 2021 Share Posted May 9, 2021 @mark41 pleased to hear that my research could bring you further! Quote Link to comment
Van Posted May 27, 2021 Share Posted May 27, 2021 (edited) On 5/9/2021 at 2:01 PM, mark41 said: @SkylinarThank you for you comment! Using script from Spaceinvaderone and your idea of editing vm`s XML from Penryn to host, I am now able to run x64 W10 on macos catalina and enjoy the advantage of easy access to snapshots. Hey, unraid newbie here - I'm wondering if you had the chance to write your process up? I'm going to wipe my opencore installation and pull the plug on unraid. docker within macOS is a must on my 3900x and wanna figure out all the steps before getting started. Cheers! Edited May 27, 2021 by Van Quote Link to comment
ghost82 Posted May 27, 2021 Share Posted May 27, 2021 58 minutes ago, Van said: on my 3900x No way to do it, at least of what I'm aware.. Quote Link to comment
KptnKMan Posted May 30, 2022 Share Posted May 30, 2022 Hi, was this ever resolved to run nested virtualisation on AMD? I've been trying to get Docker to work within a Windows 10 VM, any advice? Quote Link to comment
tjb_altf4 Posted May 31, 2022 Share Posted May 31, 2022 (edited) 9 hours ago, KptnKMan said: Hi, was this ever resolved to run nested virtualisation on AMD? I've been trying to get Docker to work within a Windows 10 VM, any advice? It works for me my 1950X, has been working for about 12 months now. Steps were something like: unraid: backup vm (if nested doesn't work properly, vm will brick) unraid cmd: modprobe -r kvm_amd unraid cmd: modprobe kvm_amd nested=1 unraid: reboot win10 vm: install wsl2 (and associated restart) win10 vm: install docker desktop Performance was crap for me in 6.9 when nested virt was enabled, but perf got a lot better in 6.10 (but this may have been my other dockers being ram hogs during 6.9 time) Edited May 31, 2022 by tjb_altf4 1 Quote Link to comment
KptnKMan Posted May 31, 2022 Share Posted May 31, 2022 5 hours ago, tjb_altf4 said: It works for me my 1950X, has been working for about 12 months now. Steps were something like: unraid: backup vm (if nested doesn't work properly, vm will brick) unraid cmd: modprobe -r kvm_amd unraid cmd: modprobe kvm_amd nested=1 unraid: reboot win10 vm: install wsl2 (and associated restart) win10 vm: install docker desktop Performance was crap for me in 6.9 when nested virt was enabled, but perf got a lot better in 6.10 (but this may have been my other dockers being ram hogs during 6.9 time) Thanks, I'm gonna try this. Will report results. Quote Link to comment
KptnKMan Posted May 31, 2022 Share Posted May 31, 2022 So I got it working. I found that KVM already had AMD extensions and nested Virtualisation enabled: root@primary:~# modprobe -r kvm_amd modprobe: FATAL: Module kvm_amd is in use. root@primary:~# systool -m kvm_amd -v | grep nested nested = "1" root@primary:~# Enabling the nested module did nothing (As expected): root@primary:~# modprobe kvm_amd nested=1 root@primary:~# I've had issues with this before, because I remember last year I enabled Docker extensions in Visual Studio Code and bricked my VM (Restored from nightly backup so no big deal) but I never tried that again. So I checked my VMs, and per advice around the forums I added the vmx CPU flag: <cpu mode='host-passthrough' check='none' migratable='on'> ... <feature policy='require' name='vmx'/> ... </cpu> ...and started up my VM, installed Hyper-V and Docker and got it running in WSL2. No errors in Device Manager were seen. I didn't even need to reboot because the nested extensions were enabled. For good measure, I downloaded SpaceInvaderOne's script and enabled it (And fixed it because there is an error in there on line 20), but it is mostly redundant because the extensions are already enabled. Still, gives me a little more control if I want it in future. I have to say though, the performance took a hit. I've seen reports that SeaBIOS is more performant, but I'd rather stay with OVMF if I can. Just looking at Task Manager was a big oof! I'm going to have to assign more resources to this VM! Thanks for all your help! 1 Quote Link to comment
KptnKMan Posted May 31, 2022 Share Posted May 31, 2022 A quick followup on this... I noticed that the VM is not just more loaded, but was noticeably more "sluggish" and slow generally. I tried and failed to convert the VM to SeaBIOS, to test comparable performance, and while attempting this I noticed this error would appear each time I tried to update the GUI while the vmx CPU flag was in place (Removing the flag avoids the error): So eventually, I made a backup and removed the vmx CPU flag, expecting my VM to die or brick or something strange. Nothing happened, Hyper-V works within the VM, Docker works, WSL2 works, and it "seems" to be a little snappier. I'm not entirely sure what is happening here (It could be more related to updates in 6.10.x), but I thought I would post it just so other know, the "vmx" CPU flag might not be required after all. ¯\_(ツ)_/¯ Quote Link to comment
ghost82 Posted June 1, 2022 Share Posted June 1, 2022 (edited) 16 hours ago, KptnKMan said: while the vmx CPU flag was in place (Removing the flag avoids the error): I think this is because you are using cpu host-passthrough, but since it's an amd it doesn't have such feature (vmx). It could work if the cpu is intel emulated, like Skylake-Client-IBRS or something else supporting the vmx flag. Quote When starting a VM Guest with CPU mode hostl, libvirt will copy its model of the host CPU into the VM Guest definition. The host CPU and features copied to the VM Guest definition can be observed in the output of the virsh capabilities. What changed for amd nested virtualization compatibility is in windows itself, that added support for this, if I remember well starting when the first previews of windows 11 were available, then they were ported to windows 10 too. Anyway since it now works with amd host-passthrough, windows is able to make it work with svm (vmx equivalent for amd). Edited June 1, 2022 by ghost82 Quote Link to comment
KptnKMan Posted June 1, 2022 Share Posted June 1, 2022 49 minutes ago, ghost82 said: I think this is because you are using cpu host-passthrough, but since it's an amd it doesn't have such feature (vmx). It should work if the cpu is intel emulated, like Skylake-Client-IBRS or something else. What changed for amd nested virtualization compatibility is in windows itself, that added support for this, if I remember well starting when the first previews of windows 11 were available, then they were ported to windows 10 too. Anyway since it now works with amd host-passthrough, windows is able to make it work with svm (vmx equivalent for amd). Ah, I thought vmx was used across Intel and AMD, but it was a question of support. Like a common layer on top of VT-X/AMD-V, but alas I am mistaken. As usual, there is a separate name and term for a similar technology between Intel/AMD. ¯\_(ツ)_/¯ I also looked into KVM support for VMX/SVM and found there's quite a bit of information, like here and here. As for guest support, and guest awareness of VMX/SVM extensions, that seems to be another layer of issues that (In this case) has had some progress in Windows 10/11. Using Docker and VMs within Windows 10 has given some excitement to test Windows 11 again, now that the new OVMF-TPM BIOS is generally released in unRAID 6.10.x . Time will tell how reliable that is, but I actually found Windows 11 to be faster in VM than Windows 10, before the TPM limitations ended the fun. Quote Link to comment
ghost82 Posted June 1, 2022 Share Posted June 1, 2022 Yes, exact. Basically, Intel: VT --> intel virtualization technology This splits into VT-x and Vt-d VT-x: hardware virtualization assistance VT-d: allows direct passthrough (enables direct i/o for virtualization) vmx: virtualization extension flag in cpu You can run vms if hardware supports VT-x, even with VT-d disabled or not supported; you must have VT-d support for passed through devices. AMD: AMD-V: same as VT-x AMD-VI: same as VT-d svm: same as vmx On top of that if you emulate an intel cpu in qemu, that cpu must support also EPT (extended page tables) and UG (unrestricted guest): if you look at the first image of the first post, the user is having that issue because he was emulating a Penryn cpu (intel core 2 duo) that lacks these features. As far as I now enabling nested virtualization in a linux host, whatever it is, is not recent, same for VT-x/AMD-V, VT-d/AMD-VI and vmx/svm. Nested virtualization for intel was more or less always working for long time, amd was poorly supported by guest oses; linux guests were the firsts to fix this, then came microsoft, apple never did this and will never do for amd, although nested virtualization works for intel (passed through or emulated). 1 Quote Link to comment
facile-soot3303 Posted March 20 Share Posted March 20 Check the Libvirt storage location: wahl then other location Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.