Dynamic memory for virtual machines -- how is this actually designed to work?

[manikyath]

i've been on quite a search to figure out if i can make a VM in unraid that dynamically gets memory assigned when it needs it, or when the host doesnt need it. Either way will work really.

 

In short, my goal is to consolidate some tasks from another server into a VM on my unraid machine.

Because of some long-winded requirements i cant use a docker container, and i dont have enough RAM in my server to dedicate the amount the VM 'could' use, while still having enough memory spare for what other services on my box 'could' use. Having a shared 'pool' is fine however.

 

For solving this problem, i got interested in the ways behind the "Initial Memory" and "Max Memory" settings in the VM settings. However, the behavior doesnt seem to make much sense in my testing.

 

My current setup is a host machine with 32GB RAM, running unraid 6.9.2

The virtual machine is running debian 11.2, configured with 2GB Initial memory and 16GB Max memory.

Other services currently running are consuming approximately 

 

The behavior i'm experiencing here is as follows:

- once the VM starts, it shows to have 16GB memory allocated

- as the system has started up, the memory allocation depicted by unraid, and in the VM as listed by htop, lowers down to around the initial memory configruation: 2GB in unraid, and about 1.6GB in debian (i'm assuming driver losses here.)

- when i run a memory intensive application in the VM, it'll start eating into swapfile, and eventually crashes with an out of memory error, the VM never gets any additional memory past what it dropped down to after startup.

- i can manually assign the VM more memory in the 2 to 16GB memory range trough terminal commands on unraid, but past the initial shrink it doesnt appear to take any automatic action in either direction.

 

Is this how it is intended to work with unraid?

Is there supposed to be any sort of automated process that allocates memory as needed, and it just appears broken for my exact setup?

 

i've dug trough every bit of documentation i could find on the topic, and have come up short unfortunately.

[tomd]

I am running into the exact same situation that you have discribed.  I have installed about 10 different linux distros ranging from lightweight AntiX to fatty Ubuntu, Arch based, and Red Hat based...  They all do what you have outlined.  They are supposed to use virt-io ballooning to dynamically grow and shrink between what is specified in the VM config but they never grow:

 

Virtio balloon device:

Spoiler

 

VM config:

Spoiler

 

Htop:

Spoiler

 

While troubleshooting, I decided to disable the virtio memory ballon PCI device to see what would happen.

 

Remove PCI device:

Spoiler

 

Htop:

Spoiler

 

The bare metal memory usage didn't spike after removing it so it seemed promising.  To continue testing, I opened a crap-ton of tabs and saw the memory usage was able to surpass the initial 2.5G, which was good:

Spoiler

 

The issue I ran into was that when I closed the browser, memory usage in the VM decreased, but the host never reclaimed any of it.
 

Before VM start:

Spoiler

 

VM running and virtio balloon device removed:

Spoiler

 

After opening a lot of tabs then closing them:

Spoiler

 

So, in short... yeah, it's broken.  no idea why, but it seems to be specific to unraid as the same VM config in proxmox grows and reclaims memory properly:

Spoiler

 

I'm new to unraid but, IMO, unraid's hypervisor is subpar...

[Crad]

Did you ever find a solution for this?

[manikyath]

6 hours ago, Conrad Allan said:

Did you ever find a solution for this?

so far the only solution i've found is to switch to proxmox, which is a really disappointing to say on the unraid forums.

[unifiedmamba]

Has anything even changed here?

It seems like this topic isn't even recognized by the developers of unraid. It's a feature you pay for and it's definitely not working correctly, so something should be done about this.

I am having the same issue and was not able to find any fix for this.

 

Since Unraid just uses KVM and qemu which are standard, could this just be a configuration issue of one of these services?

Edited January 5, 2023 by unifiedmamba

[manikyath]

1 minute ago, unifiedmamba said:

Has anything even changed here?

It seems like this topic isn't even recognized by the developers of unraid. It's a feature you pay for and it's definitely not working correct, so something should be done about this.

I've got the same issue and was not able to find any fix for this.

 

Since Unraid just uses KVM and qemu which are standard, could this just be a configuration issue of one of these services?

no change on the matter, but I also havent chased it any more. for all i know the latest release of unraid has magically resolved the issue.

 

i've figured out (almost a year ago now) that dynamic memory is a bit weird in KVM, and their own documentation sort of implies that it's not supposed to be used this way. and that dynamic memory also doesnt work on a VM with IOMMU features.

 

i'm assuming that the issue is one of the following:

- unraid might accidentially set some flags on the IOMMU side that mess with dynamic memory.

- other platforms that focus a lot more on the 'VM farm' aspect  of things have just put in the R&D of making it work.

 

given the traction this thread is getting, i think it's safe to say "there's half a douzen of us" that have encountered this issue.

i've since concluded that it's better to chase my 'it must be in a VM' limitation, than to chase this oddity with dynamic memory.

[unifiedmamba]

19 hours ago, manikyath said:

for all i know the latest release of unraid has magically resolved the issue.

Sadly not for me. Just tried yesterday.

 

The documentation of unraid doesn't help either. It just mentions the feature casually but I can't find more information about it. Maybe someone "official" could help out here, it could be us using or understanding the feature wrong .

[unifiedmamba]

Bump

[unifiedmamba]

Nice, that's what I call support. 

 

I'll write an email to the official support team listed on the website and post the result here.

[HappyBobJr]

 

Edited January 29, 2023 by HappyBobJr

[unifiedmamba]

AndrewZ replied to my request:

Quote

 

Hey there,

 

Unraid simply passes the appropriate flags to the KVM system to support memory ballooning.  This feature is pretty much limited to Linux VMs and is something that is a very advanced feature, and you most likely require the virtio guest tools installed on the VM.  Googling how this gets handled (not using search terms that are unraid specific) is your best bet.  99.99% of our users (and ourselves) have no need / use case for this feature, and your best place for support on KVM and memory ballooning on Debian is on the greater internet.

 

Andrew

 

 

...

[manikyath]

17 minutes ago, unifiedmamba said:

AndrewZ replied to my request:

 

...

by that i assume that this previous assumption was correct, then:

 

On 1/5/2023 at 3:40 PM, manikyath said:

other platforms that focus a lot more on the 'VM farm' aspect  of things have just put in the R&D of making it work.

 

[JorgeB]

I've been doing some testing and ballooning is working with Unraid and Windows, this is a VM set to 4GB minimum and 16GB max, before installing the balloon driver and service:

 

 

Same VM after installing the ballon driver and service:

 

 

Note that the used RAM reported by Windows is not correct, this is normal, see this post:

https://github.com/virtio-win/kvm-guest-drivers-windows/issues/568

 

Couple of notes:

 

- Install both the balloon driver and service, though the driver seems enough for this to work the service reportedly helps with performance, you can install everything by running virtio-win-gt-x64.exe from the virtio ISO root.

 

- memory ballooning won't work if any PCI(e) device is being passed-through to that VM, this is because all the RAM must be allocated for possible DMA use.

[manikyath]

1 minute ago, JorgeB said:

I've been doing some testing and ballooning is working with Unraid and Windows, this is a VM set to 4GB minimum and 16GB max, before installing the balloon driver and service:

 

 

 

Same VM after installing the ballon driver and service:

 

 

 

Note that the used RAM reported by Windows is not correct, this is normal, see this post:

https://github.com/virtio-win/kvm-guest-drivers-windows/issues/568

 

Couple of notes:

 

- Install both the balloon driver and service, though the driver seems enough for this to work the service reportedly helps with performance, you can install everything by running virtio-win-gt-x64.exe from the virtio ISO root.

 

- memory ballooning won't work if any PCI(e) device is being passed-through to that VM, this is because all the RAM must be allocated for possible DMA use.

 

the "question" in the thread is how to make it re-release RAM to the virtual machine automagically, and if that's possible at all on unraid.

 

i know the balloon driver is needed, i know it doesnt work if PCIe devices are passed trough, the behavior you describe is how far i got as well, but after that the RAM doesnt come back to the VM on demand, like it does with for example proxmox.

[JorgeB]

7 hours ago, manikyath said:

the "question" in the thread is how to make it re-release RAM to the virtual machine automagically

Yes, and I'm sorry, I was though I was replying to a "ballooning doesn't work" thread, and that's not what this is about, I wasn't sure myself but from what I can find KVM memory balloon is not dynamic:

"You don't mention which hypervisor you're using, so I'm going to guess KVM, but my answer applies to Xen too. When you set currentMemory=16384 and maxMemory=32768, you are telling KVM to boot with 32 GB of RAM and use balloon driver to reduce that to 16 GB.Once the balloon driver has reduced the RAM to 16 GB, it will never be increased again unless you explicitly tell KVM to increase. It won't adjust itself without you telling it to."

 

And at the very top of the link I posted above:

"EDIT: For future reference, virtio memory ballooning is not dynamic by default. Meaning, if you want the guest to request more memory from the hypervisor based on something like memory pressure, that would be up to you to set up (a script talking to a virto channel should work)."

 

So I believe, and if I'm wrong someone please correct me, that it will never automatically add or remove allocated RAM to a VM, you can do it manually or with some sort of script, for example, say I want my Windows VM to usually use 2GB of RAM, but sometimes I need to run some app that requires 16GB, I set the VM to 2GB/16GB, install the balloon driver and service and after boot the VM will only use 2GB, the rest will be released to the host, when I need the extra RAM type in the Unraid console:

 

virsh setmem "Windows 10" --size 16777216

 

Of course the host needs to have enough free RAM at that time, when the VM no longer requires the extra RAM type:

 

virsh setmem "Windows 10" --size 2097152

 

And the VM goes back to using 2GB of RAM, releasing the rest to the host, basically this as implemented allows you to add or remove allocated RAM to a VM on the fly and with the VM running, since KVM does not support dynamically changing this I guess it would be possible for Unraid to change it based on the RAM pressure of the host/VM, but no idea if this is even possible to implement for all guest OSes, I guess users can make a feature request and if there is enough interest and it's feasible LT might do it in the future.

 

[manikyath]

16 hours ago, JorgeB said:

 

So I believe, and if I'm wrong someone please correct me, that it will never automatically add or remove allocated RAM to a VM, you can do it manually or with some sort of script, for example, say I want my Windows VM to usually use 2GB of RAM, but sometimes I need to run some app that requires 16GB, I set the VM to 2GB/16GB, install the balloon driver and service and after boot the VM will only use 2GB, the rest will be released to the host, when I need the extra RAM type in the Unraid console:

this is the assumption i came to so far, and the platfoms where it "works" have just implemented some sort of in-house script to automate the process.

[zaberchann]

On 3/13/2023 at 2:59 AM, JorgeB said:

Yes, and I'm sorry, I was though I was replying to a "ballooning doesn't work" thread, and that's not what this is about, I wasn't sure myself but from what I can find KVM memory balloon is not dynamic:

"You don't mention which hypervisor you're using, so I'm going to guess KVM, but my answer applies to Xen too. When you set currentMemory=16384 and maxMemory=32768, you are telling KVM to boot with 32 GB of RAM and use balloon driver to reduce that to 16 GB.Once the balloon driver has reduced the RAM to 16 GB, it will never be increased again unless you explicitly tell KVM to increase. It won't adjust itself without you telling it to."

 

And at the very top of the link I posted above:

"EDIT: For future reference, virtio memory ballooning is not dynamic by default. Meaning, if you want the guest to request more memory from the hypervisor based on something like memory pressure, that would be up to you to set up (a script talking to a virto channel should work)."

 

So I believe, and if I'm wrong someone please correct me, that it will never automatically add or remove allocated RAM to a VM, you can do it manually or with some sort of script, for example, say I want my Windows VM to usually use 2GB of RAM, but sometimes I need to run some app that requires 16GB, I set the VM to 2GB/16GB, install the balloon driver and service and after boot the VM will only use 2GB, the rest will be released to the host, when I need the extra RAM type in the Unraid console:

 

virsh setmem "Windows 10" --size 16777216

 

Of course the host needs to have enough free RAM at that time, when the VM no longer requires the extra RAM type:

 

virsh setmem "Windows 10" --size 2097152

 

And the VM goes back to using 2GB of RAM, releasing the rest to the host, basically this as implemented allows you to add or remove allocated RAM to a VM on the fly and with the VM running, since KVM does not support dynamically changing this I guess it would be possible for Unraid to change it based on the RAM pressure of the host/VM, but no idea if this is even possible to implement for all guest OSes, I guess users can make a feature request and if there is enough interest and it's feasible LT might do it in the future.

 

I agree with your point after delving into the details about kvm balloon. Currently, no official support for dynamic RAM. On Windows VMs, they do support automatic increase of the RAM. On Linux VMs, they support neither automatic increase nor decrease.

Edited October 26, 2023 by zaberchann