I am running into the exact same situation that you have discribed. I have installed about 10 different linux distros ranging from lightweight AntiX to fatty Ubuntu, Arch based, and Red Hat based... They all do what you have outlined. They are supposed to use virt-io ballooning to dynamically grow and shrink between what is specified in the VM config but they never grow:
Virtio balloon device:
VM config:
Htop:
While troubleshooting, I decided to disable the virtio memory ballon PCI device to see what would happen.
Remove PCI device:
Htop:
The bare metal memory usage didn't spike after removing it so it seemed promising. To continue testing, I opened a crap-ton of tabs and saw the memory usage was able to surpass the initial 2.5G, which was good:
The issue I ran into was that when I closed the browser, memory usage in the VM decreased, but the host never reclaimed any of it.
Before VM start:
VM running and virtio balloon device removed:
After opening a lot of tabs then closing them:
So, in short... yeah, it's broken. no idea why, but it seems to be specific to unraid as the same VM config in proxmox grows and reclaims memory properly:
I'm new to unraid but, IMO, unraid's hypervisor is subpar...