Almost exact same thing here.
I have very few plugins that aren't resource intensive at all (all of them are Dynamix extensions), not a single Docker, and 3 VMs running W10 with 4GB+4GB+3GB of RAM, with 16GB of available RAM.
If I set them up with 4GB+4GB+4GB, all of them immediately shutdown when they're all up.
I've done a full memtest and everything is correct, 0 errors. It might seems that the RAM overhead needed for each VM is roughly ~25% of it's allocated memory. unRAID reports 200MB of used RAM and 1.4GB of cached RAM when all VMs are off, which should allow plenty of room to allocate 12GB to VMs. (I can provide any logs required)
I think that there needs to be at least a tool, or configuration, that allows unRAID users to reserve some amount of RAM to all of KVM or to each VM in particular so, at least, unRAID logs some kind of error or halts the startup of a VM; instead of shutting everything down.
Edit: here's a snapshot of RAM usage with all 3 VMs up (4GB+4GB+3GB)
RAM usage with all VMs off: