RAM Usage - Killing my VM's

[poopsie]

My unraid has a very high RAM usage.  Now I do understand that it is the RAM Cache, and it is not really using it all up.  But, it is killing my VM's.  My machine has 14GB of RAM, only allocating 1GB to my Ubuntu server and 4GB to my Server 2012, plus 2 dockers running.  When they are all shut down, I am still using 84% of my RAM for unraid.  I read that there is a way to tell unraid to only use so much RAM, is it safe to do that?  What are the downsides of doing that?  Is it done in something called dirty config?

 

Thanks!!!

[BRiT]

http://www.linuxatemyram.com/

 

Unused RAM is worthless RAM.

[poopsie]

That i understand, so the next question to ask then is does UNRAID bump out anything else that is using RAM to associate it to cache?  If so, what is the point of allowing VM's than if eventually they will be taken offline for the unused RAM?

[BRiT]

Until you post up diagnostics after your VMs are killed no one can answer accurately about your situation.

[Squid]

That i understand, so the next question to ask then is does UNRAID bump out anything else that is using RAM to associate it to cache?  If so, what is the point of allowing VM's than if eventually they will be taken offline for the unused RAM?

memory that is cached is released when another process requires it.  VMs preallocate their memory and it is theirs and theirs alone. 

 

I think a better response however to your initial question is to justify it.  Where exactly are you seeing the 84% utilized and what version of unRaid are you on?  What is the output of

free -m

[poopsie]

Thanks for the replies.  Here is the output after a I rebooted and shutdown my VM's

 

              total        used        free      shared  buff/cache  available

Mem:          14026        1294        1041        6927      11690        5297

Swap:            0          0          0

 

Is it possible that if I started a VM after the ram cache was full, it can't reallocate? 

 

 

[itimpi]

Thanks for the replies.  Here is the output after a I rebooted and shutdown my VM's

 

              total        used        free      shared  buff/cache  available

Mem:          14026        1294        1041        6927      11690        5297

Swap:            0          0          0

 

Is it possible that if I started a VM after the ram cache was full, it can't reallocate?

no.  Virtually all the RAM listed under the buffer/cache column will be reclaimed and used by applications if needed.

[poopsie]

So my VM's should be taking RAM as needed correct?  So back to my original question, why did my VM shutdown when my usage on unraid was really high?

[Squid]

So my VM's should be taking RAM as needed correct?  So back to my original question, why did my VM shutdown when my usage on unraid was really high?

Without you posting your diagnostics after it happens (and before you possibly reboot), we don't know.

[poopsie]

Next time it happens, I will do that.  Thanks Squid!

[Kristijan]

Hi,

 

Now I have this problem. Cached RAM is about 50% and when i turn on new VM other shutting down. Unraid dont free up cached space. This is log for VM which killed.

 

How to fix this. I used latest stable unraid version 6.11.5

 

-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.0","addr":"0x4"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/001 Windows 10/vdisk1.img","node-name":"libvirt-3-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-3-storage"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x5","drive":"libvirt-3-format","id":"virtio-disk2","bootindex":1,"write-cache":"on"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/en_windows_10_multi-edition_version_1709_updated_sept_2017_x64_dvd_100090817.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.0","unit":0,"drive":"libvirt-2-format","id":"ide0-0-0","bootindex":2}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.160-1.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.0","unit":1,"drive":"libvirt-1-format","id":"ide0-0-1"}' \
-netdev tap,fd=38,vhost=on,vhostfd=43,id=hostnet0 \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:46:c6:b0","bus":"pci.0","addr":"0x3"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=36,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:1,websocket=5701,audiodev=audio1 \
-k en-us \
-device '{"driver":"qxl-vga","id":"video0","max_outputs":1,"ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"bus":"pci.0","addr":"0x2"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.0","addr":"0x6"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/12 (label charserial0)
qxl_send_events: spice-server bug: guest stopped, ignoring
2023-01-10T22:48:49.034467Z qemu-system-x86_64: terminating on signal 15 from pid 6937 (/usr/sbin/libvirtd)
2023-01-10 22:48:50.035+0000: shutting down, reason=destroyed
2023-01-11 09:23:29.045+0000: starting up libvirt version: 8.7.0, qemu version: 7.1.0, kernel: 5.19.17-Unraid, hostname: Tower
LC_ALL=C \
PATH=/bin:/sbin:/usr/bin:/usr/sbin \
HOME='/var/lib/libvirt/qemu/domain-9-001 Windows 10' \
XDG_DATA_HOME='/var/lib/libvirt/qemu/domain-9-001 Windows 10/.local/share' \
XDG_CACHE_HOME='/var/lib/libvirt/qemu/domain-9-001 Windows 10/.cache' \
XDG_CONFIG_HOME='/var/lib/libvirt/qemu/domain-9-001 Windows 10/.config' \
/usr/local/sbin/qemu \
-name 'guest=001 Windows 10,debug-threads=on' \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-9-001 Windows 10/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/c0a6f337-6367-466f-9268-f1e99af23c35_VARS-pure-efi.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-i440fx-3.0,usb=off,dump-guest-core=off,mem-merge=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-accel kvm \
-cpu host,migratable=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vendor-id=none \
-m 4096 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":4294967296}' \
-overcommit mem-lock=off \
-smp 4,sockets=1,dies=1,cores=4,threads=1 \
-uuid c0a6f337-6367-466f-9268-f1e99af23c35 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=39,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device '{"driver":"ich9-usb-ehci1","id":"usb","bus":"pci.0","addr":"0x7.0x7"}' \
-device '{"driver":"ich9-usb-uhci1","masterbus":"usb.0","firstport":0,"bus":"pci.0","multifunction":true,"addr":"0x7"}' \
-device '{"driver":"ich9-usb-uhci2","masterbus":"usb.0","firstport":2,"bus":"pci.0","addr":"0x7.0x1"}' \
-device '{"driver":"ich9-usb-uhci3","masterbus":"usb.0","firstport":4,"bus":"pci.0","addr":"0x7.0x2"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.0","addr":"0x4"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/001 Windows 10/vdisk1.img","node-name":"libvirt-3-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-3-storage"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x5","drive":"libvirt-3-format","id":"virtio-disk2","bootindex":1,"write-cache":"on"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/en_windows_10_multi-edition_version_1709_updated_sept_2017_x64_dvd_100090817.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.0","unit":0,"drive":"libvirt-2-format","id":"ide0-0-0","bootindex":2}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.160-1.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.0","unit":1,"drive":"libvirt-1-format","id":"ide0-0-1"}' \
-netdev tap,fd=40,vhost=on,vhostfd=46,id=hostnet0 \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:46:c6:b0","bus":"pci.0","addr":"0x3"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=38,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:2,websocket=5702,audiodev=audio1 \
-k en-us \
-device '{"driver":"qxl-vga","id":"video0","max_outputs":1,"ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"bus":"pci.0","addr":"0x2"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.0","addr":"0x6"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/31 (label charserial0)
qxl_send_events: spice-server bug: guest stopped, ignoring
2023-01-11 09:58:28.730+0000: shutting down, reason=crashed

[apandey]

You should post diagnostics as suggested. Just VM log does tell what else was going on in the system. Also, I don't know what is telling you that this is a memory issue. Looks more like the VM isn't starting and that would explain why no memory is claimed or allocated. The effect may not be the cause here

 

EDIT: I also noticed you are not the thread starter. Better to start a new thread in such cases rather than continue on someone else's problem which may not be same issue

Edited January 11, 2023 by apandey

[Kristijan]

OK ,i will start new trade. VM is normally started, and then I start new unraid kill one vm and i can see in unraid log out of memory, but I have 32GB memory and starter few vm with 4 and 2 GB.