KVM Memory Leaking/Overhead?


Recommended Posts

Hi Everyone,

 

I've mentioned this a few times in the beta topics but haven't found anything out at all

 

I have an issue where by my VMs are using a lot more memory than they are allocated - i have rather a high end system so would love to virtualize more but i refuse to buy more memory if the VMs are wasting it

 

I have attached my system log for anyone interested

 

For those that don't read the system log here is an overview of my memory usage

 

My system has 32GB of Memory and an 8GB swap file and it is allocated as follows to my VMs

 

VM1: 12GB

VM2: 8GB

VM3: 2GB

 

Totaling 22GB so the system has 10GB free (not catering in vm overhead)

 

Here is the interesting part, this is the memory usage reported by htop for the vms

 

VM1: 18.4G VIRT | 17.1G RES

VM2: 10.7G VIRT | 10.3G RES

VM3: 3889M VIRT | 2116M RES

 

All in that totals to around 29.46GB (RES) allocated to my VMs

My entire server memory usage is 33.4GB

 

I do have 5 dockers running as well but they are using a huge 450MB between all of them

Ooo and system stats reports 220MB cache, and 320MB free

 

Now i know i should expect some overhead but 8GB of memory is not a small amount to lose considering i have a VM that can only have 2GB due to this

 

I never really noticed if this was an issue on 6.1.9 but would love some help to figure out why the overhead is so bad and what i can do to get some memory back on my machine - The longer the system runs, the more the VMs use and i think going to 64GB memory is just going to allow them to bloat even more

 

Any ideas, comment, or thoughts would be appreciated

 

Regards,

Jamie

 

p.s. i'm aware this issue is probably occuring to others so any resolutions should help more people than just me

archangel-diagnostics-20160616-1923.zip

Link to comment

Hi Everyone,

 

I've mentioned this a few times in the beta topics but haven't found anything out at all

 

I have an issue where by my VMs are using a lot more memory than they are allocated - i have rather a high end system so would love to virtualize more but i refuse to buy more memory if the VMs are wasting it

 

I have attached my system log for anyone interested

 

For those that don't read the system log here is an overview of my memory usage

 

My system has 32GB of Memory and an 8GB swap file and it is allocated as follows to my VMs

 

VM1: 12GB

VM2: 8GB

VM3: 2GB

 

Totaling 22GB so the system has 10GB free (not catering in vm overhead)

 

Here is the interesting part, this is the memory usage reported by htop for the vms

 

VM1: 18.4G VIRT | 17.1G RES

VM2: 10.7G VIRT | 10.3G RES

VM3: 3889M VIRT | 2116M RES

 

All in that totals to around 29.46GB (RES) allocated to my VMs

My entire server memory usage is 33.4GB

 

I do have 5 dockers running as well but they are using a huge 450MB between all of them

Ooo and system stats reports 220MB cache, and 320MB free

 

Now i know i should expect some overhead but 8GB of memory is not a small amount to lose considering i have a VM that can only have 2GB due to this

 

I never really noticed if this was an issue on 6.1.9 but would love some help to figure out why the overhead is so bad and what i can do to get some memory back on my machine - The longer the system runs, the more the VMs use and i think going to 64GB memory is just going to allow them to bloat even more

 

Any ideas, comment, or thoughts would be appreciated

 

Regards,

Jamie

 

p.s. i'm aware this issue is probably occuring to others so any resolutions should help more people than just me

 

Hi Jamie,

 

This is an interesting subject and I'm surprised it has taken this long to come up, but now that it has, let's address it!  First, let's be clear about RAM usage for virtual machines.  There are three primary things that use memory space for VMs to run:

 

[*]The assigned guest RAM (user-defined when creating a VM)

[*]The guest video RAM (for virtual graphics, not GPU pass through)

[*]QEMU itself (for the emulated components such as virtual storage/network controllers, etc.)

 

#3 is the tricky one, because it's impossible to discern exactly how much memory we'll need for the guest at any given time.  Libvirt documentation itself even says so:

 

...Thus, when counting them, one needs to add up guest RAM, guest video RAM, and some memory overhead of QEMU itself. The last piece is hard to determine so one needs guess and try.

 

Great, so what does all this mean in relation to the OP?

 

In short, the first thing we need to determine is whether or not this is a memory leak or just an overhead issue.  For it to be considered a leak, we would have to see a persistent growth in memory consumption for guests even with no activity inside them.  If you can boot up the guests and disable any automated services from running inside them, capture the memory usage, then let them sit idle for a while and capture it again, we can compare and see if a dramatic growth was seen just by letting them sit.  That would be a useful test.

 

The second thing we need to do is discuss whether or not memory ballooning is appropriate for your guests.  First and foremost, if you have any guests that do not have any PCI devices passed through, they are eligible to use memory ballooning.  To leverage this feature, edit your VM in the webGui and turn on Advanced View mode.  From here, adjust the minimum/maximum values and apply your changes.  For multiple guests running on a host at the same time that use this feature, this lets KVM "rob peter to pay paul" when it comes to RAM.  Again, this WILL NOT WORK with guests using PCI device assignment (such as GPU pass through).

 

Lastly, I would like to see some others report this as well.  It would be helpful to know if everyone is experiencing this increase to memory consumption or if this is an isolated event.

Link to comment

Hi Jonp,

 

Thanks for the reply to this. To go over my vm's more now i know the cases, here is a bit more overhead information

 

VM1 (12gb allocated)

 

Has a passed through GPU (780)

Is using a bridged network

Has 5 passed through usb devices (from the gui)

VM operating system is reporting 4.5GB used of 12GB (just in case this helps)

This VM has CCTV recording to it and otherwise has been sat idle at 6% cpu usage since yesterday

 

Is now using 17.7G (VIRT) and 17.4G (RES) and has not done anything all day until now - memory usage is now higher in RES but less in VIRT

 

This one is my main concern as it is using an extra 5.4GB following the RES level - it has no need to do video memory, and the vm is set to have 12,288 in the GUI - following the numbers below this would mean QEMU is having an overhead of 5.4G

 

VM2 (8gb allocated)

 

Has a passed through GPU (750Ti)

Is using a bridged network

Has 0 passed through usb devices (from the gui)

VM operating system is reporting 2.1GB of 8GB usaged

Is now using 10.7G (VIRT) and 10.3G (Res) - No change in memory usage

This VM has been sitting doing nothing since yesterday and is just sitting on the windows desktop with just AVG running

 

VM3 (2gb allocated) - was restarted since the last values but usage seems to be climbing

 

Is using VNC so will be using memory for video

Is using a bridged network

Has 0 passed through usb devices

VM operating system is reporting 1.1GB of 2GB used

Is now using 2991M (VIRT) and 1913M (RES)

 

In 5 minutes while checking this i saw this VM's RES usage go from 1913M to 1921M - VIRT remains constant - it has a CCTV program running but this saves nothing to the host and writes to network shares, it is pretty much idle at all times

 

Doing a fresh server boot i see almost all memory free, i boot these VMs up and the system jumps to ~23GB used - watching the gui stats i can then see it slowly climb

 

Using the system stats plugin i can see the following memory usage details

  • Some time after boot - 28.3GB used
  • Same time the following day - 31.6GB used
  • Same time the day after - 32.28GB used
  • Same time the day after - 31.6GB used but using 2GB swap as well
  • Today - 32.2GB usage annd 2.6GB swap

 

Hope some of that helps

 

Regards,

Jamie

Link to comment

So i just wanted to post an update as things have gotten worse since my last post and now dockers are starting to not work as they can't use any memory

 

VM1: 18G VIRT, 17.7G RES

VM2: 10.9G VIRT, 10.5G RES

VM3: 4035M VIRT, 1462M RES

 

My system has 400MB showing for cache and free combine in the stats plugin, it is still using 2.42G of my swap file as well

 

Running HTOP i can see a process for /usr/sbin/libvirtd which has the following memory usage

789M VIRT, 2972 RES

 

Right now if i don;t start turning things off my system is going to become un-usuable to dockers so this is becoming a larger and larger issue

 

Is there anything at all i can post to help with this jonp? As i genuinely don't want to have to buy another set up memory to fix this problem when it seems to just want more and more

 

Regards,

Jamie

 

Edit:

Here are a few grabs from the system that may be useful

 

virsh dommemstat "Jamie - 10586"
actual 12582912
rss 18602588

 

virsh dommemstat "CCTV"
actual 2097152
rss 1510876

 

virsh dommemstat "Cat - SeaBios"
actual 8388608
rss 10982764

 

virsh dominfo "Jamie - 10586"
Id:             4
Name:           Jamie - 10586
UUID:           8db132a6-da10-1373-64e5-3f2515cb4b17
OS Type:        hvm
State:          running
CPU(s):         8
CPU time:       573928.3s
Max memory:     12582912 KiB
Used memory:    12582912 KiB
Persistent:     yes
Autostart:      disable
Managed save:   no
Security model: none
Security DOI:   0

 

virsh dominfo "CCTV"
Id:             7
Name:           CCTV
UUID:           628a7030-7ffe-040f-e7dc-b828b045586c
OS Type:        hvm
State:          running
CPU(s):         2
CPU time:       326103.0s
Max memory:     2097152 KiB
Used memory:    2097152 KiB
Persistent:     yes
Autostart:      disable
Managed save:   no
Security model: none
Security DOI:   0

 

 

virsh dominfo "Cat - SeaBios"
Id:             2
Name:           Cat - SeaBios
UUID:           311cce28-0650-0bf3-70e8-0c25f504dcb6
OS Type:        hvm
State:          running
CPU(s):         4
CPU time:       225768.1s
Max memory:     8388608 KiB
Used memory:    8388608 KiB
Persistent:     yes
Autostart:      disable
Managed save:   no
Security model: none
Security DOI:   0

 

 

virsh nodememstats
total  :             32933444 KiB
free   :               322492 KiB
buffers:                  396 KiB
cached :               278116 KiB

 

 

top - 19:27:17 up 8 days,  6:03,  1 user,  load average: 4.20, 4.37, 4.15
Tasks: 438 total,   1 running, 437 sleeping,   0 stopped,   0 zombie
%Cpu(s): 31.9 us,  5.1 sy,  0.0 ni, 63.0 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 32933444 total,   338480 free, 31958276 used,   636688 buff/cache
KiB Swap:  8388604 total,  5959020 free,  2429584 used.   396152 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
31978 root      20   0 17.999g 0.017t  22492 S 312.0 56.5   9583:06 qemu-system-x86
8102 root      20   0 10.859g 0.010t  22380 S 190.7 33.4   3766:21 qemu-system-x86
20395 root      20   0 4132712 1.441g  14704 S  75.7  4.6   5436:46 qemu-system-x86
26511 server    20   0 2374728 256828   1072 S   1.3  0.8   0:12.69 mono-sgen
4845 root      20   0  762344  66540  18252 S   0.7  0.2 117:28.30 firefox
6708 root      20   0  819840   8088   1960 S   0.7  0.0   4:18.22 exe
20407 root      20   0       0      0      0 S   0.7  0.0  77:00.04 vhost-20395
21630 root      20   0       0      0      0 S   0.7  0.0   0:30.29 kworker/0:0
31990 root      20   0       0      0      0 S   0.7  0.0  75:50.58 vhost-31978
2547 root      20   0    9684   2504   2104 S   0.3  0.0  34:47.80 cpuload
4524 root      20   0   94476   3580   3000 S   0.3  0.0  15:39.78 emhttp
5094 root      20   0       0      0      0 S   0.3  0.0   2:12.81 xfsaild/nvme0n1
5124 root      20   0 1369872  13620    592 S   0.3  0.0 108:20.09 shfs
20411 root      20   0       0      0      0 S   0.3  0.0   5:05.00 kvm-pit/20395
20958 jamie     20   0  415400   6256   5240 S   0.3  0.0  12:10.89 smbd
    1 root      20   0    4372   1476   1476 S   0.0  0.0   0:15.93 init
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.15 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:59.45 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S   0.0  0.0   3:54.68 rcu_preempt
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_sched
    9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
   10 root      rt   0       0      0      0 S   0.0  0.0   0:01.28 migration/0
   11 root      rt   0       0      0      0 S   0.0  0.0   0:01.33 migration/1
   12 root      20   0       0      0      0 S   0.0  0.0   0:01.13 ksoftirqd/1
   14 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
   15 root      rt   0       0      0      0 S   0.0  0.0   0:03.13 migration/2
   16 root      20   0       0      0      0 S   0.0  0.0   0:00.74 ksoftirqd/2
   18 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H
   19 root      rt   0       0      0      0 S   0.0  0.0   0:02.90 migration/3
   20 root      20   0       0      0      0 S   0.0  0.0   0:00.78 ksoftirqd/3
   23 root      rt   0       0      0      0 S   0.0  0.0   0:01.83 migration/4
   24 root      20   0       0      0      0 S   0.0  0.0   0:00.63 ksoftirqd/4
   26 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/4:0H

Link to comment

Hi Jonp,

 

I have been able to remove all plugins but the following:

  • SwapFile
  • Community Applications

 

If i remove the hotswap file plugin (which was installed due to this issue) my server will instantly start shutting off VMs to conserve memory as it has massively over allocated now - i do need the Libvirt Hotplug USB plugin back at some point but have been able to remove it now

 

The server memory is still increasing but i am keeping an eye on it - i can restart my one vm with the hotswap plugin removed but it may not stay on for long

 

Jamie

Link to comment

  • Hi Jonp,
     
    I have been able to remove all plugins but the following:
    • SwapFile
    • Community Applications

 

If i remove the hotswap file plugin (which was installed due to this issue) my server will instantly start shutting off VMs to conserve memory as it has massively over allocated now - i do need the Libvirt Hotplug USB plugin back at some point but have been able to remove it now

 

The server memory is still increasing but i am keeping an eye on it

 

Jamie

While jonp has a very valid point about removing plugins to test for this kind of issue, I would suggest reinstalling fix common problems and putting it into troubleshooting mode, as additional things are logged every 10 minutes (available ram, free space on /, /var/log, tmpfs) ps -aux output, etc that may also help to shed some light on this.
Link to comment

While jonp has a very valid point about removing plugins to test for this kind of issue, I would suggest reinstalling fix common problems and putting it into troubleshooting mode, as additional things are logged every 10 minutes (available ram, free space on /, /var/log, tmpfs) ps -aux output, etc that may also help to shed some light on this.

 

Jonp is it worth me putting this back on?

 

I've got everything removed other than community apps now

 

Just to put some things into perspective, here is what happened to a fresh reboot of my VM1

Allocated - 12GB

Initial start - 12.5G VIRT, 12.3 RES

After 4 Minutes - 14.6G VIRT, 14.3G RES

 

In this time i booted up windows 10 and it has sat on the desktop doing nothing

 

Jamie

Link to comment

While jonp has a very valid point about removing plugins to test for this kind of issue, I would suggest reinstalling fix common problems and putting it into troubleshooting mode, as additional things are logged every 10 minutes (available ram, free space on /, /var/log, tmpfs) ps -aux output, etc that may also help to shed some light on this.

 

Jonp is it worth me putting this back on?

 

I've got everything removed other than community apps now

 

Just to put some things into perspective, here is what happened to a fresh reboot of my VM1

Allocated - 12GB

Initial start - 12.5G VIRT, 12.3 RES

After 4 Minutes - 14.6G VIRT, 14.3G RES

 

In this time i booted up windows 10 and it has sat on the desktop doing nothing

 

Jamie

 

I haven't actually used Squid's plugin myself before, but given what I know about it, it's probably ok because I don't think it actually adds any software packages to the system itself.  That's my main concern with plugins.

 

After removing all the plugins, did you reboot the host?  Some plugins may not remove all their software correctly so rebooting the host would be recommended to ensure a clean state.

 

Also, just want to confirm if you are still running 6.2-beta and what version.  To put things in perspective, on one of my main systems, I have 16GB of physical RAM installed and I allocate 8GB to my primary Windows 10 VM with GPU, USB, and audio pass through.  Even after running this VM almost all day today, my mem usage in htop for QEMU is showing 9028M reserved (that's the only one that matters).

Link to comment

 

I haven't actually used Squid's plugin myself before, but given what I know about it, it's probably ok because I don't think it actually adds any software packages to the system itself.  That's my main concern with plugins.

 

FCP (and CA for that matter) have zero dependencies or extra packages installed that's not already included in the base OS.  FCP in troubleshooting mode merely tails the syslog to the flash drive and adds extra diagnostic logging every 10 minutes to the syslog.  Only downside is that if you leave it running in troubleshooting mode for a couple of months with out a reboot, your flash drive will fill with the logs / diagnostics. (which are easy to delete)

 

But with it's logging of the free ram, and utilization of key RAM folders and the processes running, etc it is (IMO very useful to aid in diagnostics)

Link to comment

FCP (and CA for that matter) have zero dependencies or extra packages installed that's not already included in the base OS.  FCP in troubleshooting mode merely tails the syslog to the flash drive and adds extra diagnostic logging every 10 minutes to the syslog.  Only downside is that if you leave it running in troubleshooting mode for a couple of months with out a reboot, your flash drive will fill with the logs / diagnostics. (which are easy to delete)

 

But with it's logging of the free ram, and utilization of key RAM folders and the processes running, etc it is (IMO very useful to aid in diagnostics)

 

I didn't realize it stored so much to be honest

 

I have the plugin installed again and running in troubleshoot mode as well. I took note of memory usage during the system reboot, here are my numbers

 

I'm running beta 23

870M usage after reboot with no array running

1.55G usage after starting the array

25.5G usage 30 seconds after starting all 3 VMs

28.3G after a further 5 minutes

29G after a further 10 minutes

 

VM1: 14.7G RES

VM2: 9.7G RES

VM3: 2771M RES

 

The full system is reporting 29G of 31.4G in htop

29348852 used, 342288 free, 2967696 buff/cache

 

I hope some of this helps

Link to comment

Having removed everything from my server even the powerdown plugin i received my first full system crash today - i have been 1 month free of any system crashes, so i was forced to do a hard power reset

 

Looking through the logs folder this is the last log i found before the crash - the server didn't even make 24 hours so the log isn't huge so i was able to include it

 

The last thing i can see is unraid killing things for memory and the system hard locking not long after

 

If i can't last 24 hours without a system lock i'm going to have to put the swap file plugin back on the system

 

Jamie

archangel-diagnostics-20160622-1743.zip

Link to comment

Day 2 of system lockups, again the server didn;t last 24 hours

 

I've had to put the swap file plugin back on and will see if the server stays up or not, i have attached the diagnostics for the last failure

Not a VM expert, but from what I see at the logs is that while the memory used by the VM did increase, the last one prior to OOM was not fundamentally different from the previous ones (and certainly not any different from when memory started dropping precipitously).

 

I'm just wondering if its a docker app doing this  (something misconfigured and storing stuff into RAM?)

Link to comment

If i watch the docker usage they never really change

 

Using community apps i can see the majority of my dockers are using only 60MB or 116MB memory, my highest docker is EMBY which almost always sites at 305MB

All in they are reporting 1.83% memory usage

 

I have the swap file running now so i can see what happens tomorrow - i have put fix common problems back on troubleshoot mode so i can post what happens then

 

Edit:

One thing that makes me think it isn;t dockers is that if i have my system on for s week and shut off all vms it sits at 2GB memory usage

Link to comment

So my system hasn't shut down or crashed since my last report (it is using 600M swap at the moment)

 

My plan is to host a few of my log files online as they will be too large to post on the forum directly. Will this be of any use if i just post the links jonp?

 

As i have debug mode on i get a diagnostic every 30 minutes so can post them on intervals if needed or just the latest and you can work backwards through the logs

 

This should outline the change in VM memory and everything else over a number of days so you can see what changes?

 

Regards,

Jamie

Link to comment

Ok so i had waited but things are getting stupid now, my emby docker can not stream files or even serve artwork for videos because the server has no memory left

 

Below is a link to my latest server diagnostics, it is too large to post here but shows an outline of the changes in my system over the last 8 days of running - so as an updated:

Yes i am reliant on the swapfile plugin to keep the system running

Yes i have had to reduce my main vm's memory allocation down to 8GB from 12GB just so the rest of my server is usable

 

In total out of 32GB i can now only allocate 18GB to VMs before the system just swallows the rest of it and because a mess

 

Diagnostic link: http://www.cudamining.co.uk/archangel-diagnostics-20160701-2210.zip

 

i know i'm a pest but i could really do with this one being looked into jonp as if the server keeps chewing up my memory for nothing i'm going to have to start considering changing my OS - after having just got my vms working after months of issues

 

Regards,

Jamie

Link to comment

Set the tunable num_stripes back to default (1280), as that bug was fixed.  (A small memory gain, but probably not enough to matter with your issues.)

 

You are using memcached, which is great if it works, but my understanding is it needs to be thoroughly tested in your specific environment (unRAID with VM's), and I wasn't aware of anyone doing that yet.  If someone has, then more power to you!  Otherwise, I'd be leery.

 

Syslogs are a little strange.  Something has deleted all of the ones from when you booted on the 23rd until the 27th at 4:40am.  Then something is creating last 200 line copies of each of the syslog files, which is redundant and unnecessary.  And something is rotating the syslogs at 4:40 each morning, probably the process that's deleting the older ones, which we wanted.

 

Below is a table of memory values for the 4 largest memory users, at various periods.  I am not seeing any real growth here.  It would have been nice to have seen the earlier memory figures!  Unfortunately the Troubleshooting mode is exploding your syslogs too, so may be causing some issues on its own.  It would be nice to have a configurable period for cases like this, with long running testing.  It's currently every 10 minutes, would be better for you if it was say every 6 or 12 hours.

 

Notable mem users (in GB)

               6-27 4:40   6-29 4:40   7-1 4:40   7-1 22:00
- emby (Dc  ?)     .67         .84        .60        .06
- CCTV (VM  2)    1.51        1.39       1.30       1.55
- Cat  (VM     10.85       10.85      10.88      10.88
- Jamie(VM 12)   18.68       18.71      18.85      18.92

  Totals   22+   31.71       31.79      31.63      31.41
  Swap Used       2.19        2.34       3.12       2.34

 

You can see that emby got squeezed at the end, probably causing its issues, but I don't know why it got squeezed, as the total memory used from these users actually dropped.  CCTV doesn't ever use its 2GB allocation.  Cat is using 2.8GB more than its 8GB allocation, but never grows beyond that.  Jamie wants 6.5 GB over its 12GB, with only tiny growth after that.  Your big loss therefore appears to be the extra 6GB that Jamie wants from the start, and that's where I would concentrate troubleshooting, look for why it wants so much extra.  Those memory used figures are all about 31.5 or greater, not leaving hardly any room for unRAID, the Docker engine, or system needs.  So you are starting out at boot time already practically out of memory.  From the above, swap seems irrelevant.

 

In the last hour, load averages went over 16, so it's not surprising you couldn't do anything.  Throughout the syslogs, they generally stayed above 2, which means lags were likely.  You want them to be less than one (not that I'm an expert at this).

Link to comment

Hi RobJ

 

Thanks for the reply. By menaced do you mean the swap file that is running? I'm not aware of anything else that may be causing something like that to run? To my understanding memcached is a php extension?

 

My system boots at around 26GB memory usage with everything loaded, I've posted a few logs on the previous page where the system would crash in less than 24 hours due to memory issues, these should be small enough to show everything from the time of boot?

 

If I disable the swap file plug in I am using the vms get so large that the system shuts things down shut to save memory and the system locks up. I am using around 2.4gb swap at all times with all physical memory used

 

With regards to memory usage per vm I posted some figures about these on the previous page as well. For example if I shut down the vm Jamie and restart is it will using around 13gb at boot which is fine. Over the next day or so it will grow to us almost 19gb where it tends to stay at around that point

 

I am using beta 23 in case something new is in this one for vms that I haven't seen yet

 

Thanks again RobJ for your reply

 

Jamie

Link to comment

Additionally, troubleshooting mode was never really designed to run for weeks on end

 

;)  I'm formally requesting a configuration choice, default 10 minutes, but with the option to select longer (and shorter?) values, such as 30 minutes, 1 hour, 6 hours, 24 hours.

Link to comment

I agree that may be useful

 

So after posting last night i changed my vm Jamie to be allocated 8GB memory, the same as the VM Cat

 

I've just checked htop for both machines and here are the memory usages

VM1 Jamie:

12.9G RES

Uptime: 16 hours

 

VM2 Cat:

10.4G RES

Uptime: 93 hours

 

Both vms have 8gb allocated, both vms have a gpu passed through, both vms have the same install of windows running (same disc, same date installed, even down to updated and antivirus), both have a usb controller passed through (same make and model)

 

The only difference in them are:

VM1 is OVMF, and has 4 passed through USBs

VM2 is Seabios, and has 2 passed through USBs

 

 

Yet there is 2.5G usage difference between the 2

 

Jamie

Link to comment

Thanks for the reply. By menaced do you mean the swap file that is running? I'm not aware of anything else that may be causing something like that to run? To my understanding memcached is a php extension?

No, memcached is a performance boosting tool that can be added to an app with a database.  By its nature, it uses up memory, potentially a LOT of memory, in order to provide that boost.  I believe it's best used when the database server is remote, caching objects locally in RAM that would otherwise be stored on drives elsewhere, and accessed over limited network bandwidth.  So I'm not sure why it's used here when it's all on your server, except perhaps to cache in RAM what's on your drives.  My only experience (and it's indirect) is with Perl and MySQL, client server.  It's not a drop-in booster, it needs to be carefully hand-tooled into the application.

 

My system boots at around 26GB memory usage with everything loaded, I've posted a few logs on the previous page where the system would crash in less than 24 hours due to memory issues, these should be small enough to show everything from the time of boot?

 

With regards to memory usage per vm I posted some figures about these on the previous page as well. For example if I shut down the vm Jamie and restart is it will using around 13gb at boot which is fine. Over the next day or so it will grow to us almost 19gb where it tends to stay at around that point

When I have time, I don't right now, I'll take a look, add additional figures to the table.

 

What seems obvious though is that Jamie is your memory hog, demanding much more than it perhaps should.  As I said, troubleshooting Jamie is probably the right focus.  If you have a way to disable memcached usage, that might be an interesting test.  Or switch to a different application provider that doesn't use memcached.  Another option is to disable ballooning for Jamie, since Jamie seems to want to grab any memory it can.  Another option is to drop Cat to 6GB and Jamie to 10GB, both of which seem already excessive to me (but I'm old-school).  That would free 4GB.  Let's see if Jamie still want to grab that 4GB for itself.  It's possible that memcached is better used when it isn't sharing the available memory with other VM's.  Perhaps it needs hard limits, which is why I'm suggesting trying without ballooning.  I'm sure there are others here with more relevant experience though.

Link to comment

Thanks for the reply. By menaced do you mean the swap file that is running? I'm not aware of anything else that may be causing something like that to run? To my understanding memcached is a php extension?

No, memcached is a performance boosting tool that can be added to an app with a database.  By its nature, it uses up memory, potentially a LOT of memory, in order to provide that boost.  I believe it's best used when the database server is remote, caching objects locally in RAM that would otherwise be stored on drives elsewhere, and accessed over limited network bandwidth.  So I'm not sure why it's used here when it's all on your server, except perhaps to cache in RAM what's on your drives.  My only experience (and it's indirect) is with Perl and MySQL, client server.  It's not a drop-in booster, it needs to be carefully hand-tooled into the application.

 

My system boots at around 26GB memory usage with everything loaded, I've posted a few logs on the previous page where the system would crash in less than 24 hours due to memory issues, these should be small enough to show everything from the time of boot?

 

With regards to memory usage per vm I posted some figures about these on the previous page as well. For example if I shut down the vm Jamie and restart is it will using around 13gb at boot which is fine. Over the next day or so it will grow to us almost 19gb where it tends to stay at around that point

When I have time, I don't right now, I'll take a look, add additional figures to the table.

 

What seems obvious though is that Jamie is your memory hog, demanding much more than it perhaps should.  As I said, troubleshooting Jamie is probably the right focus.  If you have a way to disable memcached usage, that might be an interesting test.  Or switch to a different application provider that doesn't use memcached.  Another option is to disable ballooning for Jamie, since Jamie seems to want to grab any memory it can.  Another option is to drop Cat to 6GB and Jamie to 10GB, both of which seem already excessive to me (but I'm old-school).  That would free 4GB.  Let's see if Jamie still want to grab that 4GB for itself.  It's possible that memcached is better used when it isn't sharing the available memory with other VM's.  Perhaps it needs hard limits, which is why I'm suggesting trying without ballooning.  I'm sure there are others here with more relevant experience though.

 

The vm Jamie at one point had 16GB allocated to it and the more it has allocated the more it tries to take from the system so i dropped it to 12GB and then down to 8 as you can see now.

My vm for CCTV wasn't around when it was 16GB however

 

I'm not very familiar with the vm terminology or how to change anything not in the web panel, how can i disable balooning and what affect will it have on the systems? The vm Jamie sits at 2.4GB memory usage almost all the time so 8GB is plenty unless i start doing large image/video editing (web developer by trade)

 

So for my usual use i would use memcache to cache webpages served from Apache or Nginx mainly when the websites are dynamic, to my knowledge i haven't enabled it on my servers as i have no access to the php config but it seems that opcache was enabled as well (again nothing i enabled)

 

I have altered my system to disable opcache on every page load since i can not control it directly so this will remove this from the loop - although this hasn't altered the memory usage at all i don't want it on anyway (good spot!)

 

My system is right now reporting 26.8G Mem, and 1.33G Swap used and it doesn't appear to be taking more for that vm so it hasn't just taken everything it can get its haands on

 

One that may be worth a try is to see if i increase the memory for vm Cat to 12gb and see if that has the same memory usage issue as vm Jamie?

 

It may help to rule out of it is a possible OVMF issue?

 

Thanks for your help again!

 

Jamie

Link to comment
  • 2 weeks later...

So here is an interesting one - i have upgraded to rc2 and my memory issue seems to have calmed down rather a lot

 

So after upgrading i found one of my vm's (Jamie) would not boot due to some weird windows issue causing an update loop which kept failing, removing itself then applying again

 

So i did a fresh re-install and put avg free edition on the vm - the vm instantly lost internet access and explorer stopped working the second avg installed - it was so bad i couldn't shutdown, reboot or uninstall avg

 

Another windows 10 install later with no avg and everything seems to be working, this install has the newer virtio drivers and memory usage has dropped substantially

 

VM Cat: 10.5G Res - virtio from pre 6.1.9

VM Jamie: 10.1G Res - latest virtio driver

VM CCTV: 2483M Res - virtio from beta 23

 

Having the 3 vms running, my dockers and everything else i have been able to remove the swapfile plugin and my system is down to 24.8G memory usage

 

So i don't know what resolved this or helps to lower the usage but in total the following has happened since my memory usage was so high

  • Unraid updates
  • New windows install
  • New virtio drivers

 

While 2.1G overhead is a little over what i want i am very happy with it being this high and hopefully things will start to go down even more as time goes by

 

So all in i have now got a drop of 2.8G memory usage on just that 1 vm since reporting this issue

 

Jamie

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.