bigjme Posted June 16, 2016 Share Posted June 16, 2016 Hi Everyone, I've mentioned this a few times in the beta topics but haven't found anything out at all I have an issue where by my VMs are using a lot more memory than they are allocated - i have rather a high end system so would love to virtualize more but i refuse to buy more memory if the VMs are wasting it I have attached my system log for anyone interested For those that don't read the system log here is an overview of my memory usage My system has 32GB of Memory and an 8GB swap file and it is allocated as follows to my VMs VM1: 12GB VM2: 8GB VM3: 2GB Totaling 22GB so the system has 10GB free (not catering in vm overhead) Here is the interesting part, this is the memory usage reported by htop for the vms VM1: 18.4G VIRT | 17.1G RES VM2: 10.7G VIRT | 10.3G RES VM3: 3889M VIRT | 2116M RES All in that totals to around 29.46GB (RES) allocated to my VMs My entire server memory usage is 33.4GB I do have 5 dockers running as well but they are using a huge 450MB between all of them Ooo and system stats reports 220MB cache, and 320MB free Now i know i should expect some overhead but 8GB of memory is not a small amount to lose considering i have a VM that can only have 2GB due to this I never really noticed if this was an issue on 6.1.9 but would love some help to figure out why the overhead is so bad and what i can do to get some memory back on my machine - The longer the system runs, the more the VMs use and i think going to 64GB memory is just going to allow them to bloat even more Any ideas, comment, or thoughts would be appreciated Regards, Jamie p.s. i'm aware this issue is probably occuring to others so any resolutions should help more people than just me archangel-diagnostics-20160616-1923.zip Quote Link to comment
jonp Posted June 17, 2016 Share Posted June 17, 2016 Hi Everyone, I've mentioned this a few times in the beta topics but haven't found anything out at all I have an issue where by my VMs are using a lot more memory than they are allocated - i have rather a high end system so would love to virtualize more but i refuse to buy more memory if the VMs are wasting it I have attached my system log for anyone interested For those that don't read the system log here is an overview of my memory usage My system has 32GB of Memory and an 8GB swap file and it is allocated as follows to my VMs VM1: 12GB VM2: 8GB VM3: 2GB Totaling 22GB so the system has 10GB free (not catering in vm overhead) Here is the interesting part, this is the memory usage reported by htop for the vms VM1: 18.4G VIRT | 17.1G RES VM2: 10.7G VIRT | 10.3G RES VM3: 3889M VIRT | 2116M RES All in that totals to around 29.46GB (RES) allocated to my VMs My entire server memory usage is 33.4GB I do have 5 dockers running as well but they are using a huge 450MB between all of them Ooo and system stats reports 220MB cache, and 320MB free Now i know i should expect some overhead but 8GB of memory is not a small amount to lose considering i have a VM that can only have 2GB due to this I never really noticed if this was an issue on 6.1.9 but would love some help to figure out why the overhead is so bad and what i can do to get some memory back on my machine - The longer the system runs, the more the VMs use and i think going to 64GB memory is just going to allow them to bloat even more Any ideas, comment, or thoughts would be appreciated Regards, Jamie p.s. i'm aware this issue is probably occuring to others so any resolutions should help more people than just me Hi Jamie, This is an interesting subject and I'm surprised it has taken this long to come up, but now that it has, let's address it! First, let's be clear about RAM usage for virtual machines. There are three primary things that use memory space for VMs to run: [*]The assigned guest RAM (user-defined when creating a VM) [*]The guest video RAM (for virtual graphics, not GPU pass through) [*]QEMU itself (for the emulated components such as virtual storage/network controllers, etc.) #3 is the tricky one, because it's impossible to discern exactly how much memory we'll need for the guest at any given time. Libvirt documentation itself even says so: ...Thus, when counting them, one needs to add up guest RAM, guest video RAM, and some memory overhead of QEMU itself. The last piece is hard to determine so one needs guess and try. Great, so what does all this mean in relation to the OP? In short, the first thing we need to determine is whether or not this is a memory leak or just an overhead issue. For it to be considered a leak, we would have to see a persistent growth in memory consumption for guests even with no activity inside them. If you can boot up the guests and disable any automated services from running inside them, capture the memory usage, then let them sit idle for a while and capture it again, we can compare and see if a dramatic growth was seen just by letting them sit. That would be a useful test. The second thing we need to do is discuss whether or not memory ballooning is appropriate for your guests. First and foremost, if you have any guests that do not have any PCI devices passed through, they are eligible to use memory ballooning. To leverage this feature, edit your VM in the webGui and turn on Advanced View mode. From here, adjust the minimum/maximum values and apply your changes. For multiple guests running on a host at the same time that use this feature, this lets KVM "rob peter to pay paul" when it comes to RAM. Again, this WILL NOT WORK with guests using PCI device assignment (such as GPU pass through). Lastly, I would like to see some others report this as well. It would be helpful to know if everyone is experiencing this increase to memory consumption or if this is an isolated event. Quote Link to comment
bigjme Posted June 17, 2016 Author Share Posted June 17, 2016 Hi Jonp, Thanks for the reply to this. To go over my vm's more now i know the cases, here is a bit more overhead information VM1 (12gb allocated) Has a passed through GPU (780) Is using a bridged network Has 5 passed through usb devices (from the gui) VM operating system is reporting 4.5GB used of 12GB (just in case this helps) This VM has CCTV recording to it and otherwise has been sat idle at 6% cpu usage since yesterday Is now using 17.7G (VIRT) and 17.4G (RES) and has not done anything all day until now - memory usage is now higher in RES but less in VIRT This one is my main concern as it is using an extra 5.4GB following the RES level - it has no need to do video memory, and the vm is set to have 12,288 in the GUI - following the numbers below this would mean QEMU is having an overhead of 5.4G VM2 (8gb allocated) Has a passed through GPU (750Ti) Is using a bridged network Has 0 passed through usb devices (from the gui) VM operating system is reporting 2.1GB of 8GB usaged Is now using 10.7G (VIRT) and 10.3G (Res) - No change in memory usage This VM has been sitting doing nothing since yesterday and is just sitting on the windows desktop with just AVG running VM3 (2gb allocated) - was restarted since the last values but usage seems to be climbing Is using VNC so will be using memory for video Is using a bridged network Has 0 passed through usb devices VM operating system is reporting 1.1GB of 2GB used Is now using 2991M (VIRT) and 1913M (RES) In 5 minutes while checking this i saw this VM's RES usage go from 1913M to 1921M - VIRT remains constant - it has a CCTV program running but this saves nothing to the host and writes to network shares, it is pretty much idle at all times Doing a fresh server boot i see almost all memory free, i boot these VMs up and the system jumps to ~23GB used - watching the gui stats i can then see it slowly climb Using the system stats plugin i can see the following memory usage details Some time after boot - 28.3GB used Same time the following day - 31.6GB used Same time the day after - 32.28GB used Same time the day after - 31.6GB used but using 2GB swap as well Today - 32.2GB usage annd 2.6GB swap Hope some of that helps Regards, Jamie Quote Link to comment
bigjme Posted June 20, 2016 Author Share Posted June 20, 2016 So i just wanted to post an update as things have gotten worse since my last post and now dockers are starting to not work as they can't use any memory VM1: 18G VIRT, 17.7G RES VM2: 10.9G VIRT, 10.5G RES VM3: 4035M VIRT, 1462M RES My system has 400MB showing for cache and free combine in the stats plugin, it is still using 2.42G of my swap file as well Running HTOP i can see a process for /usr/sbin/libvirtd which has the following memory usage 789M VIRT, 2972 RES Right now if i don;t start turning things off my system is going to become un-usuable to dockers so this is becoming a larger and larger issue Is there anything at all i can post to help with this jonp? As i genuinely don't want to have to buy another set up memory to fix this problem when it seems to just want more and more Regards, Jamie Edit: Here are a few grabs from the system that may be useful virsh dommemstat "Jamie - 10586" actual 12582912 rss 18602588 virsh dommemstat "CCTV" actual 2097152 rss 1510876 virsh dommemstat "Cat - SeaBios" actual 8388608 rss 10982764 virsh dominfo "Jamie - 10586" Id: 4 Name: Jamie - 10586 UUID: 8db132a6-da10-1373-64e5-3f2515cb4b17 OS Type: hvm State: running CPU(s): 8 CPU time: 573928.3s Max memory: 12582912 KiB Used memory: 12582912 KiB Persistent: yes Autostart: disable Managed save: no Security model: none Security DOI: 0 virsh dominfo "CCTV" Id: 7 Name: CCTV UUID: 628a7030-7ffe-040f-e7dc-b828b045586c OS Type: hvm State: running CPU(s): 2 CPU time: 326103.0s Max memory: 2097152 KiB Used memory: 2097152 KiB Persistent: yes Autostart: disable Managed save: no Security model: none Security DOI: 0 virsh dominfo "Cat - SeaBios" Id: 2 Name: Cat - SeaBios UUID: 311cce28-0650-0bf3-70e8-0c25f504dcb6 OS Type: hvm State: running CPU(s): 4 CPU time: 225768.1s Max memory: 8388608 KiB Used memory: 8388608 KiB Persistent: yes Autostart: disable Managed save: no Security model: none Security DOI: 0 virsh nodememstats total : 32933444 KiB free : 322492 KiB buffers: 396 KiB cached : 278116 KiB top - 19:27:17 up 8 days, 6:03, 1 user, load average: 4.20, 4.37, 4.15 Tasks: 438 total, 1 running, 437 sleeping, 0 stopped, 0 zombie %Cpu(s): 31.9 us, 5.1 sy, 0.0 ni, 63.0 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st KiB Mem : 32933444 total, 338480 free, 31958276 used, 636688 buff/cache KiB Swap: 8388604 total, 5959020 free, 2429584 used. 396152 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31978 root 20 0 17.999g 0.017t 22492 S 312.0 56.5 9583:06 qemu-system-x86 8102 root 20 0 10.859g 0.010t 22380 S 190.7 33.4 3766:21 qemu-system-x86 20395 root 20 0 4132712 1.441g 14704 S 75.7 4.6 5436:46 qemu-system-x86 26511 server 20 0 2374728 256828 1072 S 1.3 0.8 0:12.69 mono-sgen 4845 root 20 0 762344 66540 18252 S 0.7 0.2 117:28.30 firefox 6708 root 20 0 819840 8088 1960 S 0.7 0.0 4:18.22 exe 20407 root 20 0 0 0 0 S 0.7 0.0 77:00.04 vhost-20395 21630 root 20 0 0 0 0 S 0.7 0.0 0:30.29 kworker/0:0 31990 root 20 0 0 0 0 S 0.7 0.0 75:50.58 vhost-31978 2547 root 20 0 9684 2504 2104 S 0.3 0.0 34:47.80 cpuload 4524 root 20 0 94476 3580 3000 S 0.3 0.0 15:39.78 emhttp 5094 root 20 0 0 0 0 S 0.3 0.0 2:12.81 xfsaild/nvme0n1 5124 root 20 0 1369872 13620 592 S 0.3 0.0 108:20.09 shfs 20411 root 20 0 0 0 0 S 0.3 0.0 5:05.00 kvm-pit/20395 20958 jamie 20 0 415400 6256 5240 S 0.3 0.0 12:10.89 smbd 1 root 20 0 4372 1476 1476 S 0.0 0.0 0:15.93 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.15 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:59.45 ksoftirqd/0 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H 7 root 20 0 0 0 0 S 0.0 0.0 3:54.68 rcu_preempt 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_sched 9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh 10 root rt 0 0 0 0 S 0.0 0.0 0:01.28 migration/0 11 root rt 0 0 0 0 S 0.0 0.0 0:01.33 migration/1 12 root 20 0 0 0 0 S 0.0 0.0 0:01.13 ksoftirqd/1 14 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H 15 root rt 0 0 0 0 S 0.0 0.0 0:03.13 migration/2 16 root 20 0 0 0 0 S 0.0 0.0 0:00.74 ksoftirqd/2 18 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H 19 root rt 0 0 0 0 S 0.0 0.0 0:02.90 migration/3 20 root 20 0 0 0 0 S 0.0 0.0 0:00.78 ksoftirqd/3 23 root rt 0 0 0 0 S 0.0 0.0 0:01.83 migration/4 24 root 20 0 0 0 0 S 0.0 0.0 0:00.63 ksoftirqd/4 26 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:0H Quote Link to comment
jonp Posted June 21, 2016 Share Posted June 21, 2016 bigjme, Have you tried recreating your tests with no plugins installed? Community Apps should be fine, but please remove all the extras you've added so we can narrow this down. Quote Link to comment
bigjme Posted June 21, 2016 Author Share Posted June 21, 2016 Hi Jonp, I have been able to remove all plugins but the following: SwapFile Community Applications If i remove the hotswap file plugin (which was installed due to this issue) my server will instantly start shutting off VMs to conserve memory as it has massively over allocated now - i do need the Libvirt Hotplug USB plugin back at some point but have been able to remove it now The server memory is still increasing but i am keeping an eye on it - i can restart my one vm with the hotswap plugin removed but it may not stay on for long Jamie Quote Link to comment
Squid Posted June 21, 2016 Share Posted June 21, 2016 Hi Jonp, I have been able to remove all plugins but the following:SwapFile Community Applications If i remove the hotswap file plugin (which was installed due to this issue) my server will instantly start shutting off VMs to conserve memory as it has massively over allocated now - i do need the Libvirt Hotplug USB plugin back at some point but have been able to remove it now The server memory is still increasing but i am keeping an eye on it Jamie While jonp has a very valid point about removing plugins to test for this kind of issue, I would suggest reinstalling fix common problems and putting it into troubleshooting mode, as additional things are logged every 10 minutes (available ram, free space on /, /var/log, tmpfs) ps -aux output, etc that may also help to shed some light on this. Quote Link to comment
bigjme Posted June 21, 2016 Author Share Posted June 21, 2016 While jonp has a very valid point about removing plugins to test for this kind of issue, I would suggest reinstalling fix common problems and putting it into troubleshooting mode, as additional things are logged every 10 minutes (available ram, free space on /, /var/log, tmpfs) ps -aux output, etc that may also help to shed some light on this. Jonp is it worth me putting this back on? I've got everything removed other than community apps now Just to put some things into perspective, here is what happened to a fresh reboot of my VM1 Allocated - 12GB Initial start - 12.5G VIRT, 12.3 RES After 4 Minutes - 14.6G VIRT, 14.3G RES In this time i booted up windows 10 and it has sat on the desktop doing nothing Jamie Quote Link to comment
jonp Posted June 21, 2016 Share Posted June 21, 2016 While jonp has a very valid point about removing plugins to test for this kind of issue, I would suggest reinstalling fix common problems and putting it into troubleshooting mode, as additional things are logged every 10 minutes (available ram, free space on /, /var/log, tmpfs) ps -aux output, etc that may also help to shed some light on this. Jonp is it worth me putting this back on? I've got everything removed other than community apps now Just to put some things into perspective, here is what happened to a fresh reboot of my VM1 Allocated - 12GB Initial start - 12.5G VIRT, 12.3 RES After 4 Minutes - 14.6G VIRT, 14.3G RES In this time i booted up windows 10 and it has sat on the desktop doing nothing Jamie I haven't actually used Squid's plugin myself before, but given what I know about it, it's probably ok because I don't think it actually adds any software packages to the system itself. That's my main concern with plugins. After removing all the plugins, did you reboot the host? Some plugins may not remove all their software correctly so rebooting the host would be recommended to ensure a clean state. Also, just want to confirm if you are still running 6.2-beta and what version. To put things in perspective, on one of my main systems, I have 16GB of physical RAM installed and I allocate 8GB to my primary Windows 10 VM with GPU, USB, and audio pass through. Even after running this VM almost all day today, my mem usage in htop for QEMU is showing 9028M reserved (that's the only one that matters). Quote Link to comment
Squid Posted June 21, 2016 Share Posted June 21, 2016 I haven't actually used Squid's plugin myself before, but given what I know about it, it's probably ok because I don't think it actually adds any software packages to the system itself. That's my main concern with plugins. FCP (and CA for that matter) have zero dependencies or extra packages installed that's not already included in the base OS. FCP in troubleshooting mode merely tails the syslog to the flash drive and adds extra diagnostic logging every 10 minutes to the syslog. Only downside is that if you leave it running in troubleshooting mode for a couple of months with out a reboot, your flash drive will fill with the logs / diagnostics. (which are easy to delete) But with it's logging of the free ram, and utilization of key RAM folders and the processes running, etc it is (IMO very useful to aid in diagnostics) Quote Link to comment
bigjme Posted June 21, 2016 Author Share Posted June 21, 2016 FCP (and CA for that matter) have zero dependencies or extra packages installed that's not already included in the base OS. FCP in troubleshooting mode merely tails the syslog to the flash drive and adds extra diagnostic logging every 10 minutes to the syslog. Only downside is that if you leave it running in troubleshooting mode for a couple of months with out a reboot, your flash drive will fill with the logs / diagnostics. (which are easy to delete) But with it's logging of the free ram, and utilization of key RAM folders and the processes running, etc it is (IMO very useful to aid in diagnostics) I didn't realize it stored so much to be honest I have the plugin installed again and running in troubleshoot mode as well. I took note of memory usage during the system reboot, here are my numbers I'm running beta 23 870M usage after reboot with no array running 1.55G usage after starting the array 25.5G usage 30 seconds after starting all 3 VMs 28.3G after a further 5 minutes 29G after a further 10 minutes VM1: 14.7G RES VM2: 9.7G RES VM3: 2771M RES The full system is reporting 29G of 31.4G in htop 29348852 used, 342288 free, 2967696 buff/cache I hope some of this helps Quote Link to comment
bigjme Posted June 22, 2016 Author Share Posted June 22, 2016 Having removed everything from my server even the powerdown plugin i received my first full system crash today - i have been 1 month free of any system crashes, so i was forced to do a hard power reset Looking through the logs folder this is the last log i found before the crash - the server didn't even make 24 hours so the log isn't huge so i was able to include it The last thing i can see is unraid killing things for memory and the system hard locking not long after If i can't last 24 hours without a system lock i'm going to have to put the swap file plugin back on the system Jamie archangel-diagnostics-20160622-1743.zip Quote Link to comment
bigjme Posted June 23, 2016 Author Share Posted June 23, 2016 Day 2 of system lockups, again the server didn;t last 24 hours I've had to put the swap file plugin back on and will see if the server stays up or not, i have attached the diagnostics for the last failure archangel-diagnostics-20160623-1756.zip Quote Link to comment
Squid Posted June 23, 2016 Share Posted June 23, 2016 Day 2 of system lockups, again the server didn;t last 24 hours I've had to put the swap file plugin back on and will see if the server stays up or not, i have attached the diagnostics for the last failure Not a VM expert, but from what I see at the logs is that while the memory used by the VM did increase, the last one prior to OOM was not fundamentally different from the previous ones (and certainly not any different from when memory started dropping precipitously). I'm just wondering if its a docker app doing this (something misconfigured and storing stuff into RAM?) Quote Link to comment
bigjme Posted June 23, 2016 Author Share Posted June 23, 2016 If i watch the docker usage they never really change Using community apps i can see the majority of my dockers are using only 60MB or 116MB memory, my highest docker is EMBY which almost always sites at 305MB All in they are reporting 1.83% memory usage I have the swap file running now so i can see what happens tomorrow - i have put fix common problems back on troubleshoot mode so i can post what happens then Edit: One thing that makes me think it isn;t dockers is that if i have my system on for s week and shut off all vms it sits at 2GB memory usage Quote Link to comment
bigjme Posted June 25, 2016 Author Share Posted June 25, 2016 So my system hasn't shut down or crashed since my last report (it is using 600M swap at the moment) My plan is to host a few of my log files online as they will be too large to post on the forum directly. Will this be of any use if i just post the links jonp? As i have debug mode on i get a diagnostic every 30 minutes so can post them on intervals if needed or just the latest and you can work backwards through the logs This should outline the change in VM memory and everything else over a number of days so you can see what changes? Regards, Jamie Quote Link to comment
bigjme Posted July 1, 2016 Author Share Posted July 1, 2016 Ok so i had waited but things are getting stupid now, my emby docker can not stream files or even serve artwork for videos because the server has no memory left Below is a link to my latest server diagnostics, it is too large to post here but shows an outline of the changes in my system over the last 8 days of running - so as an updated: Yes i am reliant on the swapfile plugin to keep the system running Yes i have had to reduce my main vm's memory allocation down to 8GB from 12GB just so the rest of my server is usable In total out of 32GB i can now only allocate 18GB to VMs before the system just swallows the rest of it and because a mess Diagnostic link: http://www.cudamining.co.uk/archangel-diagnostics-20160701-2210.zip i know i'm a pest but i could really do with this one being looked into jonp as if the server keeps chewing up my memory for nothing i'm going to have to start considering changing my OS - after having just got my vms working after months of issues Regards, Jamie Quote Link to comment
RobJ Posted July 2, 2016 Share Posted July 2, 2016 Set the tunable num_stripes back to default (1280), as that bug was fixed. (A small memory gain, but probably not enough to matter with your issues.) You are using memcached, which is great if it works, but my understanding is it needs to be thoroughly tested in your specific environment (unRAID with VM's), and I wasn't aware of anyone doing that yet. If someone has, then more power to you! Otherwise, I'd be leery. Syslogs are a little strange. Something has deleted all of the ones from when you booted on the 23rd until the 27th at 4:40am. Then something is creating last 200 line copies of each of the syslog files, which is redundant and unnecessary. And something is rotating the syslogs at 4:40 each morning, probably the process that's deleting the older ones, which we wanted. Below is a table of memory values for the 4 largest memory users, at various periods. I am not seeing any real growth here. It would have been nice to have seen the earlier memory figures! Unfortunately the Troubleshooting mode is exploding your syslogs too, so may be causing some issues on its own. It would be nice to have a configurable period for cases like this, with long running testing. It's currently every 10 minutes, would be better for you if it was say every 6 or 12 hours. Notable mem users (in GB) 6-27 4:40 6-29 4:40 7-1 4:40 7-1 22:00 - emby (Dc ?) .67 .84 .60 .06 - CCTV (VM 2) 1.51 1.39 1.30 1.55 - Cat (VM 10.85 10.85 10.88 10.88 - Jamie(VM 12) 18.68 18.71 18.85 18.92 Totals 22+ 31.71 31.79 31.63 31.41 Swap Used 2.19 2.34 3.12 2.34 You can see that emby got squeezed at the end, probably causing its issues, but I don't know why it got squeezed, as the total memory used from these users actually dropped. CCTV doesn't ever use its 2GB allocation. Cat is using 2.8GB more than its 8GB allocation, but never grows beyond that. Jamie wants 6.5 GB over its 12GB, with only tiny growth after that. Your big loss therefore appears to be the extra 6GB that Jamie wants from the start, and that's where I would concentrate troubleshooting, look for why it wants so much extra. Those memory used figures are all about 31.5 or greater, not leaving hardly any room for unRAID, the Docker engine, or system needs. So you are starting out at boot time already practically out of memory. From the above, swap seems irrelevant. In the last hour, load averages went over 16, so it's not surprising you couldn't do anything. Throughout the syslogs, they generally stayed above 2, which means lags were likely. You want them to be less than one (not that I'm an expert at this). Quote Link to comment
bigjme Posted July 2, 2016 Author Share Posted July 2, 2016 Hi RobJ Thanks for the reply. By menaced do you mean the swap file that is running? I'm not aware of anything else that may be causing something like that to run? To my understanding memcached is a php extension? My system boots at around 26GB memory usage with everything loaded, I've posted a few logs on the previous page where the system would crash in less than 24 hours due to memory issues, these should be small enough to show everything from the time of boot? If I disable the swap file plug in I am using the vms get so large that the system shuts things down shut to save memory and the system locks up. I am using around 2.4gb swap at all times with all physical memory used With regards to memory usage per vm I posted some figures about these on the previous page as well. For example if I shut down the vm Jamie and restart is it will using around 13gb at boot which is fine. Over the next day or so it will grow to us almost 19gb where it tends to stay at around that point I am using beta 23 in case something new is in this one for vms that I haven't seen yet Thanks again RobJ for your reply Jamie Quote Link to comment
Squid Posted July 2, 2016 Share Posted July 2, 2016 Additionally, troubleshooting mode was never really designed to run for weeks on end Sent from my LG-D852 using Tapatalk Quote Link to comment
RobJ Posted July 2, 2016 Share Posted July 2, 2016 Additionally, troubleshooting mode was never really designed to run for weeks on end I'm formally requesting a configuration choice, default 10 minutes, but with the option to select longer (and shorter?) values, such as 30 minutes, 1 hour, 6 hours, 24 hours. Quote Link to comment
bigjme Posted July 2, 2016 Author Share Posted July 2, 2016 I agree that may be useful So after posting last night i changed my vm Jamie to be allocated 8GB memory, the same as the VM Cat I've just checked htop for both machines and here are the memory usages VM1 Jamie: 12.9G RES Uptime: 16 hours VM2 Cat: 10.4G RES Uptime: 93 hours Both vms have 8gb allocated, both vms have a gpu passed through, both vms have the same install of windows running (same disc, same date installed, even down to updated and antivirus), both have a usb controller passed through (same make and model) The only difference in them are: VM1 is OVMF, and has 4 passed through USBs VM2 is Seabios, and has 2 passed through USBs Yet there is 2.5G usage difference between the 2 Jamie Quote Link to comment
RobJ Posted July 2, 2016 Share Posted July 2, 2016 Thanks for the reply. By menaced do you mean the swap file that is running? I'm not aware of anything else that may be causing something like that to run? To my understanding memcached is a php extension? No, memcached is a performance boosting tool that can be added to an app with a database. By its nature, it uses up memory, potentially a LOT of memory, in order to provide that boost. I believe it's best used when the database server is remote, caching objects locally in RAM that would otherwise be stored on drives elsewhere, and accessed over limited network bandwidth. So I'm not sure why it's used here when it's all on your server, except perhaps to cache in RAM what's on your drives. My only experience (and it's indirect) is with Perl and MySQL, client server. It's not a drop-in booster, it needs to be carefully hand-tooled into the application. My system boots at around 26GB memory usage with everything loaded, I've posted a few logs on the previous page where the system would crash in less than 24 hours due to memory issues, these should be small enough to show everything from the time of boot? With regards to memory usage per vm I posted some figures about these on the previous page as well. For example if I shut down the vm Jamie and restart is it will using around 13gb at boot which is fine. Over the next day or so it will grow to us almost 19gb where it tends to stay at around that point When I have time, I don't right now, I'll take a look, add additional figures to the table. What seems obvious though is that Jamie is your memory hog, demanding much more than it perhaps should. As I said, troubleshooting Jamie is probably the right focus. If you have a way to disable memcached usage, that might be an interesting test. Or switch to a different application provider that doesn't use memcached. Another option is to disable ballooning for Jamie, since Jamie seems to want to grab any memory it can. Another option is to drop Cat to 6GB and Jamie to 10GB, both of which seem already excessive to me (but I'm old-school). That would free 4GB. Let's see if Jamie still want to grab that 4GB for itself. It's possible that memcached is better used when it isn't sharing the available memory with other VM's. Perhaps it needs hard limits, which is why I'm suggesting trying without ballooning. I'm sure there are others here with more relevant experience though. Quote Link to comment
bigjme Posted July 2, 2016 Author Share Posted July 2, 2016 Thanks for the reply. By menaced do you mean the swap file that is running? I'm not aware of anything else that may be causing something like that to run? To my understanding memcached is a php extension? No, memcached is a performance boosting tool that can be added to an app with a database. By its nature, it uses up memory, potentially a LOT of memory, in order to provide that boost. I believe it's best used when the database server is remote, caching objects locally in RAM that would otherwise be stored on drives elsewhere, and accessed over limited network bandwidth. So I'm not sure why it's used here when it's all on your server, except perhaps to cache in RAM what's on your drives. My only experience (and it's indirect) is with Perl and MySQL, client server. It's not a drop-in booster, it needs to be carefully hand-tooled into the application. My system boots at around 26GB memory usage with everything loaded, I've posted a few logs on the previous page where the system would crash in less than 24 hours due to memory issues, these should be small enough to show everything from the time of boot? With regards to memory usage per vm I posted some figures about these on the previous page as well. For example if I shut down the vm Jamie and restart is it will using around 13gb at boot which is fine. Over the next day or so it will grow to us almost 19gb where it tends to stay at around that point When I have time, I don't right now, I'll take a look, add additional figures to the table. What seems obvious though is that Jamie is your memory hog, demanding much more than it perhaps should. As I said, troubleshooting Jamie is probably the right focus. If you have a way to disable memcached usage, that might be an interesting test. Or switch to a different application provider that doesn't use memcached. Another option is to disable ballooning for Jamie, since Jamie seems to want to grab any memory it can. Another option is to drop Cat to 6GB and Jamie to 10GB, both of which seem already excessive to me (but I'm old-school). That would free 4GB. Let's see if Jamie still want to grab that 4GB for itself. It's possible that memcached is better used when it isn't sharing the available memory with other VM's. Perhaps it needs hard limits, which is why I'm suggesting trying without ballooning. I'm sure there are others here with more relevant experience though. The vm Jamie at one point had 16GB allocated to it and the more it has allocated the more it tries to take from the system so i dropped it to 12GB and then down to 8 as you can see now. My vm for CCTV wasn't around when it was 16GB however I'm not very familiar with the vm terminology or how to change anything not in the web panel, how can i disable balooning and what affect will it have on the systems? The vm Jamie sits at 2.4GB memory usage almost all the time so 8GB is plenty unless i start doing large image/video editing (web developer by trade) So for my usual use i would use memcache to cache webpages served from Apache or Nginx mainly when the websites are dynamic, to my knowledge i haven't enabled it on my servers as i have no access to the php config but it seems that opcache was enabled as well (again nothing i enabled) I have altered my system to disable opcache on every page load since i can not control it directly so this will remove this from the loop - although this hasn't altered the memory usage at all i don't want it on anyway (good spot!) My system is right now reporting 26.8G Mem, and 1.33G Swap used and it doesn't appear to be taking more for that vm so it hasn't just taken everything it can get its haands on One that may be worth a try is to see if i increase the memory for vm Cat to 12gb and see if that has the same memory usage issue as vm Jamie? It may help to rule out of it is a possible OVMF issue? Thanks for your help again! Jamie Quote Link to comment
bigjme Posted July 16, 2016 Author Share Posted July 16, 2016 So here is an interesting one - i have upgraded to rc2 and my memory issue seems to have calmed down rather a lot So after upgrading i found one of my vm's (Jamie) would not boot due to some weird windows issue causing an update loop which kept failing, removing itself then applying again So i did a fresh re-install and put avg free edition on the vm - the vm instantly lost internet access and explorer stopped working the second avg installed - it was so bad i couldn't shutdown, reboot or uninstall avg Another windows 10 install later with no avg and everything seems to be working, this install has the newer virtio drivers and memory usage has dropped substantially VM Cat: 10.5G Res - virtio from pre 6.1.9 VM Jamie: 10.1G Res - latest virtio driver VM CCTV: 2483M Res - virtio from beta 23 Having the 3 vms running, my dockers and everything else i have been able to remove the swapfile plugin and my system is down to 24.8G memory usage So i don't know what resolved this or helps to lower the usage but in total the following has happened since my memory usage was so high Unraid updates New windows install New virtio drivers While 2.1G overhead is a little over what i want i am very happy with it being this high and hopefully things will start to go down even more as time goes by So all in i have now got a drop of 2.8G memory usage on just that 1 vm since reporting this issue Jamie 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.