Odd behaviour VM / docker (memory leak gallore)

[Caennanu]

Good day all,

 

So, since i started with unraid, i've always had the thought in mind to host my own VM's, so that i could run several applications in them and ofcourse learn from the different operating systems. However, i been running into an issue when it comes to atleast linux based systems.

 

For home security purposes, i'm running Shinobi. I currently have 9 camera's connected to it and the software itsself is great as it fulfills all my needs. However, in the docker i had the issue that it would not release memory. So ofcourse, i contact the developers for support. They advised me to switch to a linux. So we install a VM with Ubuntu 20.04LTS. The installation works great, the software works great, but lo and behold. The issue is there again. Once again it will not release memory eventually causing a crash in the VM (or rather, out of memory errors in the VM and the docker crashes and will restart).

 

Since ofcourse we want to exclude anything from the possibilities, we switch over to CentOS 8. The install once again works great. But here too, the exact same issue is happening. It won't release memory and will eventually crash, eventually being usually within 4 days. The testing parameters will be down below.

Now we know for sure that Docker, Ubuntu 20.04LTS and CentOS8 will all eventually crash, or atleast cause the software to stop working as intended. Can't be an unraid issue right? As most posts related to the topic always say its somewhere on the VM level and its the user or hardware fault. Right!

 

So, let's put that to the test. Bye bye unraid (for now) and lets make this a baremetal linux box. We install ubuntu 20.04LTS on the same system. Install Shinobi. And add the camera's, all fine. the exact same bare minimal configuration as all the test cases. No recording, no motion detection no nothing. Just adding camera's and displaying them on the webui dashboard.

 

And what do you geuss? instead of locking up due to out of memory issues, even tho the baremetal box has 128gb of ecc memory available and the VM's only had 32gb asigned to them. The software only uses 1gb memory over the course of 5 days, which compared to out of memory issues on 32gb over the course of 4 days in the other tests, seems like a breeze.

 

So i would love to know, what is happening with the hypervisor in unraid that causes these linux VM's AND docker to not release memory?

 

Testing parameters:

Spoiler

See signature for hardware specs.

 

CentOS VM: 4 cores 4 threads pinned / isolated, 32gb memory assigned.

Machine: Q35-5.1

Bios: OVMF

 

Ubuntu 20.04 LTS: 4 cores 4 threads pinned / isolated, 32gb memory assigned.

Machine: Q35-5.1

Bios: SeaBios

Vdisk location: Array, share + prefer:cache

 

Docker: 6 cores 6 threads assigned, not isolated. limited to 32gb memory.

img location: Array, share + prefer:cache 

 

caennanu-diagnostics-20211020-0806.zip

Edited October 20, 2021 by Caennanu
added diagnostics, editted titel

[Thirs]

I am able to reproduce the memory leak issue with an Unraid VM running Shinobi - 6 camera system, Nvidia 1050Ti pass-through to the VM. In this case the camera is used for the TensorFlow object detection which is supported by Shinobi.
Tested with Unraid v.6.8.3 and v.6.92

Ubuntu 20.04 LTS: 3 cores 3 threads pinned / isolated, 8gb memory assigned.

Machine: Q35-4.2

Bios: OVMF

Vdisk location: Cache;only
Unassigned drive pass-through to the VM for recording from the cameras.

It seems I can also reproduce the memory leak issue with a Docker container running a different NVR software. In this case the software is: doitandbedone/ispyagentdvr. In 24 hours the memory consumed by this Docker container has doubled. The test has run for a limited time so I will update this thread as time progresses. Update: After a few days the memory consumed may have stabilized. I'll continue to log the memory consumed and update this thread.
 

Date | Container | Memory consumed | Memory limit assigned to container

10/18 8am doitandbedone/ispyagentdvr 342.1MiB / 8GiB

10/19 8am doitandbedone/ispyagentdvr 705.1MiB / 8GiB

10/20 8am doitandbedone/ispyagentdvr 762.9iB / 8GiB

10/21 8am doitandbedone/ispyagentdvr 743.3MiB / 8GiB
 

Tested with Unraid v.6.92

Docker: 3 cores 3 threads assigned, not isolated. limited to 8gb memory.

img location: Cache;only
Unassigned drive pass-through to the VM for recording from the cameras.

peon-diagnostics-20211019-0806.zip

Edited October 24, 2021 by Thirs

[Caennanu]

Nobody to reply to this issue? Nobody else experiencing the same?