[6.8.3] shfs memory leak?

Does it happen with no dockers or VMs?

Also, why do you have 100G allocated to docker image? That usually means a user has their docker applications misconfigured. 20G will usually be more than enough, and if it grows beyond that you need figure out what you have done wrong.

 

An application will write into the docker image if it is writing to a path that isn't mapped to Unraid storage. This often happens from using the wrong upper/lower case, or from using a relative instead of an absolute path.

 

On the other hand, a container will write into memory if it is mapped to a host path that isn't actual storage. Note that an unmounted Unassigned Device isn't actual storage either.

Hi @trurl, it does not seem to happen when I turn off the docker containers that use the unRAID shares.

 

The 100G docker image was just because I could. Only around 3G is in use. Prior to testing for this issue, all docker container paths were mounted to unRAID shares. The docker image was probably still on the unassigned device.

 

I can lower the size of the docker container image and move it to the unRAID shares again tomorrow, to see if that helps.

I recreated the docker image with a lower size and moved it and the mounted paths to unRAID shares.

The shfs memory usage still grows, although possibly a bit slower, when backing up to the mino docker container.

When stopping the backup, the memory usage drops quite a bit now, which I don't remember happening previously.

 

I monitored the size used by the docker containers, and there was no growth there.

 

The high shfs memory usage produced yesterday, still remained today, even after shutting down the docker and vm services.

Anyone have any other suggestions?

The problem is still here. I am able to reproduce the issue when validating backups as well, which presumably doesn't write to the server, only read.

Edited April 3, 2020 by Mikael

There's another shfs memory related report, but in that case it evens happens on a clean install without any docker/VMs running, can you confirm in your case it only happens with dockers enable?

 

Strange that only two users have this issue, especially if it's not even reproducible the same way.

Hi @johnnie.black, it only happens with dockers running.

As far as I can tell only with the minio docker. That's the only one that is able to produce over 40 GB of memory usage for shfs in a couple hours.

Did you ever get a resolution to this? I am having the same issue suddenly. Wake up to 98% memory utilization out of 128G of memory on my system. Unraid 6.8.3

Hi @springsman

I didn't find a solution to this problem. I ended up not backing up one of my computers, since that one was responsible for the most rapidly increasing memory usage. Probably because the backup size was quite large.

By any chance were you using ARQ Agent for your backup software?

3 hours ago, springsman said:

By any chance were you using ARQ Agent for your backup software?

I am indeed using Arq. It's the windows based agent that seems to cause the highest memory consumption. It is also the largest backup, so the Windows part may not be a significant factor.

So it looks like it may be an ARQ issue somehow. I also had a large backup from my MacPro to my Unraid server using ARQ and Minio, and after the last update to Minio this is when the memory leak started. I deleted the entire backup folder on the Unraid and started a new backup using ARQ, and so far it seems to have worked. Will know more once the entire backup is complete.

Looks like deleting and starting a new backup on Unraid has fixed the memory leak issue. Not sure why ARQ running on my MacPro would cause Minio on my Unraid to utilize 98% of the 128GB of memory available. Hope this helps anyone else seeing this issue.