Jump to content
Sign in to follow this  
gfjardim

Limiting Docker resources

12 posts in this topic Last Reply

Recommended Posts

This is something I've been looking for sometime, but a post from grumpybutfun made me pull the plunge.

 

For a long term running containers system, it's healthy that containers run within limits, and for now we can easily set by command line two metrics, memory limit and CPU weight.

 

For memory, the operator is -m <NUM><UNIT>, eg. -m="512m" for 512MB of RAM.

 

CPU limiting is more tricking. 100% of cpu usage is equivalent to 1024 shares, so 10% is -c=102 in command line.

 

I'm ready to make some changes in the Extended Config Plugin, but I need some feedback about how to implement this. Select boxes should do the trick, but I think we shouldn't expose anything above 90% of CPU shares in the webGUI. This will obviously limit some performance in Plex transcoding, PAR2 verification in SABnzbd etc.., but I think it's for a greater good, the integrity of your unRAID system.

 

So, what do you think?

Share this post


Link to post

You have to also interface and set up some customized cgroups in Slackware. You will probably want to make those persistent too.

 

You can do this all on your own but it would be best if Lime Technologies was also in the loop. That way both of you can best decide how to proceed (in respect to them handling the Slackware side of things and you Docker side).

 

For people running something like Plex, newznab and when SABnzb is unraring a huge file... They are going to want / need more horsepower than your typical docker app (but not take all the horsepower). But... You do not want a normal Docker App that crashes / hangs to chew up 100% of the CPU, memory, Disk I/O either.

 

Find some power users who will beta test various settings you come up with and see what works best.

Share this post


Link to post

You have to also interface and set up some customized cgroups in Slackware. You will probably want to make those persistent too.

 

You can do this all on your own but it would be best if Lime Technologies was also in the loop. That way both of you can best decide how to proceed (in respect to them handling the Slackware side of things and you Docker side).

 

For people running something like Plex, newznab and when SABnzb is unraring a huge file... They are going to want / need more horsepower than your typical docker app (but not take all the horsepower). But... You do not want a normal Docker App that crashes / hangs to chew up 100% of the CPU, memory, Disk I/O either.

 

Find some power users who will beta test various settings you come up with and see what works best.

 

I'm not concerned much about disk I/O and network, but memory and CPU limits is a must have.

 

Docker automatically assigns a different cgroup for each container, they can be seen here: /sys/fs/cgroup/memory/docker, /sys/fs/cgroup/cpu/docker etc...

 

I don't know if we can set a global limit to all containers, each one having their own limits. Maybe this part must be implemented by LT.

 

I don't have any experience in cgroups by the way.

Share this post


Link to post

I'm not concerned much about disk I/O and network, but memory and CPU limits is a must have.

 

Docker automatically assigns a different cgroup for each container, they can be seen here: /sys/fs/cgroup/memory/docker, /sys/fs/cgroup/cpu/docker etc...

 

I don't know if we can set a global limit to all containers, each one having their own limits. Maybe this part must be implemented by LT.

 

A 10 Minute crash course in cgroups:

 

PaaS Under the Hood, Episode 2: cgroups

 

and...

 

Arch Wiki - cgroups

Share this post


Link to post

I'm not concerned much about disk I/O and network, but memory and CPU limits is a must have.

 

Docker automatically assigns a different cgroup for each container, they can be seen here: /sys/fs/cgroup/memory/docker, /sys/fs/cgroup/cpu/docker etc...

 

I don't know if we can set a global limit to all containers, each one having their own limits. Maybe this part must be implemented by LT.

 

A 10 Minute crash course in cgroups:

 

PaaS Under the Hood, Episode 2: cgroups

 

and...

 

Arch Wiki - cgroups

 

So, if I create a cgroup prior to docker and then add docker on start to it, all child containers should share the same cgroup limits? That would br nice.

Share this post


Link to post

This link have useful info too: http://oakbytes.wordpress.com/2012/09/02/cgroup-cpu-allocation-cpu-shares-examples/

 

If I understood it correctly, to get this to work for CPU sharing, all core unRAID services (smbd, nfsd, shfs etc..) must be assigned to one group, and docker to another. CPU sharing do not limit the cpu usage, but only its weight in case of a race for resources. It's e much nicer way to do what NICE do. It's the modern way to avoid resources hogging.

 

Is that right, grumpy?

Share this post


Link to post

In laymans terms, what's this addressing ?  Is it just giving resource priority to certain process ?  so you can have core unraid at a higher priority than containers ?

 

On a slightly related note, is there a way to trigger a pause/stop of all containers and the restart ?  The reason I ask is that I think it would be faster to run a parity check if all add ons were stopped so I'm toying with a script that will stop/pause the containers, run a parity check and restart them.

 

Would this be possible ?  Would it make a difference ?

Share this post


Link to post

Is that right, grumpy?

 

As you see, Docker works the way it is now. However, I suspect that in time with certain apps on certain hardware or certain apps crashing... it can / will be a problem that users can experience.

 

If I was Lime Technologies, I would take matters into my own hands and make sure I allocate a bare minimum for CPU, Disk I/O, Memory, etc. for the Server and if they so desire... Allow users to create groups of their own. Similar to some us who went nuts with NFS tweaking and even unRAID tunables settings.

 

I was making you and needo aware that you should take a look at it (along with Lime Technologies).  Since you and needo are running the show on Docker... I didn't want this potential issue that could crop up to reflect on you or your work.

 

I'm really biting my lip hard and trying not to be Grumpy...

 

This is really something that Lime Technology should know about or if they already do but haven't shared it... Chime in and let us know what they think / want to do, if anything.

Share this post


Link to post

Before I went to the effort or at least potentially slowed docker down in some scenarios I would put something in place to help identify if the problem is real and how wide spread they are for users.

 

Dont get me wrong having these options sounds like a very sane idea but I would be concerned if the default limits impacted a larger group than potentially the problem solves.

Share this post


Link to post

well i can tell you the problem is there

it was there when we had the normal plugins in unraid and it is still there with docker

but with docker i don't have these nasty crashes any more (at least till now)

 

biggest offenders are

 

rsync eating up all i/o cycles

have crashplan then running a scan

and plex also doing a transcode then cpu gets a serious workout and swapfile is also getting bigger and bigger

 

but like i said so far no crashes ... not even a docker that quits...

i think unraid itself is getting the resources it needs but in my case i think it is the cache drive that is killing itself....

as swapfile / dockers / download are all going to the cache drive ........

so the extra app/docker drive might be a good choice to add to the standard unraid setup

 

thing is that choosing what should have how much resources is a difficult thing to choose i guess....

i would not know where to even begin at that ...

so if you make this option available then i would at least provide some insight how people should choose wisely the settings for each app

 

Share this post


Link to post

well i can tell you the problem is there

it was there when we had the normal plugins in unraid and it is still there with docker

but with docker i don't have these nasty crashes any more (at least till now)

 

biggest offenders are

 

rsync eating up all i/o cycles

have crashplan then running a scan

and plex also doing a transcode then cpu gets a serious workout and swapfile is also getting bigger and bigger

 

but like i said so far no crashes ... not even a docker that quits...

i think unraid itself is getting the resources it needs but in my case i think it is the cache drive that is killing itself....

as swapfile / dockers / download are all going to the cache drive ........

so the extra app/docker drive might be a good choice to add to the standard unraid setup

 

thing is that choosing what should have how much resources is a difficult thing to choose i guess....

i would not know where to even begin at that ...

so if you make this option available then i would at least provide some insight how people should choose wisely the settings for each app

 

I'm not talking in limit each container individually anymore. I'm interested if we can make two control groups, one for core and one for apps, in the way apps couldn't exhaust every cpu cycle or every ram byte. I'm talking about avoid "freemem" crash dumps, etc....

Share this post


Link to post

Is there any update on this?

 

I was running a couple of Arch VMs until a corrupted USB drive caused kernel panics and I decided to jump from Beta5 to 10a.

 

Thanks to DockerManager (gfjardim you rock!) and the little Docker knowledge I picked up from the forums I've switched to Docker. However, the server isn't stable anymore.

 

I had a feeling that Plex and Sick both rebuilding their libraries could be hogging CPU cycles and crashing unraid. So I let Plex run, then once it had finished launched the Sick docker. I left it running overnight but this morning the server was unresponsive.

 

As an experiment I've launched the Sick docker with -c 512 and -m 2048m. Will this limit CPU usage to "half of 1 cpu" or "half of all cpus"?

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this