containered.toml -- log-level error [htop] - After a few days constant extra 10% CPU usage


Recommended Posts

After a few days of running, I can see in the stats window, that my cpu usage is 15%, compared to the normal 4-5% constantly (if nothing else is running, on a daily, weekly stats chart)

 

If I restart docker, it doesn't solve it (In settings, docker enable No, than back to yes)

If I restart the server, I think it solves it sometimes, for a few days, than it starts again.

 

I think it started around the time, when I changed some docker log settings or after i switch my cache to encrypted xfs

It starts to bother me now, cos if this writes to SSD so much log constantly, that it takes up 10% CPU, it will just kill my SSD in a few months (Or var/run/ is all in RAM?)

 

my docker.img is less than 50%, log is 1% in the dashboard.

 

If I run a htop, I get the following results:

image.png.411234707f96e72a3c97f3850c08669a.thumb.png.2c2b2060c1617536b777f4fb7573e9e6.png

My cache drive is now XFS encrypted for a few months, i though it would solve this occasional btrfs csum error. I was wrong, as the docker img is still btrfs...

Any ideas?

Thank you!

Link to comment
  • 2 weeks later...
On 5/26/2020 at 4:08 AM, LSL1337 said:

This is a paid software, right?

Yes, but by and large the community of volunteers is who does the support

 

And without diagnostics being posted, no one is going to be able to help.  But, your first course of diagnosis is to start shutting down your containers one at a time and then see when that disappears.  Personally though, I don't particularly worry about CPU % being utilized.  If the CPU has something to do, then it has something to do.

 

And the stats window isn't particularly the best place to guage what the server is actually doing because both it (and the dashboard) do consume CPU cycles.  

 

Your load average over the last 15 minutes of the screen shot is 0.68 which is basically nothing on an 8 core system.

 

Now, if your writes to the SSD are increasing exponentially, then that is a different story.

Link to comment
1 hour ago, Squid said:

Yes, but by and large the community of volunteers is who does the support

 

And without diagnostics being posted, no one is going to be able to help.  But, your first course of diagnosis is to start shutting down your containers one at a time and then see when that disappears.  Personally though, I don't particularly worry about CPU % being utilized.  If the CPU has something to do, then it has something to do.

 

And the stats window isn't particularly the best place to guage what the server is actually doing because both it (and the dashboard) do consume CPU cycles.  

 

Your load average over the last 15 minutes of the screen shot is 0.68 which is basically nothing on an 8 core system.

 

Now, if your writes to the SSD are increasing exponentially, then that is a different story.

first of all, thanks for the reply.

second: if the baseline would be 15%, i wouldn't care about it, but it is less than 5% (has been for years, without plex transcoding), so the system is doing something, which it shouldn't/ didn't before.

based on htop, it might have something to do with containered.toml, which could indicate possible  drive usage. this is a 1 line txt file. if it has 10% cpu usage, that would mean something is very wrong...

 

i stopped my containers one by one, and the cpu usage decreased proportionally, which means it's not a single docker which causes the issue, it could be with the docker engine itself.

 

Diagnostics attached

lsl-nas-diagnostics-20200527-1300.zip

Link to comment
  • 4 weeks later...
  • 2 months later...
  • 2 weeks later...

are you f'in kidding me? @ich777

it works

why does it have any impact even, and why is it ok for a few days after boot?

is this a bug, or a feature?

and it has impact, when the ui is not even open

 

I can't even find the words...

how many people does this impact btw?

 

@INTEL check the previous post

Edited by LSL1337
Link to comment
Just now, LSL1337 said:

are you f'in kidding me? @ich777

it works

I know ;)

 

1 minute ago, LSL1337 said:

why does it have any impact even, and why is it ok for a few days after boot?

is this a bug, or a feature?

Both, I will report that directly to @limetech it's more kind of a bug but not sure if it's easily fixable... :P

 

1 minute ago, LSL1337 said:

and it has impact, when the ui is not even open

Yep, I won't get to technical. :D

 

I know what it is and I will report back about my findings. ;)

 

Link to comment
  • 4 months later...
On 10/13/2020 at 2:00 PM, ich777 said:

@LSL1337 & @INTEL

Try to disable 'Advanced View' on the docker page and report back, I bet it will be the solution. ;)

 

Found a few of these listed when running an htop:
 

containerd --config /var/run/docker/container/containerd.toml --log-level error unraid

 

Not sure if it was of any concern but it did cost about 4% of my CPU. Switching to basic view immediately remedied this. Thanks for that.

 

But is that "error" process an issue?

 

 

 

Link to comment
Just now, adminmat said:

But is that "error" process an issue?

That is not an error... :)

 

These are just the commands that the process was started with.

This process starts if you turn on the Advanced View (to show the CPU und RAM usage from your containers) on the Docker page and won't go away if you switch back to the Basic View (even if you logout or close the WebGUI).

Link to comment
6 minutes ago, ich777 said:

This process starts if you turn on the Advanced View (to show the CPU und RAM usage from your containers) on the Docker page and won't go away if you switch back to the Basic View (even if you logout or close the WebGUI).

 

ok. So those process are still running but just not consuming as much resources when showing the basic view. So I guess nothing to be concerned about. I probably check my logs for errors a little too much. This was actually the first time I've run an htop... when I was troubleshooting the wsdd network today issue that had one CPU core at 100%.

Edited by adminmat
Link to comment
3 minutes ago, adminmat said:

So those process are still running but just not consuming as much resources when showing the basic view.

No, in basic view the process isn't running only if you turn on advanced view the process is started and ended if you turn on basic view again, that's because I keep the advanced view turned off... ;)

Link to comment
  • 6 months later...

What is the reason for this process? After starting the docker service without starting a container it's already present:

image.thumb.png.6089159e787304d07fce36e33e820b8d.png

 

This means it is something related to docker service itself. I found this documentation. It mentions the log-driver, which should be "json-file" as it is the default if no /etc/docker/daemon.js exists. So it would write logs to a json file, but is there really a log file for the docker service itself? EDIT: Yes there is a daemon log file under /var/log/docker.log, but this file is tiny and its last entries are several minutes old as well.

 

And it looks like the process does not even access the log file:

# ps -aef | grep containerd
root      7815  7791  0 10:40 ?        00:00:03 containerd --config /var/run/docker/containerd/containerd.toml --log-level error
root     29548 11758  0 10:55 pts/1    00:00:00 grep containerd

 

Or is this something which can't be further influenced?

Link to comment
36 minutes ago, mgutt said:

What is the reason for this process? After starting the docker service without starting a container it's already present

Because this process is the backend for docker, it manages the pulls, networking, storage,... it also supervises the running containers and this is one reason why it's creating more load when you turn on the "Advanced View".

 

The "docker" command is only the command line tool, if you issue a command with for example "docker run WHATEVER" it will actually invoke "containerd" to pull the image itself if they are not available locally and assign it to the specified network, hand it over to runc to actually run the container.

If you install Docker on a Linux machine, or even Windows machine, it will also install containerd with docker and runc.

 

It's not only the log driver it's more than that... see the creation/start process from a container like this:

 

docker -> containerd -> runc -> actually run the container

Link to comment
  • 5 months later...

Interesting. I've never turned on advanced view under docker before, but I'm still getting a huge number of these commands running at random points on my server. Is there anything else that can be causing it? Of course I don't really mind since the usage isn't over 5-10%, but I'd like to figure out what the actual issue is!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.