I've searched around but can't find any information on correcting the docker daemon to limit it's log file size. I recently got a FCP warning, the /var/log/docker.log was 128MB filling up the entire log fs. I deleted the extra content, but dockerd was holding onto a reference so the filesystem didn't reduce used space. I gave up and restarted and started tailing /var/log/docker.log. Here's the spam lines, searching around I don't see much mention of this information.

 

Will post diagnostics if needed but this seems to be a particular issue with the docker daemon setup?

 

I thought about configuring /etc/docker/daemon.json but I realized it would be wiped upon restart so I am posting to see if anyone else has any workarounds for this.

 

I also was getting various messages about the cgroups as well. 

 

Edited April 17, 20224 yr by gvkhna
solved

16 minutes ago, gvkhna said:

limit it's log file size

Disable Docker in Settings, Enable Docker LOG Rotation, set Docker LOG maximum file size, Enable Docker

1 minute ago, trurl said:

Disable Docker in Settings, Enable Docker LOG Rotation, set Docker LOG maximum file size, Enable Docker

 

It looks like I do have that on with these settings. Will change Docker LOG maximum file size to 10MB and try that as well now.

Based on this post and my settings, it looks like indeed these Docker settings are for container logs but not the daemon/engine. 

 

I've observed this, because the /var/log/docker.1 was 100mb rotated but I set rotation at 50mb. So these settings probably don't apply here. 

 

I also can't find a single piece of information about just stopping it from logging this error, and sure enough I restarted and it has started spamming this log file again with this error about collecting stats.

 

1 hour ago, gvkhna said:

post diagnostics

 

Here you go. Thank you kindly for your help

 

 

 

Edited April 15, 20224 yr by gvkhna
going to double check no keys are leaked on second thought

Maybe unrelated but why do you have 50G docker.img? Have you had problems filling it?

Well I know the recommendation is 30, I'm currently at 17gb and do plan on adding more things so I made it 50G just in case, I do have a lot of things I still need to setup on this system.

I ended up making the following changes which I guess will solve the problem. First time making changes to my flash drive config. If you see any potential issues please let me know.

 

/boot/config/go

#!/bin/bash
# Start the Management Utility
cp /boot/etc/cron.d/logrotate /etc/cron.d/logrotate
chmod 755 /etc/cron.d/logrotate
cp /boot/etc/logrotate.d/docker /etc/logrotate.d/docker
chmod 755 /etc/logrotate.d/docker
/usr/local/sbin/emhttp &

 

/boot/etc/logrotate.d/docker

/var/log/docker.log {
  rotate 0
  notifempty
  missingok
  size=5M
  nocompress
}

 

/boot/etc/cron.d/logrotate

*/15 * * * *  logrotate /etc/logrotate.conf

 

Obviously I'm brute-forcing this approach with a 15m log rotate interval but I'm ok with that if it prevents the /var/log from getting filled up. I'm also getting spam for this "cgroups: cannot find cgroup mount destination" 

 

Will see if this works out anyway.

 

Edited April 15, 20224 yr by gvkhna

Unfortunately this did not solve the issue. A few days later I'm still getting the /var/log is filled error. It turns out docker is holding onto the references for the docker.log.1 etc. 

 

I can see this with the following command:

 

lsof +L1

 

The output showing:

COMMAND       PID     USER   FD   TYPE DEVICE  SIZE/OFF NLINK  NODE NAME
dockerd      6701     root    1w   REG   0,35 131702784     0    55 /var/log/docker.log.1 (deleted)
dockerd      6701     root    2w   REG   0,35 131702784     0    55 /var/log/docker.log.1 (deleted)
container    6731     root    1w   REG   0,35 131702784     0    55 /var/log/docker.log.1 (deleted)
container    6731     root    2w   REG   0,35 131702784     0    55 /var/log/docker.log.1 (deleted)
lsof: no pwd entry for UID 5050

 

So it looks like dockerd would indeed have to be restarted. Anyone know what to do here, maybe how to delete these log files without restarting the containers?

Pretty sure I solved it with the following change to this config (read earlier post)

 

/boot/etc/logrotate.d/docker

/var/log/docker.log {
  rotate 0
  notifempty
  missingok
  copytruncate
  nocreate
  size=5M
  nocompress
}

Needed nocreate and copytruncate in the logrotate file and I now see it the logfile filling up but then logrotate truncates it in place. Looks like it's working well enough.

 

I´m having the same problem, "no metrics received" spamed 380mb docker.log. And my Docker Container Metrics ("Advanced view" in Docker Tab) don´t show CPU/RAM Usage of my containers. Did you encounter this, too? did your fix work?

Edited May 15, 20224 yr by Tranceport