Hello!

 

I'm not entirely sure what happened, not why it happened, I'm hoping for some clues.

Today I woke up to half of my containers dead and unable to restart because `/run` was full.

It was full of PID files like this:

/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/995bd40dfae5a3fcf5c000315d14cbd13b1951c5241cd43fffdeb1ce3614f1a3/65b8849422136e83b69bf164146333fab04d3328444a6f760ab07ebf9473ae35.pid

 

There were 120 MB (!) of such files, each 4KB in disk-size, most of them pointing to nonexistent processes.

 

Currently I do not have any data relating to whether it was a one-time surge or a steady ongoing problem. I'll be monitoring the situation.

I am running portainer and most of the container management I'm doing via that.

 

Could someone point me where to look to understand and permanently fix the issue?

 

// For anyone stumbling at the same problem

Temporary solution was to increase the size of that tmpfs: 

mount -t tmpfs tmpfs /run -o remount,size=512M

You can start your containers now.

 

Then remove all of the stale files with a script like this:

#!/bin/bash

pid_files=$(find /run/docker -name '*.pid')
for file in $pid_files
do
  pid=$(< "$file")
  if [[ -n "$pid" ]]; then
    if ps --pid $pid > /dev/null; then
        echo -n ""
    else
      rm "$file"
    fi
  fi
done

 

Might want to look at syslog / check you don't have a misconfigured container that keeps restarting in a loop.

I haven't seen anything like that in the log. I do have quite a bit of churn in the containers as I'm running sever CI build runners on the server, which basically means that there are a lot of short-lived containers getting created.

 

At any rate, I would imagine that stale PID files should be getting cleaned up.

enbyted, thanks for script! 

 

Faced same issue on Ubuntu 20 04 (quite fresh docker version) only path to pid files little different (/run/containerd/io.containerd.runtime.v2.task/moby/{CONTAINER_ID})

 

In my case problem is in health check timeout. Please check your container with CONTAINER_ID. Maybe helps for you.

I have report my issue under https://github.com/moby/moby/issues/48908. With reproduce steps.

Let see what developers say.

 

Again thanks for script! 🚀