enbyted

I haven't seen anything like that in the log. I do have quite a bit of churn in the containers as I'm running sever CI build runners on the server, which basically means that there are a lot of short-lived containers getting created. At any rate, I would imagine that stale PID files should be getting cleaned up.

Hello! I'm not entirely sure what happened, not why it happened, I'm hoping for some clues. Today I woke up to half of my containers dead and unable to restart because `/run` was full. It was full of PID files like this: /run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/995bd40dfae5a3fcf5c000315d14cbd13b1951c5241cd43fffdeb1ce3614f1a3/65b8849422136e83b69bf164146333fab04d3328444a6f760ab07ebf9473ae35.pid There were 120 MB (!) of such files, each 4KB in disk-size, most of them pointing to nonexistent processes. Currently I do not have any data relating to whether it was a one-time surge or a steady ongoing problem. I'll be monitoring the situation. I am running portainer and most of the container management I'm doing via that. Could someone point me where to look to understand and permanently fix the issue? // For anyone stumbling at the same problem Temporary solution was to increase the size of that tmpfs: mount -t tmpfs tmpfs /run -o remount,size=512M You can start your containers now. Then remove all of the stale files with a script like this: #!/bin/bash pid_files=$(find /run/docker -name '*.pid') for file in $pid_files do pid=$(< "$file") if [[ -n "$pid" ]]; then if ps --pid $pid > /dev/null; then echo -n "" else rm "$file" fi fi done