[6.12.10] Cannot fork: Resource temporarily unavailable

I ran the exact same setup in a fresh Proxmox Debian KVM. The Debian Docker implementation is not vulnerable to Docker 'fork bombs'. The PIDs are successfully limited to 7000, and the only effect is the container status changing to unhealthy. This is achieved with no additional configuration whatsoever.

This confirms that this is not a problem with Docker itself, but rather with Unraid's implementation of Docker.

Thanks, I've already reported this issue to LT to make sure they are aware, and to see if they can do something to prevent this.

On 6/25/2024 at 2:45 AM, Hendrik112 said:

I ran the exact same setup in a fresh Proxmox Debian KVM. The Debian Docker implementation is not vulnerable to Docker 'fork bombs'. The PIDs are successfully limited to 7000, and the only effect is the container status changing to unhealthy. This is achieved with no additional configuration whatsoever.

This confirms that this is not a problem with Docker itself, but rather with Unraid's implementation of Docker.

 

Thank you for tracking down this issue.  Here is a way to limit the number of processes any container is permitted to create:

 

From a terminal session, edit the file /boot/config/docker.cfg and add this line to the end:

 

DOCKER_OPTS="--default-ulimit nproc=256:1024"

 

You must then disable Docker and then re-enable for settings to take effect.

 

The string 256:1024 sets the soft and hard limits for the number of processes the container can create.  This setting will apply to all containers.  There is a way to set these limits per-container but that would require a code change.  Of course you can set different values.

 

Please let me know if this solves the 'fork bomb' issue.

1 hour ago, limetech said:

 

Thank you for tracking down this issue.  Here is a way to limit the number of processes any container is permitted to create:

 

From a terminal session, edit the file /boot/config/docker.cfg and add this line to the end:

 

DOCKER_OPTS="--default-ulimit nproc=256:1024"

 

You must then disable Docker and then re-enable for settings to take effect.

 

The string 256:1024 sets the soft and hard limits for the number of processes the container can create.  This setting will apply to all containers.  There is a way to set these limits per-container but that would require a code change.  Of course you can set different values.

 

Please let me know if this solves the 'fork bomb' issue.

Hi, thank you for looking into this.

I set the DOCKER_OPTS according to your instructions and quickly noticed that 256:1024 doesn't seem sufficient for my 30 docker containers. High CPU usage and multiple containers reported that they were running out of memory and failed to start.
After increasing nproc to 1024:2048 all containers started successfully.

The slow 'fork bomb' was however not stopped by this setting, using over 7500 PIDs when I stopped it to not crash my system again.

I then used compose down to remove all containers and lowered nproc to its initial 256:1024 limit. After restarting docker I only created the authelia stack, and it again used over 7500 PIDs. The docker setting that can contain it is setting pids_limit: 50 in a compose file. This seems to be a very hard limit compared to the one set by debian causing the container to crash.

In case you want to reproduce this issue yourself, I created a detailed bug report on the authelia github which should contain all the necessary steps.

Edited June 28 by Hendrik112

After changing the DOCKER_OPTS line did you go to Settings/Docker settings and disable docker, hit Apply, then enable docker and hit Apply?
You can open console for a container and type:

ulimit -a

and see if the limits were applied.  Look for 'max_user_processes' in the output.

 

Also: "256:1024" is just a guess - I don't know what is ultimately going to be the best setting.  In a future release we'll probably make this a per-container config setting.

Yes, my workflow is the following:
1. Settings->Docker->Enable Docker: No->Apply
2. Set docker.cfg
3. Settings->Docker->Enable Docker: Yes->Apply

Output of ulimit -a inside the authelia docker container:
 

core file size (blocks)         (-c) 0
data seg size (kb)              (-d) unlimited
scheduling priority             (-e) 0
file size (blocks)              (-f) unlimited
pending signals                 (-i) 514764
max locked memory (kb)          (-l) unlimited
max memory size (kb)            (-m) unlimited
open files                      (-n) 40960
POSIX message queues (bytes)    (-q) 819200
real-time priority              (-r) 0
stack size (kb)                 (-s) unlimited
cpu time (seconds)              (-t) unlimited
max user processes              (-u) 256
virtual memory (kb)             (-v) unlimited
file locks                      (-x) unlimited

Output of ps inside the authelia docker container:
 

PID   USER     TIME  COMMAND
    1 99        0:00 authelia
   99 root      0:00 [ssl_client]
  110 root      0:00 [ssl_client]
  118 root      0:00 [ssl_client]
  126 root      0:00 [ssl_client]
  134 root      0:00 [ssl_client]
  142 root      0:00 [ssl_client]
...
 1397 root      0:00 [ssl_client]
 1398 root      0:00 {healthcheck.sh} /bin/sh /app/healthcheck.sh
 1404 root      0:00 wget --quiet --no-check-certificate --tries=1 --spider https://localhost:9091/api/health
 1405 root      0:00 ssl_client -s3 -n localhost -I
 1406 root      0:00 ps

Output of ps -e | wc -l inside the authelia docker container:
 

1177

 

Edit: The setting seems to be applied correctly, but it doesn't limit the amount of processes at all. I have the nvidia-driver plugin installed if that maybe makes a difference.

Edited June 28 by Hendrik112

Ok forget about that DOCKER_OPTS string, you can remove it.

 

Instead, edit the authelia container, select Advanced View (upper right toggle) and include this string in the 'Extra Parameters' setting:

 

--pids-limit 1000

 

You can set any value besied 1000 of course.  The total number of processes inside that container should now be limited.

That did it. I guess --pids-limit is just the docker run equivalent to the pids_limit I already tried in docker compose.

I added --pids-limit 1000 and --health-interval 0.01s to speed up the process to the 'Extra Parameters' setting. The processes were successfully limited at 1000 and the container became unhealthy as intended.
This solution requires pid limiting for each container, which could be automated by the gui. Sadly, that won't apply to people who use docker compose via the compose plugin or portainer.

Edited June 28 by Hendrik112