Unable to open Docker console. exec failed: unable to start container process: open /dev/ptmx: no space left on device: unknown

[capino]

After testing a self build container, I'm no longer able to open the console of any of the running dockers.
I already removed the self build container and all unused volumes and images.
When trying to connect to the console of a docker, I get the following error:

OCI runtime exec failed: exec failed: unable to start container process: open /dev/ptmx: no space left on device: unknown

I also have tried to stop an start docker, but this did not help.
All running dockers seem to be running without problems.

Does anybody have any idea what could have created this issue and how to resolve this?

P.S.

These are the values of docker container sizes:
Total size                                        29.3 GB                  7.05 GB                  2.19 GB

[raun]

I'm seeing the same thing when opening the console window to any running docker container.  A reboot fixes it a few hours, but at some point this happens again.  Same symptoms as you - stopping/restarting doesn't help.  My docker.img is not full. 

[trurl]

attach diagnostics to your NEXT post in this thread

[capino]

After doing loads of google searches, I finally found the correct syntax and a workaround.
I seamed like the pts sessions are not correctly closed or unallocated.

the amount of pseudo terminals is so high
bij running: 

ls /dev/pts | wc -l

I see there are 3432.

As a workaround, I upped the amount by running.

 

sysctl -w kernel.pty.max=8192

I'm not sure if this is because of something I did while creating my self build container, or this is due to an old kernel bug (https://lkml.org/lkml/2009/11/5/370)

 

Edited December 21, 2022 by capino

[raun]

Same thing on my box:

 

cat /proc/sys/kernel/pty/max  == 4096

cat /proc/sys/kernel/pty/nr == 3518

cat /proc/sys/kernel/pty/reserve == 1024

 

I'm not creating my own containers.

[capino]

I did some additional testing and it looks like, when a docker container console session is opened in a browser, multiple Pty's are allocated, but only one gets unallocated on close.
When I open Unraid Pty console session only 1 is allocated. But sometimes the browser windows is not closed on running exit (a new Pty is opened). Running exit in this second (new) session keeps both allocated.

 

Using ssh through a remote terminal seems (for now with some quick tests) not to keep Pty's allocated.

Edited December 21, 2022 by capino
sometimes more than 2 Pty's are allocated

[raun]

After 3 days after a reboot my open ptys are holding steady at 3524 (give-or-take).  After hitting this point I dont see ptys not being closed anymore.

[capino]

I have not rebooted yet, and I'm up to 4876 pays at the moment

[raun]

I'm slightly creeping up after 2 weeks.

 

root@Tower:~# !45
cat /proc/sys/kernel/pty/nr
3714
root@Tower:~# uptime
 21:28:59 up 17 days, 10:55,  0 users,  load average: 0.35, 0.29, 0.27
root@Tower:~#

[dakotatokad]

Bump.

 

Was running into this same problem which was preventing me from opening up a console for a few docker containers i needed to work in. I had somewhere around 37xx pts. Upping it to 8192 allowed me to access the consoles.

 

This is way beyond my knowledge base - anyone know what causes this or what can reliably allow us to release some of those pts "reservations?"

[MrDecepticon]

I am also running into this issue where consoles will no longer open for any docker.   I can't even SSH into it: Server refused to allocate pty. Rebooting has allowed me to open consoles and logs again. 

Edited February 3, 2023 by MrDecepticon

[Fearmonger]

 

[capino]

As mentioned in the Bug Report 

this problem seems to be fixed in 6.12.2 

[Naonak]

I just started getting this problem and can't seem to work around it.

I'm running 6.11.5

I've tried running sysctl -w kernel.pty.max=8192 and my pty.max is set to 8192, but I still can't open a docker console.

My ls /dev/pts | wc -l has 3497, so it's not at the max ptys ... I'm not sure what to do at this point?

I've started and stopped the docker (and all the docker images of course) ... they all start and stop just fine and I can disable docker and re-enabled it just fine... just can't seem to get into the console.

Edited August 12, 2023 by Naonak