"No space left on device error"

[serhan]

Hi,

 

I am having a rather large error with the system and not sure how to resolve or diagnose.

 

Essentially, containers are not updating or starting. As far as I can tell the docker size is not full.

 

The error is:

 

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: write /var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/6ce813a60cb0812e3e3aa00be23201b3639efb1602feb30f70082e169ea53c06/.init.pid: no space left on device: unknown.

 

And then I get a 'Server Error' when I try to launch any containers.

 

I have attached the docker size information and my du command.

 

I am really quite stumped what is causing this and how to solve it. Other threads say that I may have filled the OS storage in error but I am not sure how to check this.

 

I would be very grateful for a steer!

[GermGerm]

I am also having the same issue. I just updated to 6.12.1.

My cache drive has plenty of room. I can't create new docker containers.

The Error I receive when installing the docker.

docker: Error response from daemon: failed to start shim: symlink /var/lib/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/188f4efc137e795b0291fc2f1f613e04f0bcb7de1933f3da68f6bf8840c45c63 /var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/188f4efc137e795b0291fc2f1f613e04f0bcb7de1933f3da68f6bf8840c45c63/work: no space left on device: unknown.

In the unRaid logs it looks like the docker network is blocking a port for some reason.

Jun 23 10:41:57 unRaid-Plex kernel: docker0: port 6(veth9051fe0) entered blocking state
Jun 23 10:41:57 unRaid-Plex kernel: docker0: port 6(veth9051fe0) entered disabled state
Jun 23 10:41:57 unRaid-Plex kernel: device veth9051fe0 entered promiscuous mode
Jun 23 10:41:57 unRaid-Plex kernel: docker0: port 6(veth9051fe0) entered disabled state
Jun 23 10:41:57 unRaid-Plex kernel: device veth9051fe0 left promiscuous mode
Jun 23 10:41:57 unRaid-Plex kernel: docker0: port 6(veth9051fe0) entered disabled state

 

[ps0111]

I had this issue just now but I don't have a cache and I ran a backup of the Flash drive and then reboot the Unraid server and once it came back up I didnt have the issue and I was able to add new docker containers and start all of my containers without issue. I will keep and eye on my server to see if it pops back up.

If any one knows what the issue really is then I would like to know so that I can implement a proper fix.

[dlchamp]

Sorry I'm not commenting with any solution, but I'm also having a similar issue that has appeared to come from nowhere.

Can't access "console" or exec into any  running container.   Also, my GUS container that just updated will not start.
Been on 6.11.5 for a bit now. Just over 25 days without issues.  Today, an issue and nothing has changed.  Server has been online since the reboot for the update. 
 
docker.img shows 50% used on "Main". 
`OCI runtime exec failed: exec failed: unable to start container process: open /dev/ptmx: no space left on device: unknown`

[Max]

7 hours ago, dlchamp said:

Sorry I'm not commenting with any solution, but I'm also having a similar issue that has appeared to come from nowhere.

Can't access "console" or exec into any  running container.   Also, my GUS container that just updated will not start.
Been on 6.11.5 for a bit now. Just over 25 days without issues.  Today, an issue and nothing has changed.  Server has been online since the reboot for the update. 
 
docker.img shows 50% used on "Main". 
`OCI runtime exec failed: exec failed: unable to start container process: open /dev/ptmx: no space left on device: unknown`

for me it showed today only, and for me atleast it's like suddenly i have a limit of running only n number of dockers at a time. As i noticed that reduced the number of dockers that currently, the dockers that earlier the same error here OP mentioned, starts to run back normally.

And docker exec too starts running when i reduce number of dockers that are running.

[dlchamp]

1 hour ago, Max said:

for me it showed today only, and for me atleast it's like suddenly i have a limit of running only n number of dockers at a time. As i noticed that reduced the number of dockers that currently, the dockers that earlier the same error here OP mentioned, starts to run back normally.

And docker exec too starts running when i reduce number of dockers that are running.

 
The number of running containers hasn't changed for me.  
After I commented last night I increased the max PTY that could be allocated (from 4096 to 8192)

sysctl -w kernel.pty.max=8192

This worked for my specific issue, but I don't deem that a proper fix since in reality unused PTY should be closed properly.   Hopefully someone smarter can explain why this is happening, but at least there are a couple temporary solutions.

I also restarted this morning after my usual backup routine was finished which hopefully just cleared up whatever funky was happening under the hood. 

 

Now to figure out why Grafana-Unraid-Stack is crashing no matter what I've done.

Edited June 24, 2023 by dlchamp

[GermGerm]

18 hours ago, ps0111 said:

I had this issue just now but I don't have a cache and I ran a backup of the Flash drive and then reboot the Unraid server and once it came back up I didnt have the issue and I was able to add new docker containers and start all of my containers without issue. I will keep and eye on my server to see if it pops back up.

If any one knows what the issue really is then I would like to know so that I can implement a proper fix.

I was able to reboot now and am also able to create new dockers. Strange issue.

[nibbler215]

I'm getting the same thing since upgrading to 6.12.1

[SkilledAlpaca]

Upgraded to 6.12.1 and I have the same issue as others here.

 

docker run
  -d
  --name='freshrss'
  --net='bridge'
  -e TZ="America/Los_Angeles"
  -e HOST_OS="Unraid"
  -e HOST_HOSTNAME="Tower"
  -e HOST_CONTAINERNAME="freshrss"
  -e 'PUID'='99'
  -e 'PGID'='100'
  -e 'UMASK'='022'
  -l net.unraid.docker.managed=dockerman
  -l net.unraid.docker.webui='http://[IP]:[PORT:80]'
  -l net.unraid.docker.icon='https://raw.githubusercontent.com/linuxserver/docker-templates/master/linuxserver.io/img/freshrss-logo.png'
  -p '8096:80/tcp'
  -v '/mnt/user/appdata/freshrss':'/config':'rw' 'lscr.io/linuxserver/freshrss'

e1f1a6b17997a665637d3ea6732f5792b650383329722f5fd5881c1cb492f2c3
docker: Error response from daemon: failed to start shim: symlink /var/lib/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e1f1a6b17997a665637d3ea6732f5792b650383329722f5fd5881c1cb492f2c3 /var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e1f1a6b17997a665637d3ea6732f5792b650383329722f5fd5881c1cb492f2c3/work: no space left on device: unknown.

The command failed.

 

[SkilledAlpaca]

Rebooting Unraid and everything that was having issues started working again. Updated a container and it worked no issue.

[fxppc]

Okay, we clearly need someone from Lime in this thread, preferably with an official  workaround and a quick ETA on when the permanent fix is due. I'm in the same predicament. Before the .12 release, I went weeks and months in between reboots. Now I have to do it every couple of days. Waited until the x.x.1 bug fix before updating, but apparently that's giving Lime Inc too much trust!

[Margucci]

I was just able to shut down the docker service, turn it back on, then everything started up like normal. 

 

I encountered my issue after updating 3 containers. 2 updated and started fine. one did not. Also on .12.1

Edited June 28, 2023 by Margucci

[OuticNZ]

Also getting the same error - after a reboot docker containers work fine, but after 3-4 days if stopped I can't start them again.   

 

I stopped all docker containers and then stopped the docker service - restarted it and the containers can start again.  

Edited June 29, 2023 by OuticNZ

[FestiveKyle]

This sounds like it's the same problem as below - using the Nvidia runtime ends up filling the /run directory. There's a few solutions in that post - I've added '--no-healthcheck' as an extra parameter to the container using the Nvidia runtime and haven't had the issue show up again. 

 

You can check if the mount is full by running: "df -h". If it is full, you can check the problem container with "du /run". The section at the end of the path (after "moby/") is the container id, to find the name of the container run "docker ps -a --no-trunc | grep <CONTAINER ID>".

 

Hope this helps!

 

Edited July 4, 2023 by FestiveKyle

[SkilledAlpaca]

Implemented Option 1 from the thread Kyle linked above. I was able to start my docker containers after running it manually without any need to disable and renable the Docker service. At least there is a temporary workaround now.

[Koldo]

Option 1 did the trick for me too!

[Vilaran]

I am experiencing this issue too. Every 3-4 days, I can no longer start any docker containers. I get the following "no space on device" kind of error for each container.
 

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: mkdir /sys/fs/cgroup/docker/92f1ef37cbe933b33651c450bfb566b2243b365d4bae76b791f7c5f990f5ec67: no space left on device: unknown.

 

Unlike the posters above me, I am not running the nvidia container and my /run directory is not being filled. I tried the suggestions posted there anyways with no success.

 

I can't find any directory or drive that is being filled. df -h yields nothing concerning.

 

 

A reboot fixes this, but this is extremely annoying.

[itimpi]

@Vilaran You should post your system's diagnostics zip file in your next post in this thread to get more informed feedback when you next encounter this issue.  It is always a good idea to post this if your question might involve us seeing how you have things set up or to look at recent logs.