serhan Posted June 23, 2023 Share Posted June 23, 2023 Hi, I am having a rather large error with the system and not sure how to resolve or diagnose. Essentially, containers are not updating or starting. As far as I can tell the docker size is not full. The error is: docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: write /var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/6ce813a60cb0812e3e3aa00be23201b3639efb1602feb30f70082e169ea53c06/.init.pid: no space left on device: unknown. And then I get a 'Server Error' when I try to launch any containers. I have attached the docker size information and my du command. I am really quite stumped what is causing this and how to solve it. Other threads say that I may have filled the OS storage in error but I am not sure how to check this. I would be very grateful for a steer! Quote Link to comment
GermGerm Posted June 23, 2023 Share Posted June 23, 2023 I am also having the same issue. I just updated to 6.12.1. My cache drive has plenty of room. I can't create new docker containers. The Error I receive when installing the docker. docker: Error response from daemon: failed to start shim: symlink /var/lib/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/188f4efc137e795b0291fc2f1f613e04f0bcb7de1933f3da68f6bf8840c45c63 /var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/188f4efc137e795b0291fc2f1f613e04f0bcb7de1933f3da68f6bf8840c45c63/work: no space left on device: unknown. In the unRaid logs it looks like the docker network is blocking a port for some reason. Jun 23 10:41:57 unRaid-Plex kernel: docker0: port 6(veth9051fe0) entered blocking state Jun 23 10:41:57 unRaid-Plex kernel: docker0: port 6(veth9051fe0) entered disabled state Jun 23 10:41:57 unRaid-Plex kernel: device veth9051fe0 entered promiscuous mode Jun 23 10:41:57 unRaid-Plex kernel: docker0: port 6(veth9051fe0) entered disabled state Jun 23 10:41:57 unRaid-Plex kernel: device veth9051fe0 left promiscuous mode Jun 23 10:41:57 unRaid-Plex kernel: docker0: port 6(veth9051fe0) entered disabled state Quote Link to comment
ps0111 Posted June 23, 2023 Share Posted June 23, 2023 I had this issue just now but I don't have a cache and I ran a backup of the Flash drive and then reboot the Unraid server and once it came back up I didnt have the issue and I was able to add new docker containers and start all of my containers without issue. I will keep and eye on my server to see if it pops back up. If any one knows what the issue really is then I would like to know so that I can implement a proper fix. Quote Link to comment
dlchamp Posted June 24, 2023 Share Posted June 24, 2023 Sorry I'm not commenting with any solution, but I'm also having a similar issue that has appeared to come from nowhere. Can't access "console" or exec into any running container. Also, my GUS container that just updated will not start. Been on 6.11.5 for a bit now. Just over 25 days without issues. Today, an issue and nothing has changed. Server has been online since the reboot for the update. docker.img shows 50% used on "Main". `OCI runtime exec failed: exec failed: unable to start container process: open /dev/ptmx: no space left on device: unknown` Quote Link to comment
Max Posted June 24, 2023 Share Posted June 24, 2023 7 hours ago, dlchamp said: Sorry I'm not commenting with any solution, but I'm also having a similar issue that has appeared to come from nowhere. Can't access "console" or exec into any running container. Also, my GUS container that just updated will not start. Been on 6.11.5 for a bit now. Just over 25 days without issues. Today, an issue and nothing has changed. Server has been online since the reboot for the update. docker.img shows 50% used on "Main". `OCI runtime exec failed: exec failed: unable to start container process: open /dev/ptmx: no space left on device: unknown` for me it showed today only, and for me atleast it's like suddenly i have a limit of running only n number of dockers at a time. As i noticed that reduced the number of dockers that currently, the dockers that earlier the same error here OP mentioned, starts to run back normally. And docker exec too starts running when i reduce number of dockers that are running. Quote Link to comment
dlchamp Posted June 24, 2023 Share Posted June 24, 2023 (edited) 1 hour ago, Max said: for me it showed today only, and for me atleast it's like suddenly i have a limit of running only n number of dockers at a time. As i noticed that reduced the number of dockers that currently, the dockers that earlier the same error here OP mentioned, starts to run back normally. And docker exec too starts running when i reduce number of dockers that are running. The number of running containers hasn't changed for me. After I commented last night I increased the max PTY that could be allocated (from 4096 to 8192) sysctl -w kernel.pty.max=8192 This worked for my specific issue, but I don't deem that a proper fix since in reality unused PTY should be closed properly. Hopefully someone smarter can explain why this is happening, but at least there are a couple temporary solutions. I also restarted this morning after my usual backup routine was finished which hopefully just cleared up whatever funky was happening under the hood. Now to figure out why Grafana-Unraid-Stack is crashing no matter what I've done. Edited June 24, 2023 by dlchamp Quote Link to comment
GermGerm Posted June 24, 2023 Share Posted June 24, 2023 18 hours ago, ps0111 said: I had this issue just now but I don't have a cache and I ran a backup of the Flash drive and then reboot the Unraid server and once it came back up I didnt have the issue and I was able to add new docker containers and start all of my containers without issue. I will keep and eye on my server to see if it pops back up. If any one knows what the issue really is then I would like to know so that I can implement a proper fix. I was able to reboot now and am also able to create new dockers. Strange issue. Quote Link to comment
nibbler215 Posted June 25, 2023 Share Posted June 25, 2023 I'm getting the same thing since upgrading to 6.12.1 Quote Link to comment
SkilledAlpaca Posted June 26, 2023 Share Posted June 26, 2023 Upgraded to 6.12.1 and I have the same issue as others here. docker run -d --name='freshrss' --net='bridge' -e TZ="America/Los_Angeles" -e HOST_OS="Unraid" -e HOST_HOSTNAME="Tower" -e HOST_CONTAINERNAME="freshrss" -e 'PUID'='99' -e 'PGID'='100' -e 'UMASK'='022' -l net.unraid.docker.managed=dockerman -l net.unraid.docker.webui='http://[IP]:[PORT:80]' -l net.unraid.docker.icon='https://raw.githubusercontent.com/linuxserver/docker-templates/master/linuxserver.io/img/freshrss-logo.png' -p '8096:80/tcp' -v '/mnt/user/appdata/freshrss':'/config':'rw' 'lscr.io/linuxserver/freshrss' e1f1a6b17997a665637d3ea6732f5792b650383329722f5fd5881c1cb492f2c3 docker: Error response from daemon: failed to start shim: symlink /var/lib/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e1f1a6b17997a665637d3ea6732f5792b650383329722f5fd5881c1cb492f2c3 /var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e1f1a6b17997a665637d3ea6732f5792b650383329722f5fd5881c1cb492f2c3/work: no space left on device: unknown. The command failed. Quote Link to comment
SkilledAlpaca Posted June 28, 2023 Share Posted June 28, 2023 Rebooting Unraid and everything that was having issues started working again. Updated a container and it worked no issue. Quote Link to comment
fxppc Posted June 28, 2023 Share Posted June 28, 2023 Okay, we clearly need someone from Lime in this thread, preferably with an official workaround and a quick ETA on when the permanent fix is due. I'm in the same predicament. Before the .12 release, I went weeks and months in between reboots. Now I have to do it every couple of days. Waited until the x.x.1 bug fix before updating, but apparently that's giving Lime Inc too much trust! Quote Link to comment
Margucci Posted June 28, 2023 Share Posted June 28, 2023 (edited) I was just able to shut down the docker service, turn it back on, then everything started up like normal. I encountered my issue after updating 3 containers. 2 updated and started fine. one did not. Also on .12.1 Edited June 28, 2023 by Margucci Quote Link to comment
OuticNZ Posted June 29, 2023 Share Posted June 29, 2023 (edited) Also getting the same error - after a reboot docker containers work fine, but after 3-4 days if stopped I can't start them again. I stopped all docker containers and then stopped the docker service - restarted it and the containers can start again. Edited June 29, 2023 by OuticNZ Quote Link to comment
FestiveKyle Posted July 4, 2023 Share Posted July 4, 2023 (edited) This sounds like it's the same problem as below - using the Nvidia runtime ends up filling the /run directory. There's a few solutions in that post - I've added '--no-healthcheck' as an extra parameter to the container using the Nvidia runtime and haven't had the issue show up again. You can check if the mount is full by running: "df -h". If it is full, you can check the problem container with "du /run". The section at the end of the path (after "moby/") is the container id, to find the name of the container run "docker ps -a --no-trunc | grep <CONTAINER ID>". Hope this helps! Edited July 4, 2023 by FestiveKyle 3 Quote Link to comment
SkilledAlpaca Posted July 7, 2023 Share Posted July 7, 2023 Implemented Option 1 from the thread Kyle linked above. I was able to start my docker containers after running it manually without any need to disable and renable the Docker service. At least there is a temporary workaround now. Quote Link to comment
Koldo Posted September 14, 2023 Share Posted September 14, 2023 Option 1 did the trick for me too! Quote Link to comment
Vilaran Posted November 12, 2023 Share Posted November 12, 2023 I am experiencing this issue too. Every 3-4 days, I can no longer start any docker containers. I get the following "no space on device" kind of error for each container. docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: mkdir /sys/fs/cgroup/docker/92f1ef37cbe933b33651c450bfb566b2243b365d4bae76b791f7c5f990f5ec67: no space left on device: unknown. Unlike the posters above me, I am not running the nvidia container and my /run directory is not being filled. I tried the suggestions posted there anyways with no success. I can't find any directory or drive that is being filled. df -h yields nothing concerning. A reboot fixes this, but this is extremely annoying. 1 Quote Link to comment
itimpi Posted November 12, 2023 Share Posted November 12, 2023 @Vilaran You should post your system's diagnostics zip file in your next post in this thread to get more informed feedback when you next encounter this issue. It is always a good idea to post this if your question might involve us seeing how you have things set up or to look at recent logs. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.