giusemr Posted September 29, 2019 Share Posted September 29, 2019 Hello everybody, I already read multiple topics here in the forums and over on reddit about this error, but no solution for me. Since yesterday evening I noticed my cpu utilization went up to 100% because the influxdb container could not write any data. After checking the other containers I noticed similar errors in the logs. I also checked if any container writes directly to the image, like a downloader or something but the containers and volumes sizes looks fine. Down below you find some outputs from my system and my diagnostics output. I wonder if the devices /dev/loop0 and /dev/loop1 cause these issues because they are at 100% utilization. I also increased the docker image size to 80G. Previously it was on 40G. Currently I stopped all containers. df -h Filesystem Size Used Avail Use% Mounted on rootfs 5.8G 959M 4.9G 17% / tmpfs 32M 232K 32M 1% /run devtmpfs 5.8G 0 5.8G 0% /dev tmpfs 5.9G 0 5.9G 0% /dev/shm cgroup_root 8.0M 0 8.0M 0% /sys/fs/cgroup tmpfs 128M 476K 128M 1% /var/log /dev/sda1 15G 323M 15G 3% /boot /dev/loop0 12M 12M 0 100% /lib/modules /dev/loop1 12M 12M 0 100% /lib/firmware /dev/md2 5.5T 4.5T 983G 83% /mnt/disk2 /dev/md3 932G 415G 517G 45% /mnt/disk3 /dev/sde1 224G 52G 171G 24% /mnt/cache shfs 6.4T 5.0T 1.5T 78% /mnt/user0 shfs 6.6T 5.0T 1.7T 76% /mnt/user /dev/loop2 80G 13G 67G 16% /var/lib/docker /dev/loop3 1.0G 17M 905M 2% /etc/libvirt shm 64M 0 64M 0% /var/lib/docker/containers/e5c05fd2bc13be950a7ddc7cc47dfd7025fa0c857534982c437639239c2e45ce/mounts/shm shm 64M 0 64M 0% /var/lib/docker/containers/27d86e87953d222b315ef03bb583941df32f3904d360d3ee8169dce5cd431638/mounts/shm shm 64M 0 64M 0% /var/lib/docker/containers/72bfc085261798d287e7e55d3f3f2d45791c687cfc562f916f8b5d6be0d18cd0/mounts/shm shm 64M 0 64M 0% /var/lib/docker/containers/b3e9c142f3cafbd16e4de3b68898cf496fe0b88b1d20c888b46c2ec85d51dd26/mounts/shm shm 64M 0 64M 0% /var/lib/docker/containers/717477a8dc3f79e64532727d1e054ce20eb7e9e19d293dd7d176082d52262f5d/mounts/shm shm 64M 0 64M 0% /var/lib/docker/containers/c27fe5a357e6d77fe59d5d89eb84c82625d4188cf8047f7be70e2fea61a5785c/mounts/shm shm 64M 0 64M 0% /var/lib/docker/containers/928f5b9ff7ee9cbeead0bf207640ed5213c97bb542fc44eedcee9c7b539e6077/mounts/shm docker system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 28 28 8.434GB 87.06MB (1%) Containers 32 7 350.7MB 350.7MB (99%) Local Volumes 5 1 49.65MB 49.65MB (100%) Build Cache 0 0 0B 0B Docker Volume Info: btrfs filesystem show: Label: none uuid: b5a7631a-65ec-4867-a43d-19009f3496a2 Total devices 1 FS bytes used 11.62GiB devid 1 size 80.00GiB used 20.07GiB path /dev/loop2 I hope someone can help me with this. Thanks in advance. server-diagnostics-20190929-0857.zip Quote Link to comment
Squid Posted September 29, 2019 Share Posted September 29, 2019 loop0 & loop1 are always 100% full (think of them as CD images -> since they're static there's no way to add any files to those, hence 100%) There's nothing obvious though in the logs. What's a sampling of the error messages in the docker logs? Quote Link to comment
giusemr Posted September 29, 2019 Author Share Posted September 29, 2019 The error appears everytime a container tries to create or write a file. For example my plex container from plexinc throws following error: unable to create '/usr/lib/plexmediaserver/Resources/Tuner/Shared/scanners/dvbc/transponders/5244.ini.dpkg-new' (while processing './usr/lib/plexmediaserver/Resources/Tuner/Shared/scanners/dvbc/transponders/5244.ini'): No space left on device [Errno 28] No space left on device: '/etc/letsencrypt/.certbot.lock' This morning, at the moment when I wrote this post, the problem was there. After leaving the system untouched until now, everything works as expected after a test. From this morning until now, nothing has changed in the system. Even in the logs where no entries that would indicate any change. Now I started all containers and the problem seems resolved. I don't know what could cause this problem. But thank you @Squid for looking into this. Quote Link to comment
Squid Posted September 29, 2019 Share Posted September 29, 2019 when that happens again, go to the Docker Tab and hit Container Size and post the screen shot Quote Link to comment
Fearmonger Posted February 21, 2023 Share Posted February 21, 2023 sysctl -w kernel.pty.max=8192 fixed the issue for me. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.