Hello everybody, 

 

I already read multiple topics here in the forums and over on reddit about this error, but no solution for me. Since yesterday evening I noticed my cpu utilization went up to 100% because the influxdb container could not write any data. After checking the other containers I noticed similar errors in the logs. 

I also checked if any container writes directly to the image, like a downloader or something but the containers and volumes sizes looks fine. Down below you find some outputs from my system and my diagnostics output.

 

I wonder if the devices /dev/loop0 and /dev/loop1 cause these issues because they are at 100% utilization.

 

I also increased the docker image size to 80G. Previously it was on 40G.

 

Currently I stopped all containers. 

 

df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          5.8G  959M  4.9G  17% /
tmpfs            32M  232K   32M   1% /run
devtmpfs        5.8G     0  5.8G   0% /dev
tmpfs           5.9G     0  5.9G   0% /dev/shm
cgroup_root     8.0M     0  8.0M   0% /sys/fs/cgroup
tmpfs           128M  476K  128M   1% /var/log
/dev/sda1        15G  323M   15G   3% /boot
/dev/loop0       12M   12M     0 100% /lib/modules
/dev/loop1       12M   12M     0 100% /lib/firmware
/dev/md2        5.5T  4.5T  983G  83% /mnt/disk2
/dev/md3        932G  415G  517G  45% /mnt/disk3
/dev/sde1       224G   52G  171G  24% /mnt/cache
shfs            6.4T  5.0T  1.5T  78% /mnt/user0
shfs            6.6T  5.0T  1.7T  76% /mnt/user
/dev/loop2       80G   13G   67G  16% /var/lib/docker
/dev/loop3      1.0G   17M  905M   2% /etc/libvirt
shm              64M     0   64M   0% /var/lib/docker/containers/e5c05fd2bc13be950a7ddc7cc47dfd7025fa0c857534982c437639239c2e45ce/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/27d86e87953d222b315ef03bb583941df32f3904d360d3ee8169dce5cd431638/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/72bfc085261798d287e7e55d3f3f2d45791c687cfc562f916f8b5d6be0d18cd0/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/b3e9c142f3cafbd16e4de3b68898cf496fe0b88b1d20c888b46c2ec85d51dd26/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/717477a8dc3f79e64532727d1e054ce20eb7e9e19d293dd7d176082d52262f5d/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/c27fe5a357e6d77fe59d5d89eb84c82625d4188cf8047f7be70e2fea61a5785c/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/928f5b9ff7ee9cbeead0bf207640ed5213c97bb542fc44eedcee9c7b539e6077/mounts/shm

docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              28                  28                  8.434GB             87.06MB (1%)
Containers          32                  7                   350.7MB             350.7MB (99%)
Local Volumes       5                   1                   49.65MB             49.65MB (100%)
Build Cache         0                   0                   0B                  0B

Docker Volume Info: 

btrfs filesystem show:

Label: none  uuid: b5a7631a-65ec-4867-a43d-19009f3496a2
Total devices 1 FS bytes used 11.62GiB
devid    1 size 80.00GiB used 20.07GiB path /dev/loop2

 

I hope someone can help me with this. 

 

Thanks in advance. 

server-diagnostics-20190929-0857.zip

loop0 & loop1 are always 100% full (think of them as CD images -> since they're static there's no way to add any files to those, hence 100%)

 

There's nothing obvious though in the logs.  What's a sampling of the error messages in the docker logs?

The error appears everytime a container tries to create or write a file. For example my plex container from plexinc throws following error:

unable to create '/usr/lib/plexmediaserver/Resources/Tuner/Shared/scanners/dvbc/transponders/5244.ini.dpkg-new' (while processing './usr/lib/plexmediaserver/Resources/Tuner/Shared/scanners/dvbc/transponders/5244.ini'): No space left on device

[Errno 28] No space left on device: '/etc/letsencrypt/.certbot.lock'

This morning, at the moment when I wrote this post, the problem was there. After leaving the system untouched until now, everything works as expected after a test. From this morning until now, nothing has changed in the system. Even in the logs where no entries that would indicate any change. 

 

Now I started all containers and the problem seems resolved. I don't know what could cause this problem.

 

But thank you @Squid for looking into this. 

when that happens again, go to the Docker Tab and hit Container Size and post the screen shot

sysctl -w kernel.pty.max=8192

 

fixed the issue for me.