No space left on device error


Recommended Posts

Here is where I stand today.

 

After recreating the docker.img file 5 times and re-installing my dockers I still had crashplan continue to grow the img file.

 

The last time I rebuilt I did two things differently.  I've no idea why this would make a difference, but it did.

 

1. installed crashplan docker first

2. when I use my templates unRaid for some reason always puts an empty line in the container volumes section.  It is the first line so you cannot remove it.  So I took the last line /mnt for me, cut-n-pasted it into the first line, then removed the last (empty) line. So I had no empty container volume entries.

 

I then did the same thing for the other containers and everything has been running fine with no growth.  Like I said I have no idea why this would make a difference, but I"m still at the ~5GB img file I have after recreating all my containers.  No growth.

 

As a caveat, I have noticed that with Crashplan my brother has not been backing up to me in the last couple days.  He happens to be on vacation and his Crashplan stopped working while he was away.  We'll see if docker.img growth returns when he gets his backup running again.

 

david

Link to comment
  • 1 year later...

Hello everyone,

 

since I updated my UnRaid to 6.3.3 I get the "Error response from daemon: write /var/run/docker/libcontainerd/.../config.json: no space left on device." error  when I want to start a container. The Docker Settings page tells me that there is plenty of space left for Docker. In addition the number of running container did not increase after the update. 

I have no idea what happend I hope someone can help me.

 

 

Here some system information:

docker info
Containers: 22
 Running: 15
 Paused: 0
 Stopped: 7
Images: 103
Server Version: 1.12.6
Storage Driver: btrfs
 Build Version: Btrfs v4.7.2
 Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host overlay bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 4.9.19-unRAID
Operating System: Slackware 14.2
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.59 GiB
Name: Scruffy
ID: C7JW:L6DT:2NEI:CBHJ:DODU:4WB2:S5MK:KO3U:BLKP:V2HV:D66L:DUK3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8
docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.7.4
 Git commit:   6b644ec
 Built:        Mon Jan 16 05:45:42 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.7.4
 Git commit:   6b644ec
 Built:        Mon Jan 16 05:45:42 2017
 OS/Arch:      linux/amd64
 df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          7.8G  7.8G     0 100% /
tmpfs           7.8G  340K  7.8G   1% /run
devtmpfs        7.8G  4.0K  7.8G   1% /dev
cgroup_root     7.8G     0  7.8G   0% /sys/fs/cgroup
tmpfs           128M  4.4M  124M   4% /var/log
/dev/sda1       7.5G  760M  6.7G  10% /boot
/dev/md1        3.7T 1023G  2.7T  28% /mnt/disk1
/dev/sdc1       239G   79G  160G  33% /mnt/cache
shfs            3.7T 1023G  2.7T  28% /mnt/user0
shfs            3.9T  1.1T  2.8T  28% /mnt/user
/dev/sdb1       932G  195G  736G  21% /mnt/disks/ST1000LM035_1RK172_WDE65Y0Q
/dev/loop0       75G   19G   52G  26% /var/lib/docker
shm              64M     0   64M   0% /var/lib/docker/containers/63d0b4dbdba3bc5589ee889e3a2dd44b3c26470dbd7c676ee5d6ac8f153d6030/shm
shm              64M     0   64M   0% /var/lib/docker/containers/ba0e8375ca8d140d700e437ab528ac521912acf560e61c4a0ea867999c5c6002/shm
shm              64M  4.0K   64M   1% /var/lib/docker/containers/350e8f30e5a6c76226308c784d067b536742bcf57adbf9657955ac45bdb82f12/shm
shm              64M     0   64M   0% /var/lib/docker/containers/fd555658558e580fef776e98ebe240433ef2955477d9f7783ce1baf08dce1033/shm
shm              64M     0   64M   0% /var/lib/docker/containers/82c2f68bb1e32e2af9b51195e3c1224696d681db46334736f0fbb5c37e119b43/shm
shm              64M     0   64M   0% /var/lib/docker/containers/b292b1394be3a759cc94ec47f0e7a213137d34162506d437f6a859a284c5c87e/shm
shm              64M     0   64M   0% /var/lib/docker/containers/b1e1e8a2b8b93821fa58579ee1c83c4c42957ecc4572d85b4c66535857d5e419/shm
shm              64M     0   64M   0% /var/lib/docker/containers/9559c72564c5e018e991ac7c0c0a2434acc51921ce40d84debf719ae3e562233/shm
shm              64M     0   64M   0% /var/lib/docker/containers/754acbf42975ae504146b6054dc7a203916ecd9291b8100d6d1708c939b4ae18/shm
shm              64M     0   64M   0% /var/lib/docker/containers/6ec58284c1fef41919b092e3a2acdb5d6037ff6635c60d4a2d618394032e3944/shm
shm              64M     0   64M   0% /var/lib/docker/containers/62119ccf186090093dba47e3952fda7b0eb0a145fa5b8774edb650eb6d625156/shm
/dev/loop1      1.0G   17M  905M   2% /etc/libvirt
shm              64M     0   64M   0% /var/lib/docker/containers/45bf130aea7b0616794d4f3291c930035e7d6e6fe5df990912403b4dece46a06/shm
shm              64M     0   64M   0% /var/lib/docker/containers/d7938d9471e3c3773bc174edf8d52271ef563c10c0cbe73c513db3bcd483181c/shm
shm              64M     0   64M   0% /var/lib/docker/containers/10edc7baea8f314157d2c3d0f1fab013448484ea70801c6ec30ef9ccd23bcc5c/shm
shm              64M     0   64M   0% /var/lib/docker/containers/71caf3ffb00c3810e89b933377b9ef6b42f780678ff05b08a33d671e8e37f020/shm
docker images
REPOSITORY                               TAG                 IMAGE ID            CREATED             SIZE
ttobias/logfile-notifications            latest              5ca8d7c5bbdb        7 days ago          103.7 MB
wonderfall/nextcloud                     latest              d7bad793fcef        8 days ago          280.5 MB
gitea/gitea                              latest              9801a66b7f64        9 days ago          73.58 MB
binhex/arch-minidlna                     latest              c823ae86db35        3 weeks ago         993.8 MB
jrcs/letsencrypt-nginx-proxy-companion   latest              37305499ed8d        3 weeks ago         68.37 MB
jwilder/nginx-proxy                      latest              696388b8d0ba        3 weeks ago         247.5 MB
mbentley/teamspeak                       latest              2b7687efa053        8 weeks ago         151.5 MB
cyrilix/subsonic                         latest              f469de58eda7        8 weeks ago         515.4 MB
mariadb                                  latest              56741a13bbb9        10 weeks ago        393.4 MB
themattrix/mapcrafter                    latest              a25af4c85b92        10 weeks ago        228.8 MB
openhab/openhab                          2.0.0-amd64         41d102dac4dc        3 months ago        644.9 MB
topdockercat/minio-unraid                latest              764562230e73        3 months ago        275.4 MB
rockyhb/fhem                             latest              b714da13182d        3 months ago        238.8 MB
nginx                                    latest              01f818af747d        4 months ago        181.6 MB
gfjardim/crashplan                       latest              3517caa72cbc        6 months ago        793.6 MB
topdockercat/terraria-server-vanilla     latest              f369c03e26f1        6 months ago        211.9 MB
hexparrot/mineos                         node-crux           195e248819d6        8 months ago        658.4 MB
debian                                   latest              031143c1c662        8 months ago        125.1 MB
devtelegramservice                       latest              015d7124151c        9 months ago        9.532 MB
topdockercat/terraria-server             latest              232e336887f2        9 months ago        223.6 MB
alpine                                   edge                dc4ab3a6c342        10 months ago       4.795 MB
alpine                                   3.4                 4e38e38c8ce0        10 months ago       4.795 MB
alpine                                   latest              4e38e38c8ce0        10 months ago       4.795 MB
xataz/letsencrypt                        latest              9a023097952d        11 months ago       92.73 MB
topdockercat/inform-minecraft            latest              d304d3834e20        13 months ago       714.5 MB
kvaps/kolab                              latest              b347be66c842        13 months ago       1.547 GB
mjeries/mineos-node                      latest              ed4256a29b7c        14 months ago       886.9 MB
topdockercat/nsupdate                    latest              d37bfa3817d8        14 months ago       259.6 MB
nachinius/c-dev                          latest              fcb764c8c090        17 months ago       427.9 MB
voltairemc/minecraft-site                latest              abbdb363ecba        20 months ago       1.202 GB
elsdoerfer/hibiscus-server               latest              1dd61cbbaafe        2 years ago         780.5 MB
--------------------------------------------------------------------------------------------------------------
                                                                                                   12410.88 MB

 

Link to comment
  • 2 weeks later...

Sorry for the late replay. I just testet 6.3.4 but the problem still remains.

If I type diagnostics into the console I got this

 diagnostics
Starting diagnostics collection...
Warning: file_put_contents(): Only 0 of 5 bytes written, possibly out of free di                                                                                  sk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 69

Warning: file_put_contents(): Only 0 of 15 bytes written, possibly out of free d                                                                                  isk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74

Warning: file_put_contents(): Only 0 of 228 bytes written, possibly out of free                                                                                   disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74

Warning: file_put_contents(): Only 0 of 176 bytes written, possibly out of free                                                                                   disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74

Warning: file_put_contents(): Only 0 of 4241 bytes written, possibly out of free                                                                                   disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74

Warning: file_put_contents(): Only 0 of 789 bytes written, possibly out of free                                                                                   disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74

Warning: file_put_contents(): Only 0 of 2705 bytes written, possibly out of free                                                                                   disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74

Warning: file_put_contents(): Only 0 of 2515 bytes written, possibly out of free                                                                                   disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74

Warning: file_put_contents(): Only 0 of 1625 bytes written, possibly out of free                                                                                   disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74

Warning: file_put_contents(): Only 0 of 4326 bytes written, possibly out of free                                                                                   disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74

Warning: file_put_contents(): Only 0 of 3751 bytes written, possibly out of free                                                                                   disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74

Warning: file_put_contents(): Only 0 of 2 bytes written, possibly out of free di                                                                                  sk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 90

Warning: file_put_contents(): Only 0 of 34 bytes written, possibly out of free d                                                                                  isk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 92

Warning: file_put_contents(): Only 0 of 2 bytes written, possibly out of free di                                                                                  sk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 90

Warning: file_put_contents(): Only 0 of 34 bytes written, possibly out of free d                                                                                  isk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 92
sh: line 0: echo: write error: No space left on device
sh: line 0: echo: write error: No space left on device
sh: line 0: echo: write error: No space left on device
sh: line 0: echo: write error: No space left on device
sh: line 0: echo: write error: No space left on device
sh: line 0: echo: write error: No space left on device
sh: line 0: echo: write error: No space left on device
sh: line 0: echo: write error: No space left on device
sh: line 0: echo: write error: No space left on device
done.
ZIP file '/boot/logs/scruffy-diagnostics-20170520-2129.zip' created.

And a zip file with empty text files.

Link to comment

Your rootfs is 100% used, meaning no RAM memory left to story data.

 

Something is filling your RAM and you need to find out what. I suggest the following:

 

1. Start in safemode, this will disable any plugins and check memory usage

2. Stop all dockers and start them one at the time to check their behaviour

  • Upvote 1
Link to comment
14 minutes ago, bonienl said:

Your rootfs is 100% used, meaning no RAM memory left to story data.

 

Something is filling your RAM and you need to find out what. I suggest the following:

 

1. Start in safemode, this will disable any plugins and check memory usage

2. Stop all dockers and start them one at the time to check their behaviour

Thanks for the hint. I must had overlooked this. -.-.  I will give it a try.

 

Link to comment

Thanks for your help. I have found the problem.

I had setup a sceduled script wich uses the minio client mc to mirror my backup to an offsite S3 compatible storage. https://docs.minio.io/docs/minio-client-quickstart-guide

 

The command looked like this.

./mc mirror <some direcory> <hostalias>/<some directory>

I'm not sure why, but the minio client has forgotten the hostalias. In this case mc mirror the backup to the home folder of the executing user ~/<hostalias>/<some directory>. This directory is located on the rootfs. Every night the offsite mirroring runs, it fills up my rootfs. -.-

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.