lovingHDTV Posted August 11, 2015 Author Share Posted August 11, 2015 Here is where I stand today. After recreating the docker.img file 5 times and re-installing my dockers I still had crashplan continue to grow the img file. The last time I rebuilt I did two things differently. I've no idea why this would make a difference, but it did. 1. installed crashplan docker first 2. when I use my templates unRaid for some reason always puts an empty line in the container volumes section. It is the first line so you cannot remove it. So I took the last line /mnt for me, cut-n-pasted it into the first line, then removed the last (empty) line. So I had no empty container volume entries. I then did the same thing for the other containers and everything has been running fine with no growth. Like I said I have no idea why this would make a difference, but I"m still at the ~5GB img file I have after recreating all my containers. No growth. As a caveat, I have noticed that with Crashplan my brother has not been backing up to me in the last couple days. He happens to be on vacation and his Crashplan stopped working while he was away. We'll see if docker.img growth returns when he gets his backup running again. david Quote Link to comment
topcat Posted May 9, 2017 Share Posted May 9, 2017 Hello everyone, since I updated my UnRaid to 6.3.3 I get the "Error response from daemon: write /var/run/docker/libcontainerd/.../config.json: no space left on device." error when I want to start a container. The Docker Settings page tells me that there is plenty of space left for Docker. In addition the number of running container did not increase after the update. I have no idea what happend I hope someone can help me. Here some system information: docker info Containers: 22 Running: 15 Paused: 0 Stopped: 7 Images: 103 Server Version: 1.12.6 Storage Driver: btrfs Build Version: Btrfs v4.7.2 Library Version: 101 Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: null host overlay bridge Swarm: inactive Runtimes: runc Default Runtime: runc Security Options: Kernel Version: 4.9.19-unRAID Operating System: Slackware 14.2 OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 15.59 GiB Name: Scruffy ID: C7JW:L6DT:2NEI:CBHJ:DODU:4WB2:S5MK:KO3U:BLKP:V2HV:D66L:DUK3 Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Insecure Registries: 127.0.0.0/8 docker version Client: Version: 1.12.6 API version: 1.24 Go version: go1.7.4 Git commit: 6b644ec Built: Mon Jan 16 05:45:42 2017 OS/Arch: linux/amd64 Server: Version: 1.12.6 API version: 1.24 Go version: go1.7.4 Git commit: 6b644ec Built: Mon Jan 16 05:45:42 2017 OS/Arch: linux/amd64 df -h Filesystem Size Used Avail Use% Mounted on rootfs 7.8G 7.8G 0 100% / tmpfs 7.8G 340K 7.8G 1% /run devtmpfs 7.8G 4.0K 7.8G 1% /dev cgroup_root 7.8G 0 7.8G 0% /sys/fs/cgroup tmpfs 128M 4.4M 124M 4% /var/log /dev/sda1 7.5G 760M 6.7G 10% /boot /dev/md1 3.7T 1023G 2.7T 28% /mnt/disk1 /dev/sdc1 239G 79G 160G 33% /mnt/cache shfs 3.7T 1023G 2.7T 28% /mnt/user0 shfs 3.9T 1.1T 2.8T 28% /mnt/user /dev/sdb1 932G 195G 736G 21% /mnt/disks/ST1000LM035_1RK172_WDE65Y0Q /dev/loop0 75G 19G 52G 26% /var/lib/docker shm 64M 0 64M 0% /var/lib/docker/containers/63d0b4dbdba3bc5589ee889e3a2dd44b3c26470dbd7c676ee5d6ac8f153d6030/shm shm 64M 0 64M 0% /var/lib/docker/containers/ba0e8375ca8d140d700e437ab528ac521912acf560e61c4a0ea867999c5c6002/shm shm 64M 4.0K 64M 1% /var/lib/docker/containers/350e8f30e5a6c76226308c784d067b536742bcf57adbf9657955ac45bdb82f12/shm shm 64M 0 64M 0% /var/lib/docker/containers/fd555658558e580fef776e98ebe240433ef2955477d9f7783ce1baf08dce1033/shm shm 64M 0 64M 0% /var/lib/docker/containers/82c2f68bb1e32e2af9b51195e3c1224696d681db46334736f0fbb5c37e119b43/shm shm 64M 0 64M 0% /var/lib/docker/containers/b292b1394be3a759cc94ec47f0e7a213137d34162506d437f6a859a284c5c87e/shm shm 64M 0 64M 0% /var/lib/docker/containers/b1e1e8a2b8b93821fa58579ee1c83c4c42957ecc4572d85b4c66535857d5e419/shm shm 64M 0 64M 0% /var/lib/docker/containers/9559c72564c5e018e991ac7c0c0a2434acc51921ce40d84debf719ae3e562233/shm shm 64M 0 64M 0% /var/lib/docker/containers/754acbf42975ae504146b6054dc7a203916ecd9291b8100d6d1708c939b4ae18/shm shm 64M 0 64M 0% /var/lib/docker/containers/6ec58284c1fef41919b092e3a2acdb5d6037ff6635c60d4a2d618394032e3944/shm shm 64M 0 64M 0% /var/lib/docker/containers/62119ccf186090093dba47e3952fda7b0eb0a145fa5b8774edb650eb6d625156/shm /dev/loop1 1.0G 17M 905M 2% /etc/libvirt shm 64M 0 64M 0% /var/lib/docker/containers/45bf130aea7b0616794d4f3291c930035e7d6e6fe5df990912403b4dece46a06/shm shm 64M 0 64M 0% /var/lib/docker/containers/d7938d9471e3c3773bc174edf8d52271ef563c10c0cbe73c513db3bcd483181c/shm shm 64M 0 64M 0% /var/lib/docker/containers/10edc7baea8f314157d2c3d0f1fab013448484ea70801c6ec30ef9ccd23bcc5c/shm shm 64M 0 64M 0% /var/lib/docker/containers/71caf3ffb00c3810e89b933377b9ef6b42f780678ff05b08a33d671e8e37f020/shm docker images REPOSITORY TAG IMAGE ID CREATED SIZE ttobias/logfile-notifications latest 5ca8d7c5bbdb 7 days ago 103.7 MB wonderfall/nextcloud latest d7bad793fcef 8 days ago 280.5 MB gitea/gitea latest 9801a66b7f64 9 days ago 73.58 MB binhex/arch-minidlna latest c823ae86db35 3 weeks ago 993.8 MB jrcs/letsencrypt-nginx-proxy-companion latest 37305499ed8d 3 weeks ago 68.37 MB jwilder/nginx-proxy latest 696388b8d0ba 3 weeks ago 247.5 MB mbentley/teamspeak latest 2b7687efa053 8 weeks ago 151.5 MB cyrilix/subsonic latest f469de58eda7 8 weeks ago 515.4 MB mariadb latest 56741a13bbb9 10 weeks ago 393.4 MB themattrix/mapcrafter latest a25af4c85b92 10 weeks ago 228.8 MB openhab/openhab 2.0.0-amd64 41d102dac4dc 3 months ago 644.9 MB topdockercat/minio-unraid latest 764562230e73 3 months ago 275.4 MB rockyhb/fhem latest b714da13182d 3 months ago 238.8 MB nginx latest 01f818af747d 4 months ago 181.6 MB gfjardim/crashplan latest 3517caa72cbc 6 months ago 793.6 MB topdockercat/terraria-server-vanilla latest f369c03e26f1 6 months ago 211.9 MB hexparrot/mineos node-crux 195e248819d6 8 months ago 658.4 MB debian latest 031143c1c662 8 months ago 125.1 MB devtelegramservice latest 015d7124151c 9 months ago 9.532 MB topdockercat/terraria-server latest 232e336887f2 9 months ago 223.6 MB alpine edge dc4ab3a6c342 10 months ago 4.795 MB alpine 3.4 4e38e38c8ce0 10 months ago 4.795 MB alpine latest 4e38e38c8ce0 10 months ago 4.795 MB xataz/letsencrypt latest 9a023097952d 11 months ago 92.73 MB topdockercat/inform-minecraft latest d304d3834e20 13 months ago 714.5 MB kvaps/kolab latest b347be66c842 13 months ago 1.547 GB mjeries/mineos-node latest ed4256a29b7c 14 months ago 886.9 MB topdockercat/nsupdate latest d37bfa3817d8 14 months ago 259.6 MB nachinius/c-dev latest fcb764c8c090 17 months ago 427.9 MB voltairemc/minecraft-site latest abbdb363ecba 20 months ago 1.202 GB elsdoerfer/hibiscus-server latest 1dd61cbbaafe 2 years ago 780.5 MB -------------------------------------------------------------------------------------------------------------- 12410.88 MB Quote Link to comment
topcat Posted May 20, 2017 Share Posted May 20, 2017 Sorry for the late replay. I just testet 6.3.4 but the problem still remains. If I type diagnostics into the console I got this diagnostics Starting diagnostics collection... Warning: file_put_contents(): Only 0 of 5 bytes written, possibly out of free di sk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 69 Warning: file_put_contents(): Only 0 of 15 bytes written, possibly out of free d isk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74 Warning: file_put_contents(): Only 0 of 228 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74 Warning: file_put_contents(): Only 0 of 176 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74 Warning: file_put_contents(): Only 0 of 4241 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74 Warning: file_put_contents(): Only 0 of 789 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74 Warning: file_put_contents(): Only 0 of 2705 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74 Warning: file_put_contents(): Only 0 of 2515 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74 Warning: file_put_contents(): Only 0 of 1625 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74 Warning: file_put_contents(): Only 0 of 4326 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74 Warning: file_put_contents(): Only 0 of 3751 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 74 Warning: file_put_contents(): Only 0 of 2 bytes written, possibly out of free di sk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 90 Warning: file_put_contents(): Only 0 of 34 bytes written, possibly out of free d isk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 92 Warning: file_put_contents(): Only 0 of 2 bytes written, possibly out of free di sk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 90 Warning: file_put_contents(): Only 0 of 34 bytes written, possibly out of free d isk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 92 sh: line 0: echo: write error: No space left on device sh: line 0: echo: write error: No space left on device sh: line 0: echo: write error: No space left on device sh: line 0: echo: write error: No space left on device sh: line 0: echo: write error: No space left on device sh: line 0: echo: write error: No space left on device sh: line 0: echo: write error: No space left on device sh: line 0: echo: write error: No space left on device sh: line 0: echo: write error: No space left on device done. ZIP file '/boot/logs/scruffy-diagnostics-20170520-2129.zip' created. And a zip file with empty text files. Quote Link to comment
bonienl Posted May 20, 2017 Share Posted May 20, 2017 Your rootfs is 100% used, meaning no RAM memory left to story data. Something is filling your RAM and you need to find out what. I suggest the following: 1. Start in safemode, this will disable any plugins and check memory usage 2. Stop all dockers and start them one at the time to check their behaviour 1 Quote Link to comment
topcat Posted May 20, 2017 Share Posted May 20, 2017 14 minutes ago, bonienl said: Your rootfs is 100% used, meaning no RAM memory left to story data. Something is filling your RAM and you need to find out what. I suggest the following: 1. Start in safemode, this will disable any plugins and check memory usage 2. Stop all dockers and start them one at the time to check their behaviour Thanks for the hint. I must had overlooked this. -.-. I will give it a try. Quote Link to comment
topcat Posted May 21, 2017 Share Posted May 21, 2017 Thanks for your help. I have found the problem. I had setup a sceduled script wich uses the minio client mc to mirror my backup to an offsite S3 compatible storage. https://docs.minio.io/docs/minio-client-quickstart-guide The command looked like this. ./mc mirror <some direcory> <hostalias>/<some directory> I'm not sure why, but the minio client has forgotten the hostalias. In this case mc mirror the backup to the home folder of the executing user ~/<hostalias>/<some directory>. This directory is located on the rootfs. Every night the offsite mirroring runs, it fills up my rootfs. -.- Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.