3dee Posted April 4, 2021 Share Posted April 4, 2021 Hi! One of my Docker Container writes data to a share, wich is totally fine! But as soon as a drive reached 100%, all the docker containers stopped working. Also RAM and CPU went both to 100% and something begins to read disks.. See the Screenshots: My top Output is this: Spoiler top - 10:43:41 up 10:48, 0 users, load average: 216.56, 208.60, 205.09 Tasks: 489 total, 17 running, 468 sleeping, 0 stopped, 4 zombie %Cpu(s): 0.2 us, 97.7 sy, 0.0 ni, 0.0 id, 2.0 wa, 0.0 hi, 0.2 si, 0.0 st MiB Mem : 15929.6 total, 131.8 free, 14628.9 used, 1168.9 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 76.7 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 712 root 20 0 0 0 0 R 83.5 0.0 223:44.62 kswapd0 16332 nobody 20 0 2305384 159152 4 D 26.9 1.0 11:29.30 mono 15669 nobody 20 0 2321512 169996 4 S 15.2 1.0 11:19.22 Plex Media Serv 11264 nobody 20 0 7890296 346300 0 S 13.3 2.1 10:50.68 java 19324 nobody 20 0 1136212 19592 0 S 12.9 0.1 8:29.67 ts3server 11368 nobody 20 0 1878360 81824 0 S 12.6 0.5 8:02.47 mysqld 6966 root 20 0 2333984 77168 42232 S 12.3 0.5 1:50.98 dockerd 6783 root 20 0 1159260 21636 924 S 12.0 0.1 8:35.28 shfs 7947 root 20 0 866028 672268 21332 S 11.7 4.1 17:17.28 qemu-system-x86 17069 nobody 20 0 2196000 12040 0 S 10.7 0.1 5:48.96 Plex DLNA Serve 8797 root 20 0 1056848 5748 0 S 9.4 0.0 3:35.50 bitwarden_rs 16374 nobody 20 0 473840 51204 0 R 6.5 0.3 3:42.09 deluged 6982 root 20 0 2100864 38408 20596 S 5.8 0.2 8:21.78 containerd 8029 root 20 0 1108812 488188 22696 S 4.5 3.0 7:18.76 qemu-system-x86 16502 nobody 20 0 887572 7900 0 S 4.2 0.0 2:31.15 qbittorrent-nox 13783 nobody 20 0 13.1g 11.2g 0 D 3.9 71.9 10:22.68 rtorrent main 10606 root 20 0 31876 16724 0 D 3.6 0.1 1:47.66 supervisord 13847 root 20 0 3624 2564 1812 S 3.6 0.0 0:00.33 lsof 22620 root 20 0 11612 272 0 D 3.6 0.0 0:22.84 curl 12258 root 20 0 28160 10868 0 S 3.2 0.1 5:08.67 fail2ban-server 13836 root 20 0 6956 3540 2488 R 3.2 0.0 0:00.64 top 17076 nobody 20 0 438648 1928 0 R 3.2 0.0 3:12.72 Plex Tuner Serv 21878 root 20 0 11628 284 0 D 3.2 0.0 0:34.59 curl 25156 root 20 0 11628 280 0 D 3.2 0.0 0:30.06 curl 414 nobody 20 0 27264 1924 0 R 2.9 0.0 0:33.14 php7 6548 root 20 0 283700 4148 3536 S 2.9 0.0 2:25.86 emhttpd 8089 root 20 0 31988 16824 0 D 2.9 0.1 1:41.08 supervisord 12973 root 20 0 31484 16208 0 D 2.9 0.1 1:43.41 supervisord 14805 root 20 0 31484 16220 0 R 2.9 0.1 1:42.80 supervisord 18987 root 20 0 93032 34884 536 D 2.9 0.2 1:42.54 virt-manager 12257 root 20 0 1592 60 0 R 2.6 0.0 0:40.36 crond 12427 root 20 0 2344 80 0 D 2.6 0.0 0:58.25 tini 8577 root 20 0 32264 16900 0 D 2.3 0.1 0:25.31 supervisord 11795 root 20 0 19824 1516 0 D 2.3 0.0 11:36.26 openvpn 13819 nobody 20 0 1031824 17392 0 S 2.3 0.1 2:36.94 qbittorrent-nox 15221 nobody 20 0 4598232 119252 0 S 2.3 0.7 3:36.12 jackett 19586 nobody 20 0 180136 6188 160 D 2.3 0.0 0:41.94 nginx 25549 nobody 20 0 61628 2012 0 D 2.3 0.0 0:31.20 php7 26132 root 20 0 2932 56 0 D 2.3 0.0 0:24.26 grep 2091 root 20 0 22652 444 0 D 1.9 0.0 0:30.67 curl 2576 nobody 20 0 25608 1780 0 D 1.9 0.0 0:25.68 php7 3105 root 20 0 480 80 0 D 1.9 0.0 0:15.03 sleep 10723 nobody 20 0 7520 664 0 D 1.9 0.0 0:23.67 logrotate.sh 17190 nobody 20 0 70472 38932 0 D 1.9 0.2 1:05.24 deluge-web 18881 root 20 0 31364 16288 0 D 1.9 0.1 1:45.61 supervisord 25011 nobody 20 0 11160 980 0 D 1.9 0.0 1:22.28 php7 6676 root 20 0 0 0 0 S 1.6 0.0 14:46.90 unraidd2 7816 root 20 0 2572 48 0 D 1.6 0.0 0:16.89 ash 10226 root 20 0 1592 56 0 D 1.6 0.0 1:06.69 crond 10520 root 20 0 2372 96 4 D 1.6 0.0 1:00.48 tini 18834 root 20 0 27964 12400 4572 D 1.6 0.1 0:50.95 Xvfb 19588 nobody 20 0 180128 6196 180 R 1.6 0.0 0:01.57 nginx 20434 nobody 20 0 70856 2572 0 D 1.6 0.0 0:22.77 php 22769 root 20 0 11612 272 0 D 1.6 0.0 0:21.53 curl 28191 root 20 0 484 80 0 D 1.6 0.0 0:26.61 sleep 8257 root 20 0 32304 16948 0 D 1.3 0.1 0:35.01 supervisord 9822 root 20 0 26588 12832 0 R 1.3 0.1 1:24.58 supervisord 14656 nobody 20 0 25608 1720 0 D 1.3 0.0 0:24.87 php7 Also I cannot open the Docker Tab on the unraid web UI nor open the Docker Settings under Settings Tab → Docker It's like that for around 10 hours now, hoping it will fix itself, but no luck so far. This happened before, I rebooted "clean" then, but since then my Pi-Hole Docker is gone (can I bring it back?) and some latest changes to different docker containers are "rolled back". After that reboot the filled disk was not 100% anymore, it went down to like 99 or 98%. Now I let it fill back to 100% in the hope that this was just a bug, but now it's happening again. I attached diagnostic files. Thank you so much! serverpc-diagnostics-20210404-1042.zip Quote Link to comment
Squid Posted April 4, 2021 Share Posted April 4, 2021 rtorrent has trashed your system. It's consuming an insane amount of memory right now. 2 hours ago, 3dee said: Pi-Hole Docker is gone (can I bring it back?) Apps - Previous Apps 2 hours ago, 3dee said: some latest changes to different docker containers are "rolled back". If you are manually making changes inside the container (ie: docker exec -it rutorrent /bin/bash or using the container's console), then after a reinstallation or upgrade of the container, those changes will be gone. This is one of the big features touted by docker Are you sure that you've got rutorrent set up correctly? As in it's not downloading to RAM? 1 Quote Link to comment
3dee Posted April 4, 2021 Author Share Posted April 4, 2021 (edited) 6 hours ago, Squid said: It's consuming an insane amount of memory right now. Before the disk was full it was fine 6 hours ago, Squid said: Apps - Previous Apps I will try as soon my Docker works again 6 hours ago, Squid said: If you are manually making changes inside the container Changes were made via GUI only 6 hours ago, Squid said: Are you sure that you've got rutorrent set up correctly? As in it's not downloading to RAM? I set up the data path to /mnt/user/media/Torrent and the downloads are being saved to /data/ The RAM started filling up when the HDD went full. Did I maybe accidentally set it up the write to the RAM? I am not even sure how I could do that.. Whats a good way out of this situation? Stop and start the array? Reboot the Server? Still 100% RAM and CPU. How can I make sure next time the container won't fill the disk to 100% so this wont happen again? Edited April 4, 2021 by 3dee Quote Link to comment
trurl Posted April 5, 2021 Share Posted April 5, 2021 Post your docker run command for your rtorrent docker as explained at the very first link in the Docker FAQ 1 Quote Link to comment
3dee Posted April 5, 2021 Author Share Posted April 5, 2021 (edited) I was able to stop and start the array. Downloads are paused now, when I start it, it begins to write to disk 2, which is 500MB left. If I keep downloads running, RAM and CPU will fill up when disk 2 is full again, I'm very sure. Docker run: root@localhost:# /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker run -d --name='binhex-rtorrentvpn' --net='br0' --ip='192.168.1.18' --privileged=true -e TZ="Europe/Berlin" -e HOST_OS="Unraid" -e 'TCP_PORT_5000'='5000' -e 'TCP_PORT_9080'='9080' -e 'TCP_PORT_9443'='9443' -e 'TCP_PORT_8118'='8118' -e 'VPN_ENABLED'='yes' -e 'VPN_USER'='##############' -e 'VPN_PASS'='##############' -e 'VPN_PROV'='pia' -e 'VPN_CLIENT'='openvpn' -e 'VPN_OPTIONS'='' -e 'STRICT_PORT_FORWARD'='yes' -e 'ENABLE_PRIVOXY'='yes' -e 'ENABLE_AUTODL_IRSSI'='no' -e 'ENABLE_RPC2'='yes' -e 'ENABLE_RPC2_AUTH'='yes' -e 'ENABLE_WEBUI_AUTH'='yes' -e 'RPC2_USER'='admin' -e 'RPC2_PASS'='admin' -e 'WEBUI_USER'='admin' -e 'WEBUI_PASS'='admin' -e 'LAN_NETWORK'='192.168.1.0/24' -e 'NAME_SERVERS'='209.222.18.222,209.222.18.218' -e 'VPN_INPUT_PORTS'='' -e 'VPN_OUTPUT_PORTS'='' -e 'DEBUG'='false' -e 'PHP_TZ'='UTC' -e 'UMASK'='000' -e 'PUID'='99' -e 'PGID'='100' -v '/mnt/user/media/Torrents/':'/data':'rw' -v '/mnt/user/appdata/binhex-rtorrentvpn':'/config':'rw' --sysctl="net.ipv4.conf.all.src_valid_mark=1" 'binhex/arch-rtorrentvpn' ce6c73a7cac6d38b6b16e670ff4053c754f25bf4bd1b91871fa11577224996b2 The command finished successfully! Edited April 5, 2021 by 3dee Quote Link to comment
trurl Posted April 5, 2021 Share Posted April 5, 2021 Those mappings look OK. 27 minutes ago, 3dee said: disk 2 is full again Why are you letting this happen? Unraid has no way to know how large a file will become when it chooses a disk for it. If a disk has more than Minimum Free for the share, that disk can be chosen, and if it runs out of space it will fail. You must set Minimum Free for each of your shares to larger than the largest file you expect to write for the share. 1 Quote Link to comment
3dee Posted April 5, 2021 Author Share Posted April 5, 2021 On 4/4/2021 at 2:01 PM, Squid said: Apps - Previous Apps Installed it again, most of the settings are back 1 hour ago, trurl said: You must set Minimum Free for each of your shares to larger than the largest file you expect to write for the share. I was ignorant enough not to do that. Minimum free space is set now, I think I won't run into that issue again. Thank you so much @Squid and @trurl !! Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.