June 21, 20242 yr I’m posting here for further help investigating an issue I’m having with my server. I don’t know what exactly happened, but it seems like my docker image grew full. When I woke up this morning my server was unresponsive. I force restarted it and checked common problems, updated my docker image size, and everything worked for a bit. When I got home, I logged into the server via CLI and tried to troubleshoot. the command "df -h" output this: Last login: Thu Jun 20 17:09:00 2024 Linux 6.1.79-Unraid. root@MediaMonsta:~# df -h Filesystem Size Used Avail Use% Mounted on rootfs 16G 350M 16G 3% / tmpfs 128M 9.6M 119M 8% /run /dev/sda1 30G 1017M 29G 4% /boot overlay 16G 350M 16G 3% /lib overlay 16G 350M 16G 3% /usr devtmpfs 8.0M 0 8.0M 0% /dev tmpfs 16G 0 16G 0% /dev/shm tmpfs 128M 760K 128M 1% /var/log tmpfs 1.0M 0 1.0M 0% /mnt/disks tmpfs 1.0M 0 1.0M 0% /mnt/remotes tmpfs 1.0M 0 1.0M 0% /mnt/addons tmpfs 1.0M 0 1.0M 0% /mnt/rootshare /dev/md1p1 17T 9.0T 7.5T 55% /mnt/disk1 /dev/md2p1 17T 312G 17T 2% /mnt/disk2 /dev/md3p1 13T 13T 165G 99% /mnt/disk3 /dev/md4p1 13T 13T 18G 100% /mnt/disk4 /dev/md5p1 13T 4.7T 8.1T 37% /mnt/disk5 /dev/md6p1 13T 3.0T 9.9T 23% /mnt/disk6 /dev/md7p1 11T 78G 11T 1% /mnt/disk7 /dev/md8p1 11T 78G 11T 1% /mnt/disk8 /dev/nvme0n1p1 932G 184G 744G 20% /mnt/cache_appdata_m_ii /dev/sdg1 932G 104G 824G 12% /mnt/downloads shfs 106T 43T 64T 41% /mnt/user0 shfs 106T 43T 64T 41% /mnt/user /dev/loop2 256G 127G 129G 50% /var/lib/docker /dev/loop3 1.0G 4.1M 905M 1% /etc/libvirt tmpfs 3.2G 0 3.2G 0% /run/user/0 root@MediaMonsta:~# After that, I tried to stop all my docker containers in an attempt to speed up the webGUI using the command "docker stop $(docker ps -a -q)". That worked, so I dug further and tried to run the space invaders "check docker image" script, which didn’t output anything at all because docker failed to start. I checked common problems again and got the error “Out Of Memory errors detected on your server.” Any help would be greatly appreciated! mediamonsta-diagnostics-20240620-2343.zip
June 21, 20242 yr Community Expert Start by running a correcting scrub on the pool and post the results.
June 21, 20242 yr Author 2 hours ago, JorgeB said: Click on the pool then scrub. Which pool am I scrubbing?
June 21, 20242 yr Community Expert You can scrub all, but downloads is the one with corruption detected.
June 21, 20242 yr Author Are you sure, I just scrubbed all the pools and found no errors. Edited June 21, 20242 yr by tommykmusic
June 21, 20242 yr Community Expert 27 minutes ago, tommykmusic said: Are you sure Jun 20 09:28:33 MediaMonsta kernel: BTRFS info (device sdg1): bdev /dev/sdg1 errs: wr 0, rd 0, flush 0, corrupt 18, gen 0 They could be old errors, in that case reset the stats and see if no more corruption is detected
June 21, 20242 yr Author 1 hour ago, JorgeB said: Jun 20 09:28:33 MediaMonsta kernel: BTRFS info (device sdg1): bdev /dev/sdg1 errs: wr 0, rd 0, flush 0, corrupt 18, gen 0 They could be old errors, in that case reset the stats and see if no more corruption is detected Just to make sure, I reset the stats using the command "btrfs dev stats -z /mnt/cache" and then check for corruption again?
June 21, 20242 yr Community Expert Just now, tommykmusic said: btrfs dev stats -z /mnt/cache" The pool is called downloads, not cache, so /mnt/downloads
June 22, 20242 yr Author root@MediaMonsta:~# btrfs dev stats -z /mnt/downloads [/dev/sdg1].write_io_errs 0 [/dev/sdg1].read_io_errs 0 [/dev/sdg1].flush_io_errs 0 [/dev/sdg1].corruption_errs 18 [/dev/sdg1].generation_errs 0 root@MediaMonsta:~# btrfs dev stats /mnt/downloads [/dev/sdg1].write_io_errs 0 [/dev/sdg1].read_io_errs 0 [/dev/sdg1].flush_io_errs 0 [/dev/sdg1].corruption_errs 0 [/dev/sdg1].generation_errs 0 root@MediaMonsta:~# This is the output for the downloads.
June 22, 20242 yr Community Expert Stats have been reset, keep monitoring to see if more errors appear in the near future, if yes, there's still a problem.
June 23, 20242 yr Author 2 hours ago, JorgeB said: Stats have been reset, keep monitoring to see if more errors appear in the near future, if yes, there's still a problem. My docker service still says "docker service failed to start" and fix common problems still has the same errors. Should I reboot the server?
June 23, 20242 yr Community Expert Yes, and post new diags if the docker service still fails to start.
June 23, 20242 yr Author I still seem to be having the same issue. My webgui is slow because something in the docker is filling up memory. I can better access everything through the CLI. Is there a command to stop all the docker containers or just turn docker off? That will help me post the new diags as I will be ale to download them from the web gui.
June 24, 20242 yr Community Expert /etc/rc.d/rc.docker <command> Available commands: start stop restart status
June 24, 20242 yr Community Expert Docker image is showing some corruption, it's just one event, still it may be worth to recreate it: https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file Then: https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications Also see below if you have any custom docker networks: https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks
June 26, 20242 yr Author I instead opted to buy and install a 2tb m.2 ssd into the server and create a cache pool for my system share and move the whole system share onto the cache pool. Now I am monitoring for any changes. So far I noticed that docker loads up super fast! I don't know why I didn’t do this earlier.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.