max007 Posted April 17, 2021 Share Posted April 17, 2021 (edited) Hello, for the past few days I've been seeing very high memory use of unraid. It seems to grow and than cap out on 99% and full CPU usage. I am trying to find a cause of this, but I was not yet successful. I have turned off all the dockers and VMs and the memory use still stayed on 50+%. Below are some commands I ran to try and figure out what is using all the memory. I have also tried running all commands on https://www.thegeekdiary.com/how-to-clear-the-buffer-pagecache-disk-cache-under-linux/ without any noticable results. Only reboot seems to help. I also cannot start VM and the system does not clear ram. Could anyone please give me some advise? ps aux | awk '{print $6/1024 " MB\t\t" $11}' | sort -n Quote 0.0898438 MB /usr/sbin/acpid 0.101562 MB /usr/sbin/wsdd 0.109375 MB /usr/sbin/avahi-dnsconfd 0.179688 MB /usr/sbin/dnsmasq 0.179688 MB /usr/sbin/dnsmasq 0.25 MB avahi-daemon: 0.320312 MB /usr/bin/sshfs 0.363281 MB /sbin/apcupsd 0.519531 MB /usr/local/sbin/shfs 0.679688 MB /usr/bin/unionfs 0.679688 MB sleep 0.742188 MB sleep 1.41797 MB /usr/sbin/atd 1.66406 MB /sbin/agetty 1.67188 MB /usr/sbin/crond 1.69141 MB /sbin/agetty 1.69141 MB /usr/sbin/inetd 1.70312 MB init 1.74609 MB /sbin/agetty 1.76953 MB /sbin/agetty 1.78125 MB /sbin/agetty 1.78125 MB /sbin/agetty 1.86328 MB sort 2.01953 MB /usr/bin/unionfs 2.16797 MB dhcpcd 2.17578 MB /sbin/rpcbind 2.20703 MB /usr/bin/dbus-daemon 2.33594 MB ps 2.76172 MB /bin/sh 2.78516 MB /bin/bash 2.80469 MB sh 2.83984 MB /bin/bash 2.85156 MB /bin/bash 2.88281 MB /bin/bash 2.88672 MB /bin/bash 2.90234 MB sh 2.98047 MB awk 2.99219 MB /bin/bash 3.47656 MB avahi-daemon: 3.69531 MB /usr/bin/unionfs 3.69531 MB sshd: 3.74609 MB nginx: 4.07812 MB /usr/local/sbin/emhttpd 4.16406 MB -bash 4.1875 MB /sbin/udevd 4.21484 MB /usr/sbin/rsyslogd 4.32422 MB /usr/bin/sshfs 4.32812 MB /usr/sbin/ntpd 4.60547 MB ttyd 4.66406 MB ssh 4.77344 MB ssh 4.84766 MB /sbin/udevd 5.80469 MB /sbin/haveged 5.90234 MB /sbin/rpc.statd 6.07031 MB /usr/sbin/nmbd 6.34766 MB /usr/sbin/smbd 6.79297 MB sshd: 7.08984 MB /usr/bin/unionfs 9.14453 MB /usr/sbin/smbd 9.33984 MB /usr/sbin/virtlockd 9.80859 MB /usr/bin/unionfs 10.2734 MB nginx: 10.75 MB /usr/sbin/virtlogd 10.9336 MB /usr/sbin/winbindd 10.9766 MB /usr/sbin/winbindd 11.1953 MB /usr/sbin/winbindd 11.4531 MB /usr/local/emhttp/plugins/unbalance/unbalance 11.7539 MB php-fpm: 13.1875 MB php-fpm: 13.9609 MB php-fpm: 15.5234 MB /usr/sbin/smbd 15.6094 MB /usr/sbin/smbd 16.3047 MB /usr/sbin/winbindd 16.6289 MB /usr/sbin/smbd 17.3867 MB /usr/sbin/smbd 18.6875 MB php-fpm: 19.6406 MB php-fpm: 19.9297 MB php-fpm: 19.9336 MB php-fpm: 20.0117 MB php-fpm: 20.0625 MB php-fpm: 20.6953 MB /usr/sbin/libvirtd 26.8203 MB /usr/bin/php 27.0039 MB /usr/bin/php 36.125 MB /usr/local/sbin/shfs 43.3359 MB containerd 53.3477 MB rcloneorig 56.6133 MB rcloneorig 69.3047 MB rcloneorig 88.8359 MB /usr/bin/dockerd 203.684 MB rcloneorig free -m Quote total used free shared buff/cache available Mem: 24033 845 10823 10649 12365 12153 Swap: 0 0 0 df -BK | grep tmpfs Quote devtmpfs 12197548K 0K 12197548K 0% /dev tmpfs 12305204K 0K 12305204K 0% /dev/shm tmpfs 131072K 392K 130680K 1% /var/log tmpfs 1024K 0K 1024K 0% /mnt/disks tmpfs 1024K 0K 1024K 0% /mnt/remotes df -h -t tmpfs Quote Filesystem Size Used Avail Use% Mounted on tmpfs 12G 0 12G 0% /dev/shm cgroup_root 8.0M 0 8.0M 0% /sys/fs/cgroup tmpfs 128M 396K 128M 1% /var/log tmpfs 1.0M 0 1.0M 0% /mnt/disks tmpfs 1.0M 0 1.0M 0% /mnt/remotes du -sh /tmp Quote 13M /tmp grep Shmem /proc/meminfo Quote Shmem: 11080544 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB ipcs -a --human Quote ------ Message Queues -------- key msqid owner perms size messages ------ Shared Memory Segments -------- key shmid owner perms size nattch status ------ Semaphore Arrays -------- key semid owner perms nsems Quote root@Tower:/var/log# sync root@Tower:/var/log# echo 3 > /proc/sys/vm/drop_caches root@Tower:/var/log# free -m total used free shared buff/cache available Mem: 24033 2760 9630 10916 11642 9981 Swap: 0 0 0 echo 3 | sudo tee /proc/sys/vm/drop_caches - No result. root@Tower:/var/log# dmesg | grep oom-killer Quote [92915.097680] qemu-system-x86 invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0 root@Tower:/var/log# free && sync && echo 3 > /proc/sys/vm/drop_caches && free Quote total used free shared buff/cache available Mem: 24610412 3139644 8825028 11195404 12645740 9878800 Swap: 0 0 0 total used free shared buff/cache available Mem: 24610412 3138524 9896896 11195404 11574992 9893720 Swap: 0 0 0 After reboot: Quote root@Tower:~# free -h total used free shared buff/cache available Mem: 23Gi 2.1Gi 16Gi 1.9Gi 5.4Gi 19Gi Swap: 0B 0B 0B Edited April 17, 2021 by max007 Quote Link to comment
John_M Posted April 17, 2021 Share Posted April 17, 2021 Have you tried safe mode? Post your diagnostics. Quote Link to comment
max007 Posted April 18, 2021 Author Share Posted April 18, 2021 (edited) I think I might have solved it. I think it was tvheadend docker saving recordings into nonexisting folder which was somehow accumulating in RAM. I will keep an eye on it and update this post when something new comes up. Edited April 18, 2021 by max007 Quote Link to comment
John_M Posted April 18, 2021 Share Posted April 18, 2021 That's very likely. Most of the Unraid file system is in RAM. Only paths like /boot (where the boot flash device is mounted) and specific subdirectories of /mnt are not in RAM so writes to a nonexistent folder are very likely to end up in RAM. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.