December 2, 20223 yr Had a major unraid crash today. So once more the oom memory ran while the mover ran and killed my biggest VM, it seems like whenever the fuse process is taking place it consumes a massive amount of extra virtual memory that is not visible via the normal tools to look at the memory pool. After the oom-killer had killed my VM the memory consumption while mover was running according to unraid was 30%, my VM consumes 50% of the total memory, that means 20% should have been left over. I start the VM again without waiting for mover to stop. Once i do this, the webgui instantly becomes unresponsive. I SSH into the machine and try to see whats consuming memory. I start from the heaviest docker and try to to stop it via command "docker stop name" this yelds no result, it cannot be stopped because something is not responding (dont remember exactly how the message was formulated) i try command docker kill name this yeild the exact same result as docker stop. next i try to do virtsh list -all to see if my vm is running, the command just hangs in the air, never respond, i wait over 10 minutes. next i do top and sort for whatever is consuming most virtual memory, i see that SHFS is asking for a massive amount of virtual memory so i do a kill -9 PID to free up memory process is killed but nothing really happens, webgui still completely unresponsive, no docker containers can be shut down via cli, trying to interact with virsh yields no result completely unresponsive. finally i give up, i issue command powerdown in the cli. looking at my logserver, it actually does somthing this time, it actually shutsdown all the docker containers and unmounts the disks, (why could i not shut down the dockers is my main question here). eventually it locks up trying to umount several of processes, such as umount sys/fs/cgroup target is busy umount sys/fs/cgroup/cpu target is busy umount sys/fs/cgroup/cpu/freezer target is busy umount /mnt target is busy i wait for some minutes then i hard reset using the hardware reset button. i am out of options here, my main 2 questions are: 1. why is so much virtual memory being randomly consumed by the fuse operation at random times? How can i put a cap on this? I use unraid for VM hosting and its becoming unreliable because the fuse system just eats memory at random and then oom-killer defaults to killing the process eating most memory which is always going to be my VM because it consumes 50% of the total memory when running statically. 2. any ideas why docker and virsh was both unoperationable, virsh was actually tototally unresponsive issuing any command to it yielded no response, but the strange part is docker was responding just fine but i couldn't shut down any of the containers using stop or kill, both responded in the same way that something (i cannot rememeber exactly what it said) is not responding and therefor it cannot be stopped. thanks.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.