zoylendt

Since I could not resart the server via SSH or the WebUI, I decided to force a shutdown via the power button. Luckily I was able to stop the array (via WebUI) before that. It looks like all but two files from /var/lib/nginx/client_body were deleted (or rather 'not preserved'), so the issue is 'fixed' - for now. Maybe a clean reinstall will prevent this Nginx problems in the future? root@Datengrab:~# df -h Filesystem Size Used Avail Use% Mounted on rootfs 7.6G 2.0G 5.6G 27% / tmpfs 32M 676K 32M 3% /run /dev/sda1 29G 1001M 28G 4% /boot overlay 7.6G 2.0G 5.6G 27% /lib/firmware overlay 7.6G 2.0G 5.6G 27% /lib/modules devtmpfs 8.0M 0 8.0M 0% /dev tmpfs 7.7G 0 7.7G 0% /dev/shm cgroup_root 8.0M 0 8.0M 0% /sys/fs/cgroup tmpfs 128M 264K 128M 1% /var/log tmpfs 1.6G 0 1.6G 0% /run/user/0 root@Datengrab:~# du -h --max-depth=1 /var/lib/nginx | sort -n 0 /var/lib/nginx/fastcgi 0 /var/lib/nginx/proxy 0 /var/lib/nginx/scgi 0 /var/lib/nginx/uwsgi 40K /var/lib/nginx 40K /var/lib/nginx/client_body root@Datengrab:~# cd /var/lib/nginx/client_body root@Datengrab:/var/lib/nginx/client_body# ls -alhU total 40K drwx------ 2 root root 80 Aug 2 20:33 ./ drwx------ 7 root root 140 Aug 2 20:24 ../ -rw------- 1 root root 21K Aug 2 20:33 0000000407 -rw------- 1 root root 13K Aug 2 20:33 0000000403

Hi everyone, today at midnight my monthly scheduled parity check started, which usually takes ~2,5 days (9 drives + 1 parity, max HDD size 12TB). Just 8 hours later I recieved this mail: Followed by several mails, claiming the parity check started again, finished successfully (after claiming to have run for 2 days, again), over and over, about quater-hourly until I stopped and disabled the scheduled parity check. Some months ago I had a problem with the /var/log partition getting filled (caused by nginx logs, fixed by temporarily increasing the partition size), but this time it appears to be slightly different. The Web interface was nearly unusable (elements popping in and out, missing informations, not all drives listed, array falsely marked as offline, no cache SSDs detected, reboot and diagnostic tool not working, etc ...) so I searched via SSH for the root cause. To me it looks like that for some reason the directory /var/lib/nginx/client_body has been filling up, probably by the nginx process? The rootfs is completely filled, all ~8GB (the system has 16GB RAM installed). I would appreciate help of any kind, since I myself have no idea how to fix this issue. Terminal outputs of my debugging attempts: (Array running): root@Datengrab:~# df -h Filesystem Size Used Avail Use% Mounted on rootfs 7.6G 7.6G 0 100% / tmpfs 32M 1.6M 31M 5% /run /dev/sda1 29G 1001M 28G 4% /boot overlay 7.6G 7.6G 0 100% /lib/firmware overlay 7.6G 7.6G 0 100% /lib/modules devtmpfs 8.0M 0 8.0M 0% /dev tmpfs 7.7G 0 7.7G 0% /dev/shm cgroup_root 8.0M 0 8.0M 0% /sys/fs/cgroup tmpfs 512M 343M 170M 67% /var/log /dev/mapper/md1 3.7T 3.5T 219G 95% /mnt/disk1 /dev/mapper/md2 3.7T 3.5T 219G 95% /mnt/disk2 /dev/mapper/md3 7.3T 7.0T 287G 97% /mnt/disk3 /dev/mapper/md4 7.3T 7.1T 273G 97% /mnt/disk4 /dev/mapper/md5 11T 11T 381G 97% /mnt/disk5 /dev/mapper/md6 11T 11T 273G 98% /mnt/disk6 /dev/mapper/md7 11T 11T 439G 97% /mnt/disk7 /dev/mapper/md8 11T 11T 464G 96% /mnt/disk8 /dev/mapper/sdc1 224G 43G 180G 20% /mnt/cache shfs 73T 66T 7.2T 91% /mnt/user0 shfs 73T 66T 7.2T 91% /mnt/user /dev/loop2 30G 16G 14G 55% /var/lib/docker /dev/loop3 1.0G 4.1M 905M 1% /etc/libvirt /dev/mapper/md9 7.3T 2.7T 4.7T 37% /mnt/disk9 tmpfs 1.6G 0 1.6G 0% /run/user/0 (Array stopped): root@Datengrab:/# df -h Filesystem Size Used Avail Use% Mounted on rootfs 7.6G 7.6G 4.0K 100% / tmpfs 32M 680K 32M 3% /run /dev/sda1 29G 1001M 28G 4% /boot overlay 7.6G 7.6G 4.0K 100% /lib/firmware overlay 7.6G 7.6G 4.0K 100% /lib/modules devtmpfs 8.0M 0 8.0M 0% /dev tmpfs 7.7G 0 7.7G 0% /dev/shm cgroup_root 8.0M 0 8.0M 0% /sys/fs/cgroup tmpfs 512M 343M 170M 67% /var/log tmpfs 1.6G 0 1.6G 0% /run/user/0 (search biggest files in /): root@Datengrab:/# find / -xdev -type f -size +100M -exec ls -lha {} \; | sort -nk 5 -rwxr-xr-x 1 root root 113M Dec 9 2022 /usr/lib64/libnvidia-rtcore.so.525.60.13 -rwxr-xr-x 1 root root 192M Dec 9 2022 /usr/lib64/libnvoptix.so.525.60.13 (search biggest folders in /): root@Datengrab:/# du -h --max-depth=1 / 2>/dev/null | sort -n 0 /dev 0 /home 0 /hugetlbfs 0 /mnt 0 /opt 0 /proc 0 /sys 1.8G /usr 4.0K /--help 6.0G /var 9.2G / 12M /bin 15M /etc 16K /root 23M /sbin 30M /lib64 34M /tmp 349M /lib 680K /run 1001M /boot (search biggest folders in /var/lib/nginx): root@Datengrab:/# du -h --max-depth=1 /var/lib/nginx | sort -n 0 /var/lib/nginx/fastcgi 0 /var/lib/nginx/proxy 0 /var/lib/nginx/scgi 0 /var/lib/nginx/uwsgi 5.6G /var/lib/nginx 5.6G /var/lib/nginx/client_body (# of files in /var/lib/nginx/client_body): root@Datengrab:/var/lib/nginx/client_body# ls | wc -l 362016 (# of directories in /var/lib/nginx/client_body): root@Datengrab:/var/lib/nginx/client_body# tree -d 0 directories (example of files in /var/lib/nginx/client_body): root@Datengrab:/var/lib/nginx/client_body# ls -alhU | head -10 total 5.6G drwx------ 2 root root 7.0M Aug 1 10:49 ./ drwx------ 7 root root 140 Apr 30 01:33 ../ -rw------- 1 root root 0 Aug 1 10:49 0007547602 -rw------- 1 root root 0 Aug 1 10:49 0007547600 -rw------- 1 root root 4.0K Aug 1 10:49 0007547594 -rw------- 1 root root 0 Aug 1 10:49 0007547593 -rw------- 1 root root 12K Aug 1 10:49 0007547592 -rw------- 1 root root 4.0K Aug 1 10:49 0007547591 -rw------- 1 root root 0 Aug 1 10:49 0007547590