14 hours ago14 hr What is this and how to fix it? When I downgrade to 7.3.2 everything is back to "normal". hal9000-diagnostics-20261023-1701.zip
14 hours ago14 hr Community Expert The diagnostics show the shfs process for /mnt/user using about two CPU cores the whole time since boot, with high system CPU. That is not normal, and I would like to see what it is doing. With the problem present, open a terminal and run:P=$(pgrep -f 'shfs /mnt/user -disks')top -H -b -n1 -p $P | head -30 > /boot/shfs-debug.txtgrep -c FLOCK /proc/locks >> /boot/shfs-debug.txtfor i in 1 2 3; do cat /proc/$P/task/*/stack >> /boot/shfs-debug.txt; echo ---- >> /boot/shfs-debug.txt; sleep 3; doneThen post the shfs-debug.txt file from your flash drive.After that, please also try these tests and say whether shfs CPU goes down after each one:1. Settings > Docker, set Enable Docker to No. If the CPU drops, start the containers again a few at a time to find which one causes it.2. NFS is enabled but you are not exporting any share. Disable it in Settings > NFS, reboot, and check again.Separately, your cache pool has a more important problem. The syslog shows Btrfs "parent transid verify failed" errors on both copies of the metadata, so a scrub cannot fix them, and the docker image also shows corruption. Back up anything important on the cache now, and plan to reformat the pool and recreate the docker image. Both MX500 SSDs are also well past their rated endurance (189% and 253% used), so consider replacing them at the same time.When you recreate it, set the system share to use the cache only, with no secondary storage, and enable Settings > Global Share Settings > Permit exclusive shares. Do the same for appdata after its files on disk3 and disk4 are moved to the cache. The docker image and appdata will then bypass shfs completely, which is better for performance anyway.
12 hours ago12 hr Author 1 hour ago, JorgeB said:open a terminal and run:Oh, you speak magic? ;)1 hour ago, JorgeB said:Then post the shfs-debug.txt file from your flash drive.doneThe other tests I will run next morning. Backing up /mnt/cache as we speak.Thanks a lot for your help in advance. shfs-debug.txt
12 hours ago12 hr Community Expert Thanks, that helps. Only a few file locks are held, so shfs is not stuck on locks. Two shfs threads are reading continuously from a file on the cache pool, and most of the CPU time is the kernel checking Btrfs checksums for those reads. So shfs is passing on a heavy, nonstop read load, most likely from the docker image or appdata.When you have the problem again, please run this to see which file is being read:P=$(pgrep -f 'shfs /mnt/user -disks')for n in $(seq 40); do for t in /proc/$P/task/*; do set -- $(cat $t/syscall 2>/dev/null); [ "$1" = 17 ] && readlink /proc/$P/fd/$(($2)); done; sleep 0.5; done | sort | uniq -c > /boot/shfs-files.txtdocker stats --no-stream >> /boot/shfs-files.txtThen post shfs-files.txt from your flash drive.After that, the Docker test from my last post is the most important one. Set Enable Docker to No in Settings > Docker and check whether shfs CPU drops. If it does, start the containers a few at a time. The BLOCK I/O column from docker stats should also help find the one doing all the reading. The NFS test is still worth doing, but it is less likely to matter now.Good call on backing up the cache first.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.