6.12.4 caused constant read on boot flash drive

[Canaan]

I have suffered from this occasionally, the webpage and ssh got almost frozen. 

I can see costant read on the boot flash drive and it can last for hours.

 

inotifywatch -r -t 30 /boot
Establishing watches...
Finished establishing watches, now collecting statistics.
total  access  close_nowrite  open  filename
28699  28641   29             29    /boot/
408    168     120            120   /boot/config/
219    73      73             73    /boot/config/plugins/dynamix/
24     12      6              6     /boot/config/plugins/dynamix/notifications/agents/
24     12      6              6     /boot/config/plugins/dynamix/notifications/
 

Last time I made a post here, but maybe no one noticed it. Can someone help me solve this?

 

tower-diagnostics-20231110-0058.zip

[Canaan]

Can anyone help me on this? I can only reboot the server to avoid this.

[JorgeB]

Try booting in safe mode.

[zachzaf]

Did you by any chance manage to solve this? I seem to have a very similar issue. For some reason docker seems to spike to 100% cpu usage very frequently and it can last for hours. The docker containers and the VMs seem to be working ok (albeit with a noticeable slowdown), but the web gui and ssh become unresponsive.

 

After some digging in this forum and netdata I found that the bottleneck seems to be iowait (up to 40-50% when this happens) and it seems to be connected to the flash drive being read constantly:

 

 

# inotifywatch -r -t 30 /boot

Establishing watches...
Finished establishing watches, now collecting statistics.
total  access  close_nowrite  open  filename
88538  88493   23             22    /boot/
897    299     299            299   /boot/config/shares/
296    114     92             90    /boot/config/

 

Any ideas?