System completely hangs/unresponsive to physical keyboard occasionally; maybe upon invoking oom-killer

[nick5429]

System: Supermicro X8DTE-F, 24GB ram, Xeon L5640
HBA330, bunch of spinning data drives, 1 out-of-pool SSD for misc tinkering and docker storage

 

At least once a week lately, my system will completely hang and be unresponsive to any input. No unraid networking page, nor even respond to physical keyboard attached. Cursor doesn't blink, "enter" doesn't insert a new console line, numlock doesn't toggle, etc.

 

This time, it happened when I was starting a VM which triggered the OOM process. Frequently other times, it will happen 'randomly' as far as I can tell, though with indications an OOM was involved.  I'm not sure this even makes a ton of sense outside of this VM-induced crash today, as generally I run no VMs and just typical collection of dockers with 24GB of RAM. `free -m` would typically show at least 1+GB completely free and ~8GB buffered/cached which could be reclaimed under memory pressure rather than killed-for.

 

Aside: I'm playing around with ZFS pools on the 'cache' mount, but is used only as a data drive/pool. I'm unconcerned about associated reported errors in the syslog. Various zfs things use 10-12GB, but nevertheless -- an OOM event should not cause a complete system hang [especially the more routine 'random' ones that aren't requesting 8GB all at once, which aren't captured here]

Diagnostics attached; syslog mirrored to flash and captured in diagnostics
Screenshot from IPMI "physical console" showing some killed when hung

 

It looks like process 'unraid-api' gets killed at some point; is that a noteworthy problem?

nickserver-diagnostics-20240911-0957.zip

[JorgeB]

Before the VM other processes got killed by the OOM reaper, including Connect and Plex, you may have a container or plugin casing this.