Tried turning the computer off with the power button which took down docker but didn't complete shutting down.
I was able to ssh in and
/etc/rc.d/rc.php-fpm restart
allowed me to reach the gui again.
I saw the server was prevented from shutting down because it ran into trouble shutting down the array. Shutting down cache drive around 18:41
Couldn't get the diagnostics because it was freezing at nvidia-smi
Gui went unresponsive a few times while navigating the server. the php restart got it back every time. Ultimately the restart button in the gui got it to restart.
Looking at the syslogs
Oct 20 07:47:36 Monolith webGUI: Successful login user root from 209.107.189.2
[...]
Oct 20 07:48:47 Monolith php-fpm[5983]: [WARNING] [pool www] server reached max_children setting (50), consider raising it
At 07:47 was the last time I was able to log in. The other line I included above I wonder if it is the problem. Not sure why I'm hitting so many max children.
Oct 20 20:18:48 Monolith kernel: _swap_info_get: Bad swap file entry 3ffffffffffbf
Oct 20 20:18:48 Monolith kernel: BUG: Bad page map in process php-fpm pte:00008000 pmd:1a6f38067
Oct 20 20:18:48 Monolith kernel: addr:0000154493f33000 vm_flags:00000075 anon_vma:0000000000000000 mapping:ffff888107c6f3e8 index:ee
Oct 20 20:18:48 Monolith kernel: file:libc-2.36.so fault:shmem_fault mmap:shmem_mmap read_folio:0x0
Oct 20 20:18:48 Monolith kernel: CPU: 0 PID: 29542 Comm: php-fpm Tainted: P D W O 5.19.14-Unraid #1
Oct 20 20:18:48 Monolith kernel: Hardware name: Gigabyte Technology Co., Ltd. Z390 AORUS PRO WIFI/Z390 AORUS PRO WIFI-CF, BIOS F12 11/05/2021
This bit also looked suspicious
And then apologies for the end of the syslogs I had to reboot a few times to get access and download them and diagnostics (different issue for another time)
monolith-diagnostics-20221020-2114.zip
syslog