Tried turning the computer off with the power button which took down docker but didn't complete shutting down.

I was able to ssh in and

/etc/rc.d/rc.php-fpm restart

allowed me to reach the gui again.
I saw the server was prevented from shutting down because it ran into trouble shutting down the array. Shutting down cache drive around 18:41

Couldn't get the diagnostics because it was freezing at nvidia-smi

Gui went unresponsive a few times while navigating the server. the php restart got it back every time. Ultimately the restart button in the gui got it to restart.

 

Looking at the syslogs
 

Oct 20 07:47:36 Monolith webGUI: Successful login user root from 209.107.189.2
[...]
Oct 20 07:48:47 Monolith  php-fpm[5983]: [WARNING] [pool www] server reached max_children setting (50), consider raising it

 

At 07:47 was the last time I was able to log in. The other line I included above I wonder if it is the problem. Not sure why I'm hitting so many max children.
 

Oct 20 20:18:48 Monolith kernel: _swap_info_get: Bad swap file entry 3ffffffffffbf
Oct 20 20:18:48 Monolith kernel: BUG: Bad page map in process php-fpm  pte:00008000 pmd:1a6f38067
Oct 20 20:18:48 Monolith kernel: addr:0000154493f33000 vm_flags:00000075 anon_vma:0000000000000000 mapping:ffff888107c6f3e8 index:ee
Oct 20 20:18:48 Monolith kernel: file:libc-2.36.so fault:shmem_fault mmap:shmem_mmap read_folio:0x0
Oct 20 20:18:48 Monolith kernel: CPU: 0 PID: 29542 Comm: php-fpm Tainted: P      D W  O      5.19.14-Unraid #1
Oct 20 20:18:48 Monolith kernel: Hardware name: Gigabyte Technology Co., Ltd. Z390 AORUS PRO WIFI/Z390 AORUS PRO WIFI-CF, BIOS F12 11/05/2021

 

This bit also looked suspicious

 

 

And then apologies for the end of the syslogs I had to reboot a few times to get access and download them and diagnostics (different issue for another time)

monolith-diagnostics-20221020-2114.zip syslog

You know what, I think I also sent my syslogs to a share on the cache drive which might be why it wouldn't go offline from the physical power button

Problem appears to start with a Nvidia driver crash:

 

Oct 20 05:00:18 Monolith kernel: general protection fault, probably for non-canonical address 0x841f0f2e66c3f3: 0000 [#1] PREEMPT SMP NOPTI
Oct 20 05:00:18 Monolith kernel: CPU: 2 PID: 23970 Comm: PMS Timer Tainted: P           O      5.19.14-Unraid #1
Oct 20 05:00:18 Monolith kernel: Hardware name: Gigabyte Technology Co., Ltd. Z390 AORUS PRO WIFI/Z390 AORUS PRO WIFI-CF, BIOS F12 11/05/2021
Oct 20 05:00:18 Monolith kernel: RIP: 0010:_nv027480rm+0x12/0x20 [nvidia]
Oct 20 05:00:18 Monolith kernel: Code: f6 b8 01 00 00 00 89 d1 d3 e0 09 04 b7 c3 66 2e 0f 1f 84 00 00 00 00 00 48 8b bf 80 01 00 00 89 f6 b8 fe ff ff ff 89 d1 d3 c0 <21> 04 b7 c3 66 2e 0f 1f 84 00 00 00 00 00 48 8b bf 80 01 00 00 89
Oct 20 05:00:18 Monolith kernel: RSP: 0018:ffffc900037df8d0 EFLAGS: 00010246
Oct 20 05:00:18 Monolith kernel: RAX: 00000000fffffffe RBX: ffff8881ab1b0008 RCX: 0000000000000000
Oct 20 05:00:18 Monolith kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00841f0f2e66c3f3
Oct 20 05:00:18 Monolith kernel: RBP: ffff88814fd45b90 R08: 0000000000000020 R09: ffff88814fd45b78
Oct 20 05:00:18 Monolith kernel: R10: ffff8881ab1b0008 R11: 00000001010f0000 R12: ffff8881ab1b0008
Oct 20 05:00:18 Monolith kernel: R13: ffff8881ab1b0008 R14: 0000000000000000 R15: 0000000000000001
Oct 20 05:00:18 Monolith kernel: FS:  0000000000000000(0000) GS:ffff88889dc80000(0000) knlGS:0000000000000000
Oct 20 05:00:18 Monolith kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 20 05:00:18 Monolith kernel: CR2: 000014e33580ff1c CR3: 000000000200a006 CR4: 00000000003706e0
Oct 20 05:00:18 Monolith kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 20 05:00:18 Monolith kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Oct 20 05:00:18 Monolith kernel: Call Trace:
Oct 20 05:00:18 Monolith kernel: <TASK>
Oct 20 05:00:18 Monolith kernel: ? _nv019327rm+0x8b/0x100 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? _nv019776rm+0x340/0x7f0 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? _nv019777rm+0xda/0x1e0 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? _nv019777rm+0xb8/0x1e0 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? _nv013890rm+0x438/0x920 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? _nv013890rm+0x418/0x920 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? _nv013878rm+0x78/0xf0 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? _nv014058rm+0x196/0x7f0 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? _nv035922rm+0xac/0xe0 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? _nv037394rm+0xac/0x140 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? _nv037393rm+0x2f7/0x4d0 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? _nv035835rm+0xbe/0x140 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? _nv035836rm+0x42/0x70 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? _nv031339rm+0x113/0x230 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? rm_gpu_ops_channel_destroy+0x1c/0x60 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? nvUvmGetSafeStack+0x42/0x92 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? nvUvmInterfaceChannelDestroy+0x1e/0x29 [nvidia]
Oct 20 05:00:18 Monolith kernel: ? channel_destroy+0x89/0xc0 [nvidia_uvm]
Oct 20 05:00:18 Monolith kernel: ? uvm_channel_manager_destroy.part.0+0x4c/0xa0 [nvidia_uvm]
Oct 20 05:00:18 Monolith kernel: ? remove_gpu+0x183/0x3a0 [nvidia_uvm]
Oct 20 05:00:18 Monolith kernel: ? uvm_gpu_release_locked+0x1f/0x40 [nvidia_uvm]
Oct 20 05:00:18 Monolith kernel: ? uvm_va_space_destroy+0x45f/0x4c0 [nvidia_uvm]
Oct 20 05:00:18 Monolith kernel: ? uvm_release.constprop.0+0x3d/0xa0 [nvidia_uvm]
Oct 20 05:00:18 Monolith kernel: ? uvm_release_entry.part.0.isra.0+0x7a/0xb0 [nvidia_uvm]
Oct 20 05:00:18 Monolith kernel: ? __fput+0x101/0x1c8
Oct 20 05:00:18 Monolith kernel: ? task_work_run+0x66/0x7e
Oct 20 05:00:18 Monolith kernel: ? do_exit+0x39f/0x8e5
Oct 20 05:00:18 Monolith kernel: ? _raw_spin_unlock_irqrestore+0x24/0x3a
Oct 20 05:00:18 Monolith kernel: ? do_group_exit+0x8f/0x8f
Oct 20 05:00:18 Monolith kernel: ? get_signal+0x606/0x63e
Oct 20 05:00:18 Monolith kernel: ? hrtimer_init_sleeper+0x41/0x41
Oct 20 05:00:18 Monolith kernel: ? arch_do_signal_or_restart+0x36/0x607
Oct 20 05:00:18 Monolith kernel: ? do_futex+0xcd/0x143
Oct 20 05:00:18 Monolith kernel: ? exit_to_user_mode_prepare+0x58/0x10d
Oct 20 05:00:18 Monolith kernel: ? syscall_exit_to_user_mode+0x18/0x27
Oct 20 05:00:18 Monolith kernel: ? do_syscall_64+0x77/0x81
Oct 20 05:00:18 Monolith kernel: ? entry_SYSCALL_64_after_hwframe+0x63/0xcd
Oct 20 05:00:18 Monolith kernel: </TASK>
Oct 20 05:00:18 Monolith kernel: Modules linked in: xt_mark xt_nat veth tcp_diag udp_diag inet_diag nvidia_uvm(PO) xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs input_leds led_class joydev r8152 mii md_mod ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc ipv6 nvidia_drm(PO) nvidia_modeset(PO) x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm nvidia(PO) crct10dif_pclmul crc32_pclmul gigabyte_wmi wmi_bmof intel_wmi_thunderbolt mxm_wmi crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl drm_kms_helper intel_cstate intel_uncore nvme i2c_i801 e1000e drm thunderbolt nvme_core i2c_smbus ahci i2c_core libahci syscopyarea sysfillrect sysimgblt fb_sys_fops intel_pch_thermal thermal fan tpm_crb tpm_tis video
Oct 20 05:00:18 Monolith kernel: tpm_tis_core wmi backlight tpm acpi_pad button unix
Oct 20 05:00:18 Monolith kernel: ---[ end trace 0000000000000000 ]---

 

I'm still having issues this time no nvidia crash in the logs. Nginx keeps giving me issues.

syslog

Are you using ipv6? If not disable it.

It is disabled on my router and in my network settings. This weekend I had my longest period of uptime in a while from Friday till 4AM Monday. I'll have to pull the logs again when I get home.

Booted up windows to get the log and ran into a memory related blue screen. Running memtest64 getting errors on 4, 6 & 7.

 

So this is probably the cause of my issues.