Fairly new Unraid server been running fine for 4 months as of this morning the server was locked up and required a hard restart.  Been trouble shooting all day,  the system locks up within 10 minutes of starting the array does not matter if its in safe mode or not  the GUI is accessible for hours until the array is started. I've done the following test:

 

1.) 5 hours of Mem testing - OK

2.) replaced my USB Unraid thumb drive.  fresh install and moved over the config files.

3.) completed test on all 8 of my NVME m.2 - all passed.

4.) disabled the docker - still crashes.

5.) Ran in Safe Mode. - still crashes

 

I also can run the syslog and I get the following messages for the system locks up.

 

May  4 17:37:43 FastNas kernel: BUG: unable to handle page fault for address: 000000000001cb80
May  4 17:37:43 FastNas kernel: #PF: supervisor read access in kernel mode
May  4 17:37:43 FastNas kernel: #PF: error_code(0x0000) - not-present page
May  4 17:37:43 FastNas kernel: PGD 5f964e067 P4D 5f964e067 PUD 8d3635067 PMD 0 
May  4 17:37:43 FastNas kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI
May  4 17:37:43 FastNas kernel: CPU: 8 PID: 12186 Comm: zfs Tainted: P           O       6.1.79-Unraid #1
May  4 17:37:43 FastNas kernel: Hardware name: To Be Filled By O.E.M. Z690D4U-2L2T/G5/Z690D4U-2L2T/G5, BIOS 21.01 12/19/2023
May  4 17:37:43 FastNas kernel: RIP: 0010:mutex_lock+0x11/0x2e
May  4 17:37:43 FastNas kernel: Code: eb c9 31 c0 5a c3 cc cc cc cc 0f 1f 44 00 00 be 02 00 00 00 e9 88 fc ff ff 0f 1f 44 00 00 51 48 89 3c 24 e8 13 f3 ff ff 31 c0 <48> 8b 3c 24 65 48 8b 14 25 80 cb 01 00 f0 48 0f b1 17 74 03 5a eb
May  4 17:37:43 FastNas kernel: RSP: 0018:ffffc9005927f450 EFLAGS: 00010246
May  4 17:37:43 FastNas kernel: RAX: 0000000000000000 RBX: 0000000000009c90 RCX: a9cafb63649cba82
May  4 17:37:43 FastNas kernel: RDX: 0000000080000000 RSI: 2348ce917e0a730c RDI: ffff8881734cac28
May  4 17:37:43 FastNas kernel: RBP: 0000000000000343 R08: 9ae16a3b2f90408f R09: 9ae16a3b2f90404f
May  4 17:37:43 FastNas kernel: R10: 0000000000000000 R11: 0000008000000000 R12: ffff8881734cab80
May  4 17:37:43 FastNas kernel: R13: 0000000000000000 R14: 0000000000000fa8 R15: 0000000000000000
May  4 17:37:43 FastNas kernel: FS:  000014ae2982e800(0000) GS:ffff889fff200000(0000) knlGS:0000000000000000
May  4 17:37:43 FastNas kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May  4 17:37:43 FastNas kernel: CR2: 000000000001cb80 CR3: 000000064c164000 CR4: 0000000000752ee0
May  4 17:37:43 FastNas kernel: PKRU: 55555554
May  4 17:37:43 FastNas kernel: Call Trace:
May  4 17:37:43 FastNas kernel: <TASK>
May  4 17:37:43 FastNas kernel: ? __die_body+0x1a/0x5c
May  4 17:37:43 FastNas kernel: ? page_fault_oops+0x329/0x376
May  4 17:37:43 FastNas kernel: ? do_user_addr_fault+0x12e/0x48d
May  4 17:37:43 FastNas kernel: ? exc_page_fault+0xfb/0x11d
May  4 17:37:43 FastNas kernel: ? asm_exc_page_fault+0x22/0x30
May  4 17:37:43 FastNas kernel: ? mutex_lock+0x11/0x2e
May  4 17:37:43 FastNas kernel: dbuf_find+0xac/0x146 [zfs]
May  4 17:37:43 FastNas kernel: dbuf_hold_impl+0x66/0x558 [zfs]
May  4 17:37:43 FastNas kernel: dbuf_hold_level+0x2b/0x57 [zfs]
May  4 17:37:43 FastNas kernel: dnode_hold_impl+0x159/0x9a9 [zfs]
May  4 17:37:43 FastNas kernel: ? kmem_cache_free+0xc9/0x154
May  4 17:37:43 FastNas kernel: ? slab_free_freelist_hook.constprop.0+0x3b/0xaf
May  4 17:37:43 FastNas kernel: ? memcg_slab_free_hook+0x20/0xcf
May  4 17:37:43 FastNas kernel: ? slab_free_freelist_hook.constprop.0+0x3b/0xaf
May  4 17:37:43 FastNas kernel: dmu_buf_hold_noread+0x37/0xf9 [zfs]
May  4 17:37:43 FastNas kernel: dmu_buf_hold+0x38/0x75 [zfs]
May  4 17:37:43 FastNas kernel: zap_lockdir+0x4e/0xaf [zfs]
May  4 17:37:43 FastNas kernel: ? _raw_spin_lock+0x13/0x1c
May  4 17:37:43 FastNas kernel: zap_lookup_norm+0x5a/0xcb [zfs]
May  4 17:37:43 FastNas kernel: zap_contains+0x1a/0x2f [zfs]
May  4 17:37:43 FastNas kernel: dsl_dataset_hold_obj+0x2f3/0x811 [zfs]
May  4 17:37:43 FastNas kernel: ? aggsum_add+0x6d/0x153 [zfs]
May  4 17:37:43 FastNas kernel: dsl_dataset_hold_obj+0x423/0x811 [zfs]
May  4 17:37:43 FastNas kernel: ? try_to_wake_up+0x20e/0x248
May  4 17:37:43 FastNas kernel: ? luaS_newlstr+0x72/0x15c [zlua]
May  4 17:37:43 FastNas kernel: zcp_snapshots_iter+0x69/0x1c7 [zfs]
May  4 17:37:43 FastNas kernel: ? spl_kvmalloc+0x55/0x8a [spl]
May  4 17:37:43 FastNas kernel: ? __kmalloc_node+0x9f/0xb1
May  4 17:37:43 FastNas kernel: ? spl_kvmalloc+0x55/0x8a [spl]
May  4 17:37:43 FastNas kernel: ? spl_kmem_alloc_impl+0xb2/0xf2 [spl]
May  4 17:37:43 FastNas kernel: ? zcp_lua_alloc+0x64/0xdf [zfs]
May  4 17:37:43 FastNas kernel: ? luaM_realloc_+0x32/0x94 [zlua]
May  4 17:37:43 FastNas kernel: ? luaC_newobj+0x2a/0x58 [zlua]
May  4 17:37:43 FastNas kernel: ? zcp_user_props_iter+0xe9/0xe9 [zfs]
May  4 17:37:43 FastNas kernel: luaD_precall+0xcd/0x30e [zlua]
May  4 17:37:43 FastNas kernel: luaD_call+0xda/0x110 [zlua]
May  4 17:37:43 FastNas kernel: luaV_execute+0xe0c/0x1110 [zlua]
May  4 17:37:43 FastNas kernel: luaD_call+0xca/0x110 [zlua]
May  4 17:37:43 FastNas kernel: luaD_rawrunprotected+0x98/0xa6 [zlua]
May  4 17:37:43 FastNas kernel: ? lua_setmetatable+0xe9/0xe9 [zlua]
May  4 17:37:43 FastNas kernel: ? luaD_rawrunprotected+0x52/0xa6 [zlua]
May  4 17:37:43 FastNas kernel: luaD_pcall+0x31/0x87 [zlua]
May  4 17:37:43 FastNas kernel: lua_pcallk+0x82/0x10c [zlua]
May  4 17:37:43 FastNas kernel: zcp_eval_impl+0x104/0x40d [zfs]
May  4 17:37:43 FastNas kernel: zcp_eval+0x6fc/0x78b [zfs]
May  4 17:37:43 FastNas kernel: ? nvlist_lookup_nvpair_ei_sep+0x22d/0x33f [znvpair]
May  4 17:37:43 FastNas kernel: ? nvt_lookup_name_type.isra.0+0x44/0x6e [znvpair]
May  4 17:37:43 FastNas kernel: zfs_ioc_channel_program+0x115/0x13a [zfs]
May  4 17:37:43 FastNas kernel: zfsdev_ioctl_common+0x518/0x726 [zfs]
May  4 17:37:43 FastNas kernel: zfsdev_ioctl+0x5b/0xb4 [zfs]
May  4 17:37:43 FastNas kernel: vfs_ioctl+0x1b/0x2f
May  4 17:37:43 FastNas kernel: __do_sys_ioctl+0x52/0x78
May  4 17:37:43 FastNas kernel: do_syscall_64+0x68/0x81
May  4 17:37:43 FastNas kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
May  4 17:37:43 FastNas kernel: RIP: 0033:0x14ae29ad14e8
May  4 17:37:43 FastNas kernel: Code: 00 00 48 8d 44 24 08 48 89 54 24 e0 48 89 44 24 c0 48 8d 44 24 d0 48 89 44 24 c8 b8 10 00 00 00 c7 44 24 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 07 89 d0 c3 0f 1f 40 00 48 8b 15 f9 e8 0d
May  4 17:37:43 FastNas kernel: RSP: 002b:00007ffcfd9fcb08 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
May  4 17:37:43 FastNas kernel: RAX: ffffffffffffffda RBX: 0000000000005a48 RCX: 000014ae29ad14e8
May  4 17:37:43 FastNas kernel: RDX: 00007ffcfd9fcb30 RSI: 0000000000005a48 RDI: 000000000000000c
May  4 17:37:43 FastNas kernel: RBP: 00007ffcfda00110 R08: 00000000ffffffff R09: 0000000000000000
May  4 17:37:43 FastNas kernel: R10: 000014ae299db370 R11: 0000000000000202 R12: 00007ffcfd9fcb30
May  4 17:37:43 FastNas kernel: R13: 0000000000005a48 R14: 0000000000427b01 R15: 00007ffcfda001e8
May  4 17:37:43 FastNas kernel: </TASK>
May  4 17:37:43 FastNas kernel: Modules linked in: xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc bonding tls i40e igb intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp zfs(PO) i915 kvm_intel zunicode(PO) zzstd(O) kvm ast iosf_mbi drm_buddy drm_vram_helper drm_ttm_helper ttm zlua(O) drm_display_helper zavl(PO) drm_kms_helper icp(PO) crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 drm sha1_ssse3 aesni_intel zcommon(PO) intel_gtt crypto_simd mei_hdcp mei_pxp znvpair(PO) agpgart i2c_i801 cryptd spl(O) ipmi_ssif rapl wmi_bmof i2c_algo_bit intel_cstate i2c_smbus syscopyarea tpm_crb cdc_ether
May  4 17:37:43 FastNas kernel: nvme mei_me usbnet ahci video sysfillrect tpm_tis input_leds sysimgblt intel_uncore acpi_ipmi i2c_core joydev led_class mei libahci nvme_core mii fb_sys_fops tpm_tis_core wmi ipmi_si tpm intel_pmc_core backlight acpi_pad acpi_tad button unix [last unloaded: md_mod]
May  4 17:37:43 FastNas kernel: CR2: 000000000001cb80
May  4 17:37:43 FastNas kernel: ---[ end trace 0000000000000000 ]---
May  4 17:37:43 FastNas kernel: RIP: 0010:mutex_lock+0x11/0x2e
May  4 17:37:43 FastNas kernel: Code: eb c9 31 c0 5a c3 cc cc cc cc 0f 1f 44 00 00 be 02 00 00 00 e9 88 fc ff ff 0f 1f 44 00 00 51 48 89 3c 24 e8 13 f3 ff ff 31 c0 <48> 8b 3c 24 65 48 8b 14 25 80 cb 01 00 f0 48 0f b1 17 74 03 5a eb
May  4 17:37:43 FastNas kernel: RSP: 0018:ffffc9005927f450 EFLAGS: 00010246
May  4 17:37:43 FastNas kernel: RAX: 0000000000000000 RBX: 0000000000009c90 RCX: a9cafb63649cba82
May  4 17:37:43 FastNas kernel: RDX: 0000000080000000 RSI: 2348ce917e0a730c RDI: ffff8881734cac28
May  4 17:37:43 FastNas kernel: RBP: 0000000000000343 R08: 9ae16a3b2f90408f R09: 9ae16a3b2f90404f
May  4 17:37:43 FastNas kernel: R10: 0000000000000000 R11: 0000008000000000 R12: ffff8881734cab80
May  4 17:37:43 FastNas kernel: R13: 0000000000000000 R14: 0000000000000fa8 R15: 0000000000000000
May  4 17:37:43 FastNas kernel: FS:  000014ae2982e800(0000) GS:ffff889fff200000(0000) knlGS:0000000000000000
May  4 17:37:43 FastNas kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May  4 17:37:43 FastNas kernel: CR2: 000000000001cb80 CR3: 000000064c164000 CR4: 0000000000752ee0
May  4 17:37:43 FastNas kernel: PKRU: 55555554
May  4 17:37:43 FastNas kernel: note: zfs[12186] exited with irqs disabled

fastnas-diagnostics-20240504-2025.zip

Crash is related to zfs, it would suggest some hardware issue or a problem with that filesystem, since memtest is only definitive if it finds errors, I would retry with just a pair of DIMMs, if the same try the other pair, that will basically rule out bad RAM, or a board issue with all four DIMMs installed, if it still crashes with either pair I would backup and recreate the filesystem.