Motherboard: Gigbyte Z690 AORUS MASTER
CPU: Intel 13900k
RAM: Kingston Fury Beast 128gb (4x32) 4800MT DDR5 - set at 4000MT as enabling XMP seems to cause more issues
Unraid version: 6.12.4

Over the last few months I have been having intermittent crashes. They have ranged from nearly daily, to sometimes weekly, but each time the system just completely hangs and the server cannot be interacted with in any capacity.

I am getting the below error in the syslog every hour or so:

Quote

Nov 15 19:25:29 Valhalla kernel: BUG: kernel NULL pointer dereference, address: 0000000000000038
Nov 15 19:25:29 Valhalla kernel: #PF: supervisor read access in kernel mode
Nov 15 19:25:29 Valhalla kernel: #PF: error_code(0x0000) - not-present page
Nov 15 19:25:29 Valhalla kernel: PGD 1036e3067 P4D 1036e3067 PUD 3c52cb067 PMD 0 
Nov 15 19:25:29 Valhalla kernel: Oops: 0000 [#18] PREEMPT SMP NOPTI
Nov 15 19:25:29 Valhalla kernel: CPU: 12 PID: 25313 Comm: find Tainted: P      D    O       6.1.49-Unraid #1
Nov 15 19:25:29 Valhalla kernel: Hardware name: Gigabyte Technology Co., Ltd. Z690 AORUS MASTER/Z690 AORUS MASTER, BIOS F28b 11/09/2023
Nov 15 19:25:29 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 15 19:25:29 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 15 19:25:29 Valhalla kernel: RSP: 0018:ffffc90025867dd0 EFLAGS: 00010202
Nov 15 19:25:29 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 15 19:25:29 Valhalla kernel: RDX: ffffc90025867e20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 15 19:25:29 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 15 19:25:29 Valhalla kernel: R10: ffffc90025867e10 R11: ffffc90025867e94 R12: 0000000000000000
Nov 15 19:25:29 Valhalla kernel: R13: ffffc90025867e90 R14: ffffc90025867e20 R15: 0000000000000000
Nov 15 19:25:29 Valhalla kernel: FS:  0000152156638740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 15 19:25:29 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 15 19:25:29 Valhalla kernel: CR2: 0000000000000038 CR3: 00000005c15ec000 CR4: 0000000000752ee0
Nov 15 19:25:29 Valhalla kernel: PKRU: 55555554
Nov 15 19:25:29 Valhalla kernel: Call Trace:
Nov 15 19:25:29 Valhalla kernel: <TASK>
Nov 15 19:25:29 Valhalla kernel: ? __die_body+0x1a/0x5c
Nov 15 19:25:29 Valhalla kernel: ? page_fault_oops+0x329/0x376
Nov 15 19:25:29 Valhalla kernel: ? do_user_addr_fault+0x12e/0x48d
Nov 15 19:25:29 Valhalla kernel: ? exc_page_fault+0xfb/0x11d
Nov 15 19:25:29 Valhalla kernel: ? asm_exc_page_fault+0x22/0x30
Nov 15 19:25:29 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 15 19:25:29 Valhalla kernel: ? memcg_slab_free_hook+0x28/0xcf
Nov 15 19:25:29 Valhalla kernel: kmem_cache_free+0xb7/0x154
Nov 15 19:25:29 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 15 19:25:29 Valhalla kernel: vfs_fstatat+0x52/0x62
Nov 15 19:25:29 Valhalla kernel: __do_sys_newfstatat+0x26/0x5c
Nov 15 19:25:29 Valhalla kernel: do_syscall_64+0x68/0x81
Nov 15 19:25:29 Valhalla kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Nov 15 19:25:29 Valhalla kernel: RIP: 0033:0x15215673e1ca
Nov 15 19:25:29 Valhalla kernel: Code: 48 89 f2 b9 00 01 00 00 48 89 fe bf 9c ff ff ff e9 0b 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 90 41 89 ca b8 06 01 00 00 0f 05 <3d> 00 f0 ff ff 77 07 31 c0 c3 0f 1f 40 00 48 8b 15 19 4c 0e 00 f7
Nov 15 19:25:29 Valhalla kernel: RSP: 002b:00007fff81f83608 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
Nov 15 19:25:29 Valhalla kernel: RAX: ffffffffffffffda RBX: 0000000000479ae0 RCX: 000015215673e1ca
Nov 15 19:25:29 Valhalla kernel: RDX: 0000000000479b50 RSI: 0000000000479be0 RDI: 0000000000000008
Nov 15 19:25:29 Valhalla kernel: RBP: 0000000000479b50 R08: 00000000ffffffff R09: 000000000044a290
Nov 15 19:25:29 Valhalla kernel: R10: 0000000000000100 R11: 0000000000000246 R12: 0000000000479ae0
Nov 15 19:25:29 Valhalla kernel: R13: 0000000000000000 R14: 00007fff81f83f16 R15: 0000000000000004
Nov 15 19:25:29 Valhalla kernel: </TASK>
Nov 15 19:25:29 Valhalla kernel: Modules linked in: rpcsec_gss_krb5 xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap tun xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nct7904 watchdog coretemp xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper kvm drm_kms_helper drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel crypto_simd sr_mod btusb cryptd cdrom btrtl i2c_i801
Nov 15 19:25:29 Valhalla kernel: intel_gtt rapl btbcm btintel intel_cstate mei_hdcp mei_pxp gigabyte_wmi wmi_bmof mxm_wmi mpt3sas bluetooth intel_uncore agpgart i2c_smbus nvme atlantic raid_class i2c_core input_leds mei_me scsi_transport_sas led_class nvme_core ahci joydev mei syscopyarea ecdh_generic libahci ecc sysfillrect sysimgblt thermal fb_sys_fops fan tpm_crb tpm_tis video tpm_tis_core tpm wmi backlight intel_pmc_core acpi_tad acpi_pad button unix [last unloaded: coretemp]
Nov 15 19:25:29 Valhalla kernel: CR2: 0000000000000038
Nov 15 19:25:29 Valhalla kernel: ---[ end trace 0000000000000000 ]---
Nov 15 19:25:29 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 15 19:25:29 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 15 19:25:29 Valhalla kernel: RSP: 0018:ffffc90027b0fdd0 EFLAGS: 00010202
Nov 15 19:25:29 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 15 19:25:29 Valhalla kernel: RDX: ffffc90027b0fe20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 15 19:25:29 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 15 19:25:29 Valhalla kernel: R10: ffffc90027b0fe10 R11: ffffc90027b0fe94 R12: 0000000000000000
Nov 15 19:25:29 Valhalla kernel: R13: ffffc90027b0fe90 R14: ffffc90027b0fe20 R15: 0000000000000000
Nov 15 19:25:29 Valhalla kernel: FS:  0000152156638740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 15 19:25:29 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 15 19:25:29 Valhalla kernel: CR2: 0000000000000038 CR3: 00000005c15ec000 CR4: 0000000000752ee0
Nov 15 19:25:29 Valhalla kernel: PKRU: 55555554
Nov 15 19:25:29 Valhalla kernel: note: find[25313] exited with irqs disabled

 

I have searched for other cases of similar errors and I've seen a few common "causes", i.e. docker set to macvlan, and docker containers that use libtorrent 2.x.  I don't have any torrent containers at the moment (used to have deluge but that got removed a long time ago). I've switched to ipvlan in docker settings.

One thing to call out though is that I have containers like radarr/sonarr routed through a gluetunVPN container and Rebuild-DNDC that rebuilds the containers whenever the vpn endpoint changes. I currently have all these turned off whilst I try to diagnose the issue I'm having.

I've run memtest86 and it didn't report any issues (unless I turn on XMP and run it, then it has a lot of failures)

At this point I'm at a loss for what to do, other than trying different RAM, but thought I'd post here before purchasing any new hardware.

Let me know if you need any further details.

Hope someone have help out here. Thanks!

 

valhalla-diagnostics-20231115-2043.zip

Edited November 15, 20232 yr by humaintenance

It crashed again last night at about 04:22

This is the last thing written to syslog before it crashed:

 

Quote

Nov 16 04:08:29 Valhalla kernel: BUG: kernel NULL pointer dereference, address: 0000000000000038
Nov 16 04:08:29 Valhalla kernel: #PF: supervisor read access in kernel mode
Nov 16 04:08:29 Valhalla kernel: #PF: error_code(0x0000) - not-present page
Nov 16 04:08:29 Valhalla kernel: PGD 5d8332067 P4D 5d8332067 PUD 953adf067 PMD 0 
Nov 16 04:08:29 Valhalla kernel: Oops: 0000 [#63] PREEMPT SMP NOPTI
Nov 16 04:08:29 Valhalla kernel: CPU: 12 PID: 27164 Comm: find Tainted: P      D    O       6.1.49-Unraid #1
Nov 16 04:08:29 Valhalla kernel: Hardware name: Gigabyte Technology Co., Ltd. Z690 AORUS MASTER/Z690 AORUS MASTER, BIOS F28b 11/09/2023
Nov 16 04:08:29 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 04:08:29 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 04:08:29 Valhalla kernel: RSP: 0018:ffffc900218c7dd0 EFLAGS: 00010202
Nov 16 04:08:29 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 04:08:29 Valhalla kernel: RDX: ffffc900218c7e20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 04:08:29 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 04:08:29 Valhalla kernel: R10: ffffc900218c7e10 R11: ffffc900218c7e94 R12: 0000000000000000
Nov 16 04:08:29 Valhalla kernel: R13: ffffc900218c7e90 R14: ffffc900218c7e20 R15: 0000000000000000
Nov 16 04:08:29 Valhalla kernel: FS:  0000153ad169f740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 04:08:29 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 04:08:29 Valhalla kernel: CR2: 0000000000000038 CR3: 00000008cc374000 CR4: 0000000000752ee0
Nov 16 04:08:29 Valhalla kernel: PKRU: 55555554
Nov 16 04:08:29 Valhalla kernel: Call Trace:
Nov 16 04:08:29 Valhalla kernel: <TASK>
Nov 16 04:08:29 Valhalla kernel: ? __die_body+0x1a/0x5c
Nov 16 04:08:29 Valhalla kernel: ? page_fault_oops+0x329/0x376
Nov 16 04:08:29 Valhalla kernel: ? do_user_addr_fault+0x12e/0x48d
Nov 16 04:08:29 Valhalla kernel: ? exc_page_fault+0xfb/0x11d
Nov 16 04:08:29 Valhalla kernel: ? asm_exc_page_fault+0x22/0x30
Nov 16 04:08:29 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 04:08:29 Valhalla kernel: ? memcg_slab_free_hook+0x28/0xcf
Nov 16 04:08:29 Valhalla kernel: kmem_cache_free+0xb7/0x154
Nov 16 04:08:29 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 04:08:29 Valhalla kernel: vfs_fstatat+0x52/0x62
Nov 16 04:08:29 Valhalla kernel: __do_sys_newfstatat+0x26/0x5c
Nov 16 04:08:29 Valhalla kernel: do_syscall_64+0x68/0x81
Nov 16 04:08:29 Valhalla kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Nov 16 04:08:29 Valhalla kernel: RIP: 0033:0x153ad17a51ca
Nov 16 04:08:29 Valhalla kernel: Code: 48 89 f2 b9 00 01 00 00 48 89 fe bf 9c ff ff ff e9 0b 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 90 41 89 ca b8 06 01 00 00 0f 05 <3d> 00 f0 ff ff 77 07 31 c0 c3 0f 1f 40 00 48 8b 15 19 4c 0e 00 f7
Nov 16 04:08:29 Valhalla kernel: RSP: 002b:00007ffd58c11848 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
Nov 16 04:08:29 Valhalla kernel: RAX: ffffffffffffffda RBX: 000000000044e650 RCX: 0000153ad17a51ca
Nov 16 04:08:29 Valhalla kernel: RDX: 000000000044e6c0 RSI: 000000000044e750 RDI: 000000000000000a
Nov 16 04:08:29 Valhalla kernel: RBP: 000000000044e6c0 R08: 00000000ffffffff R09: 00000000004960d0
Nov 16 04:08:29 Valhalla kernel: R10: 0000000000000100 R11: 0000000000000246 R12: 000000000044e650
Nov 16 04:08:29 Valhalla kernel: R13: 0000000000000000 R14: 00007ffd58c12f15 R15: 0000000000000004
Nov 16 04:08:29 Valhalla kernel: </TASK>
Nov 16 04:08:29 Valhalla kernel: Modules linked in: rpcsec_gss_krb5 xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap tun xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nct7904 watchdog coretemp xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper kvm drm_kms_helper drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel crypto_simd sr_mod btusb cryptd cdrom btrtl i2c_i801
Nov 16 04:08:29 Valhalla kernel: intel_gtt rapl btbcm btintel intel_cstate mei_hdcp mei_pxp gigabyte_wmi wmi_bmof mxm_wmi mpt3sas bluetooth intel_uncore agpgart i2c_smbus nvme atlantic raid_class i2c_core input_leds mei_me scsi_transport_sas led_class nvme_core ahci joydev mei syscopyarea ecdh_generic libahci ecc sysfillrect sysimgblt thermal fb_sys_fops fan tpm_crb tpm_tis video tpm_tis_core tpm wmi backlight intel_pmc_core acpi_tad acpi_pad button unix [last unloaded: coretemp]
Nov 16 04:08:29 Valhalla kernel: CR2: 0000000000000038
Nov 16 04:08:29 Valhalla kernel: ---[ end trace 0000000000000000 ]---
Nov 16 04:08:29 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 04:08:29 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 04:08:29 Valhalla kernel: RSP: 0018:ffffc90027b0fdd0 EFLAGS: 00010202
Nov 16 04:08:29 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 04:08:29 Valhalla kernel: RDX: ffffc90027b0fe20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 04:08:29 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 04:08:29 Valhalla kernel: R10: ffffc90027b0fe10 R11: ffffc90027b0fe94 R12: 0000000000000000
Nov 16 04:08:29 Valhalla kernel: R13: ffffc90027b0fe90 R14: ffffc90027b0fe20 R15: 0000000000000000
Nov 16 04:08:29 Valhalla kernel: FS:  0000153ad169f740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 04:08:29 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 04:08:29 Valhalla kernel: CR2: 0000000000000038 CR3: 00000008cc374000 CR4: 0000000000752ee0
Nov 16 04:08:29 Valhalla kernel: PKRU: 55555554
Nov 16 04:08:29 Valhalla kernel: note: find[27164] exited with irqs disabled

 

Edited November 16, 20232 yr by humaintenance

You can enable the syslog server and post that after a crash in case there's something else that helps, but those look to me more hardware related.

Already had the syslog server enabled. It crashed again a little earlier so here is the syslog from the error that was thrown before the crash up to and including the entire boot sequence following the hard reset:

 

Quote

Nov 16 14:55:32 Valhalla kernel: smartctl_type[32706]: segfault at 201 ip 0000000000000201 sp 00007fff29b27428 error 14 in php[400000+3b000] likely on CPU 13 (core 24, socket 0)
Nov 16 14:55:32 Valhalla kernel: Code: Unable to access opcode bytes at 0x1d7.
Nov 16 14:56:48 Valhalla kernel: BUG: kernel NULL pointer dereference, address: 0000000000000038
Nov 16 14:56:48 Valhalla kernel: #PF: supervisor read access in kernel mode
Nov 16 14:56:48 Valhalla kernel: #PF: error_code(0x0000) - not-present page
Nov 16 14:56:48 Valhalla kernel: PGD 141a9b067 P4D 141a9b067 PUD 25e2ca067 PMD 0 
Nov 16 14:56:48 Valhalla kernel: Oops: 0000 [#37] PREEMPT SMP NOPTI
Nov 16 14:56:48 Valhalla kernel: CPU: 12 PID: 3276 Comm: find Tainted: P      D    O       6.1.49-Unraid #1
Nov 16 14:56:48 Valhalla kernel: Hardware name: Gigabyte Technology Co., Ltd. Z690 AORUS MASTER/Z690 AORUS MASTER, BIOS F28b 11/09/2023
Nov 16 14:56:48 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:56:48 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:56:48 Valhalla kernel: RSP: 0018:ffffc9002b623dd0 EFLAGS: 00010202
Nov 16 14:56:48 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:56:48 Valhalla kernel: RDX: ffffc9002b623e20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:56:48 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:56:48 Valhalla kernel: R10: ffffc9002b623e10 R11: ffffc9002b623e94 R12: 0000000000000000
Nov 16 14:56:48 Valhalla kernel: R13: ffffc9002b623e90 R14: ffffc9002b623e20 R15: 0000000000000000
Nov 16 14:56:48 Valhalla kernel: FS:  000015151aaed740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:56:48 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:56:48 Valhalla kernel: CR2: 0000000000000038 CR3: 000000054464c000 CR4: 0000000000752ee0
Nov 16 14:56:48 Valhalla kernel: PKRU: 55555554
Nov 16 14:56:48 Valhalla kernel: Call Trace:
Nov 16 14:56:48 Valhalla kernel: <TASK>
Nov 16 14:56:48 Valhalla kernel: ? __die_body+0x1a/0x5c
Nov 16 14:56:48 Valhalla kernel: ? page_fault_oops+0x329/0x376
Nov 16 14:56:48 Valhalla kernel: ? do_user_addr_fault+0x12e/0x48d
Nov 16 14:56:48 Valhalla kernel: ? exc_page_fault+0xfb/0x11d
Nov 16 14:56:48 Valhalla kernel: ? asm_exc_page_fault+0x22/0x30
Nov 16 14:56:48 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:56:48 Valhalla kernel: ? memcg_slab_free_hook+0x28/0xcf
Nov 16 14:56:48 Valhalla kernel: kmem_cache_free+0xb7/0x154
Nov 16 14:56:48 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:56:48 Valhalla kernel: vfs_fstatat+0x52/0x62
Nov 16 14:56:48 Valhalla kernel: __do_sys_newfstatat+0x26/0x5c
Nov 16 14:56:48 Valhalla kernel: do_syscall_64+0x68/0x81
Nov 16 14:56:48 Valhalla kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Nov 16 14:56:48 Valhalla kernel: RIP: 0033:0x15151abf31ca
Nov 16 14:56:48 Valhalla kernel: Code: 48 89 f2 b9 00 01 00 00 48 89 fe bf 9c ff ff ff e9 0b 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 90 41 89 ca b8 06 01 00 00 0f 05 <3d> 00 f0 ff ff 77 07 31 c0 c3 0f 1f 40 00 48 8b 15 19 4c 0e 00 f7
Nov 16 14:56:48 Valhalla kernel: RSP: 002b:00007ffc52a4b308 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
Nov 16 14:56:48 Valhalla kernel: RAX: ffffffffffffffda RBX: 0000000000494f20 RCX: 000015151abf31ca
Nov 16 14:56:48 Valhalla kernel: RDX: 0000000000494f90 RSI: 0000000000495020 RDI: 000000000000000a
Nov 16 14:56:48 Valhalla kernel: RBP: 0000000000494f90 R08: 0000000000000007 R09: 0000000000499b60
Nov 16 14:56:48 Valhalla kernel: R10: 0000000000000100 R11: 0000000000000246 R12: 0000000000494f20
Nov 16 14:56:48 Valhalla kernel: R13: 0000000000499b60 R14: 00007ffc52a4bf15 R15: 0000000000000004
Nov 16 14:56:48 Valhalla kernel: </TASK>
Nov 16 14:56:48 Valhalla kernel: Modules linked in: rpcsec_gss_krb5 xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap tun xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nct7904 watchdog coretemp xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper kvm drm_kms_helper drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel btusb sr_mod crypto_simd cdrom mei_hdcp mei_pxp btrtl
Nov 16 14:56:48 Valhalla kernel: intel_gtt i2c_i801 cryptd btbcm rapl intel_cstate mpt3sas btintel gigabyte_wmi wmi_bmof mxm_wmi nvme agpgart mei_me i2c_smbus raid_class input_leds bluetooth intel_uncore ahci atlantic nvme_core scsi_transport_sas joydev i2c_core led_class mei libahci ecdh_generic syscopyarea ecc sysfillrect sysimgblt thermal fb_sys_fops fan tpm_crb video tpm_tis tpm_tis_core tpm wmi intel_pmc_core backlight acpi_pad acpi_tad button unix [last unloaded: coretemp]
Nov 16 14:56:48 Valhalla kernel: CR2: 0000000000000038
Nov 16 14:56:48 Valhalla kernel: ---[ end trace 0000000000000000 ]---
Nov 16 14:56:48 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:56:48 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:56:48 Valhalla kernel: RSP: 0018:ffffc900048dbdd0 EFLAGS: 00010202
Nov 16 14:56:48 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:56:48 Valhalla kernel: RDX: ffffc900048dbe20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:56:48 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:56:48 Valhalla kernel: R10: ffffc900048dbe10 R11: ffffc900048dbe94 R12: 0000000000000000
Nov 16 14:56:48 Valhalla kernel: R13: ffffc900048dbe90 R14: ffffc900048dbe20 R15: 0000000000000000
Nov 16 14:56:48 Valhalla kernel: FS:  000015151aaed740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:56:48 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:56:48 Valhalla kernel: CR2: 0000000000000038 CR3: 000000054464c000 CR4: 0000000000752ee0
Nov 16 14:56:48 Valhalla kernel: PKRU: 55555554
Nov 16 14:56:48 Valhalla kernel: note: find[3276] exited with irqs disabled
Nov 16 15:04:19 Valhalla kernel: mdcmd (36): set md_write_method 1
Nov 16 15:04:19 Valhalla kernel: 
Nov 16 15:04:19 Valhalla cache_dirs: Arguments=-l off
Nov 16 15:04:19 Valhalla cache_dirs: Max Scan Secs=10, Min Scan Secs=1
Nov 16 15:04:19 Valhalla cache_dirs: Scan Type=adaptive
Nov 16 15:04:19 Valhalla cache_dirs: Min Scan Depth=4
Nov 16 15:04:19 Valhalla cache_dirs: Max Scan Depth=none
Nov 16 15:04:19 Valhalla cache_dirs: Use Command='find -noleaf'
Nov 16 15:04:19 Valhalla cache_dirs: ---------- Caching Directories ---------------
Nov 16 15:04:19 Valhalla cache_dirs: .Trash-99
Nov 16 15:04:19 Valhalla cache_dirs: Disk
Nov 16 15:04:19 Valhalla cache_dirs: Root Shares
Nov 16 15:04:19 Valhalla cache_dirs: Unassigned
Nov 16 15:04:19 Valhalla cache_dirs: User Shares
Nov 16 15:04:19 Valhalla cache_dirs: appdata
Nov 16 15:04:19 Valhalla cache_dirs: appdataBackup
Nov 16 15:04:19 Valhalla cache_dirs: domains
Nov 16 15:04:19 Valhalla cache_dirs: isos
Nov 16 15:04:19 Valhalla cache_dirs: media
Nov 16 15:04:19 Valhalla cache_dirs: nas
Nov 16 15:04:19 Valhalla cache_dirs: plexlibrary
Nov 16 15:04:19 Valhalla cache_dirs: syslog
Nov 16 15:04:19 Valhalla cache_dirs: system
Nov 16 15:04:19 Valhalla cache_dirs: ----------------------------------------------
Nov 16 15:04:19 Valhalla cache_dirs: Setting Included dirs: 
Nov 16 15:04:19 Valhalla cache_dirs: Setting Excluded dirs: 
Nov 16 15:04:19 Valhalla cache_dirs: min_disk_idle_before_restarting_scan_sec=60
Nov 16 15:04:19 Valhalla cache_dirs: scan_timeout_sec_idle=150
Nov 16 15:04:19 Valhalla cache_dirs: scan_timeout_sec_busy=30
Nov 16 15:04:19 Valhalla cache_dirs: scan_timeout_sec_stable=30
Nov 16 15:04:19 Valhalla cache_dirs: frequency_of_full_depth_scan_sec=604800
Nov 16 15:04:19 Valhalla cache_dirs: cache_dirs service rc.cachedirs: Started: '/usr/local/emhttp/plugins/dynamix.cache.dirs/scripts/cache_dirs -l off 2>/dev/null'
Nov 16 15:04:19 Valhalla root: Delaying execution of fix common problems scan for 10 minutes
Nov 16 15:04:19 Valhalla Recycle Bin: Starting Recycle Bin
Nov 16 15:04:19 Valhalla emhttpd: Starting Recycle Bin...
Nov 16 15:04:20 Valhalla unassigned.devices: Mounting 'Auto Mount' Devices...
Nov 16 15:04:20 Valhalla unassigned.devices: Disk with ID 'ST16000NM001G-REDACTED (dev1)' is not set to auto mount.
Nov 16 15:04:20 Valhalla emhttpd: /usr/local/emhttp/plugins/user.scripts/backgroundScript.sh "/tmp/user.scripts/tmpScripts/modprobe/script" >/dev/null 2>&1/usr/local/emhttp/plugins/user.scripts/backgroundScript.sh "/tmp/user.scripts/tmpScripts/NFS_staticPorts/script" >/dev/null 2>&1
Nov 16 15:04:20 Valhalla emhttpd: Starting services...
Nov 16 15:04:20 Valhalla kernel: coretemp: unknown parameter 'force' ignored
Nov 16 15:04:20 Valhalla kernel: nct6775: unknown parameter 'force' ignored
Nov 16 15:04:20 Valhalla emhttpd: shcmd (114): /etc/rc.d/rc.samba restart
Nov 16 15:04:20 Valhalla wsdd2[14713]: 'Terminated' signal received.
Nov 16 15:04:20 Valhalla nmbd[14670]: [2023/11/16 15:04:20.701714,  0] ../../source3/nmbd/nmbd.c:59(terminate)
Nov 16 15:04:20 Valhalla nmbd[14670]:   Got SIGTERM: going down...
Nov 16 15:04:20 Valhalla winbindd[14728]: [2023/11/16 15:04:20.701728,  0] ../../source3/winbindd/winbindd_dual.c:1950(winbindd_sig_term_handler)
Nov 16 15:04:20 Valhalla winbindd[14728]:   Got sig[15] terminate (is_parent=1)
Nov 16 15:04:20 Valhalla wsdd2[14713]: terminating.
Nov 16 15:04:20 Valhalla winbindd[14732]: [2023/11/16 15:04:20.701789,  0] ../../source3/winbindd/winbindd_dual.c:1950(winbindd_sig_term_handler)
Nov 16 15:04:20 Valhalla winbindd[14732]:   Got sig[15] terminate (is_parent=0)
Nov 16 15:04:20 Valhalla winbindd[16551]: [2023/11/16 15:04:20.701801,  0] ../../source3/winbindd/winbindd_dual.c:1950(winbindd_sig_term_handler)
Nov 16 15:04:20 Valhalla winbindd[16551]:   Got sig[15] terminate (is_parent=0)
Nov 16 15:04:20 Valhalla kernel: nct7904: unknown parameter 'force' ignored
Nov 16 15:04:20 Valhalla rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="17554" x-info="https://www.rsyslog.com"] start
Nov 16 15:04:22 Valhalla root: Starting Samba:  /usr/sbin/smbd -D
Nov 16 15:04:22 Valhalla root:                  /usr/sbin/nmbd -D
Nov 16 15:04:22 Valhalla root:                  /usr/sbin/wsdd2 -d -4
Nov 16 15:04:22 Valhalla wsdd2[18247]: starting.
Nov 16 15:04:22 Valhalla root:                  /usr/sbin/winbindd -D
Nov 16 15:04:22 Valhalla emhttpd: shcmd (119): /etc/rc.d/rc.avahidaemon restart
Nov 16 15:04:22 Valhalla root: Stopping Avahi mDNS/DNS-SD Daemon: stopped
Nov 16 15:04:22 Valhalla avahi-dnsconfd[16081]: read(): EOF
Nov 16 15:04:23 Valhalla root: Starting Avahi mDNS/DNS-SD Daemon: /usr/sbin/avahi-daemon -D
Nov 16 15:04:23 Valhalla avahi-daemon[18290]: Found user 'avahi' (UID 61) and group 'avahi' (GID 214).
Nov 16 15:04:23 Valhalla avahi-daemon[18290]: Successfully dropped root privileges.
Nov 16 15:04:23 Valhalla avahi-daemon[18290]: avahi-daemon 0.8 starting up.
Nov 16 15:04:23 Valhalla avahi-daemon[18290]: Successfully called chroot().
Nov 16 15:04:23 Valhalla avahi-daemon[18290]: Successfully dropped remaining capabilities.
Nov 16 15:04:23 Valhalla avahi-daemon[18290]: Loading service file /services/sftp-ssh.service.
Nov 16 15:04:23 Valhalla avahi-daemon[18290]: Loading service file /services/smb.service.
Nov 16 15:04:23 Valhalla avahi-daemon[18290]: Loading service file /services/ssh.service.
Nov 16 15:04:23 Valhalla avahi-daemon[18290]: Joining mDNS multicast group on interface br0.IPv4 with address 192.168.1.31.
Nov 16 15:04:23 Valhalla avahi-daemon[18290]: New relevant interface br0.IPv4 for mDNS.
Nov 16 15:04:23 Valhalla avahi-daemon[18290]: Network interface enumeration completed.
Nov 16 15:04:23 Valhalla avahi-daemon[18290]: Registering new address record for 192.168.1.31 on br0.IPv4.
Nov 16 15:04:23 Valhalla emhttpd: shcmd (120): /etc/rc.d/rc.avahidnsconfd restart
Nov 16 15:04:23 Valhalla root: Stopping Avahi mDNS/DNS-SD DNS Server Configuration Daemon: stopped
Nov 16 15:04:23 Valhalla root: Starting Avahi mDNS/DNS-SD DNS Server Configuration Daemon:  /usr/sbin/avahi-dnsconfd -D
Nov 16 15:04:23 Valhalla avahi-dnsconfd[18299]: Successfully connected to Avahi daemon.
Nov 16 15:04:23 Valhalla emhttpd: shcmd (130): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 50
Nov 16 15:04:23 Valhalla rpc.mountd[16036]: No host name given with /mnt/user/syslog (ro,async,wdelay,hide,nocrossmnt,secure,root_squash,no_all_squash,no_subtree_check,secure_locks,acl,no_pnfs,fsid=114,anonuid=65534,anongid=65534,sec=sys,ro,secure,root_squash,no_all_squash), suggest *(ro,async,wdelay,hide,nocrossmnt,secure,root_squash,no_all_squash,no_subtree_check,secure_locks,acl,no_pnfs,fsid=114,anonuid=65534,anongid=65534,sec=sys,ro,secure,root_squash,no_all_squash) to avoid warning
Nov 16 15:04:23 Valhalla rpc.mountd[16036]: v4.2 client attached: 0xa5038db365562f64 from "192.168.1.32:755"
Nov 16 15:04:23 Valhalla kernel: loop2: detected capacity change from 0 to 209715200
Nov 16 15:04:23 Valhalla kernel: BTRFS: device fsid cdb2d92b-6edc-4b62-8e84-ad4c1fcd8901 devid 1 transid 8857206 /dev/loop2 scanned by mount (18349)
Nov 16 15:04:23 Valhalla kernel: BTRFS info (device loop2): using crc32c (crc32c-intel) checksum algorithm
Nov 16 15:04:23 Valhalla kernel: BTRFS info (device loop2): using free space tree
Nov 16 15:04:24 Valhalla avahi-daemon[18290]: Server startup complete. Host name is Valhalla.local. Local service cookie is 3434773968.
Nov 16 15:04:24 Valhalla kernel: BTRFS info (device loop2): enabling ssd optimizations
Nov 16 15:04:24 Valhalla root: Resize device id 1 (/dev/loop2) from 100.00GiB to max
Nov 16 15:04:24 Valhalla emhttpd: shcmd (132): /etc/rc.d/rc.docker start
Nov 16 15:04:24 Valhalla root: starting dockerd ...
Nov 16 15:04:24 Valhalla rpc.statd[18425]: Version 2.6.2 starting
Nov 16 15:04:24 Valhalla sm-notify[18426]: Version 2.6.2 starting
Nov 16 15:04:24 Valhalla sm-notify[18426]: Already notifying clients; Exiting!
Nov 16 15:04:24 Valhalla avahi-daemon[18290]: Service "Valhalla" (/services/ssh.service) successfully established.
Nov 16 15:04:24 Valhalla avahi-daemon[18290]: Service "Valhalla" (/services/smb.service) successfully established.
Nov 16 15:04:24 Valhalla avahi-daemon[18290]: Service "Valhalla" (/services/sftp-ssh.service) successfully established.
Nov 16 15:04:25 Valhalla rpc.mountd[16036]: Caught signal 15, un-registering and exiting.
Nov 16 15:04:25 Valhalla kernel: Bridge firewalling registered
Nov 16 15:04:25 Valhalla kernel: Initializing XFRM netlink socket
Nov 16 15:04:26 Valhalla kernel: docker0: port 1(veth92278ba) entered blocking state
Nov 16 15:04:26 Valhalla kernel: docker0: port 1(veth92278ba) entered disabled state
Nov 16 15:04:26 Valhalla kernel: device veth92278ba entered promiscuous mode
Nov 16 15:04:26 Valhalla kernel: docker0: port 1(veth92278ba) entered blocking state
Nov 16 15:04:26 Valhalla kernel: docker0: port 1(veth92278ba) entered forwarding state
Nov 16 15:04:26 Valhalla kernel: IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready
Nov 16 15:04:26 Valhalla kernel: docker0: port 1(veth92278ba) entered disabled state
Nov 16 15:04:26 Valhalla kernel: nfsd: last server has exited, flushing export cache
Nov 16 15:04:26 Valhalla kernel: eth0: renamed from veth6ab2218
Nov 16 15:04:26 Valhalla kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth92278ba: link becomes ready
Nov 16 15:04:26 Valhalla kernel: docker0: port 1(veth92278ba) entered blocking state
Nov 16 15:04:26 Valhalla kernel: docker0: port 1(veth92278ba) entered forwarding state
Nov 16 15:04:27 Valhalla kernel: tun: Universal TUN/TAP device driver, 1.6
Nov 16 15:04:27 Valhalla emhttpd: shcmd (146): /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 1
Nov 16 15:04:27 Valhalla kernel: loop3: detected capacity change from 0 to 2097152
Nov 16 15:04:27 Valhalla kernel: BTRFS: device fsid e6cd4a00-ca11-4141-9ad0-51eedb07202a devid 1 transid 957 /dev/loop3 scanned by mount (20266)
Nov 16 15:04:27 Valhalla kernel: BTRFS info (device loop3): using crc32c (crc32c-intel) checksum algorithm
Nov 16 15:04:27 Valhalla kernel: BTRFS info (device loop3): using free space tree
Nov 16 15:04:27 Valhalla kernel: BTRFS info (device loop3): enabling ssd optimizations
Nov 16 15:04:27 Valhalla root: Resize device id 1 (/dev/loop3) from 1.00GiB to max
Nov 16 15:04:27 Valhalla emhttpd: shcmd (148): /etc/rc.d/rc.libvirt start
Nov 16 15:04:27 Valhalla root: Starting virtlockd...
Nov 16 15:04:27 Valhalla root: Starting virtlogd...
Nov 16 15:04:27 Valhalla root: Starting libvirtd...
Nov 16 15:04:27 Valhalla kernel: mdcmd (37): check correct
Nov 16 15:04:27 Valhalla kernel: md: recovery thread: check P Q ...
Nov 16 15:04:27 Valhalla dnsmasq[20504]: started, version 2.89 cachesize 150
Nov 16 15:04:27 Valhalla dnsmasq[20504]: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset no-nftset auth cryptohash DNSSEC loop-detect inotify dumpfile
Nov 16 15:04:27 Valhalla dnsmasq-dhcp[20504]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h
Nov 16 15:04:27 Valhalla dnsmasq-dhcp[20504]: DHCP, sockets bound exclusively to interface virbr0
Nov 16 15:04:27 Valhalla dnsmasq[20504]: reading /etc/resolv.conf
Nov 16 15:04:27 Valhalla dnsmasq[20504]: using nameserver 1.1.1.1#53
Nov 16 15:04:27 Valhalla dnsmasq[20504]: read /etc/hosts - 3 names
Nov 16 15:04:27 Valhalla dnsmasq[20504]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 names
Nov 16 15:04:27 Valhalla dnsmasq-dhcp[20504]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Nov 16 15:04:27 Valhalla rc.docker: plex: started succesfully!
Nov 16 15:04:27 Valhalla rc.docker: plex: wait 5 seconds
Nov 16 15:04:27 Valhalla kernel: mpt3sas 0000:05:00.0: invalid VPD tag 0x00 (size 0) at offset 0; assume missing optional EEPROM
Nov 16 15:04:28 Valhalla kernel: br0: port 2(vnet0) entered blocking state
Nov 16 15:04:28 Valhalla kernel: br0: port 2(vnet0) entered disabled state
Nov 16 15:04:28 Valhalla kernel: device vnet0 entered promiscuous mode
Nov 16 15:04:28 Valhalla kernel: br0: port 2(vnet0) entered blocking state
Nov 16 15:04:28 Valhalla kernel: br0: port 2(vnet0) entered forwarding state
Nov 16 15:04:28 Valhalla kernel: NFSD: Using UMH upcall client tracking operations.
Nov 16 15:04:28 Valhalla kernel: NFSD: starting 90-second grace period (net f0000000)
Nov 16 15:04:28 Valhalla rpc.mountd[20860]: Version 2.6.2 starting
Nov 16 15:04:29 Valhalla enhanced.log: Enabling Syslog filter...
Nov 16 15:04:29 Valhalla kernel: XFS (md1p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x1cb47bd dinode
Nov 16 15:04:29 Valhalla kernel: XFS (md1p1): Unmount and run xfs_repair
Nov 16 15:04:29 Valhalla kernel: XFS (md1p1): First 128 bytes of corrupted metadata buffer:
Nov 16 15:04:29 Valhalla kernel: 00000000: 49 4e 81 b6 03 02 00 00 00 00 00 63 00 00 00 64  IN.........c...d
Nov 16 15:04:29 Valhalla kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 16 15:04:29 Valhalla kernel: 00000020: 34 6c 69 fa b7 02 ec 98 34 6c 69 f9 ee 5a 2e ac  4li.....4li..Z..
Nov 16 15:04:29 Valhalla kernel: 00000030: 35 4b bb 51 44 6a e6 17 00 00 00 00 00 02 c5 be  5K.QDj..........
Nov 16 15:04:29 Valhalla kernel: 00000040: 00 00 00 00 00 00 00 2d 00 00 00 00 00 00 00 01  .......-........
Nov 16 15:04:29 Valhalla kernel: 00000050: 00 00 24 01 00 00 00 00 00 00 00 00 14 6a e1 7f  ..$..........j..
Nov 16 15:04:29 Valhalla kernel: 00000060: ff ff ff ff 54 35 7f 38 00 00 00 00 00 00 00 13  ....T5.8........
Nov 16 15:04:29 Valhalla kernel: 00000070: 00 00 00 07 00 22 82 ec 00 00 00 00 00 00 00 08  ....."..........
Nov 16 15:04:32 Valhalla rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="20997" x-info="https://www.rsyslog.com"] start
Nov 16 15:04:32 Valhalla Parity Check Tuning: No restart information present
Nov 16 15:04:33 Valhalla rc.docker: binhex-radarr: started succesfully!
Nov 16 15:04:33 Valhalla rc.docker: binhex-radarr: wait 5 seconds
Nov 16 15:04:38 Valhalla rc.docker: binhex-sonarr: started succesfully!
Nov 16 15:04:38 Valhalla rc.docker: binhex-sonarr: wait 5 seconds
Nov 16 15:04:38 Valhalla tips.and.tweaks: Tweaks Applied
Nov 16 15:04:38 Valhalla unassigned.devices: Mounting 'Auto Mount' Remote Shares...
Nov 16 15:04:38 Valhalla unassigned.devices: Using Gateway '192.168.1.1' to Ping Remote Shares.
Nov 16 15:04:38 Valhalla unassigned.devices: Waiting 5 secs before mounting Remote Shares...
Nov 16 15:04:38 Valhalla sudo:     root : PWD=/ ; USER=root ; COMMAND=/bin/bash -c '/usr/local/emhttp/plugins/unbalance/unbalance -port 6237'
Nov 16 15:04:38 Valhalla sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Nov 16 15:04:43 Valhalla rc.docker: prowlarr: started succesfully!
Nov 16 15:04:43 Valhalla rc.docker: prowlarr: wait 5 seconds
Nov 16 15:04:48 Valhalla rc.docker: overseerr: started succesfully!
Nov 16 15:04:48 Valhalla rc.docker: overseerr: wait 5 seconds
Nov 16 15:04:53 Valhalla kernel: docker0: port 2(veth5f0e16c) entered blocking state
Nov 16 15:04:53 Valhalla kernel: docker0: port 2(veth5f0e16c) entered disabled state
Nov 16 15:04:53 Valhalla kernel: device veth5f0e16c entered promiscuous mode
Nov 16 15:04:53 Valhalla kernel: eth0: renamed from veth68a8558
Nov 16 15:04:53 Valhalla kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth5f0e16c: link becomes ready
Nov 16 15:04:53 Valhalla kernel: docker0: port 2(veth5f0e16c) entered blocking state
Nov 16 15:04:53 Valhalla kernel: docker0: port 2(veth5f0e16c) entered forwarding state
Nov 16 15:04:53 Valhalla rc.docker: tautulli: started succesfully!
Nov 16 15:04:53 Valhalla rc.docker: tautulli: wait 5 seconds
Nov 16 15:04:58 Valhalla kernel: docker0: port 3(veth4647dba) entered blocking state
Nov 16 15:04:58 Valhalla kernel: docker0: port 3(veth4647dba) entered disabled state
Nov 16 15:04:58 Valhalla kernel: device veth4647dba entered promiscuous mode
Nov 16 15:04:58 Valhalla kernel: docker0: port 3(veth4647dba) entered blocking state
Nov 16 15:04:58 Valhalla kernel: docker0: port 3(veth4647dba) entered forwarding state
Nov 16 15:04:58 Valhalla kernel: docker0: port 3(veth4647dba) entered disabled state
Nov 16 15:04:59 Valhalla kernel: eth0: renamed from veth22efe46
Nov 16 15:04:59 Valhalla kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth4647dba: link becomes ready
Nov 16 15:04:59 Valhalla kernel: docker0: port 3(veth4647dba) entered blocking state
Nov 16 15:04:59 Valhalla kernel: docker0: port 3(veth4647dba) entered forwarding state
Nov 16 15:04:59 Valhalla rc.docker: Rebuild-DNDC: started succesfully!
Nov 16 15:04:59 Valhalla rc.docker: Rebuild-DNDC: wait 5 seconds
Nov 16 15:05:10 Valhalla kernel: traps: node[25521] trap invalid opcode ip:559fe37db1bc sp:7fff3d67c3e0 error:0 in node[559fe2f4c000+117f000]
Nov 16 15:05:23 Valhalla rpc.mountd[20860]: No host name given with /mnt/user/syslog (ro,async,wdelay,hide,nocrossmnt,secure,root_squash,no_all_squash,no_subtree_check,secure_locks,acl,no_pnfs,fsid=114,anonuid=65534,anongid=65534,sec=sys,ro,secure,root_squash,no_all_squash), suggest *(ro,async,wdelay,hide,nocrossmnt,secure,root_squash,no_all_squash,no_subtree_check,secure_locks,acl,no_pnfs,fsid=114,anonuid=65534,anongid=65534,sec=sys,ro,secure,root_squash,no_all_squash) to avoid warning
Nov 16 15:06:41 Valhalla Parity Check Tuning: Automatic Correcting Parity-Check detected
Nov 16 15:14:00 Valhalla root: Fix Common Problems Version 2023.10.08a

 

The point would be to look at it before the error, in case there's some more clues, not after.

Here is the syslog for 30mins leading up to the crash:

 

Quote

Nov 16 14:25:34 Valhalla kernel: BUG: kernel NULL pointer dereference, address: 0000000000000038
Nov 16 14:25:34 Valhalla kernel: #PF: supervisor read access in kernel mode
Nov 16 14:25:34 Valhalla kernel: #PF: error_code(0x0000) - not-present page
Nov 16 14:25:34 Valhalla kernel: PGD 2ec964067 P4D 2ec964067 PUD 53ce41067 PMD 0 
Nov 16 14:25:34 Valhalla kernel: Oops: 0000 [#30] PREEMPT SMP NOPTI
Nov 16 14:25:34 Valhalla kernel: CPU: 12 PID: 20152 Comm: find Tainted: P      D    O       6.1.49-Unraid #1
Nov 16 14:25:34 Valhalla kernel: Hardware name: Gigabyte Technology Co., Ltd. Z690 AORUS MASTER/Z690 AORUS MASTER, BIOS F28b 11/09/2023
Nov 16 14:25:34 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:25:34 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:25:34 Valhalla kernel: RSP: 0018:ffffc90040297dd0 EFLAGS: 00010202
Nov 16 14:25:34 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:25:34 Valhalla kernel: RDX: ffffc90040297e20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:25:34 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:25:34 Valhalla kernel: R10: ffffc90040297e10 R11: ffffc90040297e94 R12: 0000000000000000
Nov 16 14:25:34 Valhalla kernel: R13: ffffc90040297e90 R14: ffffc90040297e20 R15: 0000000000000000
Nov 16 14:25:34 Valhalla kernel: FS:  000014e42cd93740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:25:34 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:25:34 Valhalla kernel: CR2: 0000000000000038 CR3: 0000000566640000 CR4: 0000000000752ee0
Nov 16 14:25:34 Valhalla kernel: PKRU: 55555554
Nov 16 14:25:34 Valhalla kernel: Call Trace:
Nov 16 14:25:34 Valhalla kernel: <TASK>
Nov 16 14:25:34 Valhalla kernel: ? __die_body+0x1a/0x5c
Nov 16 14:25:34 Valhalla kernel: ? page_fault_oops+0x329/0x376
Nov 16 14:25:34 Valhalla kernel: ? do_user_addr_fault+0x12e/0x48d
Nov 16 14:25:34 Valhalla kernel: ? exc_page_fault+0xfb/0x11d
Nov 16 14:25:34 Valhalla kernel: ? asm_exc_page_fault+0x22/0x30
Nov 16 14:25:34 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:25:34 Valhalla kernel: ? memcg_slab_free_hook+0x28/0xcf
Nov 16 14:25:34 Valhalla kernel: kmem_cache_free+0xb7/0x154
Nov 16 14:25:34 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:25:34 Valhalla kernel: vfs_fstatat+0x52/0x62
Nov 16 14:25:34 Valhalla kernel: __do_sys_newfstatat+0x26/0x5c
Nov 16 14:25:34 Valhalla kernel: do_syscall_64+0x68/0x81
Nov 16 14:25:34 Valhalla kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Nov 16 14:25:34 Valhalla kernel: RIP: 0033:0x14e42ce991ca
Nov 16 14:25:34 Valhalla kernel: Code: 48 89 f2 b9 00 01 00 00 48 89 fe bf 9c ff ff ff e9 0b 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 90 41 89 ca b8 06 01 00 00 0f 05 <3d> 00 f0 ff ff 77 07 31 c0 c3 0f 1f 40 00 48 8b 15 19 4c 0e 00 f7
Nov 16 14:25:34 Valhalla kernel: RSP: 002b:00007fffce142d78 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
Nov 16 14:25:34 Valhalla kernel: RAX: ffffffffffffffda RBX: 0000000000490000 RCX: 000014e42ce991ca
Nov 16 14:25:34 Valhalla kernel: RDX: 0000000000490070 RSI: 0000000000490100 RDI: 000000000000000a
Nov 16 14:25:34 Valhalla kernel: RBP: 0000000000490070 R08: 0000000000000007 R09: 000000000048fef0
Nov 16 14:25:34 Valhalla kernel: R10: 0000000000000100 R11: 0000000000000246 R12: 0000000000490000
Nov 16 14:25:34 Valhalla kernel: R13: 000000000048fef0 R14: 00007fffce143f15 R15: 0000000000000002
Nov 16 14:25:34 Valhalla kernel: </TASK>
Nov 16 14:25:34 Valhalla kernel: Modules linked in: rpcsec_gss_krb5 xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap tun xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nct7904 watchdog coretemp xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper kvm drm_kms_helper drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel btusb sr_mod crypto_simd cdrom mei_hdcp mei_pxp btrtl
Nov 16 14:25:34 Valhalla kernel: intel_gtt i2c_i801 cryptd btbcm rapl intel_cstate mpt3sas btintel gigabyte_wmi wmi_bmof mxm_wmi nvme agpgart mei_me i2c_smbus raid_class input_leds bluetooth intel_uncore ahci atlantic nvme_core scsi_transport_sas joydev i2c_core led_class mei libahci ecdh_generic syscopyarea ecc sysfillrect sysimgblt thermal fb_sys_fops fan tpm_crb video tpm_tis tpm_tis_core tpm wmi intel_pmc_core backlight acpi_pad acpi_tad button unix [last unloaded: coretemp]
Nov 16 14:25:34 Valhalla kernel: CR2: 0000000000000038
Nov 16 14:25:34 Valhalla kernel: ---[ end trace 0000000000000000 ]---
Nov 16 14:25:34 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:25:34 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:25:34 Valhalla kernel: RSP: 0018:ffffc900048dbdd0 EFLAGS: 00010202
Nov 16 14:25:34 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:25:34 Valhalla kernel: RDX: ffffc900048dbe20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:25:34 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:25:34 Valhalla kernel: R10: ffffc900048dbe10 R11: ffffc900048dbe94 R12: 0000000000000000
Nov 16 14:25:34 Valhalla kernel: R13: ffffc900048dbe90 R14: ffffc900048dbe20 R15: 0000000000000000
Nov 16 14:25:34 Valhalla kernel: FS:  000014e42cd93740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:25:34 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:25:34 Valhalla kernel: CR2: 0000000000000038 CR3: 0000000566640000 CR4: 0000000000752ee0
Nov 16 14:25:34 Valhalla kernel: PKRU: 55555554
Nov 16 14:25:34 Valhalla kernel: note: find[20152] exited with irqs disabled
Nov 16 14:28:06 Valhalla kernel: BUG: kernel NULL pointer dereference, address: 0000000000000038
Nov 16 14:28:06 Valhalla kernel: #PF: supervisor read access in kernel mode
Nov 16 14:28:06 Valhalla kernel: #PF: error_code(0x0000) - not-present page
Nov 16 14:28:06 Valhalla kernel: PGD 52a8f6067 P4D 52a8f6067 PUD 139941067 PMD 0 
Nov 16 14:28:06 Valhalla kernel: Oops: 0000 [#31] PREEMPT SMP NOPTI
Nov 16 14:28:06 Valhalla kernel: CPU: 12 PID: 25429 Comm: find Tainted: P      D    O       6.1.49-Unraid #1
Nov 16 14:28:06 Valhalla kernel: Hardware name: Gigabyte Technology Co., Ltd. Z690 AORUS MASTER/Z690 AORUS MASTER, BIOS F28b 11/09/2023
Nov 16 14:28:06 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:28:06 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:28:06 Valhalla kernel: RSP: 0018:ffffc90022327dd0 EFLAGS: 00010202
Nov 16 14:28:06 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:28:06 Valhalla kernel: RDX: ffffc90022327e20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:28:06 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:28:06 Valhalla kernel: R10: ffffc90022327e10 R11: ffffc90022327e94 R12: 0000000000000000
Nov 16 14:28:06 Valhalla kernel: R13: ffffc90022327e90 R14: ffffc90022327e20 R15: 0000000000000000
Nov 16 14:28:06 Valhalla kernel: FS:  0000153ac3181740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:28:06 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:28:06 Valhalla kernel: CR2: 0000000000000038 CR3: 000000014325c000 CR4: 0000000000752ee0
Nov 16 14:28:06 Valhalla kernel: PKRU: 55555554
Nov 16 14:28:06 Valhalla kernel: Call Trace:
Nov 16 14:28:06 Valhalla kernel: <TASK>
Nov 16 14:28:06 Valhalla kernel: ? __die_body+0x1a/0x5c
Nov 16 14:28:06 Valhalla kernel: ? page_fault_oops+0x329/0x376
Nov 16 14:28:06 Valhalla kernel: ? do_user_addr_fault+0x12e/0x48d
Nov 16 14:28:06 Valhalla kernel: ? exc_page_fault+0xfb/0x11d
Nov 16 14:28:06 Valhalla kernel: ? asm_exc_page_fault+0x22/0x30
Nov 16 14:28:06 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:28:06 Valhalla kernel: ? memcg_slab_free_hook+0x28/0xcf
Nov 16 14:28:06 Valhalla kernel: kmem_cache_free+0xb7/0x154
Nov 16 14:28:06 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:28:06 Valhalla kernel: vfs_fstatat+0x52/0x62
Nov 16 14:28:06 Valhalla kernel: __do_sys_newfstatat+0x26/0x5c
Nov 16 14:28:06 Valhalla kernel: do_syscall_64+0x68/0x81
Nov 16 14:28:06 Valhalla kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Nov 16 14:28:06 Valhalla kernel: RIP: 0033:0x153ac32871ca
Nov 16 14:28:06 Valhalla kernel: Code: 48 89 f2 b9 00 01 00 00 48 89 fe bf 9c ff ff ff e9 0b 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 90 41 89 ca b8 06 01 00 00 0f 05 <3d> 00 f0 ff ff 77 07 31 c0 c3 0f 1f 40 00 48 8b 15 19 4c 0e 00 f7
Nov 16 14:28:06 Valhalla kernel: RSP: 002b:00007ffd22c0d628 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
Nov 16 14:28:06 Valhalla kernel: RAX: ffffffffffffffda RBX: 00000000005178a0 RCX: 0000153ac32871ca
Nov 16 14:28:06 Valhalla kernel: RDX: 0000000000517910 RSI: 00000000005179a0 RDI: 000000000000000b
Nov 16 14:28:06 Valhalla kernel: RBP: 0000000000517910 R08: 0000000000000007 R09: 0000000000494330
Nov 16 14:28:06 Valhalla kernel: R10: 0000000000000100 R11: 0000000000000246 R12: 00000000005178a0
Nov 16 14:28:06 Valhalla kernel: R13: 0000000000494330 R14: 00007ffd22c0df15 R15: 0000000000000003
Nov 16 14:28:06 Valhalla kernel: </TASK>
Nov 16 14:28:06 Valhalla kernel: Modules linked in: rpcsec_gss_krb5 xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap tun xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nct7904 watchdog coretemp xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper kvm drm_kms_helper drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel btusb sr_mod crypto_simd cdrom mei_hdcp mei_pxp btrtl
Nov 16 14:28:06 Valhalla kernel: intel_gtt i2c_i801 cryptd btbcm rapl intel_cstate mpt3sas btintel gigabyte_wmi wmi_bmof mxm_wmi nvme agpgart mei_me i2c_smbus raid_class input_leds bluetooth intel_uncore ahci atlantic nvme_core scsi_transport_sas joydev i2c_core led_class mei libahci ecdh_generic syscopyarea ecc sysfillrect sysimgblt thermal fb_sys_fops fan tpm_crb video tpm_tis tpm_tis_core tpm wmi intel_pmc_core backlight acpi_pad acpi_tad button unix [last unloaded: coretemp]
Nov 16 14:28:06 Valhalla kernel: CR2: 0000000000000038
Nov 16 14:28:06 Valhalla kernel: ---[ end trace 0000000000000000 ]---
Nov 16 14:28:06 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:28:06 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:28:06 Valhalla kernel: RSP: 0018:ffffc900048dbdd0 EFLAGS: 00010202
Nov 16 14:28:06 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:28:06 Valhalla kernel: RDX: ffffc900048dbe20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:28:06 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:28:06 Valhalla kernel: R10: ffffc900048dbe10 R11: ffffc900048dbe94 R12: 0000000000000000
Nov 16 14:28:06 Valhalla kernel: R13: ffffc900048dbe90 R14: ffffc900048dbe20 R15: 0000000000000000
Nov 16 14:28:06 Valhalla kernel: FS:  0000153ac3181740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:28:06 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:28:06 Valhalla kernel: CR2: 0000000000000038 CR3: 000000014325c000 CR4: 0000000000752ee0
Nov 16 14:28:06 Valhalla kernel: PKRU: 55555554
Nov 16 14:28:06 Valhalla kernel: note: find[25429] exited with irqs disabled
Nov 16 14:29:47 Valhalla kernel: BUG: kernel NULL pointer dereference, address: 0000000000000038
Nov 16 14:29:47 Valhalla kernel: #PF: supervisor read access in kernel mode
Nov 16 14:29:47 Valhalla kernel: #PF: error_code(0x0000) - not-present page
Nov 16 14:29:47 Valhalla kernel: PGD 3407aa067 P4D 3407aa067 PUD 566d2b067 PMD 0 
Nov 16 14:29:47 Valhalla kernel: Oops: 0000 [#32] PREEMPT SMP NOPTI
Nov 16 14:29:47 Valhalla kernel: CPU: 12 PID: 29787 Comm: find Tainted: P      D    O       6.1.49-Unraid #1
Nov 16 14:29:47 Valhalla kernel: Hardware name: Gigabyte Technology Co., Ltd. Z690 AORUS MASTER/Z690 AORUS MASTER, BIOS F28b 11/09/2023
Nov 16 14:29:47 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:29:47 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:29:47 Valhalla kernel: RSP: 0018:ffffc90024f77dd0 EFLAGS: 00010202
Nov 16 14:29:47 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:29:47 Valhalla kernel: RDX: ffffc90024f77e20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:29:47 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:29:47 Valhalla kernel: R10: ffffc90024f77e10 R11: ffffc90024f77e94 R12: 0000000000000000
Nov 16 14:29:47 Valhalla kernel: R13: ffffc90024f77e90 R14: ffffc90024f77e20 R15: 0000000000000000
Nov 16 14:29:47 Valhalla kernel: FS:  000015510e7c7740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:29:47 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:29:47 Valhalla kernel: CR2: 0000000000000038 CR3: 0000000140582000 CR4: 0000000000752ee0
Nov 16 14:29:47 Valhalla kernel: PKRU: 55555554
Nov 16 14:29:47 Valhalla kernel: Call Trace:
Nov 16 14:29:47 Valhalla kernel: <TASK>
Nov 16 14:29:47 Valhalla kernel: ? __die_body+0x1a/0x5c
Nov 16 14:29:47 Valhalla kernel: ? page_fault_oops+0x329/0x376
Nov 16 14:29:47 Valhalla kernel: ? do_user_addr_fault+0x12e/0x48d
Nov 16 14:29:47 Valhalla kernel: ? exc_page_fault+0xfb/0x11d
Nov 16 14:29:47 Valhalla kernel: ? asm_exc_page_fault+0x22/0x30
Nov 16 14:29:47 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:29:47 Valhalla kernel: ? memcg_slab_free_hook+0x28/0xcf
Nov 16 14:29:47 Valhalla kernel: kmem_cache_free+0xb7/0x154
Nov 16 14:29:47 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:29:47 Valhalla kernel: vfs_fstatat+0x52/0x62
Nov 16 14:29:47 Valhalla kernel: __do_sys_newfstatat+0x26/0x5c
Nov 16 14:29:47 Valhalla kernel: do_syscall_64+0x68/0x81
Nov 16 14:29:47 Valhalla kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Nov 16 14:29:47 Valhalla kernel: RIP: 0033:0x15510e8cd1ca
Nov 16 14:29:47 Valhalla kernel: Code: 48 89 f2 b9 00 01 00 00 48 89 fe bf 9c ff ff ff e9 0b 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 90 41 89 ca b8 06 01 00 00 0f 05 <3d> 00 f0 ff ff 77 07 31 c0 c3 0f 1f 40 00 48 8b 15 19 4c 0e 00 f7
Nov 16 14:29:47 Valhalla kernel: RSP: 002b:00007ffef3519c38 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
Nov 16 14:29:47 Valhalla kernel: RAX: ffffffffffffffda RBX: 0000000000462430 RCX: 000015510e8cd1ca
Nov 16 14:29:47 Valhalla kernel: RDX: 00000000004624a0 RSI: 0000000000462530 RDI: 0000000000000009
Nov 16 14:29:47 Valhalla kernel: RBP: 00000000004624a0 R08: 0000000000000007 R09: 0000000000446060
Nov 16 14:29:47 Valhalla kernel: R10: 0000000000000100 R11: 0000000000000246 R12: 0000000000462430
Nov 16 14:29:47 Valhalla kernel: R13: 0000000000446060 R14: 00007ffef351bf15 R15: 0000000000000006
Nov 16 14:29:47 Valhalla kernel: </TASK>
Nov 16 14:29:47 Valhalla kernel: Modules linked in: rpcsec_gss_krb5 xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap tun xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nct7904 watchdog coretemp xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper kvm drm_kms_helper drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel btusb sr_mod crypto_simd cdrom mei_hdcp mei_pxp btrtl
Nov 16 14:29:47 Valhalla kernel: intel_gtt i2c_i801 cryptd btbcm rapl intel_cstate mpt3sas btintel gigabyte_wmi wmi_bmof mxm_wmi nvme agpgart mei_me i2c_smbus raid_class input_leds bluetooth intel_uncore ahci atlantic nvme_core scsi_transport_sas joydev i2c_core led_class mei libahci ecdh_generic syscopyarea ecc sysfillrect sysimgblt thermal fb_sys_fops fan tpm_crb video tpm_tis tpm_tis_core tpm wmi intel_pmc_core backlight acpi_pad acpi_tad button unix [last unloaded: coretemp]
Nov 16 14:29:47 Valhalla kernel: CR2: 0000000000000038
Nov 16 14:29:47 Valhalla kernel: ---[ end trace 0000000000000000 ]---
Nov 16 14:29:47 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:29:47 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:29:47 Valhalla kernel: RSP: 0018:ffffc900048dbdd0 EFLAGS: 00010202
Nov 16 14:29:47 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:29:47 Valhalla kernel: RDX: ffffc900048dbe20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:29:47 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:29:47 Valhalla kernel: R10: ffffc900048dbe10 R11: ffffc900048dbe94 R12: 0000000000000000
Nov 16 14:29:47 Valhalla kernel: R13: ffffc900048dbe90 R14: ffffc900048dbe20 R15: 0000000000000000
Nov 16 14:29:47 Valhalla kernel: FS:  000015510e7c7740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:29:47 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:29:47 Valhalla kernel: CR2: 0000000000000038 CR3: 0000000140582000 CR4: 0000000000752ee0
Nov 16 14:29:47 Valhalla kernel: PKRU: 55555554
Nov 16 14:29:47 Valhalla kernel: note: find[29787] exited with irqs disabled
Nov 16 14:32:19 Valhalla kernel: BUG: kernel NULL pointer dereference, address: 0000000000000038
Nov 16 14:32:19 Valhalla kernel: #PF: supervisor read access in kernel mode
Nov 16 14:32:19 Valhalla kernel: #PF: error_code(0x0000) - not-present page
Nov 16 14:32:19 Valhalla kernel: PGD 192c92067 P4D 192c92067 PUD 566899067 PMD 0 
Nov 16 14:32:19 Valhalla kernel: Oops: 0000 [#33] PREEMPT SMP NOPTI
Nov 16 14:32:19 Valhalla kernel: CPU: 12 PID: 4069 Comm: find Tainted: P      D    O       6.1.49-Unraid #1
Nov 16 14:32:19 Valhalla kernel: Hardware name: Gigabyte Technology Co., Ltd. Z690 AORUS MASTER/Z690 AORUS MASTER, BIOS F28b 11/09/2023
Nov 16 14:32:19 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:32:19 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:32:19 Valhalla kernel: RSP: 0018:ffffc9002bfc3dd0 EFLAGS: 00010202
Nov 16 14:32:19 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:32:19 Valhalla kernel: RDX: ffffc9002bfc3e20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:32:19 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:32:19 Valhalla kernel: R10: ffffc9002bfc3e10 R11: ffffc9002bfc3e94 R12: 0000000000000000
Nov 16 14:32:19 Valhalla kernel: R13: ffffc9002bfc3e90 R14: ffffc9002bfc3e20 R15: 0000000000000000
Nov 16 14:32:19 Valhalla kernel: FS:  000015325ed39740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:32:19 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:32:19 Valhalla kernel: CR2: 0000000000000038 CR3: 00000002e15a0000 CR4: 0000000000752ee0
Nov 16 14:32:19 Valhalla kernel: PKRU: 55555554
Nov 16 14:32:19 Valhalla kernel: Call Trace:
Nov 16 14:32:19 Valhalla kernel: <TASK>
Nov 16 14:32:19 Valhalla kernel: ? __die_body+0x1a/0x5c
Nov 16 14:32:19 Valhalla kernel: ? page_fault_oops+0x329/0x376
Nov 16 14:32:19 Valhalla kernel: ? do_user_addr_fault+0x12e/0x48d
Nov 16 14:32:19 Valhalla kernel: ? exc_page_fault+0xfb/0x11d
Nov 16 14:32:19 Valhalla kernel: ? asm_exc_page_fault+0x22/0x30
Nov 16 14:32:19 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:32:19 Valhalla kernel: ? memcg_slab_free_hook+0x28/0xcf
Nov 16 14:32:19 Valhalla kernel: kmem_cache_free+0xb7/0x154
Nov 16 14:32:19 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:32:19 Valhalla kernel: vfs_fstatat+0x52/0x62
Nov 16 14:32:19 Valhalla kernel: __do_sys_newfstatat+0x26/0x5c
Nov 16 14:32:19 Valhalla kernel: do_syscall_64+0x68/0x81
Nov 16 14:32:19 Valhalla kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Nov 16 14:32:19 Valhalla kernel: RIP: 0033:0x15325ee3f1ca
Nov 16 14:32:19 Valhalla kernel: Code: 48 89 f2 b9 00 01 00 00 48 89 fe bf 9c ff ff ff e9 0b 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 90 41 89 ca b8 06 01 00 00 0f 05 <3d> 00 f0 ff ff 77 07 31 c0 c3 0f 1f 40 00 48 8b 15 19 4c 0e 00 f7
Nov 16 14:32:19 Valhalla kernel: RSP: 002b:00007ffebfd91048 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
Nov 16 14:32:19 Valhalla kernel: RAX: ffffffffffffffda RBX: 0000000000455ce0 RCX: 000015325ee3f1ca
Nov 16 14:32:19 Valhalla kernel: RDX: 0000000000455d50 RSI: 0000000000455de0 RDI: 000000000000000b
Nov 16 14:32:19 Valhalla kernel: RBP: 0000000000455d50 R08: 0000000000000007 R09: 00000000004d0e20
Nov 16 14:32:19 Valhalla kernel: R10: 0000000000000100 R11: 0000000000000246 R12: 0000000000455ce0
Nov 16 14:32:19 Valhalla kernel: R13: 00000000004d0e20 R14: 00007ffebfd92f16 R15: 0000000000000005
Nov 16 14:32:19 Valhalla kernel: </TASK>
Nov 16 14:32:19 Valhalla kernel: Modules linked in: rpcsec_gss_krb5 xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap tun xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nct7904 watchdog coretemp xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper kvm drm_kms_helper drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel btusb sr_mod crypto_simd cdrom mei_hdcp mei_pxp btrtl
Nov 16 14:32:19 Valhalla kernel: intel_gtt i2c_i801 cryptd btbcm rapl intel_cstate mpt3sas btintel gigabyte_wmi wmi_bmof mxm_wmi nvme agpgart mei_me i2c_smbus raid_class input_leds bluetooth intel_uncore ahci atlantic nvme_core scsi_transport_sas joydev i2c_core led_class mei libahci ecdh_generic syscopyarea ecc sysfillrect sysimgblt thermal fb_sys_fops fan tpm_crb video tpm_tis tpm_tis_core tpm wmi intel_pmc_core backlight acpi_pad acpi_tad button unix [last unloaded: coretemp]
Nov 16 14:32:19 Valhalla kernel: CR2: 0000000000000038
Nov 16 14:32:19 Valhalla kernel: ---[ end trace 0000000000000000 ]---
Nov 16 14:32:19 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:32:19 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:32:19 Valhalla kernel: RSP: 0018:ffffc900048dbdd0 EFLAGS: 00010202
Nov 16 14:32:19 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:32:19 Valhalla kernel: RDX: ffffc900048dbe20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:32:19 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:32:19 Valhalla kernel: R10: ffffc900048dbe10 R11: ffffc900048dbe94 R12: 0000000000000000
Nov 16 14:32:19 Valhalla kernel: R13: ffffc900048dbe90 R14: ffffc900048dbe20 R15: 0000000000000000
Nov 16 14:32:19 Valhalla kernel: FS:  000015325ed39740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:32:19 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:32:19 Valhalla kernel: CR2: 0000000000000038 CR3: 00000002e15a0000 CR4: 0000000000752ee0
Nov 16 14:32:19 Valhalla kernel: PKRU: 55555554
Nov 16 14:32:19 Valhalla kernel: note: find[4069] exited with irqs disabled
Nov 16 14:35:04 Valhalla kernel: BUG: kernel NULL pointer dereference, address: 0000000000000038
Nov 16 14:35:04 Valhalla kernel: #PF: supervisor read access in kernel mode
Nov 16 14:35:04 Valhalla kernel: #PF: error_code(0x0000) - not-present page
Nov 16 14:35:04 Valhalla kernel: PGD 140f9a067 P4D 140f9a067 PUD 51fe9b067 PMD 0 
Nov 16 14:35:04 Valhalla kernel: Oops: 0000 [#34] PREEMPT SMP NOPTI
Nov 16 14:35:04 Valhalla kernel: CPU: 12 PID: 11352 Comm: find Tainted: P      D    O       6.1.49-Unraid #1
Nov 16 14:35:04 Valhalla kernel: Hardware name: Gigabyte Technology Co., Ltd. Z690 AORUS MASTER/Z690 AORUS MASTER, BIOS F28b 11/09/2023
Nov 16 14:35:04 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:35:04 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:35:04 Valhalla kernel: RSP: 0018:ffffc9000545fdd0 EFLAGS: 00010202
Nov 16 14:35:04 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:35:04 Valhalla kernel: RDX: ffffc9000545fe20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:35:04 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:35:04 Valhalla kernel: R10: ffffc9000545fe10 R11: ffffc9000545fe94 R12: 0000000000000000
Nov 16 14:35:04 Valhalla kernel: R13: ffffc9000545fe90 R14: ffffc9000545fe20 R15: 0000000000000000
Nov 16 14:35:04 Valhalla kernel: FS:  0000145b2c50f740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:35:04 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:35:04 Valhalla kernel: CR2: 0000000000000038 CR3: 00000001a567e000 CR4: 0000000000752ee0
Nov 16 14:35:04 Valhalla kernel: PKRU: 55555554
Nov 16 14:35:04 Valhalla kernel: Call Trace:
Nov 16 14:35:04 Valhalla kernel: <TASK>
Nov 16 14:35:04 Valhalla kernel: ? __die_body+0x1a/0x5c
Nov 16 14:35:04 Valhalla kernel: ? page_fault_oops+0x329/0x376
Nov 16 14:35:04 Valhalla kernel: ? do_user_addr_fault+0x12e/0x48d
Nov 16 14:35:04 Valhalla kernel: ? exc_page_fault+0xfb/0x11d
Nov 16 14:35:04 Valhalla kernel: ? asm_exc_page_fault+0x22/0x30
Nov 16 14:35:04 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:35:04 Valhalla kernel: ? memcg_slab_free_hook+0x28/0xcf
Nov 16 14:35:04 Valhalla kernel: kmem_cache_free+0xb7/0x154
Nov 16 14:35:04 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:35:04 Valhalla kernel: vfs_fstatat+0x52/0x62
Nov 16 14:35:04 Valhalla kernel: __do_sys_newfstatat+0x26/0x5c
Nov 16 14:35:04 Valhalla kernel: do_syscall_64+0x68/0x81
Nov 16 14:35:04 Valhalla kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Nov 16 14:35:04 Valhalla kernel: RIP: 0033:0x145b2c6151ca
Nov 16 14:35:04 Valhalla kernel: Code: 48 89 f2 b9 00 01 00 00 48 89 fe bf 9c ff ff ff e9 0b 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 90 41 89 ca b8 06 01 00 00 0f 05 <3d> 00 f0 ff ff 77 07 31 c0 c3 0f 1f 40 00 48 8b 15 19 4c 0e 00 f7
Nov 16 14:35:04 Valhalla kernel: RSP: 002b:00007ffe22c7a288 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
Nov 16 14:35:04 Valhalla kernel: RAX: ffffffffffffffda RBX: 000000000044ccd0 RCX: 0000145b2c6151ca
Nov 16 14:35:04 Valhalla kernel: RDX: 000000000044cd40 RSI: 000000000044cdd0 RDI: 000000000000000b
Nov 16 14:35:04 Valhalla kernel: RBP: 000000000044cd40 R08: 00000000ffffffff R09: 000000000044d560
Nov 16 14:35:04 Valhalla kernel: R10: 0000000000000100 R11: 0000000000000246 R12: 000000000044ccd0
Nov 16 14:35:04 Valhalla kernel: R13: 0000000000000000 R14: 00007ffe22c7bf16 R15: 0000000000000005
Nov 16 14:35:04 Valhalla kernel: </TASK>
Nov 16 14:35:04 Valhalla kernel: Modules linked in: rpcsec_gss_krb5 xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap tun xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nct7904 watchdog coretemp xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper kvm drm_kms_helper drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel btusb sr_mod crypto_simd cdrom mei_hdcp mei_pxp btrtl
Nov 16 14:35:04 Valhalla kernel: intel_gtt i2c_i801 cryptd btbcm rapl intel_cstate mpt3sas btintel gigabyte_wmi wmi_bmof mxm_wmi nvme agpgart mei_me i2c_smbus raid_class input_leds bluetooth intel_uncore ahci atlantic nvme_core scsi_transport_sas joydev i2c_core led_class mei libahci ecdh_generic syscopyarea ecc sysfillrect sysimgblt thermal fb_sys_fops fan tpm_crb video tpm_tis tpm_tis_core tpm wmi intel_pmc_core backlight acpi_pad acpi_tad button unix [last unloaded: coretemp]
Nov 16 14:35:04 Valhalla kernel: CR2: 0000000000000038
Nov 16 14:35:04 Valhalla kernel: ---[ end trace 0000000000000000 ]---
Nov 16 14:35:04 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:35:04 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:35:04 Valhalla kernel: RSP: 0018:ffffc900048dbdd0 EFLAGS: 00010202
Nov 16 14:35:04 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:35:04 Valhalla kernel: RDX: ffffc900048dbe20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:35:04 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:35:04 Valhalla kernel: R10: ffffc900048dbe10 R11: ffffc900048dbe94 R12: 0000000000000000
Nov 16 14:35:04 Valhalla kernel: R13: ffffc900048dbe90 R14: ffffc900048dbe20 R15: 0000000000000000
Nov 16 14:35:04 Valhalla kernel: FS:  0000145b2c50f740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:35:04 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:35:04 Valhalla kernel: CR2: 0000000000000038 CR3: 00000001a567e000 CR4: 0000000000752ee0
Nov 16 14:35:04 Valhalla kernel: PKRU: 55555554
Nov 16 14:35:04 Valhalla kernel: note: find[11352] exited with irqs disabled
Nov 16 14:39:42 Valhalla kernel: BUG: kernel NULL pointer dereference, address: 0000000000000038
Nov 16 14:39:42 Valhalla kernel: #PF: supervisor read access in kernel mode
Nov 16 14:39:42 Valhalla kernel: #PF: error_code(0x0000) - not-present page
Nov 16 14:39:42 Valhalla kernel: PGD 143708067 P4D 143708067 PUD 57817c067 PMD 0 
Nov 16 14:39:42 Valhalla kernel: Oops: 0000 [#35] PREEMPT SMP NOPTI
Nov 16 14:39:42 Valhalla kernel: CPU: 12 PID: 23190 Comm: find Tainted: P      D    O       6.1.49-Unraid #1
Nov 16 14:39:42 Valhalla kernel: Hardware name: Gigabyte Technology Co., Ltd. Z690 AORUS MASTER/Z690 AORUS MASTER, BIOS F28b 11/09/2023
Nov 16 14:39:42 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:39:42 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:39:42 Valhalla kernel: RSP: 0018:ffffc90029f73dd0 EFLAGS: 00010202
Nov 16 14:39:42 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:39:42 Valhalla kernel: RDX: ffffc90029f73e20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:39:42 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:39:42 Valhalla kernel: R10: ffffc90029f73e10 R11: ffffc90029f73e94 R12: 0000000000000000
Nov 16 14:39:42 Valhalla kernel: R13: ffffc90029f73e90 R14: ffffc90029f73e20 R15: 0000000000000000
Nov 16 14:39:42 Valhalla kernel: FS:  0000145ab7420740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:39:42 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:39:42 Valhalla kernel: CR2: 0000000000000038 CR3: 0000000286af4000 CR4: 0000000000752ee0
Nov 16 14:39:42 Valhalla kernel: PKRU: 55555554
Nov 16 14:39:42 Valhalla kernel: Call Trace:
Nov 16 14:39:42 Valhalla kernel: <TASK>
Nov 16 14:39:42 Valhalla kernel: ? __die_body+0x1a/0x5c
Nov 16 14:39:42 Valhalla kernel: ? page_fault_oops+0x329/0x376
Nov 16 14:39:42 Valhalla kernel: ? do_user_addr_fault+0x12e/0x48d
Nov 16 14:39:42 Valhalla kernel: ? exc_page_fault+0xfb/0x11d
Nov 16 14:39:42 Valhalla kernel: ? asm_exc_page_fault+0x22/0x30
Nov 16 14:39:42 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:39:42 Valhalla kernel: ? memcg_slab_free_hook+0x28/0xcf
Nov 16 14:39:42 Valhalla kernel: kmem_cache_free+0xb7/0x154
Nov 16 14:39:42 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:39:42 Valhalla kernel: vfs_fstatat+0x52/0x62
Nov 16 14:39:42 Valhalla kernel: __do_sys_newfstatat+0x26/0x5c
Nov 16 14:39:42 Valhalla kernel: do_syscall_64+0x68/0x81
Nov 16 14:39:42 Valhalla kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Nov 16 14:39:42 Valhalla kernel: RIP: 0033:0x145ab75261ca
Nov 16 14:39:42 Valhalla kernel: Code: 48 89 f2 b9 00 01 00 00 48 89 fe bf 9c ff ff ff e9 0b 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 90 41 89 ca b8 06 01 00 00 0f 05 <3d> 00 f0 ff ff 77 07 31 c0 c3 0f 1f 40 00 48 8b 15 19 4c 0e 00 f7
Nov 16 14:39:42 Valhalla kernel: RSP: 002b:00007ffc0ce7d168 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
Nov 16 14:39:42 Valhalla kernel: RAX: ffffffffffffffda RBX: 000000000044cd80 RCX: 0000145ab75261ca
Nov 16 14:39:42 Valhalla kernel: RDX: 000000000044cdf0 RSI: 000000000044ce80 RDI: 000000000000000a
Nov 16 14:39:42 Valhalla kernel: RBP: 000000000044cdf0 R08: 0000000000000007 R09: 0000000000449560
Nov 16 14:39:42 Valhalla kernel: R10: 0000000000000100 R11: 0000000000000246 R12: 000000000044cd80
Nov 16 14:39:42 Valhalla kernel: R13: 0000000000449560 R14: 00007ffc0ce7df16 R15: 0000000000000004
Nov 16 14:39:42 Valhalla kernel: </TASK>
Nov 16 14:39:42 Valhalla kernel: Modules linked in: rpcsec_gss_krb5 xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap tun xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nct7904 watchdog coretemp xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper kvm drm_kms_helper drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel btusb sr_mod crypto_simd cdrom mei_hdcp mei_pxp btrtl
Nov 16 14:39:42 Valhalla kernel: intel_gtt i2c_i801 cryptd btbcm rapl intel_cstate mpt3sas btintel gigabyte_wmi wmi_bmof mxm_wmi nvme agpgart mei_me i2c_smbus raid_class input_leds bluetooth intel_uncore ahci atlantic nvme_core scsi_transport_sas joydev i2c_core led_class mei libahci ecdh_generic syscopyarea ecc sysfillrect sysimgblt thermal fb_sys_fops fan tpm_crb video tpm_tis tpm_tis_core tpm wmi intel_pmc_core backlight acpi_pad acpi_tad button unix [last unloaded: coretemp]
Nov 16 14:39:42 Valhalla kernel: CR2: 0000000000000038
Nov 16 14:39:42 Valhalla kernel: ---[ end trace 0000000000000000 ]---
Nov 16 14:39:42 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:39:42 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:39:42 Valhalla kernel: RSP: 0018:ffffc900048dbdd0 EFLAGS: 00010202
Nov 16 14:39:42 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:39:42 Valhalla kernel: RDX: ffffc900048dbe20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:39:42 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:39:42 Valhalla kernel: R10: ffffc900048dbe10 R11: ffffc900048dbe94 R12: 0000000000000000
Nov 16 14:39:42 Valhalla kernel: R13: ffffc900048dbe90 R14: ffffc900048dbe20 R15: 0000000000000000
Nov 16 14:39:42 Valhalla kernel: FS:  0000145ab7420740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:39:42 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:39:42 Valhalla kernel: CR2: 0000000000000038 CR3: 0000000286af4000 CR4: 0000000000752ee0
Nov 16 14:39:42 Valhalla kernel: PKRU: 55555554
Nov 16 14:39:42 Valhalla kernel: note: find[23190] exited with irqs disabled
Nov 16 14:50:41 Valhalla kernel: BUG: kernel NULL pointer dereference, address: 0000000000000038
Nov 16 14:50:41 Valhalla kernel: #PF: supervisor read access in kernel mode
Nov 16 14:50:41 Valhalla kernel: #PF: error_code(0x0000) - not-present page
Nov 16 14:50:41 Valhalla kernel: PGD 181bb5067 P4D 181bb5067 PUD 502d71067 PMD 0 
Nov 16 14:50:41 Valhalla kernel: Oops: 0000 [#36] PREEMPT SMP NOPTI
Nov 16 14:50:41 Valhalla kernel: CPU: 12 PID: 19730 Comm: find Tainted: P      D    O       6.1.49-Unraid #1
Nov 16 14:50:41 Valhalla kernel: Hardware name: Gigabyte Technology Co., Ltd. Z690 AORUS MASTER/Z690 AORUS MASTER, BIOS F28b 11/09/2023
Nov 16 14:50:41 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:50:41 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:50:41 Valhalla kernel: RSP: 0018:ffffc9004056fdd0 EFLAGS: 00010202
Nov 16 14:50:41 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:50:41 Valhalla kernel: RDX: ffffc9004056fe20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:50:41 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:50:41 Valhalla kernel: R10: ffffc9004056fe10 R11: ffffc9004056fe94 R12: 0000000000000000
Nov 16 14:50:41 Valhalla kernel: R13: ffffc9004056fe90 R14: ffffc9004056fe20 R15: 0000000000000000
Nov 16 14:50:41 Valhalla kernel: FS:  000015383bd00740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:50:41 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:50:41 Valhalla kernel: CR2: 0000000000000038 CR3: 00000001a5628000 CR4: 0000000000752ee0
Nov 16 14:50:41 Valhalla kernel: PKRU: 55555554
Nov 16 14:50:41 Valhalla kernel: Call Trace:
Nov 16 14:50:41 Valhalla kernel: <TASK>
Nov 16 14:50:41 Valhalla kernel: ? __die_body+0x1a/0x5c
Nov 16 14:50:41 Valhalla kernel: ? page_fault_oops+0x329/0x376
Nov 16 14:50:41 Valhalla kernel: ? do_user_addr_fault+0x12e/0x48d
Nov 16 14:50:41 Valhalla kernel: ? exc_page_fault+0xfb/0x11d
Nov 16 14:50:41 Valhalla kernel: ? asm_exc_page_fault+0x22/0x30
Nov 16 14:50:41 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:50:41 Valhalla kernel: ? memcg_slab_free_hook+0x28/0xcf
Nov 16 14:50:41 Valhalla kernel: kmem_cache_free+0xb7/0x154
Nov 16 14:50:41 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:50:41 Valhalla kernel: vfs_fstatat+0x52/0x62
Nov 16 14:50:41 Valhalla kernel: __do_sys_newfstatat+0x26/0x5c
Nov 16 14:50:41 Valhalla kernel: do_syscall_64+0x68/0x81
Nov 16 14:50:41 Valhalla kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Nov 16 14:50:41 Valhalla kernel: RIP: 0033:0x15383be061ca
Nov 16 14:50:41 Valhalla kernel: Code: 48 89 f2 b9 00 01 00 00 48 89 fe bf 9c ff ff ff e9 0b 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 90 41 89 ca b8 06 01 00 00 0f 05 <3d> 00 f0 ff ff 77 07 31 c0 c3 0f 1f 40 00 48 8b 15 19 4c 0e 00 f7
Nov 16 14:50:41 Valhalla kernel: RSP: 002b:00007ffcf0a0c068 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
Nov 16 14:50:41 Valhalla kernel: RAX: ffffffffffffffda RBX: 0000000000465d20 RCX: 000015383be061ca
Nov 16 14:50:41 Valhalla kernel: RDX: 0000000000465d90 RSI: 0000000000465e20 RDI: 0000000000000009
Nov 16 14:50:41 Valhalla kernel: RBP: 0000000000465d90 R08: 0000000000000007 R09: 000000000044aee0
Nov 16 14:50:41 Valhalla kernel: R10: 0000000000000100 R11: 0000000000000246 R12: 0000000000465d20
Nov 16 14:50:41 Valhalla kernel: R13: 000000000044aee0 R14: 00007ffcf0a0cf16 R15: 0000000000000006
Nov 16 14:50:41 Valhalla kernel: </TASK>
Nov 16 14:50:41 Valhalla kernel: Modules linked in: rpcsec_gss_krb5 xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap tun xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nct7904 watchdog coretemp xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper kvm drm_kms_helper drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel btusb sr_mod crypto_simd cdrom mei_hdcp mei_pxp btrtl
Nov 16 14:50:41 Valhalla kernel: intel_gtt i2c_i801 cryptd btbcm rapl intel_cstate mpt3sas btintel gigabyte_wmi wmi_bmof mxm_wmi nvme agpgart mei_me i2c_smbus raid_class input_leds bluetooth intel_uncore ahci atlantic nvme_core scsi_transport_sas joydev i2c_core led_class mei libahci ecdh_generic syscopyarea ecc sysfillrect sysimgblt thermal fb_sys_fops fan tpm_crb video tpm_tis tpm_tis_core tpm wmi intel_pmc_core backlight acpi_pad acpi_tad button unix [last unloaded: coretemp]
Nov 16 14:50:41 Valhalla kernel: CR2: 0000000000000038
Nov 16 14:50:41 Valhalla kernel: ---[ end trace 0000000000000000 ]---
Nov 16 14:50:41 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:50:41 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:50:41 Valhalla kernel: RSP: 0018:ffffc900048dbdd0 EFLAGS: 00010202
Nov 16 14:50:41 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:50:41 Valhalla kernel: RDX: ffffc900048dbe20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:50:41 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:50:41 Valhalla kernel: R10: ffffc900048dbe10 R11: ffffc900048dbe94 R12: 0000000000000000
Nov 16 14:50:41 Valhalla kernel: R13: ffffc900048dbe90 R14: ffffc900048dbe20 R15: 0000000000000000
Nov 16 14:50:41 Valhalla kernel: FS:  000015383bd00740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:50:41 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:50:41 Valhalla kernel: CR2: 0000000000000038 CR3: 00000001a5628000 CR4: 0000000000752ee0
Nov 16 14:50:41 Valhalla kernel: PKRU: 55555554
Nov 16 14:50:41 Valhalla kernel: note: find[19730] exited with irqs disabled
Nov 16 14:55:32 Valhalla kernel: smartctl_type[32706]: segfault at 201 ip 0000000000000201 sp 00007fff29b27428 error 14 in php[400000+3b000] likely on CPU 13 (core 24, socket 0)
Nov 16 14:55:32 Valhalla kernel: Code: Unable to access opcode bytes at 0x1d7.
Nov 16 14:56:48 Valhalla kernel: BUG: kernel NULL pointer dereference, address: 0000000000000038
Nov 16 14:56:48 Valhalla kernel: #PF: supervisor read access in kernel mode
Nov 16 14:56:48 Valhalla kernel: #PF: error_code(0x0000) - not-present page
Nov 16 14:56:48 Valhalla kernel: PGD 141a9b067 P4D 141a9b067 PUD 25e2ca067 PMD 0 
Nov 16 14:56:48 Valhalla kernel: Oops: 0000 [#37] PREEMPT SMP NOPTI
Nov 16 14:56:48 Valhalla kernel: CPU: 12 PID: 3276 Comm: find Tainted: P      D    O       6.1.49-Unraid #1
Nov 16 14:56:48 Valhalla kernel: Hardware name: Gigabyte Technology Co., Ltd. Z690 AORUS MASTER/Z690 AORUS MASTER, BIOS F28b 11/09/2023
Nov 16 14:56:48 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:56:48 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:56:48 Valhalla kernel: RSP: 0018:ffffc9002b623dd0 EFLAGS: 00010202
Nov 16 14:56:48 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:56:48 Valhalla kernel: RDX: ffffc9002b623e20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:56:48 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:56:48 Valhalla kernel: R10: ffffc9002b623e10 R11: ffffc9002b623e94 R12: 0000000000000000
Nov 16 14:56:48 Valhalla kernel: R13: ffffc9002b623e90 R14: ffffc9002b623e20 R15: 0000000000000000
Nov 16 14:56:48 Valhalla kernel: FS:  000015151aaed740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:56:48 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:56:48 Valhalla kernel: CR2: 0000000000000038 CR3: 000000054464c000 CR4: 0000000000752ee0
Nov 16 14:56:48 Valhalla kernel: PKRU: 55555554
Nov 16 14:56:48 Valhalla kernel: Call Trace:
Nov 16 14:56:48 Valhalla kernel: <TASK>
Nov 16 14:56:48 Valhalla kernel: ? __die_body+0x1a/0x5c
Nov 16 14:56:48 Valhalla kernel: ? page_fault_oops+0x329/0x376
Nov 16 14:56:48 Valhalla kernel: ? do_user_addr_fault+0x12e/0x48d
Nov 16 14:56:48 Valhalla kernel: ? exc_page_fault+0xfb/0x11d
Nov 16 14:56:48 Valhalla kernel: ? asm_exc_page_fault+0x22/0x30
Nov 16 14:56:48 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:56:48 Valhalla kernel: ? memcg_slab_free_hook+0x28/0xcf
Nov 16 14:56:48 Valhalla kernel: kmem_cache_free+0xb7/0x154
Nov 16 14:56:48 Valhalla kernel: ? vfs_fstatat+0x52/0x62
Nov 16 14:56:48 Valhalla kernel: vfs_fstatat+0x52/0x62
Nov 16 14:56:48 Valhalla kernel: __do_sys_newfstatat+0x26/0x5c
Nov 16 14:56:48 Valhalla kernel: do_syscall_64+0x68/0x81
Nov 16 14:56:48 Valhalla kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Nov 16 14:56:48 Valhalla kernel: RIP: 0033:0x15151abf31ca
Nov 16 14:56:48 Valhalla kernel: Code: 48 89 f2 b9 00 01 00 00 48 89 fe bf 9c ff ff ff e9 0b 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 90 41 89 ca b8 06 01 00 00 0f 05 <3d> 00 f0 ff ff 77 07 31 c0 c3 0f 1f 40 00 48 8b 15 19 4c 0e 00 f7
Nov 16 14:56:48 Valhalla kernel: RSP: 002b:00007ffc52a4b308 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
Nov 16 14:56:48 Valhalla kernel: RAX: ffffffffffffffda RBX: 0000000000494f20 RCX: 000015151abf31ca
Nov 16 14:56:48 Valhalla kernel: RDX: 0000000000494f90 RSI: 0000000000495020 RDI: 000000000000000a
Nov 16 14:56:48 Valhalla kernel: RBP: 0000000000494f90 R08: 0000000000000007 R09: 0000000000499b60
Nov 16 14:56:48 Valhalla kernel: R10: 0000000000000100 R11: 0000000000000246 R12: 0000000000494f20
Nov 16 14:56:48 Valhalla kernel: R13: 0000000000499b60 R14: 00007ffc52a4bf15 R15: 0000000000000004
Nov 16 14:56:48 Valhalla kernel: </TASK>
Nov 16 14:56:48 Valhalla kernel: Modules linked in: rpcsec_gss_krb5 xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net vhost vhost_iotlb tap tun xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nct7904 watchdog coretemp xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper kvm drm_kms_helper drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel btusb sr_mod crypto_simd cdrom mei_hdcp mei_pxp btrtl
Nov 16 14:56:48 Valhalla kernel: intel_gtt i2c_i801 cryptd btbcm rapl intel_cstate mpt3sas btintel gigabyte_wmi wmi_bmof mxm_wmi nvme agpgart mei_me i2c_smbus raid_class input_leds bluetooth intel_uncore ahci atlantic nvme_core scsi_transport_sas joydev i2c_core led_class mei libahci ecdh_generic syscopyarea ecc sysfillrect sysimgblt thermal fb_sys_fops fan tpm_crb video tpm_tis tpm_tis_core tpm wmi intel_pmc_core backlight acpi_pad acpi_tad button unix [last unloaded: coretemp]
Nov 16 14:56:48 Valhalla kernel: CR2: 0000000000000038
Nov 16 14:56:48 Valhalla kernel: ---[ end trace 0000000000000000 ]---
Nov 16 14:56:48 Valhalla kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf
Nov 16 14:56:48 Valhalla kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41
Nov 16 14:56:48 Valhalla kernel: RSP: 0018:ffffc900048dbdd0 EFLAGS: 00010202
Nov 16 14:56:48 Valhalla kernel: RAX: 0000000000000001 RBX: ffff8881001e4e00 RCX: 0000000000000001
Nov 16 14:56:48 Valhalla kernel: RDX: ffffc900048dbe20 RSI: 0000000000000000 RDI: ffff8881001e4e00
Nov 16 14:56:48 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff812463cc
Nov 16 14:56:48 Valhalla kernel: R10: ffffc900048dbe10 R11: ffffc900048dbe94 R12: 0000000000000000
Nov 16 14:56:48 Valhalla kernel: R13: ffffc900048dbe90 R14: ffffc900048dbe20 R15: 0000000000000000
Nov 16 14:56:48 Valhalla kernel: FS:  000015151aaed740(0000) GS:ffff88a03f700000(0000) knlGS:0000000000000000
Nov 16 14:56:48 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 16 14:56:48 Valhalla kernel: CR2: 0000000000000038 CR3: 000000054464c000 CR4: 0000000000752ee0
Nov 16 14:56:48 Valhalla kernel: PKRU: 55555554
Nov 16 14:56:48 Valhalla kernel: note: find[3276] exited with irqs disabled

CRASHED

 

Had a spare set of RAM so I chucked that in to see if it helped, and unfortunately no it did not. Still seeing these errors. It always seems to be address: 0000000000000038 that has the error, though I'm not sure if that's indicative of anything.

I also experienced a crash very soon after boot. So soon that none of these errors (or any others) appeared in the syslog. I.e. it just shows boot sequence, no errors, then it crashed.

Edited November 17, 20232 yr by humaintenance

You still haven't posted the complete syslog, looking at the errors in contest may help.

My apologies, I didn't realise you wanted the entire syslog, so I only provided snippets from it. See attached.

syslog-192.168.1.31.log

14 hours ago, humaintenance said:

be address: 0000000000000038 that has the error

Most are but there are also different errors, e.g.:

 

Nov 14 15:44:15 Valhalla kernel: general protection fault, probably for non-canonical address 0xff7fffff81e26318: 0000 [#1] PREEMPT SMP NOPTI

 

And the errors are constant, when you tried with the different RAM set I assume you removed the other one? This looks more hardware related to me, if it's not RAM could be board or CPU, I would just try with a single RAM stick, since some boards may have issues with the increased load, if the same try a different single stick, that will basically rule out bad RAM.

2 minutes ago, JorgeB said:

And the errors are constant, when you tried with the different RAM set I assume you removed the other one?

Yes I removed all 4 sticks of RAM and replaced with a completely different set of 4 sticks.

 

ill try with single sticks of RAM like you suggested. Failing that I’ll swap out the mobo with another I’ve got and see if that helps. 

Update:

So I tried each stick individually and also tried them in channel A and then channel B. Each time I was still getting the errors. 
I have swapped out the motherboard and so far so good... No errors no crashes, but still early days at the moment. Will report back if the issue reoccurs.

Thank you JorgeB for your help here.

EDIT:

Have now had it running for 10 hours and not a single error in the syslog, where as I would have had several by now before. Issue was with the motherboard, or perhaps memory compatibility with the motherboard.

Edited November 20, 20232 yr by humaintenance

Unfortunately I seem to have jumped the gun on thinking it was resolved. I'm still seeing the errors and I had a crash last night. The frequency of the errors has decreased a vast amount though.

Only thing left to swap out would be the CPU. Might put the 13700k from my main pc into it and see if the errors still occur.

Worth a shot.

On 11/21/2023 at 10:30 PM, humaintenance said:

Unfortunately I seem to have jumped the gun on thinking it was resolved. I'm still seeing the errors and I had a crash last night. The frequency of the errors has decreased a vast amount though.

Only thing left to swap out would be the CPU. Might put the 13700k from my main pc into it and see if the errors still occur.

Hi, does it works by swap the CPU? I have met this problem about a year, it makes me very unhappy.

On 12/7/2023 at 1:36 PM, 奇幻树海 said:

Hi, does it works by swap the CPU? I have met this problem about a year, it makes me very unhappy.

 

I haven't had a chance to swap the CPU's around yet unfortunately. 

I've been having the same probably for about a year now too. I believe it started around when I upgraded to the 6.11.x versions IIRC. 


At current I'm seeing different (and stranger) behaviour, but I'm still seeing crashes. More recently I've been seeing it kinda "soft" crash where docker my containers seems to still "appear" to be running, but at least from the GUI but then they aren't actually working if I try to access any of the contains webUIs. Then the Unraid UI starts to act weird, where if I try to access certain parts of the UI it won't load, then my entire web session wont load anything. However if I open up and incognito browser window and log back in, it works again, until I try to access that same UI area (usually the Docker tab), then it breaks incognito too (untill I close and reopen a new session). 

I had a crash a little earlier and what I'm seeing in the syslog differs to what I've posted in this thread, but its still surrounding issues with Memory

 

Dec  9 20:02:18 Valhalla kernel: ------------[ cut here ]------------
Dec  9 20:02:18 Valhalla kernel: kernel BUG at drivers/md/unraid.c:1617!
Dec  9 20:02:18 Valhalla kernel: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
Dec  9 20:02:18 Valhalla kernel: CPU: 8 PID: 15870 Comm: unraidd6 Tainted: P           O       6.1.49-Unraid #1
Dec  9 20:02:18 Valhalla kernel: Hardware name: ASUS System Product Name/ProArt Z790-CREATOR WIFI, BIOS 1501 10/06/2023
Dec  9 20:02:18 Valhalla kernel: RIP: 0010:unraidd+0x1051/0x1140 [md_mod]
Dec  9 20:02:18 Valhalla kernel: Code: 00 83 3d 83 50 00 00 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 63 54 a0 48 8b 73 20 e8 db 8a 31 e1 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 10 41 c7 46 b0 00 10 00 00 49 8b 56 10
Dec  9 20:02:18 Valhalla kernel: RSP: 0018:ffffc9000bd57df0 EFLAGS: 00010246
Dec  9 20:02:18 Valhalla kernel: RAX: 0000000000000000 RBX: ffff88816b8ba4b8 RCX: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: RDX: 0000000000000000 RSI: ffffffff829e7f00 RDI: ffff88812c6e9e38
Dec  9 20:02:18 Valhalla kernel: RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8883806b2118
Dec  9 20:02:18 Valhalla kernel: R13: ffff88816b8ba760 R14: ffff88816b8ba7d8 R15: ffff889f146fd398
Dec  9 20:02:18 Valhalla kernel: FS:  0000000000000000(0000) GS:ffff88a03f200000(0000) knlGS:0000000000000000
Dec  9 20:02:18 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  9 20:02:18 Valhalla kernel: CR2: 000014970a4218c8 CR3: 0000000f65fc4000 CR4: 0000000000750ee0
Dec  9 20:02:18 Valhalla kernel: PKRU: 55555554
Dec  9 20:02:18 Valhalla kernel: Call Trace:
Dec  9 20:02:18 Valhalla kernel: <TASK>
Dec  9 20:02:18 Valhalla kernel: ? __die_body+0x1a/0x5c
Dec  9 20:02:18 Valhalla kernel: ? die+0x30/0x49
Dec  9 20:02:18 Valhalla kernel: ? do_trap+0x7b/0xfe
Dec  9 20:02:18 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod]
Dec  9 20:02:18 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod]
Dec  9 20:02:18 Valhalla kernel: ? do_error_trap+0x6e/0x98
Dec  9 20:02:18 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod]
Dec  9 20:02:18 Valhalla kernel: ? exc_invalid_op+0x4c/0x60
Dec  9 20:02:18 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod]
Dec  9 20:02:18 Valhalla kernel: ? asm_exc_invalid_op+0x16/0x20
Dec  9 20:02:18 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod]
Dec  9 20:02:18 Valhalla kernel: md_thread+0xf4/0x122 [md_mod]
Dec  9 20:02:18 Valhalla kernel: ? _raw_spin_rq_lock_irqsave+0x20/0x20
Dec  9 20:02:18 Valhalla kernel: ? signal_pending+0x1d/0x1d [md_mod]
Dec  9 20:02:18 Valhalla kernel: kthread+0xe4/0xef
Dec  9 20:02:18 Valhalla kernel: ? kthread_complete_and_exit+0x1b/0x1b
Dec  9 20:02:18 Valhalla kernel: ret_from_fork+0x1f/0x30
Dec  9 20:02:18 Valhalla kernel: </TASK>
Dec  9 20:02:18 Valhalla kernel: Modules linked in: vhost_net tun vhost tap kvm_intel kvm md_mod ipvlan xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_iotlb wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775 nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet bridge 8021q garp mrp stp llc bonding tls igc atlantic i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper drm_kms_helper btusb btrtl
Dec  9 20:02:18 Valhalla kernel: crct10dif_pclmul btbcm crc32_pclmul crc32c_intel btintel ghash_clmulni_intel sha512_ssse3 drm bluetooth aesni_intel crypto_simd cryptd rapl intel_gtt intel_cstate input_leds ecdh_generic mei_hdcp mei_pxp wmi_bmof joydev led_class ecc i2c_i801 mpt3sas i2c_smbus agpgart nvme mei_me ahci raid_class intel_uncore thunderbolt i2c_core scsi_transport_sas nvme_core mei libahci syscopyarea sysfillrect vmd sysimgblt thermal fb_sys_fops fan video tpm_crb tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_tad acpi_pad button unix [last unloaded: md_mod]
Dec  9 20:02:18 Valhalla kernel: ---[ end trace 0000000000000000 ]---
Dec  9 20:02:18 Valhalla kernel: RIP: 0010:unraidd+0x1051/0x1140 [md_mod]
Dec  9 20:02:18 Valhalla kernel: Code: 00 83 3d 83 50 00 00 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 63 54 a0 48 8b 73 20 e8 db 8a 31 e1 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 10 41 c7 46 b0 00 10 00 00 49 8b 56 10
Dec  9 20:02:18 Valhalla kernel: RSP: 0018:ffffc9000bd57df0 EFLAGS: 00010246
Dec  9 20:02:18 Valhalla kernel: RAX: 0000000000000000 RBX: ffff88816b8ba4b8 RCX: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: RDX: 0000000000000000 RSI: ffffffff829e7f00 RDI: ffff88812c6e9e38
Dec  9 20:02:18 Valhalla kernel: RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8883806b2118
Dec  9 20:02:18 Valhalla kernel: R13: ffff88816b8ba760 R14: ffff88816b8ba7d8 R15: ffff889f146fd398
Dec  9 20:02:18 Valhalla kernel: FS:  0000000000000000(0000) GS:ffff88a03f200000(0000) knlGS:0000000000000000
Dec  9 20:02:18 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  9 20:02:18 Valhalla kernel: CR2: 000014970a4218c8 CR3: 0000000f65fc4000 CR4: 0000000000750ee0
Dec  9 20:02:18 Valhalla kernel: PKRU: 55555554
Dec  9 20:02:18 Valhalla kernel: ------------[ cut here ]------------
Dec  9 20:02:18 Valhalla kernel: WARNING: CPU: 8 PID: 15870 at kernel/exit.c:814 do_exit+0x87/0x923
Dec  9 20:02:18 Valhalla kernel: Modules linked in: vhost_net tun vhost tap kvm_intel kvm md_mod ipvlan xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_iotlb wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775 nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet bridge 8021q garp mrp stp llc bonding tls igc atlantic i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper drm_kms_helper btusb btrtl
Dec  9 20:02:18 Valhalla kernel: crct10dif_pclmul btbcm crc32_pclmul crc32c_intel btintel ghash_clmulni_intel sha512_ssse3 drm bluetooth aesni_intel crypto_simd cryptd rapl intel_gtt intel_cstate input_leds ecdh_generic mei_hdcp mei_pxp wmi_bmof joydev led_class ecc i2c_i801 mpt3sas i2c_smbus agpgart nvme mei_me ahci raid_class intel_uncore thunderbolt i2c_core scsi_transport_sas nvme_core mei libahci syscopyarea sysfillrect vmd sysimgblt thermal fb_sys_fops fan video tpm_crb tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_tad acpi_pad button unix [last unloaded: md_mod]
Dec  9 20:02:18 Valhalla kernel: CPU: 8 PID: 15870 Comm: unraidd6 Tainted: P      D    O       6.1.49-Unraid #1
Dec  9 20:02:18 Valhalla kernel: Hardware name: ASUS System Product Name/ProArt Z790-CREATOR WIFI, BIOS 1501 10/06/2023
Dec  9 20:02:18 Valhalla kernel: RIP: 0010:do_exit+0x87/0x923
Dec  9 20:02:18 Valhalla kernel: Code: 24 74 04 75 13 b8 01 00 00 00 41 89 6c 24 60 48 c1 e0 22 49 89 44 24 70 4c 89 ef e8 e7 ae 80 00 48 83 bb b0 07 00 00 00 74 02 <0f> 0b 48 8b bb d8 06 00 00 e8 e9 ad 80 00 48 8b 83 d0 06 00 00 83
Dec  9 20:02:18 Valhalla kernel: RSP: 0018:ffffc9000bd57ee0 EFLAGS: 00010286
Dec  9 20:02:18 Valhalla kernel: RAX: 0000000000000000 RBX: ffff8892920f5000 RCX: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: RDX: 0000000000000001 RSI: 0000000000002710 RDI: 00000000ffffffff
Dec  9 20:02:18 Valhalla kernel: RBP: 000000000000000b R08: 0000000000000000 R09: 0000000000aaaaaa
Dec  9 20:02:18 Valhalla kernel: R10: 0000000000000001 R11: 0000000000000001 R12: ffff888107785000
Dec  9 20:02:18 Valhalla kernel: R13: ffff888109516300 R14: 0000000000000002 R15: ffffffff820c6fef
Dec  9 20:02:18 Valhalla kernel: FS:  0000000000000000(0000) GS:ffff88a03f200000(0000) knlGS:0000000000000000
Dec  9 20:02:18 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  9 20:02:18 Valhalla kernel: CR2: 000014970a4218c8 CR3: 0000000f65fc4000 CR4: 0000000000750ee0
Dec  9 20:02:18 Valhalla kernel: PKRU: 55555554
Dec  9 20:02:18 Valhalla kernel: Call Trace:
Dec  9 20:02:18 Valhalla kernel: <TASK>
Dec  9 20:02:18 Valhalla kernel: ? __warn+0xab/0x122
Dec  9 20:02:18 Valhalla kernel: ? report_bug+0x109/0x17e
Dec  9 20:02:18 Valhalla kernel: ? do_exit+0x87/0x923
Dec  9 20:02:18 Valhalla kernel: ? handle_bug+0x41/0x6f
Dec  9 20:02:18 Valhalla kernel: ? exc_invalid_op+0x13/0x60
Dec  9 20:02:18 Valhalla kernel: ? asm_exc_invalid_op+0x16/0x20
Dec  9 20:02:18 Valhalla kernel: ? do_exit+0x87/0x923
Dec  9 20:02:18 Valhalla kernel: make_task_dead+0x11c/0x11c
Dec  9 20:02:18 Valhalla kernel: rewind_stack_and_make_dead+0x17/0x17
Dec  9 20:02:18 Valhalla kernel: RIP: 0000:0x0
Dec  9 20:02:18 Valhalla kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6.
Dec  9 20:02:18 Valhalla kernel: RSP: 0000:0000000000000000 EFLAGS: 00000000 ORIG_RAX: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: </TASK>
Dec  9 20:02:18 Valhalla kernel: ---[ end trace 0000000000000000 ]---

 

I will try to find the time to swap out the CPU's.

 

On 12/10/2023 at 7:49 AM, humaintenance said:

 

I haven't had a chance to swap the CPU's around yet unfortunately. 

I've been having the same probably for about a year now too. I believe it started around when I upgraded to the 6.11.x versions IIRC. 


At current I'm seeing different (and stranger) behaviour, but I'm still seeing crashes. More recently I've been seeing it kinda "soft" crash where docker my containers seems to still "appear" to be running, but at least from the GUI but then they aren't actually working if I try to access any of the contains webUIs. Then the Unraid UI starts to act weird, where if I try to access certain parts of the UI it won't load, then my entire web session wont load anything. However if I open up and incognito browser window and log back in, it works again, until I try to access that same UI area (usually the Docker tab), then it breaks incognito too (untill I close and reopen a new session). 

I had a crash a little earlier and what I'm seeing in the syslog differs to what I've posted in this thread, but its still surrounding issues with Memory

 

Dec  9 20:02:18 Valhalla kernel: ------------[ cut here ]------------
Dec  9 20:02:18 Valhalla kernel: kernel BUG at drivers/md/unraid.c:1617!
Dec  9 20:02:18 Valhalla kernel: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
Dec  9 20:02:18 Valhalla kernel: CPU: 8 PID: 15870 Comm: unraidd6 Tainted: P           O       6.1.49-Unraid #1
Dec  9 20:02:18 Valhalla kernel: Hardware name: ASUS System Product Name/ProArt Z790-CREATOR WIFI, BIOS 1501 10/06/2023
Dec  9 20:02:18 Valhalla kernel: RIP: 0010:unraidd+0x1051/0x1140 [md_mod]
Dec  9 20:02:18 Valhalla kernel: Code: 00 83 3d 83 50 00 00 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 63 54 a0 48 8b 73 20 e8 db 8a 31 e1 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 10 41 c7 46 b0 00 10 00 00 49 8b 56 10
Dec  9 20:02:18 Valhalla kernel: RSP: 0018:ffffc9000bd57df0 EFLAGS: 00010246
Dec  9 20:02:18 Valhalla kernel: RAX: 0000000000000000 RBX: ffff88816b8ba4b8 RCX: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: RDX: 0000000000000000 RSI: ffffffff829e7f00 RDI: ffff88812c6e9e38
Dec  9 20:02:18 Valhalla kernel: RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8883806b2118
Dec  9 20:02:18 Valhalla kernel: R13: ffff88816b8ba760 R14: ffff88816b8ba7d8 R15: ffff889f146fd398
Dec  9 20:02:18 Valhalla kernel: FS:  0000000000000000(0000) GS:ffff88a03f200000(0000) knlGS:0000000000000000
Dec  9 20:02:18 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  9 20:02:18 Valhalla kernel: CR2: 000014970a4218c8 CR3: 0000000f65fc4000 CR4: 0000000000750ee0
Dec  9 20:02:18 Valhalla kernel: PKRU: 55555554
Dec  9 20:02:18 Valhalla kernel: Call Trace:
Dec  9 20:02:18 Valhalla kernel: <TASK>
Dec  9 20:02:18 Valhalla kernel: ? __die_body+0x1a/0x5c
Dec  9 20:02:18 Valhalla kernel: ? die+0x30/0x49
Dec  9 20:02:18 Valhalla kernel: ? do_trap+0x7b/0xfe
Dec  9 20:02:18 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod]
Dec  9 20:02:18 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod]
Dec  9 20:02:18 Valhalla kernel: ? do_error_trap+0x6e/0x98
Dec  9 20:02:18 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod]
Dec  9 20:02:18 Valhalla kernel: ? exc_invalid_op+0x4c/0x60
Dec  9 20:02:18 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod]
Dec  9 20:02:18 Valhalla kernel: ? asm_exc_invalid_op+0x16/0x20
Dec  9 20:02:18 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod]
Dec  9 20:02:18 Valhalla kernel: md_thread+0xf4/0x122 [md_mod]
Dec  9 20:02:18 Valhalla kernel: ? _raw_spin_rq_lock_irqsave+0x20/0x20
Dec  9 20:02:18 Valhalla kernel: ? signal_pending+0x1d/0x1d [md_mod]
Dec  9 20:02:18 Valhalla kernel: kthread+0xe4/0xef
Dec  9 20:02:18 Valhalla kernel: ? kthread_complete_and_exit+0x1b/0x1b
Dec  9 20:02:18 Valhalla kernel: ret_from_fork+0x1f/0x30
Dec  9 20:02:18 Valhalla kernel: </TASK>
Dec  9 20:02:18 Valhalla kernel: Modules linked in: vhost_net tun vhost tap kvm_intel kvm md_mod ipvlan xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_iotlb wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775 nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet bridge 8021q garp mrp stp llc bonding tls igc atlantic i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper drm_kms_helper btusb btrtl
Dec  9 20:02:18 Valhalla kernel: crct10dif_pclmul btbcm crc32_pclmul crc32c_intel btintel ghash_clmulni_intel sha512_ssse3 drm bluetooth aesni_intel crypto_simd cryptd rapl intel_gtt intel_cstate input_leds ecdh_generic mei_hdcp mei_pxp wmi_bmof joydev led_class ecc i2c_i801 mpt3sas i2c_smbus agpgart nvme mei_me ahci raid_class intel_uncore thunderbolt i2c_core scsi_transport_sas nvme_core mei libahci syscopyarea sysfillrect vmd sysimgblt thermal fb_sys_fops fan video tpm_crb tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_tad acpi_pad button unix [last unloaded: md_mod]
Dec  9 20:02:18 Valhalla kernel: ---[ end trace 0000000000000000 ]---
Dec  9 20:02:18 Valhalla kernel: RIP: 0010:unraidd+0x1051/0x1140 [md_mod]
Dec  9 20:02:18 Valhalla kernel: Code: 00 83 3d 83 50 00 00 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 63 54 a0 48 8b 73 20 e8 db 8a 31 e1 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 10 41 c7 46 b0 00 10 00 00 49 8b 56 10
Dec  9 20:02:18 Valhalla kernel: RSP: 0018:ffffc9000bd57df0 EFLAGS: 00010246
Dec  9 20:02:18 Valhalla kernel: RAX: 0000000000000000 RBX: ffff88816b8ba4b8 RCX: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: RDX: 0000000000000000 RSI: ffffffff829e7f00 RDI: ffff88812c6e9e38
Dec  9 20:02:18 Valhalla kernel: RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8883806b2118
Dec  9 20:02:18 Valhalla kernel: R13: ffff88816b8ba760 R14: ffff88816b8ba7d8 R15: ffff889f146fd398
Dec  9 20:02:18 Valhalla kernel: FS:  0000000000000000(0000) GS:ffff88a03f200000(0000) knlGS:0000000000000000
Dec  9 20:02:18 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  9 20:02:18 Valhalla kernel: CR2: 000014970a4218c8 CR3: 0000000f65fc4000 CR4: 0000000000750ee0
Dec  9 20:02:18 Valhalla kernel: PKRU: 55555554
Dec  9 20:02:18 Valhalla kernel: ------------[ cut here ]------------
Dec  9 20:02:18 Valhalla kernel: WARNING: CPU: 8 PID: 15870 at kernel/exit.c:814 do_exit+0x87/0x923
Dec  9 20:02:18 Valhalla kernel: Modules linked in: vhost_net tun vhost tap kvm_intel kvm md_mod ipvlan xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_iotlb wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775 nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet bridge 8021q garp mrp stp llc bonding tls igc atlantic i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper drm_kms_helper btusb btrtl
Dec  9 20:02:18 Valhalla kernel: crct10dif_pclmul btbcm crc32_pclmul crc32c_intel btintel ghash_clmulni_intel sha512_ssse3 drm bluetooth aesni_intel crypto_simd cryptd rapl intel_gtt intel_cstate input_leds ecdh_generic mei_hdcp mei_pxp wmi_bmof joydev led_class ecc i2c_i801 mpt3sas i2c_smbus agpgart nvme mei_me ahci raid_class intel_uncore thunderbolt i2c_core scsi_transport_sas nvme_core mei libahci syscopyarea sysfillrect vmd sysimgblt thermal fb_sys_fops fan video tpm_crb tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_tad acpi_pad button unix [last unloaded: md_mod]
Dec  9 20:02:18 Valhalla kernel: CPU: 8 PID: 15870 Comm: unraidd6 Tainted: P      D    O       6.1.49-Unraid #1
Dec  9 20:02:18 Valhalla kernel: Hardware name: ASUS System Product Name/ProArt Z790-CREATOR WIFI, BIOS 1501 10/06/2023
Dec  9 20:02:18 Valhalla kernel: RIP: 0010:do_exit+0x87/0x923
Dec  9 20:02:18 Valhalla kernel: Code: 24 74 04 75 13 b8 01 00 00 00 41 89 6c 24 60 48 c1 e0 22 49 89 44 24 70 4c 89 ef e8 e7 ae 80 00 48 83 bb b0 07 00 00 00 74 02 <0f> 0b 48 8b bb d8 06 00 00 e8 e9 ad 80 00 48 8b 83 d0 06 00 00 83
Dec  9 20:02:18 Valhalla kernel: RSP: 0018:ffffc9000bd57ee0 EFLAGS: 00010286
Dec  9 20:02:18 Valhalla kernel: RAX: 0000000000000000 RBX: ffff8892920f5000 RCX: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: RDX: 0000000000000001 RSI: 0000000000002710 RDI: 00000000ffffffff
Dec  9 20:02:18 Valhalla kernel: RBP: 000000000000000b R08: 0000000000000000 R09: 0000000000aaaaaa
Dec  9 20:02:18 Valhalla kernel: R10: 0000000000000001 R11: 0000000000000001 R12: ffff888107785000
Dec  9 20:02:18 Valhalla kernel: R13: ffff888109516300 R14: 0000000000000002 R15: ffffffff820c6fef
Dec  9 20:02:18 Valhalla kernel: FS:  0000000000000000(0000) GS:ffff88a03f200000(0000) knlGS:0000000000000000
Dec  9 20:02:18 Valhalla kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  9 20:02:18 Valhalla kernel: CR2: 000014970a4218c8 CR3: 0000000f65fc4000 CR4: 0000000000750ee0
Dec  9 20:02:18 Valhalla kernel: PKRU: 55555554
Dec  9 20:02:18 Valhalla kernel: Call Trace:
Dec  9 20:02:18 Valhalla kernel: <TASK>
Dec  9 20:02:18 Valhalla kernel: ? __warn+0xab/0x122
Dec  9 20:02:18 Valhalla kernel: ? report_bug+0x109/0x17e
Dec  9 20:02:18 Valhalla kernel: ? do_exit+0x87/0x923
Dec  9 20:02:18 Valhalla kernel: ? handle_bug+0x41/0x6f
Dec  9 20:02:18 Valhalla kernel: ? exc_invalid_op+0x13/0x60
Dec  9 20:02:18 Valhalla kernel: ? asm_exc_invalid_op+0x16/0x20
Dec  9 20:02:18 Valhalla kernel: ? do_exit+0x87/0x923
Dec  9 20:02:18 Valhalla kernel: make_task_dead+0x11c/0x11c
Dec  9 20:02:18 Valhalla kernel: rewind_stack_and_make_dead+0x17/0x17
Dec  9 20:02:18 Valhalla kernel: RIP: 0000:0x0
Dec  9 20:02:18 Valhalla kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6.
Dec  9 20:02:18 Valhalla kernel: RSP: 0000:0000000000000000 EFLAGS: 00000000 ORIG_RAX: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  9 20:02:18 Valhalla kernel: </TASK>
Dec  9 20:02:18 Valhalla kernel: ---[ end trace 0000000000000000 ]---

 

I will try to find the time to swap out the CPU's.

 

I have tried change the unraid version from 6.12.5 to 6.9.2 last week, but the unraid also crashed when I am seeing a movie by jellyfin docker and the machine didn't repsonse at all, but this time, unraid didn't print anything about the error trace log, so I don't know what happened to it. I wonder if the hard disk's problem will cause unraid's crash like top, I have 7*16TB hard disks installed.

So I still have not swapped out the CPU, but I've also not received a crash or error in a while now... Not to say I still won't get one, but its been a "relatively" long time now where I've not had an error/crash. Before I couldn't go like 2/3 days without having a crash, but I have not had an error now for 8 days.

Things I have changed since the last crash:
1) upgraded to 6.12.6
2) Removed A LOT of data from my array (roughly 50TBs). I did this for unrelated reasons, but I've currently condensed my array from 15 drives down to 8.

Continuing to monitor things, and will be looking to reintroduce some of the drives I'm removed at some point.

On 12/23/2023 at 10:45 AM, humaintenance said:

So I still have not swapped out the CPU, but I've also not received a crash or error in a while now... Not to say I still won't get one, but its been a "relatively" long time now where I've not had an error/crash. Before I couldn't go like 2/3 days without having a crash, but I have not had an error now for 8 days.

Things I have changed since the last crash:
1) upgraded to 6.12.6
2) Removed A LOT of data from my array (roughly 50TBs). I did this for unrelated reasons, but I've currently condensed my array from 15 drives down to 8.

Continuing to monitor things, and will be looking to reintroduce some of the drives I'm removed at some point.

I also found remove a lot of data (about 3TB, include some of tiny files) from unraid significantly decrease the crash rate, when I copy them back to the unraid, crash increase again, so I wonder if the data or hard disk's problem will cause unraid crash, hope official workers of unraid could give some ideas.

I have resoleved this problem after 1.5 years later!! it's the memory cause this problem, after replaced the memory to Kinston fury, the problem has gone.