[6.12.13] BUG: unable to handle page fault for address: ffff88810a81cc90

Aug  5 04:42:57 NAS kernel: kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
Aug  5 04:42:57 NAS kernel: BUG: unable to handle page fault for address: ffff88810a81cc90
Aug  5 04:42:57 NAS kernel: #PF: supervisor instruction fetch in kernel mode
Aug  5 04:42:57 NAS kernel: #PF: error_code(0x0011) - permissions violation
Aug  5 04:42:57 NAS kernel: PGD 4c01067 P4D 4c01067 PUD 101587063 PMD 15bea6063 PTE 800000010a81c163
Aug  5 04:42:57 NAS kernel: Oops: 0011 [#1] PREEMPT SMP NOPTI
Aug  5 04:42:57 NAS kernel: CPU: 8 PID: 2917 Comm: zfs Tainted: P     U     O       6.1.99-Unraid #1
Aug  5 04:42:57 NAS kernel: Hardware name: ASUS System Product Name/PRIME Z790-P WIFI, BIOS 1402 09/08/2023
Aug  5 04:42:57 NAS kernel: RIP: 0010:0xffff88810a81cc90
Aug  5 04:42:57 NAS kernel: Code: 00 00 68 cc 81 0a 81 88 ff ff 68 cc 81 0a 81 88 ff ff ca 0c 10 00 00 00 00 00 18 3d 8c 6c 81 88 ff ff d8 31 8c 6c 81 88 ff ff <01> 00 00 00 00 00 00 00 00 c0 62 83 82 88 ff ff 00 00 00 00 00 00
Aug  5 04:42:57 NAS kernel: RSP: 0018:ffffc900452f3b88 EFLAGS: 00010246
Aug  5 04:42:57 NAS kernel: RAX: 0000000000000000 RBX: ffff8885cae34cf0 RCX: 0000000000000000
Aug  5 04:42:57 NAS kernel: RDX: 0000000000000000 RSI: 000014e49c4d9000 RDI: 00000000ffffffff
Aug  5 04:42:57 NAS kernel: RBP: 000014e49c4dc000 R08: ffff888638b26a10 R09: 0000000000000000
Aug  5 04:42:57 NAS kernel: R10: ffff888638b27508 R11: ffff888638b2750c R12: ffff8885cae34360
Aug  5 04:42:57 NAS kernel: R13: ffff88857e007ab0 R14: 0000000000000000 R15: ffff8885cae34cf0
Aug  5 04:42:57 NAS kernel: FS:  0000000000000000(0000) GS:ffff88903f400000(0000) knlGS:0000000000000000
Aug  5 04:42:57 NAS kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug  5 04:42:57 NAS kernel: CR2: ffff88810a81cc90 CR3: 000000073dd38003 CR4: 0000000000770ee0
Aug  5 04:42:57 NAS kernel: PKRU: 55555554
Aug  5 04:42:57 NAS kernel: Call Trace:
Aug  5 04:42:57 NAS kernel: <TASK>
Aug  5 04:42:57 NAS kernel: ? __die_body+0x1a/0x5c
Aug  5 04:42:57 NAS kernel: ? page_fault_oops+0x329/0x376
Aug  5 04:42:57 NAS kernel: ? fixup_exception+0x22/0x24b
Aug  5 04:42:57 NAS kernel: ? exc_page_fault+0xf4/0x11d
Aug  5 04:42:57 NAS kernel: ? asm_exc_page_fault+0x22/0x30
Aug  5 04:42:57 NAS kernel: ? __vma_adjust+0x24c/0x7b1
Aug  5 04:42:57 NAS kernel: ? __split_vma+0x105/0x16b
Aug  5 04:42:57 NAS kernel: ? do_mas_align_munmap.constprop.0+0x17a/0x3c6
Aug  5 04:42:57 NAS kernel: ? mmap_region+0xcc/0x61e
Aug  5 04:42:57 NAS kernel: ? shmem_get_unmapped_area+0xc5/0x270
Aug  5 04:42:57 NAS kernel: ? do_mmap+0x3bc/0x428
Aug  5 04:42:57 NAS kernel: ? vm_mmap_pgoff+0xbb/0x112
Aug  5 04:42:57 NAS kernel: ? ksys_mmap_pgoff+0x138/0x166
Aug  5 04:42:57 NAS kernel: ? do_syscall_64+0x65/0x7b
Aug  5 04:42:57 NAS kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Aug  5 04:42:57 NAS kernel: </TASK>
Aug  5 04:42:57 NAS kernel: Modules linked in: xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xfs cmac cifs asn1_decoder cifs_arc4 cifs_md4 dns_resolver nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod tcp_diag inet_diag kvmgt mdev i915 drm_buddy i2c_algo_bit ttm drm_display_helper intel_gtt agpgart iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs 8021q garp mrp bridge stp llc bonding tls atlantic r8169 realtek zfs(PO) nvidia_drm(PO) nvidia_modeset(PO) zunicode(PO) intel_rapl_msr intel_rapl_common zzstd(O) iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel zlua(O) zavl(PO)
Aug  5 04:42:57 NAS kernel: kvm icp(PO) nvidia(PO) btusb crct10dif_pclmul crc32_pclmul btrtl crc32c_intel btbcm ghash_clmulni_intel btintel sha512_ssse3 sha256_ssse3 sha1_ssse3 bluetooth aesni_intel crypto_simd cryptd zcommon(PO) drm_kms_helper znvpair(PO) mei_hdcp mei_pxp i2c_i801 rapl ecdh_generic spl(O) ecc intel_cstate wmi_bmof drm mpt3sas intel_uncore nvme i2c_smbus mei_me ahci raid_class i2c_core nvme_core mei scsi_transport_sas libahci syscopyarea sysfillrect vmd sysimgblt fb_sys_fops thermal fan video tpm_crb tpm_tis tpm_tis_core tpm wmi intel_pmc_core backlight acpi_pad acpi_tad button unix [last unloaded: atlantic]
Aug  5 04:42:57 NAS kernel: CR2: ffff88810a81cc90
Aug  5 04:42:57 NAS kernel: ---[ end trace 0000000000000000 ]---
Aug  5 04:42:57 NAS kernel: RIP: 0010:0xffff88810a81cc90
Aug  5 04:42:57 NAS kernel: Code: 00 00 68 cc 81 0a 81 88 ff ff 68 cc 81 0a 81 88 ff ff ca 0c 10 00 00 00 00 00 18 3d 8c 6c 81 88 ff ff d8 31 8c 6c 81 88 ff ff <03> 00 00 00 00 00 00 00 00 c0 62 83 82 88 ff ff 00 00 00 00 00 00
Aug  5 04:42:57 NAS kernel: RSP: 0018:ffffc900452f3b88 EFLAGS: 00010246
Aug  5 04:42:57 NAS kernel: RAX: 0000000000000000 RBX: ffff8885cae34cf0 RCX: 0000000000000000
Aug  5 04:42:57 NAS kernel: RDX: 0000000000000000 RSI: 000014e49c4d9000 RDI: 00000000ffffffff
Aug  5 04:42:57 NAS kernel: RBP: 000014e49c4dc000 R08: ffff888638b26a10 R09: 0000000000000000
Aug  5 04:42:57 NAS kernel: R10: ffff888638b27508 R11: ffff888638b2750c R12: ffff8885cae34360
Aug  5 04:42:57 NAS kernel: R13: ffff88857e007ab0 R14: 0000000000000000 R15: ffff8885cae34cf0
Aug  5 04:42:57 NAS kernel: FS:  0000000000000000(0000) GS:ffff88903f400000(0000) knlGS:0000000000000000
Aug  5 04:42:57 NAS kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug  5 04:42:57 NAS kernel: CR2: ffff88810a81cc90 CR3: 000000073dd38003 CR4: 0000000000770ee0
Aug  5 04:42:57 NAS kernel: PKRU: 55555554
Aug  5 04:42:57 NAS kernel: note: zfs[2917] exited with irqs disabled

Hi Everyone,

Ever since I updated to 6.12.11 from 6.12.10 I have been experiencing server crashes every night. I turned on syslog finally and caught what appears to be the reason or at least the information about the crash. 

I don't have the diagnostics because I downgraded back down to 6.12.10. I was just looking for common causes of this error. If you needed my diagnostics to have any sort of clue, I apologize. 

I will post the diagnostics if I get the same crash after downgrading to 6.12.10. Thanks.

Edit: Aug 6th 2024

So I downgraded to 6.12.10 and everything is stable again. I think there may be some issue with the latest update for my system. Going to skip it and pray that version 7 works through the bug. Else I will have to consider changing operating systems.

I noticed that this bug may be due to the version of zfs paired with the kernel version from researching online. It seems the bug may be present in the beta versions as well. Unsure of the configuration that leads to this crash but I have no errors from the fix common problems plugin.

Edit: Aug 8th 2024

So I downgraded and my server crashed again with all containers running. Looks like I will enable them one by one to see which is the culprit. My server seems to crash around 5AM consistently. Only thing to note is that my Plex Server does its scheduled tasks from 2AM to 5AM. So going to keep Plex running all day and see what happens.

Edit: Aug 9th 2024

Plex didn't crash the server. Nor did another container I use crash it. I am testing to see if another one is crashing it by enabling it. Will find out eventually.

Edit: Sep 8th 2024

Updated to 6.12.13 and crashed overnight again. Doesn't seem like it was related to the original crash. Syslog isn't being helpful with regards to the crash. Just shows ssh restarting every hour because I configured it to behave like that. Posted my diagnostics from 6.12.13.