humaintenance

So I didn't rollback to an earlier release yet, I instead did things like run memtest (parallel) for a day), all fine. I then saw 6.12.10 was released and it was a kernel rollback. Since upgrading to .10 I've not had it crash on me, so potentially that was it, though I'm still cautious so monitoring it closely. Will report back if I get the issue again. Edit Thursday 11th: So I fairly confident the issues I was having were related to the kernel update, as I've had no crash since updating to 6.12.10. I've run a successful parity sync/corrections all while having docker engine on and dockers running. Gonna mark this as resolved.

So the downgrade available to me through the UI is 6.12.8 which also had the issue. I found 6.11.5 via https://docs.unraid.net/unraid-os/download_list/, so I'll use that to roll back and test.

I have been experiencing numerous crashes and hanging of the system. I noticed recently that it appears to be whenever I've got VM's and/or docker enabled. If parity check is running and VM's are enabled then parity check speed lowers drastically (i'm aware this expected behaviour), but I also then start seeing kernel panics in syslog and the parity check gets stuck and cannot be paused/cancelled/killed forcing me to hard reset the system. If I disable VM and docker, the parity check will run till completion, and similarly if parity check is not running then VM's and docker will run without issue. If I then turn on either docker or VM I seem to immediately trigger a kernel panic and parity check crashes. This just happened to me. I had parity check running with VM and Docker disabled and it was chugging along nicely for hours. I then turned on VM to grab some config off it quickly (thinking if I'm quick enough it'd be OK), but it instantly triggered the issue. I was still able to interact with the UI and SSH, but I couldn't kill parity check, so I downloaded diagnostics (attached) and hard reset the system. I've never had an issue before with parity checks whilst Docker is enabled, and I don't want to have to turn off everything for a day for me to run my monthly parity checks. Any help you can provide is greatly appreciated Here is the syslogged kernel panic: Apr 2 14:06:13 Valhalla kernel: ------------[ cut here ]------------ Apr 2 14:06:13 Valhalla kernel: kernel BUG at drivers/md/unraid.c:1617! Apr 2 14:06:13 Valhalla kernel: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI Apr 2 14:06:13 Valhalla kernel: CPU: 8 PID: 14807 Comm: unraidd0 Tainted: P O 6.1.82-Unraid #1 Apr 2 14:06:13 Valhalla kernel: Hardware name: ASUS System Product Name/ProArt Z790-CREATOR WIFI, BIOS 2102 03/15/2024 Apr 2 14:06:13 Valhalla kernel: RIP: 0010:unraidd+0x1051/0x1140 [md_mod] Apr 2 14:06:13 Valhalla kernel: Code: 00 83 3d 83 50 00 00 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 f3 72 a0 48 8b 73 20 e8 fb 56 13 e1 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 10 41 c7 46 b0 00 10 00 00 49 8b 56 10 Apr 2 14:06:13 Valhalla kernel: RSP: 0018:ffffc9000b32fdf0 EFLAGS: 00010246 Apr 2 14:06:13 Valhalla kernel: RAX: 0000000000000000 RBX: ffff888182bfe4d8 RCX: 0000000000000000 Apr 2 14:06:13 Valhalla kernel: RDX: 0000000000000000 RSI: ffffffff829ec720 RDI: ffff888101afe038 Apr 2 14:06:13 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 Apr 2 14:06:13 Valhalla kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff88812c450108 Apr 2 14:06:13 Valhalla kernel: R13: ffff888182bfe620 R14: ffff888182bfe698 R15: ffff88816ee41218 Apr 2 14:06:13 Valhalla kernel: FS: 0000000000000000(0000) GS:ffff88a03f200000(0000) knlGS:0000000000000000 Apr 2 14:06:13 Valhalla kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 2 14:06:13 Valhalla kernel: CR2: 0000150729e808c8 CR3: 000000000420a000 CR4: 0000000000750ee0 Apr 2 14:06:13 Valhalla kernel: PKRU: 55555554 Apr 2 14:06:13 Valhalla kernel: Call Trace: Apr 2 14:06:13 Valhalla kernel: <TASK> Apr 2 14:06:13 Valhalla kernel: ? __die_body+0x1a/0x5c Apr 2 14:06:13 Valhalla kernel: ? die+0x30/0x49 Apr 2 14:06:13 Valhalla kernel: ? do_trap+0x7b/0xfe Apr 2 14:06:13 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod] ### [PREVIOUS LINE REPEATED 1 TIMES] ### Apr 2 14:06:13 Valhalla kernel: ? do_error_trap+0x6e/0x98 Apr 2 14:06:13 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod] Apr 2 14:06:13 Valhalla kernel: ? exc_invalid_op+0x4c/0x60 Apr 2 14:06:13 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod] Apr 2 14:06:13 Valhalla kernel: ? asm_exc_invalid_op+0x16/0x20 Apr 2 14:06:13 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod] Apr 2 14:06:13 Valhalla kernel: md_thread+0xf4/0x122 [md_mod] Apr 2 14:06:13 Valhalla kernel: ? _raw_spin_rq_lock_irqsave+0x20/0x20 Apr 2 14:06:13 Valhalla kernel: ? signal_pending+0x1d/0x1d [md_mod] Apr 2 14:06:13 Valhalla kernel: kthread+0xe4/0xef Apr 2 14:06:13 Valhalla kernel: ? kthread_complete_and_exit+0x1b/0x1b Apr 2 14:06:13 Valhalla kernel: ret_from_fork+0x1f/0x30 Apr 2 14:06:13 Valhalla kernel: </TASK> Apr 2 14:06:13 Valhalla kernel: Modules linked in: vhost_net tun vhost tap kvm_intel kvm xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_iotlb veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) nfsd auth_rpcgss oid_registry lockd grace sunrpc tcp_diag inet_diag nct6775 nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls igc atlantic i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp iosf_mbi drm_buddy i2c_algo_bit ttm btusb btrtl btbcm drm_display_helper btintel drm_kms_helper bluetooth drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel sr_mod Apr 2 14:06:13 Valhalla kernel: crypto_simd intel_gtt cryptd rapl ecdh_generic input_leds intel_cstate mei_hdcp mei_pxp wmi_bmof cdrom joydev led_class ecc mpt3sas intel_uncore i2c_i801 thunderbolt agpgart nvme i2c_smbus ahci mei_me raid_class scsi_transport_sas i2c_core nvme_core libahci mei syscopyarea sysfillrect video vmd sysimgblt fb_sys_fops thermal fan tpm_crb tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_pad acpi_tad button unix [last unloaded: kvm] Apr 2 14:06:13 Valhalla kernel: ---[ end trace 0000000000000000 ]--- Apr 2 14:06:13 Valhalla kernel: RIP: 0010:unraidd+0x1051/0x1140 [md_mod] Apr 2 14:06:13 Valhalla kernel: Code: 00 83 3d 83 50 00 00 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 f3 72 a0 48 8b 73 20 e8 fb 56 13 e1 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 10 41 c7 46 b0 00 10 00 00 49 8b 56 10 Apr 2 14:06:13 Valhalla kernel: RSP: 0018:ffffc9000b32fdf0 EFLAGS: 00010246 Apr 2 14:06:13 Valhalla kernel: RAX: 0000000000000000 RBX: ffff888182bfe4d8 RCX: 0000000000000000 Apr 2 14:06:13 Valhalla kernel: RDX: 0000000000000000 RSI: ffffffff829ec720 RDI: ffff888101afe038 Apr 2 14:06:13 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 Apr 2 14:06:13 Valhalla kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff88812c450108 Apr 2 14:06:13 Valhalla kernel: R13: ffff888182bfe620 R14: ffff888182bfe698 R15: ffff88816ee41218 Apr 2 14:06:13 Valhalla kernel: FS: 0000000000000000(0000) GS:ffff88a03f200000(0000) knlGS:0000000000000000 Apr 2 14:06:13 Valhalla kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 2 14:06:13 Valhalla kernel: CR2: 0000150729e808c8 CR3: 0000000286308000 CR4: 0000000000750ee0 Apr 2 14:06:13 Valhalla kernel: PKRU: 55555554 Apr 2 14:06:13 Valhalla kernel: ------------[ cut here ]------------ Apr 2 14:06:13 Valhalla kernel: WARNING: CPU: 8 PID: 14807 at kernel/exit.c:814 do_exit+0x87/0x923 Apr 2 14:06:13 Valhalla kernel: Modules linked in: vhost_net tun vhost tap kvm_intel kvm xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_iotlb veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) nfsd auth_rpcgss oid_registry lockd grace sunrpc tcp_diag inet_diag nct6775 nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls igc atlantic i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp iosf_mbi drm_buddy i2c_algo_bit ttm btusb btrtl btbcm drm_display_helper btintel drm_kms_helper bluetooth drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel sr_mod Apr 2 14:06:13 Valhalla kernel: crypto_simd intel_gtt cryptd rapl ecdh_generic input_leds intel_cstate mei_hdcp mei_pxp wmi_bmof cdrom joydev led_class ecc mpt3sas intel_uncore i2c_i801 thunderbolt agpgart nvme i2c_smbus ahci mei_me raid_class scsi_transport_sas i2c_core nvme_core libahci mei syscopyarea sysfillrect video vmd sysimgblt fb_sys_fops thermal fan tpm_crb tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_pad acpi_tad button unix [last unloaded: kvm] Apr 2 14:06:13 Valhalla kernel: CPU: 8 PID: 14807 Comm: unraidd0 Tainted: P D O 6.1.82-Unraid #1 Apr 2 14:06:13 Valhalla kernel: Hardware name: ASUS System Product Name/ProArt Z790-CREATOR WIFI, BIOS 2102 03/15/2024 Apr 2 14:06:13 Valhalla kernel: RIP: 0010:do_exit+0x87/0x923 Apr 2 14:06:13 Valhalla kernel: Code: 24 74 04 75 13 b8 01 00 00 00 41 89 6c 24 60 48 c1 e0 22 49 89 44 24 70 4c 89 ef e8 69 0b 81 00 48 83 bb b0 07 00 00 00 74 02 <0f> 0b 48 8b bb d8 06 00 00 e8 6b 0a 81 00 48 8b 83 d0 06 00 00 83 Apr 2 14:06:13 Valhalla kernel: RSP: 0018:ffffc9000b32fee0 EFLAGS: 00010286 Apr 2 14:06:13 Valhalla kernel: RAX: 0000000000000000 RBX: ffff88815bc4b000 RCX: 0000000000000000 Apr 2 14:06:13 Valhalla kernel: RDX: 0000000000000001 RSI: 0000000000002710 RDI: 00000000ffffffff Apr 2 14:06:13 Valhalla kernel: RBP: 000000000000000b R08: 0000000000000000 R09: 0000000000aaaaaa Apr 2 14:06:13 Valhalla kernel: R10: 0000000000000001 R11: 0000000000000001 R12: ffff88815751b000 Apr 2 14:06:13 Valhalla kernel: R13: ffff88816ee839c0 R14: 0000000000000002 R15: ffffffff820b2735 Apr 2 14:06:13 Valhalla kernel: FS: 0000000000000000(0000) GS:ffff88a03f200000(0000) knlGS:0000000000000000 Apr 2 14:06:13 Valhalla kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 2 14:06:13 Valhalla kernel: CR2: 0000150729e808c8 CR3: 0000000286308000 CR4: 0000000000750ee0 Apr 2 14:06:13 Valhalla kernel: PKRU: 55555554 Apr 2 14:06:13 Valhalla kernel: Call Trace: Apr 2 14:06:13 Valhalla kernel: <TASK> Apr 2 14:06:13 Valhalla kernel: ? __warn+0xab/0x122 Apr 2 14:06:13 Valhalla kernel: ? report_bug+0x109/0x17e Apr 2 14:06:13 Valhalla kernel: ? do_exit+0x87/0x923 Apr 2 14:06:13 Valhalla kernel: ? handle_bug+0x41/0x6f Apr 2 14:06:13 Valhalla kernel: ? exc_invalid_op+0x13/0x60 Apr 2 14:06:13 Valhalla kernel: ? asm_exc_invalid_op+0x16/0x20 Apr 2 14:06:13 Valhalla kernel: ? do_exit+0x87/0x923 Apr 2 14:06:13 Valhalla kernel: make_task_dead+0x11c/0x11c Apr 2 14:06:13 Valhalla kernel: rewind_stack_and_make_dead+0x17/0x17 Apr 2 14:06:13 Valhalla kernel: RIP: 0000:0x0 Apr 2 14:06:13 Valhalla kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6. Apr 2 14:06:13 Valhalla kernel: RSP: 0000:0000000000000000 EFLAGS: 00000000 ORIG_RAX: 0000000000000000 Apr 2 14:06:13 Valhalla kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 Apr 2 14:06:13 Valhalla kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 Apr 2 14:06:13 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 Apr 2 14:06:13 Valhalla kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 Apr 2 14:06:13 Valhalla kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 Apr 2 14:06:13 Valhalla kernel: </TASK> Apr 2 14:06:13 Valhalla kernel: ---[ end trace 0000000000000000 ]--- valhalla-diagnostics-20240402-1425.zip

So I still have not swapped out the CPU, but I've also not received a crash or error in a while now... Not to say I still won't get one, but its been a "relatively" long time now where I've not had an error/crash. Before I couldn't go like 2/3 days without having a crash, but I have not had an error now for 8 days. Things I have changed since the last crash: 1) upgraded to 6.12.6 2) Removed A LOT of data from my array (roughly 50TBs). I did this for unrelated reasons, but I've currently condensed my array from 15 drives down to 8. Continuing to monitor things, and will be looking to reintroduce some of the drives I'm removed at some point.

I haven't had a chance to swap the CPU's around yet unfortunately. I've been having the same probably for about a year now too. I believe it started around when I upgraded to the 6.11.x versions IIRC. At current I'm seeing different (and stranger) behaviour, but I'm still seeing crashes. More recently I've been seeing it kinda "soft" crash where docker my containers seems to still "appear" to be running, but at least from the GUI but then they aren't actually working if I try to access any of the contains webUIs. Then the Unraid UI starts to act weird, where if I try to access certain parts of the UI it won't load, then my entire web session wont load anything. However if I open up and incognito browser window and log back in, it works again, until I try to access that same UI area (usually the Docker tab), then it breaks incognito too (untill I close and reopen a new session). I had a crash a little earlier and what I'm seeing in the syslog differs to what I've posted in this thread, but its still surrounding issues with Memory Dec 9 20:02:18 Valhalla kernel: ------------[ cut here ]------------ Dec 9 20:02:18 Valhalla kernel: kernel BUG at drivers/md/unraid.c:1617! Dec 9 20:02:18 Valhalla kernel: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI Dec 9 20:02:18 Valhalla kernel: CPU: 8 PID: 15870 Comm: unraidd6 Tainted: P O 6.1.49-Unraid #1 Dec 9 20:02:18 Valhalla kernel: Hardware name: ASUS System Product Name/ProArt Z790-CREATOR WIFI, BIOS 1501 10/06/2023 Dec 9 20:02:18 Valhalla kernel: RIP: 0010:unraidd+0x1051/0x1140 [md_mod] Dec 9 20:02:18 Valhalla kernel: Code: 00 83 3d 83 50 00 00 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 63 54 a0 48 8b 73 20 e8 db 8a 31 e1 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 10 41 c7 46 b0 00 10 00 00 49 8b 56 10 Dec 9 20:02:18 Valhalla kernel: RSP: 0018:ffffc9000bd57df0 EFLAGS: 00010246 Dec 9 20:02:18 Valhalla kernel: RAX: 0000000000000000 RBX: ffff88816b8ba4b8 RCX: 0000000000000000 Dec 9 20:02:18 Valhalla kernel: RDX: 0000000000000000 RSI: ffffffff829e7f00 RDI: ffff88812c6e9e38 Dec 9 20:02:18 Valhalla kernel: RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000000 Dec 9 20:02:18 Valhalla kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8883806b2118 Dec 9 20:02:18 Valhalla kernel: R13: ffff88816b8ba760 R14: ffff88816b8ba7d8 R15: ffff889f146fd398 Dec 9 20:02:18 Valhalla kernel: FS: 0000000000000000(0000) GS:ffff88a03f200000(0000) knlGS:0000000000000000 Dec 9 20:02:18 Valhalla kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Dec 9 20:02:18 Valhalla kernel: CR2: 000014970a4218c8 CR3: 0000000f65fc4000 CR4: 0000000000750ee0 Dec 9 20:02:18 Valhalla kernel: PKRU: 55555554 Dec 9 20:02:18 Valhalla kernel: Call Trace: Dec 9 20:02:18 Valhalla kernel: <TASK> Dec 9 20:02:18 Valhalla kernel: ? __die_body+0x1a/0x5c Dec 9 20:02:18 Valhalla kernel: ? die+0x30/0x49 Dec 9 20:02:18 Valhalla kernel: ? do_trap+0x7b/0xfe Dec 9 20:02:18 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod] Dec 9 20:02:18 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod] Dec 9 20:02:18 Valhalla kernel: ? do_error_trap+0x6e/0x98 Dec 9 20:02:18 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod] Dec 9 20:02:18 Valhalla kernel: ? exc_invalid_op+0x4c/0x60 Dec 9 20:02:18 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod] Dec 9 20:02:18 Valhalla kernel: ? asm_exc_invalid_op+0x16/0x20 Dec 9 20:02:18 Valhalla kernel: ? unraidd+0x1051/0x1140 [md_mod] Dec 9 20:02:18 Valhalla kernel: md_thread+0xf4/0x122 [md_mod] Dec 9 20:02:18 Valhalla kernel: ? _raw_spin_rq_lock_irqsave+0x20/0x20 Dec 9 20:02:18 Valhalla kernel: ? signal_pending+0x1d/0x1d [md_mod] Dec 9 20:02:18 Valhalla kernel: kthread+0xe4/0xef Dec 9 20:02:18 Valhalla kernel: ? kthread_complete_and_exit+0x1b/0x1b Dec 9 20:02:18 Valhalla kernel: ret_from_fork+0x1f/0x30 Dec 9 20:02:18 Valhalla kernel: </TASK> Dec 9 20:02:18 Valhalla kernel: Modules linked in: vhost_net tun vhost tap kvm_intel kvm md_mod ipvlan xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_iotlb wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775 nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet bridge 8021q garp mrp stp llc bonding tls igc atlantic i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper drm_kms_helper btusb btrtl Dec 9 20:02:18 Valhalla kernel: crct10dif_pclmul btbcm crc32_pclmul crc32c_intel btintel ghash_clmulni_intel sha512_ssse3 drm bluetooth aesni_intel crypto_simd cryptd rapl intel_gtt intel_cstate input_leds ecdh_generic mei_hdcp mei_pxp wmi_bmof joydev led_class ecc i2c_i801 mpt3sas i2c_smbus agpgart nvme mei_me ahci raid_class intel_uncore thunderbolt i2c_core scsi_transport_sas nvme_core mei libahci syscopyarea sysfillrect vmd sysimgblt thermal fb_sys_fops fan video tpm_crb tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_tad acpi_pad button unix [last unloaded: md_mod] Dec 9 20:02:18 Valhalla kernel: ---[ end trace 0000000000000000 ]--- Dec 9 20:02:18 Valhalla kernel: RIP: 0010:unraidd+0x1051/0x1140 [md_mod] Dec 9 20:02:18 Valhalla kernel: Code: 00 83 3d 83 50 00 00 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 63 54 a0 48 8b 73 20 e8 db 8a 31 e1 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 10 41 c7 46 b0 00 10 00 00 49 8b 56 10 Dec 9 20:02:18 Valhalla kernel: RSP: 0018:ffffc9000bd57df0 EFLAGS: 00010246 Dec 9 20:02:18 Valhalla kernel: RAX: 0000000000000000 RBX: ffff88816b8ba4b8 RCX: 0000000000000000 Dec 9 20:02:18 Valhalla kernel: RDX: 0000000000000000 RSI: ffffffff829e7f00 RDI: ffff88812c6e9e38 Dec 9 20:02:18 Valhalla kernel: RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000000 Dec 9 20:02:18 Valhalla kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8883806b2118 Dec 9 20:02:18 Valhalla kernel: R13: ffff88816b8ba760 R14: ffff88816b8ba7d8 R15: ffff889f146fd398 Dec 9 20:02:18 Valhalla kernel: FS: 0000000000000000(0000) GS:ffff88a03f200000(0000) knlGS:0000000000000000 Dec 9 20:02:18 Valhalla kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Dec 9 20:02:18 Valhalla kernel: CR2: 000014970a4218c8 CR3: 0000000f65fc4000 CR4: 0000000000750ee0 Dec 9 20:02:18 Valhalla kernel: PKRU: 55555554 Dec 9 20:02:18 Valhalla kernel: ------------[ cut here ]------------ Dec 9 20:02:18 Valhalla kernel: WARNING: CPU: 8 PID: 15870 at kernel/exit.c:814 do_exit+0x87/0x923 Dec 9 20:02:18 Valhalla kernel: Modules linked in: vhost_net tun vhost tap kvm_intel kvm md_mod ipvlan xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_iotlb wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775 nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet bridge 8021q garp mrp stp llc bonding tls igc atlantic i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper drm_kms_helper btusb btrtl Dec 9 20:02:18 Valhalla kernel: crct10dif_pclmul btbcm crc32_pclmul crc32c_intel btintel ghash_clmulni_intel sha512_ssse3 drm bluetooth aesni_intel crypto_simd cryptd rapl intel_gtt intel_cstate input_leds ecdh_generic mei_hdcp mei_pxp wmi_bmof joydev led_class ecc i2c_i801 mpt3sas i2c_smbus agpgart nvme mei_me ahci raid_class intel_uncore thunderbolt i2c_core scsi_transport_sas nvme_core mei libahci syscopyarea sysfillrect vmd sysimgblt thermal fb_sys_fops fan video tpm_crb tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_tad acpi_pad button unix [last unloaded: md_mod] Dec 9 20:02:18 Valhalla kernel: CPU: 8 PID: 15870 Comm: unraidd6 Tainted: P D O 6.1.49-Unraid #1 Dec 9 20:02:18 Valhalla kernel: Hardware name: ASUS System Product Name/ProArt Z790-CREATOR WIFI, BIOS 1501 10/06/2023 Dec 9 20:02:18 Valhalla kernel: RIP: 0010:do_exit+0x87/0x923 Dec 9 20:02:18 Valhalla kernel: Code: 24 74 04 75 13 b8 01 00 00 00 41 89 6c 24 60 48 c1 e0 22 49 89 44 24 70 4c 89 ef e8 e7 ae 80 00 48 83 bb b0 07 00 00 00 74 02 <0f> 0b 48 8b bb d8 06 00 00 e8 e9 ad 80 00 48 8b 83 d0 06 00 00 83 Dec 9 20:02:18 Valhalla kernel: RSP: 0018:ffffc9000bd57ee0 EFLAGS: 00010286 Dec 9 20:02:18 Valhalla kernel: RAX: 0000000000000000 RBX: ffff8892920f5000 RCX: 0000000000000000 Dec 9 20:02:18 Valhalla kernel: RDX: 0000000000000001 RSI: 0000000000002710 RDI: 00000000ffffffff Dec 9 20:02:18 Valhalla kernel: RBP: 000000000000000b R08: 0000000000000000 R09: 0000000000aaaaaa Dec 9 20:02:18 Valhalla kernel: R10: 0000000000000001 R11: 0000000000000001 R12: ffff888107785000 Dec 9 20:02:18 Valhalla kernel: R13: ffff888109516300 R14: 0000000000000002 R15: ffffffff820c6fef Dec 9 20:02:18 Valhalla kernel: FS: 0000000000000000(0000) GS:ffff88a03f200000(0000) knlGS:0000000000000000 Dec 9 20:02:18 Valhalla kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Dec 9 20:02:18 Valhalla kernel: CR2: 000014970a4218c8 CR3: 0000000f65fc4000 CR4: 0000000000750ee0 Dec 9 20:02:18 Valhalla kernel: PKRU: 55555554 Dec 9 20:02:18 Valhalla kernel: Call Trace: Dec 9 20:02:18 Valhalla kernel: <TASK> Dec 9 20:02:18 Valhalla kernel: ? __warn+0xab/0x122 Dec 9 20:02:18 Valhalla kernel: ? report_bug+0x109/0x17e Dec 9 20:02:18 Valhalla kernel: ? do_exit+0x87/0x923 Dec 9 20:02:18 Valhalla kernel: ? handle_bug+0x41/0x6f Dec 9 20:02:18 Valhalla kernel: ? exc_invalid_op+0x13/0x60 Dec 9 20:02:18 Valhalla kernel: ? asm_exc_invalid_op+0x16/0x20 Dec 9 20:02:18 Valhalla kernel: ? do_exit+0x87/0x923 Dec 9 20:02:18 Valhalla kernel: make_task_dead+0x11c/0x11c Dec 9 20:02:18 Valhalla kernel: rewind_stack_and_make_dead+0x17/0x17 Dec 9 20:02:18 Valhalla kernel: RIP: 0000:0x0 Dec 9 20:02:18 Valhalla kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6. Dec 9 20:02:18 Valhalla kernel: RSP: 0000:0000000000000000 EFLAGS: 00000000 ORIG_RAX: 0000000000000000 Dec 9 20:02:18 Valhalla kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 Dec 9 20:02:18 Valhalla kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 Dec 9 20:02:18 Valhalla kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 Dec 9 20:02:18 Valhalla kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 Dec 9 20:02:18 Valhalla kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 Dec 9 20:02:18 Valhalla kernel: </TASK> Dec 9 20:02:18 Valhalla kernel: ---[ end trace 0000000000000000 ]--- I will try to find the time to swap out the CPU's.

Unfortunately I seem to have jumped the gun on thinking it was resolved. I'm still seeing the errors and I had a crash last night. The frequency of the errors has decreased a vast amount though. Only thing left to swap out would be the CPU. Might put the 13700k from my main pc into it and see if the errors still occur.

Update: So I tried each stick individually and also tried them in channel A and then channel B. Each time I was still getting the errors. I have swapped out the motherboard and so far so good... No errors no crashes, but still early days at the moment. Will report back if the issue reoccurs. Thank you JorgeB for your help here. EDIT: Have now had it running for 10 hours and not a single error in the syslog, where as I would have had several by now before. Issue was with the motherboard, or perhaps memory compatibility with the motherboard.

Yes I removed all 4 sticks of RAM and replaced with a completely different set of 4 sticks. ill try with single sticks of RAM like you suggested. Failing that I’ll swap out the mobo with another I’ve got and see if that helps.

My apologies, I didn't realise you wanted the entire syslog, so I only provided snippets from it. See attached. syslog-192.168.1.31.log

Had a spare set of RAM so I chucked that in to see if it helped, and unfortunately no it did not. Still seeing these errors. It always seems to be address: 0000000000000038 that has the error, though I'm not sure if that's indicative of anything. I also experienced a crash very soon after boot. So soon that none of these errors (or any others) appeared in the syslog. I.e. it just shows boot sequence, no errors, then it crashed.

Here is the syslog for 30mins leading up to the crash:

Already had the syslog server enabled. It crashed again a little earlier so here is the syslog from the error that was thrown before the crash up to and including the entire boot sequence following the hard reset:

It crashed again last night at about 04:22 This is the last thing written to syslog before it crashed:

Motherboard: Gigbyte Z690 AORUS MASTER CPU: Intel 13900k RAM: Kingston Fury Beast 128gb (4x32) 4800MT DDR5 - set at 4000MT as enabling XMP seems to cause more issues Unraid version: 6.12.4 Over the last few months I have been having intermittent crashes. They have ranged from nearly daily, to sometimes weekly, but each time the system just completely hangs and the server cannot be interacted with in any capacity. I am getting the below error in the syslog every hour or so: I have searched for other cases of similar errors and I've seen a few common "causes", i.e. docker set to macvlan, and docker containers that use libtorrent 2.x. I don't have any torrent containers at the moment (used to have deluge but that got removed a long time ago). I've switched to ipvlan in docker settings. One thing to call out though is that I have containers like radarr/sonarr routed through a gluetunVPN container and Rebuild-DNDC that rebuilds the containers whenever the vpn endpoint changes. I currently have all these turned off whilst I try to diagnose the issue I'm having. I've run memtest86 and it didn't report any issues (unless I turn on XMP and run it, then it has a lot of failures) At this point I'm at a loss for what to do, other than trying different RAM, but thought I'd post here before purchasing any new hardware. Let me know if you need any further details. Hope someone have help out here. Thanks! valhalla-diagnostics-20231115-2043.zip