tknz

Running Unraid 7.0.1 I have been having parity check crashes around the 70% or higher mark after a couple of days. The system otherwise seems stable, and the array functions normally except during parity operations. Once the bug / fault triggers unraid parity check slows down from 100MB/s to a few KB/s but I presume this is because the unraidd thread has crashed or had a panic. I completed a memtest last night with no issues. I do run the new ZFS plugin across 3 NVME cache drives, and I also (have for years) run SR-IOV to virtual GPU pass through to a windows VM running BlueIris CCTV. The Kernel BUG log is extracted from syslog here: May 8 03:44:02 Tower kernel: ------------[ cut here ]------------ May 8 03:44:02 Tower kernel: kernel BUG at drivers/md/unraid.c:1617! May 8 03:44:02 Tower kernel: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI May 8 03:44:02 Tower kernel: CPU: 19 PID: 12987 Comm: unraidd0 Tainted: P U O 6.6.78-Unraid #2 May 8 03:44:02 Tower kernel: Hardware name: Gigabyte Technology Co., Ltd. Z790 UD AX/Z790 UD AX, BIOS F10 05/08/2024 May 8 03:44:02 Tower kernel: RIP: 0010:unraidd+0x1189/0x1278 [md_mod] May 8 03:44:02 Tower kernel: Code: 00 83 3d 04 30 ee ff 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 e3 04 a1 48 8b 73 20 e8 68 15 09 e0 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 18 41 c7 46 b0 00 10 00 00 49 8b 56 10 May 8 03:44:02 Tower kernel: RSP: 0000:ffffc9000f807da8 EFLAGS: 00010246 May 8 03:44:02 Tower kernel: RAX: 0000000000000000 RBX: ffff888250a0c8e8 RCX: 0000000000000000 May 8 03:44:02 Tower kernel: RDX: 0000000000000000 RSI: ffffffff82cbf420 RDI: ffff888108363438 May 8 03:44:02 Tower kernel: RBP: 0000000000000004 R08: 0000000000000000 R09: 0000000000000000 May 8 03:44:02 Tower kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff888114eee928 May 8 03:44:02 Tower kernel: R13: ffff888250a0ccf0 R14: ffff888250a0cd68 R15: ffff8881ad295540 May 8 03:44:02 Tower kernel: FS: 0000000000000000(0000) GS:ffff88a03f8c0000(0000) knlGS:0000000000000000 May 8 03:44:02 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 8 03:44:02 Tower kernel: CR2: 000014942802a000 CR3: 0000000005416000 CR4: 0000000000752ee0 May 8 03:44:02 Tower kernel: PKRU: 55555554 May 8 03:44:02 Tower kernel: Call Trace: May 8 03:44:02 Tower kernel: <TASK> May 8 03:44:02 Tower kernel: ? __die_body+0x1a/0x5c May 8 03:44:02 Tower kernel: ? die+0x30/0x49 May 8 03:44:02 Tower kernel: ? do_trap+0x7b/0xfe May 8 03:44:02 Tower kernel: ? unraidd+0x1189/0x1278 [md_mod] ### [PREVIOUS LINE REPEATED 1 TIMES] ### May 8 03:44:02 Tower kernel: ? do_error_trap+0x6e/0x98 May 8 03:44:02 Tower kernel: ? unraidd+0x1189/0x1278 [md_mod] May 8 03:44:02 Tower kernel: ? exc_invalid_op+0x4c/0x60 May 8 03:44:02 Tower kernel: ? unraidd+0x1189/0x1278 [md_mod] May 8 03:44:02 Tower kernel: ? asm_exc_invalid_op+0x16/0x20 May 8 03:44:02 Tower kernel: ? unraidd+0x1189/0x1278 [md_mod] May 8 03:44:02 Tower kernel: ? unraidd+0x1159/0x1278 [md_mod] May 8 03:44:02 Tower kernel: ? preempt_latency_start+0x2b/0x46 ### [PREVIOUS LINE REPEATED 1 TIMES] ### May 8 03:44:02 Tower kernel: md_thread+0xf7/0x127 [md_mod] May 8 03:44:02 Tower kernel: ? __pfx_autoremove_wake_function+0x10/0x10 May 8 03:44:02 Tower kernel: ? __pfx_md_thread+0x10/0x10 [md_mod] May 8 03:44:02 Tower kernel: kthread+0xf1/0xfc May 8 03:44:02 Tower kernel: ? __pfx_kthread+0x10/0x10 May 8 03:44:02 Tower kernel: ret_from_fork+0x21/0x36 May 8 03:44:02 Tower kernel: ? __pfx_kthread+0x10/0x10 May 8 03:44:02 Tower kernel: ret_from_fork_asm+0x1b/0x30 May 8 03:44:02 Tower kernel: </TASK> May 8 03:44:02 Tower kernel: Modules linked in: cmac cifs cifs_arc4 nls_ucs2_utils cifs_md4 dns_resolver af_packet veth xt_set ip_set nft_chain_nat nft_compat nf_tables nvidia_uvm(PO) xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle iptable_mangle vhost_net tun vhost vhost_iotlb tap xt_nat xt_tcpudp macvlan xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo ip6table_nat iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nfsd auth_rpcgss lockd grace sunrpc md_mod ntfs3 tcp_diag inet_diag i915(O) drm_buddy ttm i2c_algo_bit drm_display_helper intel_gtt agpgart ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge 8021q garp mrp stp llc bonding tls mlx4_en mlx4_core r8169 realtek nvidia_drm(PO) nvidia_modeset(PO) intel_rapl_common iosf_mbi iwlmvm x86_pkg_temp_thermal intel_powerclamp mac80211 coretemp kvm_intel libarc4 kvm nvidia(PO) btusb btrtl btbcm btintel crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 May 8 03:44:02 Tower kernel: bluetooth sha1_ssse3 iwlwifi aesni_intel drm_kms_helper crypto_simd cryptd zfs(PO) input_leds rapl ecdh_generic spl(O) mei_pxp intel_cstate mei_hdcp gigabyte_wmi wmi_bmof joydev led_class ecc cfg80211 drm mpt3sas nvme i2c_i801 intel_uncore mei_me i2c_smbus raid_class ahci nvme_core mei scsi_transport_sas i2c_core libahci rfkill thermal fan tpm_crb video tpm_tis tpm_tis_core wmi tpm backlight acpi_tad acpi_pad button [last unloaded: mlx4_core] May 8 03:44:02 Tower kernel: ---[ end trace 0000000000000000 ]--- May 8 03:44:02 Tower kernel: RIP: 0010:unraidd+0x1189/0x1278 [md_mod] May 8 03:44:02 Tower kernel: Code: 00 83 3d 04 30 ee ff 03 7e 16 41 8b 56 98 89 e9 48 c7 c7 21 e3 04 a1 48 8b 73 20 e8 68 15 09 e0 41 f6 86 69 ff ff ff 02 75 02 <0f> 0b 48 8b 43 20 49 03 47 18 41 c7 46 b0 00 10 00 00 49 8b 56 10 May 8 03:44:02 Tower kernel: RSP: 0000:ffffc9000f807da8 EFLAGS: 00010246 May 8 03:44:02 Tower kernel: RAX: 0000000000000000 RBX: ffff888250a0c8e8 RCX: 0000000000000000 May 8 03:44:02 Tower kernel: RDX: 0000000000000000 RSI: ffffffff82cbf420 RDI: ffff888108363438 May 8 03:44:02 Tower kernel: RBP: 0000000000000004 R08: 0000000000000000 R09: 0000000000000000 May 8 03:44:02 Tower kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff888114eee928 May 8 03:44:02 Tower kernel: R13: ffff888250a0ccf0 R14: ffff888250a0cd68 R15: ffff8881ad295540 May 8 03:44:02 Tower kernel: FS: 0000000000000000(0000) GS:ffff88a03f8c0000(0000) knlGS:0000000000000000 May 8 03:44:02 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 8 03:44:02 Tower kernel: CR2: 000014942802a000 CR3: 0000000674bfe000 CR4: 0000000000752ee0 May 8 03:44:02 Tower kernel: PKRU: 55555554 May 8 03:44:02 Tower kernel: ------------[ cut here ]------------ May 8 03:44:02 Tower kernel: WARNING: CPU: 19 PID: 12987 at kernel/exit.c:820 do_exit+0x81/0x90b May 8 03:44:02 Tower kernel: Modules linked in: cmac cifs cifs_arc4 nls_ucs2_utils cifs_md4 dns_resolver af_packet veth xt_set ip_set nft_chain_nat nft_compat nf_tables nvidia_uvm(PO) xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle iptable_mangle vhost_net tun vhost vhost_iotlb tap xt_nat xt_tcpudp macvlan xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo ip6table_nat iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nfsd auth_rpcgss lockd grace sunrpc md_mod ntfs3 tcp_diag inet_diag i915(O) drm_buddy ttm i2c_algo_bit drm_display_helper intel_gtt agpgart ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge 8021q garp mrp stp llc bonding tls mlx4_en mlx4_core r8169 realtek nvidia_drm(PO) nvidia_modeset(PO) intel_rapl_common iosf_mbi iwlmvm x86_pkg_temp_thermal intel_powerclamp mac80211 coretemp kvm_intel libarc4 kvm nvidia(PO) btusb btrtl btbcm btintel crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 May 8 03:44:02 Tower kernel: bluetooth sha1_ssse3 iwlwifi aesni_intel drm_kms_helper crypto_simd cryptd zfs(PO) input_leds rapl ecdh_generic spl(O) mei_pxp intel_cstate mei_hdcp gigabyte_wmi wmi_bmof joydev led_class ecc cfg80211 drm mpt3sas nvme i2c_i801 intel_uncore mei_me i2c_smbus raid_class ahci nvme_core mei scsi_transport_sas i2c_core libahci rfkill thermal fan tpm_crb video tpm_tis tpm_tis_core wmi tpm backlight acpi_tad acpi_pad button [last unloaded: mlx4_core] May 8 03:44:02 Tower kernel: CPU: 19 PID: 12987 Comm: unraidd0 Tainted: P UD O 6.6.78-Unraid #2 May 8 03:44:02 Tower kernel: Hardware name: Gigabyte Technology Co., Ltd. Z790 UD AX/Z790 UD AX, BIOS F10 05/08/2024 May 8 03:44:02 Tower kernel: RIP: 0010:do_exit+0x81/0x90b May 8 03:44:02 Tower kernel: Code: 24 74 04 75 13 b8 01 00 00 00 41 89 6c 24 60 48 c1 e0 22 49 89 44 24 70 4c 89 ef e8 62 d9 9c 00 48 83 bb c0 07 00 00 00 74 02 <0f> 0b 48 8b bb d8 06 00 00 e8 2c d8 9c 00 48 8b 83 d0 06 00 00 83 May 8 03:44:02 Tower kernel: RSP: 0000:ffffc9000f807ee0 EFLAGS: 00010286 May 8 03:44:02 Tower kernel: RAX: 0000000000000000 RBX: ffff888108305000 RCX: 0000000000000000 May 8 03:44:02 Tower kernel: RDX: 0000000000000001 RSI: 0000000000002710 RDI: 00000000ffffffff May 8 03:44:02 Tower kernel: RBP: 000000000000000b R08: 0000000000000000 R09: ffff8881034a2400 May 8 03:44:02 Tower kernel: R10: 0000000000000001 R11: 0000000000aaaaaa R12: ffff8881097b1980 May 8 03:44:02 Tower kernel: R13: ffff88817a40ca40 R14: 0000000000000002 R15: ffffffff8222c44c May 8 03:44:02 Tower kernel: FS: 0000000000000000(0000) GS:ffff88a03f8c0000(0000) knlGS:0000000000000000 May 8 03:44:02 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 8 03:44:02 Tower kernel: CR2: 000014942802a000 CR3: 0000000674bfe000 CR4: 0000000000752ee0 May 8 03:44:02 Tower kernel: PKRU: 55555554 May 8 03:44:02 Tower kernel: Call Trace: May 8 03:44:02 Tower kernel: <TASK> May 8 03:44:02 Tower kernel: ? __warn+0x99/0x11a May 8 03:44:02 Tower kernel: ? report_bug+0xd9/0x153 May 8 03:44:02 Tower kernel: ? do_exit+0x81/0x90b May 8 03:44:02 Tower kernel: ? handle_bug+0x53/0x7c May 8 03:44:02 Tower kernel: ? exc_invalid_op+0x13/0x60 May 8 03:44:02 Tower kernel: ? asm_exc_invalid_op+0x16/0x20 May 8 03:44:02 Tower kernel: ? do_exit+0x81/0x90b May 8 03:44:02 Tower kernel: ? __pfx_md_thread+0x10/0x10 [md_mod] May 8 03:44:02 Tower kernel: ? kthread+0xf1/0xfc May 8 03:44:02 Tower kernel: make_task_dead+0x113/0x113 May 8 03:44:02 Tower kernel: rewind_stack_and_make_dead+0x17/0x17 May 8 03:44:02 Tower kernel: RIP: 0000:0x0 May 8 03:44:02 Tower kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6. May 8 03:44:02 Tower kernel: RSP: 0000:0000000000000000 EFLAGS: 00000000 ORIG_RAX: 0000000000000000 May 8 03:44:02 Tower kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 May 8 03:44:02 Tower kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 May 8 03:44:02 Tower kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 May 8 03:44:02 Tower kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 May 8 03:44:02 Tower kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 May 8 03:44:02 Tower kernel: </TASK> May 8 03:44:02 Tower kernel: ---[ end trace 0000000000000000 ]--- May 8 03:59:03 Tower emhttpd: spinning down /dev/sdf May 8 03:59:03 Tower emhttpd: spinning down /dev/sdi May 8 03:59:20 Tower emhttpd: spinning down /dev/sdh