DannyR83

Thank you! I found your post here and I will try the first solution. Interestingly I don't see anything at all about the error till the first instance I posted happens and the server crashes after that, seems like others get some errors prior to the crash.

I have a new issue, problem is that its a kernel panic so I can't get much out of the logs after it disconnects the "tail -f" I have been running on a separate machine. Below I posted the last crash I got plus what was shown on the monitor attached to the server. The error on the monitor matches the last 3 kernel panics I have gotten with "nf_nat_setup_info+0x146/0x773". I read somewhere that others running into similar errors switched from "macvlan" to "ipvlan"? I am re-running a full memtest, but last run had no issues found. tail -f /var/log/syslog: Jul 11 21:40:01 Unimatrix kernel: general protection fault, probably for non-canonical address 0x770c000000000040: 0000 [#1] SMP PTI Jul 11 21:40:01 Unimatrix kernel: CPU: 0 PID: 10577 Comm: isc-worker0000 Tainted: G W I 5.15.46-Unraid #1 Jul 11 21:40:01 Unimatrix kernel: Hardware name: Dell Inc. PowerEdge R710/0YMXG9, BIOS 6.6.0 05/22/2018 Jul 11 21:40:01 Unimatrix kernel: RIP: 0010:nf_nat_setup_info+0x146/0x773 [nf_nat] Jul 11 21:40:01 Unimatrix kernel: Code: 15 c0 6a 00 00 89 c0 48 8d 04 c2 48 8b 10 48 85 d2 74 07 48 81 ea 98 00 00 00 45 31 c0 48 85 d2 0f 84 6b ff ff ff 8a 44 24 4e <38> 42 46 74 19 48 8b 92 98 00 00 00 48 85 d2 0f 84 52 ff ff ff 48 Jul 11 21:40:01 Unimatrix kernel: RSP: 0018:ffffc90000003790 EFLAGS: 00010206 Jul 11 21:40:01 Unimatrix kernel: RAX: ffff88814f415f11 RBX: ffff888199735900 RCX: 917e0ff665e1d99b Jul 11 21:40:01 Unimatrix kernel: RDX: 770bfffffffffffa RSI: c7b19645379efdef RDI: ffffc90000003768 Jul 11 21:40:01 Unimatrix kernel: RBP: ffffc90000003860 R08: 0000000000000000 R09: 889f1bfb869f2ede Jul 11 21:40:01 Unimatrix kernel: R10: 976f444f8dfdc34f R11: 8d7ae8674c487d4e R12: 0000000000000000 Jul 11 21:40:01 Unimatrix kernel: R13: ffffc90000003874 R14: 0000000000000000 R15: ffffc900000037b8 Jul 11 21:40:01 Unimatrix kernel: FS: 00001502b65b0b20(0000) GS:ffff8897df800000(0000) knlGS:0000000000000000 Jul 11 21:40:01 Unimatrix kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 11 21:40:01 Unimatrix kernel: CR2: 0000555cb4692000 CR3: 0000000413e00004 CR4: 00000000000206f0 Jul 11 21:40:01 Unimatrix kernel: Call Trace: Jul 11 21:40:01 Unimatrix kernel: <IRQ> Jul 11 21:40:01 Unimatrix kernel: ? __fib_lookup+0x5f/0x9b Jul 11 21:40:01 Unimatrix kernel: ? __ksize+0x1e/0x76 Jul 11 21:40:01 Unimatrix kernel: ? krealloc+0x31/0x89 Jul 11 21:40:01 Unimatrix kernel: nf_nat_masquerade_ipv4+0x111/0x137 [nf_nat] Jul 11 21:40:01 Unimatrix kernel: masquerade_tg+0x48/0x62 [xt_MASQUERADE] Jul 11 21:40:01 Unimatrix kernel: ? fib4_rule_action+0x6c/0x74 Jul 11 21:40:01 Unimatrix kernel: ipt_do_table+0x50d/0x5b0 [ip_tables] Jul 11 21:40:01 Unimatrix kernel: ? find_exception+0x22/0xb5 Jul 11 21:40:01 Unimatrix kernel: ? ipt_do_table+0x563/0x5b0 [ip_tables] Jul 11 21:40:01 Unimatrix kernel: nf_nat_inet_fn+0x106/0x187 [nf_nat] Jul 11 21:40:01 Unimatrix kernel: nf_nat_ipv4_out+0x15/0x8b [nf_nat] Jul 11 21:40:01 Unimatrix kernel: nf_hook_slow+0x3e/0x93 Jul 11 21:40:01 Unimatrix kernel: nf_hook+0xaf/0xd6 Jul 11 21:40:01 Unimatrix kernel: ? __ip_finish_output+0x14e/0x14e Jul 11 21:40:01 Unimatrix kernel: ip_output+0x82/0x8f Jul 11 21:40:01 Unimatrix kernel: ? __ip_finish_output+0x14e/0x14e Jul 11 21:40:01 Unimatrix kernel: ip_sabotage_in+0x4c/0x59 [br_netfilter] Jul 11 21:40:01 Unimatrix kernel: nf_hook_slow+0x3e/0x93 Jul 11 21:40:01 Unimatrix kernel: ? ip_rcv_finish_core.constprop.0+0x358/0x358 Jul 11 21:40:01 Unimatrix kernel: NF_HOOK.constprop.0+0x73/0xce Jul 11 21:40:01 Unimatrix kernel: ? ip_rcv_finish_core.constprop.0+0x358/0x358 Jul 11 21:40:01 Unimatrix kernel: __netif_receive_skb_one_core+0x79/0x9a Jul 11 21:40:01 Unimatrix kernel: netif_receive_skb+0xa3/0xf7 Jul 11 21:40:01 Unimatrix kernel: br_handle_frame_finish+0x3c5/0x407 Jul 11 21:40:01 Unimatrix kernel: ? br_handle_local_finish+0x13/0x13 Jul 11 21:40:01 Unimatrix kernel: br_nf_hook_thresh+0xc6/0xf7 [br_netfilter] Jul 11 21:40:01 Unimatrix kernel: ? br_handle_local_finish+0x13/0x13 Jul 11 21:40:01 Unimatrix kernel: br_nf_pre_routing_finish+0x29d/0x2c4 [br_netfilter] Jul 11 21:40:01 Unimatrix kernel: ? br_handle_local_finish+0x13/0x13 Jul 11 21:40:01 Unimatrix kernel: ? NF_HOOK.isra.0+0xc5/0x119 [br_netfilter] Jul 11 21:40:01 Unimatrix kernel: br_nf_pre_routing+0x226/0x236 [br_netfilter] Jul 11 21:40:01 Unimatrix kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter] Jul 11 21:40:01 Unimatrix kernel: br_handle_frame+0x292/0x2da Jul 11 21:40:01 Unimatrix kernel: ? br_handle_local_finish+0x13/0x13 Jul 11 21:40:01 Unimatrix kernel: __netif_receive_skb_core.constprop.0+0x4b6/0x685 Jul 11 21:40:01 Unimatrix kernel: ? udp_queue_rcv_one_skb+0x1c9/0x306 Jul 11 21:40:01 Unimatrix kernel: ? udp_unicast_rcv_skb+0x6b/0x7e Jul 11 21:40:01 Unimatrix kernel: __netif_receive_skb_one_core+0x42/0x9a Jul 11 21:40:01 Unimatrix kernel: process_backlog+0xab/0x143 Jul 11 21:40:01 Unimatrix kernel: __napi_poll.constprop.0+0x2a/0x114 Jul 11 21:40:01 Unimatrix kernel: net_rx_action+0xe8/0x1f2 Jul 11 21:40:01 Unimatrix kernel: __do_softirq+0xef/0x218 Jul 11 21:40:01 Unimatrix kernel: do_softirq+0x50/0x68 Jul 11 21:40:01 Unimatrix kernel: </IRQ> Jul 11 21:40:01 Unimatrix kernel: <TASK> Jul 11 21:40:01 Unimatrix kernel: __local_bh_enable_ip+0x40/0x48 Jul 11 21:40:01 Unimatrix kernel: ip_finish_output2+0x2d3/0x305 Jul 11 21:40:01 Unimatrix kernel: ip_send_skb+0x15/0x37 Jul 11 21:40:01 Unimatrix kernel: udp_send_skb+0x25c/0x2be Jul 11 21:40:01 Unimatrix kernel: udp_sendmsg+0x688/0x8b2 Jul 11 21:40:01 Unimatrix kernel: ? ip_select_ident_segs+0x51/0x51 Jul 11 21:40:01 Unimatrix kernel: ? _copy_from_user+0x28/0x46 Jul 11 21:40:01 Unimatrix kernel: ? _copy_from_user+0x28/0x46 Jul 11 21:40:01 Unimatrix kernel: ? iovec_from_user+0xc9/0x118 Jul 11 21:40:01 Unimatrix kernel: ? sock_sendmsg_nosec+0x2b/0x3c Jul 11 21:40:01 Unimatrix kernel: sock_sendmsg_nosec+0x2b/0x3c Jul 11 21:40:01 Unimatrix kernel: ____sys_sendmsg+0x143/0x1a8 Jul 11 21:40:01 Unimatrix kernel: ___sys_sendmsg+0x7f/0xb7 Jul 11 21:40:01 Unimatrix kernel: ? pipe_write+0x488/0x4f0 Jul 11 21:40:01 Unimatrix kernel: ? ___sys_recvmsg+0x89/0xb2 Jul 11 21:40:01 Unimatrix kernel: ? new_sync_write+0x7f/0xb7 Jul 11 21:40:01 Unimatrix kernel: ? intel_pmu_pebs_add+0x5c/0x60 Jul 11 21:40:01 Unimatrix kernel: ? __fget+0x29/0x2f Jul 11 21:40:01 Unimatrix kernel: __sys_sendmsg+0x60/0x93 Jul 11 21:40:01 Unimatrix kernel: do_syscall_64+0x83/0xa5 Jul 11 21:40:01 Unimatrix kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae Jul 11 21:40:01 Unimatrix kernel: RIP: 0033:0x1502b6e97878 Jul 11 21:40:01 Unimatrix kernel: Code: c3 8b 07 85 c0 75 24 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 <c3> e9 5e d3 ff ff 41 54 b8 02 00 00 00 49 89 f4 be 00 08 08 00 55 Jul 11 21:40:01 Unimatrix kernel: RSP: 002b:00001502b65b00e8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e Jul 11 21:40:01 Unimatrix kernel: RAX: ffffffffffffffda RBX: 00001502b65b0b20 RCX: 00001502b6e97878 Jul 11 21:40:01 Unimatrix kernel: RDX: 0000000000000000 RSI: 00001502b65b0148 RDI: 0000000000000014 Jul 11 21:40:01 Unimatrix kernel: RBP: 0000000000000014 R08: 0000000000000000 R09: 0000000000000000 Jul 11 21:40:01 Unimatrix kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000002e Jul 11 21:40:01 Unimatrix kernel: R13: 0000000000000014 R14: 00001502b65b0148 R15: 00001502b65b05c8 Jul 11 21:40:01 Unimatrix kernel: </TASK> Jul 11 21:40:01 Unimatrix kernel: Modules linked in: ccp macvlan veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xfs md_mod iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bonding bnx2 ipmi_ssif i2c_core intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd intel_cstate input_leds intel_uncore mpt3sas led_class ata_piix ipmi_si raid_class scsi_transport_sas wmi acpi_power_meter button acpi_cpufreq [last unloaded: bnx2] Jul 11 21:40:01 Unimatrix kernel: ---[ end trace 3d6c5e747724fdb7 ]--- client_loop: send disconnect: Broken pipe last lines on screen: curve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel up_tunnel libchacha ip6table _filter ip6_tables iptable_filter ip_tables x_tables bonding bnx2 ipmi_ssif i2c_core intel_powerclamp coretemp kvm_intel kvm crct10dif _pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto simd cryptd intel_cstate input_leds intel_uncore mpt3sas led_class ata_piix ipmi_si raid _class scsi_transport_sas wmi acpi_power _meter button acpi_cpufreq [last unloaded: bnx2] ---[ end trace 3d6c5e747724fdb7 ]--- RIP: 0010:nf_nat_setup_info+0x146/Ox773 [nf_nat] Code: 15 c0 6a 00 00 89 c0 48 8d 04 c2 48 8b 10 48 85 d2 74 07 48 81 ea 98 00 00 00 45 31 c0 48 85 d2 Of 84 6b ff ff ff 8a 44 24 4e <38> 42 46 74 19 48 8b 92 98 00 00 00 48 85 d2 Of 84 52 ff ff ff 48 RSP: 0018:ffffc90000003790 FLAGS: 00010206 RAX: ffff88814f415f11 RBX: ffff888199735900 RCX: 917e0ff665e1d99b RDX: 770bfffffffffffa RSI: c7b19645379efdef RDI: ffffc90000003768 RBP: ffffc90000003860 R08: 0000000000000000 R09: 889f1bfb869f2ede R10: 976f444f8df dc34f R11: 8d7ae8674c487d4e R12: 0000000000000000 R13: ffffc90000003874 R14: 0000000000000000 R15: ffffc90000003768 FS: 00001502b65bOb20(0000) GS:fff18897df800000(0000) knIGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CRO: 0000000080050033 CRZ: 0000555cb4692000 CR3: 0000000413e00004 CR4: 00000000000206f0 Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: disabled ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

Thank you!! Upgraded to v6.10.3-rc1 (there was a hick-up trying to restart, had to cold reboot, assuming same issues lead to this) and there were no errors at all. I will reformat my cache pool tomorrow and remake my docker.img and if I run into further issues I will send my diagnostics file.

I have a Dell R710 with a dual nic (second disabled) Broadcom BCM5709 with kernel driver bnx2. I have seen similar kernel DMAR errors as others have posted here. I also ran into an issue with the bzfirmware file not having the correct checksum after each new RC upgrade until I switched the flash to my front USB port which made upgrades correctly extract bzfirmware. My cache drive scrum does report 4 uncorrectable errors and docker (i believe) has on multiple occasions hung the entire system where I could not reboot. Finally maybe unrelated I did run into two CRC errors on my parity drive which I had to acknowledge and re-add my parity drive to recover from. Currently I am working on backing up my appdata off the cache pool to rebuilt it because of the uncorrectable csum errors and also wanted to delete and rebuild my docker.img. In bios my virtualization is set to true as I would like to setup HomeAssistant, but I can hold off to see if I am in the same boat as others and I should disable virtualization for now before going through and rebuilding my cache pool and docker image. I have docker off for now and no errors have taken place. NIC Info (note NIC2 is disabled): 01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) DeviceName: Embedded NIC 1 Subsystem: Dell PowerEdge R710 BCM5709 Gigabit Ethernet Kernel driver in use: bnx2 01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) DeviceName: Embedded NIC 2 Subsystem: Dell PowerEdge R710 BCM5709 Gigabit Ethernet Kernel driver in use: bnx2 BTRFS errors after boot scrub (and start of DMAR errors): Jun 9 18:18:14 kernel: BTRFS info (device sdb1): scrub: started on devid 1 Jun 9 18:18:14 kernel: BTRFS info (device sdb1): scrub: started on devid 2 Jun 9 18:18:48 kernel: BTRFS warning (device sdb1): checksum error at logical 11833892864 on dev /dev/sdb1, physical 6465183744, root 5, inode 3598121, offset 200704, length 4096, links 1 (path: appdata/sabnzbd/logs/sabnzbd.log.1) Jun 9 18:18:48 kernel: BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 12, gen 0 Jun 9 18:18:48 kernel: BTRFS error (device sdb1): unable to fixup (regular) error at logical 11833892864 on dev /dev/sdb1 Jun 9 18:18:48 kernel: BTRFS warning (device sdb1): checksum error at logical 11833892864 on dev /dev/sdc1, physical 6444212224, root 5, inode 3598121, offset 200704, length 4096, links 1 (path: appdata/sabnzbd/logs/sabnzbd.log.1) Jun 9 18:18:48 kernel: BTRFS error (device sdb1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 Jun 9 18:18:48 kernel: BTRFS warning (device sdb1): checksum error at logical 11833896960 on dev /dev/sdb1, physical 6465187840, root 5, inode 3598121, offset 204800, length 4096, links 1 (path: appdata/sabnzbd/logs/sabnzbd.log.1) Jun 9 18:18:48 kernel: BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 13, gen 0 Jun 9 18:18:48 kernel: BTRFS error (device sdb1): unable to fixup (regular) error at logical 11833892864 on dev /dev/sdc1 Jun 9 18:18:48 kernel: BTRFS error (device sdb1): unable to fixup (regular) error at logical 11833896960 on dev /dev/sdb1 Jun 9 18:18:48 kernel: BTRFS warning (device sdb1): checksum error at logical 11833896960 on dev /dev/sdc1, physical 6444216320, root 5, inode 3598121, offset 204800, length 4096, links 1 (path: appdata/sabnzbd/logs/sabnzbd.log.1) Jun 9 18:18:48 kernel: BTRFS error (device sdb1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 Jun 9 18:18:48 kernel: BTRFS error (device sdb1): unable to fixup (regular) error at logical 11833896960 on dev /dev/sdc1 Jun 9 18:19:05 kernel: DMAR: ERROR: DMA PTE for vPFN 0xbf4df already set (to bf4df003 not 1011b0801) Jun 9 18:19:05 kernel: ------------[ cut here ]------------ Jun 9 18:19:05 kernel: WARNING: CPU: 20 PID: 7515 at drivers/iommu/intel/iommu.c:2408 __domain_mapping+0x2e5/0x390 Jun 9 18:19:05 kernel: Modules linked in: xfs md_mod iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bonding bnx2 ipmi_ssif i2c_core intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd intel_cstate intel_uncore mpt3sas input_leds led_class ata_piix ipmi_si raid_class wmi acpi_power_meter scsi_transport_sas button acpi_cpufreq [last unloaded: bnx2] Jun 9 18:19:05 kernel: CPU: 20 PID: 7515 Comm: apcupsd Tainted: G I 5.15.43-Unraid #1 Jun 9 18:19:05 kernel: Hardware name: Dell Inc. PowerEdge R710/0YMXG9, BIOS 6.6.0 05/22/2018 kernel DMAR errors (these repeat several times on boot): Jun 9 18:19:06 kernel: DMAR: ERROR: DMA PTE for vPFN 0xbf4de already set (to bf4de003 not 1011b0801) Jun 9 18:19:06 kernel: ------------[ cut here ]------------ Jun 9 18:19:06 kernel: WARNING: CPU: 20 PID: 7515 at drivers/iommu/intel/iommu.c:2408 __domain_mapping+0x2e5/0x390 Jun 9 18:19:06 kernel: Modules linked in: xfs md_mod iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bonding bnx2 ipmi_ssif i2c_core intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd intel_cstate intel_uncore mpt3sas input_leds led_class ata_piix ipmi_si raid_class wmi acpi_power_meter scsi_transport_sas button acpi_cpufreq [last unloaded: bnx2] Jun 9 18:19:06 kernel: CPU: 20 PID: 7515 Comm: apcupsd Tainted: G W I 5.15.43-Unraid #1 Jun 9 18:19:06 kernel: Hardware name: Dell Inc. PowerEdge R710/0YMXG9, BIOS 6.6.0 05/22/2018 Jun 9 18:19:06 kernel: RIP: 0010:__domain_mapping+0x2e5/0x390 Jun 9 18:19:06 kernel: Code: 2b 48 8b 4c 24 08 48 89 c2 4c 89 e6 48 c7 c7 fe b1 11 82 e8 ef 87 2d 00 8b 05 1d 9b df 00 85 c0 74 08 ff c8 89 05 11 9b df 00 <0f> 0b 8b 74 24 38 b8 34 00 00 00 8d 0c f6 83 e9 09 39 c1 0f 4f c8 Jun 9 18:19:06 kernel: RSP: 0018:ffffc900081b7ad8 EFLAGS: 00010006 Jun 9 18:19:06 kernel: RAX: 0000000000000003 RBX: ffff8881087136f0 RCX: 0000000000000000 Jun 9 18:19:06 kernel: RDX: 0000000000000000 RSI: ffff8897dfd1c510 RDI: ffff8897dfd1c510 Jun 9 18:19:06 kernel: RBP: ffff88810806e300 R08: ffff88983ffa3e68 R09: 0000000000000000 Jun 9 18:19:06 kernel: R10: 2931303830623131 R11: 623131303120746f R12: 00000000000bf4de Jun 9 18:19:06 kernel: R13: ffff8881087136f0 R14: 0000000000001000 R15: 00000000bf4de000 Jun 9 18:19:06 kernel: FS: 000014a59cb33f00(0000) GS:ffff8897dfd00000(0000) knlGS:0000000000000000 Jun 9 18:19:06 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 9 18:19:06 kernel: CR2: 0000000000462cb8 CR3: 0000000100da6005 CR4: 00000000000206e0 Jun 9 18:19:06 kernel: Call Trace: Jun 9 18:19:06 kernel: <TASK> Jun 9 18:19:06 kernel: ? alloc_iova+0x30/0x1a8 Jun 9 18:19:06 kernel: intel_iommu_map_pages+0xf3/0x102 Jun 9 18:19:06 kernel: __iommu_map+0x138/0x211 Jun 9 18:19:06 kernel: _iommu_map+0x23/0x51 Jun 9 18:19:06 kernel: __iommu_dma_map+0x94/0xc4 Jun 9 18:19:06 kernel: iommu_dma_map_page+0x100/0x142 Jun 9 18:19:06 kernel: usb_hcd_map_urb_for_dma+0xe5/0x2ee Jun 9 18:19:06 kernel: usb_hcd_submit_urb+0x675/0x72b Jun 9 18:19:06 kernel: ? update_load_avg+0x43/0x2ce Jun 9 18:19:06 kernel: ? get_sd_balance_interval+0x18/0x3b Jun 9 18:19:06 kernel: ? rpm_resume+0x45d/0x484 Jun 9 18:19:06 kernel: ? newidle_balance+0x21c/0x291 Jun 9 18:19:06 kernel: ? dequeue_entity+0x1e0/0x203 Jun 9 18:19:06 kernel: ? usb_submit_urb+0x2d6/0x4ed Jun 9 18:19:06 kernel: hid_submit_ctrl+0x1c4/0x201 Jun 9 18:19:06 kernel: usbhid_restart_ctrl_queue.isra.0+0x8c/0xc7 Jun 9 18:19:06 kernel: usbhid_submit_report+0x25b/0x2e6 Jun 9 18:19:06 kernel: hiddev_ioctl+0x301/0x558 Jun 9 18:19:06 kernel: ? hrtimer_nanosleep+0x75/0xdf Jun 9 18:19:06 kernel: ? hrtimer_init_sleeper+0x3d/0x3d Jun 9 18:19:06 kernel: vfs_ioctl+0x1e/0x2b Jun 9 18:19:06 kernel: __do_sys_ioctl+0x51/0x74 Jun 9 18:19:06 kernel: do_syscall_64+0x83/0xa5 Jun 9 18:19:06 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae Jun 9 18:19:06 kernel: RIP: 0033:0x14a59cdfd067 Jun 9 18:19:06 kernel: Code: 3c 1c e8 2c ff ff ff 85 c0 79 97 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d1 1d 0d 00 f7 d8 64 89 01 48 Jun 9 18:19:06 kernel: RSP: 002b:00007ffcb5fa99f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 Jun 9 18:19:06 kernel: RAX: ffffffffffffffda RBX: 0000000000437370 RCX: 000014a59cdfd067 Jun 9 18:19:06 kernel: RDX: 00007ffcb5fa9a04 RSI: 00000000400c4807 RDI: 0000000000000006 Jun 9 18:19:06 kernel: RBP: 00000000004711c0 R08: 0000000000000000 R09: 0000000000000000 Jun 9 18:19:06 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000474650 Jun 9 18:19:06 kernel: R13: 00007ffcb5fa9a70 R14: 000000000000a000 R15: 0000000000009fff Jun 9 18:19:06 kernel: </TASK> Jun 9 18:19:06 kernel: ---[ end trace 8747f0981c782280 ]--- Manual BTRFS scrub results: UUID: <redacted> Scrub started: Thu Jun 9 23:43:02 2022 Status: finished Duration: 0:06:09 Total to scrub: 143.61GiB Rate: 350.23MiB/s Error summary: csum=4 Corrected: 0 Uncorrectable: 4 Unverified: 0 @JorgeB I can run and send a diagnostics, I don't believe I have one when the server hung and needed a hard reboot.

I actually do have a Dell R710, thank you for the link I updated my original post, just copied the bzfirmware file from the zip on a separate computer. I double checked the sha256 checksum and it matched, when I originally checked the bzfirmware file that was unpacked as a part of the GUI upgrade in unraid the checksum was off.

When I updated via GUI and rebooted I run into: Verifying bzfirmware checksum ... bzfirmware checksum error - press ENTER key to reboot... I restored the contents of the /previous folder on my boot USB and returned to RC3 and booted fine, tried the update again through the GUI and same error occurred. I am going to try and download the update zip directly and unpack just the bzfirmware files and see if that helps. update: downloaded the rc4 zip on a separate machine confirmed the shasum -a 256 returned correctly on bzfirmware i had downloaded, copied it over to the USB and booted just fine