Unraid 6.12.4 Kernel Panic

[werkkrew]

Ever since upgrading to 6.12.4 my system has been experiencing a kernel panic every few hours.

 

I have attached the diagnostic bundle as well as as chunk of the syslog from my remote syslog server if anyone wants to take a stab at trying to diagnose this.

 

Nov 11 22:08:54 arthas kernel: ------------[ cut here ]------------
Nov 11 22:08:54 arthas kernel: WARNING: CPU: 33 PID: 13137 at net/netfilter/nf_nat_core.c:594 nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Nov 11 22:08:54 arthas kernel: Modules linked in: xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth macvlan xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet 8021q garp mrp bridge stp llc bonding tls e1000e amdgpu gpu_sched drm_buddy radeon intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel video i2c_algo_bit drm_ttm_helper ttm drm_display_helper drm_kms_helper tpm_infineon kvm drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel crypto_simd backlight cryptd mei_hdcp mei_pxp agpgart i2c_i801 rapl tpm_tis intel_cstate wmi_bmof mpt3sas
Nov 11 22:08:54 arthas kernel: isci firewire_ohci tpm_tis_core i2c_smbus syscopyarea nvme mei_me libsas ahci sysfillrect raid_class sysimgblt intel_uncore firewire_core i2c_core nvme_core mei libahci scsi_transport_sas fb_sys_fops wmi tpm button unix [last unloaded: e1000e]
Nov 11 22:08:54 arthas kernel: CPU: 33 PID: 13137 Comm: kworker/u97:6 Tainted: P        W  O       6.1.49-Unraid #1
Nov 11 22:08:54 arthas kernel: Hardware name: Hewlett-Packard HP Z820 Workstation/158B, BIOS J63 v03.96 10/28/2019
Nov 11 22:08:54 arthas kernel: Workqueue: events_unbound macvlan_process_broadcast [macvlan]
Nov 11 22:08:54 arthas kernel: RIP: 0010:nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Nov 11 22:08:54 arthas kernel: Code: a8 80 75 26 48 8d 73 58 48 8d 7c 24 20 e8 18 9b fc ff 48 8d 43 0c 4c 8b bb 88 00 00 00 48 89 44 24 18 eb 54 0f ba e0 08 73 07 <0f> 0b e9 75 06 00 00 48 8d 73 58 48 8d 7c 24 20 e8 eb 9a fc ff 48
Nov 11 22:08:54 arthas kernel: RSP: 0018:ffffc90006c00c78 EFLAGS: 00010282
Nov 11 22:08:54 arthas kernel: RAX: 0000000000000180 RBX: ffff8882683a6c00 RCX: ffff88903b8f3480
Nov 11 22:08:54 arthas kernel: RDX: 0000000000000000 RSI: ffffc90006c00d5c RDI: ffff8882683a6c00
Nov 11 22:08:54 arthas kernel: RBP: ffffc90006c00d40 R08: 000000001501a8c0 R09: 0000000000000000
Nov 11 22:08:54 arthas kernel: R10: 0000000000000158 R11: 0000000000000000 R12: ffffc90006c00d5c
Nov 11 22:08:54 arthas kernel: R13: 0000000000000000 R14: ffffc90006c00e40 R15: 0000000000000001
Nov 11 22:08:54 arthas kernel: FS:  0000000000000000(0000) GS:ffff888fefd40000(0000) knlGS:0000000000000000
Nov 11 22:08:54 arthas kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 22:08:54 arthas kernel: CR2: 0000152dabebce80 CR3: 000000000220a004 CR4: 00000000001706e0
Nov 11 22:08:54 arthas kernel: Call Trace:
Nov 11 22:08:54 arthas kernel: <IRQ>
Nov 11 22:08:54 arthas kernel: ? __warn+0xab/0x122
Nov 11 22:08:54 arthas kernel: ? report_bug+0x109/0x17e
Nov 11 22:08:54 arthas kernel: ? nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Nov 11 22:08:54 arthas kernel: ? handle_bug+0x41/0x6f
Nov 11 22:08:54 arthas kernel: ? exc_invalid_op+0x13/0x60
Nov 11 22:08:54 arthas kernel: ? asm_exc_invalid_op+0x16/0x20
Nov 11 22:08:54 arthas kernel: ? nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Nov 11 22:08:54 arthas kernel: ? nf_nat_setup_info+0x44/0x7d1 [nf_nat]
Nov 11 22:08:54 arthas kernel: ? xt_write_recseq_end+0xf/0x1c [ip_tables]
Nov 11 22:08:54 arthas kernel: ? __local_bh_enable_ip+0x56/0x6b
Nov 11 22:08:54 arthas kernel: ? ipt_do_table+0x57a/0x5bf [ip_tables]
Nov 11 22:08:54 arthas kernel: ? xt_write_recseq_end+0xf/0x1c [ip_tables]
Nov 11 22:08:54 arthas kernel: __nf_nat_alloc_null_binding+0x66/0x81 [nf_nat]
Nov 11 22:08:54 arthas kernel: nf_nat_inet_fn+0xc0/0x1a8 [nf_nat]
Nov 11 22:08:54 arthas kernel: nf_nat_ipv4_local_in+0x2a/0xaa [nf_nat]
Nov 11 22:08:54 arthas kernel: nf_hook_slow+0x3d/0x96
Nov 11 22:08:54 arthas kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Nov 11 22:08:54 arthas kernel: NF_HOOK.constprop.0+0x79/0xd9
Nov 11 22:08:54 arthas kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Nov 11 22:08:54 arthas kernel: __netif_receive_skb_one_core+0x77/0x9c
Nov 11 22:08:54 arthas kernel: process_backlog+0x8c/0x116
Nov 11 22:08:54 arthas kernel: __napi_poll.constprop.0+0x2b/0x124
Nov 11 22:08:54 arthas kernel: net_rx_action+0x159/0x24f

 

arthas-diagnostics-20231112-0925.zip

[JorgeB]

Switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

[werkkrew]

I have made the change and will upgrade again soon, will report back if this solves the problem.  Thanks.

[JorgeB]

Don't forget to reboot after the change.

[werkkrew]

So I haven't had a Kernel panic since I made this change but today after rebooting my router I suddenly cannot get any internet access from my Unraid box, or the containers it runs.

 

Based on some searching this seems likely related to the IPVLAN change but I can't find any clear resolution. Any idea?

[JorgeB]

Server should still have internet with ipvlan, if it were just the dockers maybe it could be that.

[werkkrew]

When I woke up this morning now everything is working again, but I have no idea why.

 

I will do some digging, it could have been on the router side also.