Kernel Error - Happening in 6.9.2 and 6.10.3

[JayC]

This started happening after I upgraded to 6.9.2 and upgrading to 6.10.3 didn't fix it.  Any diea what is causing this crash, and a way to fix it?

 

Sep 12 14:17:33 BFS kernel: ------------[ cut here ]------------
Sep 12 14:17:33 BFS kernel: WARNING: CPU: 7 PID: 48 at net/netfilter/nf_conntrack_core.c:1120 __nf_conntrack_confirm+0x9b/0x1e6 [nf_conntrack]
Sep 12 14:17:33 BFS kernel: Modules linked in: veth xt_nat xt_tcpudp macvlan xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs md_mod nfsd lockd grace sunrpc ip6table_filter ip6_tables iptable_filter ip_tables x_tables igb i2c_algo_bit intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper i2c_i801 mpt3sas ahci intel_cstate i2c_smbus nvme i2c_core intel_uncore raid_class input_leds nvme_core i7core_edac libahci led_class scsi_transport_sas i5500_temp button acpi_cpufreq [last unloaded: md_mod]
Sep 12 14:17:33 BFS kernel: CPU: 7 PID: 48 Comm: kworker/7:0 Not tainted 5.10.28-Unraid #1
Sep 12 14:17:33 BFS kernel: Hardware name: Supermicro X8DT3/X8DT3, BIOS 2.1     03/17/2012
Sep 12 14:17:33 BFS kernel: Workqueue: events macvlan_process_broadcast [macvlan]
Sep 12 14:17:33 BFS kernel: RIP: 0010:__nf_conntrack_confirm+0x9b/0x1e6 [nf_conntrack]
Sep 12 14:17:33 BFS kernel: Code: e8 dc f8 ff ff 44 89 fa 89 c6 41 89 c4 48 c1 eb 20 89 df 41 89 de e8 36 f6 ff ff 84 c0 75 bb 48 8b 85 80 00 00 00 a8 08 74 18 <0f> 0b 89 df 44 89 e6 31 db e8 6d f3 ff ff e8 35 f5 ff ff e9 22 01
Sep 12 14:17:33 BFS kernel: RSP: 0018:ffffc900064acdd8 EFLAGS: 00010202
Sep 12 14:17:33 BFS kernel: RAX: 0000000000000188 RBX: 00000000000036e2 RCX: 00000000a5311ccf
Sep 12 14:17:33 BFS kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffa023bd88
Sep 12 14:17:33 BFS kernel: RBP: ffff888695f13cc0 R08: 00000000e95c212f R09: ffff888147f10d60
Sep 12 14:17:33 BFS kernel: R10: 0000000000000000 R11: ffff888686710900 R12: 000000000000128b
Sep 12 14:17:33 BFS kernel: R13: ffffffff8210b440 R14: 00000000000036e2 R15: 0000000000000000
Sep 12 14:17:33 BFS kernel: FS:  0000000000000000(0000) GS:ffff888c27a40000(0000) knlGS:0000000000000000
Sep 12 14:17:33 BFS kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 12 14:17:33 BFS kernel: CR2: 0000148e9858d718 CR3: 000000000200a001 CR4: 00000000000206e0
Sep 12 14:17:33 BFS kernel: Call Trace:
Sep 12 14:17:33 BFS kernel: <IRQ>
Sep 12 14:17:33 BFS kernel: nf_conntrack_confirm+0x2f/0x36 [nf_conntrack]
Sep 12 14:17:33 BFS kernel: nf_hook_slow+0x39/0x8e
Sep 12 14:17:33 BFS kernel: nf_hook.constprop.0+0xb1/0xd8
Sep 12 14:17:33 BFS kernel: ? ip_protocol_deliver_rcu+0xfe/0xfe
Sep 12 14:17:33 BFS kernel: ip_local_deliver+0x49/0x75
Sep 12 14:17:33 BFS kernel: __netif_receive_skb_one_core+0x74/0x95
Sep 12 14:17:33 BFS kernel: process_backlog+0xa3/0x13b
Sep 12 14:17:33 BFS kernel: net_rx_action+0xf4/0x29d
Sep 12 14:17:33 BFS kernel: __do_softirq+0xc4/0x1c2
Sep 12 14:17:33 BFS kernel: asm_call_irq_on_stack+0x12/0x20
Sep 12 14:17:33 BFS kernel: </IRQ>
Sep 12 14:17:33 BFS kernel: do_softirq_own_stack+0x2c/0x39
Sep 12 14:17:33 BFS kernel: do_softirq+0x3a/0x44
Sep 12 14:17:33 BFS kernel: netif_rx_ni+0x1c/0x22
Sep 12 14:17:33 BFS kernel: macvlan_broadcast+0x10e/0x13c [macvlan]
Sep 12 14:17:33 BFS kernel: macvlan_process_broadcast+0xf8/0x143 [macvlan]
Sep 12 14:17:33 BFS kernel: process_one_work+0x13c/0x1d5
Sep 12 14:17:33 BFS kernel: worker_thread+0x18b/0x22f
Sep 12 14:17:33 BFS kernel: ? process_scheduled_works+0x27/0x27
Sep 12 14:17:33 BFS kernel: kthread+0xe5/0xea
Sep 12 14:17:33 BFS kernel: ? __kthread_bind_mask+0x57/0x57
Sep 12 14:17:33 BFS kernel: ret_from_fork+0x22/0x30
Sep 12 14:17:33 BFS kernel: ---[ end trace 03639e18427da19f ]---

 

From time to time this server is locking up, I'll keep posting any log entries like this, hopefully somebody knows what is causing the issue, and it's fixable   Thanks!

[trurl]

Attach Diagnostics to your NEXT post in this thread. 

 

Setup syslog server. 

[JayC]

I'm attaching the diagnostics, and have enabled syslog as well.

bfs-diagnostics-20220912-1540.zip

[trurl]

Not related to your problem, but you should disable mover logging in Scheduler Settings unless you are trying to debug a mover problem.

  

3 hours ago, JayC said:

what is causing this crash

 

 

Sep 12 14:17:33 BFS kernel: Workqueue: events macvlan_process_broadcast [macvlan]

 

[JorgeB]

Upgrading to v6.10 and switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).