Crashes after 6.9.1 --> 6.10.3

[stultus]

6.9.1 was beautifully stable with 6+ month uptimes for me. I upgraded to 6.10 around July 4th, got a lot of PCIe errors, and so moved to 6.10.3 to solve that known bug. Ever since then my system dies every week or too.

 

This is the last thing I see on the monitor, always related to nf_conntrack which appears to be a docker/network issue. Last weekend I changed from macvlan to ipvlan but that didn't solve my issue. Any ideas where to start poking around?

 

Aug  3 01:54:07 Sophos kernel: md: recovery thread: check P Q ...
Aug  3 01:54:56 Sophos flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Aug  3 02:25:19 Sophos kernel: ------------[ cut here ]------------
Aug  3 02:25:19 Sophos kernel: WARNING: CPU: 1 PID: 14065 at net/netfilter/nf_conntrack_core.c:1192 __nf_conntrack_confirm+0xb8/0x254 [nf_conntrack]
Aug  3 02:25:19 Sophos kernel: Modules linked in: tun nvidia_modeset(PO) nvidia_uvm(PO) veth macvlan xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xfs md_mod nvidia(PO) drm backlight nct6775 hwmon_vid efivarfs iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables btusb btrtl btbcm edac_mce_amd kvm_amd kvm crct10dif_pclmul wmi_bmof mxm_wmi btintel crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd bluetooth igb rapl input_leds i2c_piix4 i2c_algo_bit ccp ecdh_generic hid_lg_g15 k10temp i2c_core ahci led_class ecc libahci wmi button acpi_cpufreq
Aug  3 02:25:19 Sophos kernel: CPU: 1 PID: 14065 Comm: kworker/1:1 Tainted: P           O      5.15.46-Unraid #1
Aug  3 02:25:19 Sophos kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X370 Taichi, BIOS P6.40 07/28/2020
Aug  3 02:25:19 Sophos kernel: Workqueue: events macvlan_process_broadcast [macvlan]
Aug  3 02:25:19 Sophos kernel: RIP: 0010:__nf_conntrack_confirm+0xb8/0x254 [nf_conntrack]
Aug  3 02:25:19 Sophos kernel: Code: 89 c6 48 89 44 24 18 e8 53 e4 ff ff 44 89 f2 44 89 ef 89 c6 89 c5 e8 2c e8 ff ff 84 c0 75 9f 49 8b 87 80 00 00 00 a8 08 74 19 <0f> 0b 89 ee 44 89 ef 45 31 e4 e8 dc df ff ff e8 ae e4 ff ff e9 71
Aug  3 02:25:19 Sophos kernel: RSP: 0018:ffffc900001dcdb0 EFLAGS: 00010202
Aug  3 02:25:19 Sophos kernel: RAX: 0000000000000188 RBX: ffffffff828e1500 RCX: 0000000000000000
Aug  3 02:25:19 Sophos kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffa02c1648
Aug  3 02:25:19 Sophos kernel: RBP: 000000000000fca9 R08: 3705f11d51bb3b46 R09: 9409ee1e63995df3
Aug  3 02:25:19 Sophos kernel: R10: aef9ca04f70f31cc R11: 8711e21c188c2656 R12: 0000000000005102
Aug  3 02:25:19 Sophos kernel: R13: 0000000000005102 R14: 0000000000000000 R15: ffff88814e840dc0
Aug  3 02:25:19 Sophos kernel: FS:  0000000000000000(0000) GS:ffff8887fe840000(0000) knlGS:0000000000000000
Aug  3 02:25:19 Sophos kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug  3 02:25:19 Sophos kernel: CR2: 000000c00014d000 CR3: 0000000148b04000 CR4: 0000000000350ee0
Aug  3 02:25:19 Sophos kernel: Call Trace:
Aug  3 02:25:19 Sophos kernel: <IRQ>
Aug  3 02:25:19 Sophos kernel: nf_conntrack_confirm+0x26/0x3f [nf_conntrack]
Aug  3 02:25:19 Sophos kernel: nf_hook_slow+0x3e/0x93
Aug  3 02:25:19 Sophos kernel: ? ip_protocol_deliver_rcu+0x135/0x135
Aug  3 02:25:19 Sophos kernel: NF_HOOK.constprop.0+0x73/0xce
Aug  3 02:25:19 Sophos kernel: ? ip_protocol_deliver_rcu+0x135/0x135
Aug  3 02:25:19 Sophos kernel: __netif_receive_skb_one_core+0x79/0x9a
Aug  3 02:25:19 Sophos kernel: process_backlog+0xab/0x143
Aug  3 02:25:19 Sophos kernel: __napi_poll.constprop.0+0x2a/0x114
Aug  3 02:25:19 Sophos kernel: net_rx_action+0xe8/0x1f2
Aug  3 02:25:19 Sophos kernel: __do_softirq+0xef/0x218
Aug  3 02:25:19 Sophos kernel: do_softirq+0x50/0x68
Aug  3 02:25:19 Sophos kernel: </IRQ>
Aug  3 02:25:19 Sophos kernel: <TASK>
Aug  3 02:25:19 Sophos kernel: netif_rx_ni+0x53/0x85
Aug  3 02:25:19 Sophos kernel: macvlan_broadcast+0x116/0x144 [macvlan]
Aug  3 02:25:19 Sophos kernel: macvlan_process_broadcast+0xc7/0x110 [macvlan]
Aug  3 02:25:19 Sophos kernel: process_one_work+0x198/0x27a
Aug  3 02:25:19 Sophos kernel: worker_thread+0x19c/0x240
Aug  3 02:25:19 Sophos kernel: ? rescuer_thread+0x28b/0x28b
Aug  3 02:25:19 Sophos kernel: kthread+0xde/0xe3
Aug  3 02:25:19 Sophos kernel: ? set_kthread_struct+0x32/0x32
Aug  3 02:25:19 Sophos kernel: ret_from_fork+0x22/0x30
Aug  3 02:25:19 Sophos kernel: </TASK>
Aug  3 02:25:19 Sophos kernel: ---[ end trace f391be142f2ac721 ]---

 

sophos-diagnostics-20220812-1638.zip

[JonathanM]

3 hours ago, stultus said:

Last weekend I changed from macvlan to ipvlan but that didn't solve my issue.

Are you sure the change was applied?

3 hours ago, stultus said:

Aug  3 02:25:19 Sophos kernel: Modules linked in: tun nvidia_modeset(PO) nvidia_uvm(PO) veth macvlan xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xfs md_mod nvidia(PO) drm backlight nct6775 hwmon_vid efivarfs iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables btusb btrtl btbcm edac_mce_amd kvm_amd kvm crct10dif_pclmul wmi_bmof mxm_wmi btintel crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd bluetooth igb rapl input_leds i2c_piix4 i2c_algo_bit ccp ecdh_generic hid_lg_g15 k10temp i2c_core ahci led_class ecc libahci wmi button acpi_cpufreq
Aug  3 02:25:19 Sophos kernel: CPU: 1 PID: 14065 Comm: kworker/1:1 Tainted: P           O      5.15.46-Unraid #1
Aug  3 02:25:19 Sophos kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X370 Taichi, BIOS P6.40 07/28/2020
Aug  3 02:25:19 Sophos kernel: Workqueue: events macvlan_process_broadcast [macvlan]

 

[JorgeB]

12 hours ago, stultus said:

Aug  3 02:25:19 Sophos kernel: macvlan_broadcast+0x116/0x144 [macvlan]
Aug  3 02:25:19 Sophos kernel: macvlan_process_broadcast+0xc7/0x110 [macvlan]

It's still using macvlan.