• 6.12.0-rc4 "macvlan call traces found", but not on <=6.11.x


    sonic6
    • Solved Minor

    Hello,

     

    I got the following output in my syslog: https://pastebin.cloud-becker.de/5ede1251c0f8 (Diagnostic attached)

     

    I know, the general fix for this is using "ipvlan" instead of "mcvlan".

    But in my (and other people) case, this isn't an option.

    The AVM Fritzbox (7595 in my case) isn't compatible with ipvlan.

     

    I came from the latest 6.11.x stable without any problems, the same for 6.10.x .

     

    @alturismo got the same problem with 6.12.x, also when he was problemless on 6.11.x . Maybe he can post some more details from his setup.

     

    So I hope it is fixable, especially when version before run without this problem.

    unraid-1-diagnostics-20230429-1014.zip

    • Upvote 2



    User Feedback

    Recommended Comments



    1 hour ago, sonic6 said:

    Here we are again:

     

    Does it crash after this?

    I have seen something similar but server keeps working.

     

    Link to comment
    2 minutes ago, bonienl said:

    Does it crash after this?

    No, Server is still working, but will crash in a few days, like before.

    Link to comment

    Are you able to start something on your server which makes "continues" call traces or does it stay with this single call trace?

    Link to comment

    Looks like it will stay with this single trace for a long time.

     

    But, when you look here there was first a single trace when the server starts: 

    May  2 06:56:23 Unraid-1 kernel: ------------[ cut here ]------------
    May  2 06:56:23 Unraid-1 kernel: WARNING: CPU: 5 PID: 30624 at net/netfilter/nf_conntrack_core.c:1211 __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
    May  2 06:56:23 Unraid-1 kernel: Modules linked in: cmac cifs asn1_decoder cifs_arc4 cifs_md4 dns_resolver tls nft_chain_nat xt_owner nft_compat nf_tables macvlan xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_nat xt_tcpudp ip6table_mangle iptable_mangle vhost_net tun vhost vhost_iotlb tap xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat xt_addrtype br_netfilter veth xfs xt_MASQUERADE ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod tcp_diag inet_diag af_packet kvmgt mdev i915 iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper drm_kms_helper drm intel_gtt agpgart syscopyarea sysfillrect sysimgblt fb_sys_fops nct6775 nct6775_core hwmon_vid efivarfs wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables 8021q garp mrp bridge stp llc x86_pkg_temp_thermal intel_powerclamp
    May  2 06:56:23 Unraid-1 kernel: coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel crypto_simd cryptd mei_hdcp mei_pxp rapl intel_cstate i2c_i801 nvme mei_me wmi_bmof i2c_smbus mpt3sas e1000e intel_uncore nvme_core i2c_core cp210x mei intel_pch_thermal raid_class usbserial joydev scsi_transport_sas tpm_crb video tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_pad acpi_tad button unix
    May  2 06:56:23 Unraid-1 kernel: CPU: 5 PID: 30624 Comm: core Tainted: G     U             6.1.27-Unraid #1
    May  2 06:56:23 Unraid-1 kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H470M-ITX/ac, BIOS L1.22 12/07/2020
    May  2 06:56:23 Unraid-1 kernel: RIP: 0010:__nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
    May  2 06:56:23 Unraid-1 kernel: Code: 44 24 10 e8 f4 e1 ff ff 8b 7c 24 04 89 ea 89 c6 89 04 24 e8 76 e6 ff ff 84 c0 75 a2 48 89 df e8 ad e2 ff ff 85 c0 89 c5 74 18 <0f> 0b 8b 34 24 8b 7c 24 04 e8 2a dd ff ff e8 8b e3 ff ff e9 72 01
    May  2 06:56:23 Unraid-1 kernel: RSP: 0000:ffffc90006117808 EFLAGS: 00010202
    May  2 06:56:23 Unraid-1 kernel: RAX: 0000000000000001 RBX: ffff88836f618500 RCX: 75b9cb1795d9325b
    May  2 06:56:23 Unraid-1 kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88836f618500
    May  2 06:56:23 Unraid-1 kernel: RBP: 0000000000000001 R08: 892664b1afc23242 R09: 4dfe4a4532a4deda
    May  2 06:56:23 Unraid-1 kernel: R10: a2013e3c290bb2d9 R11: ffffc900061177d0 R12: ffffffff82a0e440
    May  2 06:56:23 Unraid-1 kernel: R13: 0000000000003edb R14: ffff88836f604c00 R15: 0000000000000000
    May  2 06:56:23 Unraid-1 kernel: FS:  0000153394d3e3c0(0000) GS:ffff88883f740000(0000) knlGS:0000000000000000
    May  2 06:56:23 Unraid-1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    May  2 06:56:23 Unraid-1 kernel: CR2: 0000557163404014 CR3: 0000000350120004 CR4: 00000000007706e0
    May  2 06:56:23 Unraid-1 kernel: PKRU: 55555554
    May  2 06:56:23 Unraid-1 kernel: Call Trace:
    May  2 06:56:23 Unraid-1 kernel: <TASK>
    May  2 06:56:23 Unraid-1 kernel: ? nf_nat_inet_fn+0x60/0x1a8 [nf_nat]
    May  2 06:56:23 Unraid-1 kernel: nf_conntrack_confirm+0x25/0x54 [nf_conntrack]
    May  2 06:56:23 Unraid-1 kernel: nf_hook_slow+0x3a/0x96
    May  2 06:56:23 Unraid-1 kernel: ? ip_protocol_deliver_rcu+0x164/0x164
    May  2 06:56:23 Unraid-1 kernel: NF_HOOK.constprop.0+0x79/0xd9
    May  2 06:56:23 Unraid-1 kernel: ? ip_protocol_deliver_rcu+0x164/0x164
    May  2 06:56:23 Unraid-1 kernel: ip_sabotage_in+0x4f/0x60 [br_netfilter]
    May  2 06:56:23 Unraid-1 kernel: nf_hook_slow+0x3a/0x96
    May  2 06:56:23 Unraid-1 kernel: ? ip_rcv_finish_core.constprop.0+0x3e8/0x3e8
    May  2 06:56:23 Unraid-1 kernel: NF_HOOK.constprop.0+0x79/0xd9
    May  2 06:56:23 Unraid-1 kernel: ? ip_rcv_finish_core.constprop.0+0x3e8/0x3e8
    May  2 06:56:23 Unraid-1 kernel: __netif_receive_skb_one_core+0x77/0x9c
    May  2 06:56:23 Unraid-1 kernel: netif_receive_skb+0xbf/0x127
    May  2 06:56:23 Unraid-1 kernel: br_handle_frame_finish+0x438/0x472 [bridge]
    May  2 06:56:23 Unraid-1 kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
    May  2 06:56:23 Unraid-1 kernel: br_nf_hook_thresh+0xe2/0x109 [br_netfilter]
    May  2 06:56:23 Unraid-1 kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
    May  2 06:56:23 Unraid-1 kernel: br_nf_pre_routing_finish+0x2c1/0x2ec [br_netfilter]
    May  2 06:56:23 Unraid-1 kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
    May  2 06:56:23 Unraid-1 kernel: ? NF_HOOK.isra.0+0xe4/0x140 [br_netfilter]
    May  2 06:56:23 Unraid-1 kernel: ? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
    May  2 06:56:23 Unraid-1 kernel: br_nf_pre_routing+0x236/0x24a [br_netfilter]
    May  2 06:56:23 Unraid-1 kernel: ? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
    May  2 06:56:23 Unraid-1 kernel: br_handle_frame+0x277/0x2e0 [bridge]
    May  2 06:56:23 Unraid-1 kernel: ? kernel_restart_prepare+0x2a/0x32
    May  2 06:56:23 Unraid-1 kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
    May  2 06:56:23 Unraid-1 kernel: __netif_receive_skb_core.constprop.0+0x4fa/0x6e9
    May  2 06:56:23 Unraid-1 kernel: ? place_entity+0x6e/0xae
    May  2 06:56:23 Unraid-1 kernel: ? paravirt_write_msr+0xb/0x11
    May  2 06:56:23 Unraid-1 kernel: ? check_preempt_wakeup+0x174/0x1f2
    May  2 06:56:23 Unraid-1 kernel: __netif_receive_skb_list_core+0x8a/0x11e
    May  2 06:56:23 Unraid-1 kernel: netif_receive_skb_list_internal+0x1d2/0x20b
    May  2 06:56:23 Unraid-1 kernel: gro_normal_list+0x1d/0x3f
    May  2 06:56:23 Unraid-1 kernel: napi_complete_done+0x7b/0x11a
    May  2 06:56:23 Unraid-1 kernel: e1000e_poll+0x9e/0x23e [e1000e]
    May  2 06:56:23 Unraid-1 kernel: __napi_poll.constprop.0+0x28/0x124
    May  2 06:56:23 Unraid-1 kernel: net_rx_action+0x159/0x24f
    May  2 06:56:23 Unraid-1 kernel: ? e1000_intr_msi+0x118/0x124 [e1000e]
    May  2 06:56:23 Unraid-1 kernel: __do_softirq+0x126/0x288
    May  2 06:56:23 Unraid-1 kernel: __irq_exit_rcu+0x5e/0xb8
    May  2 06:56:23 Unraid-1 kernel: common_interrupt+0x3b/0xc1
    May  2 06:56:23 Unraid-1 kernel: asm_common_interrupt+0x22/0x40
    May  2 06:56:23 Unraid-1 kernel: RIP: 0033:0x1533844d49f6
    May  2 06:56:23 Unraid-1 kernel: Code: 1f 44 00 00 48 8b 54 24 18 48 8b fa e8 03 67 ff ff 90 49 3b c6 0f 84 d7 fe ff ff 48 8b 78 18 49 8d 0c 3e 48 3b 4c 24 20 74 9a <48> 85 ff 0f 84 19 00 00 00 48 8b 7c 24 20 48 8b f1 48 8b 44 24 30
    May  2 06:56:23 Unraid-1 kernel: RSP: 002b:00007ffc1826be40 EFLAGS: 00000212
    May  2 06:56:23 Unraid-1 kernel: RAX: 000015337f823618 RBX: 000015337ed4b838 RCX: 000015337e555810
    May  2 06:56:23 Unraid-1 kernel: RDX: 0000000000000400 RSI: 0000000003b23618 RDI: 0000000002855810
    May  2 06:56:23 Unraid-1 kernel: RBP: 000015337f823618 R08: 00000000145c7530 R09: 000015336bcdf7b0
    May  2 06:56:23 Unraid-1 kernel: R10: 000015336a28cc10 R11: 000015336bcd0ad8 R12: 000015336bcdf7b0
    May  2 06:56:23 Unraid-1 kernel: R13: 000015337fc7f6a8 R14: 000015337bd00000 R15: 0000153394c6d300
    May  2 06:56:23 Unraid-1 kernel: </TASK>
    May  2 06:56:23 Unraid-1 kernel: ---[ end trace 0000000000000000 ]---
    May  2 06:56:25 Unraid-1 kernel: process '4a104bf4d9746c31192addd040baa60c09299ac6cc5fba22a56231040e70ff26/usr/bin/par2' started with executable stack
    May  2 06:59:56 Unraid-1 ntpd[1331]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

     

    and a second and third one in the logs from syslog server:

     

    2023-05-05T22:13:09+02:00 Unraid-1 kernel: ------------[ cut here ]------------
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: refcount_t: addition on 0; use-after-free.
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: WARNING: CPU: 9 PID: 152 at lib/refcount.c:25 refcount_warn_saturate+0x90/0x100
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: Modules linked in: cmac cifs asn1_decoder cifs_arc4 cifs_md4 dns_resolver tls nft_chain_nat xt_owner nft_compat nf_tables xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_nat xt_tcpudp ip6table_mangle iptable_mangle vhost_net tun vhost vhost_iotlb tap macvlan xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat xt_addrtype br_netfilter veth xfs xt_MASQUERADE ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod tcp_diag inet_diag af_packet kvmgt mdev i915 iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper drm_kms_helper drm intel_gtt agpgart syscopyarea sysfillrect sysimgblt fb_sys_fops nct6775 nct6775_core hwmon_vid efivarfs wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables 8021q garp mrp bridge stp llc x86_pkg_temp_thermal intel_powerclamp
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel crypto_simd cryptd rapl intel_cstate mei_hdcp mei_pxp wmi_bmof i2c_i801 nvme mpt3sas intel_uncore e1000e mei_me i2c_smbus nvme_core cp210x intel_pch_thermal mei i2c_core joydev usbserial raid_class scsi_transport_sas tpm_crb video tpm_tis tpm_tis_core wmi backlight tpm intel_pmc_core acpi_pad acpi_tad button unix
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: CPU: 9 PID: 152 Comm: kswapd0 Tainted: G     U  W          6.1.27-Unraid #1
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H470M-ITX/ac, BIOS L1.22 12/07/2020
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: RIP: 0010:refcount_warn_saturate+0x90/0x100
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: Code: 00 01 e8 85 94 c3 ff 0f 0b c3 cc cc cc cc 80 3d 6e bc f5 00 00 75 7e 48 c7 c7 bb 8f 0b 82 c6 05 5e bc f5 00 01 e8 62 94 c3 ff <0f> 0b c3 cc cc cc cc 80 3d 4a bc f5 00 00 75 5b 48 c7 c7 e7 8f 0b
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: RSP: 0018:ffffc900005e79d8 EFLAGS: 00010282
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: RAX: 0000000000000000 RBX: ffff888107fe0b88 RCX: 0000000000000027
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: RDX: 0000000000000003 RSI: ffffffff8208f7cd RDI: 00000000ffffffff
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: RBP: ffff8881630b1de0 R08: 0000000000000000 R09: ffffffff82245e30
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: R10: 00007fffffffffff R11: 0000000000000000 R12: 0000000000000000
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: R13: ffff8881de978d98 R14: 0000000000000000 R15: 0000000000000000
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: FS:  0000000000000000(0000) GS:ffff88883f840000(0000) knlGS:0000000000000000
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: CR2: 00001480936e2000 CR3: 00000001dc554002 CR4: 00000000007706e0
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: PKRU: 55555554
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: Call Trace:
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: <TASK>
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: __set_extent_bit+0x27f/0x499
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: ? btrfs_drop_extent_map_range+0x78/0x453
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: lock_extent+0x46/0xcf
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: btrfs_evict_inode+0x114/0x340
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: ? autoremove_wake_function+0x33/0x33
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: evict+0xb9/0x150
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: dispose_list+0x35/0x42
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: prune_icache_sb+0x51/0x73
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: super_cache_scan+0x123/0x17c
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: do_shrink_slab+0x188/0x2a1
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: shrink_slab+0x1f9/0x267
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: shrink_node+0x318/0x549
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: balance_pgdat+0x4e9/0x6a2
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: ? newidle_balance+0x289/0x30a
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: ? _raw_spin_unlock+0x14/0x29
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: ? raw_spin_rq_unlock_irq+0x5/0x10
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: ? finish_task_switch.isra.0+0x140/0x218
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: kswapd+0x2f0/0x333
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: ? _raw_spin_rq_lock_irqsave+0x20/0x20
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: ? balance_pgdat+0x6a2/0x6a2
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: kthread+0xe4/0xef
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: ? kthread_complete_and_exit+0x1b/0x1b
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: ret_from_fork+0x1f/0x30
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: </TASK>
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: ---[ end trace 0000000000000000 ]---
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: ------------[ cut here ]------------
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: refcount_t: underflow; use-after-free.
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: WARNING: CPU: 9 PID: 152 at lib/refcount.c:28 refcount_warn_saturate+0xb3/0x100
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: Modules linked in: cmac cifs asn1_decoder cifs_arc4 cifs_md4 dns_resolver tls nft_chain_nat xt_owner nft_compat nf_tables xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_nat xt_tcpudp ip6table_mangle iptable_mangle vhost_net tun vhost vhost_iotlb tap macvlan xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat xt_addrtype br_netfilter veth xfs xt_MASQUERADE ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod tcp_diag inet_diag af_packet kvmgt mdev i915 iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper drm_kms_helper drm intel_gtt agpgart syscopyarea sysfillrect sysimgblt fb_sys_fops nct6775 nct6775_core hwmon_vid efivarfs wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables 8021q garp mrp bridge stp llc x86_pkg_temp_thermal intel_powerclamp
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel crypto_simd cryptd rapl intel_cstate mei_hdcp mei_pxp wmi_bmof i2c_i801 nvme mpt3sas intel_uncore e1000e mei_me i2c_smbus nvme_core cp210x intel_pch_thermal mei i2c_core joydev usbserial raid_class scsi_transport_sas tpm_crb video tpm_tis tpm_tis_core wmi backlight tpm intel_pmc_core acpi_pad acpi_tad button unix
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: CPU: 9 PID: 152 Comm: kswapd0 Tainted: G     U  W          6.1.27-Unraid #1
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H470M-ITX/ac, BIOS L1.22 12/07/2020
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: RIP: 0010:refcount_warn_saturate+0xb3/0x100
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: Code: 00 01 e8 62 94 c3 ff 0f 0b c3 cc cc cc cc 80 3d 4a bc f5 00 00 75 5b 48 c7 c7 e7 8f 0b 82 c6 05 3a bc f5 00 01 e8 3f 94 c3 ff <0f> 0b c3 cc cc cc cc 80 3d 26 bc f5 00 00 75 38 48 c7 c7 0f 90 0b
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: RSP: 0018:ffffc900005e7a38 EFLAGS: 00010282
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: RAX: 0000000000000000 RBX: ffff888107fe0b88 RCX: 0000000000000027
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: RDX: 0000000000000003 RSI: ffffffff8208f7cd RDI: 00000000ffffffff
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffffff82245e30
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: R10: 00007fffffffffff R11: 0000000000000000 R12: 00000000ffff7fff
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: R13: ffff8881630b1de0 R14: 0000000000000000 R15: ffffc900005e7ad0
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: FS:  0000000000000000(0000) GS:ffff88883f840000(0000) knlGS:0000000000000000
    2023-05-05T22:13:09+02:00 Unraid-1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    2023-05-05T22:22:46+02:00 Unraid-1 kernel: mdcmd (93): set md_write_method 0
    2023-05-05T22:22:46+02:00 Unraid-1 kernel: 
    2023-05-05T22:42:46+02:00 Unraid-1 kernel: mdcmd (94): set md_write_method 1
    2023-05-05T22:42:46+02:00 Unraid-1 kernel: 
    2023-05-05T23:52:46+02:00 Unraid-1 kernel: mdcmd (95): set md_write_method 0
    2023-05-05T23:52:46+02:00 Unraid-1 kernel: 
    2023-05-06T00:02:46+02:00 Unraid-1 kernel: mdcmd (96): set md_write_method 1
    2023-05-06T00:02:46+02:00 Unraid-1 kernel: 

     

    two hours before the server crashed at 2023-05-06T00:02:46+02:00

     

    this happend when i was sleeping.

    I checked my schedules, crons, userscript, and container, but i didn't found anything which kicked in on 00:00 AM.

    Link to comment

    Thanks.

    Those second and third call traces are caused by something else, don't know exactly what but it is not by  macvlan.

     

    Link to comment
    24 minutes ago, bonienl said:

    Those second and third call traces are caused by something else, don't know exactly what but it is not by  macvlan.

    Just to add that this is fairly typical with the macvaln issue, first you see some macvlan call traces, then there might be some more which might not look related until the server crashes, but since the first call trace is about macvaln it leaves a trace in the syslog, making it more easy to identify the root cause.

    Link to comment

    ok, Server hard crashed again

     

    i putted some traffic on there yesterday, some today ... now i wanted to check logs again and its down.

     

    @bonienl sadly the "change" didnt work out, i ll give it another try with host access off (i turned it on with the last change)

     

    there was visible only 1 macvlan error on 20230506 ~ 2.10 am (idle), no more to see here ...

     

    it went offline pretty sure here with 0 traffic (just a HA meter but ...)

     

    image.png.bdd4e3e334744b988ea7dbffa1757651.png

     

    no web, smb, ssh, docker(s), ... short, dead ;)

    Link to comment

    Thanks for testing.

    For the moment, it seems the only reliable macvlan usage is without bridging enabled.

     

    Link to comment
    9 hours ago, bonienl said:

    For the moment, it seems the only reliable macvlan usage is without bridging enabled.

     

    ok, thanks for taking the time.

     

    just as small note, i rebooted yesterday like described, had it completely idle ... no errors in log.

     

    now just restarted a docker and put small traffic on tvheadend and ... here the snipplet from starting docker.d <esterday to today ...

     

    May  7 20:40:36 AlsServerII root: starting dockerd ...
    May  7 20:40:38 AlsServerII rc.docker: created network br0 with subnets: 192.168.1.0/24; 2a02:810b:56bf:dc30::/64; 
    May  7 20:40:49 AlsServerII kernel: eth0: renamed from veth49a7b60
    May  7 20:40:49 AlsServerII kernel: device br0 entered promiscuous mode
    May  7 20:59:04 AlsServerII emhttpd: spinning down /dev/sdb
    May  7 21:19:48 AlsServerII emhttpd: read SMART /dev/sdb
    May  7 21:36:49 AlsServerII emhttpd: spinning down /dev/sdb
    May  8 00:10:04 AlsServerII emhttpd: read SMART /dev/sdb
    May  8 00:27:00 AlsServerII emhttpd: spinning down /dev/sdb
    May  8 01:00:01 AlsServerII Docker Auto Update: Community Applications Docker Autoupdate running
    May  8 01:00:01 AlsServerII Docker Auto Update: Checking for available updates
    May  8 01:00:04 AlsServerII emhttpd: read SMART /dev/sdb
    May  8 01:00:09 AlsServerII Docker Auto Update: No updates will be installed
    May  8 01:16:51 AlsServerII emhttpd: spinning down /dev/sdb
    May  8 01:20:01 AlsServerII Plugin Auto Update: Checking for available plugin updates
    May  8 01:20:08 AlsServerII Plugin Auto Update: Auto Updating community.applications.plg
    May  8 01:20:09 AlsServerII root: plugin: running: anonymous
    May  8 01:20:09 AlsServerII root: plugin: running: anonymous
    May  8 01:20:09 AlsServerII root: plugin: creating: /boot/config/plugins/community.applications/community.applications-2023.05.07a-x86_64-1.txz - downloading from URL https://raw.githubusercontent.com/Squidly271/community.applications/master/archive/community.applications-2023.05.07a-x86_64-1.txz
    May  8 01:20:10 AlsServerII root: plugin: checking: /boot/config/plugins/community.applications/community.applications-2023.05.07a-x86_64-1.txz - MD5
    May  8 01:20:10 AlsServerII root: plugin: running: upgradepkg --install-new --reinstall /boot/config/plugins/community.applications/community.applications-2023.05.07a-x86_64-1.txz
    May  8 01:20:10 AlsServerII root: plugin: running: anonymous
    May  8 01:20:10 AlsServerII root: plugin: community.applications.plg updated
    May  8 01:20:12 AlsServerII Plugin Auto Update: Auto Updating parity.check.tuning.plg
    May  8 01:20:12 AlsServerII root: plugin: running: anonymous
    May  8 01:20:12 AlsServerII root: plugin: creating: /boot/config/plugins/parity.check.tuning/parity.check.tuning-2023.05.07.txz - downloading from URL https://raw.githubusercontent.com/itimpi/parity.check.tuning/master/archives/parity.check.tuning-2023.05.07.txz
    May  8 01:20:12 AlsServerII root: plugin: running: upgradepkg --install-new /boot/config/plugins/parity.check.tuning/parity.check.tuning-2023.05.07.txz
    May  8 01:20:13 AlsServerII root: plugin: running: anonymous
    May  8 01:20:13 AlsServerII root: plugin: parity.check.tuning.plg updated
    May  8 01:20:14 AlsServerII Plugin Auto Update: Checking for language updates
    May  8 01:20:15 AlsServerII Plugin Auto Update: Community Applications Plugin Auto Update finished
    May  8 05:57:33 AlsServerII emhttpd: read SMART /dev/sdb
    May  8 06:06:58 AlsServerII kernel: device br0 left promiscuous mode
    May  8 06:06:58 AlsServerII kernel: veth49a7b60: renamed from eth0
    May  8 06:06:59 AlsServerII kernel: eth0: renamed from veth8d82cef
    May  8 06:06:59 AlsServerII kernel: device br0 entered promiscuous mode
    May  8 06:26:33 AlsServerII kernel: ------------[ cut here ]------------
    May  8 06:26:33 AlsServerII kernel: WARNING: CPU: 1 PID: 15585 at net/netfilter/nf_conntrack_core.c:1211 __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
    May  8 06:26:33 AlsServerII kernel: Modules linked in: xt_nat xt_tcpudp macvlan bluetooth ecdh_generic ecc cmac cifs asn1_decoder cifs_arc4 cifs_md4 oid_registry dns_resolver xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat xt_addrtype br_netfilter xfs xt_MASQUERADE ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 md_mod tcp_diag inet_diag nct6775 nct6775_core hwmon_vid efivarfs ip6table_filter ip6_tables iptable_filter ip_tables x_tables 8021q garp mrp bridge stp llc x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel i915 kvm iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper crct10dif_pclmul crc32_pclmul crc32c_intel drm_kms_helper ghash_clmulni_intel sha512_ssse3 mei_pxp mei_hdcp drm aesni_intel crypto_simd intel_gtt cryptd rapl i2c_i801 agpgart intel_cstate mei_me i2c_smbus r8169 i2c_core ahci realtek libahci mei syscopyarea sysfillrect sysimgblt fb_sys_fops thermal fan button video wmi backlight intel_pmc_core unix
    May  8 06:26:33 AlsServerII kernel: CPU: 1 PID: 15585 Comm: kworker/u8:2 Not tainted 6.1.27-Unraid #1
    May  8 06:26:33 AlsServerII kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./J3355M, BIOS P1.90 11/27/2018
    May  8 06:26:33 AlsServerII kernel: Workqueue: events_unbound macvlan_process_broadcast [macvlan]
    May  8 06:26:33 AlsServerII kernel: RIP: 0010:__nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
    May  8 06:26:33 AlsServerII kernel: Code: 44 24 10 e8 f4 e1 ff ff 8b 7c 24 04 89 ea 89 c6 89 04 24 e8 76 e6 ff ff 84 c0 75 a2 48 89 df e8 ad e2 ff ff 85 c0 89 c5 74 18 <0f> 0b 8b 34 24 8b 7c 24 04 e8 2a dd ff ff e8 8b e3 ff ff e9 72 01
    May  8 06:26:33 AlsServerII kernel: RSP: 0018:ffffc900000fcd98 EFLAGS: 00010202
    May  8 06:26:33 AlsServerII kernel: RAX: 0000000000000001 RBX: ffff888147964300 RCX: df117f1db0e5f651
    May  8 06:26:33 AlsServerII kernel: RDX: 0000000000000000 RSI: 0000000000000112 RDI: ffff888147964300
    May  8 06:26:33 AlsServerII kernel: RBP: 0000000000000001 R08: 4bd3c8213a28cba8 R09: 24428f35b7be7bf4
    May  8 06:26:33 AlsServerII kernel: R10: 90e86a8e9e578041 R11: ffffc900000fcd60 R12: ffffffff82a0e440
    May  8 06:26:33 AlsServerII kernel: R13: 000000000000c6ab R14: ffff88815a192500 R15: 0000000000000000
    May  8 06:26:33 AlsServerII kernel: FS:  0000000000000000(0000) GS:ffff888277e80000(0000) knlGS:0000000000000000
    May  8 06:26:33 AlsServerII kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    May  8 06:26:33 AlsServerII kernel: CR2: 00001456add1e1e0 CR3: 000000000420a000 CR4: 00000000003506e0
    May  8 06:26:33 AlsServerII kernel: Call Trace:
    May  8 06:26:33 AlsServerII kernel: <IRQ>
    May  8 06:26:33 AlsServerII kernel: ? nf_nat_inet_fn+0x60/0x1a8 [nf_nat]
    May  8 06:26:33 AlsServerII kernel: nf_conntrack_confirm+0x25/0x54 [nf_conntrack]
    May  8 06:26:33 AlsServerII kernel: nf_hook_slow+0x3a/0x96
    May  8 06:26:33 AlsServerII kernel: ? ip_protocol_deliver_rcu+0x164/0x164
    May  8 06:26:33 AlsServerII kernel: NF_HOOK.constprop.0+0x79/0xd9
    May  8 06:26:33 AlsServerII kernel: ? ip_protocol_deliver_rcu+0x164/0x164
    May  8 06:26:33 AlsServerII kernel: __netif_receive_skb_one_core+0x77/0x9c
    May  8 06:26:33 AlsServerII kernel: process_backlog+0x8c/0x116
    May  8 06:26:33 AlsServerII kernel: __napi_poll.constprop.0+0x28/0x124
    May  8 06:26:33 AlsServerII kernel: net_rx_action+0x159/0x24f
    May  8 06:26:33 AlsServerII kernel: __do_softirq+0x126/0x288
    May  8 06:26:33 AlsServerII kernel: do_softirq+0x7f/0xab
    May  8 06:26:33 AlsServerII kernel: </IRQ>
    May  8 06:26:33 AlsServerII kernel: <TASK>
    May  8 06:26:33 AlsServerII kernel: __local_bh_enable_ip+0x4c/0x6b
    May  8 06:26:33 AlsServerII kernel: netif_rx+0x52/0x5a
    May  8 06:26:33 AlsServerII kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
    May  8 06:26:33 AlsServerII kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]
    May  8 06:26:33 AlsServerII kernel: process_one_work+0x1a8/0x295
    May  8 06:26:33 AlsServerII kernel: worker_thread+0x18b/0x244
    May  8 06:26:33 AlsServerII kernel: ? rescuer_thread+0x281/0x281
    May  8 06:26:33 AlsServerII kernel: kthread+0xe4/0xef
    May  8 06:26:33 AlsServerII kernel: ? kthread_complete_and_exit+0x1b/0x1b
    May  8 06:26:33 AlsServerII kernel: ret_from_fork+0x1f/0x30
    May  8 06:26:33 AlsServerII kernel: </TASK>
    May  8 06:26:33 AlsServerII kernel: ---[ end trace 0000000000000000 ]---

     

    so in the end, traffic on this specific network is causing it (and prolly also speeding up) and crashing something then.

     

    i ll try now again the eth0 method (no bridging) and recheck this, last time the server completely crashed there too (but different).

    Link to comment

    my server crashed 4 times in the last two day, so i had to go back back to 6.11.5.

    if there is something more to try/test, let me know.

    Link to comment
    On 5/7/2023 at 8:36 PM, bonienl said:

    Thanks for testing.

    For the moment, it seems the only reliable macvlan usage is without bridging enabled.

    i think for now i can confirm this

     

    with more traffic on the mashine (small testserver with eth0 setup) i get 0 errors currently.

     

    root@AlsServerII:~# uptime 
     04:38:23 up 4 days, 15:18,  0 users,  load average: 0.03, 0.07, 0.02
    root@AlsServerII:~#

     

    only once nginx had some issues, but i assume its more related to the 8 GB Ram only ;) not related to this topic.

     

    syslog attached just as comparision.

     

    as note, the main Server with 6.11.5 downgraded and bridged macvlan "normal" setup here.

    root@AlsServer:~# uptime 
     04:41:35 up 20 days, 20:42,  0 users,  load average: 0.45, 0.33, 0.29
    root@AlsServer:~# 

    also no more issues since downgraded ... ;)

    alsserverii-syslog-20230513-0237.zip

    Link to comment
    2 hours ago, alturismo said:

    also no more issues since downgraded ... ;)

    i can confirm this, after downgrade my server runs for 3 days without any traces.

    Link to comment
    On 5/13/2023 at 3:45 AM, alturismo said:
    On 5/7/2023 at 7:36 PM, bonienl said:

    Thanks for testing.

    For the moment, it seems the only reliable macvlan usage is without bridging enabled.

    i think for now i can confirm this

     

    with more traffic on the mashine (small testserver with eth0 setup) i get 0 errors currently.

    @alturismo just to confirm, if bridge is enabled, you see the call traces even without any VM using the bridge at the same time correct?

    Link to comment
    1 hour ago, JorgeB said:

    @alturismo just to confirm, if bridge is enabled, you see the call traces even without any VM using the bridge at the same time correct?

    the small Test Server doesnt even have VM enabled, so its not VM related (what i 1st thought from my Main Server)

     

    and if we talk about bridge, macvlan br0 mode is the one which is causing the issues ...

     

    sample (now from the 6.11.5 mashine which is working) from a docker setup

    image.thumb.png.233e7b5ad153d0bf0e86ec36fa00ea24.png

     

    the 6.12rc Test Server is now running fine since 7 days + with eth0 setup ...

    image.thumb.png.87d6ad9423cb728831f34349bc2038f6.png

     

    if i use br0 mode on 6.12rc (2 local mashines, totally different hardware) the Server's will always startup with the posted errors and always will crash after few hours or few days (max where 4 days or so without traffic on the Server), doesnt matter if VM is on, off, active, disabled, ...

     

    hope its understandable ;)

    • Like 2
    Link to comment

    I have some macvlan errors but no crashes so far. With 6.11 everything was fine.

     

    Regarding the fix by disable bridging, is that even an option? if I disable bridging how can I give an static IP to every docker?

    Link to comment
    2 hours ago, L0rdRaiden said:

    I have some macvlan errors but no crashes so far. With 6.11 everything was fine.

     

    Regarding the fix by disable bridging, is that even an option? if I disable bridging how can I give an static IP to every docker?

     

    Sure it is an option if you are NOT using VMs.

    When bridge is disabled you need to use the regular interfaces, like eth0 (see docker settings).

    Assignment of static IP addresses needs to be redone for the new interface(s) for each container, but works as before.

     

    • Thanks 1
    Link to comment

    Following as this release broke it for me as well. I switched to ipvlan but it really sucks because I can’t have static ips/macs now. All ips share 1 mac meaning I can no longer manage in my firewall. Quite a pain but I spose better than crashing all the time. 

    Link to comment

    @alturismocan you re-test with rc7? Newer kernel appears do have some possible related changes, most likely it will be the same but still worth a try.

    Link to comment
    7 hours ago, JorgeB said:

    can you re-test with rc7?

    sure, just give me some days, currently on vacation in Greece, when i find a spot i ll do so ;)

    • Like 1
    Link to comment

    @JorgeB I did a short test, but it doen't look good:

     

    Jun  7 08:08:58 Unraid-1 kernel: docker0: port 16(vethcb01385) entered forwarding state
    Jun  7 08:09:59 Unraid-1 webGUI: Successful login user root from 192.168.0.6
    Jun  7 08:16:00 Unraid-1 root: Fix Common Problems Version 2023.04.26
    Jun  7 08:16:01 Unraid-1 root: Fix Common Problems: Warning: Docker Update Check not enabled ** Ignored
    Jun  7 08:16:22 Unraid-1 root: Fix Common Problems: Warning: The plugin un-get.plg is not known to Community Applications and is possibly incompatible with your server ** Ignored
    Jun  7 08:16:23 Unraid-1 root: Fix Common Problems: Warning: The plugin unraid-versionchange.plg is not known to Community Applications and is possibly incompatible with your server ** Ignored
    Jun  7 08:19:11 Unraid-1 kernel: ------------[ cut here ]------------
    Jun  7 08:19:11 Unraid-1 kernel: WARNING: CPU: 7 PID: 29046 at net/netfilter/nf_conntrack_core.c:1210 __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
    Jun  7 08:19:11 Unraid-1 kernel: Modules linked in: cmac cifs asn1_decoder cifs_arc4 cifs_md4 dns_resolver tls nft_chain_nat xt_owner nft_compat nf_tables xt_nat xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle iptable_mangle vhost_net tun vhost vhost_iotlb tap macvlan xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat xt_addrtype br_netfilter veth xfs xt_MASQUERADE ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag af_packet nct6775 nct6775_core hwmon_vid wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs 8021q garp mrp bridge stp llc i915 x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm iosf_mbi drm_buddy i2c_algo_bit ttm
    Jun  7 08:19:11 Unraid-1 kernel: crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 drm_display_helper aesni_intel crypto_simd drm_kms_helper cryptd mei_hdcp mei_pxp rapl intel_cstate drm wmi_bmof mei_me nvme mpt3sas intel_uncore e1000e intel_gtt i2c_i801 agpgart nvme_core i2c_smbus cp210x mei syscopyarea i2c_core sysfillrect usbserial sysimgblt joydev raid_class fb_sys_fops scsi_transport_sas intel_pch_thermal tpm_crb video tpm_tis tpm_tis_core wmi tpm backlight acpi_tad intel_pmc_core acpi_pad button unix
    Jun  7 08:19:11 Unraid-1 kernel: CPU: 7 PID: 29046 Comm: kworker/u24:18 Tainted: P           O       6.1.32-Unraid #1
    Jun  7 08:19:11 Unraid-1 kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H470M-ITX/ac, BIOS L1.22 12/07/2020
    Jun  7 08:19:11 Unraid-1 kernel: Workqueue: events_unbound macvlan_process_broadcast [macvlan]
    Jun  7 08:19:11 Unraid-1 kernel: RIP: 0010:__nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
    Jun  7 08:19:11 Unraid-1 kernel: Code: 44 24 10 e8 e2 e1 ff ff 8b 7c 24 04 89 ea 89 c6 89 04 24 e8 7e e6 ff ff 84 c0 75 a2 48 89 df e8 9b e2 ff ff 85 c0 89 c5 74 18 <0f> 0b 8b 34 24 8b 7c 24 04 e8 18 dd ff ff e8 93 e3 ff ff e9 72 01
    Jun  7 08:19:11 Unraid-1 kernel: RSP: 0018:ffffc900002a0d98 EFLAGS: 00010202
    Jun  7 08:19:11 Unraid-1 kernel: RAX: 0000000000000001 RBX: ffff8881deab9600 RCX: f50c3c8cd53db18b
    Jun  7 08:19:11 Unraid-1 kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8881deab9600
    Jun  7 08:19:11 Unraid-1 kernel: RBP: 0000000000000001 R08: 4ec3617a21e80860 R09: 7cbf210d91d63b67
    Jun  7 08:19:11 Unraid-1 kernel: R10: ac3d1ea7a12092c9 R11: ffffc900002a0d60 R12: ffffffff82a11440
    Jun  7 08:19:11 Unraid-1 kernel: R13: 0000000000037df1 R14: ffff888102c19e00 R15: 0000000000000000
    Jun  7 08:19:11 Unraid-1 kernel: FS:  0000000000000000(0000) GS:ffff88883f7c0000(0000) knlGS:0000000000000000
    Jun  7 08:19:11 Unraid-1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    Jun  7 08:19:11 Unraid-1 kernel: CR2: 00007fffe876e038 CR3: 000000000420a002 CR4: 00000000007706e0
    Jun  7 08:19:11 Unraid-1 kernel: PKRU: 55555554
    Jun  7 08:19:11 Unraid-1 kernel: Call Trace:
    Jun  7 08:19:11 Unraid-1 kernel: <IRQ>
    Jun  7 08:19:11 Unraid-1 kernel: ? __warn+0xab/0x122
    Jun  7 08:19:11 Unraid-1 kernel: ? report_bug+0x109/0x17e
    Jun  7 08:19:11 Unraid-1 kernel: ? __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
    Jun  7 08:19:11 Unraid-1 kernel: ? handle_bug+0x41/0x6f
    Jun  7 08:19:11 Unraid-1 kernel: ? exc_invalid_op+0x13/0x60
    Jun  7 08:19:11 Unraid-1 kernel: ? asm_exc_invalid_op+0x16/0x20
    Jun  7 08:19:11 Unraid-1 kernel: ? __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
    Jun  7 08:19:11 Unraid-1 kernel: ? __nf_conntrack_confirm+0x9e/0x2b0 [nf_conntrack]
    Jun  7 08:19:11 Unraid-1 kernel: ? nf_nat_inet_fn+0x123/0x1a8 [nf_nat]
    Jun  7 08:19:11 Unraid-1 kernel: nf_conntrack_confirm+0x25/0x54 [nf_conntrack]
    Jun  7 08:19:11 Unraid-1 kernel: nf_hook_slow+0x3a/0x96
    Jun  7 08:19:11 Unraid-1 kernel: ? ip_protocol_deliver_rcu+0x164/0x164
    Jun  7 08:19:11 Unraid-1 kernel: NF_HOOK.constprop.0+0x79/0xd9
    Jun  7 08:19:11 Unraid-1 kernel: ? ip_protocol_deliver_rcu+0x164/0x164
    Jun  7 08:19:11 Unraid-1 kernel: __netif_receive_skb_one_core+0x77/0x9c
    Jun  7 08:19:11 Unraid-1 kernel: process_backlog+0x8c/0x116
    Jun  7 08:19:11 Unraid-1 kernel: __napi_poll.constprop.0+0x28/0x124
    Jun  7 08:19:11 Unraid-1 kernel: net_rx_action+0x159/0x24f
    Jun  7 08:19:11 Unraid-1 kernel: __do_softirq+0x126/0x288
    Jun  7 08:19:11 Unraid-1 kernel: do_softirq+0x7f/0xab
    Jun  7 08:19:11 Unraid-1 kernel: </IRQ>
    Jun  7 08:19:11 Unraid-1 kernel: <TASK>
    Jun  7 08:19:11 Unraid-1 kernel: __local_bh_enable_ip+0x4c/0x6b
    Jun  7 08:19:11 Unraid-1 kernel: netif_rx+0x52/0x5a
    Jun  7 08:19:11 Unraid-1 kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
    Jun  7 08:19:11 Unraid-1 kernel: ? _raw_spin_unlock+0x14/0x29
    Jun  7 08:19:11 Unraid-1 kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]
    Jun  7 08:19:11 Unraid-1 kernel: process_one_work+0x1a8/0x295
    Jun  7 08:19:11 Unraid-1 kernel: worker_thread+0x18b/0x244
    Jun  7 08:19:11 Unraid-1 kernel: ? rescuer_thread+0x281/0x281
    Jun  7 08:19:11 Unraid-1 kernel: kthread+0xe4/0xef
    Jun  7 08:19:11 Unraid-1 kernel: ? kthread_complete_and_exit+0x1b/0x1b
    Jun  7 08:19:11 Unraid-1 kernel: ret_from_fork+0x1f/0x30
    Jun  7 08:19:11 Unraid-1 kernel: </TASK>
    Jun  7 08:19:11 Unraid-1 kernel: ---[ end trace 0000000000000000 ]---

     

    unraid-1-diagnostics-20230607-0825.zip

    • Like 1
    Link to comment

    Hi. I'm using 6.12.1 and I can see a trace  on my QNAP NAS - but no crash of server / everything works fine

    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ------------[ cut here ]------------
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: WARNING: CPU: 1 PID: 14256 at net/netfilter/nf_conntrack_core.c:1210 __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: Modules linked in: udp_diag macvlan xt_nat xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs dm_crypt dm_mod md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs af_packet 8021q garp mrp bridge stp llc bonding tls i915 x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm iosf_mbi drm_buddy crct10dif_pclmul crc32_pclmul ttm crc32c_intel drm_display_helper ghash_clmulni_intel sha512_ssse3 mei_hdcp mei_pxp drm_kms_helper drm aesni_intel igb crypto_simd cryptd rapl intel_cstate ahci mei_me intel_gtt agpgart i2c_i801 i2c_algo_bit i2c_smbus i2c_core libahci mei syscopyarea sysfillrect sysimgblt
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: fb_sys_fops thermal fan video wmi backlight intel_pmc_core button unix
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: CPU: 1 PID: 14256 Comm: kworker/u8:2 Tainted: P           O       6.1.34-Unraid #1
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: Hardware name: Default string Default string/Default string, BIOS Q04WAR07 03/12/2020
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: Workqueue: events_unbound macvlan_process_broadcast [macvlan]
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: RIP: 0010:__nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: Code: 44 24 10 e8 e2 e1 ff ff 8b 7c 24 04 89 ea 89 c6 89 04 24 e8 7e e6 ff ff 84 c0 75 a2 48 89 df e8 9b e2 ff ff 85 c0 89 c5 74 18 <0f> 0b 8b 34 24 8b 7c 24 04 e8 18 dd ff ff e8 93 e3 ff ff e9 72 01
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: RSP: 0018:ffffc90000100d98 EFLAGS: 00010202
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: RAX: 0000000000000001 RBX: ffff88810628c400 RCX: f5f194a9dae5fa12
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88810628c400
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: RBP: 0000000000000001 R08: 1b693dbe4dee8d35 R09: 1fc6fb5782e3218f
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: R10: 0c6b8e75b627422c R11: ffffc90000100d60 R12: ffffffff82a11440
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: R13: 000000000003a40a R14: ffff88813f4bd400 R15: 0000000000000000
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: FS:  0000000000000000(0000) GS:ffff8881f9c80000(0000) knlGS:0000000000000000
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: CR2: 0000000000453cd8 CR3: 000000000420a000 CR4: 0000000000350ee0
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: Call Trace:
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: <IRQ>
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ? __warn+0xab/0x122
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ? report_bug+0x109/0x17e
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ? __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ? handle_bug+0x41/0x6f
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ? exc_invalid_op+0x13/0x60
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ? asm_exc_invalid_op+0x16/0x20
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ? __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ? __nf_conntrack_confirm+0x9e/0x2b0 [nf_conntrack]
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ? nf_nat_inet_fn+0x123/0x1a8 [nf_nat]
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: nf_conntrack_confirm+0x25/0x54 [nf_conntrack]
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: nf_hook_slow+0x3a/0x96
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ? ip_protocol_deliver_rcu+0x164/0x164
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: NF_HOOK.constprop.0+0x79/0xd9
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ? ip_protocol_deliver_rcu+0x164/0x164
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: __netif_receive_skb_one_core+0x77/0x9c
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: process_backlog+0x8c/0x116
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: __napi_poll.constprop.0+0x28/0x124
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: net_rx_action+0x159/0x24f
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: __do_softirq+0x126/0x288
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: do_softirq+0x7f/0xab
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: </IRQ>
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: <TASK>
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: __local_bh_enable_ip+0x4c/0x6b
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: netif_rx+0x52/0x5a
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: process_one_work+0x1a8/0x295
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: worker_thread+0x18b/0x244
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ? rescuer_thread+0x281/0x281
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: kthread+0xe4/0xef
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ? kthread_complete_and_exit+0x1b/0x1b
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ret_from_fork+0x1f/0x30
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: </TASK>
    192.168.1.11 :  <4>Jun 24 05:39:35 Tower kernel: ---[ end trace 0000000000000000 ]---

    It happened during the night all of a sudden.

    Edited by ds9
    Link to comment

    may a small update, after 2 month uptime on 6.11.5 on my main Server, changing all dockers to bridge mode and ipvlan and all traces are also gone as expected on 6.12.2, so hardware is fine, its definately a software issue in unraid caming up on ~6.12beta7 ...

     

    changing to bridge btw is really a pain with some services like tvheadend (satip usage), homeassistant (discovery), ... and so on, so not really a final solution ... sure, can also startup with a mixed mode (some in host, some in bridge) but thats also only working until a certain point ... like dlna Services are all using port 1900 as simple sample ...

     

    as note as more and more are upgrading and running into issues and dont even know why their servers are crashing (macvlan) or why their services have connectivity issues (ipvlan on some router hardware) it would be really nice to look futher into it.

     

    may also as note, we have the first issues coming up with unify routers and firewall rules as their also the mac address is the leader and ipvlan not really a solution ...

     

    just as friendly reminder that this is still a issue ... ;) sadly only a few people reporting this issue and returning to 6.11 for now as they think its just a bug which will be fixed soon.

    • Like 1
    • Thanks 1
    Link to comment

    Soo i fell over this forum :)
    I want to point out , that I DONT HAVE UNRAID , but my kernel panic errors directed me to this site.
    Im running OMV / Debian
    And my server started these CPU errors and i found out (at least i think) that this was related to network / docker / bridge
    And right now im on testing phase for possible solution.
    Here's what i did :

    I simply added forward/accept br0 connection:
    iptables -A FORWARD -p all -i br0 -j ACCEPT

    Im not sure if this will help you guys or you already have tried this.
     

    EDIT:
    It seems this didn't work :(
    Im at work right now, and i cannot remote in to my pc.. So its properly in "kernel panic"

    Edited by kekec777
    • Like 1
    • Thanks 2
    Link to comment

    Absolutely beyond me how this remained an issue for so long when macvlan was FINE before.

     

    Having static IPs for dockers is no niche need and getting server lockups as frequently as I got them the past few weeks when 6.9.2 ran like a roadrunner for over a year without downtime is... It's just really weird.

     

    If the issue is higher level I'm sure LimeTech could get in touch with those who are closer to the meat and potatoes of this issue, the OS may be based on FOSS, but in the end it's a paid product advertising to be a cheaper, but more customizable and functional alternative to off-the-shelf NAS systems for tech savvy folks.

     

    The most painful thing is that I had been wondering forever what my troubles could be and what I'm doing wrong. Turns out, I'm not doing anything crazy that shouldn't be possible and just in time I'm joining the ranks of FritzBox users in the coming days. Oh Lordie...

     

    So even if I didn't need static IPs for some dockers, I'd be SOL the moment I switch to my new router.

    • Upvote 1
    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.