nerv Posted March 26, 2023 Share Posted March 26, 2023 Ever since upgrading to 6.11.5 my Unraid system has started crashing after having never previously crashed before. It was every 2 weeks or so, but recently it has increased to every 2-3 days. I've attached diagnostics and my syslog, any ideas on the cause? The most recent crash was sometime on the 25th, and I turned the server back on the morning of the 26th. There's a CPU warning in the logs right before then (see below), and I see something similar a few other places. Is this potentially related? Any help greatly appreciated. Thanks! Mar 25 06:55:24 Media kernel: ------------[ cut here ]------------ Mar 25 06:55:24 Media kernel: WARNING: CPU: 16 PID: 6805 at net/netfilter/nf_nat_core.c:594 nf_nat_setup_info+0x73/0x7b1 [nf_nat] Mar 25 06:55:24 Media kernel: Modules linked in: af_packet tcp_diag udp_diag inet_diag xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat vhost_net tun vhost vhost_iotlb tap veth macvlan xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user iptable_nat nf_nat br_netfilter xfs md_mod it87 hwmon_vid xt_connmark nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_mark iptable_mangle xt_comment xt_addrtype iptable_raw wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge 8021q garp mrp stp llc bonding tls ixgbe xfrm_algo mdio x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm crct10dif_pclmul crc32_pclmul crc32c_intel ast ghash_clmulni_intel drm_vram_helper i2c_algo_bit aesni_intel drm_ttm_helper ttm crypto_simd cryptd drm_kms_helper rapl intel_cstate drm intel_uncore mpt3sas backlight agpgart Mar 25 06:55:24 Media kernel: i2c_i801 i2c_smbus ahci syscopyarea i2c_core sysfillrect raid_class sysimgblt libahci fb_sys_fops scsi_transport_sas wmi acpi_ipmi ipmi_si button unix [last unloaded: xfrm_algo] Mar 25 06:55:24 Media kernel: CPU: 16 PID: 6805 Comm: kworker/16:2 Tainted: G W 5.19.17-Unraid #2 Mar 25 06:55:24 Media kernel: Hardware name: Cirrascale VB1416/GA-7PESH2, BIOS R17 06/26/2018 Mar 25 06:55:24 Media kernel: Workqueue: events macvlan_process_broadcast [macvlan] Mar 25 06:55:24 Media kernel: RIP: 0010:nf_nat_setup_info+0x73/0x7b1 [nf_nat] Mar 25 06:55:24 Media kernel: Code: 48 8b 87 80 00 00 00 48 89 fb 49 89 f4 76 04 0f 0b eb 0e 83 7c 24 1c 00 75 07 25 80 00 00 00 eb 05 25 00 01 00 00 85 c0 74 07 <0f> 0b e9 6a 06 00 00 48 8b 83 88 00 00 00 48 8d 73 58 48 8d 7c 24 Mar 25 06:55:24 Media kernel: RSP: 0018:ffffc90006784bc8 EFLAGS: 00010202 Mar 25 06:55:24 Media kernel: RAX: 0000000000000080 RBX: ffff8882a61a5700 RCX: ffff889099a38400 Mar 25 06:55:24 Media kernel: RDX: 0000000000000000 RSI: ffffc90006784cac RDI: ffff8882a61a5700 Mar 25 06:55:24 Media kernel: RBP: ffffc90006784c90 R08: 000000006756a8c0 R09: 0000000000000000 Mar 25 06:55:24 Media kernel: R10: 0000000000000158 R11: 0000000000000000 R12: ffffc90006784cac Mar 25 06:55:24 Media kernel: R13: 000000006756a800 R14: ffffc90006784d90 R15: 0000000000000000 Mar 25 06:55:24 Media kernel: FS: 0000000000000000(0000) GS:ffff88903fc00000(0000) knlGS:0000000000000000 Mar 25 06:55:24 Media kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Mar 25 06:55:24 Media kernel: CR2: 0000151c1dd49bf8 CR3: 000000000220a006 CR4: 00000000001726e0 Mar 25 06:55:24 Media kernel: Call Trace: Mar 25 06:55:24 Media kernel: <IRQ> Mar 25 06:55:24 Media kernel: ? xt_write_recseq_end+0xf/0x1c [ip_tables] Mar 25 06:55:24 Media kernel: ? __local_bh_enable_ip+0x56/0x6b Mar 25 06:55:24 Media kernel: ? ipt_do_table+0x57a/0x5bf [ip_tables] Mar 25 06:55:24 Media kernel: ? xt_write_recseq_end+0xf/0x1c [ip_tables] Mar 25 06:55:24 Media kernel: __nf_nat_alloc_null_binding+0x66/0x81 [nf_nat] Mar 25 06:55:24 Media kernel: nf_nat_inet_fn+0xc0/0x1a8 [nf_nat] Mar 25 06:55:24 Media kernel: nf_nat_ipv4_local_in+0x2a/0xaa [nf_nat] Mar 25 06:55:24 Media kernel: nf_hook_slow+0x3d/0x96 Mar 25 06:55:24 Media kernel: ? ip_protocol_deliver_rcu+0x164/0x164 Mar 25 06:55:24 Media kernel: NF_HOOK.constprop.0+0x79/0xd9 Mar 25 06:55:24 Media kernel: ? ip_protocol_deliver_rcu+0x164/0x164 Mar 25 06:55:24 Media kernel: ip_sabotage_in+0x4a/0x58 [br_netfilter] Mar 25 06:55:24 Media kernel: nf_hook_slow+0x3d/0x96 Mar 25 06:55:24 Media kernel: ? ip_rcv_finish_core.constprop.0+0x3b7/0x3b7 Mar 25 06:55:24 Media kernel: NF_HOOK.constprop.0+0x79/0xd9 Mar 25 06:55:24 Media kernel: ? ip_rcv_finish_core.constprop.0+0x3b7/0x3b7 Mar 25 06:55:24 Media kernel: __netif_receive_skb_one_core+0x77/0x9c Mar 25 06:55:24 Media kernel: process_backlog+0x8c/0x116 Mar 25 06:55:24 Media kernel: __napi_poll.constprop.0+0x2b/0x124 Mar 25 06:55:24 Media kernel: net_rx_action+0x159/0x24f Mar 25 06:55:24 Media kernel: __do_softirq+0x129/0x288 Mar 25 06:55:24 Media kernel: do_softirq+0x7f/0xab Mar 25 06:55:24 Media kernel: </IRQ> Mar 25 06:55:24 Media kernel: <TASK> Mar 25 06:55:24 Media kernel: __local_bh_enable_ip+0x4c/0x6b Mar 25 06:55:24 Media kernel: netif_rx+0x52/0x5a Mar 25 06:55:24 Media kernel: macvlan_broadcast+0x10a/0x150 [macvlan] Mar 25 06:55:24 Media kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan] Mar 25 06:55:24 Media kernel: process_one_work+0x1ab/0x295 Mar 25 06:55:24 Media kernel: worker_thread+0x18b/0x244 Mar 25 06:55:24 Media kernel: ? rescuer_thread+0x281/0x281 Mar 25 06:55:24 Media kernel: kthread+0xe7/0xef Mar 25 06:55:24 Media kernel: ? kthread_complete_and_exit+0x1b/0x1b Mar 25 06:55:24 Media kernel: ret_from_fork+0x22/0x30 Mar 25 06:55:24 Media kernel: </TASK> Mar 25 06:55:24 Media kernel: ---[ end trace 0000000000000000 ]--- Mar 26 08:10:34 Media kernel: mdcmd (36): set md_write_method 1 peer-Media-wg0-1 (1).zip syslog-192.168.86.42 (1).log Quote Link to comment
Solution JorgeB Posted March 27, 2023 Solution Share Posted March 27, 2023 Macvlan call traces are usually the result of having dockers with a custom IP address and will end up crashing the server, upgrading to v6.10 or later and switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)) Quote Link to comment
nerv Posted March 27, 2023 Author Share Posted March 27, 2023 5 hours ago, JorgeB said: Macvlan call traces are usually the result of having dockers with a custom IP address and will end up crashing the server, upgrading to v6.10 or later and switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)) Got it, I changed that. I'll follow up if that fixes the issue. That's actually the first time I've seen that log fwiw, or at least the first time fix common errors alerted me to it was the other day. I had assumed that was because Mullvad took down my VPN for maintenance and I had to switch. Maybe that's unrelated though. I'm not sure where that setting actually came from. Maybe an older legacy thing as I've had unraid for ~10 years? I do have a fairly unusual setup with dockers with individual dual IPs on a vlan + on a wireguard tunnel, but as far as I can tell I didn't set it that way due to one of those, and everything is still working with the new setting. Doesn't matter, just curious :). Quote Link to comment
macmanluke Posted March 28, 2023 Share Posted March 28, 2023 18 hours ago, JorgeB said: Macvlan call traces are usually the result of having dockers with a custom IP address and will end up crashing the server, upgrading to v6.10 or later and switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)) This is something i just picked up on after fix common problems started alerting about it What has changed to make that an issue? iv had dockers with custom IPs since i started using unraid in 6.8 which was default to macvlan all the way up to this week when fix common problems warned me (and iv been having crashing issues lately but previously months of stability etc) Quote Link to comment
JorgeB Posted March 28, 2023 Share Posted March 28, 2023 It's common that this issue can appear after an OS/kernel update. Quote Link to comment
nerv Posted April 1, 2023 Author Share Posted April 1, 2023 So far so good, so I'm going to mark this solved and cross my fingers. Will reopen if I get a new crash + logs. Thanks @JorgeB! 1 Quote Link to comment
nerv Posted April 23, 2023 Author Share Posted April 23, 2023 (edited) No crashes still for a month+ so that definitely seems to have fixed it. Weirdly though I just got an alert from fix common errors that it found Macvlan call traces. I confirmed I still have docker set to ipvlan. Should I ignore this? I also have some VMs running. Edited April 23, 2023 by nerv Quote Link to comment
JorgeB Posted April 23, 2023 Share Posted April 23, 2023 38 minutes ago, nerv said: I confirmed I still have docker set to ipvlan. Should I ignore this? Don't understand, I suggest changing that above and assumed you did. Quote Link to comment
nerv Posted April 23, 2023 Author Share Posted April 23, 2023 2 hours ago, JorgeB said: Don't understand, I suggest changing that above and assumed you did. Yea I did change the setting and the crashes stopped, but I got the warning again. Not sure why I'm getting the warning again. Quote Link to comment
JorgeB Posted April 24, 2023 Share Posted April 24, 2023 Did you reboot? FCP will show old warnings until you do. Quote Link to comment
nerv Posted April 24, 2023 Author Share Posted April 24, 2023 Hmm, no that's probably it. The warning just showed up for the first time after a month which is odd, but that makes the most sense. Thanks! Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.