Random Hard Crash Lockups

[Jaybau]

Every few days Unraid is hard crashing and locks-up requiring a hard reset.

Display monitor is dead (can't see dumps), keyboard is dead, machine is totally unresponsive.

 

It looks to me something happened between 12/16/2023 7:58 AM thru 8:30 AM in the log.

 

I wasn't paying 100% attention when this was happening, but here are some suspicions:

• Docker containers causing instability to the host.  I'm very suspicious a docker container (or something related) can take down a host.
• VNC'd into a docker container.
• Docker container network traffic (I run a lot of network traffic through multiple containers).
• Docker container + network traffic + macvlan = crashing host?

 

Dec 16 08:30:39 Tower kernel: Call Trace:
Dec 16 08:30:39 Tower kernel: <IRQ>
Dec 16 08:30:39 Tower kernel: ? __warn+0xab/0x122
Dec 16 08:30:39 Tower kernel: ? report_bug+0x109/0x17e
Dec 16 08:30:39 Tower kernel: ? nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Dec 16 08:30:39 Tower kernel: ? handle_bug+0x41/0x6f
Dec 16 08:30:39 Tower kernel: ? exc_invalid_op+0x13/0x60
Dec 16 08:30:39 Tower kernel: ? asm_exc_invalid_op+0x16/0x20
Dec 16 08:30:39 Tower kernel: ? nf_nat_setup_info+0x8c/0x7d1 [nf_nat]
Dec 16 08:30:39 Tower kernel: ? nf_nat_setup_info+0x44/0x7d1 [nf_nat]
Dec 16 08:30:39 Tower kernel: ? xt_write_recseq_end+0xf/0x1c [ip_tables]
Dec 16 08:30:39 Tower kernel: ? __local_bh_enable_ip+0x56/0x6b
Dec 16 08:30:39 Tower kernel: ? ipt_do_table+0x575/0x5ba [ip_tables]
Dec 16 08:30:39 Tower kernel: ? xt_write_recseq_end+0xf/0x1c [ip_tables]
Dec 16 08:30:39 Tower kernel: __nf_nat_alloc_null_binding+0x66/0x81 [nf_nat]
Dec 16 08:30:39 Tower kernel: nf_nat_inet_fn+0xc0/0x1a8 [nf_nat]
Dec 16 08:30:39 Tower kernel: nf_nat_ipv4_local_in+0x2a/0xaa [nf_nat]
Dec 16 08:30:39 Tower kernel: nf_hook_slow+0x3d/0x96
Dec 16 08:30:39 Tower kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Dec 16 08:30:39 Tower kernel: NF_HOOK.constprop.0+0x79/0xd9
Dec 16 08:30:39 Tower kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Dec 16 08:30:39 Tower kernel: ip_sabotage_in+0x52/0x60 [br_netfilter]
Dec 16 08:30:39 Tower kernel: nf_hook_slow+0x3d/0x96
Dec 16 08:30:39 Tower kernel: ? ip_rcv_finish_core.constprop.0+0x3e8/0x3e8
Dec 16 08:30:39 Tower kernel: NF_HOOK.constprop.0+0x79/0xd9
Dec 16 08:30:39 Tower kernel: ? ip_rcv_finish_core.constprop.0+0x3e8/0x3e8
Dec 16 08:30:39 Tower kernel: __netif_receive_skb_one_core+0x77/0x9c
Dec 16 08:30:39 Tower kernel: process_backlog+0x8c/0x116
Dec 16 08:30:39 Tower kernel: __napi_poll.constprop.0+0x2b/0x124
Dec 16 08:30:39 Tower kernel: net_rx_action+0x159/0x24f
Dec 16 08:30:39 Tower kernel: __do_softirq+0x129/0x288
Dec 16 08:30:39 Tower kernel: do_softirq+0x7f/0xab
Dec 16 08:30:39 Tower kernel: </IRQ>
Dec 16 08:30:39 Tower kernel: <TASK>
Dec 16 08:30:39 Tower kernel: __local_bh_enable_ip+0x4c/0x6b
Dec 16 08:30:39 Tower kernel: netif_rx+0x52/0x5a
Dec 16 08:30:39 Tower kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Dec 16 08:30:39 Tower kernel: ? _raw_spin_unlock+0x14/0x29
Dec 16 08:30:39 Tower kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]
Dec 16 08:30:39 Tower kernel: process_one_work+0x1ab/0x295
Dec 16 08:30:39 Tower kernel: worker_thread+0x18b/0x244
Dec 16 08:30:39 Tower kernel: ? rescuer_thread+0x281/0x281
Dec 16 08:30:39 Tower kernel: kthread+0xe7/0xef
Dec 16 08:30:39 Tower kernel: ? kthread_complete_and_exit+0x1b/0x1b
Dec 16 08:30:39 Tower kernel: ret_from_fork+0x22/0x30
Dec 16 08:30:39 Tower kernel: </TASK>
Dec 16 08:30:39 Tower kernel: ---[ end trace 0000000000000000 ]---

 

Thank you

 

 

tower-diagnostics-20231216-0859.zip

[itimpi]

Macvlan crashes are known to eventually crash the server on the 6.12.x Unraid releases.   Ideally you should switch to using ipvlan, or if you need to continue using macvlan make sure you follow the instructions in the 6.12.5 release notes.

[Jaybau]

7 minutes ago, itimpi said:

Macvlan crashes are known to eventually crash the server on the 6.12.x Unraid releases.   Ideally you should switch to using ipvlan, or if you need to continue using macvlan make sure you follow the instructions in the 6.12.5 release notes.

 

I was hoping the macvlan issue was fixed:

https://docs.unraid.net/unraid-os/release-notes/6.12.4/#fix-for-macvlan-call-traces

Quote

The big news in this release is that we have resolved issues related to macvlan call traces and crashes!

 

The reason why I am using macvlan is so my router can assign IP addresses based on MAC address and use DHCP.  Otherwise if I use ipvlan, I have to hardcode the IP addresses, and make sure I don't assign an already used IP.

 

But if macvlan is truly still unstable, I can manage ipvlan.

 

Thank you.

 

[JorgeB]

15 hours ago, Jaybau said:

I was hoping the macvlan issue was fixed:

It is, but to use it you need to disable bridging for eth0, see the release notes.

[Jaybau]

7 hours ago, JorgeB said:

need to disable bridging for eth0

 

Thanks. I updated the Network Settings below: