Hi there, folks –

 

I've had some crashes recently that have been happening with increasing frequency. I thought it might have been a temperature problem (the room my server is in gets very hot in the summer), but I added more cooling and got another crash this afternoon 80% of the way into a 20 hour parity sync (sigh).

 

I have my diagnostic file and my logs (attached), but I'm finding the log a bit tough to interpret; I'm hoping someone with more specific knowledge might be able to give me a better idea of what the log might be suggesting is the problem. My layman interpretation is it's the CPU experiencing hardware failure and I may need to replace it, but that's almost a guess based on skimming the log.

 

Appreciate any help. Let me know if you need more background.

 

Excerpt right before the crash from the log – more in the attached file:

 

Spoiler

Aug 19 14:56:06 Tower kernel: Call Trace:
Aug 19 14:56:06 Tower kernel: <IRQ>
Aug 19 14:56:06 Tower kernel: dump_stack_lvl+0x46/0x5a
Aug 19 14:56:06 Tower kernel: nmi_cpu_backtrace+0xae/0xd2
Aug 19 14:56:06 Tower kernel: ? lapic_can_unplug_cpu+0x93/0x93
Aug 19 14:56:06 Tower kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
Aug 19 14:56:06 Tower kernel: rcu_dump_cpu_stacks+0xc3/0xea
Aug 19 14:56:06 Tower kernel: rcu_sched_clock_irq+0x22d/0x631
Aug 19 14:56:06 Tower kernel: ? _raw_spin_unlock_irqrestore+0xe/0x1b
Aug 19 14:56:06 Tower kernel: ? tick_sched_do_timer+0x3e/0x3e
Aug 19 14:56:06 Tower kernel: update_process_times+0x8c/0xab
Aug 19 14:56:06 Tower kernel: tick_sched_timer+0x38/0x65
Aug 19 14:56:06 Tower kernel: __hrtimer_run_queues+0xf8/0x18a
Aug 19 14:56:06 Tower kernel: hrtimer_interrupt+0x92/0x160
Aug 19 14:56:06 Tower kernel: __sysvec_apic_timer_interrupt+0x96/0xdb
Aug 19 14:56:06 Tower kernel: sysvec_apic_timer_interrupt+0x37/0x7d
Aug 19 14:56:06 Tower kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Aug 19 14:56:06 Tower kernel: RIP: 0010:nf_ct_tuple_equal+0x3/0x47 [nf_conntrack]
Aug 19 14:56:06 Tower kernel: Code: 80 e8 08 00 00 48 85 c0 74 1c 53 89 fb 48 89 f7 be 04 00 00 00 e8 8a ff ff ff 48 85 c0 74 05 f0 48 0f ab 18 5b c3 c3 48 8b 17 <31> c0 48 33 16 48 8b 4f 08 48 33 4e 08 48 09 ca 75 2e 8b 4e 10 39
Aug 19 14:56:06 Tower kernel: RSP: 0018:ffffc900001c88d8 EFLAGS: 00000246
Aug 19 14:56:06 Tower kernel: RAX: 00000000034d257e RBX: ffff8881c6e3f7c0 RCX: ffffffff828e1500
Aug 19 14:56:06 Tower kernel: RDX: 00000000fb0000e0 RSI: ffff8881c6e3f6d8 RDI: ffffc900001c8970
Aug 19 14:56:06 Tower kernel: RBP: ffff8881c6e3f6c8 R08: 0000000000000003 R09: 7d4f2cc31c5d048f
Aug 19 14:56:06 Tower kernel: R10: 8247bea796618e2e R11: b8e78e770e080bf8 R12: 0000000000000000
Aug 19 14:56:06 Tower kernel: R13: ffff8881c6e3f7cc R14: ffffffff828e1500 R15: ffff8881c6e3f680
Aug 19 14:56:06 Tower kernel: nf_ct_key_equal+0x20/0x70 [nf_conntrack]
Aug 19 14:56:06 Tower kernel: nf_conntrack_tuple_taken+0xc9/0x156 [nf_conntrack]
Aug 19 14:56:06 Tower kernel: nf_nat_used_tuple+0x33/0x4e [nf_nat]
Aug 19 14:56:06 Tower kernel: nf_nat_setup_info+0x3f5/0x773 [nf_nat]
Aug 19 14:56:06 Tower kernel: ? addrtype_mt_v1+0xc2/0x100 [xt_addrtype]
Aug 19 14:56:06 Tower kernel: __nf_nat_alloc_null_binding+0x68/0x7f [nf_nat]
Aug 19 14:56:06 Tower kernel: nf_nat_inet_fn+0xa6/0x187 [nf_nat]
Aug 19 14:56:06 Tower kernel: nf_nat_ipv4_pre_routing+0x23/0x4e [nf_nat]
Aug 19 14:56:06 Tower kernel: nf_hook_slow+0x3b/0x93
Aug 19 14:56:06 Tower kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter]
Aug 19 14:56:06 Tower kernel: NF_HOOK.isra.0+0xbd/0x119 [br_netfilter]
Aug 19 14:56:06 Tower kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter]
Aug 19 14:56:06 Tower kernel: br_nf_pre_routing+0x226/0x236 [br_netfilter]
Aug 19 14:56:06 Tower kernel: ? br_nf_forward_finish+0xdc/0xdc [br_netfilter]
Aug 19 14:56:06 Tower kernel: br_handle_frame+0x28f/0x2da
Aug 19 14:56:06 Tower kernel: ? tcp_v4_destroy_sock+0x50/0x18c
Aug 19 14:56:06 Tower kernel: ? br_handle_local_finish+0x13/0x13
Aug 19 14:56:06 Tower kernel: __netif_receive_skb_core.constprop.0+0x4b3/0x685
Aug 19 14:56:06 Tower kernel: __netif_receive_skb_list_core+0x81/0x112
Aug 19 14:56:06 Tower kernel: netif_receive_skb_list_internal+0x1c7/0x1fc
Aug 19 14:56:06 Tower kernel: gro_normal_list+0x22/0x3e
Aug 19 14:56:06 Tower kernel: napi_complete_done+0x7f/0x118
Aug 19 14:56:06 Tower kernel: igb_poll+0xd69/0xf96 [igb]
Aug 19 14:56:06 Tower kernel: ? tick_sched_do_timer+0x3e/0x3e
Aug 19 14:56:06 Tower kernel: __napi_poll.constprop.0+0x27/0x114
Aug 19 14:56:06 Tower kernel: net_rx_action+0xe8/0x1f2
Aug 19 14:56:06 Tower kernel: __do_softirq+0xec/0x218
Aug 19 14:56:06 Tower kernel: __irq_exit_rcu+0x4d/0x88
Aug 19 14:56:06 Tower kernel: common_interrupt+0x73/0x8b
Aug 19 14:56:06 Tower kernel: </IRQ>
Aug 19 14:56:06 Tower kernel: <TASK>
Aug 19 14:56:06 Tower kernel: asm_common_interrupt+0x1e/0x40
Aug 19 14:56:06 Tower kernel: RIP: 0010:arch_local_irq_enable+0x7/0x8
Aug 19 14:56:06 Tower kernel: Code: a2 bc 1c 00 85 db 48 89 e8 79 03 48 63 c3 5b 5d 41 5c c3 9c 58 0f 1f 44 00 00 c3 fa 66 0f 1f 44 00 00 c3 fb 66 0f 1f 44 00 00 <c3> 0f 1f 44 00 00 55 49 89 d3 48 81 c7 b0 00 00 00 48 83 c6 70 53
Aug 19 14:56:06 Tower kernel: RSP: 0018:ffffc90000107ea0 EFLAGS: 00000246
Aug 19 14:56:06 Tower kernel: RAX: ffff88884bd2bb40 RBX: 0000000000000006 RCX: 000000000000001f
Aug 19 14:56:06 Tower kernel: RDX: 0000000000000004 RSI: 0000000000000004 RDI: 0000000000000000
Aug 19 14:56:06 Tower kernel: RBP: ffff88884bd34c00 R08: 00000000ffffffff R09: 071c71c71c71c71c
Aug 19 14:56:06 Tower kernel: R10: 0000000000000020 R11: 00000000000008cf R12: ffffffff82311d20
Aug 19 14:56:06 Tower kernel: R13: 0000000000000006 R14: 0000347a39b12379 R15: 0000000000000000
Aug 19 14:56:06 Tower kernel: cpuidle_enter_state+0x117/0x1db
Aug 19 14:56:06 Tower kernel: cpuidle_enter+0x2a/0x36
Aug 19 14:56:06 Tower kernel: do_idle+0x1b7/0x225
Aug 19 14:56:06 Tower kernel: cpu_startup_entry+0x1d/0x1f
Aug 19 14:56:06 Tower kernel: secondary_startup_64_no_verify+0xb0/0xbb
Aug 19 14:56:06 Tower kernel: </TASK>
Aug 19 14:56:42 Tower kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 1-... 4-... } 2046033 jiffies s: 10229 root: 0x12/.
Aug 19 14:56:42 Tower kernel: rcu: blocking rcu_node structures (internal RCU debug):
Aug 19 14:56:42 Tower kernel: Task dump for CPU 1:
Aug 19 14:56:42 Tower kernel: task:kworker/u12:7   state:R  running task     stack:    0 pid:  533 ppid:     2 flags:0x00004008
Aug 19 14:56:42 Tower kernel: Workqueue: events_power_efficient gc_worker [nf_conntrack]
Aug 19 14:56:42 Tower kernel: Call Trace:
Aug 19 14:56:42 Tower kernel: <TASK>
Aug 19 14:56:42 Tower kernel: ? gc_worker+0xb2/0x30b [nf_conntrack]
Aug 19 14:56:42 Tower kernel: ? process_one_work+0x195/0x27a
Aug 19 14:56:42 Tower kernel: ? worker_thread+0x19c/0x240
Aug 19 14:56:42 Tower kernel: ? rescuer_thread+0x28b/0x28b
Aug 19 14:56:42 Tower kernel: ? kthread+0xdc/0xe3
Aug 19 14:56:42 Tower kernel: ? set_kthread_struct+0x32/0x32
Aug 19 14:56:42 Tower kernel: ? ret_from_fork+0x1f/0x30
Aug 19 14:56:42 Tower kernel: </TASK>
Aug 19 14:56:42 Tower kernel: Task dump for CPU 4:
Aug 19 14:56:42 Tower kernel: task:swapper/4       state:R  running task     stack:    0 pid:    0 ppid:     1 flags:0x00004008
Aug 19 14:56:42 Tower kernel: Call Trace:
Aug 19 14:56:42 Tower kernel: <TASK>
Aug 19 14:56:42 Tower kernel: ? arch_local_irq_enable+0x7/0x8
Aug 19 14:56:42 Tower kernel: ? cpuidle_enter_state+0x117/0x1db
Aug 19 14:56:42 Tower kernel: ? cpuidle_enter+0x2a/0x36
Aug 19 14:56:42 Tower kernel: ? do_idle+0x1b7/0x225
Aug 19 14:56:42 Tower kernel: ? cpu_startup_entry+0x1d/0x1f
Aug 19 14:56:42 Tower kernel: ? secondary_startup_64_no_verify+0xb0/0xbb
Aug 19 14:56:42 Tower kernel: </TASK>

 

syslog.log tower-diagnostics-20220819-1506.zip

Edited August 20, 20223 yr by Refrigerator
My excerpt of log there in the spoiler tag didn't show.

Try this,  switch to ipvlan (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

Thanks for the suggestion, @JorgeB – I was able to complete the parity sync and switch to ipvlan. So far, the only issue since the change is that Wireguard isn't working properly, which is odd. Anyhow, I'll update the thread in a while to say for sure if this ended up preventing the problem after several days of uptime.

Edited August 22, 20223 yr by Refrigerator
Detail correction.