[6.11+] Server getting unresponsive

[ivangoetelek]

Hi All,

about 3 Months back or so, I tried updating to 6.11 - my machine became unstable as many others are describing it here.

It works all fine after a clean startup, and after a finite amount of time (usually between 0.5h to 3h) it is just gone.

 

I enabled syslog back then, but didn't feel like doing anything about it, so to regain stability I downgraded to 6.10.3

My machine was up again for 3 Months then (stable - no issues).

 

Today I decided to update yet again - this time to 6.12.4

Same story: The update and the machine were all good and fine - and after an uptime of about 1.5h it was gone again.

Thankfully I knew what I needed to do: I downgraded to 6.10.3 again.

 

I have the syslog and the diagnostics attached.

 

From what I was able to gather from it, I think it might have to do with the following segment:

Quote

Nov  6 17:23:00 Mini kernel: RSP: 0018:ffffc900002b8d98 EFLAGS: 00010202
Nov  6 17:23:00 Mini kernel: RAX: 0000000000000001 RBX: ffff88815b484000 RCX: 17752a5a0d688103
Nov  6 17:23:00 Mini kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88815b484000
Nov  6 17:23:00 Mini kernel: RBP: 0000000000000001 R08: 167ef409c21de0dc R09: 2141924b61fcc642
Nov  6 17:23:00 Mini kernel: R10: da9aaa77ba57ef47 R11: ffffc900002b8d60 R12: ffffffff82a11d00
Nov  6 17:23:00 Mini kernel: R13: 00000000000170d7 R14: ffff888100f7bd00 R15: 0000000000000000
Nov  6 17:23:00 Mini kernel: FS:  0000000000000000(0000) GS:ffff888390680000(0000) knlGS:0000000000000000
Nov  6 17:23:00 Mini kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov  6 17:23:00 Mini kernel: CR2: 0000151c31416f20 CR3: 00000001315ea000 CR4: 00000000003506e0
Nov  6 17:23:00 Mini kernel: Call Trace:
Nov  6 17:23:00 Mini kernel: <IRQ>
Nov  6 17:23:00 Mini kernel: ? __warn+0xab/0x122
Nov  6 17:23:00 Mini kernel: ? report_bug+0x109/0x17e
Nov  6 17:23:00 Mini kernel: ? __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
Nov  6 17:23:00 Mini kernel: ? handle_bug+0x41/0x6f
Nov  6 17:23:00 Mini kernel: ? exc_invalid_op+0x13/0x60
Nov  6 17:23:00 Mini kernel: ? asm_exc_invalid_op+0x16/0x20
Nov  6 17:23:00 Mini kernel: ? __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
Nov  6 17:23:00 Mini kernel: ? __nf_conntrack_confirm+0x9e/0x2b0 [nf_conntrack]
Nov  6 17:23:00 Mini kernel: ? nf_nat_inet_fn+0x60/0x1a8 [nf_nat]
Nov  6 17:23:00 Mini kernel: nf_conntrack_confirm+0x25/0x54 [nf_conntrack]
Nov  6 17:23:00 Mini kernel: nf_hook_slow+0x3d/0x96
Nov  6 17:23:00 Mini kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Nov  6 17:23:00 Mini kernel: NF_HOOK.constprop.0+0x79/0xd9
Nov  6 17:23:00 Mini kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Nov  6 17:23:00 Mini kernel: __netif_receive_skb_one_core+0x77/0x9c
Nov  6 17:23:00 Mini kernel: process_backlog+0x8c/0x116
Nov  6 17:23:00 Mini kernel: __napi_poll.constprop.0+0x2b/0x124
Nov  6 17:23:00 Mini kernel: net_rx_action+0x159/0x24f
Nov  6 17:23:00 Mini kernel: __do_softirq+0x129/0x288
Nov  6 17:23:00 Mini kernel: do_softirq+0x7f/0xab
Nov  6 17:23:00 Mini kernel: </IRQ>
Nov  6 17:23:00 Mini kernel: <TASK>
Nov  6 17:23:00 Mini kernel: __local_bh_enable_ip+0x4c/0x6b
Nov  6 17:23:00 Mini kernel: netif_rx+0x52/0x5a
Nov  6 17:23:00 Mini kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Nov  6 17:23:00 Mini kernel: ? _raw_spin_unlock+0x14/0x29
Nov  6 17:23:00 Mini kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]
Nov  6 17:23:00 Mini kernel: process_one_work+0x1ab/0x295
Nov  6 17:23:00 Mini kernel: worker_thread+0x18b/0x244
Nov  6 17:23:00 Mini kernel: ? rescuer_thread+0x281/0x281
Nov  6 17:23:00 Mini kernel: kthread+0xe7/0xef
Nov  6 17:23:00 Mini kernel: ? kthread_complete_and_exit+0x1b/0x1b
Nov  6 17:23:00 Mini kernel: ret_from_fork+0x22/0x30
Nov  6 17:23:00 Mini kernel: </TASK>
Nov  6 17:23:00 Mini kernel: ---[ end trace 0000000000000000 ]---

Sadly I have no Idea what any of that means, but it feels like an issue with the Docker br0/macvlan network

 

Any help is highly appreciated!

Thanks for your support!

syslog-127.0.0.1.log mini-diagnostics-20231106-1724.zip

[JorgeB]

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).
 

[ivangoetelek]

Alright - I switched to ipvlan - I don't quite understand the difference.
Will it come with any sideeffects (besides not crashing )?

Do you know why the macvlan is standard instead of the ipvlan, if one is more stable than the other?

 

Will update to 6.12.4 then and will let you know tomorrow if it stayed intact or not.

 

Thanks so far

[JorgeB]

34 minutes ago, ivangoetelek said:

Will it come with any sideeffects (besides not crashing )?

For most users it's the same, if you notice any difference you can still use macvlan with v6.12.4, see the release notes.

[ivangoetelek]

@JorgeB you wouldn't by any chance know how to provoke the initial error?

I would like to test the setup now after the update

 

But the speed of your answer leads me to believe that you know what you're talking about - is this a known issue?

[JorgeB]

13 minutes ago, ivangoetelek said:

you wouldn't by any chance know how to provoke the initial error?

Do you mean the macvlan errors?

[ivangoetelek]

Yeah - how would I test if your solution is correct (not that I distrust it, but i consider it best practice)

[ivangoetelek]

So - so far no issues - seems to have worked!

[JorgeB]

13 hours ago, ivangoetelek said:

Yeah - how would I test if your solution is correct (not that I distrust it, but i consider it best practice)

You'd change to macvlan again and see if it kept crashing.