Repeated kernel panics during high-CPU usage states

[volcs0]

After years and years of stability, I'm now seeing occasional kernel panics during periods of high CPU usage.  This is happening more frequently - I've had three this week. I have to physically turn the machine off and on.

 

I'd like to troubleshoot this.

 

I turned on Syslog Server with "local syslog" to a cache-only share.

Is this the preferred way to troubleshoot this?

 

In case it helps, I've attached a screenshot of the kernel panic, and I've attached my diagnostics.

 

Thanks.

tower-diagnostics-20230828-0750.zip

Edited August 28, 2023 by volcs0

[JorgeB]

Based on the screenshot if may be the macvlan issue, but post the persistent syslog when you have it to confirm. If it's the macvlan issue switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

[volcs0]

23 hours ago, JorgeB said:

Based on the screenshot if may be the macvlan issue, but post the persistent syslog when you have it to confirm. If it's the macvlan issue switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

 

I turned on syslog server (local) with /mnt/user/syslog as the share (cache only).

 

That was two days ago, and nothing has been written to that folder.

 

Are there supposed to be logs written periodically to the folder?

 

How does that work for a kernel panic - when would something get written to syslog?

 

I checked and group/user is correct (nobody/users) and permissions are 777 for the share.

 

Thanks for the advice.

 

[JorgeB]

Post a screenshot of the syslog server settings, but most likely it's not correctly configured, you may want to read the instructions again.

[volcs0]

1 hour ago, JorgeB said:

Post a screenshot of the syslog server settings, but most likely it's not correctly configured, you may want to read the instructions again.

 

Here is what I have.

 

Thank you.

 

Edit - Does not appear to be listening on port 514.. Is that the problem?

 

 

Edited August 29, 2023 by volcs0

[itimpi]

You need to set the Remote syslog server to have the IP address of your Unraid server OR set the mirror to flash option.

[volcs0]

7 minutes ago, itimpi said:

You need to set the Remote syslog server to have the IP address of your Unraid server OR set the mirror to flash option.

 

That did it. Thanks.

[volcs0]

On 8/28/2023 at 9:05 AM, JorgeB said:

Based on the screenshot if may be the macvlan issue, but post the persistent syslog when you have it to confirm. If it's the macvlan issue switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

 

I didn't make any changes yet based on your suggestion above.

It happened again yesterday - around 16:30.

Here are the last 1000 lines of the syslog.

 

 

Thank you for your insights.

 

[JorgeB]

55 minutes ago, volcs0 said:

Based on the screenshot if may be the macvlan issue,

It is:
 

Aug 31 09:18:58 Tower kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Aug 31 09:18:58 Tower kernel: ? _raw_spin_unlock+0x14/0x29
Aug 31 09:18:58 Tower kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

 

[volcs0]

7 minutes ago, JorgeB said:

It is:
 

Aug 31 09:18:58 Tower kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Aug 31 09:18:58 Tower kernel: ? _raw_spin_unlock+0x14/0x29
Aug 31 09:18:58 Tower kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

 

Excellent. Thank you for taking a look. I will make the change you suggested.

 

Also, are all of those post disabled / blocked lines concerning? 

 

Thanks again for your help.

 

[JorgeB]

6 minutes ago, volcs0 said:

Also, are all of those post disabled / blocked lines concerning? 

Nope

[volcs0]

On 9/1/2023 at 7:36 AM, JorgeB said:

Nope

Sorry to pile on another question -

Since changing from macvlan to ipvlan, my cloudflare tunnel to my apps no longer works - getting DNS errors / SRV lookup errors.

If I change back to macvlan, it works again. 

I'm reading about this and trying to troubleshoot and learn.

Any thoughts about where to start with this? 

Thanks

 

[JorgeB]

Update to v6.12.4, there's a new option to still use macvlan without causing the call traces, read the release notes for more info.