Kernel Panic Crash Every Few Weeks

[sirelewop14]

Greetings unRAID users!
 

I have been an unraid user for about 6 or 7 years now. Have wrangled a few issues on and off but of course I have been very pleased with the stability of unRAID (until lately)

 

Over the past 6 months or so, I have been having issues with my unRAID server on and off. At first, the issue was bad RAM - server would crash and the IPMI would show that a DIMM had uncorrectable errors, etc.

 

So I purchased 128GB of new DDR3 ECC RAM and installed, this seemed to resolve the RAM issue.

Then, I had problems where the web interface would become unresponsive and sometimes the whole server would crash with a kernel panic. So I purchased a new USB drive and reinstalled unRAID from scratch. I reconfigured my docker containers, stripped down the plugins installed, and ensured I was running the latest stable version. (which I was running before the USB replacement) I also removed a 10Gb network card which I thought may have been causing issues.

 

Now, my web GUI issues appear to be resolved, but I am still getting kernel panics every so often. It's been a bit frustrating as I am having trouble nailing down the source. I enabled logging to disk on the unRAID server but my logs don't seem to indicate an issue before the kernel panic. The stack trace displayed on the console also doesn't seem to help - usually Kernel Panic - not syncing.

 

So, I am turning to the masters for assistance - here is the relevant information as well as log and system-diagnostic bundles.

 

The server just crashed this morning so I have fresh DIAG and logs.

 

Model: Custom
M/B: Supermicro X9DR3-F Version 0123456789 - s/n: 0123456789
BIOS: American Megatrends Inc. Version 3.3. Dated: 07/12/2018
CPU: Intel® Xeon® CPU E5-2630L 0 @ 2.00GHz
HVM: Enabled
IOMMU: Enabled
Cache: 384 KiB, 1536 KiB, 15 MB, 384 KiB, 1536 KiB, 15 MB
Memory: 128 GiB DDR3 Multi-bit ECC (max. installable capacity 192 GiB)
Network: bond0: fault-tolerance (active-backup), mtu 1500
 eth0: 1000 Mbps, full duplex, mtu 1500
 eth1: 1000 Mbps, full duplex, mtu 1500
Kernel: Linux 5.10.28-Unraid x86_64
OpenSSL: 1.1.1j
Uptime: 0 days, 00:07:11

 

I have attached Diag and logs from today, as well as some I collected in December, just before christmas - the last time the server crashed.

Reginald_Diagnostics_And_Logs.zip

[sirelewop14]

I posted this thread to Reddit as well an shave received a few responses indicating the issue may be custom docker networks.

 

I was using this feature to assign my nginx reverse proxy a static IP separate from my unRAID box but several users have indicated this was the source of kernel panics.

 

I'm going to try to remove that config and wait and see.

 

If anyone has any suggestions in the meantime I'm all ears!

 

Thanks!

[JorgeB]

Dec  5 01:37:50 Reginald kernel: macvlan_broadcast+0x10e/0x13c [macvlan]
Dec  5 01:37:50 Reginald kernel: macvlan_process_broadcast+0xf8/0x143 [macvlan]

 

8 hours ago, sirelewop14 said:

indicating the issue may be custom docker networks.

Agree, macvlan call traces are usually the result of having dockers with a custom IP address, upgrading to v6.10 and switching to ipvlan might fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enable, top right)), or see below for more info.

https://forums.unraid.net/topic/70529-650-call-traces-when-assigning-ip-address-to-docker-containers/

See also here:

https://forums.unraid.net/bug-reports/stable-releases/690691-kernel-panic-due-to-netfilter-nf_nat_setup_info-docker-static-ip-macvlan-r1356/

[Prbecker]

I tried googling around for a release date for v6.10 (unraid shows the "next" branch as 6.10.0-rc2 and "stable" as 6.9.2 which is what i'm currently on) as I'm facing this issue too but didn't have the option to change the setting to ipvlan. Is there an expected release date possibly?

[itimpi]

38 minutes ago, Prbecker said:

I tried googling around for a release date for v6.10 (unraid shows the "next" branch as 6.10.0-rc2 and "stable" as 6.9.2 which is what i'm currently on) as I'm facing this issue too but didn't have the option to change the setting to ipvlan. Is there an expected release date possibly?

Limetech never gives predicted release dates