Kernel panics


subagon

Recommended Posts

No crash to report since the last post. Since then I have updated to 6.6.2 and kept an open terminal tailing /var/log/syslog.

 

I just saw this from 30 minutes ago. It did not cause a kernel panic/crash but looked like it might provide a clue.

 

Oct 25 13:59:54 asok kernel: WARNING: CPU: 15 PID: 20898 at net/netfilter/nf_conntrack_core.c:763 __nf_conntrack_confirm+0x96/0x4fc
Oct 25 13:59:54 asok kernel: Modules linked in: macvlan xt_nat veth xt_CHECKSUM iptable_mangle ipt_REJECT ebtable_filter ebtables ip6table_filter ip6_tables vhost_net tun vhost tap ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs md_mod nct7904 igb i2c_algo_bit sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd ipmi_ssif mpt3sas glue_helper i2c_i801 i2c_core intel_cstate intel_uncore ahci intel_rapl_perf libahci raid_class scsi_transport_sas pcc_cpufreq wmi ipmi_si button [last unloaded: i2c_algo_bit]
Oct 25 13:59:54 asok kernel: CPU: 15 PID: 20898 Comm: kworker/15:1 Tainted: G        W         4.18.14-unRAID #1
Oct 25 13:59:54 asok kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0  07/09/2013
Oct 25 13:59:54 asok kernel: Workqueue: events macvlan_process_broadcast [macvlan]
Oct 25 13:59:54 asok kernel: RIP: 0010:__nf_conntrack_confirm+0x96/0x4fc
Oct 25 13:59:54 asok kernel: Code: c1 ed 20 89 2c 24 e8 26 f7 ff ff 8b 54 24 04 89 ef 89 c6 41 89 c5 e8 bc f8 ff ff 84 c0 75 b9 49 8b 86 80 00 00 00 a8 08 74 02 <0f> 0b 4c 89 f7 e8 04 ff ff ff 49 8b 86 80 00 00 00 0f ba e0 09 73
Oct 25 13:59:54 asok kernel: RSP: 0018:ffff88085f443d30 EFLAGS: 00010202
Oct 25 13:59:54 asok kernel: RAX: 0000000000000188 RBX: ffff8801986ede00 RCX: 0000000000000101
Oct 25 13:59:54 asok kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffff81e09070
Oct 25 13:59:54 asok kernel: RBP: 000000000000b1af R08: 00000000c8ff4e01 R09: 0000000000000000
Oct 25 13:59:54 asok kernel: R10: 0000000000000098 R11: 0000000000000001 R12: ffffffff81e8cc80
Oct 25 13:59:54 asok kernel: R13: 000000000000729c R14: ffff8805f75e9a40 R15: ffff8805f75e9a98
Oct 25 13:59:54 asok kernel: FS:  0000000000000000(0000) GS:ffff88085f440000(0000) knlGS:0000000000000000
Oct 25 13:59:54 asok kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 25 13:59:54 asok kernel: CR2: 0000153b5c810000 CR3: 0000000001e0a003 CR4: 00000000001606e0
Oct 25 13:59:54 asok kernel: Call Trace:
Oct 25 13:59:54 asok kernel: <IRQ>
Oct 25 13:59:54 asok kernel: ipv4_confirm+0xaf/0xb7 [nf_conntrack_ipv4]
Oct 25 13:59:54 asok kernel: nf_hook_slow+0x37/0x96
Oct 25 13:59:54 asok kernel: ip_local_deliver+0xa7/0xd5
Oct 25 13:59:54 asok kernel: ? inet_del_offload+0x3e/0x3e
Oct 25 13:59:54 asok kernel: ip_rcv+0x2dc/0x317
Oct 25 13:59:54 asok kernel: ? ip_local_deliver_finish+0x1aa/0x1aa
Oct 25 13:59:54 asok kernel: __netif_receive_skb_core+0x6b2/0x740
Oct 25 13:59:54 asok kernel: ? kmem_cache_free_bulk+0x25b/0x273
Oct 25 13:59:54 asok kernel: process_backlog+0x7e/0x116
Oct 25 13:59:54 asok kernel: net_rx_action+0x10b/0x274
Oct 25 13:59:54 asok kernel: __do_softirq+0xce/0x1c8
Oct 25 13:59:54 asok kernel: do_softirq_own_stack+0x2a/0x40
Oct 25 13:59:54 asok kernel: </IRQ>
Oct 25 13:59:54 asok kernel: do_softirq+0x4d/0x59
Oct 25 13:59:54 asok kernel: netif_rx_ni+0x1c/0x22
Oct 25 13:59:54 asok kernel: macvlan_broadcast+0x10f/0x153 [macvlan]
Oct 25 13:59:54 asok kernel: ? __switch_to_asm+0x34/0x70
Oct 25 13:59:54 asok kernel: macvlan_process_broadcast+0xd5/0x131 [macvlan]
Oct 25 13:59:54 asok kernel: process_one_work+0x16e/0x243
Oct 25 13:59:54 asok kernel: ? cancel_delayed_work_sync+0xa/0xa
Oct 25 13:59:54 asok kernel: worker_thread+0x1dc/0x2ac
Oct 25 13:59:54 asok kernel: kthread+0x10b/0x113
Oct 25 13:59:54 asok kernel: ? kthread_flush_work_fn+0x9/0x9
Oct 25 13:59:54 asok kernel: ret_from_fork+0x35/0x40
Oct 25 13:59:54 asok kernel: ---[ end trace 1da7a4f5b4f12c5a ]---

 

Link to comment
8 hours ago, subagon said:

Thank you for the information. I'll investigate the dockers with custom IPs.

FYI - it's not docker specific.  Any docker to which you may have a custom IP address may cause macvlan call traces.

 

In my case, it only happened with IP addresses on br0.  When I created a docker VLAN (br0.3) and assigned dockers custom IP addresses on br0.3 - no more call traces.  Others get them on br1.  In another thread they are discussing if Mellanox 10Gb Ethernet cards are causing them with custom IP addresses. 

 

The bottom line is that not everyone sees them and the cause varies.  I fought macvlan call traces for months until I discovered that a separate VLAN for dockers made them go away in my case.  That is not a guaranteed fix for everyone.

Edited by Hoopster
Link to comment

Quick update...

 

My system has been somewhat stable since my last post (about a week). During that time I've had a few other issues that may or may not be related to the kernel panics. Fist, I had a drive go bad and had to replace it. I can't say if the drive problem was because of the numerous crashes (unlikely), or if the drive was causing the panics, or unrelated.

 

I've also updated to 6.6.3.

 

I've seen some error messages in the syslog that seem to be related to the age of the motherboard BIOS, so a BIOS update is needed.

 

I'm also having a lot of problems with samba on my Mac desktop. I don't think it's unRAID since my Windows desktop isn't having any samba problems. I'll need to start a new thread because the samba/Mac is becoming unusable. I do see that's it's possible the Mac might be causing the kernel panics. Because of this, I've been unmounting any samba share on the Mac when not in use.

 

Right now I'm in the middle of rebuilding the lost drive from parity so I'm trying to do as little as possible on the system as not to cause a crash.

Edited by subagon
Link to comment

Been about a week since my last post and no crashes. I'm hoping that's the problem has been fixed. I've made a few changes, so I can't be sure what fixed it. I assume it was either one of the unRAID updates or replacing a failing/dead drive. I have been having a lot of problems with samba on my Mac, but don't think it's related and I'll start another thread for that topic.

Link to comment
On 11/8/2018 at 6:40 PM, subagon said:

Been about a week since my last post and no crashes. I'm hoping that's the problem has been fixed. I've made a few changes, so I can't be sure what fixed it. I assume it was either one of the unRAID updates or replacing a failing/dead drive. I have been having a lot of problems with samba on my Mac, but don't think it's related and I'll start another thread for that topic.

Can you tell me what you think may have caused this? I've just recently begun having the same type of kernel crashes. They tend to happen around 10 in the evening. The only thing, that I can think of that has a custom IP address is my docker running Pi-Hole. This was the most recent thing I setup and I believe when the issues started a couple weeks after it was up and running.

 

Thanks for any insight.

Link to comment
9 minutes ago, WillWorkForAmmo said:

Can you tell me what you think may have caused this?

Do you have macvlan call traces similar to what Subagon posted?  These are almost always the result of docker(s) with a custom IP address.  It seems to only affect certain hardware/configurations (mine being one of them). 

 

Macvlan call traces do not generally cause kernel panics/server crashes, although from time to time they would lock up my server after several days of semi-frequent call traces.

 

It doesn't really matter to which docker the custom IP address is assigned, but, I had call traces more frequently with Pihole.  The solution for me was to create a docker VLAN (br0.3).  Assigning IP addresses on this VLAN did not produce call traces whereas I got them frequently on br0. 

 

You should post diagnostics and/or screen shots of kernel panics if that is what you are experiencing.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.