subagon Posted October 12, 2018 Share Posted October 12, 2018 Started after I updated to 6.6.x. The system runs fine for several days and then is unresponsive. After power cycling the system I have to endure a 2 day parity check. Over the last 2 months I'd guess this has happened 4 times. The console shows a kernel panic (sorry for the poor image).asok-diagnostics-20181012-0708.zip Quote Link to comment
subagon Posted October 17, 2018 Author Share Posted October 17, 2018 Five days and another crash. Quote Link to comment
deadnote Posted October 18, 2018 Share Posted October 18, 2018 (edited) Same here since I updated to 6.6.2 : 2 crashes the first day. After the second hard reboot, it seems to work but I have warning in logs (in attachement). The jpg is the screen of the last crash. Do I need to rollback to 6.6.1 ? Thanks for your help tower-diagnostics-20181018-2105.zip Edited October 18, 2018 by deadnote add informations Quote Link to comment
subagon Posted October 19, 2018 Author Share Posted October 19, 2018 Another crash. This time in only 2 days. The parity check from the prior crash didn't finish. I could really use some help. Quote Link to comment
subagon Posted October 25, 2018 Author Share Posted October 25, 2018 No crash to report since the last post. Since then I have updated to 6.6.2 and kept an open terminal tailing /var/log/syslog. I just saw this from 30 minutes ago. It did not cause a kernel panic/crash but looked like it might provide a clue. Oct 25 13:59:54 asok kernel: WARNING: CPU: 15 PID: 20898 at net/netfilter/nf_conntrack_core.c:763 __nf_conntrack_confirm+0x96/0x4fc Oct 25 13:59:54 asok kernel: Modules linked in: macvlan xt_nat veth xt_CHECKSUM iptable_mangle ipt_REJECT ebtable_filter ebtables ip6table_filter ip6_tables vhost_net tun vhost tap ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs md_mod nct7904 igb i2c_algo_bit sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd ipmi_ssif mpt3sas glue_helper i2c_i801 i2c_core intel_cstate intel_uncore ahci intel_rapl_perf libahci raid_class scsi_transport_sas pcc_cpufreq wmi ipmi_si button [last unloaded: i2c_algo_bit] Oct 25 13:59:54 asok kernel: CPU: 15 PID: 20898 Comm: kworker/15:1 Tainted: G W 4.18.14-unRAID #1 Oct 25 13:59:54 asok kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0 07/09/2013 Oct 25 13:59:54 asok kernel: Workqueue: events macvlan_process_broadcast [macvlan] Oct 25 13:59:54 asok kernel: RIP: 0010:__nf_conntrack_confirm+0x96/0x4fc Oct 25 13:59:54 asok kernel: Code: c1 ed 20 89 2c 24 e8 26 f7 ff ff 8b 54 24 04 89 ef 89 c6 41 89 c5 e8 bc f8 ff ff 84 c0 75 b9 49 8b 86 80 00 00 00 a8 08 74 02 <0f> 0b 4c 89 f7 e8 04 ff ff ff 49 8b 86 80 00 00 00 0f ba e0 09 73 Oct 25 13:59:54 asok kernel: RSP: 0018:ffff88085f443d30 EFLAGS: 00010202 Oct 25 13:59:54 asok kernel: RAX: 0000000000000188 RBX: ffff8801986ede00 RCX: 0000000000000101 Oct 25 13:59:54 asok kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffff81e09070 Oct 25 13:59:54 asok kernel: RBP: 000000000000b1af R08: 00000000c8ff4e01 R09: 0000000000000000 Oct 25 13:59:54 asok kernel: R10: 0000000000000098 R11: 0000000000000001 R12: ffffffff81e8cc80 Oct 25 13:59:54 asok kernel: R13: 000000000000729c R14: ffff8805f75e9a40 R15: ffff8805f75e9a98 Oct 25 13:59:54 asok kernel: FS: 0000000000000000(0000) GS:ffff88085f440000(0000) knlGS:0000000000000000 Oct 25 13:59:54 asok kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 25 13:59:54 asok kernel: CR2: 0000153b5c810000 CR3: 0000000001e0a003 CR4: 00000000001606e0 Oct 25 13:59:54 asok kernel: Call Trace: Oct 25 13:59:54 asok kernel: <IRQ> Oct 25 13:59:54 asok kernel: ipv4_confirm+0xaf/0xb7 [nf_conntrack_ipv4] Oct 25 13:59:54 asok kernel: nf_hook_slow+0x37/0x96 Oct 25 13:59:54 asok kernel: ip_local_deliver+0xa7/0xd5 Oct 25 13:59:54 asok kernel: ? inet_del_offload+0x3e/0x3e Oct 25 13:59:54 asok kernel: ip_rcv+0x2dc/0x317 Oct 25 13:59:54 asok kernel: ? ip_local_deliver_finish+0x1aa/0x1aa Oct 25 13:59:54 asok kernel: __netif_receive_skb_core+0x6b2/0x740 Oct 25 13:59:54 asok kernel: ? kmem_cache_free_bulk+0x25b/0x273 Oct 25 13:59:54 asok kernel: process_backlog+0x7e/0x116 Oct 25 13:59:54 asok kernel: net_rx_action+0x10b/0x274 Oct 25 13:59:54 asok kernel: __do_softirq+0xce/0x1c8 Oct 25 13:59:54 asok kernel: do_softirq_own_stack+0x2a/0x40 Oct 25 13:59:54 asok kernel: </IRQ> Oct 25 13:59:54 asok kernel: do_softirq+0x4d/0x59 Oct 25 13:59:54 asok kernel: netif_rx_ni+0x1c/0x22 Oct 25 13:59:54 asok kernel: macvlan_broadcast+0x10f/0x153 [macvlan] Oct 25 13:59:54 asok kernel: ? __switch_to_asm+0x34/0x70 Oct 25 13:59:54 asok kernel: macvlan_process_broadcast+0xd5/0x131 [macvlan] Oct 25 13:59:54 asok kernel: process_one_work+0x16e/0x243 Oct 25 13:59:54 asok kernel: ? cancel_delayed_work_sync+0xa/0xa Oct 25 13:59:54 asok kernel: worker_thread+0x1dc/0x2ac Oct 25 13:59:54 asok kernel: kthread+0x10b/0x113 Oct 25 13:59:54 asok kernel: ? kthread_flush_work_fn+0x9/0x9 Oct 25 13:59:54 asok kernel: ret_from_fork+0x35/0x40 Oct 25 13:59:54 asok kernel: ---[ end trace 1da7a4f5b4f12c5a ]--- Quote Link to comment
JorgeB Posted October 25, 2018 Share Posted October 25, 2018 macvlan call traces are usually related to dockers with custom IP addresses. Quote Link to comment
subagon Posted October 25, 2018 Author Share Posted October 25, 2018 (edited) Thank you for the information. I'll investigate the dockers with custom IPs. Edited October 25, 2018 by subagon Quote Link to comment
Hoopster Posted October 25, 2018 Share Posted October 25, 2018 (edited) 8 hours ago, subagon said: Thank you for the information. I'll investigate the dockers with custom IPs. FYI - it's not docker specific. Any docker to which you may have a custom IP address may cause macvlan call traces. In my case, it only happened with IP addresses on br0. When I created a docker VLAN (br0.3) and assigned dockers custom IP addresses on br0.3 - no more call traces. Others get them on br1. In another thread they are discussing if Mellanox 10Gb Ethernet cards are causing them with custom IP addresses. The bottom line is that not everyone sees them and the cause varies. I fought macvlan call traces for months until I discovered that a separate VLAN for dockers made them go away in my case. That is not a guaranteed fix for everyone. Edited October 26, 2018 by Hoopster Quote Link to comment
subagon Posted November 2, 2018 Author Share Posted November 2, 2018 (edited) Quick update... My system has been somewhat stable since my last post (about a week). During that time I've had a few other issues that may or may not be related to the kernel panics. Fist, I had a drive go bad and had to replace it. I can't say if the drive problem was because of the numerous crashes (unlikely), or if the drive was causing the panics, or unrelated. I've also updated to 6.6.3. I've seen some error messages in the syslog that seem to be related to the age of the motherboard BIOS, so a BIOS update is needed. I'm also having a lot of problems with samba on my Mac desktop. I don't think it's unRAID since my Windows desktop isn't having any samba problems. I'll need to start a new thread because the samba/Mac is becoming unusable. I do see that's it's possible the Mac might be causing the kernel panics. Because of this, I've been unmounting any samba share on the Mac when not in use. Right now I'm in the middle of rebuilding the lost drive from parity so I'm trying to do as little as possible on the system as not to cause a crash. Edited November 2, 2018 by subagon Quote Link to comment
subagon Posted November 9, 2018 Author Share Posted November 9, 2018 Been about a week since my last post and no crashes. I'm hoping that's the problem has been fixed. I've made a few changes, so I can't be sure what fixed it. I assume it was either one of the unRAID updates or replacing a failing/dead drive. I have been having a lot of problems with samba on my Mac, but don't think it's related and I'll start another thread for that topic. Quote Link to comment
WillWorkForAmmo Posted November 16, 2018 Share Posted November 16, 2018 On 11/8/2018 at 6:40 PM, subagon said: Been about a week since my last post and no crashes. I'm hoping that's the problem has been fixed. I've made a few changes, so I can't be sure what fixed it. I assume it was either one of the unRAID updates or replacing a failing/dead drive. I have been having a lot of problems with samba on my Mac, but don't think it's related and I'll start another thread for that topic. Can you tell me what you think may have caused this? I've just recently begun having the same type of kernel crashes. They tend to happen around 10 in the evening. The only thing, that I can think of that has a custom IP address is my docker running Pi-Hole. This was the most recent thing I setup and I believe when the issues started a couple weeks after it was up and running. Thanks for any insight. Quote Link to comment
Hoopster Posted November 16, 2018 Share Posted November 16, 2018 9 minutes ago, WillWorkForAmmo said: Can you tell me what you think may have caused this? Do you have macvlan call traces similar to what Subagon posted? These are almost always the result of docker(s) with a custom IP address. It seems to only affect certain hardware/configurations (mine being one of them). Macvlan call traces do not generally cause kernel panics/server crashes, although from time to time they would lock up my server after several days of semi-frequent call traces. It doesn't really matter to which docker the custom IP address is assigned, but, I had call traces more frequently with Pihole. The solution for me was to create a docker VLAN (br0.3). Assigning IP addresses on this VLAN did not produce call traces whereas I got them frequently on br0. You should post diagnostics and/or screen shots of kernel panics if that is what you are experiencing. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.