Jump to content

Kernel panic occurring randomly

jerkke

By jerkke
in General Support

Recommended Posts

jerkke Noob

jerkke

Posted
Posted

Hello, I have strange problem with my Unraid throwing kernel panics randomly and thus freezing the system.

I was able to get the full log of event by tailing the log file trough ssh. 

  

login as: root
[email protected]'s password:
Linux 4.18.20-unRAID.
root@SRV-UNRAID:~#  tail -f /var/log/syslog
Feb  8 22:27:48 SRV-UNRAID emhttpd: shcmd (119): chmod 0777 '/mnt/user/Yleinen'
Feb  8 22:27:48 SRV-UNRAID emhttpd: shcmd (120): chown 'nobody':'users' '/mnt/us                                                                                                                                                             er/Yleinen'
Feb  8 22:27:48 SRV-UNRAID emhttpd: Starting services...
Feb  8 22:27:55 SRV-UNRAID emhttpd: req (3): shareName=Yleinen&shareExport=e&sha                                                                                                                                                             reFruit=no&shareSecurity=private&changeShareSecurity=Apply&csrf_token=**********                                                                                                                                                             ******
Feb  8 22:27:55 SRV-UNRAID emhttpd: Starting services...
Feb  8 22:27:55 SRV-UNRAID emhttpd: shcmd (141): smbcontrol smbd close-share 'Yl                                                                                                                                                             einen'
Feb  8 22:28:00 SRV-UNRAID emhttpd: req (4): shareName=Yleinen&userAccess.0=no-a                                                                                                                                                             ccess&userAccess.1=read-write&userAccess.2=read-write&changeShareAccess=Apply&cs                                                                                                                                                             rf_token=****************
Feb  8 22:28:00 SRV-UNRAID emhttpd: Starting services...
Feb  8 22:28:01 SRV-UNRAID emhttpd: shcmd (152): smbcontrol smbd close-share 'Yl                                                                                                                                                             einen'
Feb  8 23:18:16 SRV-UNRAID sshd[12951]: Accepted password for root from 192.168.                                                                                                                                                             1.47 port 58817 ssh2
Feb  8 23:19:10 SRV-UNRAID emhttpd: req (5): csrf_token=****************&title=System+Log&cmd=%2FwebGui%2Fscripts%2Ftail_log&arg1=syslog
Feb  8 23:19:10 SRV-UNRAID emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
Feb  9 00:10:24 SRV-UNRAID sSMTP[9177]: Creating SSL connection to host
Feb  9 00:10:24 SRV-UNRAID sSMTP[9177]: SSL connection using TLS_AES_256_GCM_SHA384
Feb  9 00:10:29 SRV-UNRAID sSMTP[9177]: Sent mail for <EMIALMASKED>@gmail.com (221 2.0.0 closing connection g11sm679855lfg.31 - gsmtp) uid=0 username=root outbytes=721
Feb  9 00:20:01 SRV-UNRAID sSMTP[20481]: Creating SSL connection to host
Feb  9 00:20:01 SRV-UNRAID sSMTP[20481]: SSL connection using TLS_AES_256_GCM_SHA384
Feb  9 00:20:04 SRV-UNRAID sSMTP[20481]: Sent mail for <EMIALMASKED>@gmail.com (221 2.0.0 closing connection z17sm744509lfh.9 - gsmtp) uid=0 username=root outbytes=1200
Feb  9 04:40:02 SRV-UNRAID root: Fix Common Problems Version 2019.01.19
Feb  9 05:02:10 SRV-UNRAID crond[2369]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null
Feb  9 06:00:07 SRV-UNRAID root: /var/lib/docker: 15.6 GiB (16683061248 bytes) trimmed
Feb  9 06:00:07 SRV-UNRAID root: /mnt/cache: 417.2 GiB (447918030848 bytes) trimmed
Feb  9 11:38:32 SRV-UNRAID apcupsd[4221]: Power failure.
Feb  9 11:38:35 SRV-UNRAID apcupsd[4221]: Power is back. UPS running on mains.
Feb  9 12:20:32 SRV-UNRAID apcupsd[4221]: Power failure.
Feb  9 12:20:35 SRV-UNRAID apcupsd[4221]: Power is back. UPS running on mains.
Feb  9 15:07:28 SRV-UNRAID emhttpd: req (6): csrf_token=****************&title=System+Log&cmd=%2FwebGui%2Fscripts%2Ftail_log&arg1=syslog
Feb  9 15:07:28 SRV-UNRAID emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
Feb  9 17:03:50 SRV-UNRAID emhttpd: req (7): csrf_token=****************&title=System+Log&cmd=%2FwebGui%2Fscripts%2Ftail_log&arg1=syslog
Feb  9 17:03:50 SRV-UNRAID emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
Feb  9 18:33:15 SRV-UNRAID emhttpd: req (8): csrf_token=****************&title=System+Log&cmd=%2FwebGui%2Fscripts%2Ftail_log&arg1=syslog
Feb  9 18:33:15 SRV-UNRAID emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
Feb 10 00:10:04 SRV-UNRAID sSMTP[38479]: Creating SSL connection to host
Feb 10 00:10:04 SRV-UNRAID sSMTP[38479]: SSL connection using TLS_AES_256_GCM_SHA384
Feb 10 00:10:07 SRV-UNRAID sSMTP[38479]: Sent mail for <EMIALMASKED>@gmail.com (221 2.0.0 closing connection p8-v6sm199543ljp.12 - gsmtp) uid=0 username=root outbytes=759
Feb 10 00:10:15 SRV-UNRAID sSMTP[38787]: Creating SSL connection to host
Feb 10 00:10:15 SRV-UNRAID sSMTP[38787]: SSL connection using TLS_AES_256_GCM_SHA384
Feb 10 00:10:17 SRV-UNRAID sSMTP[38787]: Sent mail for <EMIALMASKED>@gmail.com (221 2.0.0 closing connection b128sm1289430lfe.91 - gsmtp) uid=0 username=root outbytes=745
Feb 10 00:15:27 SRV-UNRAID emhttpd: req (9): cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=update&arg2=community.applications.plg&csrf_token=****************
Feb 10 00:15:27 SRV-UNRAID emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin update community.applications.plg
Feb 10 00:15:27 SRV-UNRAID root: plugin: running: anonymous
Feb 10 00:15:27 SRV-UNRAID root: plugin: running: anonymous
Feb 10 00:15:27 SRV-UNRAID root: plugin: creating: /boot/config/plugins/community.applications/community.applications-2019.02.08-x86_64-1.txz - downloading from URL https://raw.github.com/Squidly271/community.applications/master/archive/community.applications-2019.02.08-x86_64-1.txz
Feb 10 00:15:28 SRV-UNRAID root: plugin: checking: /boot/config/plugins/community.applications/community.applications-2019.02.08-x86_64-1.txz - MD5
Feb 10 00:15:28 SRV-UNRAID root: plugin: running: /boot/config/plugins/community.applications/community.applications-2019.02.08-x86_64-1.txz
Feb 10 00:15:28 SRV-UNRAID root: plugin: running: anonymous
Feb 10 00:20:01 SRV-UNRAID sSMTP[51509]: Creating SSL connection to host
Feb 10 00:20:02 SRV-UNRAID sSMTP[51509]: SSL connection using TLS_AES_256_GCM_SHA384
Feb 10 00:20:04 SRV-UNRAID sSMTP[51509]: Sent mail for <EMIALMASKED>@gmail.com (221 2.0.0 closing connection m128sm1128277lfg.88 - gsmtp) uid=0 username=root outbytes=1201
Feb 10 01:48:32 SRV-UNRAID kernel: WARNING: CPU: 1 PID: 0 at net/netfilter/nf_conntrack_core.c:763 __nf_conntrack_confirm+0x96/0x4fc
Feb 10 01:48:32 SRV-UNRAID kernel: Modules linked in: macvlan tun xt_nat veth ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs md_mod igb i2c_algo_bit sb_edac ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper hpsa intel_cstate intel_uncore intel_rapl_perf i2c_core scsi_transport_sas thermal ipmi_si button pcc_cpufreq [last unloaded: acpi_power_meter]
Feb 10 01:48:32 SRV-UNRAID kernel: CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.18.20-unRAID #1
Feb 10 01:48:32 SRV-UNRAID kernel: Hardware name: HP ProLiant DL380e Gen8, BIOS P73 01/22/2018
Feb 10 01:48:32 SRV-UNRAID kernel: RIP: 0010:__nf_conntrack_confirm+0x96/0x4fc
Feb 10 01:48:32 SRV-UNRAID kernel: Code: c1 ed 20 89 2c 24 e8 26 f7 ff ff 8b 54 24 04 89 ef 89 c6 41 89 c5 e8 bc f8 ff ff 84 c0 75 b9 49 8b 86 80 00 00 00 a8 08 74 02 <0f> 0b 4c 89 f7 e8 04 ff ff ff 49 8b 86 80 00 00 00 0f ba e0 09 73
Feb 10 01:48:32 SRV-UNRAID kernel: RSP: 0018:ffff8806676438f8 EFLAGS: 00010202
Feb 10 01:48:32 SRV-UNRAID kernel: RAX: 0000000000000188 RBX: ffff8805c5dd2000 RCX: 0000000000000100
Feb 10 01:48:32 SRV-UNRAID kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffff81e094f8
Feb 10 01:48:32 SRV-UNRAID kernel: RBP: 00000000000017be R08: 0000000082d0ba69 R09: 0000000000000000
Feb 10 01:48:32 SRV-UNRAID kernel: R10: 0000000000000158 R11: ffffffff81e8cdc0 R12: ffffffff81e8cdc0
Feb 10 01:48:32 SRV-UNRAID kernel: R13: 000000000000ddf5 R14: ffff880142977480 R15: ffff8801429774d8
Feb 10 01:48:32 SRV-UNRAID kernel: FS:  0000000000000000(0000) GS:ffff880667640000(0000) knlGS:0000000000000000
Feb 10 01:48:32 SRV-UNRAID kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 10 01:48:32 SRV-UNRAID kernel: CR2: 000000c42062d010 CR3: 0000000001e0a001 CR4: 00000000000606e0
Feb 10 01:48:32 SRV-UNRAID kernel: Call Trace:
Feb 10 01:48:32 SRV-UNRAID kernel: <IRQ>
Feb 10 01:48:32 SRV-UNRAID kernel: ipv4_confirm+0xaf/0xb7 [nf_conntrack_ipv4]
Feb 10 01:48:32 SRV-UNRAID kernel: nf_hook_slow+0x37/0x96
Feb 10 01:48:32 SRV-UNRAID kernel: ip_local_deliver+0xa7/0xd5
Feb 10 01:48:32 SRV-UNRAID kernel: ? inet_del_offload+0x3e/0x3e
Feb 10 01:48:32 SRV-UNRAID kernel: ip_sabotage_in+0x38/0x3e
Feb 10 01:48:32 SRV-UNRAID kernel: nf_hook_slow+0x37/0x96
Feb 10 01:48:32 SRV-UNRAID kernel: ip_rcv+0x2ca/0x317
Feb 10 01:48:32 SRV-UNRAID kernel: ? ip_local_deliver_finish+0x1aa/0x1aa
Feb 10 01:48:32 SRV-UNRAID kernel: __netif_receive_skb_core+0x6b2/0x740
Feb 10 01:48:32 SRV-UNRAID kernel: ? nf_conntrack_tuple_taken+0x45/0x18c
Feb 10 01:48:32 SRV-UNRAID kernel: netif_receive_skb_internal+0x9f/0xba
Feb 10 01:48:32 SRV-UNRAID kernel: br_pass_frame_up+0x121/0x143
Feb 10 01:48:32 SRV-UNRAID kernel: ? br_flood+0xa8/0x148
Feb 10 01:48:32 SRV-UNRAID kernel: ? br_fdb_update+0x56/0x140
Feb 10 01:48:32 SRV-UNRAID kernel: br_handle_frame_finish+0x331/0x376
Feb 10 01:48:32 SRV-UNRAID kernel: ? br_pass_frame_up+0x143/0x143
Feb 10 01:48:32 SRV-UNRAID kernel: br_nf_hook_thresh+0xa3/0xc3
Feb 10 01:48:32 SRV-UNRAID kernel: ? br_pass_frame_up+0x143/0x143
Feb 10 01:48:32 SRV-UNRAID kernel: br_nf_pre_routing_finish+0x22f/0x256
Feb 10 01:48:32 SRV-UNRAID kernel: ? br_pass_frame_up+0x143/0x143
Feb 10 01:48:32 SRV-UNRAID kernel: ? nf_nat_ipv4_in+0x1d/0x64 [nf_nat_ipv4]
Feb 10 01:48:32 SRV-UNRAID kernel: br_nf_pre_routing+0x2d2/0x2f7
Feb 10 01:48:32 SRV-UNRAID kernel: ? br_nf_forward_ip+0x349/0x349
Feb 10 01:48:32 SRV-UNRAID kernel: nf_hook_slow+0x37/0x96
Feb 10 01:48:32 SRV-UNRAID kernel: br_handle_frame+0x291/0x2d0
Feb 10 01:48:32 SRV-UNRAID kernel: ? br_pass_frame_up+0x143/0x143
Feb 10 01:48:32 SRV-UNRAID kernel: ? br_handle_local_finish+0x31/0x31
Feb 10 01:48:32 SRV-UNRAID kernel: __netif_receive_skb_core+0x459/0x740
Feb 10 01:48:32 SRV-UNRAID kernel: ? inet_gro_receive+0x249/0x257
Feb 10 01:48:32 SRV-UNRAID kernel: netif_receive_skb_internal+0x9f/0xba
Feb 10 01:48:32 SRV-UNRAID kernel: napi_gro_receive+0x42/0x76
Feb 10 01:48:32 SRV-UNRAID kernel: igb_poll+0xb96/0xbbc [igb]
Feb 10 01:48:32 SRV-UNRAID kernel: net_rx_action+0x10b/0x274
Feb 10 01:48:32 SRV-UNRAID kernel: __do_softirq+0xce/0x1e2
Feb 10 01:48:32 SRV-UNRAID kernel: irq_exit+0x5e/0x9d
Feb 10 01:48:32 SRV-UNRAID kernel: do_IRQ+0xa9/0xc7
Feb 10 01:48:32 SRV-UNRAID kernel: common_interrupt+0xf/0xf
Feb 10 01:48:32 SRV-UNRAID kernel: </IRQ>
Feb 10 01:48:32 SRV-UNRAID kernel: RIP: 0010:cpuidle_enter_state+0xe5/0x141
Feb 10 01:48:32 SRV-UNRAID kernel: Code: ce 32 bb ff 45 84 ff 74 1d 9c 58 66 66 90 66 90 0f ba e0 09 73 09 0f 0b fa 66 66 90 66 66 90 31 ff e8 73 af be ff fb 66 66 90 <66> 66 90 48 2b 1c 24 b8 ff ff ff 7f 48 b9 ff ff ff ff f3 01 00 00
Feb 10 01:48:32 SRV-UNRAID kernel: RSP: 0018:ffffc900062cbea0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffdb
Feb 10 01:48:32 SRV-UNRAID kernel: RAX: ffff880667660c00 RBX: 00005a27d358655e RCX: 000000000000001f
Feb 10 01:48:32 SRV-UNRAID kernel: RDX: 00005a27d358655e RSI: 000000004748524f RDI: 0000000000000000
Feb 10 01:48:32 SRV-UNRAID kernel: RBP: ffff880667669c00 R08: 0000000000000004 R09: 0000000000020480
Feb 10 01:48:32 SRV-UNRAID kernel: R10: 00000002426f95a4 R11: 0000a221f59e4f52 R12: 0000000000000005
Feb 10 01:48:32 SRV-UNRAID kernel: R13: 0000000000000005 R14: ffffffff81e58a18 R15: 0000000000000000
Feb 10 01:48:32 SRV-UNRAID kernel: do_idle+0x192/0x20e
Feb 10 01:48:32 SRV-UNRAID kernel: cpu_startup_entry+0x6a/0x6c
Feb 10 01:48:32 SRV-UNRAID kernel: start_secondary+0x197/0x1b2
Feb 10 01:48:32 SRV-UNRAID kernel: secondary_startup_64+0xa5/0xb0
Feb 10 01:48:32 SRV-UNRAID kernel: ---[ end trace effc81621f0f1ab0 ]---
Feb 10 02:22:19 SRV-UNRAID kernel: general protection fault: 0000 [#1] SMP PTI
Feb 10 02:22:19 SRV-UNRAID kernel: CPU: 17 PID: 5624 Comm: Plex DLNA Serve Tainted: G        W         4.18.20-unRAID #1
Feb 10 02:22:19 SRV-UNRAID kernel: Hardware name: HP ProLiant DL380e Gen8, BIOS P73 01/22/2018
Feb 10 02:22:19 SRV-UNRAID kernel: RIP: 0010:nf_nat_setup_info+0x178/0x5d4 [nf_nat]
Feb 10 02:22:19 SRV-UNRAID kernel: Code: 45 31 e4 e8 14 f8 ff ff 48 8b 15 3b 27 00 00 89 c0 48 8d 04 c2 48 8b 00 48 85 c0 74 2c 4c 8d a0 68 ff ff ff eb 23 8a 44 24 4e <41> 38 44 24 46 74 22 4d 8b a4 24 98 00 00 00 4d 85 e4 0f 84 2f ff
Feb 10 02:22:19 SRV-UNRAID kernel: RSP: 0018:ffffc9000d6ab918 EFLAGS: 00010202
Feb 10 02:22:19 SRV-UNRAID kernel: RAX: ffff880142977511 RBX: ffff880100716300 RCX: 0000000034aff03c
Feb 10 02:22:19 SRV-UNRAID kernel: RDX: ffff880645d00000 RSI: 00000000c465e0bd RDI: 0000000047f2ddf6
Feb 10 02:22:19 SRV-UNRAID kernel: RBP: ffffc9000d6ab9e8 R08: 0000000000000038 R09: ffff8806636d2188
Feb 10 02:22:19 SRV-UNRAID kernel: R10: 0000000000000348 R11: ffffc9000d6abacc R12: 02fffeffffffff68
Feb 10 02:22:19 SRV-UNRAID kernel: R13: ffffc9000d6ab9fc R14: ffffffffa00251c0 R15: 0000000000000000
Feb 10 02:22:19 SRV-UNRAID kernel: FS:  0000150ae63f9700(0000) GS:ffff880667840000(0000) knlGS:0000000000000000
Feb 10 02:22:19 SRV-UNRAID kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 10 02:22:19 SRV-UNRAID kernel: CR2: 0000150aecc4c000 CR3: 000000068227e004 CR4: 00000000000606e0
Feb 10 02:22:19 SRV-UNRAID kernel: Call Trace:
Feb 10 02:22:19 SRV-UNRAID kernel: ? remove_wait_queue+0xd/0x45
Feb 10 02:22:19 SRV-UNRAID kernel: ? __krealloc+0x25/0x5d
Feb 10 02:22:19 SRV-UNRAID kernel: ? nf_ct_ext_add+0x8a/0xe9
Feb 10 02:22:19 SRV-UNRAID kernel: nf_nat_masquerade_ipv4+0x106/0x126 [nf_nat_ipv4]
Feb 10 02:22:19 SRV-UNRAID kernel: masquerade_tg+0x44/0x5e [ipt_MASQUERADE]
Feb 10 02:22:19 SRV-UNRAID kernel: ? nf_nat_setup_info+0x416/0x5d4 [nf_nat]
Feb 10 02:22:19 SRV-UNRAID kernel: ipt_do_table+0x531/0x5db [ip_tables]
Feb 10 02:22:19 SRV-UNRAID kernel: nf_nat_inet_fn+0xfe/0x198 [nf_nat]
Feb 10 02:22:19 SRV-UNRAID kernel: nf_nat_ipv4_out+0xf/0x85 [nf_nat_ipv4]
Feb 10 02:22:19 SRV-UNRAID kernel: nf_hook_slow+0x37/0x96
Feb 10 02:22:19 SRV-UNRAID kernel: ip_output+0xa3/0xd4
Feb 10 02:22:19 SRV-UNRAID kernel: ? ip_fragment.constprop.5+0x6e/0x6e
Feb 10 02:22:19 SRV-UNRAID kernel: ip_send_skb+0x10/0x32
Feb 10 02:22:19 SRV-UNRAID kernel: udp_send_skb+0x260/0x2ca
Feb 10 02:22:19 SRV-UNRAID kernel: udp_sendmsg+0x51a/0x701
Feb 10 02:22:19 SRV-UNRAID kernel: ? core_sys_select+0x1e0/0x231
Feb 10 02:22:19 SRV-UNRAID kernel: ? seccomp_run_filters+0xf0/0x12f
Feb 10 02:22:19 SRV-UNRAID kernel: sock_sendmsg+0x14/0x1e
Feb 10 02:22:19 SRV-UNRAID kernel: __sys_sendto+0xcc/0x10a
Feb 10 02:22:19 SRV-UNRAID kernel: ? syscall_trace_enter+0x164/0x1a3
Feb 10 02:22:19 SRV-UNRAID kernel: __x64_sys_sendto+0x1f/0x22
Feb 10 02:22:19 SRV-UNRAID kernel: do_syscall_64+0x57/0xe6
Feb 10 02:22:19 SRV-UNRAID kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Feb 10 02:22:19 SRV-UNRAID kernel: RIP: 0033:0x150afc5849ae
Feb 10 02:22:19 SRV-UNRAID kernel: Code: d5 53 49 89 f4 89 fb 48 83 ec 10 e8 6c f6 ff ff 45 31 c9 89 c5 45 31 c0 45 89 f2 4c 89 ea 4c 89 e6 89 df b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 36 89 ef 48 89 44 24 08 e8 9e f6 ff ff 48 8b
Feb 10 02:22:19 SRV-UNRAID kernel: RSP: 002b:0000150ae63f89e0 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
Feb 10 02:22:19 SRV-UNRAID kernel: RAX: ffffffffffffffda RBX: 0000000000000029 RCX: 0000150afc5849ae
Feb 10 02:22:19 SRV-UNRAID kernel: RDX: 0000000000000155 RSI: 0000150aecc43180 RDI: 0000000000000029
Feb 10 02:22:19 SRV-UNRAID kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Feb 10 02:22:19 SRV-UNRAID kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000150aecc43180
Feb 10 02:22:19 SRV-UNRAID kernel: R13: 0000000000000155 R14: 0000000000004000 R15: 0000000000000000
Feb 10 02:22:19 SRV-UNRAID kernel: Modules linked in: macvlan tun xt_nat veth ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs md_mod igb i2c_algo_bit sb_edac ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper hpsa intel_cstate intel_uncore intel_rapl_perf i2c_core scsi_transport_sas thermal ipmi_si button pcc_cpufreq [last unloaded: acpi_power_meter]
Feb 10 02:22:19 SRV-UNRAID kernel: ---[ end trace effc81621f0f1ab1 ]---
Feb 10 02:22:19 SRV-UNRAID kernel: RIP: 0010:nf_nat_setup_info+0x178/0x5d4 [nf_nat]
Feb 10 02:22:19 SRV-UNRAID kernel: Code: 45 31 e4 e8 14 f8 ff ff 48 8b 15 3b 27 00 00 89 c0 48 8d 04 c2 48 8b 00 48 85 c0 74 2c 4c 8d a0 68 ff ff ff eb 23 8a 44 24 4e <41> 38 44 24 46 74 22 4d 8b a4 24 98 00 00 00 4d 85 e4 0f 84 2f ff
Feb 10 02:22:19 SRV-UNRAID kernel: RSP: 0018:ffffc9000d6ab918 EFLAGS: 00010202
Feb 10 02:22:19 SRV-UNRAID kernel: RAX: ffff880142977511 RBX: ffff880100716300 RCX: 0000000034aff03c
Feb 10 02:22:19 SRV-UNRAID kernel: RDX: ffff880645d00000 RSI: 00000000c465e0bd RDI: 0000000047f2ddf6
Feb 10 02:22:19 SRV-UNRAID kernel: RBP: ffffc9000d6ab9e8 R08: 0000000000000038 R09: ffff8806636d2188
Feb 10 02:22:19 SRV-UNRAID kernel: R10: 0000000000000348 R11: ffffc9000d6abacc R12: 02fffeffffffff68
Feb 10 02:22:19 SRV-UNRAID kernel: R13: ffffc9000d6ab9fc R14: ffffffffa00251c0 R15: 0000000000000000
Feb 10 02:22:19 SRV-UNRAID kernel: FS:  0000150ae63f9700(0000) GS:ffff880667840000(0000) knlGS:0000000000000000
Feb 10 02:22:19 SRV-UNRAID kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033

Message from syslogd@SRV-UNRAID at Feb 10 02:22:19 ...
 kernel:Kernel panic - not syncing: Fatal exception in interrupt
Feb 10 02:22:19 SRV-UNRAID kernel: CR2: 0000150aecc4c000 CR3: 000000068227e004 CR4: 00000000000606e0
Feb 10 02:22:19 SRV-UNRAID kernel: Kernel panic - not syncing: Fatal exception in interrupt

As seen in the logs the server ran fine for few days and then crashed with kernel panic.

I also attached diagnostics after rebooting the system. This was not first time this occurred but this time I got the full log. Before this kernel panic I did ran memtest on the server for a few passes and it did not find any errors.

 

Any help with this is much appreciated!

srv-unraid-diagnostics-20190210-1257.zip

Link to comment
jerkke Noob

jerkke

Posted
  • Author
Posted

No, I have not tried the safe mode yet. I'm currently re-building the parity. I also disabled docker for now to see if that has any effect on the kernel panics. I'll let the parity check to complete and see if I get any kernel panics without docker running. If I still get kernel panics I'll try the Safe mode next. 

 

 

Link to comment
  • 3 weeks later...
jerkke Noob

jerkke

Posted
  • Author
Posted

Yeah, I think I have been able to isolate the issue into one docker container. The container has custom IP address assigned and I think that is the reason for the issues. Short story about the investigation:

  1. I completely disabled docker and let the server run for about four days
    1. No kernel panics occurred
  2. I enabled docker and started all containers expect the one that had custom IP address assigned
    1. No kernel panics occurred

I have not yet started the problematic container but I'm going to try it with some additional changes to my network interface configurations.

I currently have two NICs. First one has static IP address and is used by the unraid and the docker containers running in host/bridge.

Second one has also static IP address assigned and used by the problematic docker container which also has an static IP address assigned. I have done some research about issues related to custom IP addresses on docker containers and one solution proposed in this forum was to create separate VLAN for the docker containers. I do not have enough knowledge to create and configure VLAN. I understand the concept but have never created one and since I have two NICs I'm going to try to get this working without VLANs. Even though it would be good practice for me to create one.

 

So, my next battle plan is to remove the static IP address from the second NIC completely (Unraid Network settings -> IPv4 addres assigment = none) and only assign custom IP trough that NIC for the docker container in question (Conatiner settings -> Network type = custom and set the fixed IP address). I have not yet tried this but maybe I find some time in the weekend to try this out.

 

21 minutes ago, MrKevbo said:

I'm having the exact same issue.  Any updates on how your investigation is going?

Do you MrKevbo have any docker containers with custom network configurations? 

 

Link to comment
Hoopster Experienced

Hoopster

Posted
Posted
1 hour ago, jerkke said:

Yeah, I think I have been able to isolate the issue into one docker container. The container has custom IP address assigned and I think that is the reason for the issues.

I had the same issue.  Although the call traces are not identical, it resulted from dockers assigned custom IP addresses on br0.  As soon as I set up a docker VLAN on br0.3 the call traces/kernel panics went away.  Others had the issue on br1.  It is not docker specific, it is simply the fact that a custom IP address is assigned to a docker on a particular interface.

 

Fortunately, it took several call traces over 2-4 days before it locked up my server.  Kernel panics were not the immediate result.

 

Here is more on the issue in case it is helpful to you:

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing
×
×
  • Create New...