BlackNL16 Posted March 9, 2021 Share Posted March 9, 2021 Hi All, Since I started using unraid I have regularly crashes ( once a 2/3 weeks or so ) Sometimes even more. Now I recently added the syslog towards my NAS, So now I am in possession of the logs before the crash. Server: Dell R630 2x 600GB SAS Drives Version: 6.9.0 If anyone has a suggestion what could be the reason of this random crashes. I would appreciate it. I've added the syslog entries as attachments. unraid-error..csv unraid-error.xlsx Quote Link to comment
Squid Posted March 9, 2021 Share Posted March 9, 2021 You need to upload your diagnostics to put everything into context. Ideally, you should also set up the syslog server to mirror to the flash drive and post the resulting file after the next crash Quote Link to comment
BlackNL16 Posted March 15, 2021 Author Share Posted March 15, 2021 today it happend again I dont know the exact time, but I Suppose this belongs to the crash. Mar 15 12:51:53 Unraid kernel: ------------[ cut here ]------------ Mar 15 12:51:53 Unraid kernel: WARNING: CPU: 0 PID: 0 at net/netfilter/nf_conntrack_core.c:1120 __nf_conntrack_confirm+0x9b/0x1e6 Mar 15 12:51:53 Unraid kernel: Modules linked in: xt_CHECKSUM ipt_REJECT xt_nat ip6table_mangle ip6table_nat nfsv3 nfs nfs_ssc iptable_mangle veth vhost_net tun vhost vhost_iotlb tap macvlan xt_MASQUERADE iptable_nat nf_nat xfs nfsd lockd grace sunrpc md_mod ip6table_filter ip6_tables iptable_filter ip_tables bonding tg3 sr_mod cdrom sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel mxm_wmi kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper rapl intel_cstate intel_uncore ftdi_sio cdc_acm usbserial usblp ipmi_ssif ahci i2c_core libahci megaraid_sas ipmi_si wmi acpi_power_meter button [last unloaded: tg3] Mar 15 12:51:53 Unraid kernel: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.10.21-Unraid #1 Mar 15 12:51:53 Unraid kernel: Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.12.1 12/04/2020 Mar 15 12:51:53 Unraid kernel: RIP: 0010:__nf_conntrack_confirm+0x9b/0x1e6 Mar 15 12:51:53 Unraid kernel: Code: e8 64 f9 ff ff 44 89 fa 89 c6 41 89 c4 48 c1 eb 20 89 df 41 89 de e8 d5 f6 ff ff 84 c0 75 bb 48 8b 85 80 00 00 00 a8 08 74 18 <0f> 0b 89 df 44 89 e6 31 db e8 5d f3 ff ff e8 30 f6 ff ff e9 22 01 Mar 15 12:51:53 Unraid kernel: RSP: 0018:ffffc90000003930 EFLAGS: 00010202 Mar 15 12:51:53 Unraid kernel: RAX: 0000000000000188 RBX: 000000000000364b RCX: 000000009770d25e Mar 15 12:51:53 Unraid kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff8200a16c Mar 15 12:51:53 Unraid kernel: RBP: ffff88a17b2e5900 R08: 00000000ca256227 R09: ffff888100d82200 Mar 15 12:51:53 Unraid kernel: R10: 0000000000000158 R11: ffff889a88aa3400 R12: 000000000000d1b2 Mar 15 12:51:53 Unraid kernel: R13: ffffffff8210db40 R14: 000000000000364b R15: 0000000000000000 Mar 15 12:51:53 Unraid kernel: FS: 0000000000000000(0000) GS:ffff88a7dfa00000(0000) knlGS:0000000000000000 Mar 15 12:51:53 Unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Mar 15 12:51:53 Unraid kernel: CR2: 000014a39b24b000 CR3: 000000016faba004 CR4: 00000000001706f0 Mar 15 12:51:53 Unraid kernel: Call Trace: Mar 15 12:51:53 Unraid kernel: <IRQ> Mar 15 12:51:53 Unraid kernel: nf_conntrack_confirm+0x2f/0x36 Mar 15 12:51:53 Unraid kernel: nf_hook_slow+0x39/0x8e Mar 15 12:51:53 Unraid kernel: nf_hook.constprop.0+0xb1/0xd8 Mar 15 12:51:53 Unraid kernel: ? ip_protocol_deliver_rcu+0xfe/0xfe Mar 15 12:51:53 Unraid kernel: ip_local_deliver+0x49/0x75 Mar 15 12:51:53 Unraid kernel: ip_sabotage_in+0x43/0x4d Mar 15 12:51:53 Unraid kernel: nf_hook_slow+0x39/0x8e Mar 15 12:51:53 Unraid kernel: nf_hook.constprop.0+0xb1/0xd8 Mar 15 12:51:53 Unraid kernel: ? l3mdev_l3_rcv.constprop.0+0x50/0x50 Mar 15 12:51:53 Unraid kernel: ip_rcv+0x41/0x61 Mar 15 12:51:53 Unraid kernel: __netif_receive_skb_one_core+0x74/0x95 Mar 15 12:51:53 Unraid kernel: netif_receive_skb+0x79/0xa1 Mar 15 12:51:53 Unraid kernel: br_handle_frame_finish+0x30d/0x351 Mar 15 12:51:53 Unraid kernel: ? ipt_do_table+0x570/0x5c0 [ip_tables] Mar 15 12:51:53 Unraid kernel: ? br_pass_frame_up+0xda/0xda Mar 15 12:51:53 Unraid kernel: br_nf_hook_thresh+0xa3/0xc3 Mar 15 12:51:53 Unraid kernel: ? br_pass_frame_up+0xda/0xda Mar 15 12:51:53 Unraid kernel: br_nf_pre_routing_finish+0x23d/0x264 Mar 15 12:51:53 Unraid kernel: ? br_pass_frame_up+0xda/0xda Mar 15 12:51:53 Unraid kernel: ? br_handle_frame_finish+0x351/0x351 Mar 15 12:51:53 Unraid kernel: ? nf_nat_ipv4_in+0x1e/0x4a [nf_nat] Mar 15 12:51:53 Unraid kernel: ? br_nf_forward_finish+0xd0/0xd0 Mar 15 12:51:53 Unraid kernel: ? br_handle_frame_finish+0x351/0x351 Mar 15 12:51:53 Unraid kernel: NF_HOOK+0xd7/0xf7 Mar 15 12:51:53 Unraid kernel: ? br_nf_forward_finish+0xd0/0xd0 Mar 15 12:51:53 Unraid kernel: br_nf_pre_routing+0x229/0x239 Mar 15 12:51:53 Unraid kernel: ? br_nf_forward_finish+0xd0/0xd0 Mar 15 12:51:53 Unraid kernel: br_handle_frame+0x25e/0x2a6 Mar 15 12:51:53 Unraid kernel: ? br_pass_frame_up+0xda/0xda Mar 15 12:51:53 Unraid kernel: __netif_receive_skb_core+0x335/0x4e7 Mar 15 12:51:53 Unraid kernel: ? inet_gro_receive+0x252/0x264 Mar 15 12:51:53 Unraid kernel: __netif_receive_skb_list_core+0x78/0x104 Mar 15 12:51:53 Unraid kernel: netif_receive_skb_list_internal+0x1bf/0x1f2 Mar 15 12:51:53 Unraid kernel: gro_normal_list+0x1d/0x39 Mar 15 12:51:53 Unraid kernel: napi_complete_done+0x79/0x104 Mar 15 12:51:53 Unraid kernel: tg3_poll_msix+0xb3/0x124 [tg3] Mar 15 12:51:53 Unraid kernel: net_rx_action+0xf4/0x29d Mar 15 12:51:53 Unraid kernel: __do_softirq+0xc4/0x1c2 Mar 15 12:51:53 Unraid kernel: asm_call_irq_on_stack+0x12/0x20 Mar 15 12:51:53 Unraid kernel: </IRQ> Mar 15 12:51:53 Unraid kernel: do_softirq_own_stack+0x2c/0x39 Mar 15 12:51:53 Unraid kernel: __irq_exit_rcu+0x45/0x80 Mar 15 12:51:53 Unraid kernel: common_interrupt+0x119/0x12e Mar 15 12:51:53 Unraid kernel: asm_common_interrupt+0x1e/0x40 Mar 15 12:51:53 Unraid kernel: RIP: 0010:arch_local_irq_enable+0x7/0x8 Mar 15 12:51:53 Unraid kernel: Code: 00 48 83 c4 28 4c 89 e0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 9c 58 0f 1f 44 00 00 c3 fa 66 0f 1f 44 00 00 c3 fb 66 0f 1f 44 00 00 <c3> 55 8b af 28 04 00 00 b8 01 00 00 00 45 31 c9 53 45 31 d2 39 c5 Mar 15 12:51:53 Unraid kernel: RSP: 0018:ffffffff82003e60 EFLAGS: 00000246 Mar 15 12:51:53 Unraid kernel: RAX: ffff88a7dfa22380 RBX: 0000000000000002 RCX: 000000000000001f Mar 15 12:51:53 Unraid kernel: RDX: 0000000000000000 RSI: 000000003555583f RDI: 0000000000000000 Mar 15 12:51:53 Unraid kernel: RBP: ffffe8ffffa3f200 R08: 000122c9255e7bb6 R09: 00000000000026fc Mar 15 12:51:53 Unraid kernel: R10: 0000000000002720 R11: 071c71c71c71c71c R12: 000122c9255e7bb6 Mar 15 12:51:53 Unraid kernel: R13: ffffffff820c7ec0 R14: 0000000000000002 R15: 0000000000000000 Mar 15 12:51:53 Unraid kernel: cpuidle_enter_state+0x101/0x1c4 Mar 15 12:51:53 Unraid kernel: cpuidle_enter+0x25/0x31 Mar 15 12:51:53 Unraid kernel: do_idle+0x1a6/0x214 Mar 15 12:51:53 Unraid kernel: cpu_startup_entry+0x18/0x1a Mar 15 12:51:53 Unraid kernel: start_kernel+0x4c0/0x4e3 Mar 15 12:51:53 Unraid kernel: secondary_startup_64_no_verify+0xb0/0xbb Mar 15 12:51:53 Unraid kernel: ---[ end trace 606bce8250f84b7e ]--- syslog unraid-diagnostics-20210315-2041.zip Quote Link to comment
Gabriel_B Posted July 5, 2021 Share Posted July 5, 2021 Same here, any solution? Quote Link to comment
BlackNL16 Posted July 5, 2021 Author Share Posted July 5, 2021 51 minutes ago, Gabriel_B said: Same here, any solution? unfortunately still no solution from my side. it still happens randomly. Quote Link to comment
JorgeB Posted July 5, 2021 Share Posted July 5, 2021 On 3/15/2021 at 8:14 PM, BlackNL16 said: nf_conntrack See if this applies to you: https://forums.unraid.net/topic/70529-650-call-traces-when-assigning-ip-address-to-docker-containers/ See also here: https://forums.unraid.net/bug-reports/stable-releases/690691-kernel-panic-due-to-netfilter-nf_nat_setup_info-docker-static-ip-macvlan-r1356/ Quote Link to comment
trurl Posted July 5, 2021 Share Posted July 5, 2021 1 hour ago, Gabriel_B said: Same here, any solution? You should start your own thread with your diagnostics. Quote Link to comment
bonienl Posted July 5, 2021 Share Posted July 5, 2021 On 3/15/2021 at 9:14 PM, BlackNL16 said: but I Suppose this belongs to the crash. No, it is not the cause of a crash. The call trace took place at 12:51, your system keeps running and reporting until 19:31 (7 hours later) One hour later at 20:36 the system restarts. Not clear why. Did you do anything? Perhaps unrelated but your syslog is littered with error messages from Pulseway, either remove it or fix it. Quote Link to comment
BlackNL16 Posted July 5, 2021 Author Share Posted July 5, 2021 6 hours ago, JorgeB said: See if this applies to you: https://forums.unraid.net/topic/70529-650-call-traces-when-assigning-ip-address-to-docker-containers/ See also here: https://forums.unraid.net/bug-reports/stable-releases/690691-kernel-panic-due-to-netfilter-nf_nat_setup_info-docker-static-ip-macvlan-r1356/ Thanks I just created an separate vlan for the dockers with a static ip address and migrated them toward it. let's see if this gives some stability again. Quote Link to comment
BlackNL16 Posted July 8, 2021 Author Share Posted July 8, 2021 (edited) Today unraid crashed again seems to me the issue today was with the ISCSI plugin?? so I deleted it. i attached the logs and diagnostics, maybe someone else sees something else which I cant find. unraid-crash-syslog.xlsx unraid-diagnostics-20210708-1912.zip Edited July 8, 2021 by BlackNL16 Quote Link to comment
JorgeB Posted July 8, 2021 Share Posted July 8, 2021 Call trace still seems related to the above. Quote Link to comment
BlackNL16 Posted July 8, 2021 Author Share Posted July 8, 2021 12 minutes ago, JorgeB said: Call trace still seems related to the above. so you still suspect it has something to do with mac vlan (docker network) despite I already moved all dockers with static IP address towards an separate vlan on my network. could it be the proxynet network which causes this crashes then? otherwise I will move them towards the other vlan also but with a dhcp server active then in that vlan. Quote Link to comment
JorgeB Posted July 8, 2021 Share Posted July 8, 2021 See this, it works for some: https://forums.unraid.net/bug-reports/stable-releases/690691-kernel-panic-due-to-netfilter-nf_nat_setup_info-docker-static-ip-macvlan-r1356/?do=findComment&comment=14530 Quote Link to comment
BlackNL16 Posted July 8, 2021 Author Share Posted July 8, 2021 (edited) 1 hour ago, JorgeB said: See this, it works for some: https://forums.unraid.net/bug-reports/stable-releases/690691-kernel-panic-due-to-netfilter-nf_nat_setup_info-docker-static-ip-macvlan-r1356/?do=findComment&comment=14530 Just moved all of my proxynet dockers towards the br0.208 network and also disabled host access to networks in docker settings. curious if this stay stable now. luckily I'm well equipped regarding router and switches XD. Edited July 8, 2021 by BlackNL16 Quote Link to comment
BlackNL16 Posted August 2, 2021 Author Share Posted August 2, 2021 Quick update: Since I disabled host access and moved all the bridged docker towards a separate vlan, I Haven't had a crash since then. Stable now for 25 Days. Still have a few of my dockers on host or default bridge, but this is running without any problems at the moment. @JorgeB, thanks for your suggestions and thoughts !!! 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.