Unraid randomly chrashed

[corgan]

Hello

 

my Unraid Server randomly crashed with a strange error message, i dont really understand.

The error happend around every 3-5 days, or like today, 2 times a day. I cant really reproduce it.

This error did not happend if i let the parity check running (for 23h) after a chrash.

 

The biggest problem is, that my complete Network is stop working at all. I cant even ping my router.

This happend on all >30 Devices in the Network till i hard reset the server. After reset the pings and the network is working again instantly.

 

So please help me to understand where the error is comming from and what i could do about it.

 

Thanks!

 

 

Running Unraid 6.8.3 (Nvidia 6.6.7 - 440,59)

 

Network:

The server is connected with 2x 10gb on a normal switch

Router: pfSense @ Dell RS210 II

WiFi: Asus RT-AX88U + Lyra

 

I have some VMs configured, but not running.

 

Running Dockers:

• Gitlab-CE
• hddtemp2influx
• JD
• mariaDB
• NginxProxy
• phpmyadmin
• telegraf
• zabbix-agent
• zerotier

 

Hardware

MB: ASRock X470D4U2-2T

CPU: AMD Ryzen 7 3700X

GPU1: Nvidia GeForce RTX 2060 @ NVMe Slot2 on AST1150 PCI-to-PCI Bridge

GPU2: Radeon RX 570 @ PCIe x8

Onboard 2x 10gb NIC

 

Array

HBA: LSI SAS2308 PCI-Express Fusion-MPT SAS-2 @ PCIe x8

8x16TB HDD

 

Cache:

Onboard  400 Series Chipset SATA Controller ASMedia Technology

5x 500gb Sandisk SSD

 

UD:

1tb nvme0n1 - SSD 970 EVO

 

PCIe ACS override active

 

error syslog

Quote

    : WARNING: CPU: 7 PID: 0 at net/netfilter/nf_conntrack_core.c:945 __nf_conntrack_confirm+0xa0/0x69e
    : Modules linked in: xt_CHECKSUM ipt_REJECT ip6table_mangle ip6table_nat nf_nat_ipv6 iptable_mangle ip6table_filter ip6_tables vhost_net tun vhost tap veth macvlan xt_nat ipt_MASQUERADE iptable_filter iptable_nat nf_nat_ipv4 nf_nat ip_tables xfs nfsd lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlua(PO) ipmi_devintf nct6775 hwmon_vid k10temp ixgbe(O) rsnvme(PO) edac_mce_amd crc32_pclmul pcbc aesni_intel aes_x86_64 glue_helper crypto_simd ghash_clmulni_intel cryptd kvm_amd kvm nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) drm_kms_helper drm mpt3sas syscopyarea sysfillrect sysimgblt fb_sys_fops raid_class scsi_transport_sas ipmi_si ahci agpgart libahci ipmi_ssif i2c_piix4 i2c_core wmi_bmof nvme ccp nvme_core wmi pcc_cpufreq crct10dif_pclmul
    : button crc32c_intel acpi_cpufreq [last unloaded: ixgbe]
    : CPU: 7 PID: 0 Comm: swapper/7 Tainted: P           O      4.19.107-Unraid #1
    : Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470D4U2-2T, BIOS P3.30 10/03/2019
    : RIP: 0010:__nf_conntrack_confirm+0xa0/0x69e
    : Code: 04 e8 56 fb ff ff 44 89 f2 44 89 ff 89 c6 41 89 c4 e8 7f f9 ff ff 48 8b 4c 24 08 84 c0 75 af 48 8b 85 80 00 00 00 a8 08 74 26 <0f> 0b 44 89 e6 44 89 ff 45 31 f6 e8 95 f1 ff ff be 00 02 00 00 48
    : RSP: 0018:ffff88881e7c38d0 EFLAGS: 00010202
    : RAX: 0000000000000188 RBX: ffff8884bc12d900 RCX: ffff8883ef863118
    : RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffffff81e08eec
    : RBP: ffff8883ef8630c0 R08: 00000000ad600214 R09: ffffffff81c8aa80
    : R10: 0000000000000158 R11: ffffffff81e91080 R12: 000000000000fcc8
    : R13: ffffffff81e91080 R14: 0000000000000000 R15: 0000000000006e3b
    : FS:  0000000000000000(0000) GS:ffff88881e7c0000(0000) knlGS:0000000000000000
    : CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    : CR2: 00007f64e86b4000 CR3: 0000000733ebc000 CR4: 0000000000340ee0
    : Call Trace:
    : <IRQ>
    : ipv4_confirm+0xaf/0xb9
    : nf_hook_slow+0x3a/0x90
    : ip_local_deliver+0xad/0xdc
    : ? ip_sublist_rcv_finish+0x54/0x54
    : ip_sabotage_in+0x38/0x3e
    : nf_hook_slow+0x3a/0x90
    : ip_rcv+0x8e/0xbe
    : ? ip_rcv_finish_core.isra.0+0x2e1/0x2e1
    : __netif_receive_skb_one_core+0x53/0x6f
    : netif_receive_skb_internal+0x79/0x94
    : br_pass_frame_up+0x128/0x14a
    : ? br_port_flags_change+0x29/0x29
    : br_handle_frame_finish+0x342/0x383
    : ? br_pass_frame_up+0x14a/0x14a
    : br_nf_hook_thresh+0xa3/0xc3
    : ? br_pass_frame_up+0x14a/0x14a
    : br_nf_pre_routing_finish+0x24a/0x271
    : ? br_pass_frame_up+0x14a/0x14a
    : ? br_handle_local_finish+0xe/0xe
    : ? nf_nat_ipv4_in+0x1e/0x62 [nf_nat_ipv4]
    : ? br_handle_local_finish+0xe/0xe
    : br_nf_pre_routing+0x31c/0x343
    : ? br_nf_forward_ip+0x362/0x362
    : nf_hook_slow+0x3a/0x90
    : br_handle_frame+0x27e/0x2bd
    : ? br_pass_frame_up+0x14a/0x14a
    : __netif_receive_skb_core+0x4a7/0x7b1
    : ? udp_gro_receive+0x4b/0x136
    : __netif_receive_skb_one_core+0x35/0x6f
    : netif_receive_skb_internal+0x79/0x94
    : napi_gro_receive+0x44/0x7b
    : ixgbe_poll+0xb97/0xce4 [ixgbe]
    : ? common_interrupt+0xf/0xf
    : net_rx_action+0x107/0x26c
    : __do_softirq+0xc9/0x1d7
    : irq_exit+0x5e/0x9d
    : do_IRQ+0xb2/0xd0
    : common_interrupt+0xf/0xf
    : </IRQ>
    : RIP: 0010:native_safe_halt+0xe/0xf
    : Code: 60 02 df f0 83 44 24 fc 00 48 8b 00 a8 08 74 0b 65 81 25 55 a9 98 7e ff ff ff 7f c3 e9 07 00 00 00 0f 00 2d fa c4 57 00 fb f4 <c3> e9 07 00 00 00 0f 00 2d eb c4 57 00 f4 c3 53 e8 44 0f a1 ff 65
    : RSP: 0018:ffffc9000326bec8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd8
    : RAX: 0000000000000007 RBX: ffff88881afe0000 RCX: ffffc9000c94fe50
    : RDX: 00000000546796ba RSI: 0000000000000083 RDI: ffff88881e7e07c0
    : RBP: 0000000000000000 R08: ffff88881e7dd000 R09: 0000000000000000
    : R10: 0000000000000002 R11: ffff88881e7dfb40 R12: 0000000000000000
    : R13: 0000000000000007 R14: 0000000000000000 R15: 0000000000000000
    : default_idle+0xc/0x16
    : do_idle+0xdb/0x1fc
    : ? do_idle+0x1f7/0x1fc
    : cpu_startup_entry+0x6a/0x6c
    : start_secondary+0x197/0x1b2
    : secondary_startup_64+0xa4/0xb0
    : ---[ end trace 6453ee61aca24421 ]---

 

serva4-diagnostics-20200402-0329.zip

Edited April 2, 2020 by corgan

[JorgeB]

This might help with the crashing.

[corgan]

thanks for the tip.

I added rcu_nocbs=0-15 to the boot option. Hope this will help.