Still Unraid randomly chrashed

[corgan]

Hello

 

my Unraid Server randomly crashed with a strange error message, i dont really understand.

The error happend around every 3-5 days, or like today, 2 times a day. I cant really reproduce it.

So please help me to understand where the error is comming from and what i could do about it.

 

Thanks!

 

Running Unraid 6.8.3

 

Network:

The server is connected with 2x 10gb on a normal switch

Router: pfSense @ Dell RS210 II

WiFi: Asus RT-AX88U + Lyra

 

I have some VMs configured, but not running.

 

Running Dockers:

Gitlab-CE

hddtemp2influx

JD

mariaDB

NginxProxy

phpmyadmin

telegraf

zabbix-agent

zerotier

 

Hardware

MB: ASRock X470D4U2-2T

CPU: AMD Ryzen 7 3700X

GPU1: Nvidia GeForce RTX 2060 @ NVMe Slot2 on AST1150 PCI-to-PCI Bridge

GPU2: Radeon RX 570 @ PCIe x8

Onboard 2x 10gb NIC

 

Array

HBA: LSI SAS2308 PCI-Express Fusion-MPT SAS-2 @ PCIe x8

8x16TB HDD

 

Cache:

Onboard  400 Series Chipset SATA Controller ASMedia Technology

5x 500gb Sandisk SSD

 

UD:

1tb nvme0n1 - SSD 970 EVO

 

PCIe ACS override active

 

Thinks i allready tryed:

 

- I added rcu_nocbs=0-15 to the boot option.

 

 

 

Quote

Apr 23 07:43:41 serva4 kernel: WARNING: CPU: 9 PID: 0 at net/netfilter/nf_conntrack_core.c:945 __nf_conntrack_confirm+0xa0/0x69e
Apr 23 07:43:41 serva4 kernel: Modules linked in: arc4 ecb md4 sha512_ssse3 sha512_generic cmac cifs ccm xt_CHECKSUM ipt_REJECT ip6table_mangle ip6table_nat nf_nat_ipv6 iptable_mangle ip6table_filter ip6_tables vhost_net tun vhost tap veth macvlan xt_nat ipt_MASQUERADE iptable_filter iptable_nat nf_nat_ipv4 nf_nat ip_tables xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid k10temp bonding ixgbe(O) edac_mce_amd kvm_amd ipmi_ssif kvm mpt3sas crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel raid_class scsi_transport_sas aes_x86_64 crypto_simd cryptd i2c_piix4 i2c_core ahci libahci nvme glue_helper wmi_bmof ccp nvme_core wmi pcc_cpufreq button ipmi_si acpi_cpufreq [last unloaded: ixgbe]
Apr 23 07:43:41 serva4 kernel: CPU: 9 PID: 0 Comm: swapper/9 Tainted: G           O      4.19.107-Unraid #1
Apr 23 07:43:41 serva4 kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470D4U2-2T, BIOS P3.30 10/03/2019
Apr 23 07:43:41 serva4 kernel: RIP: 0010:__nf_conntrack_confirm+0xa0/0x69e
Apr 23 07:43:41 serva4 kernel: Code: 04 e8 56 fb ff ff 44 89 f2 44 89 ff 89 c6 41 89 c4 e8 7f f9 ff ff 48 8b 4c 24 08 84 c0 75 af 48 8b 85 80 00 00 00 a8 08 74 26 <0f> 0b 44 89 e6 44 89 ff 45 31 f6 e8 95 f1 ff ff be 00 02 00 00 48
Apr 23 07:43:41 serva4 kernel: RSP: 0018:ffff88881e8438d0 EFLAGS: 00010202
Apr 23 07:43:41 serva4 kernel: RAX: 0000000000000188 RBX: ffff888367770600 RCX: ffff888372663898
Apr 23 07:43:41 serva4 kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffffff81e095fc
Apr 23 07:43:41 serva4 kernel: RBP: ffff888372663840 R08: 00000000dc2d73a9 R09: ffffffff81c8aa80
Apr 23 07:43:41 serva4 kernel: R10: 0000000000000158 R11: ffffffff81e91080 R12: 000000000000ebae
Apr 23 07:43:41 serva4 kernel: R13: ffffffff81e91080 R14: 0000000000000000 R15: 0000000000003fff
Apr 23 07:43:41 serva4 kernel: FS:  0000000000000000(0000) GS:ffff88881e840000(0000) knlGS:0000000000000000
Apr 23 07:43:41 serva4 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 23 07:43:41 serva4 kernel: CR2: 00000000004b70c8 CR3: 0000000371c62000 CR4: 0000000000340ee0
Apr 23 07:43:41 serva4 kernel: Call Trace:
Apr 23 07:43:41 serva4 kernel: <IRQ>
Apr 23 07:43:41 serva4 kernel: ipv4_confirm+0xaf/0xb9
Apr 23 07:43:41 serva4 kernel: nf_hook_slow+0x3a/0x90
Apr 23 07:43:41 serva4 kernel: ip_local_deliver+0xad/0xdc
Apr 23 07:43:41 serva4 kernel: ? ip_sublist_rcv_finish+0x54/0x54
Apr 23 07:43:41 serva4 kernel: ip_sabotage_in+0x38/0x3e
Apr 23 07:43:41 serva4 kernel: nf_hook_slow+0x3a/0x90
Apr 23 07:43:41 serva4 kernel: ip_rcv+0x8e/0xbe
Apr 23 07:43:41 serva4 kernel: ? ip_rcv_finish_core.isra.0+0x2e1/0x2e1
Apr 23 07:43:41 serva4 kernel: __netif_receive_skb_one_core+0x53/0x6f
Apr 23 07:43:41 serva4 kernel: netif_receive_skb_internal+0x79/0x94
Apr 23 07:43:41 serva4 kernel: br_pass_frame_up+0x128/0x14a
Apr 23 07:43:41 serva4 kernel: ? br_port_flags_change+0x29/0x29
Apr 23 07:43:41 serva4 kernel: br_handle_frame_finish+0x342/0x383
Apr 23 07:43:41 serva4 kernel: ? br_pass_frame_up+0x14a/0x14a
Apr 23 07:43:41 serva4 kernel: br_nf_hook_thresh+0xa3/0xc3
Apr 23 07:43:41 serva4 kernel: ? br_pass_frame_up+0x14a/0x14a
Apr 23 07:43:41 serva4 kernel: br_nf_pre_routing_finish+0x24a/0x271
Apr 23 07:43:41 serva4 kernel: ? br_pass_frame_up+0x14a/0x14a
Apr 23 07:43:41 serva4 kernel: ? br_handle_local_finish+0xe/0xe
Apr 23 07:43:41 serva4 kernel: ? nf_nat_ipv4_in+0x1e/0x62 [nf_nat_ipv4]
Apr 23 07:43:41 serva4 kernel: ? br_handle_local_finish+0xe/0xe
Apr 23 07:43:41 serva4 kernel: br_nf_pre_routing+0x31c/0x343
Apr 23 07:43:41 serva4 kernel: ? br_nf_forward_ip+0x362/0x362
Apr 23 07:43:41 serva4 kernel: nf_hook_slow+0x3a/0x90
Apr 23 07:43:41 serva4 kernel: br_handle_frame+0x27e/0x2bd
Apr 23 07:43:41 serva4 kernel: ? br_pass_frame_up+0x14a/0x14a
Apr 23 07:43:41 serva4 kernel: __netif_receive_skb_core+0x4a7/0x7b1
Apr 23 07:43:41 serva4 kernel: ? udp_gro_receive+0x4b/0x136
Apr 23 07:43:41 serva4 kernel: __netif_receive_skb_one_core+0x35/0x6f
Apr 23 07:43:41 serva4 kernel: netif_receive_skb_internal+0x79/0x94
Apr 23 07:43:41 serva4 kernel: napi_gro_receive+0x44/0x7b
Apr 23 07:43:41 serva4 kernel: ixgbe_poll+0xb97/0xce4 [ixgbe]
Apr 23 07:43:41 serva4 kernel: ? common_interrupt+0xf/0xf
Apr 23 07:43:41 serva4 kernel: net_rx_action+0x107/0x26c
Apr 23 07:43:41 serva4 kernel: __do_softirq+0xc9/0x1d7
Apr 23 07:43:41 serva4 kernel: irq_exit+0x5e/0x9d
Apr 23 07:43:41 serva4 kernel: do_IRQ+0xb2/0xd0
Apr 23 07:43:41 serva4 kernel: common_interrupt+0xf/0xf
Apr 23 07:43:41 serva4 kernel: </IRQ>
Apr 23 07:43:41 serva4 kernel: RIP: 0010:cpuidle_enter_state+0xe8/0x141
Apr 23 07:43:41 serva4 kernel: Code: ff 45 84 f6 74 1d 9c 58 0f 1f 44 00 00 0f ba e0 09 73 09 0f 0b fa 66 0f 1f 44 00 00 31 ff e8 7a 8d bb ff fb 66 0f 1f 44 00 00 <48> 2b 2c 24 b8 ff ff ff 7f 48 b9 ff ff ff ff f3 01 00 00 48 39 cd
Apr 23 07:43:41 serva4 kernel: RSP: 0018:ffffc9000328be98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd8
Apr 23 07:43:41 serva4 kernel: RAX: ffff88881e85fac0 RBX: ffff88881811b000 RCX: 000000000000001f
Apr 23 07:43:41 serva4 kernel: RDX: 0000000000000000 RSI: 00000000239f5376 RDI: 0000000000000000
Apr 23 07:43:41 serva4 kernel: RBP: 0000118084500fcb R08: 0000118084500fcb R09: 0000000000000dd6
Apr 23 07:43:41 serva4 kernel: R10: 00000000043c2768 R11: 071c71c71c71c71c R12: 0000000000000002
Apr 23 07:43:41 serva4 kernel: R13: ffffffff81e5e1e0 R14: 0000000000000000 R15: ffffffff81e5e2b8
Apr 23 07:43:41 serva4 kernel: ? cpuidle_enter_state+0xbf/0x141
Apr 23 07:43:41 serva4 kernel: do_idle+0x17e/0x1fc
Apr 23 07:43:41 serva4 kernel: cpu_startup_entry+0x6a/0x6c
Apr 23 07:43:41 serva4 kernel: start_secondary+0x197/0x1b2
Apr 23 07:43:41 serva4 kernel: secondary_startup_64+0xa4/0xb0
Apr 23 07:43:41 serva4 kernel: ---[ end trace 608a67040b615e17 ]---

 

serva4-diagnostics-20200424-1247.zip

[JorgeB]

Start here.

[corgan]

Thanks for the reply.

I allready did all things what mentioned in this thread.

1. added rcu_nocbs=0-15 to the boot option.
2. Bios: set Power Supply Idle Control to Typical
3. Bios: C6 Enabled (there was some missleading comments at first, but i think turning off was not correct?)
4. Latest Bios

 

So any other tips what i can do?

 

Edited April 24, 2020 by corgan