Crashing every 24-30 hours

[zarklin]

Please forgive me if I've overlooked something. Only 15 days into my trial key and very new to server side. After the previous crash I ran memtest for around 10 hours and did not show any errors. Durning this crash I was able to access the unifi docker (didn't think to try Emby) so not sure if it was a full crash. I could not access web or run a diagnostics so that was ran post crash. Additional Syslog is from the syslog server share i was still able to access  this time before restarting.

 

I'm assuming I misconfigured something as the system ran well for the first week and only in the last few days has been crashing every 24-30 hours it seems. Kennel Panic and RIP: 0010:kmem_cache_alloc+0x8d/0x130 is what I've been able to see from the IPMI preview and log. Not sure if the 504 timeouts when trying to access the web portal are related. Wondering if it may be a problem with my cache drive. I unfotunatley made a lot of changes in the days before this started. Installed a cache drive, moved appdata, setup pihole which I disabled after the previous crash and repointed the router dns to a public one to see if that was the problem.

 

Thank you in advance. Details below.

 

No VMs.

 

Installed dockers :

Binhex Krusader - was not started

Emby - was running

Unifi - was running

PiHole - wasn not started

 

Plugins :

CA Auto Update Applications, CA Bacup / Restore Appdata, CA Cleanup Appdata, Community Applications, Dynamix SSD TRIM, Fix Common Problems, Preclear Disks, Unassigned Devicves, Unassigned Devices Plus, VM Backup

 

System info :

 

M/B: Supermicro X9DRD-7LN4F Version REV:1.02

BIOS: American Megatrends Inc. Version 3.3. Dated: 08/23/2018

CPU: Intel® Xeon® CPU E5-2620 0 @ 2.00GHz

HVM: Enabled

IOMMU: Enabled

Cache: 384 KiB, 1536 KiB, 15 MB, 384 KiB, 1536 KiB, 15 MB

Memory: 32 GiB DDR3 Multi-bit ECC (max. installable capacity 1024 GiB)

Network: bond0: fault-tolerance (active-backup), mtu 1500
 eth0: 1000 Mbps, full duplex, mtu 1500
 eth1: interface down
 eth2: interface down
 eth3: interface down

Kernel: Linux 5.10.28-Unraid x86_64

OpenSSL: 1.1.1j

moria-diagnostics-20211108-2352.zip syslog-10.214.152.15.log

[ChatNoir]

1 hour ago, zarklin said:

Installed dockers :

Binhex Krusader - was not started

Emby - was running

Unifi - was running

PiHole - wasn not started

 

Nov  6 11:45:48 Moria kernel: ------------[ cut here ]------------
Nov  6 11:45:48 Moria kernel: WARNING: CPU: 2 PID: 26556 at net/netfilter/nf_conntrack_core.c:1120 __nf_conntrack_confirm+0x9b/0x1e6 [nf_conntrack]
Nov  6 11:45:48 Moria kernel: Modules linked in: xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle nf_tables vhost_net tun vhost vhost_iotlb tap xt_nat xt_tcpudp macvlan xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs md_mod ip6table_filter ip6_tables iptable_filter ip_tables x_tables bonding igb i2c_algo_bit sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper rapl intel_cstate mpt3sas ahci i2c_i801 intel_uncore i2c_smbus input_leds i2c_core libahci raid_class led_class acpi_ipmi scsi_transport_sas wmi ipmi_si button acpi_pad [last unloaded: i2c_algo_bit]
Nov  6 11:45:48 Moria kernel: CPU: 2 PID: 26556 Comm: kworker/2:2 Not tainted 5.10.28-Unraid #1
Nov  6 11:45:48 Moria kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.3 08/23/2018
Nov  6 11:45:48 Moria kernel: Workqueue: events macvlan_process_broadcast [macvlan]
Nov  6 11:45:48 Moria kernel: RIP: 0010:__nf_conntrack_confirm+0x9b/0x1e6 [nf_conntrack]
Nov  6 11:45:48 Moria kernel: Code: e8 dc f8 ff ff 44 89 fa 89 c6 41 89 c4 48 c1 eb 20 89 df 41 89 de e8 36 f6 ff ff 84 c0 75 bb 48 8b 85 80 00 00 00 a8 08 74 18 <0f> 0b 89 df 44 89 e6 31 db e8 6d f3 ff ff e8 35 f5 ff ff e9 22 01
Nov  6 11:45:48 Moria kernel: RSP: 0018:ffffc90003380dd8 EFLAGS: 00010202
Nov  6 11:45:48 Moria kernel: RAX: 0000000000000188 RBX: 0000000000003a60 RCX: 000000002605bc03
Nov  6 11:45:48 Moria kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffa0123f48
Nov  6 11:45:48 Moria kernel: RBP: ffff888164e8c500 R08: 00000000d6b7bd8e R09: 0000000000000000
Nov  6 11:45:48 Moria kernel: R10: 0000000000000098 R11: ffff88811b862800 R12: 000000000000f352
Nov  6 11:45:48 Moria kernel: R13: ffffffff8210b440 R14: 0000000000003a60 R15: 0000000000000000
Nov  6 11:45:48 Moria kernel: FS:  0000000000000000(0000) GS:ffff88846fa80000(0000) knlGS:0000000000000000
Nov  6 11:45:48 Moria kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov  6 11:45:48 Moria kernel: CR2: 0000145982658080 CR3: 000000000200a006 CR4: 00000000000606e0
Nov  6 11:45:48 Moria kernel: Call Trace:
Nov  6 11:45:48 Moria kernel: <IRQ>
Nov  6 11:45:48 Moria kernel: nf_conntrack_confirm+0x2f/0x36 [nf_conntrack]
Nov  6 11:45:48 Moria kernel: nf_hook_slow+0x39/0x8e
Nov  6 11:45:48 Moria kernel: nf_hook.constprop.0+0xb1/0xd8
Nov  6 11:45:48 Moria kernel: ? ip_protocol_deliver_rcu+0xfe/0xfe
Nov  6 11:45:48 Moria kernel: ip_local_deliver+0x49/0x75
Nov  6 11:45:48 Moria kernel: __netif_receive_skb_one_core+0x74/0x95
Nov  6 11:45:48 Moria kernel: process_backlog+0xa3/0x13b
Nov  6 11:45:48 Moria kernel: net_rx_action+0xf4/0x29d
Nov  6 11:45:48 Moria kernel: __do_softirq+0xc4/0x1c2
Nov  6 11:45:48 Moria kernel: asm_call_irq_on_stack+0x12/0x20
Nov  6 11:45:48 Moria kernel: </IRQ>
Nov  6 11:45:48 Moria kernel: do_softirq_own_stack+0x2c/0x39
Nov  6 11:45:48 Moria kernel: do_softirq+0x3a/0x44
Nov  6 11:45:48 Moria kernel: netif_rx_ni+0x1c/0x22
Nov  6 11:45:48 Moria kernel: macvlan_broadcast+0x10e/0x13c [macvlan]
Nov  6 11:45:48 Moria kernel: macvlan_process_broadcast+0xf8/0x143 [macvlan]
Nov  6 11:45:48 Moria kernel: process_one_work+0x13c/0x1d5
Nov  6 11:45:48 Moria kernel: worker_thread+0x18b/0x22f
Nov  6 11:45:48 Moria kernel: ? process_scheduled_works+0x27/0x27
Nov  6 11:45:48 Moria kernel: kthread+0xe5/0xea
Nov  6 11:45:48 Moria kernel: ? __kthread_bind_mask+0x57/0x57
Nov  6 11:45:48 Moria kernel: ret_from_fork+0x22/0x30
Nov  6 11:45:48 Moria kernel: ---[ end trace 1f1b68cca18c7de8 ]---

 

Do you have custom IP address on some of your dockers ?

[JorgeB]

Macvlan call traces are usually the result of having dockers with a custom IP address, upgrading to v6.10 and switching to ipvlan might fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enable, top right)), or see below for more info.

https://forums.unraid.net/topic/70529-650-call-traces-when-assigning-ip-address-to-docker-containers/

See also here:

https://forums.unraid.net/bug-reports/stable-releases/690691-kernel-panic-due-to-netfilter-nf_nat_setup_info-docker-static-ip-macvlan-r1356/

[zarklin]

Thanks for the info. I do have individual IPs for the dockers. I'll take a look through the link and see if I can get the Vlan setup. My home network is flat right now and the server is hanging off a dumb switch so may need to do some rearanaging