[6.8.3] My Unraid server does this (crashes?) every so often, anyone know what's going on?

[Gbcue]

Last time, it took 20+ days, this time, 4-5-ish days. Can't access shares or the web interface.  I have to hard reset via IPMI and then parity check runs, 0 errors.

 

The dockers I have running are Plex, Locast2Plex, Glances, and binhex-krusader. No VMs. I wasn't even home when the server went weird.

 

I think it might have something to do with Locast2Plex (host with its own IP) but stopping it off for a long time didn't do anything.

 

Anyone have any insight?

 

Mobo:  Supermicro X9DRH-7TF

CPU:  2x Intel Xeon E5-2630 v2

RAM:  128gb DDR3 ECC

Connected to power via UPS.

The two interfaces are bonded via 802.3ad to a switch that supports it.

 

6 array disks (WD Reds) xfs, 2 parity (WD Reds), 2 cache (WD Black) btrfs.

Edited February 5, 2021 by Gbcue

[JorgeB]

You can try this and then post that log, as well the complete diagnostics.

[Gbcue]

23 hours ago, JorgeB said:

You can try this and then post that log, as well the complete diagnostics.

 

I enabled syslog, here's an entire snip of the kernel panic.  But the server is not crashed or hung up.  Nothing prompted me to grab this log, I was just curious what it said at this hour.  I also see the BTRFS critical errors, but no SMART errors on any drives.

 

 

syslog-10.0.10.x-5.txt

serverone-diagnostics-20210206-0039.zip

Edited February 6, 2021 by Gbcue

[JorgeB]

Btrfs errors are likely the result of the previous crashes, I don't know how critical that particular error is, but it might be a good idea to re-format the pool just in case.

 

Your main issue is likely related to the macvlan call traces, these are usually the result of having dockers with a custom IP address, and may end up crashing Unraid, see below for more info.

 

[Gbcue]

Thanks, it seemed like setting a VLAN with custom IPs solved my issue.  Can you point me to more info on reformatting my cache pool?

 

 

[JorgeB]

With the array stopped use wipefs on both cache devices, first the partition then the device (all data there will be lost):

 

wipefs -a /dev/sdX1

then

wipefs -a /dev/sdX

 

Start array and there will be an option to format cache.

[Gbcue]

Thanks, I got my cache to a single drive since I didn't need a pool.  I moved my dockers to their own VLAN and they all have custom IPs.

 

But I just got this error in the syslog:

 

Feb 17 21:47:56 ServerOne kernel: WARNING: CPU: 0 PID: 151 at net/netfilter/nf_conntrack_core.c:945 __nf_conntrack_confirm+0xa0/0x69e
Feb 17 21:47:56 ServerOne kernel: Modules linked in: ccp xt_nat macvlan ipt_MASQUERADE iptable_filter iptable_nat nf_nat_ipv4 nf_nat ip_tables xfs md_mod ipmi_devintf ixgbe(O) sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp ipmi_ssif kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd mpt3sas glue_helper isci libsas intel_cstate ahci i2c_i801 intel_uncore i2c_core libahci raid_class intel_rapl_perf scsi_transport_sas pcc_cpufreq wmi ipmi_si button [last unloaded: ixgbe]
Feb 17 21:47:56 ServerOne kernel: CPU: 0 PID: 151 Comm: kworker/0:2 Tainted: G           O      4.19.107-Unraid #1
Feb 17 21:47:56 ServerOne kernel: Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.3 07/13/2018
Feb 17 21:47:56 ServerOne kernel: Workqueue: events macvlan_process_broadcast [macvlan]
Feb 17 21:47:56 ServerOne kernel: RIP: 0010:__nf_conntrack_confirm+0xa0/0x69e
Feb 17 21:47:56 ServerOne kernel: Code: 04 e8 56 fb ff ff 44 89 f2 44 89 ff 89 c6 41 89 c4 e8 7f f9 ff ff 48 8b 4c 24 08 84 c0 75 af 48 8b 85 80 00 00 00 a8 08 74 26 <0f> 0b 44 89 e6 44 89 ff 45 31 f6 e8 95 f1 ff ff be 00 02 00 00 48
Feb 17 21:47:56 ServerOne kernel: RSP: 0018:ffff88903f803d58 EFLAGS: 00010202
Feb 17 21:47:56 ServerOne kernel: RAX: 0000000000000188 RBX: ffff88903b55ef00 RCX: ffff888fd5eb1318
Feb 17 21:47:56 ServerOne kernel: RDX: 0000000000000001 RSI: 0000000000000017 RDI: ffffffff81e0865c
Feb 17 21:47:56 ServerOne kernel: RBP: ffff888fd5eb12c0 R08: 00000000e1f538c4 R09: ffffffff81c8aa80
Feb 17 21:47:56 ServerOne kernel: R10: 0000000000000158 R11: ffffffff81e91080 R12: 0000000000000c17
Feb 17 21:47:56 ServerOne kernel: R13: ffffffff81e91080 R14: 0000000000000000 R15: 000000000000c80d
Feb 17 21:47:56 ServerOne kernel: FS:  0000000000000000(0000) GS:ffff88903f800000(0000) knlGS:0000000000000000
Feb 17 21:47:56 ServerOne kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 17 21:47:56 ServerOne kernel: CR2: 00001516f289a718 CR3: 0000000001e0a004 CR4: 00000000001606f0
Feb 17 21:47:56 ServerOne kernel: Call Trace:
Feb 17 21:47:56 ServerOne kernel: <IRQ>
Feb 17 21:47:56 ServerOne kernel: ipv4_confirm+0xaf/0xb9
Feb 17 21:47:56 ServerOne kernel: nf_hook_slow+0x3a/0x90
Feb 17 21:47:56 ServerOne kernel: ip_local_deliver+0xad/0xdc
Feb 17 21:47:56 ServerOne kernel: ? ip_sublist_rcv_finish+0x54/0x54
Feb 17 21:47:56 ServerOne kernel: ip_sabotage_in+0x38/0x3e
Feb 17 21:47:56 ServerOne kernel: nf_hook_slow+0x3a/0x90
Feb 17 21:47:56 ServerOne kernel: ip_rcv+0x8e/0xbe
Feb 17 21:47:56 ServerOne kernel: ? ip_rcv_finish_core.isra.0+0x2e1/0x2e1
Feb 17 21:47:56 ServerOne kernel: __netif_receive_skb_one_core+0x53/0x6f
Feb 17 21:47:56 ServerOne kernel: process_backlog+0x77/0x10e
Feb 17 21:47:56 ServerOne kernel: net_rx_action+0x107/0x26c
Feb 17 21:47:56 ServerOne kernel: __do_softirq+0xc9/0x1d7
Feb 17 21:47:56 ServerOne kernel: do_softirq_own_stack+0x2a/0x40
Feb 17 21:47:56 ServerOne kernel: </IRQ>
Feb 17 21:47:56 ServerOne kernel: do_softirq+0x4d/0x5a
Feb 17 21:47:56 ServerOne kernel: netif_rx_ni+0x1c/0x22
Feb 17 21:47:56 ServerOne kernel: macvlan_broadcast+0x111/0x156 [macvlan]
Feb 17 21:47:56 ServerOne kernel: ? __switch_to_asm+0x41/0x70
Feb 17 21:47:56 ServerOne kernel: macvlan_process_broadcast+0xea/0x128 [macvlan]
Feb 17 21:47:56 ServerOne kernel: process_one_work+0x16e/0x24f
Feb 17 21:47:56 ServerOne kernel: worker_thread+0x1e2/0x2b8
Feb 17 21:47:56 ServerOne kernel: ? rescuer_thread+0x2a7/0x2a7
Feb 17 21:47:56 ServerOne kernel: kthread+0x10c/0x114
Feb 17 21:47:56 ServerOne kernel: ? kthread_park+0x89/0x89
Feb 17 21:47:56 ServerOne kernel: ret_from_fork+0x35/0x40
Feb 17 21:47:56 ServerOne kernel: ---[ end trace e6dff886c5bb9d83 ]---

 

But no server hangs, dockers are still working as normal.

[JorgeB]

Macvlan call traces are usually the result of having dockers with a custom IP address, more info below.

 

https://forums.unraid.net/topic/70529-650-call-traces-when-assigning-ip-address-to-docker-containers/