[6.8.3] My Unraid server does this (crashes?) every so often, anyone know what's going on?


Recommended Posts

Last time, it took 20+ days, this time, 4-5-ish days. Can't access shares or the web interface.  I have to hard reset via IPMI and then parity check runs, 0 errors.

 

The dockers I have running are Plex, Locast2Plex, Glances, and binhex-krusader. No VMs. I wasn't even home when the server went weird.

 

I think it might have something to do with Locast2Plex (host with its own IP) but stopping it off for a long time didn't do anything.

 

Anyone have any insight?

 

Mobo:  Supermicro X9DRH-7TF

CPU:  2x Intel Xeon E5-2630 v2

RAM:  128gb DDR3 ECC

Connected to power via UPS.

The two interfaces are bonded via 802.3ad to a switch that supports it.

 

6 array disks (WD Reds) xfs, 2 parity (WD Reds), 2 cache (WD Black) btrfs.

unknown.png

Edited by Gbcue
Link to comment
23 hours ago, JorgeB said:

You can try this and then post that log, as well the complete diagnostics.

 

I enabled syslog, here's an entire snip of the kernel panic.  But the server is not crashed or hung up.  Nothing prompted me to grab this log, I was just curious what it said at this hour.  I also see the BTRFS critical errors, but no SMART errors on any drives.

 

 

syslog-10.0.10.x-5.txt

serverone-diagnostics-20210206-0039.zip

Edited by Gbcue
Link to comment

Btrfs errors are likely the result of the previous crashes, I don't know how critical that particular error is, but it might be a good idea to re-format the pool just in case.

 

Your main issue is likely related to the macvlan call traces, these are usually the result of having dockers with a custom IP address, and may end up crashing Unraid, see below for more info.

 

Link to comment
  • 2 weeks later...

Thanks, I got my cache to a single drive since I didn't need a pool.  I moved my dockers to their own VLAN and they all have custom IPs.

 

But I just got this error in the syslog:

 

Feb 17 21:47:56 ServerOne kernel: WARNING: CPU: 0 PID: 151 at net/netfilter/nf_conntrack_core.c:945 __nf_conntrack_confirm+0xa0/0x69e
Feb 17 21:47:56 ServerOne kernel: Modules linked in: ccp xt_nat macvlan ipt_MASQUERADE iptable_filter iptable_nat nf_nat_ipv4 nf_nat ip_tables xfs md_mod ipmi_devintf ixgbe(O) sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp ipmi_ssif kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd mpt3sas glue_helper isci libsas intel_cstate ahci i2c_i801 intel_uncore i2c_core libahci raid_class intel_rapl_perf scsi_transport_sas pcc_cpufreq wmi ipmi_si button [last unloaded: ixgbe]
Feb 17 21:47:56 ServerOne kernel: CPU: 0 PID: 151 Comm: kworker/0:2 Tainted: G           O      4.19.107-Unraid #1
Feb 17 21:47:56 ServerOne kernel: Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.3 07/13/2018
Feb 17 21:47:56 ServerOne kernel: Workqueue: events macvlan_process_broadcast [macvlan]
Feb 17 21:47:56 ServerOne kernel: RIP: 0010:__nf_conntrack_confirm+0xa0/0x69e
Feb 17 21:47:56 ServerOne kernel: Code: 04 e8 56 fb ff ff 44 89 f2 44 89 ff 89 c6 41 89 c4 e8 7f f9 ff ff 48 8b 4c 24 08 84 c0 75 af 48 8b 85 80 00 00 00 a8 08 74 26 <0f> 0b 44 89 e6 44 89 ff 45 31 f6 e8 95 f1 ff ff be 00 02 00 00 48
Feb 17 21:47:56 ServerOne kernel: RSP: 0018:ffff88903f803d58 EFLAGS: 00010202
Feb 17 21:47:56 ServerOne kernel: RAX: 0000000000000188 RBX: ffff88903b55ef00 RCX: ffff888fd5eb1318
Feb 17 21:47:56 ServerOne kernel: RDX: 0000000000000001 RSI: 0000000000000017 RDI: ffffffff81e0865c
Feb 17 21:47:56 ServerOne kernel: RBP: ffff888fd5eb12c0 R08: 00000000e1f538c4 R09: ffffffff81c8aa80
Feb 17 21:47:56 ServerOne kernel: R10: 0000000000000158 R11: ffffffff81e91080 R12: 0000000000000c17
Feb 17 21:47:56 ServerOne kernel: R13: ffffffff81e91080 R14: 0000000000000000 R15: 000000000000c80d
Feb 17 21:47:56 ServerOne kernel: FS:  0000000000000000(0000) GS:ffff88903f800000(0000) knlGS:0000000000000000
Feb 17 21:47:56 ServerOne kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 17 21:47:56 ServerOne kernel: CR2: 00001516f289a718 CR3: 0000000001e0a004 CR4: 00000000001606f0
Feb 17 21:47:56 ServerOne kernel: Call Trace:
Feb 17 21:47:56 ServerOne kernel: <IRQ>
Feb 17 21:47:56 ServerOne kernel: ipv4_confirm+0xaf/0xb9
Feb 17 21:47:56 ServerOne kernel: nf_hook_slow+0x3a/0x90
Feb 17 21:47:56 ServerOne kernel: ip_local_deliver+0xad/0xdc
Feb 17 21:47:56 ServerOne kernel: ? ip_sublist_rcv_finish+0x54/0x54
Feb 17 21:47:56 ServerOne kernel: ip_sabotage_in+0x38/0x3e
Feb 17 21:47:56 ServerOne kernel: nf_hook_slow+0x3a/0x90
Feb 17 21:47:56 ServerOne kernel: ip_rcv+0x8e/0xbe
Feb 17 21:47:56 ServerOne kernel: ? ip_rcv_finish_core.isra.0+0x2e1/0x2e1
Feb 17 21:47:56 ServerOne kernel: __netif_receive_skb_one_core+0x53/0x6f
Feb 17 21:47:56 ServerOne kernel: process_backlog+0x77/0x10e
Feb 17 21:47:56 ServerOne kernel: net_rx_action+0x107/0x26c
Feb 17 21:47:56 ServerOne kernel: __do_softirq+0xc9/0x1d7
Feb 17 21:47:56 ServerOne kernel: do_softirq_own_stack+0x2a/0x40
Feb 17 21:47:56 ServerOne kernel: </IRQ>
Feb 17 21:47:56 ServerOne kernel: do_softirq+0x4d/0x5a
Feb 17 21:47:56 ServerOne kernel: netif_rx_ni+0x1c/0x22
Feb 17 21:47:56 ServerOne kernel: macvlan_broadcast+0x111/0x156 [macvlan]
Feb 17 21:47:56 ServerOne kernel: ? __switch_to_asm+0x41/0x70
Feb 17 21:47:56 ServerOne kernel: macvlan_process_broadcast+0xea/0x128 [macvlan]
Feb 17 21:47:56 ServerOne kernel: process_one_work+0x16e/0x24f
Feb 17 21:47:56 ServerOne kernel: worker_thread+0x1e2/0x2b8
Feb 17 21:47:56 ServerOne kernel: ? rescuer_thread+0x2a7/0x2a7
Feb 17 21:47:56 ServerOne kernel: kthread+0x10c/0x114
Feb 17 21:47:56 ServerOne kernel: ? kthread_park+0x89/0x89
Feb 17 21:47:56 ServerOne kernel: ret_from_fork+0x35/0x40
Feb 17 21:47:56 ServerOne kernel: ---[ end trace e6dff886c5bb9d83 ]---

 

But no server hangs, dockers are still working as normal.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.