My Unraid 6 crashed, need help to figure out why

[ytwytw]

I am using Unraid 6.7.0

 

Several days after a power cutoff (terrible weather in the mid-west USA)

 

My Unraid is keeping crashing

 

When i connect a monitor to the server, all i can see is the screenshot

 

 

Is there anything I can pull from the system to figure out the issue?

 

I have 4 harddrive (2 data and 2 parity) and 1 ssd cache

 

 

root@MyUnRaid:~# tail /var/log/syslog -n 100
May 30 20:57:25 MyUnRaid kernel: docker0: port 2(veth9147dc7) entered disabled state
May 30 20:57:29 MyUnRaid kernel: eth0: renamed from vethf040938
May 30 20:57:29 MyUnRaid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth9147dc7: link becomes ready
May 30 20:57:29 MyUnRaid kernel: docker0: port 2(veth9147dc7) entered blocking state
May 30 20:57:29 MyUnRaid kernel: docker0: port 2(veth9147dc7) entered forwarding state
May 30 20:57:30 MyUnRaid avahi-daemon[3311]: Joining mDNS multicast group on interface veth9147dc7.IPv6 with address fe80::1479:f4ff:fe00:4b9a.
May 30 20:57:30 MyUnRaid avahi-daemon[3311]: New relevant interface veth9147dc7.IPv6 for mDNS.
May 30 20:57:30 MyUnRaid avahi-daemon[3311]: Registering new address record for fe80::1479:f4ff:fe00:4b9a on veth9147dc7.*.
May 30 20:57:30 MyUnRaid rc.docker: mariadb: started succesfully!
May 30 20:57:30 MyUnRaid kernel: docker0: port 3(veth2601525) entered blocking state
May 30 20:57:30 MyUnRaid kernel: docker0: port 3(veth2601525) entered disabled state
May 30 20:57:30 MyUnRaid kernel: device veth2601525 entered promiscuous mode
May 30 20:57:30 MyUnRaid kernel: IPv6: ADDRCONF(NETDEV_UP): veth2601525: link is not ready
May 30 20:57:30 MyUnRaid kernel: docker0: port 3(veth2601525) entered blocking state
May 30 20:57:30 MyUnRaid kernel: docker0: port 3(veth2601525) entered forwarding state
May 30 20:57:30 MyUnRaid kernel: docker0: port 3(veth2601525) entered disabled state
May 30 20:57:34 MyUnRaid kernel: eth0: renamed from veth81f8143
May 30 20:57:34 MyUnRaid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth2601525: link becomes ready
May 30 20:57:34 MyUnRaid kernel: docker0: port 3(veth2601525) entered blocking state
May 30 20:57:34 MyUnRaid kernel: docker0: port 3(veth2601525) entered forwarding state
May 30 20:57:36 MyUnRaid rc.docker: miniflux: started succesfully!
May 30 20:57:36 MyUnRaid kernel: docker0: port 4(veth09d15be) entered blocking state
May 30 20:57:36 MyUnRaid kernel: docker0: port 4(veth09d15be) entered disabled state
May 30 20:57:36 MyUnRaid kernel: device veth09d15be entered promiscuous mode
May 30 20:57:36 MyUnRaid kernel: IPv6: ADDRCONF(NETDEV_UP): veth09d15be: link is not ready
May 30 20:57:36 MyUnRaid kernel: docker0: port 4(veth09d15be) entered blocking state
May 30 20:57:36 MyUnRaid kernel: docker0: port 4(veth09d15be) entered forwarding state
May 30 20:57:36 MyUnRaid kernel: docker0: port 4(veth09d15be) entered disabled state
May 30 20:57:36 MyUnRaid avahi-daemon[3311]: Joining mDNS multicast group on interface veth2601525.IPv6 with address fe80::82b:a4ff:fe61:dafc.
May 30 20:57:36 MyUnRaid avahi-daemon[3311]: New relevant interface veth2601525.IPv6 for mDNS.
May 30 20:57:36 MyUnRaid avahi-daemon[3311]: Registering new address record for fe80::82b:a4ff:fe61:dafc on veth2601525.*.
May 30 20:57:40 MyUnRaid kernel: eth0: renamed from veth96f9614
May 30 20:57:40 MyUnRaid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth09d15be: link becomes ready
May 30 20:57:40 MyUnRaid kernel: docker0: port 4(veth09d15be) entered blocking state
May 30 20:57:40 MyUnRaid kernel: docker0: port 4(veth09d15be) entered forwarding state
May 30 20:57:42 MyUnRaid avahi-daemon[3311]: Joining mDNS multicast group on interface veth09d15be.IPv6 with address fe80::5449:24ff:feb5:c806.
May 30 20:57:42 MyUnRaid avahi-daemon[3311]: New relevant interface veth09d15be.IPv6 for mDNS.
May 30 20:57:42 MyUnRaid avahi-daemon[3311]: Registering new address record for fe80::5449:24ff:feb5:c806 on veth09d15be.*.
May 30 20:57:42 MyUnRaid rc.docker: minifluxdb: started succesfully!
May 30 20:57:42 MyUnRaid kernel: docker0: port 5(veth3bbfa1c) entered blocking state
May 30 20:57:42 MyUnRaid kernel: docker0: port 5(veth3bbfa1c) entered disabled state
May 30 20:57:42 MyUnRaid kernel: device veth3bbfa1c entered promiscuous mode
May 30 20:57:42 MyUnRaid kernel: IPv6: ADDRCONF(NETDEV_UP): veth3bbfa1c: link is not ready
May 30 20:57:42 MyUnRaid kernel: docker0: port 5(veth3bbfa1c) entered blocking state
May 30 20:57:42 MyUnRaid kernel: docker0: port 5(veth3bbfa1c) entered forwarding state
May 30 20:57:42 MyUnRaid kernel: docker0: port 5(veth3bbfa1c) entered disabled state
May 30 20:57:48 MyUnRaid kernel: eth0: renamed from vethaa2884d
May 30 20:57:48 MyUnRaid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth3bbfa1c: link becomes ready
May 30 20:57:48 MyUnRaid kernel: docker0: port 5(veth3bbfa1c) entered blocking state
May 30 20:57:48 MyUnRaid kernel: docker0: port 5(veth3bbfa1c) entered forwarding state
May 30 20:57:49 MyUnRaid avahi-daemon[3311]: Joining mDNS multicast group on interface veth3bbfa1c.IPv6 with address fe80::7c39:60ff:feb7:eac2.
May 30 20:57:49 MyUnRaid avahi-daemon[3311]: New relevant interface veth3bbfa1c.IPv6 for mDNS.
May 30 20:57:49 MyUnRaid avahi-daemon[3311]: Registering new address record for fe80::7c39:60ff:feb7:eac2 on veth3bbfa1c.*.
May 30 20:57:50 MyUnRaid rc.docker: nginx-proxy-manager: started succesfully!
May 30 20:57:55 MyUnRaid kernel: eth0: renamed from veth47e9650
May 30 20:57:56 MyUnRaid rc.docker: unifi-controller: started succesfully!
May 30 21:01:02 MyUnRaid kernel: WARNING: CPU: 9 PID: 268 at net/netfilter/nf_conntrack_core.c:910 __nf_conntrack_confirm+0x97/0x686
May 30 21:01:02 MyUnRaid kernel: Modules linked in: xt_CHECKSUM ipt_REJECT ip6table_mangle ip6table_nat nf_nat_ipv6 iptable_mangle ip6table_filter ip6_tables vhost_net tun vhost tap veth macvlan xt_nat ipt_MASQUERADE iptable_nat nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod bonding sfc mdio tg3 sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp ipmi_ssif i2c_core kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper sr_mod ahci libahci intel_cstate intel_uncore ipmi_si cdrom intel_rapl_perf acpi_power_meter button wmi pcc_cpufreq [last unloaded: mdio]
May 30 21:01:02 MyUnRaid kernel: CPU: 9 PID: 268 Comm: kworker/9:1 Not tainted 4.19.41-Unraid #1
May 30 21:01:02 MyUnRaid kernel: Hardware name: Dell Inc. PowerEdge T320/0FDT3J, BIOS 2.6.0 06/11/2018
May 30 21:01:02 MyUnRaid kernel: Workqueue: events macvlan_process_broadcast [macvlan]
May 30 21:01:02 MyUnRaid kernel: RIP: 0010:__nf_conntrack_confirm+0x97/0x686
May 30 21:01:02 MyUnRaid kernel: Code: c1 ed 20 89 2c 24 e8 c3 fb ff ff 8b 54 24 04 89 ef 89 c6 41 89 c4 e8 a1 f8 ff ff 84 c0 75 b9 49 8b 86 80 00 00 00 a8 08 74 25 <0f> 0b 44 89 e6 89 ef 45 31 ff e8 eb f1 ff ff be 00 02 00 00 48 c7
May 30 21:01:02 MyUnRaid kernel: RSP: 0018:ffff88880f243d98 EFLAGS: 00010202
May 30 21:01:02 MyUnRaid kernel: RAX: 0000000000000188 RBX: ffff88869dd1d200 RCX: 00000000f927bdad
May 30 21:01:02 MyUnRaid kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffffff81e093ac
May 30 21:01:02 MyUnRaid kernel: RBP: 0000000000006f6a R08: ffff88868826ac30 R09: 0000000046dc2555
May 30 21:01:02 MyUnRaid kernel: R10: 0000000000000098 R11: ffff8887d0243c00 R12: 000000000000b36b
May 30 21:01:02 MyUnRaid kernel: R13: ffffffff81e8e040 R14: ffff88868826abc0 R15: ffff88868826ac18
May 30 21:01:02 MyUnRaid kernel: FS:  0000000000000000(0000) GS:ffff88880f240000(0000) knlGS:0000000000000000
May 30 21:01:02 MyUnRaid kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 30 21:01:02 MyUnRaid kernel: CR2: 0000146150ff2000 CR3: 0000000001e0a005 CR4: 00000000001606e0
May 30 21:01:02 MyUnRaid kernel: Call Trace:
May 30 21:01:02 MyUnRaid kernel: <IRQ>
May 30 21:01:02 MyUnRaid kernel: ipv4_confirm+0xaf/0xb7
May 30 21:01:02 MyUnRaid kernel: nf_hook_slow+0x37/0x96
May 30 21:01:02 MyUnRaid kernel: ip_local_deliver+0xa7/0xd5
May 30 21:01:02 MyUnRaid kernel: ? ip_sublist_rcv_finish+0x53/0x53
May 30 21:01:02 MyUnRaid kernel: ip_rcv+0x9e/0xbc
May 30 21:01:02 MyUnRaid kernel: ? ip_rcv_finish_core.isra.0+0x2e2/0x2e2
May 30 21:01:02 MyUnRaid kernel: __netif_receive_skb_one_core+0x4d/0x69
May 30 21:01:02 MyUnRaid kernel: process_backlog+0x7e/0x116
May 30 21:01:02 MyUnRaid kernel: net_rx_action+0x10b/0x274
May 30 21:01:02 MyUnRaid kernel: __do_softirq+0xce/0x1e2
May 30 21:01:02 MyUnRaid kernel: do_softirq_own_stack+0x2a/0x40
May 30 21:01:02 MyUnRaid kernel: </IRQ>
May 30 21:01:02 MyUnRaid kernel: do_softirq+0x4d/0x59
May 30 21:01:02 MyUnRaid kernel: netif_rx_ni+0x1c/0x22
May 30 21:01:02 MyUnRaid kernel: macvlan_broadcast+0x10f/0x153 [macvlan]
May 30 21:01:02 MyUnRaid kernel: ? __switch_to_asm+0x34/0x70
May 30 21:01:02 MyUnRaid kernel: macvlan_process_broadcast+0xd5/0x131 [macvlan]
May 30 21:01:02 MyUnRaid kernel: process_one_work+0x16e/0x24f
May 30 21:01:02 MyUnRaid kernel: ? pwq_unbound_release_workfn+0xb7/0xb7
May 30 21:01:02 MyUnRaid kernel: worker_thread+0x1dc/0x2ac
May 30 21:01:02 MyUnRaid kernel: kthread+0x10b/0x113
May 30 21:01:02 MyUnRaid kernel: ? kthread_park+0x71/0x71
May 30 21:01:02 MyUnRaid kernel: ret_from_fork+0x35/0x40
May 30 21:01:02 MyUnRaid kernel: ---[ end trace 3fabbb108adf8f69 ]---
May 30 21:01:43 MyUnRaid ntpd[1957]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
May 30 21:02:34 MyUnRaid login[10210]: ROOT LOGIN  on '/dev/pts/0'
root@MyUnRaid:~# 

 

 

myunraid-diagnostics-20190531-0100.zip

Edited May 31, 2019 by ytwytw

[JorgeB]

macvlan call traces are usually caused by dockers with custom IPs.

[ytwytw]

I do have pihole and plex with custom IP.

 

Is the crash caused by docker? or anything else?

 

I have a friend with similar config but he is fine...  

[Hoopster]

37 minutes ago, ytwytw said:

I do have pihole and plex with custom IP.

 

Is the crash caused by docker? or anything else?

 

I have a friend with similar config but he is fine...  

If you have docker containers with custom IP addresses on br0 and your call traces are macvlan related due to broadcast messages (which they appear to be), join the club.

 

 

I had to create a VLAN (br0.3) and assign custom IP addresses on that VLAN to avoid the problem.  Call traces do not always result in system crashes immediately. With this situation, after several days of call traces, my server eventually locked up hard and had to be powered down manually.  With the br0.3 VLAN, I have not seen these call traces in almost a year.

 

It does not happen to everyone and it does not always happen on br0, although for most of us it does. 

 

This is not a core unRAID issue.  There is not much Limetech can do to solve this as it appears to be related to the macvlan implementation in Docker.