Hello,

 

I have a basic unraid server that hosts my file server, plex, windows 10 vm and pfsense vm. I woke up Thursday morning and the server was unresponsive. I re-booted the server and turned on troubleshoot mode. Everything worked fine all day Thursday but around 1:00AM PST my internet started to go out (pfSense), the unraid web gui was unresponsive, I could actually access pfsense vm's web gui but everything was running ultra slow (aka unusable). Here is the last log dump, I see some weird activity but I don't know what it means. If anyone could help me out, that would be awesome.

 

Thank you!

skynet-diagnostics-20181227-2226.zip

There's multiple (4) call traces like the following in your syslog:

 

Dec 27 10:59:10 SkyNET kernel: WARNING: CPU: 0 PID: 0 at net/netfilter/nf_conntrack_core.c:763 __nf_conntrack_confirm+0x96/0x4fc
Dec 27 10:59:10 SkyNET kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT xt_nat macvlan ebtable_filter ebtables ip6table_filter ip6_tables vhost_net tun vhost tap ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs md_mod ipmi_devintf igb i2c_algo_bit ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_cstate intel_uncore intel_rapl_perf i2c_i801 i2c_core e1000e ahci libahci video ie31200_edac backlight pcc_cpufreq thermal button acpi_pad fan ipmi_si [last unloaded: i2c_algo_bit]
Dec 27 10:59:10 SkyNET kernel: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.18.20-unRAID #1
Dec 27 10:59:10 SkyNET kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./E3C226D2I, BIOS P3.30 06/04/2015
Dec 27 10:59:10 SkyNET kernel: RIP: 0010:__nf_conntrack_confirm+0x96/0x4fc
Dec 27 10:59:10 SkyNET kernel: Code: c1 ed 20 89 2c 24 e8 26 f7 ff ff 8b 54 24 04 89 ef 89 c6 41 89 c5 e8 bc f8 ff ff 84 c0 75 b9 49 8b 86 80 00 00 00 a8 08 74 02 <0f> 0b 4c 89 f7 e8 04 ff ff ff 49 8b 86 80 00 00 00 0f ba e0 09 73 
Dec 27 10:59:10 SkyNET kernel: RSP: 0018:ffff88041fc038f8 EFLAGS: 00010202
Dec 27 10:59:10 SkyNET kernel: RAX: 0000000000000188 RBX: ffff8804090f2800 RCX: 00000000df980d6a
Dec 27 10:59:10 SkyNET kernel: RDX: 0000000000000001 RSI: 000000000000037f RDI: ffffffff81e093fc
Dec 27 10:59:10 SkyNET kernel: RBP: 0000000000000c78 R08: 000000007637d993 R09: 0000000000000000
Dec 27 10:59:10 SkyNET kernel: R10: 0000000000000158 R11: ffffffff81e8cdc0 R12: ffffffff81e8cdc0
Dec 27 10:59:10 SkyNET kernel: R13: 000000000000137f R14: ffff8802b697c000 R15: ffff8802b697c058
Dec 27 10:59:10 SkyNET kernel: FS:  0000000000000000(0000) GS:ffff88041fc00000(0000) knlGS:0000000000000000
Dec 27 10:59:10 SkyNET kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 27 10:59:10 SkyNET kernel: CR2: 00001503206b5000 CR3: 0000000001e0a006 CR4: 00000000001626f0
Dec 27 10:59:10 SkyNET kernel: Call Trace:
Dec 27 10:59:10 SkyNET kernel: <IRQ>
Dec 27 10:59:10 SkyNET kernel: ipv4_confirm+0xaf/0xb7 [nf_conntrack_ipv4]
Dec 27 10:59:10 SkyNET kernel: nf_hook_slow+0x37/0x96
Dec 27 10:59:10 SkyNET kernel: ip_local_deliver+0xa7/0xd5
Dec 27 10:59:10 SkyNET kernel: ? inet_del_offload+0x3e/0x3e
Dec 27 10:59:10 SkyNET kernel: ip_sabotage_in+0x38/0x3e
Dec 27 10:59:10 SkyNET kernel: nf_hook_slow+0x37/0x96
Dec 27 10:59:10 SkyNET kernel: ip_rcv+0x2ca/0x317
Dec 27 10:59:10 SkyNET kernel: ? ip_local_deliver_finish+0x1aa/0x1aa
Dec 27 10:59:10 SkyNET kernel: __netif_receive_skb_core+0x6b2/0x740
Dec 27 10:59:10 SkyNET kernel: ? nf_conntrack_tuple_taken+0x45/0x18c
Dec 27 10:59:10 SkyNET kernel: netif_receive_skb_internal+0x9f/0xba
Dec 27 10:59:10 SkyNET kernel: br_pass_frame_up+0x121/0x143
Dec 27 10:59:10 SkyNET kernel: ? br_port_flags_change+0x29/0x29
Dec 27 10:59:10 SkyNET kernel: br_handle_frame_finish+0x331/0x376
Dec 27 10:59:10 SkyNET kernel: ? ipt_do_table+0x58e/0x5db [ip_tables]
Dec 27 10:59:10 SkyNET kernel: ? br_pass_frame_up+0x143/0x143
Dec 27 10:59:10 SkyNET kernel: br_nf_hook_thresh+0xa3/0xc3
Dec 27 10:59:10 SkyNET kernel: ? br_pass_frame_up+0x143/0x143
Dec 27 10:59:10 SkyNET kernel: br_nf_pre_routing_finish+0x22f/0x256
Dec 27 10:59:10 SkyNET kernel: ? br_pass_frame_up+0x143/0x143
Dec 27 10:59:10 SkyNET kernel: ? nf_nat_ipv4_in+0x1d/0x64 [nf_nat_ipv4]
Dec 27 10:59:10 SkyNET kernel: br_nf_pre_routing+0x2d2/0x2f7
Dec 27 10:59:10 SkyNET kernel: ? br_nf_forward_ip+0x349/0x349
Dec 27 10:59:10 SkyNET kernel: nf_hook_slow+0x37/0x96
Dec 27 10:59:10 SkyNET kernel: br_handle_frame+0x291/0x2d0
Dec 27 10:59:10 SkyNET kernel: ? br_pass_frame_up+0x143/0x143
Dec 27 10:59:10 SkyNET kernel: ? br_handle_local_finish+0x31/0x31
Dec 27 10:59:10 SkyNET kernel: __netif_receive_skb_core+0x459/0x740
Dec 27 10:59:10 SkyNET kernel: ? inet_gro_receive+0x249/0x257
Dec 27 10:59:10 SkyNET kernel: netif_receive_skb_internal+0x9f/0xba
Dec 27 10:59:10 SkyNET kernel: napi_gro_receive+0x42/0x76
Dec 27 10:59:10 SkyNET kernel: igb_poll+0xb96/0xbbc [igb]
Dec 27 10:59:10 SkyNET kernel: net_rx_action+0x10b/0x274
Dec 27 10:59:10 SkyNET kernel: __do_softirq+0xce/0x1e2
Dec 27 10:59:10 SkyNET kernel: irq_exit+0x5e/0x9d
Dec 27 10:59:10 SkyNET kernel: do_IRQ+0xa9/0xc7
Dec 27 10:59:10 SkyNET kernel: common_interrupt+0xf/0xf
Dec 27 10:59:10 SkyNET kernel: </IRQ>
Dec 27 10:59:10 SkyNET kernel: RIP: 0010:cpuidle_enter_state+0xe8/0x141
Dec 27 10:59:10 SkyNET kernel: Code: ff 45 84 ff 74 1d 9c 58 0f 1f 44 00 00 0f ba e0 09 73 09 0f 0b fa 66 0f 1f 44 00 00 31 ff e8 73 af be ff fb 66 0f 1f 44 00 00 <48> 2b 1c 24 b8 ff ff ff 7f 48 b9 ff ff ff ff f3 01 00 00 48 39 cb 
Dec 27 10:59:10 SkyNET kernel: RSP: 0018:ffffffff81e03e80 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffdd
Dec 27 10:59:10 SkyNET kernel: RAX: ffff88041fc20c00 RBX: 00000ac39fa97e90 RCX: 000000000000001f
Dec 27 10:59:10 SkyNET kernel: RDX: 00000ac39fa97e90 RSI: 0000000025bb2ac8 RDI: 0000000000000000
Dec 27 10:59:10 SkyNET kernel: RBP: ffff88041fc28e00 R08: 0000000000000004 R09: 0000000000020480
Dec 27 10:59:10 SkyNET kernel: R10: 00000000000f6768 R11: 000024acdc5af5a0 R12: 0000000000000005
Dec 27 10:59:10 SkyNET kernel: R13: 0000000000000005 R14: ffffffff81e58a18 R15: 0000000000000000
Dec 27 10:59:10 SkyNET kernel: do_idle+0x192/0x20e
Dec 27 10:59:10 SkyNET kernel: cpu_startup_entry+0x6a/0x6c
Dec 27 10:59:10 SkyNET kernel: start_kernel+0x43a/0x45a
Dec 27 10:59:10 SkyNET kernel: secondary_startup_64+0xa5/0xb0
Dec 27 10:59:10 SkyNET kernel: ---[ end trace 0954a7802915d576 ]---

Yeah, I noticed that, what is a call trace, and what would be causing it? Also, why would it grind the server to a screeching halt?

 

I have a 4 port intel nic stubed out for the pfSense vm, and then I have dual intel nic on the motherboard that I use for unraid.

Could this be an issue with the the 4 port NIC card or something else? How can I lookup what is causing the trace?

Took my 4 port nic out and the call trace stopped. Looks like I got a counterfeit 4 port nic card, at least it lasted two years.

Well, the call traces started back up without the NIC in the PCIE slot.

 

Can anyone read this and point me in a possible direction? Is it hardware related or will a re-install of everything fix my issue?

 

Jan  5 05:27:27 SkyNET kernel: WARNING: CPU: 5 PID: 25630 at net/netfilter/nf_conntrack_core.c:763 __nf_conntrack_confirm+0x96/0x4fc
Jan  5 05:27:27 SkyNET kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT xt_nat macvlan ebtable_filter ebtables ip6table_filter ip6_tables vhost_net tun vhost tap ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs md_mod ipmi_devintf igb i2c_algo_bit x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_cstate intel_uncore ahci libahci intel_rapl_perf i2c_i801 video i2c_core ie31200_edac backlight pcc_cpufreq thermal acpi_pad button fan ipmi_si [last unloaded: i2c_algo_bit]
Jan  5 05:27:27 SkyNET kernel: CPU: 5 PID: 25630 Comm: kworker/5:2 Tainted: G    B   W         4.18.20-unRAID #1
Jan  5 05:27:27 SkyNET kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./E3C226D2I, BIOS P3.30 06/04/2015
Jan  5 05:27:27 SkyNET kernel: Workqueue: events macvlan_process_broadcast [macvlan]
Jan  5 05:27:27 SkyNET kernel: RIP: 0010:__nf_conntrack_confirm+0x96/0x4fc
Jan  5 05:27:27 SkyNET kernel: Code: c1 ed 20 89 2c 24 e8 26 f7 ff ff 8b 54 24 04 89 ef 89 c6 41 89 c5 e8 bc f8 ff ff 84 c0 75 b9 49 8b 86 80 00 00 00 a8 08 74 02 <0f> 0b 4c 89 f7 e8 04 ff ff ff 49 8b 86 80 00 00 00 0f ba e0 09 73 
Jan  5 05:27:27 SkyNET kernel: RSP: 0018:ffff88041fd43d30 EFLAGS: 00010202
Jan  5 05:27:27 SkyNET kernel: RAX: 0000000000000188 RBX: ffff88040d061a00 RCX: 0000000000000101
Jan  5 05:27:27 SkyNET kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffff81e091dc
Jan  5 05:27:27 SkyNET kernel: RBP: 00000000000049f6 R08: 00000000dbc65500 R09: 0000000000000000
Jan  5 05:27:27 SkyNET kernel: R10: 0000000000000098 R11: ffff8803d9858000 R12: ffffffff81e8cdc0
Jan  5 05:27:27 SkyNET kernel: R13: 000000000000d6f7 R14: ffff8800dcdc6640 R15: ffff8800dcdc6698
Jan  5 05:27:27 SkyNET kernel: FS:  0000000000000000(0000) GS:ffff88041fd40000(0000) knlGS:0000000000000000
Jan  5 05:27:27 SkyNET kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan  5 05:27:27 SkyNET kernel: CR2: 0000148c4e444718 CR3: 0000000001e0a005 CR4: 00000000001606e0
Jan  5 05:27:27 SkyNET kernel: Call Trace:
Jan  5 05:27:27 SkyNET kernel: <IRQ>
Jan  5 05:27:27 SkyNET kernel: ipv4_confirm+0xaf/0xb7 [nf_conntrack_ipv4]
Jan  5 05:27:27 SkyNET kernel: nf_hook_slow+0x37/0x96
Jan  5 05:27:27 SkyNET kernel: ip_local_deliver+0xa7/0xd5
Jan  5 05:27:27 SkyNET kernel: ? inet_del_offload+0x3e/0x3e
Jan  5 05:27:27 SkyNET kernel: ip_rcv+0x2dc/0x317
Jan  5 05:27:27 SkyNET kernel: ? ip_local_deliver_finish+0x1aa/0x1aa
Jan  5 05:27:27 SkyNET kernel: __netif_receive_skb_core+0x6b2/0x740
Jan  5 05:27:27 SkyNET kernel: process_backlog+0x7e/0x116
Jan  5 05:27:27 SkyNET kernel: net_rx_action+0x10b/0x274
Jan  5 05:27:27 SkyNET kernel: __do_softirq+0xce/0x1e2
Jan  5 05:27:27 SkyNET kernel: do_softirq_own_stack+0x2a/0x40
Jan  5 05:27:27 SkyNET kernel: </IRQ>
Jan  5 05:27:27 SkyNET kernel: do_softirq+0x4d/0x59
Jan  5 05:27:27 SkyNET kernel: netif_rx_ni+0x1c/0x22
Jan  5 05:27:27 SkyNET kernel: macvlan_broadcast+0x10f/0x153 [macvlan]
Jan  5 05:27:27 SkyNET kernel: ? __switch_to_asm+0x34/0x70
Jan  5 05:27:27 SkyNET kernel: macvlan_process_broadcast+0xd5/0x131 [macvlan]
Jan  5 05:27:27 SkyNET kernel: process_one_work+0x16e/0x24f
Jan  5 05:27:27 SkyNET kernel: ? cancel_delayed_work_sync+0xa/0xa
Jan  5 05:27:27 SkyNET kernel: worker_thread+0x1dc/0x2ac
Jan  5 05:27:27 SkyNET kernel: kthread+0x10b/0x113
Jan  5 05:27:27 SkyNET kernel: ? kthread_flush_work_fn+0x9/0x9
Jan  5 05:27:27 SkyNET kernel: ret_from_fork+0x35/0x40
Jan  5 05:27:27 SkyNET kernel: ---[ end trace 068d22dfd1443f4b ]---

Thank you for any assistance you can provide.

skynet-diagnostics-20190105-0638.zip

Following up on this: I believe my actual issue was related to assigning specific IP's in Docker containers, Plex being the main culprit. I have left it in host mode and gone 40+ hours now without a call trace (Unraid [or the NIC assigned to unraid] was going completely unresponsive in 8-24 hours after a restart previously)

 

This thread helped me identify the possible issue:

 

https://forums.unraid.net/topic/74886-call-trace-issue-started-653/

 

Not sure what update via plex or unraid brought this up as my setup was unchanged for roughly 12 months. I'm just going to leave plex in host mode and call it day.

 

Thank you to those that reviewed this but did not have an answer, much appreciated!

47 minutes ago, Reznap said:

Not sure what update via plex or unraid brought this up as my setup was unchanged for roughly 12 months

I don't think it has anything to do with any particular docker.  It seems to be hardware related somehow.  On some system there are never any issues and on others (like mine), custom IP address on br0/br1 will produce call traces that lock up the server.  It did not matter what docker it was.

 

I do have Plex running as host on the unRAID server IP, however, I have tried it with a custom IP on a VLAN and had no call traces.

 

You already referenced one of the threads in which my experience with this is documented.  However, creating a docker VLAN and assigning the customer IP addresses on that VLAN (br0.3) solved the issue for me.  Strangely, br0 and br0.3 are the same physical NIC, but, the former produced call traces and the latter does not.