Help Request: Call Traces Found v6.5.0

[JasonM]

I've seen a few posts about this, but much of the analysis is beyond my ability. Relevant information appears to be in section pasted below. Full logs attached. Any assistance is appreciated.

Mar 19 15:01:37 unRAID kernel: Call Trace:
Mar 19 15:01:37 unRAID kernel: <IRQ>
Mar 19 15:01:37 unRAID kernel: ipv4_confirm+0xac/0xb4 [nf_conntrack_ipv4]
Mar 19 15:01:37 unRAID kernel: nf_hook_slow+0x37/0x96
Mar 19 15:01:37 unRAID kernel: ip_local_deliver+0xab/0xd3
Mar 19 15:01:37 unRAID kernel: ? inet_del_offload+0x3e/0x3e
Mar 19 15:01:37 unRAID kernel: ip_sabotage_in+0x2b/0x31
Mar 19 15:01:37 unRAID kernel: nf_hook_slow+0x37/0x96
Mar 19 15:01:37 unRAID kernel: ip_rcv+0x2f2/0x346
Mar 19 15:01:37 unRAID kernel: ? ip_local_deliver_finish+0x1b8/0x1b8
Mar 19 15:01:37 unRAID kernel: __netif_receive_skb_core+0x6ba/0x733
Mar 19 15:01:37 unRAID kernel: netif_receive_skb_internal+0xbb/0xd0
Mar 19 15:01:37 unRAID kernel: ? nf_nat_used_tuple+0x1e/0x24 [nf_nat]
Mar 19 15:01:37 unRAID kernel: br_pass_frame_up+0x12d/0x13a
Mar 19 15:01:37 unRAID kernel: ? br_flood+0xa6/0x146
Mar 19 15:01:37 unRAID kernel: br_handle_frame_finish+0x41a/0x44a
Mar 19 15:01:37 unRAID kernel: ? br_pass_frame_up+0x13a/0x13a
Mar 19 15:01:37 unRAID kernel: br_nf_hook_thresh+0x93/0x9e
Mar 19 15:01:37 unRAID kernel: ? br_pass_frame_up+0x13a/0x13a
Mar 19 15:01:37 unRAID kernel: br_nf_pre_routing_finish+0x268/0x27a
Mar 19 15:01:37 unRAID kernel: ? br_pass_frame_up+0x13a/0x13a
Mar 19 15:01:37 unRAID kernel: ? nf_nat_ipv4_fn+0x116/0x166 [nf_nat_ipv4]
Mar 19 15:01:37 unRAID kernel: ? nf_nat_ipv4_in+0x21/0x68 [nf_nat_ipv4]
Mar 19 15:01:37 unRAID kernel: br_nf_pre_routing+0x2d8/0x2e8
Mar 19 15:01:37 unRAID kernel: ? br_nf_forward_ip+0x32c/0x32c
Mar 19 15:01:37 unRAID kernel: nf_hook_slow+0x37/0x96
Mar 19 15:01:37 unRAID kernel: br_handle_frame+0x2a0/0x2d3
Mar 19 15:01:37 unRAID kernel: ? br_pass_frame_up+0x13a/0x13a
Mar 19 15:01:37 unRAID kernel: ? br_handle_local_finish+0x31/0x31
Mar 19 15:01:37 unRAID kernel: __netif_receive_skb_core+0x463/0x733
Mar 19 15:01:37 unRAID kernel: ? inet_gro_receive+0x25a/0x26f
Mar 19 15:01:37 unRAID kernel: ? recalibrate_cpu_khz+0x6/0x6
Mar 19 15:01:37 unRAID kernel: netif_receive_skb_internal+0xbb/0xd0
Mar 19 15:01:37 unRAID kernel: napi_gro_receive+0x42/0x76
Mar 19 15:01:37 unRAID kernel: igb_poll+0xb6b/0xb91 [igb]
Mar 19 15:01:37 unRAID kernel: net_rx_action+0xfb/0x24f
Mar 19 15:01:37 unRAID kernel: __do_softirq+0xcd/0x1c2
Mar 19 15:01:37 unRAID kernel: irq_exit+0x4f/0x8e
Mar 19 15:01:37 unRAID kernel: do_IRQ+0xa5/0xbb
Mar 19 15:01:37 unRAID kernel: common_interrupt+0x7d/0x7d
Mar 19 15:01:37 unRAID kernel: </IRQ>
Mar 19 15:01:37 unRAID kernel: RIP: 0010:cpuidle_enter_state+0xe3/0x135
Mar 19 15:01:37 unRAID kernel: RSP: 0018:ffffc9000630bef8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff1b
Mar 19 15:01:37 unRAID kernel: RAX: ffff8810779e0980 RBX: 0000000000000000 RCX: 000000000000001f
Mar 19 15:01:37 unRAID kernel: RDX: 0000141824948c1c RSI: 0000000000020180 RDI: 0000000000000000
Mar 19 15:01:37 unRAID kernel: RBP: ffff8810779e8800 R08: 000048dc057acc94 R09: 0000000000000090
Mar 19 15:01:37 unRAID kernel: R10: ffffc9000630bed8 R11: 0000000000006d68 R12: 0000000000000006
Mar 19 15:01:37 unRAID kernel: R13: 0000141824948c1c R14: ffffffff81c59398 R15: 0000141824781252
Mar 19 15:01:37 unRAID kernel: ? cpuidle_enter_state+0xbb/0x135
Mar 19 15:01:37 unRAID kernel: do_idle+0x11a/0x179
Mar 19 15:01:37 unRAID kernel: cpu_startup_entry+0x18/0x1a
Mar 19 15:01:37 unRAID kernel: secondary_startup_64+0xa5/0xb0
Mar 19 15:01:37 unRAID kernel: Code: 48 c1 eb 20 89 1c 24 e8 24 f9 ff ff 8b 54 24 04 89 df 89 c6 41 89 c5 e8 a9 fa ff ff 84 c0 75 b9 49 8b 86 80 00 00 00 a8 08 74 02 <0f> 0b 4c 89 f7 e8 03 ff ff ff 49 8b 86 80 00 00 00 0f ba e0 09 
Mar 19 15:01:37 unRAID kernel: ---[ end trace afb9e2141655eb4f ]---

 

unraid-diagnostics-20180320-1130.zip

[JorgeB]

Are you using custom docker IPs? Looks related, if yes try disabling that and see it makes a difference.

[JasonM]

Thanks. Yes, all Dockers have a dedicated IP. Unfortunately, I'll loose some functionality by disabling, but I'll give it a shot. The Call Trace error is intermittent, so it may take a few days to confirm if it's really gone.

[Hoopster]

27 minutes ago, JasonM said:

Thanks. Yes, all Dockers have a dedicated IP. Unfortunately, I'll loose some functionality by disabling, but I'll give it a shot. The Call Trace error is intermittent, so it may take a few days to confirm if it's really gone.

 

I also got call traces with separate IPs assigned to dockers.  Letting the dockers go back to the host IP eliminated the call traces.  I was doing this on an experimental basis, so, as of yet, I am not losing functionality; however, I hope to be able to assign separate IPs in the future without these issues. 

 

In my case, the call traces did not cause any issues that I could identify.  The system seemed to be functioning normally despite the call traces; however, I was concerned that it may eventually have led to problems so I removed the docker IP assignments.

 

Based on forum posts, it appears call traces are much more common than they used to be and I am hopeful that this will eventually get resolved or better explained.

[bonienl]

This issue is hard to pinpoint.

 

All my containers use a custom network, and for testing purposes I have created VLANs and a separate physical interface to let the containers communicate in different ways. Even make a combination of all different network types available.

 

Never had any call trace.

 

[Hoopster]

1 hour ago, bonienl said:

Never had any call trace.

 

It is almost impossible to solve a problem (or even know if it really is a problem) you cannot reproduce.  That's the most frustrating aspect of software development - and other things.