[6.5.0]+ Call Traces when assigning IP Address to Docker Containers


89 posts in this topic Last Reply

Recommended Posts

I think i'm having the same issue lately. 

A lot of call trace on macvlan and server hangs.

Most of my dockers and VMs are running on VLANs (br1.xxxx) on eth1, two of them "host" and one "bridge". Unraid management have interface of it's own on eth0. 

How can i troubleshoot this? Can i find out what docker container causing this?

 

Apr 23 02:21:25 Tower kernel: igb 0000:02:00.0 eth1: mixed HW and IP checksum settings.
Apr 23 02:21:25 Tower kernel: eth0: renamed from veth7ba0ded
Apr 23 02:21:26 Tower CA Backup/Restore: #######################
Apr 23 02:21:26 Tower CA Backup/Restore: appData Backup complete
Apr 23 02:21:26 Tower CA Backup/Restore: #######################
Apr 23 02:21:26 Tower CA Backup/Restore: Deleting /mnt/user/backup/unraid_docker/2020-04-16@02.00
Apr 23 02:21:26 Tower CA Backup/Restore: Backup / Restore Completed
Apr 23 02:37:22 Tower kernel: WARNING: CPU: 5 PID: 825 at mm/workingset.c:456 shadow_lru_isolate+0x9f/0x26d
Apr 23 02:37:22 Tower kernel: Modules linked in: veth nfsv3 nfs arc4 ecb md4 xt_CHECKSUM sha512_ssse3 ipt_REJECT sha512_generic cmac cifs ccm ip6table_mangle ip6table_nat nf_nat_ipv6 iptable_mangle ip6table_filter ip6_tables vhost_net tun vhost tap macvlan xt_nat ipt_MASQUERADE iptable_filter iptable_nat nf_nat_ipv4 nf_nat ip_tables xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid e1000e igb(O) wmi_bmof mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_cstate intel_uncore intel_rapl_perf i2c_i801 i2c_core mpt3sas ahci libahci cdc_acm wmi raid_class pcc_cpufreq scsi_transport_sas video backlight thermal fan button acpi_pad [last unloaded: e1000e]
Apr 23 02:37:22 Tower kernel: CPU: 5 PID: 825 Comm: kswapd0 Tainted: G        W  O      4.19.107-Unraid #1
Apr 23 02:37:22 Tower kernel: Hardware name: System manufacturer System Product Name/Z170 PRO GAMING, BIOS 3805 05/16/2018
Apr 23 02:37:22 Tower kernel: RIP: 0010:shadow_lru_isolate+0x9f/0x26d
Apr 23 02:37:22 Tower kernel: Code: 00 00 ff ca fe 4b ea 88 53 eb 49 ff 4c 24 58 48 83 c0 08 48 3d 00 02 00 00 74 33 48 8b 54 03 10 48 85 d2 74 ea 80 e2 02 75 07 <0f> 0b e9 97 01 00 00 8a 53 eb 84 d2 75 07 0f 0b e9 89 01 00 00 49
Apr 23 02:37:22 Tower kernel: RSP: 0018:ffffc90003387c60 EFLAGS: 00010046
Apr 23 02:37:22 Tower kernel: RAX: 0000000000000140 RBX: ffff888011093490 RCX: 0000000000000000
Apr 23 02:37:22 Tower kernel: RDX: ffff8887b9b5df00 RSI: ffff888011093490 RDI: ffff88880dc2e580
Apr 23 02:37:22 Tower kernel: RBP: ffff88880dc2e588 R08: 0000000000000015 R09: ffff8880110936d8
Apr 23 02:37:22 Tower kernel: R10: 0000000000000000 R11: ffffea001ddf8b40 R12: ffff888302d9b870
Apr 23 02:37:22 Tower kernel: R13: ffff88880dc2e580 R14: ffffffff811026a6 R15: 0000000000000000
Apr 23 02:37:22 Tower kernel: FS:  0000000000000000(0000) GS:ffff888816b40000(0000) knlGS:0000000000000000
Apr 23 02:37:22 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 23 02:37:22 Tower kernel: CR2: 00007fe005e39000 CR3: 0000000001e0a003 CR4: 00000000003626e0
Apr 23 02:37:22 Tower kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 23 02:37:22 Tower kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Apr 23 02:37:22 Tower kernel: Call Trace:
Apr 23 02:37:22 Tower kernel: ? memcg_drain_all_list_lrus+0x16f/0x16f
Apr 23 02:37:22 Tower kernel: __list_lru_walk_one+0x76/0xfe
Apr 23 02:37:22 Tower kernel: ? memcg_drain_all_list_lrus+0x16f/0x16f
Apr 23 02:37:22 Tower kernel: list_lru_walk_one_irq+0x4a/0x69
Apr 23 02:37:22 Tower kernel: do_shrink_slab+0x128/0x194
Apr 23 02:37:22 Tower kernel: shrink_slab+0x20c/0x276
Apr 23 02:37:22 Tower kernel: shrink_node+0x108/0x3cb
Apr 23 02:37:22 Tower kernel: kswapd+0x451/0x58a
Apr 23 02:37:22 Tower kernel: ? mem_cgroup_shrink_node+0xa4/0xa4
Apr 23 02:37:22 Tower kernel: kthread+0x10c/0x114
Apr 23 02:37:22 Tower kernel: ? kthread_park+0x89/0x89
Apr 23 02:37:22 Tower kernel: ret_from_fork+0x1f/0x40
Apr 23 02:37:22 Tower kernel: ---[ end trace 3a60ba8e5d8238cd ]---




 

Link to post
  • Replies 88
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

TLDR Version:   Since this thread gets linked often for reference when a forum member experiences macvlan/broadcast call traces, I will provide a summary of the findings and potential soluti

My network settings for eth1 My Docker Settings   Hope this helps.

So I have been running my system since Wednesday of last week and I no longer have any call traces on macvlan.   I may try the next suggestion and place 192.168.19.xxx with a DHCP pool on br

Posted Images

  • 5 months later...

Sorry to resurrect this topic, but I am also having this issue...

I am seeing a lot of UniFi docker containers in screenshots and traces. Does everyone with this issue have Ubiquiti/UniFi equipment? Could that be our issue?

Can someone please explain the steps to implement the "fix" (creating a new VLAN to register the Docker container's static IPs to). Are we saying I need to create a new "Pool" of IPs on my EdgeRouter for use only with the containers, or is this a config change only on the Unraid side? (Sorry, still kind of an Unraid n00b here).

 

Thanks in advance.

Link to post
26 minutes ago, ScottRTL said:

Can someone please explain the steps to implement the "fix" (creating a new VLAN to register the Docker container's static IPs to

I used this guide to setup my br0.3 VLAN.  I do not use a Docker DHCP pool and prefer to manually assign IP address in a range not in my router DHCP pool.

 

Edited by Hoopster
Link to post
  • 2 months later...

I was getting macvlan call traces this past year on Unraid 6.8 and 6.9. I have unifi hardware also and have always had vlans and static ip's for dockers. Here's some things I have done since I started getting the call traces. Not all of these were done to try and fix the problem. Just upgraded some hardware. I swapped my SM MB for my current x11ssi-ln4f (old one had bent pins anyway that made one of ram ports not work). I also upgraded my network USG4 router and USW 24 PoE switch to UDM Pro router and 24 PoE Pro switch. Upgraded to Mellanox 10GB network card and disabled the 4 Intel nic ports on the MB. Dedicated IPMI port was still active. Still was getting call traces through all these changes. 

 

One of 2 things fixed this for me. I noticed Unraid's network settings for my 10GB card was set to eth0 and my vlans on br0 had IPv4 address assignments setup. I changed those vlans Ipv4 assignments to none. (not sure why I had this set up as I already manually set docker IP addresses to an ip in the vlans range in the docker's config) My dedicated IPMI lan with a fixed ip (I set a fixed ip in the SM IPMI/BMC settings) was set as eth1 (not part of br0) and eth1 was setup to get an IP under IPv4 address assignment. I changed that to none also. 

 

So after setting both my vlans and eth1 IPv4 address assignment to none, I have not had a single call trace or server crash in a few weeks. Previously server would have call traces once in a while but would crash crash at least once a week. 

Link to post

Hey @Hoopster, is this still the way? I noticed that my Nextcloud wasn't running so I checked my syslog and see a call trace error. I assume it's related? I have pihole but it's been disabled since the last crash (nginx errors). I have Plex on a br0 network with a set IP. I don't have a DHCP pool set in Docker settings. The IPv4 custom network on interface br0 (in Docker settings) is the same subnet as my router's main LAN network. And the Gateway address is my router's address. I have no DHCP pool set. (in Docker)  

 

Do I have this set wrong? This was all done per Spaceinvaderone's videos. I have Nextcloud and Swag on a "proxynet" custom network per his video as well. 

 

197526084_calltrace.thumb.JPG.6795b7b3cbfe99a2d33b6a8b3b4fabb0.JPG

 

tower-syslog-20210121-0359.zip

Link to post
  • Hoopster changed the title to [6.5.0]+ Call Traces when assigning IP Address to Docker Containers
  • 2 weeks later...

Thank you for taking the time to troubleshoot and diagnose this issue. I would have been lost without this thread. I had a thread open for two months diagnosing my call trace issues. I found out that it is related to my 10gig PCIE NIC that I was using. I removed the NIC and all of my problems have went away. My server was going as far as to completely lock up requiring a hard reset. 

 

 

Link to post
On 10/16/2020 at 1:34 PM, Hoopster said:

I used this guide to setup my br0.3 VLAN.  I do not use a Docker DHCP pool and prefer to manually assign IP address in a range not in my router DHCP pool.

How does this actually work in practice though. If Pihole etc is on a different network VLAN how do your computers on your network see it ?

 

Also do you know if this problem occurs if you set the network of the container to br0 but then leave it to grab an IP via DHCP ? If so I could just put an entry in the DHCP table instead.

Link to post
17 minutes ago, anethema said:

How does this actually work in practice though. If Pihole etc is on a different network VLAN how do your computers on your network see it ?

 

It depends on how your router treats VLAN networks. My Edgerouter, by default allows communication across VLANs by default unless you add firewall rules to segregate them. Other routers may be different. 

 

17 minutes ago, anethema said:

Also do you know if this problem occurs if you set the network of the container to br0 but then leave it to grab an IP via DHCP ? If so I could just put an entry in the DHCP table instead.

 

Yes, for me it still occured when my router was providing DHCP for the containers on br0.

Link to post
2 hours ago, anethema said:

How does this actually work in practice though. If Pihole etc is on a different network VLAN how do your computers on your network see it ?

As @adminmat stated, this depends on your router.  You may have to configure firewall rules to allow network traffic to pass between VLANs, primary LAN, etc.  By default mine allows all traffic between configured "corporate" LANs (just what UniFi chooses to call LANs/VLANs, - guest networks are excluded).

 

FYI - I started out with PiHole as a docker container and switched to running it dedicated on a Raspberry Pi.  Far fewer problems all around for me this way.  It certainly can work well as a docker container, but, there are some "gotchas" you have to be aware of and plan around.

 

2 hours ago, anethema said:

Also do you know if this problem occurs if you set the network of the container to br0 but then leave it to grab an IP via DHCP ? If so I could just put an entry in the DHCP table instead.

It made no difference in my case.  I had to create a VLAN, something other than br0.  br0 and br0.3 (my VLAN) are physically on the same NIC but there is something different in Docker with it not being br0. 

Link to post

Thanks for the help I will give it a shot. I am on unifi also. I think I can probably just move unifi controller to a bridge mode and get rid of pihole.


Does this also happen with VMs though? I really need my blue iris VM to have an IP heh.

Link to post
On 2/20/2021 at 12:18 PM, anethema said:

Does this also happen with VMs though?

Seems to only be an issues with Docker containers.  I have seen no reports of IP addresses assigned to VMs causing this problem.

Link to post
  • 1 month later...

Swapped my motherboard out for a ASRockRack X470D4U2-2T to use IPMI with dual 10 gig NICs and the call trace issue has returned. I am going to follow the following post and disable the "Enable VLANs:" setting under my 10 gig NIC and we will see if that makes a difference or not.

 

Unraid 6.9.2

Router: UDM-Pro

Switch: USW-Pro-24-PoE

 

 

 

image.png.0cf44d9f0b2ce36fbd04773d1ee68036.png

Link to post

Just disabling the VLAN on the NIC did not stop the issue. The dockers that need separate IP addresses I have built onto another VLAN I will watch the logs for a week or so to see if the call traces come back again or not.

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.