April 24, 20206 yr I think i'm having the same issue lately. A lot of call trace on macvlan and server hangs. Most of my dockers and VMs are running on VLANs (br1.xxxx) on eth1, two of them "host" and one "bridge". Unraid management have interface of it's own on eth0. How can i troubleshoot this? Can i find out what docker container causing this? Apr 23 02:21:25 Tower kernel: igb 0000:02:00.0 eth1: mixed HW and IP checksum settings. Apr 23 02:21:25 Tower kernel: eth0: renamed from veth7ba0ded Apr 23 02:21:26 Tower CA Backup/Restore: ####################### Apr 23 02:21:26 Tower CA Backup/Restore: appData Backup complete Apr 23 02:21:26 Tower CA Backup/Restore: ####################### Apr 23 02:21:26 Tower CA Backup/Restore: Deleting /mnt/user/backup/unraid_docker/[email protected] Apr 23 02:21:26 Tower CA Backup/Restore: Backup / Restore Completed Apr 23 02:37:22 Tower kernel: WARNING: CPU: 5 PID: 825 at mm/workingset.c:456 shadow_lru_isolate+0x9f/0x26d Apr 23 02:37:22 Tower kernel: Modules linked in: veth nfsv3 nfs arc4 ecb md4 xt_CHECKSUM sha512_ssse3 ipt_REJECT sha512_generic cmac cifs ccm ip6table_mangle ip6table_nat nf_nat_ipv6 iptable_mangle ip6table_filter ip6_tables vhost_net tun vhost tap macvlan xt_nat ipt_MASQUERADE iptable_filter iptable_nat nf_nat_ipv4 nf_nat ip_tables xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid e1000e igb(O) wmi_bmof mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_cstate intel_uncore intel_rapl_perf i2c_i801 i2c_core mpt3sas ahci libahci cdc_acm wmi raid_class pcc_cpufreq scsi_transport_sas video backlight thermal fan button acpi_pad [last unloaded: e1000e] Apr 23 02:37:22 Tower kernel: CPU: 5 PID: 825 Comm: kswapd0 Tainted: G W O 4.19.107-Unraid #1 Apr 23 02:37:22 Tower kernel: Hardware name: System manufacturer System Product Name/Z170 PRO GAMING, BIOS 3805 05/16/2018 Apr 23 02:37:22 Tower kernel: RIP: 0010:shadow_lru_isolate+0x9f/0x26d Apr 23 02:37:22 Tower kernel: Code: 00 00 ff ca fe 4b ea 88 53 eb 49 ff 4c 24 58 48 83 c0 08 48 3d 00 02 00 00 74 33 48 8b 54 03 10 48 85 d2 74 ea 80 e2 02 75 07 <0f> 0b e9 97 01 00 00 8a 53 eb 84 d2 75 07 0f 0b e9 89 01 00 00 49 Apr 23 02:37:22 Tower kernel: RSP: 0018:ffffc90003387c60 EFLAGS: 00010046 Apr 23 02:37:22 Tower kernel: RAX: 0000000000000140 RBX: ffff888011093490 RCX: 0000000000000000 Apr 23 02:37:22 Tower kernel: RDX: ffff8887b9b5df00 RSI: ffff888011093490 RDI: ffff88880dc2e580 Apr 23 02:37:22 Tower kernel: RBP: ffff88880dc2e588 R08: 0000000000000015 R09: ffff8880110936d8 Apr 23 02:37:22 Tower kernel: R10: 0000000000000000 R11: ffffea001ddf8b40 R12: ffff888302d9b870 Apr 23 02:37:22 Tower kernel: R13: ffff88880dc2e580 R14: ffffffff811026a6 R15: 0000000000000000 Apr 23 02:37:22 Tower kernel: FS: 0000000000000000(0000) GS:ffff888816b40000(0000) knlGS:0000000000000000 Apr 23 02:37:22 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 23 02:37:22 Tower kernel: CR2: 00007fe005e39000 CR3: 0000000001e0a003 CR4: 00000000003626e0 Apr 23 02:37:22 Tower kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Apr 23 02:37:22 Tower kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Apr 23 02:37:22 Tower kernel: Call Trace: Apr 23 02:37:22 Tower kernel: ? memcg_drain_all_list_lrus+0x16f/0x16f Apr 23 02:37:22 Tower kernel: __list_lru_walk_one+0x76/0xfe Apr 23 02:37:22 Tower kernel: ? memcg_drain_all_list_lrus+0x16f/0x16f Apr 23 02:37:22 Tower kernel: list_lru_walk_one_irq+0x4a/0x69 Apr 23 02:37:22 Tower kernel: do_shrink_slab+0x128/0x194 Apr 23 02:37:22 Tower kernel: shrink_slab+0x20c/0x276 Apr 23 02:37:22 Tower kernel: shrink_node+0x108/0x3cb Apr 23 02:37:22 Tower kernel: kswapd+0x451/0x58a Apr 23 02:37:22 Tower kernel: ? mem_cgroup_shrink_node+0xa4/0xa4 Apr 23 02:37:22 Tower kernel: kthread+0x10c/0x114 Apr 23 02:37:22 Tower kernel: ? kthread_park+0x89/0x89 Apr 23 02:37:22 Tower kernel: ret_from_fork+0x1f/0x40 Apr 23 02:37:22 Tower kernel: ---[ end trace 3a60ba8e5d8238cd ]---
April 24, 20206 yr When using VMs, change the model of the interface to "virtio-net" <interface type='bridge'> ... <model type='virtio-net'/> ... </interface>
October 16, 20205 yr Sorry to resurrect this topic, but I am also having this issue... I am seeing a lot of UniFi docker containers in screenshots and traces. Does everyone with this issue have Ubiquiti/UniFi equipment? Could that be our issue? Can someone please explain the steps to implement the "fix" (creating a new VLAN to register the Docker container's static IPs to). Are we saying I need to create a new "Pool" of IPs on my EdgeRouter for use only with the containers, or is this a config change only on the Unraid side? (Sorry, still kind of an Unraid n00b here). Thanks in advance.
October 16, 20205 yr Author 26 minutes ago, ScottRTL said: Can someone please explain the steps to implement the "fix" (creating a new VLAN to register the Docker container's static IPs to I used this guide to setup my br0.3 VLAN. I do not use a Docker DHCP pool and prefer to manually assign IP address in a range not in my router DHCP pool. Edited October 16, 20205 yr by Hoopster
January 2, 20215 yr I was getting macvlan call traces this past year on Unraid 6.8 and 6.9. I have unifi hardware also and have always had vlans and static ip's for dockers. Here's some things I have done since I started getting the call traces. Not all of these were done to try and fix the problem. Just upgraded some hardware. I swapped my SM MB for my current x11ssi-ln4f (old one had bent pins anyway that made one of ram ports not work). I also upgraded my network USG4 router and USW 24 PoE switch to UDM Pro router and 24 PoE Pro switch. Upgraded to Mellanox 10GB network card and disabled the 4 Intel nic ports on the MB. Dedicated IPMI port was still active. Still was getting call traces through all these changes. One of 2 things fixed this for me. I noticed Unraid's network settings for my 10GB card was set to eth0 and my vlans on br0 had IPv4 address assignments setup. I changed those vlans Ipv4 assignments to none. (not sure why I had this set up as I already manually set docker IP addresses to an ip in the vlans range in the docker's config) My dedicated IPMI lan with a fixed ip (I set a fixed ip in the SM IPMI/BMC settings) was set as eth1 (not part of br0) and eth1 was setup to get an IP under IPv4 address assignment. I changed that to none also. So after setting both my vlans and eth1 IPv4 address assignment to none, I have not had a single call trace or server crash in a few weeks. Previously server would have call traces once in a while but would crash crash at least once a week.
January 21, 20215 yr Hey @Hoopster, is this still the way? I noticed that my Nextcloud wasn't running so I checked my syslog and see a call trace error. I assume it's related? I have pihole but it's been disabled since the last crash (nginx errors). I have Plex on a br0 network with a set IP. I don't have a DHCP pool set in Docker settings. The IPv4 custom network on interface br0 (in Docker settings) is the same subnet as my router's main LAN network. And the Gateway address is my router's address. I have no DHCP pool set. (in Docker) Do I have this set wrong? This was all done per Spaceinvaderone's videos. I have Nextcloud and Swag on a "proxynet" custom network per his video as well. tower-syslog-20210121-0359.zip
February 13, 20215 yr Thank you for taking the time to troubleshoot and diagnose this issue. I would have been lost without this thread. I had a thread open for two months diagnosing my call trace issues. I found out that it is related to my 10gig PCIE NIC that I was using. I removed the NIC and all of my problems have went away. My server was going as far as to completely lock up requiring a hard reset.
February 20, 20215 yr On 10/16/2020 at 1:34 PM, Hoopster said: I used this guide to setup my br0.3 VLAN. I do not use a Docker DHCP pool and prefer to manually assign IP address in a range not in my router DHCP pool. How does this actually work in practice though. If Pihole etc is on a different network VLAN how do your computers on your network see it ? Also do you know if this problem occurs if you set the network of the container to br0 but then leave it to grab an IP via DHCP ? If so I could just put an entry in the DHCP table instead.
February 20, 20215 yr 17 minutes ago, anethema said: How does this actually work in practice though. If Pihole etc is on a different network VLAN how do your computers on your network see it ? It depends on how your router treats VLAN networks. My Edgerouter, by default allows communication across VLANs by default unless you add firewall rules to segregate them. Other routers may be different. 17 minutes ago, anethema said: Also do you know if this problem occurs if you set the network of the container to br0 but then leave it to grab an IP via DHCP ? If so I could just put an entry in the DHCP table instead. Yes, for me it still occured when my router was providing DHCP for the containers on br0.
February 20, 20215 yr Author 2 hours ago, anethema said: How does this actually work in practice though. If Pihole etc is on a different network VLAN how do your computers on your network see it ? As @adminmat stated, this depends on your router. You may have to configure firewall rules to allow network traffic to pass between VLANs, primary LAN, etc. By default mine allows all traffic between configured "corporate" LANs (just what UniFi chooses to call LANs/VLANs, - guest networks are excluded). FYI - I started out with PiHole as a docker container and switched to running it dedicated on a Raspberry Pi. Far fewer problems all around for me this way. It certainly can work well as a docker container, but, there are some "gotchas" you have to be aware of and plan around. 2 hours ago, anethema said: Also do you know if this problem occurs if you set the network of the container to br0 but then leave it to grab an IP via DHCP ? If so I could just put an entry in the DHCP table instead. It made no difference in my case. I had to create a VLAN, something other than br0. br0 and br0.3 (my VLAN) are physically on the same NIC but there is something different in Docker with it not being br0.
February 20, 20215 yr Thanks for the help I will give it a shot. I am on unifi also. I think I can probably just move unifi controller to a bridge mode and get rid of pihole. Does this also happen with VMs though? I really need my blue iris VM to have an IP heh.
February 22, 20215 yr Author On 2/20/2021 at 12:18 PM, anethema said: Does this also happen with VMs though? Seems to only be an issues with Docker containers. I have seen no reports of IP addresses assigned to VMs causing this problem.
April 9, 20215 yr Swapped my motherboard out for a ASRockRack X470D4U2-2T to use IPMI with dual 10 gig NICs and the call trace issue has returned. I am going to follow the following post and disable the "Enable VLANs:" setting under my 10 gig NIC and we will see if that makes a difference or not. Unraid 6.9.2 Router: UDM-Pro Switch: USW-Pro-24-PoE
April 12, 20215 yr Just disabling the VLAN on the NIC did not stop the issue. The dockers that need separate IP addresses I have built onto another VLAN I will watch the logs for a week or so to see if the call traces come back again or not.
October 4, 20214 yr Well here i am, also fighting this issue it seems. First i had issues with BTRFS, so i switched to XFS and a single cache drive (bummer). But now i get these stacktraces with nf_conntrack_core macvlan. Oct 2 10:55:55 Winston kernel: ------------[ cut here ]------------ Oct 2 10:55:55 Winston kernel: WARNING: CPU: 10 PID: 19944 at net/netfilter/nf_conntrack_core.c:1120 __nf_conntrack_confirm+0x9b/0x1e6 Oct 2 10:55:55 Winston kernel: Modules linked in: tun veth macvlan xt_nat xt_MASQUERADE iptable_nat nf_nat xfs ecb crypto_simd cryptd glue_helper xts dm_crypt dm_mod dax md_mod ipmi_devintf ip6table_filter ip6_tables iptable_filter ip_tables bonding igb i2c_algo_bit intel_powerclamp coretemp kvm_intel kvm i2c_i801 i2c_smbus crc32c_intel i2c_core input_leds intel_cstate led_class i5500_temp intel_uncore ipmi_si cdc_acm ata_piix i7core_edac button acpi_cpufreq [last unloaded: i2c_algo_bit] Oct 2 10:55:55 Winston kernel: CPU: 10 PID: 19944 Comm: kworker/10:0 Not tainted 5.10.21-Unraid #1 Oct 2 10:55:55 Winston kernel: Hardware name: Supermicro X8DT3/X8DT3, BIOS 2.2 07/09/2018 Oct 2 10:55:55 Winston kernel: Workqueue: events macvlan_process_broadcast [macvlan] Oct 2 10:55:55 Winston kernel: RIP: 0010:__nf_conntrack_confirm+0x9b/0x1e6 Oct 2 10:55:55 Winston kernel: Code: e8 64 f9 ff ff 44 89 fa 89 c6 41 89 c4 48 c1 eb 20 89 df 41 89 de e8 d5 f6 ff ff 84 c0 75 bb 48 8b 85 80 00 00 00 a8 08 74 18 <0f> 0b 89 df 44 89 e6 31 db e8 5d f3 ff ff e8 30 f6 ff ff e9 22 01 Oct 2 10:55:55 Winston kernel: RSP: 0000:ffffc90003414dd8 EFLAGS: 00010202 Oct 2 10:55:55 Winston kernel: RAX: 0000000000000188 RBX: 000000000000453f RCX: 0000000090f81b21 Oct 2 10:55:55 Winston kernel: RDX: 0000000000000000 RSI: 000000000000026b RDI: ffffffff8200a1ec Oct 2 10:55:55 Winston kernel: RBP: ffff88825c527cc0 R08: 00000000ea80c49f R09: ffff88834286f8e0 Oct 2 10:55:55 Winston kernel: R10: 0000000000000000 R11: ffff8883f69c9600 R12: 000000000000fe6b Oct 2 10:55:55 Winston kernel: R13: ffffffff8210db40 R14: 000000000000453f R15: 0000000000000000 Oct 2 10:55:55 Winston kernel: FS: 0000000000000000(0000) GS:ffff888633b00000(0000) knlGS:0000000000000000 Oct 2 10:55:55 Winston kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 2 10:55:55 Winston kernel: CR2: 0000151fb09ec000 CR3: 00000003af620003 CR4: 00000000000206e0 Oct 2 10:55:55 Winston kernel: Call Trace: Oct 2 10:55:55 Winston kernel: <IRQ> Oct 2 10:55:55 Winston kernel: nf_conntrack_confirm+0x2f/0x36 Oct 2 10:55:55 Winston kernel: nf_hook_slow+0x39/0x8e Oct 2 10:55:55 Winston kernel: nf_hook.constprop.0+0xb1/0xd8 Oct 2 10:55:55 Winston kernel: ? ip_protocol_deliver_rcu+0xfe/0xfe Oct 2 10:55:55 Winston kernel: ip_local_deliver+0x49/0x75 Oct 2 10:55:55 Winston kernel: __netif_receive_skb_one_core+0x74/0x95 Oct 2 10:55:55 Winston kernel: process_backlog+0xa3/0x13b Oct 2 10:55:55 Winston kernel: net_rx_action+0xf4/0x29d Oct 2 10:55:55 Winston kernel: __do_softirq+0xc4/0x1c2 Oct 2 10:55:55 Winston kernel: asm_call_irq_on_stack+0x12/0x20 Oct 2 10:55:55 Winston kernel: </IRQ> Oct 2 10:55:55 Winston kernel: do_softirq_own_stack+0x2c/0x39 Oct 2 10:55:55 Winston kernel: do_softirq+0x3a/0x44 Oct 2 10:55:55 Winston kernel: netif_rx_ni+0x1c/0x22 Oct 2 10:55:55 Winston kernel: macvlan_broadcast+0x10e/0x13c [macvlan] Oct 2 10:55:55 Winston kernel: macvlan_process_broadcast+0xf8/0x143 [macvlan] Oct 2 10:55:55 Winston kernel: process_one_work+0x13c/0x1d5 Oct 2 10:55:55 Winston kernel: worker_thread+0x18b/0x22f Oct 2 10:55:55 Winston kernel: ? process_scheduled_works+0x27/0x27 Oct 2 10:55:55 Winston kernel: kthread+0xe5/0xea Oct 2 10:55:55 Winston kernel: ? __kthread_bind_mask+0x57/0x57 Oct 2 10:55:55 Winston kernel: ret_from_fork+0x22/0x30 Oct 2 10:55:55 Winston kernel: ---[ end trace 39e64ba254a07726 ]--- Will now try to seperate my dockers into a seperate VLAN.
October 4, 20214 yr 50 minutes ago, martinjuhasz said: Will now try to seperate my dockers into a seperate VLAN. You could also try 6.10RC1, it seems there are general improvements as well as a new mode IPVLAN that you can use instead of MACVLAN.
October 4, 20214 yr cool, ok thats easier. will give that a try first than having to create new firewall rules for a new VLAN and remapping everything -.- Is there some guide or information already on IPVLAN on unraid? Improvements/Drawbacks on using it and how to use it? // edit: running RC1 now with IPVLAN as docker network type. fingers crossed Edited October 4, 20214 yr by martinjuhasz
October 26, 20214 yr I swapped out a cache drive, and did a bunch of updating — I don’t know if I moved from 6.8.3 to 6.9.2 at this time, but I was definitely running 6.9.2 when the crashes started happening a few weeks ago. I re-seated RAM, etc. and saw no improvement. I found this thread and narrowed in on a HOOBS Docker running custom on br0. Uninstalling HOOBS and switching to Homebridge docker in bridged mode has kept the system running nicely for about 5 days now. A frustrating few weeks of crashes, but glad that it seems to be stable now. Not 100% sure that it was macvlan calls (server runs headless, and never got the syslog server running before fixing the HOOBS docker), but assume it was. EDIT: Ten days later without a crash. Looking good. Edited October 31, 20214 yr by imagoodusername Updating to confirm stability.
November 29, 20214 yr Hi Guys! I'm so glad I'm not alone in this. I couldn't understand for a very long time why my Server will become "Nuked" after anytime between 24 hours through to 7 days sometimes. I have a good handful of Dockers using Set IP's. of course, don't need it for some of the Dockers but its for a reason / preference [Have a good few more than this on that specific server so it will tend to crash way before my 2nd Server] H Hope this gets Patched or something for a long term fix. I have another Server with a similar config-ish and have the same issue but far less prevalent (more likely to crash 3days+) Thanks All! Edited November 29, 20214 yr by Stan464
November 29, 20214 yr 1 hour ago, Stan464 said: Hope this gets Patched or something for a long term fix. As previously stated, the 6.10 branch offers the IPVLAN option instead of MACVLAN. People report better stability. You can try the RC2.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.