[GUIDE] How to solve macvlan and ipvlan issues with containers on a custom network

[bonienl]

Some users experience kernel panics when they have Docker containers set up using a custom network connection, which uses by default a docker macvlan network type.

 

In such a case the general advice is to switch the connection to a docker ipvlan network type, which usually solves the issue, but for some users may introduce a network connectivity issue, depending on the network equipment (router) in use and if it can handle the specifics of ipvlan.

 

In such a situation neither macvlan nor ipvlan is a 100% perfect solution and becomes a trade-off between stability and connectivity.

To counter this situation the following solution is proposed to solve the macvlan / ipvlan issue once and for all.

 

THE SOLUTION - USE A DEDICATED ETHERNET PORT FOR DOCKER ONLY

 

You will need an additional ethernet port of your server to make this solution work. This additional port is connected to your local router or switch just like the main port. No network modifications are required for your main connection eth0/br0 unless it is configured as bond or bridge interface with multiple ports and you need to free up one port from the bond or bridge interface and turn it into a dedicated port for docker connections.

 

1. Configure the dedicated interface in network settings (array must be stopped).

    - Enable bridging for this interface

    - Use IPv4 only or IPv4 and IPv6 as per your case

    - No IP addresses are assigned to this interface

 

 

2. Configure Docker to use this dedicated interface

    - Use default values for custom network

 

 

    - Disable the IP assignment(s) of eth0 / br0 which is going to be replaced

 

 

   - Assign manually the "old" assignments to the new dedicated interface

 

 

3. Configure the Docker containers with the new custom network

    - IP addresses may be fixed (as in the example below) or dynamic using DHCP (configured in the previous step)

 

 

THAT'S IT - NOW YOUR DOCKER CONTAINER(S) OPERATE ON A DEDICATED CUSTOM NETWORK 

 

[FredrikJL]

I've tried a bunch of different things to get rid of messages like this without any luck. I usually get one or two per day, as well as a random crash per week or so. 

Mar  2 22:21:04 Unraid kernel: WARNING: CPU: 1 PID: 7504 at net/netfilter/nf_conntrack_core.c:1208 __nf_conntrack_confirm+0xa5/0x2cb [nf_conntrack]
Mar  2 22:21:04 Unraid kernel: Modules linked in: xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap macvlan xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs md_mod ipmi_devintf jc42 efivarfs ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls igb intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ast drm_vram_helper drm_ttm_helper ttm aesni_intel crypto_simd drm_kms_helper ipmi_ssif cryptd i2c_i801 intel_cstate joydev i2c_smbus drm input_leds backlight led_class agpgart syscopyarea sysfillrect sysimgblt i2c_algo_bit acpi_ipmi ahci fb_sys_fops i2c_core libahci ipmi_si button acpi_cpufreq unix [last unloaded: igb]

 

Implemented the changes suggested above and no problems so far. Three days ago and counting...

On an ASRock Rack C2750D4I that never gives my enough sleep.

[Bizquick]

Is this what I need to do if I followed SpaceInvader's old videos on setting up like my own BitWarden server etc... I'm just getting that message now in Fix Common problems and not sure what steps I'm going to need to take to switch from MacVlan to IPVLan.

[FredrikJL]

This is what I did to fix my macvlan. Setting up/using ipvlan never worked for me

[Omri]

Hi

Tried to follow your guide

But I guess I miss something

There isn't br1 in my containers config

Although I configured eth1 like the screenshot you provided and configured docker setting with br1 instead of br0.

 

[FredrikJL]

8 hours ago, Omri said:

Hi

Tried to follow your guide

But I guess I miss something

There isn't br1 in my containers config

Although I configured eth1 like the screenshot you provided and configured docker setting with br1 instead of br0.

 

Make sure to have “IPv4 address assignment:” set to “none” for your eth1. (As well as for ipv6). 

 

[FredrikJL]

[Omri]

I do

Still br1 doesn't exist in containers config

It doesn't listed in routing table also

Do I need to create a custom network via cli?

Edited April 15 by Omri

[wassereimer]

6 hours ago, Omri said:

I do

Still br1 doesn't exist in containers config

It doesn't listed in routing table also

Do I need to create a custom network via cli?

 

Same problem for me.

[Omri]

@bonienl

Hi

Can you post a screenshot of your routing table under network settings?

Thanks

[insomnia417]

这不是个很完美的解决方案，我折腾了2周，被死机搞烦了，现在已经降级回6.11.5，等正式版内核看看会不会修复macvlan的bug

[wassereimer]

12 hours ago, insomnia417 said:

这不是个很完美的解决方案，我折腾了2周，被死机搞烦了，现在已经降级回6.11.5，等正式版内核看看会不会修复macvlan的bug

 

 @insomnia417 said (from deepl): This is not a perfect solution, I tossed 2 weeks, was bored by the crash, and now have downgraded back to 6.11.5, waiting for the official kernel to see if the macvlan bug will be fixed

Edited April 17 by wassereimer
@ corrected

[FredrikJL]

I got this working and the macvlan issue remains solved a couple of weeks later. No crashes since I implemented this (on 11.5). Uptime is 3 weeks and counting or the first time ever. Sorry to hear that there seems to be problems for people to get this configured as described by @bonienl

[wassereimer]

On 3/28/2023 at 7:05 PM, bonienl said:

 

    - Disable the IP assignment(s) of eth0 / br0 which is going to be replaced

 

 

   - Assign manually the "old" assignments to the new dedicated interface

 

 

 

I found something. You write "Assign manually the "old" assignments to the new dedicated interface" but you didn't do that. Look at the ipv6 Gateway at br0 and br1. And it works for me if I don't enable the ipv6 custom network on br1, but just ipv4. Then Unraid creates the br1 macvlan network in docker (but you can't see it in the routing table). So there must be something wrong with taking the "old" assignments from br0.

Edited April 17 by wassereimer

[Omri]

I think it's a bug in 6.12 (rc3)

BR1 isn't being created

even when I disable bridging on eth0 and enabling it only on eth1

docker filling the right subnet/gateway on br1 but docker doesn't create br1 network (although it's visible in ifconfig)

will be glad if someone who got it working will post docker.cfg and network.cfg from /boot/config

 

Thanks in advance

[wassereimer]

Yeah. Tested it further and it doesn't work. I'm also on 6.12 (rc3). Did you create a bug report if you think it is one? I don't know if it would work in a stable version, because I'm new to Unraid and started with 6.12 because of zfs stuff.

[bonienl]

3 hours ago, wassereimer said:

Tested it further and it doesn't work. I'm also on 6.12 (rc3).

 

It works with Unraid version 6.12. Please upload your diagnostics, likely there is a configuration error.

 

[Omri]

@bonienl

Can you share docker.cfg and network.cfg?

Thanks

[wassereimer]

5 hours ago, bonienl said:

 

It works with Unraid version 6.12. Please upload your diagnostics, likely there is a configuration error.

 

 

Of course. Thank you for looking into it. If I change the "IPv4/IPv6 address assignment" to "Automatic", the br1 Network is correctly created and visible in the routing table. If set to none, that doesn't happen. Even after setting everything in the Docker Settings and enabling the Service.

homeserver-diagnostics-20230418-1850.zip

[Omri]

Here is my diagnostics

bridging enabled only for eth1

br1 is visible with right settings in routing table

is visible in ifconfig

can't select it in container settings

 

 

Edited April 21 by Omri
Removed attachment

[ceyo14]

Thanks for posting this, I had been using ipvlan without issue with Untangle and OPNsense, but recently switched to Sophos XG and it immediately had issues with it, I changed it to macvlan and have had no further issues but remember I switched to ipvlan for a reason, I mean I can't remember why now but I was curious why Sophos didn't like ipvlan... but this seems like a much better option regardless.

[Omri]

On 4/18/2023 at 2:05 PM, bonienl said:

 

It works with Unraid version 6.12. Please upload your diagnostics, likely there is a configuration error.

 

Did you find the configuration error?

[Omri]

Well, figured out a solution (although not perfect)

eth0 was using automatic setting for IPV4+IPV6

after setting it to static, docker network br1 appeared and it's working (IPV4+IPV6).

 

eth1 is using automatic setting for IPV4+IPV6 so I didn't need to set addresses in docker settings (which is good because my IPV6 prefix isn't fixed, and a manual setting would not be valid after my prefix will change).

 

The only "problem" now is that Unraid GUI has two IP's in same subnet.

[wassereimer]

On 4/18/2023 at 1:05 PM, bonienl said:

 

It works with Unraid version 6.12. Please upload your diagnostics, likely there is a configuration error.

 

 

Did you had time to look into the data?  

[Omri]

Don't use auto IPV4/IPV6 on eth0 - use static and define IP/gateway/dns

than you need to feel the subnets in docker settings under br1 (but don't check the pool - just the subnet and gateway)