MACVLan vs IPVLan a discussion.....

[aglyons]

I run a lot of Unifi hardware in my network. Pretty much the whole sha-bang.

 

Running Unraid using IPVLan is confusing Unifi to no ends as clients listed will flip from one DNS name to another under the same IP address.

 

So, I started digging around. 

 

It was originally suggested to move to IPVLan as the MACVlan implementation was causing Kernel Panic freezes with Unraid. This would lock up the system. As such, apparently MACVlan was going to be removed from future versions of Unraid.

 

But, I see that even in v6.11.5, both IPVLan and MACVlan are still an option. There are also links to descriptions as to which situations to use each one. Has the issues with Kernel Panic been resolved using MACVlan on Unraid?

 

Reading the Docker docs on both networks it reads to me that MACVLan is the more appropriate choice given the need to have a container on the hosts physical network. With one caveat which I am still trying to figure out regarding Unifi hardware. "Your networking equipment needs to be able to handle “promiscuous mode”, where one physical interface can be assigned multiple MAC addresses". I have confirmed that the NIC in the Unraid server (AQC107 NBase-T/IEEE 802.3bz Ethernet Controller) does support promiscuous and it is enabled in the driver.

 

Quote

Some applications, especially legacy applications or applications which monitor network traffic, expect to be directly connected to the physical network. In this type of situation, you can use the macvlan network driver to assign a MAC address to each container’s virtual network interface, making it appear to be a physical network interface directly connected to the physical network.

 

This one part stood out to me that would solve the Unifi network confusion is that apparently is states that "you can use the macvlan network driver to assign a MAC address to each container’s virtual network interface". Does this happen automatically in Unraid? Is this something that needs to be added as a variable to the Docker containers templates?

 

In summary, some answers I'm hoping someone can answer,

 

1. Has the issues with Kernel Panic been resolved using MACVlan on Unraid?

2. Does MACVLan mode assign a virtual MAC address to the containers?

     2b. If not, is this a variable that we need to add to containers templates?

 

3. If using MACVlan, do you 'have' to use the internal DOcker DHCP or can you use the host network DHCP (with reserved IP's) to assign addresses on the containers? Could this implementation be the root cause of the Kernel Panic issues? (because that's how I had it set up originally)

 

Thanks all in advance. Looking forward to the discussion.

 

A.

 

[Hoopster]

52 minutes ago, aglyons said:

In summary, some answers I'm hoping someone can answer

Some general observations (not direct answers to your questions):

 

1. I also have Unifi hardware exclusively in my LAN (USG, two switches, five access points)

2. I used to get frequent server crashes (macvlan broadcast call traces) with macvlan (this was years before ipvlan was even an option).

3. The call traces happened when I had docker containers with IP addresses on br0 (no problems if they were in host or bridge mode using server IP address)

4. I created a VLAN (br0.3) in Unifi and unRAID just for docker containers and the macvlan call traces/server crashes disappeared

5. I switched docker network type over to ipvlan as soon as it was an option (although I suspect it was not necessary as I had already resolved the problem with macvlan)

6. As far as I have observed, I have experienced no DNS issues with Unifi hardware and ipvlan but then I am doing nothing in particular with it other than pointing clients to PiHole for DNS

7. I assign IP addresses to docker containers manually and do not have a DHCP pool set for the Docker network in unRAID.  The DHCP range in Unifi controller is set to not include the range of addresses I assign to docker containers

 

I cannot answer the questions about how unRAID and its implementation of Docker behaves with respect to network interfaces, MAC addresses, etc.

 

For users experiencing server lockups related to macvlan, the recommendation is to switch to ipvlan and it resolves the issue in most cases.

Edited December 29, 2022 by Hoopster

[BRiT]

From what I recall, (some details may be off, but overall it's very close). I don't have any screenshots of how I had macvlan setup, so I can't validate exactly how it behaved before.

 

Docker would not use DHCP from an outside DHCP server, so if you are setting macvlan or even ipvlan you still need to reserve the static ipaddress in your DHCP server and setup reverse dns if you want to resolve from a Name to ip lookups. I always had to setup my router/dhcp server in addition to the static configuration settings for each individual docker container.

 

Switching over to IPVLAN only required changes on each individual Docker container to remove the mac-address settings as the containers refused to start with the Mac address specified.

 

[schale01]

After a recent upgrade to 6.11.5 I started getting server lock ups.  No webgui, no ssh, no ping response, not even output from console.  It seemed to happen when using apps such as Plex or transmission.  Never had the issue before so I've currently rolled back to 6.10.3 and am no longer seeing issues.  I'm starting to think that the recent kernel update really hates macvlan more then ever.  When I have some time I'll try to update again and switch over all my Dockers to ipvlan to confirm if that was the cause.

[aglyons]

59 minutes ago, schale01 said:

After a recent upgrade to 6.11.5 I started getting server lock ups.  No webgui, no ssh, no ping response, not even output from console.  It seemed to happen when using apps such as Plex or transmission.  Never had the issue before so I've currently rolled back to 6.10.3 and am no longer seeing issues.  I'm starting to think that the recent kernel update really hates macvlan more then ever.  When I have some time I'll try to update again and switch over all my Dockers to ipvlan to confirm if that was the cause.

 

This was happening months ago. I posted about it and was advised to switch to IPVLan. But, so far I am seeing really odd things happening with my network which I suspect is due to IPVLan. I've read up on how the two methods operate and with MACVlan, each container gets a virtualized MAC address that the host network can see. With IPVLan, they don't. They share the MAC address of the host NIC. Some NICs don't play nice as they cannot operate in promiscuous mode (which allows for multiple IP addresses associated with the same NIC). Not only does the NIC need to support that mode but also all the other hardware on your network.

 

While I am using Ubiquiti throughout my network. I suspect that they are currently having issues with how they are handling that.

 

All that being said, switching over to IPVLan did stop the kernel panic issues with Unraid. 

 

[Waddoo]

I have been getting there macvlan kernel panics, at least I think I am...

 

Mar 22 19:39:10 Alexandria kernel: ------------[ cut here ]------------
Mar 22 19:39:10 Alexandria kernel: WARNING: CPU: 5 PID: 1298 at net/netfilter/nf_conntrack_core.c:1120 __nf_conntrack_confirm+0x9b/0x1e6 [nf_conntrack]
Mar 22 19:39:10 Alexandria kernel: Modules linked in: nvidia_uvm(PO) xt_mark macvlan xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle nf_tables vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xfs nfsd lockd grace sunrpc md_mod nvidia_drm(PO) nvidia_modeset(PO) drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops nvidia(PO) drm backlight agpgart iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables mlx4_en mlx4_core r8169 realtek edac_mce_amd kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd mpt3sas wmi_bmof glue_helper i2c_piix4 wmi k10temp raid_class scsi_transport_sas i2c_core
Mar 22 19:39:10 Alexandria kernel: ccp rapl ahci libahci button acpi_cpufreq [last unloaded: mlx4_core]
Mar 22 19:39:10 Alexandria kernel: CPU: 5 PID: 1298 Comm: kworker/5:1 Tainted: P S         O      5.10.28-Unraid #1
Mar 22 19:39:10 Alexandria kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B79/X470 GAMING PLUS MAX (MS-7B79), BIOS H.60 06/11/2020
Mar 22 19:39:10 Alexandria kernel: Workqueue: events macvlan_process_broadcast [macvlan]
Mar 22 19:39:10 Alexandria kernel: RIP: 0010:__nf_conntrack_confirm+0x9b/0x1e6 [nf_conntrack]

 

I am also using a UDMP-SE. How would I go about switching to IPVLan? Reading through threads it reads like it is something to be changed in Unraid?

[aglyons]

The option is in the Unraid Docker settings. You'll need to stop Docker first, then you can change the option.

 

I don't think you can see it until Docker is down.

Edited March 24, 2023 by aglyons

[tjb_altf4]

@Waddoo further to the above reply, you need to stop the Docker service, not just running docker containers.

To do this:

• go to SETTINGS > DOCKER
• with advanced view toggled (top right)
• set Enable Docker: No,
• then set Docker custom network type: ipvlan

Once complete:

• set Enable Docker: Yes to restart docker service.

Report back with fresh diagnostics if kernel panics continue.

[Waddoo]

Will the changing of the network type affect my driver network as it is? Or nginx for example?

[tjb_altf4]

1 hour ago, Waddoo said:

Will the changing of the network type affect my driver network as it is? Or nginx for example?

Only if you are using the mac addresses assigned to those containers for some purpose, so for nginx and other things in layer 3 it won't/shouldn't have an impact.

The one thing I miss from using macvlan is seeing each container appear in my router software (unifi), whereas ipvlan all share the host/interface mac.

[aglyons]

1 hour ago, tjb_altf4 said:

The one thing I miss from using macvlan is seeing each container appear in my router software (unifi), whereas ipvlan all share the host/interface mac.

 

That's my biggest issue with IPVLAN. I'm not 100% sure if it might be causing problems with Unifi, specifically Unifi's threat detection/blocking. I'm not 100% sure how they approach this feature, whether it is IP based or MAC based and how it affects what I will call a 'floating MAC'; the same MAC for multiple systems.

[cpthook]

On 3/23/2023 at 10:13 PM, tjb_altf4 said:

@Waddoo further to the above reply, you need to stop the Docker service, not just running docker containers.

To do this:

• go to SETTINGS > DOCKER
• with advanced view toggled (top right)
• set Enable Docker: No,
• then set Docker custom network type: ipvlan

 

 

I also began to receive Kernel panics after upgrading to 6.11.5.  Is making this change the only thing required for configuration?  When I made the change I found that containers no longer had access to the public network.  More specifically GITHUB and were not able to update.  I must be missing something?   

[aglyons]

I have noticed that when you make this level of change to the Docker network, you will have to go through all of your containers networking config. You might find that IP addresses may have changed or the network type may be missing. I've even seen it where it all looks right but doesn't work until you force update the container for it to be rebuilt.

[Waddoo]

32 minutes ago, cpthook said:

When I made the change I found that containers no longer had access to the public network.

All of my"public" containers remain accessible through my domain on nginx. I'm not sure if that is because I do not hard code IPs but unsure of y'all's setup.

[aglyons]

9 minutes ago, Waddoo said:

because I do not hard code IPs

 

I'll make the assumption that you are running containers in bridge mode with br0. This would also include you NPM container. Another assumption, all the "services" you are running are talking to each other via the internal Docker network, 172.xxx.xxx.xxx. It would also mean that you access all of your services through the hosts IP with different port assignments, 192.168.xxx.xxx:port. If that is the case and you are not able to access those services, I would check the network settings in UR and look at the bridge settings there. Although it is interesting that you state you can access them through the domain.

 

I run things a little different in that I have most public accessible services in their own VLAN running from a 2nd NIC, seperate from the hosts network. This keeps them segregated from my local network systems. In that instance, each service gets its own dedicated IP on that VLAN manually assigned. Docker containers either don't request or respond to DHCP so they won't be assigned an IP. I wish that wasn't the case as I would rather reserve IP's in my router. 

[bonienl]

I made a guide how to set up a dedicated connection for Docker containers on a custom network.

This solution solves the issues some users are having with macvlan or ipvlan.

 

See 

 

 

[cpthook]

Thank you!!  This guide worked for me.  I had to set static IP's for all docker containers though.  But no kernel errors so far.  Fingers crossed

Edited March 29, 2023 by cpthook

[BurntOC]

@bonienl  I have been using macvlan since like day 1, and while I did have to move some things off br0 a couple of years ago due to crashes.  I have br0, br2, and br3 now.  I don't have any containers running on br0, just a couple on bridge and the others on br2 and br3, but I am getting the macvlan call traces error from FCP.  Is that just something that can be ignored, or do I gain any sort of network benefits from ipvlan if things aren't crashing (e.g. fewer broadcasts or such)?

[bonienl]

The macvlan issues occur on the shared interface br0. If no containers make use of this interface then disable it in the docker settings, this should help.

 

[BurntOC]

Okay, so I decided to just switch to ipvlan on one of my Unraid servers as a test.  Today I wake up to find I am still getting macvlan call trace warnings from FCP.  Any idea what's up with that?  To be clear, it's not a timing of the message - I ran a check again manually and it is reporting the same thing, but I also checked my Docker settings and yup, ipvlan.

Edited April 2, 2023 by BurntOC

[Squid]

FCP will still see any previous macvlan traces in the log.  Rebooting clears them out.

[BurntOC]

Got it.  I thought it was just checking events since the last run.  Good to know.

[nraygun]

I've had no problems prior to going to 6.11.5.

I recently shrank my array to remove a 500GB drive and started seeing the macvlan message in FCP.

I tried changing the custom network type to ipvlan as suggested in a few posts, but then that almost renders my server inoperable. My host can't ping host names(www.google.com, www.microsoft.com, etc.). And the Apps area doesn't update its contents(probably because it can't resolve host names).

 

Also, my DuckDNS container won't update and I can't ping hosts inside of that either.

 

Putting the custom network back to macvlan restores operation of my host and the DuckDNS container(updates and pings hosts).

 

Even though I get what appears to be some sort of crash in the logs, things are operational. Containers work, GUI is fine, shares are fine, etc.

 

Should I just stay with macvlan and leave well enough alone?

Edited April 23, 2023 by nraygun

[nraygun]

On 3/23/2023 at 8:39 PM, Waddoo said:

I have been getting there macvlan kernel panics, at least I think I am...

 

Mar 22 19:39:10 Alexandria kernel: ------------[ cut here ]------------
Mar 22 19:39:10 Alexandria kernel: WARNING: CPU: 5 PID: 1298 at net/netfilter/nf_conntrack_core.c:1120 __nf_conntrack_confirm+0x9b/0x1e6 [nf_conntrack]
Mar 22 19:39:10 Alexandria kernel: Modules linked in: nvidia_uvm(PO) xt_mark macvlan xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle nf_tables vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xfs nfsd lockd grace sunrpc md_mod nvidia_drm(PO) nvidia_modeset(PO) drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops nvidia(PO) drm backlight agpgart iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables mlx4_en mlx4_core r8169 realtek edac_mce_amd kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd mpt3sas wmi_bmof glue_helper i2c_piix4 wmi k10temp raid_class scsi_transport_sas i2c_core
Mar 22 19:39:10 Alexandria kernel: ccp rapl ahci libahci button acpi_cpufreq [last unloaded: mlx4_core]
Mar 22 19:39:10 Alexandria kernel: CPU: 5 PID: 1298 Comm: kworker/5:1 Tainted: P S         O      5.10.28-Unraid #1
Mar 22 19:39:10 Alexandria kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B79/X470 GAMING PLUS MAX (MS-7B79), BIOS H.60 06/11/2020
Mar 22 19:39:10 Alexandria kernel: Workqueue: events macvlan_process_broadcast [macvlan]
Mar 22 19:39:10 Alexandria kernel: RIP: 0010:__nf_conntrack_confirm+0x9b/0x1e6 [nf_conntrack]

 

I am also using a UDMP-SE. How would I go about switching to IPVLan? Reading through threads it reads like it is something to be changed in Unraid?

 

I get similar entries in my log but the system continues to run.

Does your server crash?

 

Here's what I get:

Apr 22 22:57:49 flores kernel: WARNING: CPU: 23 PID: 3924 at net/netfilter/nf_conntrack_core.c:1208 __nf_conntrack_confirm+0xa5/0x2cb [nf_conntrack]
Apr 22 22:57:49 flores kernel: Modules linked in: macvlan vhost_net tun vhost tap kvm_intel kvm xt_mark xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_iotlb xt_nat xt_tcpudp veth ipvlan xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod tcp_diag inet_diag ipmi_devintf iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bnx2 mgag200 drm_shmem_helper i2c_algo_bit drm_kms_helper drm sr_mod cdrom ipmi_ssif backlight intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd intel_cstate intel_uncore mpt3sas i2c_core input_leds ahci syscopyarea led_class joydev sysfillrect ata_piix
Apr 22 22:57:49 flores kernel: sysimgblt libahci fb_sys_fops raid_class scsi_transport_sas wmi ipmi_si acpi_power_meter button acpi_cpufreq unix [last unloaded: kvm]
Apr 22 22:57:49 flores kernel: CPU: 23 PID: 3924 Comm: kworker/23:1 Tainted: G        W I       5.19.17-Unraid #2
Apr 22 22:57:49 flores kernel: Hardware name: Dell Inc. PowerEdge R710/0YDJK3, BIOS 6.6.0 05/22/2018
Apr 22 22:57:49 flores kernel: Workqueue: events macvlan_process_broadcast [macvlan]
Apr 22 22:57:49 flores kernel: RIP: 0010:__nf_conntrack_confirm+0xa5/0x2cb [nf_conntrack]

 

[Waddoo]

On 4/23/2023 at 1:28 AM, nraygun said:

 

I get similar entries in my log but the system continues to run.

Does your server crash?

 

Yes, in-fact it just crashed recently...

 

Apr 25 12:04:41 Alexandria kernel: WARNING: CPU: 9 PID: 20303 at net/netfilter/nf_conntrack_core.c:1120 __nf_conntrack_confirm+0x9b/0x1e6 [nf_conntrack]
Apr 25 12:04:41 Alexandria kernel: Modules linked in: nvidia_uvm(PO) xt_mark macvlan xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle nf_tables vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xfs nfsd lockd grace sunrpc md_mod nvidia_drm(PO) nvidia_modeset(PO) drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops nvidia(PO) drm backlight agpgart iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables mlx4_en mlx4_core r8169 realtek edac_mce_amd kvm_amd kvm mpt3sas crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd wmi_bmof i2c_piix4 glue_helper i2c_core wmi raid_class scsi_transport_sas k10temp
Apr 25 12:04:41 Alexandria kernel: ccp ahci rapl libahci button acpi_cpufreq [last unloaded: mlx4_core]
Apr 25 12:04:41 Alexandria kernel: CPU: 9 PID: 20303 Comm: kworker/9:2 Tainted: P S         O      5.10.28-Unraid #1
Apr 25 12:04:41 Alexandria kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B79/X470 GAMING PLUS MAX (MS-7B79), BIOS H.60 06/11/2020
Apr 25 12:04:41 Alexandria kernel: Workqueue: events macvlan_process_broadcast [macvlan]
Apr 25 12:04:41 Alexandria kernel: RIP: 0010:__nf_conntrack_confirm+0x9b/0x1e6 [nf_conntrack]

 

Unsure what to do next about this issue, we aren't even running similar hardware based on your logs.