vhost0@eth0 is using the same IP as eth0 -> this is an IP collision, how to do better?

[murkus]

arpwatch and truenas are complaining that the IP of my unraid server is flapping between the MAC of its eth0 (one of 4 interfaces, but eth1-3 are not used) and vhost0@eth0. And from all I know about IP networking having two interfaces with different MACs using the same IP is not a good condition.

 

I have investigated a little and it looks like vhost0@eth0 is there even if docker is off, but then it has no IP assigned. If I turn docker on, it gets the IP of eth0 assigned. (VM manager is using virbr0, which uses a different IP and also goes away when turnin VM manager off)

The curious thing is that vhost0 hast a routing metric of 0 and is thus preferred over eth0 (metric 1007).

 

Such a construct is unseen on all of my Debian docker hosts, so it is certainly possible to have docker working fine without such a unsane setup.

 

Is it possible to run unraid with docker in a way that each interface has an on IP (without assigning individual non-docker-network IPs to the containers), and how?

 

[ich777]

1 hour ago, murkus said:

Is it possible to run unraid with docker in a way that each interface has an on IP (without assigning individual non-docker-network IPs to the containers), and how?

You should really include your Diagnostics since no one knows how your Unraid server is configured in terms of network.

 

This would be also be better suited in the Bug Reports subforums however, I think you are the first that reports such an issue.

@JorgeB & @itimpi & @JonathanM did you hear about that before?

[JorgeB]

Not that I recall, if @bonienlis around he may be able to confirm if it could be a bug, but like mentioned please post the diagnostics.

[murkus]

I am not the first, when I searched for the problem in the forum I found 5+ threads of people reporting this and asking for help and others commenting +1. Unfortunately none of the threads received any working help.

[JorgeB]

In that case recommend creating a bug report.

[murkus]

I investigated further wit different settings for Docker and I found that vhost0@eth0 will only then get an IP (and then it is the same IP as eth0) if both of these are enabled (if only one of these is enabled, vhost0 does not get an IP):

• Host access to custom networks
• IPv4 custom network on interface eth0 (optional)

I have no ides why both were enabled on my server (and it is quite possible I did this myself without knowing what I was doing).

 

Just to clarify whether I really need those, could someone confirm or correct my assumptions:

• IPv4 custom network on interface eth0 (optional): I only need this, if I want to have a containter to use an IP from the subnet in which the IP of eth0 is located. Correct?
• Host access to custom networks: I only need this if the unraid host should access a service provided by some of the containers running on the the unraid host. Example an agent running natively on the unraid host needs to connect to a service (like NMS or backup) in a container. Correct?

 

 

[murkus]

And I thought I had enable ipvlan, but actually macvlan is enabled. ipvlan is shown as an option, but it is greyed out and cannot be selected. Why would that be the case?

[ljm42]

11 minutes ago, murkus said:

And I thought I had enable ipvlan, but actually macvlan is enabled. ipvlan is shown as an option, but it is greyed out and cannot be selected. Why would that be the case?

 

Most likely, you disabled bridging in order to enable macvlan. If you want to switch to ipvlan you need to re-enable bridging first. For more info see:

https://docs.unraid.net/unraid-os/release-notes/6.12.4/#fix-for-macvlan-call-traces

 

Being able to enable macvlan without getting call traces/crashes was a big enhancement of 6.12.4. A side effect of the network changes is that your router may complain about duplicate IPs on different interfaces. Aside from the complaint from your router it should not cause any actual problems.

[JorgeB]

15 minutes ago, murkus said:

ipvlan is shown as an option, but it is greyed

That will happen if bridging is disabled, enable bridging and you can use ipvlan, and with ipvlan you should no longer see the IP collision warning, I believe that only for users that really need to use macvlan, the IP collision can be an issue.

[murkus]

@ljm42 @JorgeB thanks I got that working now.

For the benefit of all people who had the same problem and posted to a bunch of other threads (which have been leinked to this thread now) here is my summary of what I have learnt:

 

• to be able to use macvlan driver for docker: disable bridging in network settings; here eth0 and possibly vhost0@eth0 have the IP address of the unraid host
• to be able to use ipvlan driver for docker: enable bridging in network settings; here eth0 has no IP address, but is connected to br0 which has the IP address of the unraid host
• to avoid vhost0@eth0 to use the same IP as eth0 (will be alarmed by arpwatch, pfSense, TrueNAS, etc.): do no NOT enable BOTH of IPv4 custom network on interface eth0 (optional) (default is ON) and Host access to custom networks (default is OFF)

I am currently using these settings successfully:

• in this setting eth0-eth3 have no IP address but are connected to the bridge br0, which has the IP address of the unraid host 
• network settings: Bridging: yes
• docker settings: Docker custom network type: ipvlan;  Host access to custom networks: disabled; IPv4 custom network on interface eth0 (optional): enabled

[aglyons]

On 3/7/2024 at 6:40 PM, murkus said:

to avoid vhost0@eth0 to use the same IP as eth0 (will be alarmed by arpwatch, pfSense, TrueNAS, etc.): do no NOT enable BOTH of IPv4 custom network on interface eth0 (optional) (default is ON) and Host access to custom networks (default is OFF)

 

Does this step not interfere with the new workaround to solve the MACVLAN call trace issues? In the config docs it states "Settings > Docker > Host access to custom networks = Enabled". If I recall correctly, this was required to allow the workaround to function properly as a custom network is created when docker is started.

 

[murkus]

On 3/9/2024 at 11:57 PM, aglyons said:

 

Does this step not interfere with the new workaround to solve the MACVLAN call trace issues? In the config docs it states "Settings > Docker > Host access to custom networks = Enabled". If I recall correctly, this was required to allow the workaround to function properly as a custom network is created when docker is started.

 

 

no problems here so far, but note that I have enabled bridging and I use ipvlan driver for Docker.
The recommendation you mentioned is for the case when people use macvlan driver for Docker and no bridging because they otherwise had problems with network equipement (Fritzbox, Unifi).
However, I do not experience problems with the Unifi network application and the IP-flapping issue with arpwatch and truenas is resolved now, although I am using ipvlan.
 

Edited March 12 by murkus

[murkus]

On 3/9/2024 at 11:57 PM, aglyons said:

 

Does this step not interfere with the new workaround to solve the MACVLAN call trace issues? In the config docs it states "Settings > Docker > Host access to custom networks = Enabled". If I recall correctly, this was required to allow the workaround to function properly as a custom network is created when docker is started.

 

maybe @ljm42 or @JorgeB have some comment about your concern

[aglyons]

On 3/7/2024 at 12:03 PM, ljm42 said:

 

Most likely, you disabled bridging in order to enable macvlan. If you want to switch to ipvlan you need to re-enable bridging first. For more info see:

https://docs.unraid.net/unraid-os/release-notes/6.12.4/#fix-for-macvlan-call-traces

 

Being able to enable macvlan without getting call traces/crashes was a big enhancement of 6.12.4. A side effect of the network changes is that your router may complain about duplicate IPs on different interfaces. Aside from the complaint from your router it should not cause any actual problems.

 

I think this may be causing problems when it comes to VM's going through the MACVTAP. I posted a new thread as it's similar but not the same.

 

 

What do you think @JorgeB ?

 

[JorgeB]

I don't know, sorry, @bonienl should be able to answer that, maybe @ljm42

[nicx]

any solution when using macvlan instead of ipvlan? I have the same problem. and as far as I understood the Unraid howto I should enable both options "Host access to custom networks" and "IPv4 custom network on interface eth0 (optional)" when using macvlan. but with this options enabled, I have the problem with duplicate IP address.

[ich777]

3 hours ago, nicx said:

any solution when using macvlan instead of ipvlan? I have the same problem. and as far as I understood the Unraid howto I should enable both options "Host access to custom networks" and "IPv4 custom network on interface eth0 (optional)" when using macvlan. but with this options enabled, I have the problem with duplicate IP address.

If you are on 6.12.11 or on 7.0.0-beta.2 it is now safe again to enable macvlan in Docker and the bridge in the Network settings.

[nicx]

@ich777 thanks for that hint. but what is "best practice" when using macvlan?

 

with

• "bridge" diabled on eth0
• "Host access to custom networks" disabled
• "IPv4 custom network on interface eth0 (optional)" enabled

it seems to work.

 

why should I enable "bridge"?

why should I enable "Host access to custom networks"?

 

 

 

 

[ich777]

1 hour ago, nicx said:

but what is "best practice" when using macvlan?

Before version 6.12.11 it was best practice to disable Bridge (in network settings) in combination with macvlan because a bug in the Kernel caused Kernel traces which lead to crashing the system.

This bug is now solved and won‘t crash the system anymore so it is safe to use the Bridge (in network settings) with macvlan (in Docker settings) on Unraid 6.12.11+
 

1 hour ago, nicx said:

why should I enable "bridge"?

This is up to you.

 

1 hour ago, nicx said:

why should I enable "Host access to custom networks"?

So that containers that you assign a static IP on br0 or eth0 can talk to the host or other containers which are on the default bridge network.

If you are not enabling Host access to custom networks containers on br0 or eth0 won‘t be able to communicate with containers on the bridge network (please note this paragraph is about Docker networks).

 

 

[nicx]

What I want to achieve: Some containers with own IP address, some containers bridged to the unraid host ip. All containers should communicate to the other containers.

 

just did a few tests regarding this: is it possible in ANY way to have fixed IPs for docker containers in my main network without having duplicate IP address for the unraid host in the unifi router when activated "Host access to custom networks" (which is needed in any case for the inter-container-communcation as far as I understood)?

 

I get either eth0 and vhost0 with the same ip address, or br0 and shim-br0. In both cases my Unifi router complains about duplicate IP address in the network.

 

@ich777 any further hints?

Edited July 19 by nicx

[ich777]

On 7/18/2024 at 10:54 AM, nicx said:

@ich777 any further hints?

If you are using macvlan in Docker then Docker creates a virtual network adapter on the physical network interface from the host and each individual container gets it's own MAC address so to speak you have multiple MAC addresses for on IP address (which is nothing to worry about since this is on most routers/firewalls just a information, only a few routers/firewalls have issues with that eg: Fritzbox).

 

If you are using ipvlan (I think this is done on the Layer 3 from the network stack) in Docker then Docker only has one IP and one MAC address which are all shared between all containers (the MAC address from the host) regardless if you assign the container a dedicated IP they will always have the same MAC address.

 

However, on 6.11.12 and 7.0.0-beta.2 it is save to enable the network Bridge (in the network settings) and to use macvlan in Docker since as said above the bug in the Kernel has been fixed where this constellation resulted in a Kernel Panic.

[nicx]

@ich777 ok thanks again. so it is just not possible to have mixed containers (fixed ip and host ip) in macvlan mode with only 1 MAC address for the host itself presented to the network.

[ich777]

1 hour ago, nicx said:

@ich777 ok thanks again. so it is just not possible to have mixed containers (fixed ip and host ip) in macvlan mode with only 1 MAC address for the host itself presented to the network.

Nope, but you can do that in IPVLAN mode, then you have a shared MAC address whether you assign a static IP to the container or you leave it in the default bridge.

[nicx]

@ich777 yes sure, but with ipvlan mode I cannot use my firewall anymore. Thats why I need explicit MAC and IP addresses for some containers.