IPVLAN/MACVLAN > pulling my hair out

[aglyons]

I've posted other threads about issues and have been given some advice which I appreciate. But I still can't seem to figure out a solution.

 

Docker MACVLAN was causing kernel panic crashes so I was advised to switch to IPVLAN, great.

 

I noticed that IPVLAN, how it operates, assigns the containers set static IP address to the host MAC address. While this may be kosher, I have noticed that my router, Unifi UDMProSE, has issues with this approach.

 

It appears that the UDM tracks clients by MAC address. So, the host MAC address in Unifi will have it's IP address flip amongst the various IP's set as static in the container's config. If the host IP ends with 88 and a containers ends with 249 and another at 250. The router will show the host MAC associated with each of those IP's, at random, and only one at a time. If you query the UDM client list for one of the IP's that is not showing, it will not know anything about it. It's still ping-able from other clients, UDM just has no knowledge of it on the LAN.

 

This can cause a problem as the router 'could' assign a used IP via DHCP to a new client on the network at any time causing an IP conflict.

 

Being that IPVLAN is being used, I can't set a DHCP reservation as there is no unique MAC address associated with the container.

 

I switched back to MACVLAN in Docker and went the route of assigning a fixed IP in the UDM for the different containers. I understood that the kernel panic issues were caused by static IP's on the container settings. But then ran into another issue. When you update a container, the MAC address changes. So setting a reserved IP on the DHCP is useless.

 

I have tried going the VLAN route but following all the guides am not getting the same results as expected. I added a VLAN to the network config for docker use. But that VLAN is not shown in the container settings. Just the default br0.

 

 

 

So, to wrap things up, we can't used MACVLAN without static IP's or services will break. Set a static IP with MACVLAN and it crashes the box at random. We can't use IPVLAN as it confuses routers and can cause DHCP lease conflicts.

 

Call me frustrated!

 

EDIT: If anyone would be so gracious, I would be more than willing to have a Skype or Zoom call and I can share my screen. That might help speed up both my explanation and a possible resolution to what I am seeing.

 

Edited September 3, 2022 by aglyons

[Vr2Io]

4 hours ago, aglyons said:

So setting a reserved IP on the DHCP is useless.

No matter MACVLAN or IPVLAN, DHCP always useless because docker won't get IP from router/external DHCP.

 

4 hours ago, aglyons said:

causing an IP conflict.

This also won't happen, you should setting an IP range for DHCP purpose, and all non-DHCP application use out of this range. Some SMART enough router also work well if you bind MAC address out of DHCP range IP.

 

4 hours ago, aglyons said:

The router will show the host MAC associated with each of those IP's, at random, and only one at a time.

 

You can use br'x' network mode then static a IP for each docker, only some docker must use host mode. ( Of course all with same MAC address )

 

Edited September 3, 2022 by Vr2Io

[Hoopster]

@aglyons As mentioned above by @Vr2Io, DHCP for docker and DHCP on your router cannot compete in the range of IP addresses they are allowed to assign.  Docker does not know anything about your router DHCP.  I assign the Docker VLAN in the USG a DHCP Range of 192.168.3.100 to 192.168.3.150.  However, I do NOT enable DHCP on the docker VLAN in unRAID and manually assign addresses in this range.

 

 

On the VLAN side in unRAID, I do not enable DHCP on the br0.3 interface

 

Settings --> Network

 

 

Settings --> Docker

 

For each docker container to which I want to assign a static IP address, I specify the IP addresses beginning with 192.168.3.100

 

Here is my Plex container (192.168.3.110) for example:

 

I am not sure how this differs from what you are doing but I do not run into the issues you describe.

 

 

Edited September 3, 2022 by Hoopster

[aglyons]

So let me get this straight. Do Docker networks operate differently than VM networks?

 

I am coming from using an ESXi server where each VM gets its own IP on the same network as the host. My UDM DHCP server sees them and assigns an IP. I then go to that device in the UDM and set it to always get that same IP. It' just simplified things IMO. I can see all the devices and their respective IP in the UDM UI. As well, if I choose to, I can change the IP of that virtual device/container from the UDM UI.

 

If I understand correctly, Docker containers won't get assigned an IP from the UDM at all? I would have to manually assign IP's to containers even though they are on the same network as the host? Or, using DHCP, it would only respond to the Docker networks internal DHCP server?

Edited September 4, 2022 by aglyons

[Hoopster]

6 hours ago, aglyons said:

If I understand correctly, Docker containers won't get assigned an IP from the UDM at all? I would have to manually assign IP's to containers even though they are on the same network as the host? Or, using DHCP, it would only respond to the Docker networks internal DHCP server?

If you assign a DHCP pool of addresses for use by a docker network you must be careful that these addresses do not overlap with the DHCP pool of your router on the same subnet. If they overlap you could have the same IP address assigned by Docker DHCP and router DHCP since they do not know about each other.  I have nothing but Docker using the 192.168.3.xxx subnet so this is of little concern in my situation.

 

Manual IP address assignment to docker containers is not required; you can let them use docker DHCP to assign from the pool.  I just choose to manually assign IP addresses to docker containers on my Docker VLAN rather than defining a DHCP pool.

Edited September 4, 2022 by Hoopster

[aglyons]

5 hours ago, Hoopster said:

If you assign a DHCP pool of addresses for use by a docker network you must be careful that these addresses do not overlap with the DHCP pool of your router on the same subnet. If they overlap you could have the same IP address assigned by Docker DHCP and router DHCP since they do not know about each other.  I have nothing but Docker using the 192.168.3.xxx subnet so this is of little concern in my situation.

 

Manual IP address assignment to docker containers is not required; you can let them use docker DHCP to assign from the pool.  I just choose to manually assign IP addresses to docker containers on my Docker VLAN rather than defining a DHCP pool.

I understand what you're talking about, but, I'm curious if the Docker containers will be assigned an IP from the routers DHCP server. They are all exposed to that network. I can't see why the router couldn't look after assignment. Yes, keep the Docker DHCP server off. Then there's no duplicate IPs.

 

I'll have to do some tests myself

[Hoopster]

20 minutes ago, aglyons said:

I'm curious if the Docker containers will be assigned an IP from the routers DHCP server

Apparently not.  You might find this discussion informative.