Containers on br1


Spritzup

Recommended Posts

You can assign an IP address to eth1 in the same range as your eth0 interface. You will need to make an IP plan to ensure everything is properly addressed and no IP conflicts can occur. The router must support a secondary IP address on the same interface in this case.

 

The below picture gives an example.

 

The IP range used is 192.168.1.0/24

The router has a fixed address of 192.168.1.1 - gateway for eth0

The router has a fixed secondary address of 192.168.1.11 - gateway for eth1 (docker containers)

unRAID eth0 has a fixed address of 192.168.1.2

unRAID eth1 has a fixed address of 192.168.1.3

Router DHCP range is set from 192.168.1.50 to 192.168.1.150

Docker DHCP range is set from 192.168.1.192 to 192.168.1.254

PCs use dynamic address assignment

Containers use dynamic address assignment

image.png.b5cc8c19331daf6eb2df4e2e2eeb8eae.png

 

Perhaps a better approach is to use two different networks for eth0 and eth1, but the router must support a secondary network on the same interface or the router allows two LAN interfaces to be configured in different networks.

 

The IP range used for eth0 is 192.168.1.0/24 (primary)

The IP range used for eth1 is 192.168.2.0/24 (secondary)

The router has a fixed primary address of 192.168.1.1 - gateway for eth0

The router has a fixed secondary address of 192.168.2.1 - gateway for eth1 (docker containers)

unRAID eth0 has a fixed address of 192.168.1.2

unRAID eth1 has a fixed address of 192.168.2.2

Router DHCP range is set from 192.168.1.50 to 192.168.1.150

Docker DHCP range is set from 192.168.2.192 to 192.168.2.254

PCs use dynamic address assignment

Containers use dynamic address assignment

image.png.fea4d8c289960c6795822f248e24828c.png

Edited by bonienl
Link to comment
41 minutes ago, ken-ji said:

@bonienl Last I checked, that solution won't work, since docker needs to create 2 networks br0 and br1 with the same gateway - which docker won't do for whatever unclear reason.

@Spritzup see https://lime-technology.com/forums/topic/54882-630-how-to-setup-dockers-without-sharing-unraid-ip-address for a way to do this

 

 

Made a quick test and you are right. Docker does create the second interface but does NOT set the gateway.

You can create a manual custom network, but need 6.4.0-rc1 and set user networks to "preserve".

 

An alternative - if your router allows this - is to add a secondary IP address on the router's interface (e.g. 192.168.1.11) and use that as gateway for eth1.

 

Edited by bonienl
Link to comment

Thinking about it more...

It is actually logical that two interfaces must not point to the same default gateway. This would make it impossible for the default route to choose the outgoing interface.

Different default routes with different gateways can work, because in this situation a metric is used to set the priority of the choices.

 

If you want docker containers to communicate on a different interface then eth0 (br0) then two choices:

- If the second interface is in the same network as the first interface, a different default gateway must be given. This gateway address must be configured on the router as a secondary address (note: most consumer routers don't allow this)

- If the second interface is in a different network as the first interface, it will have a different default gateway by design. In this case the router must be able to use a secondary network on the same interface or two different interfaces are used, each with their own network configuration.

 

Instead of physical interfaces it is also possible to use logical interfaces (VLANs), but this requires a switch (if used) and router with VLAN support.

Link to comment

logical for a single machine. but for docker use cases not as much.

2 hours ago, bonienl said:

- If the second interface is in the same network as the first interface, a different default gateway must be given. This gateway address must be configured on the router as a secondary address (note: most consumer routers don't allow this)

Yes, unless you don't auto-create the docker network for the first interface. :D

  • Like 1
Link to comment
18 hours ago, ken-ji said:

logical for a single machine. but for docker use cases not as much.

Yes, unless you don't auto-create the docker network for the first interface. :D

 

I see what you're saying @bonienl and I think pfSense will let me do that, but I think @ken-ji hit the nail on the head.  It can be an advanced setting to create the docker container on an alternative interface.  Or it can check for multiple interfaces, and if it sees more than one, ask the user what they want to do.

 

~Spritz

Link to comment
On 1/29/2018 at 3:35 PM, Spritzup said:

 

I see what you're saying @bonienl and I think pfSense will let me do that, but I think @ken-ji hit the nail on the head.  It can be an advanced setting to create the docker container on an alternative interface.  Or it can check for multiple interfaces, and if it sees more than one, ask the user what they want to do.

 

~Spritz

 

Yes, pfSense will allow you to do that - it's quite a beast. In my case, I've created a VLAN specifically for dockers I'd want to assign on the br1 interface. This VLAN was both configured on unRAID (eth1) and pfSense (and you'll need a VLAN capable switch.) On the unRAID side of things ('cause I'm using ESXi), I was able to create a second virtual NIC (eth1) and use that as br1. I did not assign eth1 an address in unRAID but configured an IP for the VLAN and used the VLAN's default gateway as specified in the VLAN interface on pfSense.

 

The ultimate goal was being able to route dockers on br1 via an establish VPN connection on pfSense and direct host base communication (unRAID -> eth0) via the default ISP route. More so, because pfSense sees both networks, you can apply firewall rules to secure you VLAN <-> LAN communication.

 

I really only have two physical NICs on my box, one for WAN and another for LAN.  On pfSense you can create multiple interfaces (VLANs) on the same parent device, which is what I did. Furthermore, I created a second VNIC for unRAID for br1 (as explained above). I did have to set the vSwitch in ESXi into promiscuous mode but I'm totally surprised it works, even if I still don't fully grasp the true extent of what I did, lol...

 

 

Link to comment
  • 3 years later...
On 1/28/2018 at 6:40 PM, bonienl said:

Thinking about it more...

It is actually logical that two interfaces must not point to the same default gateway. This would make it impossible for the default route to choose the outgoing interface.

Different default routes with different gateways can work, because in this situation a metric is used to set the priority of the choices.

 

If you want docker containers to communicate on a different interface then eth0 (br0) then two choices:

- If the second interface is in the same network as the first interface, a different default gateway must be given. This gateway address must be configured on the router as a secondary address (note: most consumer routers don't allow this)

- If the second interface is in a different network as the first interface, it will have a different default gateway by design. In this case the router must be able to use a secondary network on the same interface or two different interfaces are used, each with their own network configuration.

 

Instead of physical interfaces it is also possible to use logical interfaces (VLANs), but this requires a switch (if used) and router with VLAN support.

 

Apologies for raising a thread from the dead, but I've finally got around attempting to implement this, albeit slightly different.  I'm looking to have my VM's on their own bridge, but I lose connectivity to containers running on the host (though not the host itself).  I've tried the suggestions you made in this thread @bonienl, but nothing worked.  So either I need to keep everything on the same bridge (which is not my preference), or using VLAN's for my VM's.

 

~Spritz

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.