Custom Docker Network on br1

[SggCnn93]

Hi everyone Been looking around all day but couldn't find anything about the topic.

I have 2 NICs in my unraid and when I create a custom docker network it gets routed through br0, is there a way to create a custom network that gets routed through br1?

I tried using Custom: br1 on a docker but it gets a personal IP address and it doesn't let me change the ports like this.

What I need is exactly the same kind of network that gets created with "docker network create XXXXX" but on the second NIC, while the rest of the dockers stay on "bridge" on br0.

Sorry I'm not too expert on custom networks and routing, any help would be really appreciated

Thank you very much!

[SggCnn93]

Tried using this command I found on the forum:

 

docker network create -o "com.docker.network.bridge.name=br1" my-net

 

In theory: "

This creates a new docker bridge named my-net of type bridge (the type you get

when no type is specified with switch  -d ...) that uses the br1 Linux bridge as the
external interface."

 

What it seems to do instead is replace br1 with this new network.

I tried booting a Swag docker on this network and at first it wouldn't boot not sure why, after trying a bit and deleting and re-creating the custom network "my-net" I manages to start Swag.

 

The problem now was that would get the correct "internal" IP from my-net but in my network it was still getting an IP from br0 and trying to check with "curl ifconfig.io" confirmed that the public IP was from br0.

 

Here are some screenshots of the process:

 

This is my initial routing table status (br-a13a0a2c93c4 is an already existing custom network created using "docker network create customnetworkname") :

 

These are my docker networks:

 

This is the routing table after using "docker network create -o "com.docker.network.bridge.name=br1" my-net":

 

And these are the docker networks:

 

Tried looking at the options for "docker network create" but couldn't find a solution, if anyone knows the right command to use please let me know cause I have no clue XD

Ty very much again

 

 

Edit: Here's a link to the forum post where I found the command:

 

Edited December 2, 2022 by SggCnn93

[SggCnn93]

After some more research I figured out what I need for this to work, but not how to do it yet XD

 

Basically to achieve what I want to do I need to create in docker network a user-defined bridge, that's what you create when using "docker network create XXXX", but I need it to connect to br1.

 

To do this I think I need to specify the driver (?), tried looking here and there are a lot of infos but I couldn't figure out how to do this:

https://docs.docker.com/engine/reference/commandline/network_create/#specify-advanced-options

[Vr2Io]

What router you use ? If you set GW 11.1 & 23.1, are you confirm router support multiple subnet ? ( subnet separate not truly network separate ). Otherwise you mess yourself.

 

For truly network separate, you need VLAN.

 

 

Or, to do your expect outcome, a solution was add a router, WAN to connect 11.1 and set the LAN be 23.1 then connect to eth1.

 

Or, you can setup software router, but this some complicated.

 

If you have advance router, you can do all in one even single ethernet.

 

 

 

Edited December 3, 2022 by Vr2Io

[SggCnn93]

I have 2 routers with 2 different ISPs, so I have 2 separate networks 

 

I have an Ubiquiti USG so I could connect both WANs to it instead of using 2 routers, but I would have the same problem I think.

 

 

Edited December 3, 2022 by SggCnn93

[SggCnn93]

I also tried blocking the custom network on br0 with iptables, using the command in this post:

https://stackoverflow.com/questions/32289417/how-to-make-docker-only-use-a-eth1-interface-to-communicate-with-other-hosts

 

Sadly nothing happened, tried "iptables -A FORWARD -i NETWORKNAME -o eth0 -j DROP", tried changing "-A" to "-I" as suggested in the comment, tried changin eth0 with br0 with both -A and -I.

The docker on "NETWORKNAME" still gets the Unraid interface IP on br0 instead of the one on br1 and checking with "curl ifconfig.io" confirm that it uses the network on eth0.

Maybe it only works with the default docker bridge "docker0" and not with user created networks?

[Vr2Io]

I can't reproduce the problem as you mention, I just simple assign docker to the network then it will route to correct gateway, no extra command or touch IP table need.

 

12 hours ago, SggCnn93 said:

I have an Ubiquiti USG so I could connect both WANs to it instead of using 2 routers, but I would have the same problem I think.

Use 2 independent router would make thing simple, I also agree change to one (USG) no use. It really doesn't make sense br1 will route traffic to br0 ( or vice reverse ).

 

 

I haven't assign IP to eth0.2 ( VLAN ), just set IP in docker setting, but I don't think this is the reason for the routing problem you mention.

 

Edited December 3, 2022 by Vr2Io

[SggCnn93]

33 minutes ago, Vr2Io said:

I can't reproduce the problem as you mention, I just simple assign docker to the network then it will route to correct gateway, no extra command or touch IP table need.

 

If my Unraid has IP 192.168.11.24 on eth0 and 192.168.23.24 on eth1 and I create a custom network (using "docker network create proxynet") when I assing this network to a docker like Swag it will get an "internal" IP like 172.18.0.2:port and an IP on my network which would be my Unraid IP on br0, so something like 192.168.11.24:port, and I can change the docker port on my network. Like this:

If instead of a custom network you use the default bridge it gets an "internal" IP like 172.17.0.2:port and on my network it still gets my Unraid IP on br0, so 192.168.11.24:port.

 

If I assign br1 to a docker it does not do this, the docker would get an "internal" IP on the second network like 192.168.23.2:port and the same IP 192.168.23.2:port on my second network. Like this:

In this situation you also cannot change the docker ports in the template, you're stuck with the default ones.

 

This is why I would need to create a custom network and route it through br1, so that it would get an "internal" IP like 172.18.0.2:port and an IP on my second network which would be my Unraid IP on br1 so 192.168.23.24:port, like this I am also able to change the dafault docker port using the template.

 

I could just assign the network br1 to my dockers and give every docker it's own IP, but I absolutely need to be able to change the default ports and I'm pretty sure you can't do that in this configuration, at least not using the docker template.

If there is a way to change the default ports on a docker on br1 with it's own IP that would also solve my problem

 

I tried using iptables because on the post I linked before on StackOverflow it says:

"The bridge created by docker isn't attached to any physical interface. External access is mediated by layer 3 forwarding and NAT rules in your iptables nat table."

So by blocking traffic on eth0 for the docker network it should be routed through eth1, but I think it olny works on the default bridge "docker0" since it did absolutely nothing when I tried using the command on the custom network "proxynet".

Edited December 3, 2022 by SggCnn93

[SggCnn93]

2 hours ago, Vr2Io said:

Use 2 independent router would make thing simple, I also agree change to one (USG) no use. It really doesn't make sense br1 will route traffic to br0 ( or vice reverse ).

 

 

I haven't assign IP to eth0.2 ( VLAN ), just set IP in docker setting, but I don't think this is the reason for the routing problem you mention.

 

One difference I see is that you use ipvlan for the custom network type while I use macvlan, don't know if it makes a different or if I could switch without issues tho.

 

I tried using this command:

docker network create -d macvlan   --subnet=192.168.23.0/24   --gateway=192.168.23.1   -o parent=eth1 proxytel

 

and this one:

docker network create -d ipvlan     --subnet=192.168.23.0/24     --gateway=192.168.23.1     -o ipvlan_mode=l2     -o parent=eth1 proxytel

 

Found them respectively here:

https://docs.docker.com/network/macvlan/

 

and here:

https://docs.docker.com/network/ipvlan/

 

But with both I get this error:

"Error response from daemon: Pool overlaps with other one on this address space"

 

I don't know why I get this error since on the second link (the one about ipvlan networks) it literally say:

"Use the network from the host’s interface as the --subnet in the docker network create. The container will be attached to the same network as the host interface as set via the -o parent= option."

 

Maybe it's because the command doesn't work if I don't change the custom network type to ipvlan? No clue XD

[Vr2Io]

For my understanding, there have three NAT docker network, but it still under br0 or br1, If br1 gateway unavailable then it shouldn't route to br0 gateway. This still what I expect.

 

10 hours ago, SggCnn93 said:

The bridge created by docker isn't attached to any physical interface. External access is mediated by layer 3 forwarding and NAT rules in your iptables nat table."

Yes

 

10 hours ago, SggCnn93 said:

So by blocking traffic on eth0 for the docker network it should be routed through eth1.

I doubt that, in general.

 

8 hours ago, SggCnn93 said:

One difference I see is that you use ipvlan for the custom network type while I use macvlan, don't know if it makes a different or if I could switch without issues tho.

Doesn't look like especially you use two independent router. If one router it may relate but I could be wrong.

 

Due to large different network environment between you and me, so I have difficulties to perform more test or verify.

 

Hope otherone can help you.

 

 

 

 

Edited December 4, 2022 by Vr2Io

[SggCnn93]

2 minutes ago, Vr2Io said:

Due to large different network environment between you and me, so I have difficulties to perform more test or verify.

 

Hope otherone can help you.

 

Yeah it seems our configuration is very different, thank you anyway for all the help!

 

I feel like the solution should be pretty simple, but I can't find it XD

 

Hope someone can help me find it 

[Hoopster]

3 hours ago, SggCnn93 said:

Hope someone can help me find it

Have you seen this topic?  Are those the instructions you used in the Secondary NIC is Available section?

[SggCnn93]

5 hours ago, Hoopster said:

Have you seen this topic?  Are those the instructions you used in the Secondary NIC is Available section?

 

Checked the topic but I think they're doing the opposite of what I need, they're trying to assign a different IP to every docker while I need the dockers to share the IP with Unraid like they do with the default bridge "docker0".

It could work if I found a way to assign custom ports to dockers with a dedicate IP address, but I couldn't find a way to do that.

 

I don't understand the second question sorry

[Hoopster]

14 hours ago, SggCnn93 said:

Checked the topic but I think they're doing the opposite of what I need, they're trying to assign a different IP to every docker while I need the dockers to share the IP with Unraid like they do with the default bridge "docker0".

 

Yes, I misunderstood what you are trying to do.  Personally, I have all my docker containers on a VLAN (br0.3) with separate IP addresses for each.

Edited December 4, 2022 by Hoopster

[positronicP]

On 12/3/2022 at 3:36 PM, SggCnn93 said:

But with both I get this error:

"Error response from daemon: Pool overlaps with other one on this address space"

 

Believe you have to set a DHCP pool range for each network interface when using multiple. You have two pipes coming in, each with it's own DHCP service within docker, and they are both drawing from the same default pool. Since neither DHCP service knows what the other is doing, you have to manually specify the pool space for each. From the photos: 'DHCP pool: not set'

 

I'm not 100% on docker network architecture, but I'm trying to do the same thing you were and that's my understanding.

[Alpha.Ars]

Hey did you find anything to make it work ?
I'm in exactly the same situation

[truetype]

Hey, same here. Did you solve it? I am also trying to find a solution to this.

[SggCnn93]

No sorry, in the end I just ended up not using a custom docker network. In my case it wasn't really necessary, I just had to use the docker IPs instead of the docker names in the various configs which imho is probably better anyway.

[caldegräl]

Chiming in because i was just looking for the same thing and got solution.

 

Unraid docker uses ipvlan for bridging and if you want to set things up so that Unraid respects your choices, you should follow whatever Unraid does.

 

First thing to do is - check how Unraid configures docker

 

~# docker network inspect br0
[
    {
        "Name": "br0",
        "Id": "...",
        "Created": "...",
        "Scope": "local",
        "Driver": "ipvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.1.0/24",
                    "Gateway": "192.168.1.1",
                    "AuxiliaryAddresses": {
                        "server": "192.168.1.3"
                    }
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
        },
        "Options": {
            "parent": "br0"
        },
        "Labels": {}
    }
]

 

This tells us a few very important config parameters, among which you will find the network type (ipvlan), "wiring", driver and other stuff.

from here it was a matter of simply doing:

 

~# docker network create -d ipvlan --subnet=192.168.16.0/24 --gateway=192.168.16.1 --aux-address=server=192.168.16.100 -o parent=br1 br1 --scope=local
6b8081041456c806253dbdfcbb1c1c074aa936d7046176312fc8b7d0cba12f55

 

I did not try to alter anything about how Unraid sets things up or how it names things, but just as I created that network, I was greeted with a new option in my container settings: "Custom : br1 - Aux"