Dual LAN & Dual WAN pointing to one UNRAID SERVER

[hus2020]

Dear all, let me brief my situation.

I have an unraid server with two NICs each connecting to two separate routers with two separate ISPs.

1) eth0 is on 192.168.0.x    

2) eth1 is on 192.168.1.x

 

I run unraid primarily as my storage hub with few dockers such as nextcloud, plex, letsencrypt nginx, qbittorrent.

 

For the two WAN ips, i have tied to a custom domain name and have DNS records pointing to their individual WAN IPs.

 

To connect wan ip's to the unraid dockers, I am using Linuxservers excellent letsencrypt docker container, where I have created two sets of these dockers to allow me to fetch the ssl for two domains seperately and reverse proxy both wan ips. 

 

What I want is to be able to reach the unraid server's docker container using any one of the custom domain.

 

Logically if I call custom domain (1) it will redirect to wan ip 1 and reverse proxy me to the dockers

 

If I call custom domain (2) it will redirect to wan ip 2 and reverse proxy to the same dockers

 

THE ISSUE :
The issue I am facing is that the domain name only redirects to the right docker when the particular wan connection is set at eth0.

So if domain1.abc.com is WAN 1, LAN 1 (192.168.0.x) and this is eth0 in unraid network settings, the proxying will work.

 

If I set WAN1, LAN1 to eth1, it stops responding.

 

So basically whichever WAN is connected to eth0 it will proxy properly to all the dockers.

For all my dockers, I am using bridge mode, except for plex which runs on host mode.

 

I have attached my unraid network configuration  for reference.

 

Kindly advise what do I tweak to allow both WAN to proxy to all my dockers properly.

 

Tq.

 

 

 

[ken-ji]

Run the LE containers in custom network mode. this gives each container a dedicated IP (on eth0 & eth1 respectively)

then set wan1.domain.com to point to the ip of wan1.

on router 1, port forward to the LE container IP on eth0

now, set wan2.domain.com to point to the ip of wan2.

on router 2, port forward to the LE container IP on eth1

 

This should resolve you issue.

[hus2020]

4 hours ago, ken-ji said:

Run the LE containers in custom network mode. this gives each container a dedicated IP (on eth0 & eth1 respectively)

then set wan1.domain.com to point to the ip of wan1.

on router 1, port forward to the LE container IP on eth0

now, set wan2.domain.com to point to the ip of wan2.

on router 2, port forward to the LE container IP on eth1

 

This should resolve you issue.

Thanks for the suggestion Kenji. I did everything but didn't work. The one on eth0 only receives the incoming connection, the WAN set to eth1 fails.

 

Did u check my unraid network setting? Is it right? I have not enabled bonding as I want the NIC to works independently. I hope this is correct.

 

Also, I noticed in UNraid network setting, the value for DNS can only be set for eth0 and not eth1. Can this be the reason? 

 

 

[ken-ji]

in your current setup

You should have wan2.domain.com -> wan.ip.of.router2

if this is not correct then your setup won't work as you want.

router2 then forwards connections to wan.ip.of.router2:port (80/443) -> 192.168.1.110 (le container 2 port)

 

your network seems right.

DNS servers are actually independent of the interface as the OS will try whatever route to reach the specified DNS servers. so it doesn't matter that its only specified for eth0

Edited September 25, 2018 by ken-ji

[hus2020]

2 minutes ago, ken-ji said:

in your current setup

You should have wan2.domain.com -> wan.ip.of.router2

if this is not correct then your setup won't work as you want.

router2 then forwards connections to wan.ip.of.router2:port (80/443) -> 192.168.1.110 (le container 2 port)

 

your network seems right.

DNS servers are actually independent of the interface as the OS will try whatever route to reach the specified DNS servers. so it doesn't matter that its only specified for eth0

I am positive on my firewall/router settings. I have rechecked this several times.

 

On a 3rd network ie. my mobile data, I can ping to wan1.domain.com ---> and get the right wan1 ip reply

and i can also ping wan2.domain.com --> get the right wan2 ip reply

 

Its getting out from the domain name to my wan ip and wan is forwarding to lan ip, but somehow i can't confirm whether it is reaching the destination (unraid server or not).

 

As I said before, the moment I flip eth0 and eth1 with the other NIC, the second WAN works and first one don't.

 

Can this be a bug in unraid for the second NIC (eth1), where the incoming packets are not passed to the destination?

[ken-ji]

Have you tried to connected to the LE2 docker from within your network? ie https://192.168.1.110:(whatever the port is) or https://192.168.0.111:(whatever the port is) - does both work?

 

I haven't worked with multiple ips on Unraid in a long while and AFAIK docker in regular bridge mode will use all interfaces to do port mapping

I'm assuming router 1 and router 2 are not able to talk to each other so from Unraid pov they are completely separate networks.

[hus2020]

LAN 1 can open all the dockers using the unraid ip 192.168.1.110:xx

LAN2 can open all the dockers using the unraid ip 192.168.0.111:xx

Since both the networks are isolated I cannot use LAN2 unraid ip on LAN1 and vise versa.

 

If I set the LE1 & LE2 dockers on the default bridge mode, then I can go to either one of the LE dockers's console and ping the other LEs LAN IP as docker uses the consolidated bridge which is combining and forwarding request across both networks.

 

Then, when I setup both LE as custom network, they stay confined within thier own subnet 192.168.1.x or 192.168.0.x

 

It's just bugging me why changing the NIC position to eth0 makes it work and the NIC on eth1 do not work?

 

Perhaps something to ponder. Can it be that the docker engine network works exclusively on the eth0 network?

[ken-ji]

its not supposed too... lets see:

LE docker needs two ports mapped something for 80 and 443

since unRAID is using that (at least 80 by default) lets assume you set LE1 to 8080 and 8443, and LE2 to 9080 and 9443

so from a pc on LAN2, you should be able to access the LE1 using LAN2 IP - 192.168.1.110:8080 and 192.168.1.110:8443

and by access I mean browse using a browser

Also, to be clear router 2 is port forwarding to 192.168.1.110:9080/9443 right?

Edited September 25, 2018 by ken-ji

[hus2020]

Yes precisely, all my dockers can be accessed on any lan on the particular subnet. On LAN 1 I use LAN 1 subnet and on LAN 2 I use LAN 2 subnet. 

 

I went one step further to even combine the two WANs. Since I have a pfsense router for WAN1, I pushed the WAN2 connection to pfsense box as WAN1b and did a load balancing which at a high level now allow me to cross and ping any ip across both 1.x and 0.x subnets from the pfsense LAN1. 

 

Still same thing. I cannot figure out any other reason. 

 

Can it be that the LE docker is hard coded to only accept incoming requests on eth0 and not eth1? This is the only idea I can think of

[hus2020]

Under my docker settings, I see docker automatically two bridge network based on two NIC eth0 and eth1 which is labelled br0 and br1.

 

Since eth0 is tied to br0 it received and shows its gateway value which is the lan ip of my router.

 

eth0 is tied to br1 which shows and empty gateway? Can this be the reason that the WAN that is tied to eth1 cannot get its connection routed?

 

Is there a way I can force docker to also identify the gateway for the second NIC (br1)

 

 

 

[ken-ji]

The missing gateway will definitely have an effect on the docker network not being able to use WAN2, Unraid itself would be unable to reach the WAN2 using router2, as it only knows about WAN1

Disable the docker service (Enable Docker: No) above to be able to edit the information for br1

 

Since you have a merged network now (routed and firewalled by pfsense) it might be better to delegate all the load balancing to it.

[hus2020]

HI ken-ji,

 

I disabled docker service, but the value for br1 gateway is non editable. In fact non of the docker network values are editable.

I can only enable and disable the subnet using the check box. 

 

[ken-ji]

Feels like a bug @bonienl Is this a bug?

You can try this from the Unraid CLI

docker network inspect br1 which would tell us if this misconfigured part of the docker networking is giving us fits.

But your docker containers are in bridged mode right? or are they on br0/br1 respectively?

[hus2020]

Yes all my dockers are on default bridge mode. Below is the result of docker network inspect br0 and br1:

If you see only br0 has gateway value, br1 does not. So that is the reason why when I swap the WAN2 to eth0, it works because it then gets its gateway configured properly.

 

BR0

{
        "Name": "br0",
        "Id": "a4e2c0c0fd4fbd13efb5397fcbf036e8e593c6d7e595dafb3f4fc7e57b6daa10",
        "Created": "2018-09-26T08:38:16.557359047+08:00",
        "Scope": "local",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.0.0/24",
                    "IPRange": "192.168.0.0/25",
                    "Gateway": "192.168.0.1",
                    "AuxiliaryAddresses": {
                        "server": "192.168.0.111"
                    }
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "parent": "br0"
        },
        "Labels": {}
    }

BR1

{
        "Name": "br1",
        "Id": "d3a6bb0dc39b8c1e72eea65848a270a88cdc795e3466801bf276992a2edd7ead",
        "Created": "2018-09-26T08:38:16.834005583+08:00",
        "Scope": "local",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.1.0/24",
                    "IPRange": "192.168.1.0/25",
                    "AuxiliaryAddresses": {
                        "server": "192.168.1.110"
                    }
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "parent": "br1"
        },
        "Labels": {}
    }
]

 

[bonienl]

6 hours ago, ken-ji said:

Feels like a bug @bonienl Is this a bug?

No, not a bug.

When an interface is given an IP address, all fields are automatically filled in based on the network settings of the interface.

To make Docker fields editable the interface must be set to NONE as IP address.

[hus2020]

But even if it's fetching the ip automatically, why is my gateway address for eth1 is not populated?

For your second line, how do I set NONE as ip address? 

[bonienl]

Just now, hus2020 said:

But even if it's fetching the ip automatically, why is my gateway address for eth1 is not populated?

When DHCP is used it is your DHCP server which must announce the gateway address in order to populate it.

1 minute ago, hus2020 said:

For your second line, how do I set NONE as ip address? 

Stop docker and vm services, then go to Settings -> Network settings

Change "IPv4 address assignment" for eth1 (br1) to NONE

You can now edit under Docker settings the IP assignments for br1

[hus2020]

I did it everything accordingly. 

Setting my second NIC to none allowed me adjust the setting in docker. I believe in this way, the second nic is not used by unraid and becomes exclusive for docker engine. 

 

However, my  custom domain2 is still not reachable to my dockers. 

 

The forwarding only work when the subnet is set to eth0. 

[hus2020]

@ken-ji @bonienl   I did further checking on my issue. To confirm that my netowrk setup is right, I created a ubuntu VM using KVM unraid and tagged to br1 NIC (2nd WAN), installed docker ce on it, and installed the same LSIO LE container (the same that is  published on CA).

 

With this setup, all my subdomain.customdomain2.com request reaches the destination properly.  i.e. I now have 2 wan of different isp both pointing to the same unraid server docker.

 

To conclude, I believe there is some bug in the unraid docker network. Second NIC on eth1, if exposed to internet will never gets its incoming packets forwarded to its destination.

 

Only if I tag the nic to eth0 the incoming connection works.

 

I hope for the next release, this issue can be looked into.

 

Thanks.

[ken-ji]

It just occurred to me that I've never seen a multi nic server work properly ie send the reply to a packet on the interface + gateway it came in from.

this is because by default on a OS with multiple interfaces and gateways assigned per interface

the routing table specifies that packets to anywhere else (0.0.0.0/0) can ge reached by the default gateway - this usually is the gateway for eth0.

 

my guess is its like this. (take a look at your routing table) 

default can be reached by two gateways router1 and router2, they probably don't have equal metrics and this means the one with the lowest metric is the highest priority.

 

so when you send packets thru wan2 to unraid docker containers, they respond using gateway from LAN1

and when you swap the NICs - whichever is eth0 works.

 

there is a fix for this using some sysctl settings in Linux OS, but I don't remeber which they are and it can get really convoluted.

 

So my suggestions in rough:

* since you have merged to a single pfsense router, just port forward all the access from WAN1 and WAN2 to the same Unraid IP on eth0 and on Unraid, you probably want to turn on bonding, so eth0 and eth1 becomes a single interface bond0.  The limitation being Unraid no longer has a LAN2 IP and must be accessed from LAN1 or using LAN1 IP (routed via LAN2).

* or, you continue with having dual NICs, but you need to not give Unraid an IP on eth1, and program the custom network on br1 with the desired settings. - then set LE2 to be on br1 respectively and assign it a static IPs in the LAN2. with this approach dockers on br1 do not need a port mapping and have an IP that's on LAN2 and independent of Unraid. The same limitation being Unraid no longer has a LAN2 IP and must be accessed from LAN1 or using LAN1 IP (routed via LAN2).

 

There are a few more solutions I can think of but they require a more featured network (ie VLAN support, QoS, etc)

Edited September 27, 2018 by ken-ji

[hus2020]

Thank you very much kenji. I really appreciate your valuable thoughts and recommendations. 

 

You are right it may be an issue with the gateway with lowest metric takes precedence and it's gateway translates all request. 

 

With VM, I virtualize br1 as eth0 in the virtualized environment, thus the packets reach destination. 

 

For now, I am contended with solution of using the vm for my second LE container. 

 

Combing the wan in pfsense and forwarding ports there is not an option for me. My second wan connection is not pppoe but rather a static ip connection coming from an isp router from other house in the back lane. Pfsense is not dialing this connection to isp. Also, port forwarding rules are on isp router and not pfsense. 

 

That's why, having 2 nic, I send one lan from isp2 router directly to unraid (this is fastest speed connection).

 

Isp1 connection terminates at my own house so my pfsense box dials to the isp with pppoe directly and the nat port forwarding is managed by pfsense directly. 

 

Load balancing this two connection with pfsense amplifies my outgoing speed only. For incoming I cannot load balance with two isp because of limitation above, thus the need arise for me to have 2 nic in unraid. 

 

Vlan tagging etc will requires managed switch which are not at my disposal, and since my unraid is for personal use and not commercial, I wouldn't bother upgrading my switches anyways. 

 

If you stumble upon or think of some way where I can program static routing in unraid ip tables where incoming packets on wan2 are managed by wan2 gateway, then let me know. I am comfortable with cli and doing some tweaking like that. 

 

But on top of all that, tons and tons of thanks to you for giving your valuable input in this. I am truly grateful to you.