Network Masquerade Not Working When in Host Networking (OpenVPN, Docker)

[kmwoley]

Hey folks,

I'm beating my head against a wall here and could use some help reasoning through a networking problem.

 

I've got an OpenVPN container (kylemanna/openvpn) that works just fine when it's in bridge networking mode (clients can reach internet, local network). However, when I put it into host networking the clients connected via OpenVPN cannot reach the internet or devices on the local network.

 

I've narrowed it down to what I think is a masquerading problem.

 

When OpenVPN is in bridge networking mode, tcpdump shows traffic originating from OpenVPN clients leaving the ethernet adapter on the server have been masqueraded correctly to the server's IP (10.4.10.8).

 

In this example, I'm successfully connecting to a endpoint at 10.4.40.20 from a OpenVPN client:

# tcpdump -i eth0 dst 10.4.40.20
09:21:37.044954 IP 10.4.10.8.64693 > 10.4.40.20.http: Flags [SEW], seq 1466305596, win 65535, options [mss 1361,nop,wscale 6,nop,nop,TS val 827422948 ecr 0,sackOK,eol], length 0
09:21:37.104329 IP 10.4.10.8.64693 > 10.4.40.20.http: Flags [.], ack 750528488, win 2065, options [nop,nop,TS val 827423017 ecr 232700051], length 0
09:21:37.247838 IP 10.4.10.8.64693 > 10.4.40.20.http: Flags [P.], seq 0:361, ack 1, win 2065, options [nop,nop,TS val 827423152 ecr 232700051], length 361: HTTP: POST /onvif/device_service HTTP/1.0

 

When OpenVPN is in host networking mode, tcpdump shows that same traffic which is coming from OpenVPN client as having the IP address assigned by OpenVPN (192.168.255.200):

# tcpdump -i eth0 dst 10.4.40.20
23:50:27.172338 IP 192.168.255.200.64379 > 10.4.40.20.http: Flags [S], seq 119222741, win 65535, options [mss 1361,nop,wscale 6,nop,nop,TS val 814857219 ecr 0,sackOK,eol], length 0

For some reason which is unknown to me, that traffic isn't getting masqueraded correctly. So, 192.168.255.0/24 OpenVPN client IPs are leaking out to my local network which clearly has no idea how to route them back.

 

Here's the relevant iptables from the server:

# iptables -t nat -v -L POSTROUTING -n --line-number
num   pkts bytes target     prot opt in     out     source               destination
1      102  6434 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0
2        0     0 MASQUERADE  all  --  *      eth0    192.168.255.0/24     0.0.0.0/0
3        0     0 MASQUERADE  all  --  *      tun0    192.168.255.0/24     0.0.0.0/0

Line 2 is part of the default OpenVPN configuration. Line 3 I added in attempt to see if it'd help (it didn't make any difference).

 

I've checked that, when OpenVPN is in the host networking config, the OpenVPN container itself has the expected access to the internet and local network (i.e. attaching to the container and pinging things works as expected).

 

I've tried a handful of things, but not being an expert in networking, I've reached my limit of knowledge. 

 

Let me know if there's other config/settings that'd be useful in debugging this. Any hints would be very appreciated.

 

Thanks.

[Squid]

2 minutes ago, kmwoley said:

(kylemanna/openvpn)

Does the OpenVPN server that most people around here use (OpenVPN-AS from lsio) exhibit the same behaviour?

[kmwoley]

52 minutes ago, Squid said:

Does the OpenVPN server that most people around here use (OpenVPN-AS from lsio) exhibit the same behaviour?

I don’t know. I’ve always used the one from kylemanna as it has a pretty big user base elsewhere. And I was already familiar with the bare OpenVPN over OpenVPN-AS

 

If someone has the lsio one installed and configured under bridge networking, it’d be an easy test to switch the network type to host and see what happens. 

Edited December 1, 2018 by kmwoley

[Squid]

Tell me what you want me to do.

[kmwoley]

6 minutes ago, Squid said:

Tell me what you want me to do.

Thanks for offering!

 

Under the OpenVPN-AS Docker config settings, change networking from “bridge” to “host”. I would expect that your VPN clients can access the internet, local in whatever way you already had them configured. 

[Squid]

That's already done.  Default for the container.

[Squid]

iptables -t nat -v -L POSTROUTING -n --line-number
Chain POSTROUTING (policy ACCEPT 1430 packets, 66323 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 AS0_NAT_POST_REL_EST  all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
2      160 32561 AS0_NAT_PRE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x2000000/0x2000000
3     1008 72776 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0
4        0     0 MASQUERADE  tcp  --  *      *       172.17.0.2           172.17.0.2           tcp dpt:8200
5        0     0 MASQUERADE  tcp  --  *      *       172.17.0.3           172.17.0.3           tcp dpt:6080
6        0     0 MASQUERADE  tcp  --  *      *       172.17.0.4           172.17.0.4           tcp dpt:3306
7        0     0 MASQUERADE  tcp  --  *      *       172.17.0.5           172.17.0.5           tcp dpt:6789
8        0     0 MASQUERADE  tcp  --  *      *       172.17.0.6           172.17.0.6           tcp dpt:7878
9        0     0 MASQUERADE  tcp  --  *      *       172.17.0.7           172.17.0.7           tcp dpt:8989
10       0     0 MASQUERADE  tcp  --  *      *       172.17.0.8           172.17.0.8           tcp dpt:2203
11       0     0 MASQUERADE  tcp  --  *      *       172.17.0.8           172.17.0.8           tcp dpt:2202

Client's IP is 172.27.234.2

[kmwoley]

Thanks, @Squid - it's useful to see other folks config.  Looking at your iptables, I'm even more confused. I don't see an entry which would handle the masquerade of your OpenVPN client IP (172.27.234.2). I'm at a loss as far as where to look next to understand why putting the container in 'bridge' networking works, but 'host' does not.

 

 

[bonienl]

In host mode there is no masquerading (network address translation). Only in bridge mode.

[kmwoley]

@bonienl thanks - that’s incredibly useful to know. Is that an unraid specific feature/limitation, or a general networking characteristic?

 

Do you know how OpenVPN-AS (or other VPNs which run in host mode) and their hosts are configured to properly route the traffic?

 

I could probably put a NAT entry on my router, but I don’t like the idea of solving this issue “off box” if there’s a cleaner way to do it. 

[bonienl]

This is how Docker has implemented their networking stack, it is not Unraid specific.

 

I don't use OpenVPN myself, can't give specific advice, but general advice when running in host mode, the docker container uses the IP address of your system and you have to make sure there are no port conflicts. By default the GUI uses port 80 and/or 443.

[kmwoley]

I honestly don't have a clue then how OpenVPN-AS (or any other VPN) running in Docker host networking mode is masquerading it's client traffic, then. It just doesn't make sense to me that there's no way to masquerade the OpenVPN client IP addresses on their way out the door.

 

In the end, I gave up trying to get my host to masquerade the traffic and instead added static routes for my OpenVPN client address space to my networking stack outside of the unraid host. It's a fine solution, just annoying.

 

If someone trips across this post in the future and finds a solution, I'd be interested in knowing what you figure out.

 

Thanks.

 

 

[bonienl]

Masquerading is usually done by your router. It translates a local private IP address to a public IP address.

[kmwoley]

You're right that masquerading is typically done for traffic exiting your local network, but it can also be used for things like the OpenVPN config.

 

In the case of OpenVPN it's handing out IP addresses to the clients in a different range than the local network. In the bridged docker network configuration installation of OpenVPN, those remote client IP addresses were being masqueraded somewhere along the path before they left the host so they showed up as my host server's IP on the local network.

 

Switching to host networking exposed those client IPs to the local network. At some point I'd like to understand why that was... but for now, I can configure my local network to work around the problem. At this point it's kinda academic for me to learn why that was; I'm curious.