Separate Docker network on separate NIC


jthacker48

Recommended Posts

Unraid 6.6.7

Dell T20 w/ 6 NIC

 

I've got pfsense 2.4 running as a VM in Unraid.  I've passed through a quad NIC to pfsense with the following:

Domain - mypersonaldomain.com

WAN

LAN - 192.168.1.0/24 (Secure LAN)

IoT - 192.168.2.0/24 (Unsecure LAN)

DMZ - 192.168.3.0/24 (Docker Servers)

 

In addition, I'm running HAProxy as a package in pfsense for my Docker usenet servers (i.e. nzbget.mypersonaldomain.com).

 

Unraid has 2 NICs:

eth0 - br0 - 192.168.1.0 (Unraid)

eth1 - br1 - 192.168.3.0 (Docker)

 

I have assigned static ip addresses for my Docker servers using 192.168.3.X but am not able to access them.  In my Docker settings, it doesn't show a gateway for br1 despite it being assigned in the Network settings.  Most of the documentation that I've seen is discussing vLANs which is what I'm trying to avoid. 

 

BTW, I don't believe it has anything to do with my pfsense settings as all of this was working prior to me implementing the DMZ and eth1/br1.  HAProxy was working and everything was communicating when it was running on br0 alone.  As of right now, I have allowed DMZ to pass any traffic through pfsense so it's not blocked at all at the moment. 

 

Any input on which settings I need to change would be appreciated.

 

 

 

Docker Settings.jpg

Network - eth0.jpg

Network - eth1.jpg

Network Routing.jpg

Link to comment

you can't use a gateway that's outside of the network/subnet - o 192.168.1.1 is not a valid gateway for 192.168.3.0/24

and unless you absolutely need to have stuff in 192.168.3.0/24 access Unraid via the 192.168.3.7 ip instead of 192.168.1.7, you should keep br1 without an IP address, and just define the network details for dockers. This will prevent some ugly situations like Unraid trying to reach the internet over br1 rather than br0

or trying to respond with the wrong interface.

  • Upvote 1
Link to comment
On 2/28/2019 at 4:51 PM, ken-ji said:

you can't use a gateway that's outside of the network/subnet - o 192.168.1.1 is not a valid gateway for 192.168.3.0/24

and unless you absolutely need to have stuff in 192.168.3.0/24 access Unraid via the 192.168.3.7 ip instead of 192.168.1.7, you should keep br1 without an IP address, and just define the network details for dockers. This will prevent some ugly situations like Unraid trying to reach the internet over br1 rather than br0

or trying to respond with the wrong interface.

You're the man!  That worked for me.

 

Although, Unraid won't allow you to enter the ip range without assigning a network address to the server...at least not through the GUI.  This is how things look at the moment.  It's working but if you're saying I'm going to run into issues, I'm open to making changes.  

 

image.thumb.png.581316c4d68d925ec4599c8f80165408.png

Link to comment
  • 1 month later...

My br0 is 172.16.0.0/24

My docker0 is 172.17.0.0

Pfsense VM LAN is 172.16.1.x on a passed-through NIC

 

Why is pfsense see traffic src from 172.17.0.0 (docker0) and of course its being denied by FW rules, should the docker0 subnet be bridged with br0 and all traffic src from 172.16.1.0/24 ?

Edited by guruleenyc
Link to comment
4 hours ago, ken-ji said:

This didn't provide enough info on what's connected to what and how.

but answer to the question is no.

Sorry about that; so unraid is on br0 (eth0) and pfsense LAN is on same subnet as bri0 using pass-thru NIC port (eth2). The pfsense WAN interface (eth3) is not on br0 or the same subnet as unraid mgmt network. Eth3 connects to an upstream switch.

I only have one bridge (br0) in unraid.

Any ideas why pfsense is seeing docker0 subnet traffic coming in on the LAN interface?

Link to comment

post you diagnostics file too.

something is quite right with your config if eth2 and eth0 are on the same physical LAN, yet the pfsense VM has a different subnet 172.16.1.0/24 (?) and still be able to see the docker0 (172.17.0.0/24) traffic. are you doing any form of bridging by cli?

  • Like 1
Link to comment
16 hours ago, ken-ji said:

post you diagnostics file too.

something is quite right with your config if eth2 and eth0 are on the same physical LAN, yet the pfsense VM has a different subnet 172.16.1.0/24 (?) and still be able to see the docker0 (172.17.0.0/24) traffic. are you doing any form of bridging by cli?

Allow me to clarify...

Unraid mgmt: br0/eth0 - on 172.16.1.0/24

pfsense LAN interface: eth2 - on 172.16.1.0/24 (passed-thru NIC)

pfsense WAN interface: eth3 - on 192.168.1.0/24 (passed-thru NIC)

 

***NIC for pfsense is not blacklisted in syslinux config, rather just allowing unsafe interrupts and specifying NIC in VM XML

 

That being said, pfsense LAN interface is seeing traffic for docker0 (172.17.1.0/24) in firewall logs and being denied. Should this be expected?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.