Hi All,

 

So this may have been asked before but i can't seem to find exactly what i need

 

I'm currently attempting to desperate out and secure my docker's on my unraid server. I've been on 6.4 beta for a while now and didn't know the option for setting separate IP's existed until today as i don't tend to mess about with my docker layout due to it using so many different forwarded ports

 

I have a number of web servers and database servers in dockers on my server and am looking to set up some mail servers soon as well. I know Dockers aren't extremely secure but i'm looking to use them to host various different things and potential allow access to the servers for use by others for hosting websites etc.

 

What i would like to do is set all of my dockers into isolated IP ranges and then use my router to prevent access between the ranges so as to prevent them accessing my home network. So in essence i would like to be able to separate things like so to make the dockers a little more secure:

 

Private Home Range: 192.168.0.0/24

Home VPN Range 1: 192.168.1.0/24

Home VPN Range 2: 192.168.2.0/24

 

Docker Range 1: 192.168.20.0/24

Docker Range 2: 192.168.21.0/24

 

In short the top 3 ranges can talk together, the 2 docker ranges should be able to only talk to themselves. I can set up rules in my home network to prevent the cross ip range talking but i need to check a few things:

1. How do i allow the dockers to use the other ip range that is different to my main server ip range? I have this article but i'm not sure if the ip range change is supported or not when adding in different ip rnages: 
2. Since both the docker rnages would be using the macvlan driver, i know the dockers would be unable to access unraid due to security which is what i want to achieve however my concern is that as unraid does some internal routing for vm's accessing network drives for example, would the dockers try to route traffic internally to vm ip's and to other docker ip's thus bypassing any rules i set up in my network firewall as they stay routed internally to the server?  i.e. would it still try and route the requests internally or would the requests go out and be view able (thus manageable) by the firewall on the network?

 

Again, sorry if this has been asked before but i don't want to start adding in docker networks and altering firewall rules if its not supported yet

 

Regards,

Jamie

My article is for the 6.3.x series as there is no GUI support then.

You are on 6.4 and the GUI has native support this - which I'm not sure how to use ATM since I'm still on 6.3.5

 

But you need to be sure to have the following info:

1. A router IP for each subnet (this is usually 192.168.x.1 or similar)

2. A DNS server for each subnet - again usually the router

3. A DHCP server for the docker subnets is not necessary for the docker network as dockers still used a built-in internal address assignment scheme.

 

8 hours ago, bigjme said:

Since both the docker rnages would be using the macvlan driver, i know the dockers would be unable to access unraid due to security which is what i want to achieve however my concern is that as unraid does some internal routing for vm's accessing network drives for example, would the dockers try to route traffic internally to vm ip's and to other docker ip's thus bypassing any rules i set up in my network firewall as they stay routed internally to the server?  i.e. would it still try and route the requests internally or would the requests go out and be view able (thus manageable) by the firewall on the network?

That normally won't work:

• as the docker network would be created as you have specified and thus not routable on a different subnet
• the docker containers don't allow the internal user to fiddle with network settings (special cases do apply, but then permissions need to be set and applied to the containers to do that in the first place)
• you're VMs do not have routing enabled (normally they won't)
• Also, you failed to note what network your VMs show up in. the initial default is an internal virtual network (vibr0) in unRAID that the docker containers can't reach.

 

Thanks for the reply

 

I had a feeling I may have missed something or missunderstood the way things work, honestly I don't know fully how everything is separated by unraid in terms of network interfaces. Everything has changed a lot and I haven't really kept up to date on it. 

 

Vm's are using br0 at the moment and just get a normal ip off the router by dhcp. The vm's can also access my dockers which are almost all set up as either bridged or set as host (they have been set up over a year now) 

 

I can give my router dhcp access to the different docker ranges and also put it on the ip range if needed using a virtual alias so that's not a problem although I will likely block the docker access to the router on the Web and ssh ports to prevent potential issues

 

All I'm trying to achieve at the moment is the following, it may be doable in a much easier way than I imagine due to the ip and system restrictions from unraid

 

Set up 3 dockers; Web, database, mail on the ip range 192.168.20.0/24

 

Set up another set on the ip range 192.168.21.0/24

 

My vm's and private network are located on the ip range 192.168.0.0/24 via dhcp from the router

 

I have a few vpn ip address ranges on the network as well

 

So this is what I want to achieve. My private network can access the vpn ips and can access the docker ip ranges without limitation

 

The docker ips can only access themselves so they can't access each other, my private network, or my vpns

 

I would then have completely separated out 2 full sets of hosting assets which could be used without being able to interact with each other or my personal network

 

To give an example of my current issue, I have my Web server docker set up as bridged. This Web server is able to do Web requests to my unraid and router login pages meaning if I was to allow other people to host files on this server, they could create a way for my unraid and router login pages to be accessible by the Internet, hence me wanting to isolate things and secure them down

 

Again I may be being dim here and missing something simple that makes what I want very easy

 

Regards,

Jamie

Hi Ken-ji

 

I think i've found the settings screen you meant in the gui. Was this the correct one?

 

I haven;t messed with this just in case since i can't give anything names etc.

 

I would presume this is likely correct as i can't find anything else but it doesn't seem to request half the fields posted in the instructions so i don't want to break anything

 

This is likely something i'm unlikely to get a response from until people jump to 6.4 on official release so for now i will wait and see

 

Regards,

Jamie

Hi Ken-ji

 

Ok so i think i've got a better solution for this

 

I have a managed switch that sites between unraid and my router. Whilst i've never done it before as networking is entirely new to me, would it not be easier to set up unraid to use vlans and just route everything that way?

 

In short i would do as follows:

• Unraid uses br0 as always - from what i can gather br0 also broadcats on vlan tag id 10 as well? i.e. br0.10?
• VM's use br0 as well (as these shouldn't be restricted)
• Dockers are separated to different vlans based on where i want them (br.20, br.21 etc.) as these should be isolated entirely from everything else

I then make my switch aware of all vlans that will be accessing it and of course set up the VLANs in the router. In this instance the switch should then receive the vlan'd packets, route them to other vlan'd ips automatically and anything not in the vlan is sent to the router where i can then route as needed for the internet and other networks.

 

The dockers should then get an ip from the router in their appropriate vlan range (i presume) and i can then use the router to manage the communications between them. My only concern is that if i want to associate a static ip to a docker, it still won't be aware of the vlan ip range i presume and therefore i would have to just reserve the IP at the router level?

 

I may be 100% wrong but as i'm off work now for Christmas, its a great time to start tatting (hopefully i can't break access to my entire network doing this)

 

Regards,

Jamie

Edited December 22, 20178 yr by bigjme

Maybe this can help.

 

I think it's safe to say that you have saved me a whole mess of trouble trying to figure this out for sure! Thanks so much bonienl

 

So the dockers have to be assigned an ip by unraids built in dhcp server so it can't get one off my router at all? Is there a reason behind this? More curious than anything 

It is the way Docker has implemented their macvlan solution. I believe there is/was a request made to use an external DHCP server (your router) instead, but not sure if that is going to happen.

 

As long as Docker is configured to not conflict with your DHCP server, it is a workable solution.

 

Hi Bonienl

 

Perfect, thank you

 

I will have to have a play with this a little over the weekend

 

Regards, 

Jamie 

Well so far so good, i've got all the vlans set up for testing, just 2 web servers in it right now

 

IP's are allocated and everything is routing as expected. I put a small script on the one server to try and connect to the other web server and then a resource outside the vlan and its working like a charm. One thing i have notices is my webserver can be accessed using port 80 from br0 but not on port 443 (ssl)

 

I'm 100% sure this is down to firewall routing which i'm currently working on but worth mentioning in case its something odd that i would never expect in the docker system. Extremely unlikely but worth noting

 

Regards,

Jamie

 

Edit

Ok so i've found a problem/items worth mentioning, i set up in the docker config with the DHCP pool as 192.168.20.128/26 like in the guide, except this means that the dockers have a very odd default gateway in that range, i think it was 192.168.20.253

 

As all i'm using is dockers with fixed ip's i have disabled those DHCP pools entirely, told my router to never give out DHCP, and i can allow unraid to do the rest which gave the dockers a default gateway of .1 which fixed all sorts of ip routing issues and allowed the docker to access the internet when i allow it to in the firewall

Edited December 24, 20178 yr by bigjme

There is a regression error in this version which doesn't set the correct gateway for Docker. It is fixed for the upcoming version.

 

2 hours ago, bonienl said:

There is a regression error in this version which doesn't set the correct gateway for Docker. It is fixed for the upcoming version.

 

I've been having some networking problems - e.g. kodi clients randomly can't play network files (says not available when others can) and only some weird combination of rebooting Unifi APs and/or Kodi/mariadb docker seems to fix temporarily. 

 

Could this be my problem?  I don't recognise the docker0 or virbr0 routes below.  br0 - br0.50 for my VLANs and the default Gateways I do recognise.

 

If so, is there a way to manually fix?  Thanks

 

 

How is your docker set up? Vlans, host, etc. 

2 minutes ago, bigjme said:

How is your docker set up? Vlans, host, etc. 

All of my dockers are on VLANs and my W10 VMs.

 

I've just remembered one other networking problem I've been having.  My SSH connection to unRAID keeps disconnecting after a few minutes - from VMs (br0.50) and from laptop (br0.50).  I have to use screen to stop problems with  connection dropping.  I've had this problem for a few RCs I think.

 

You can check the network used by Docker

docker network inspect br0.50

Check if It has a default gateway setting

Btw if this problem was already existing prior to this release, the problem is elsewhere.

OK so your having ssh issues and accessing dockers. 

 

Since your network is all vlan'd, have you checked your router isn't dropping/blocking any packets? 

 

For example i have my vlan and lan ports monitored by snort, some rules in snort were periodically blocking requests until I allowed the rule

 

At the same time, I presume all your devices like your vm's and laptop can ping the gateway without issues when you loose access? 

 

I know one thing I found was that for some reason, when I added my vlans to unraid it crashes my network switch entirely. I had sporadic connections between the vlans until I rebooted it

Edited December 24, 20178 yr by bigjme

8 minutes ago, bonienl said:

You can check the network used by Docker

docker network inspect br0.50

Check if It has a default gateway setting

Btw if this problem was already existing prior to this release, the problem is elsewhere.

Just checked and all looks like it should.

8 minutes ago, bigjme said:

 

 

For example i have my vlan and lan ports monitored by snort, some rules in snort were periodically blocking requests until I allowed the rule

 

Ahh, that's one thing I haven't checked.  I'm fairly confident all my subnet<-->subnet rules are all setup correctly, but I hadn't thought to check snort.  What rule did you create?  I haven't touched snort other than following the pfsense setup guide, so I'm not familiar with it

 

https://doc.pfsense.org/index.php/Setup_Snort_Package

Honestly I hadn't played with snort much. I had it enabled on all my interfaces but mine was set to block bad traffic. So all I did was go into alerts and monitor the interfaces for a while. Clear the alerts and blocks, Try connecting to what ever was being broken and watch for alerts from ips in the subnet having issues. The just allow them as needed (red x in the far right of the alert) 

 

I had rather a few connections being blocked by snort going to my Web server so you may have some being blocked for streams etc. 

 

If blocking isn't enabled on the rules then it won't be snort as at that point it's just logging potential issues

20 minutes ago, bigjme said:

Honestly I hadn't played with snort much. I had it enabled on all my interfaces but mine was set to block bad traffic. So all I did was go into alerts and monitor the interfaces for a while. Clear the alerts and blocks, Try connecting to what ever was being broken and watch for alerts from ips in the subnet having issues. The just allow them as needed (red x in the far right of the alert) 

 

I had rather a few connections being blocked by snort going to my Web server so you may have some being blocked for streams etc. 

 

If blocking isn't enabled on the rules then it won't be snort as at that point it's just logging potential issues

Thanks - will monitor.  I've only got snort enabled on my WAN, so I don't think it will be the problem.  I've done a bit of research and clicked on WAN in Snort Interfaces, and my HOME_NET has all subnets, so they should be whitelisted.

 

Even so, I've benefited from checking my snort alerts for the first time in anger as there were a few things getting blocked like acoustid.com for beets and giganews that shouldn't have been.  When i click the red x to remove the block, where does snort keep this list so I can keep track e.g. if I make a mistake?

 

EDIT:

 

Should suppress not disable and can then track:

 

https://doc.pfsense.org/index.php/Snort_suppress_list

 

Suppression Lists allow control over the alerts generated by Snort rules. When an alert is suppressed, then Snort no longer logs an alert entry (or blocks the IP address if block offenders is enabled) when a particular rule fires. Snort still inspects all network traffic against the rule, but even when traffic matches the rule signature, no alert will be generated. This is different from disabling a rule. When a rule is disabled, Snort no longer tries to match it to any network traffic. 

Edited December 24, 20178 yr by DZMM

If you suppress the alert it will still block the connection, suppression just means it won't tell you when its done it, where as disabling it stops it triggering (thus allowing the connection)

 

As you don't have snort on your lan then it isn't going to be snort blocking it. My recommendation would be to try pinging something you can access normally when you have say ssh errors and see if its still connecting fine. Just to rule out any sort of networking error

hmm, I read it differently - that suppression doesn't block as well - will read again