Hello,

I'm trying to troubleshoot an issue and don't understand several things happening on my server.

I have 2 sites.

On my main site i have an unraid server with many containers, an opnsense router.

On a secondary site i have a proxmox server with lxc containers and vms, an asus router.

The 2 sites communicate through an openvpn tunnel since at least a year without trouble.

Since last week troubles arise:

• no backup from remote homeassistant vm to my unraid server

• no backup from remote proxmox to my proxmox backup server container on unraid

On the remote site the lan address is 192.168.50.0/24

On my main site : LAN: 192.168.2.0/25 (unraid is 192.168.2.50 no bonding no bridging)

IOT : 192.169.10.0/24

SERVER: 192.168.20.0/24

DMZ: 192.168.30.0/24

unraid has a sub interface on each of these vlans.

I have containers setup with templates and others setup with stacks in portainer.

When i was trying to find the cause of communication error i found that:

• ping from my main router to remote home assistant vm was OK

• ping from my laptop to remote home assistant vm on LAN was OK

• ping from remote home assistant vm to unraid was NOK

• ping from unraid to home asisstant vm was NOK

For the last one here is the result:

ping 192.168.50.10
PING 192.168.50.10 (192.168.50.10) 56(84) bytes of data.
From 192.168.48.1 icmp_seq=1 Destination Host Unreachable
From 192.168.48.1 icmp_seq=2 Destination Host Unreachable
From 192.168.48.1 icmp_seq=3 Destination Host Unreachable
From 192.168.48.1 icmp_seq=4 Destination Host Unreachable

So i was quite confused as i don't have any address on my site nor on the remote site that looks like 192.168.48.1

I discover that address was in fact used by a docker network using default bridge driver.

As soon as i bring the stack down communications between unraid and the remote site were ok.

So here is the first thing i don't understand : why docker picked up that network address. All the other networks created the same way have an address like : 172.17.0.0/16 for the default docker0 network and 172.x.0.0/16 for custom bridge networks where x starts at 18.

If i bring the stack up this time it picked up a custom bridge network like that : 192.168.32.0/20

With that network address there is no communication problem.

Attached is my unraid routing table.

The second thing i don't understand : why the hell unraid got an answer from 192.168.48.1 when pinging 192.168.50.10 ?

Second attachment is my unraid routing table at the time the incriminated stack picked up an adress on 192.168.48.0 network (i don't remember the network mask)

The 3rd attachement is the result of trace route before bringing down the stack and after

Edited September 26, 2025Sep 26 by caplam

Please post a diag file....

This isn’t Unraid “bugging out” — Docker created a network that overlapped your VPN routes, and the kernel routing table dutifully sent your packets there. Constraining Docker’s address pool should fix it permanently.

As this looks like a reuse of a subnet and issues with ip routes and confusion between mutiple networks...
It doesn’t look like a bug in Unraid itself, but rather Docker picking network subnets that overlap/confuse your site-to-site routing.

Why Docker picked 192.168.48.0/20

By default, when Docker creates new bridge networks, it will try to pick from private address ranges. If it finds that the usual 172.17.x.x / 172.18.x.x pools are “taken” (or it thinks they are), it will fall back to other RFC1918 ranges — including 192.168.x.x. That’s how you ended up with 192.168.48.0/20.
Since your VPN tunnel and remote site are also in the 192.168.x.x space, Docker unintentionally created an overlapping route.

Why pings returned from 192.168.48.1

Once that bridge network came up, the Unraid host added a kernel route for 192.168.48.0/20 pointing into the docker bridge. So when you pinged 192.168.50.10, your kernel looked at the routing table and decided “that lives on the 192.168.48.0/20 subnet → send to Docker.” The ARP reply you saw (192.168.48.1) was just Docker’s virtual gateway answering because it owned that subnet, even though the real destination was across the VPN.

*In other words, the Docker network route “hijacked” traffic meant for your remote site.

Why it worked when you tore the stack down

When you removed that stack, the conflicting route (192.168.48.0/20) was deleted. The kernel then fell back to the correct VPN route, and your cross-site traffic worked again. When the stack later came up using 192.168.32.0/20, there was no overlap with your real subnets, so routing worked fine.

SO? how to fix--

How to fix it long-term

You want to stop Docker from ever using the 192.168.x.x ranges that overlap with your LAN/VPN. A few options:

*Hard to do on unraid to survie a reboot.

Global default pool: Edit /etc/docker/daemon.json (create it if missing) and add something like

{
  "default-address-pools": [
    {"base":"172.30.0.0/16","size":24}
  ]
}

This forces Docker to allocate new bridge networks from 172.30.0.0/16 instead of randomly choosing from 192.168.x.x.

What I recommend:

Per-network control: When you create a custom network (via Portainer, compose, or CLI), explicitly set the subnet:

*Make a docker netowrk before hand... example:

docker network create \
  --driver=bridge \
  --subnet=172.31.1.0/24 \
  mynet

As a general rule of thumb ipv4 has private address blocks. don't use 192.x opr 10.x networks... (these should be used for comute and device network connecitons. ISP etc) use teh 172.x private address sapce...

So General rule: Avoid using the same private ranges for Docker bridges that you’re using for LAN, VLANs, or VPNs (192.168.x.x / 10.x.x.x). Stick with non-overlapping blocks (172.16.0.0/12 is usually the safest).

review:

hello,

Thank you for your answer.

I understand the mechanism but made a mistake (actually 2) : i didn't realize that automatic docker network address provisioning exhausted the 172.16.0.0/12 range.

I never setup the pool range on docker network because automatic assignment is convenient. i won't use it anymore.

Now i have many containers (more than 100; see the screenshot) and new containers are deployed with portainer stack and i transition old ones with stacks too. So each stack create a default docker network which makes numerous docker networks. I also have to reconfigure many containers as i turned off health monitoring when it killed my ssd in no time in unraid 6.8.

So what i will do is to delete custom docker networks and recreate them with a /24 (to keep it simple). This way i can stay in the 172.16 range.

I also didn't realize that 192.168.48.0/20 overlap with 192.168.50.0/24

I don't know why docker network pickup by default a /16 network. It doesn't make much sense.Who would put his containers in such a large network ?

For the vpn :

remote lan is 192.168.50.0/24

protocol is openvpn and tunnel address is 10.8.0.0/24. The 2nd site router which is the server has a route to my main site LAN. The problem is that this router is an asus xd6 which doesn't offer many possibilities (if i remember correctly i didn't find a tuning for tunnel network mask).

My main site router (opnsense) has also a route to the remote site lan through the openvpn gateway. And of course firewall rules which let pass only traffic to the backup server.

Next addition to my container apps could be phpipam to help manage network addresses.

Before moving on to redoing my docker custom networks i need some better understanding.

There are so many possibilities. I like to use hostnames in my network and i setup my dns servers (unbound as recursive and dnsmasq for local resolution) and reverse proxy to use hostnames for everything.

I have to review each one my containers and decide where i want to put them taking into account which of other containers they need to communicate with and of course if they need to be accessible from the outside. Doing this using hostnames is much more convenient.

I can put a container in a bridge network attached to unraid interface or subinterface; it can be in a macvlan network which has an unraid parent interface (or sub interface).

It's easy for single container app but gets more complicated in a stack when you have a webserver, a database, a redis, a worker,...

And if you have several databases mysql, postgres,... you have to decide if you setup a single instance for all databases or an instance for each database or a mix of that.

Stacks are also more convenient for backups and upgrades as you can have dependancy between containers. Unraid should really improve on that and docker in general.

You were refering to daemon.json. I don't have that file in my system and if i create one i will have to do it each time i reboot. It can certainly be done with user script but not very convenient.

I also started to manage container updates with watchtower and docker labels. I have to do it for all my containers and add backup before update.

I discovered docker labels not a long time ago. It offers many possibilities to automate things.

So all that to say that my journey through containers, networking and security is long and i constantly learn new things that i have to ingest. The more i learn the more i have to spend time on old things i setup long time ago to improve confidentiality, availability, integrity and ease of use. I'm far from having all these things setup as i want.

Edited September 25, 2025Sep 25 by caplam

These are excellent questions. I need to know if you are going to forgo unraid in favor of poratiner per your pictures. As this may dictate areas and explanations on what may have happened...

so yes:

• Create your own /24 networks up front; don’t let Docker auto-assign.

• Add a daemon.json with default-address-pools under /boot/config/docker so Unraid/Docker never touches 192.168.x.x again.

• Keep DBs and internal services on private bridge nets; only expose what needs to talk across sites.

• Use Portainer stacks, but pin their networks explicitly.

*That’ll prevent the VPN “hijack” from ever happening again and give you predictable container addressing going forward.

You’ve got the mechanism exactly right now, but Docker wasn’t “bugging,” it just fell back to grabbing a 192.168.x.x block once it thought your 172.16/12 pool was “used up.” A few points that should clear up your next questions:

1. Why Docker hands out big /16 (or bigger) subnets

By default, Docker assumes new bridge networks may host many containers, so it allocates a /16. That’s overly generous for most home labs. Nothing stops you from constraining it — you can explicitly create a /24 per stack or service. Example:

docker network create \

--driver=bridge \

--subnet=172.20.10.0/24 \

my_stack_net

Portainer will happily attach stacks to that network if you define it first.

2. Preventing overlaps permanently

On Unraid, you can define a custom IPAM pool in /boot/config/docker/daemon.json. Example:

{

"default-address-pools":[

{"base":"172.30.0.0/16","size":24},

{"base":"172.31.0.0/16","size":24}

]

}

That tells Docker: “When auto-allocating, only hand out /24s carved out of 172.30/16 or 172.31/16.”

On Unraid you’ll need to reboot or restart the Docker service for this to apply. The file is persistent if you keep it under /boot/config/docker/daemon.json. If you don’t see it now, it just means Unraid hasn’t needed one yet. You can create it safely.

3. Bridge vs. macvlan vs. parent interface

• bridge: good for intra-stack/service communication; simplest default.

• macvlan: puts containers directly on your LAN/VLAN with their own MACs. Good if you need them to look like first-class hosts, but beware Unraid host > container traffic quirks (sometimes requires ipvlan workaround).

• interface-attached bridge: common for stacks where you want all containers to live in a segment but not have their own MACs.

Most people keep “internal-only” services (databases, redis, workers) on an isolated bridge network, then attach just the front-end / reverse proxy to a LAN-facing network. That keeps routing simple.

Bridge vs. macvlan vs. subinterface networks

• Bridge network (per-stack):
  Good for “internal plumbing” like webserver > DB > redis. Containers resolve each other by name automatically within that bridge. With Portainer stacks you can declare the bridge explicitly so you always know what subnet it’s in.

• Macvlan/ipvlan:
  Useful when you want a container to look like a first-class host on your LAN/VLAN (e.g. Plex, Pi-hole, reverse proxy). Downsides: Unraid host > container traffic is tricky unless you use ipvlan mode.

• Unraid interface/subinterface attachment:
  This is mostly for when you want containers to be bound to a specific VLAN. It can make sense if you already segment by function (IoT, DMZ, Server).

A common pattern: keep stacks on their own bridge for internal communication, then expose only the front-end container (proxy, API, etc.) on a LAN-facing macvlan

4. Managing multiple stacks with shared services

If you want a single database instance to serve multiple apps, put both stacks on the same custom bridge. If you want complete isolation per app, spin up one DB per stack. There’s no wrong answer — it’s a tradeoff between resource use vs. security/isolation.

5. Automation and labels

You’re on the right track with Watchtower + labels. Combine that with named networks and backups, and you’ll avoid surprises. phpIPAM is also a solid idea to keep address planning organized, especially since you’re already spanning multiple /24s.

You’re asking all the right questions — this is exactly the “next layer” of Docker/Unraid once the basics work. A few clarifications that may help you settle on a structure:

6. Hostnames and DNS

You’re right that hostnames make life easier. A couple of approaches:

• Inside each Docker network, container names already resolve (db resolves to db:3306 inside the same bridge).

• Across stacks or across your LAN, let Unbound/dnsmasq do the work. Just make sure each container that needs to be reachable is on a LAN-facing macvlan with a static IP or DHCP reservation, then publish a DNS record for it.

• For reverse proxy setups (nginx, Traefik, Caddy), you can lean on labels in Portainer stacks to automatically publish hostnames → container IPs.

7. One database vs. many

It comes down to isolation vs. efficiency:

• Single shared instance (e.g. one MySQL, one Postgres): lighter on resources, easier to back up, but every app shares the same DB server (upgrade risk, permission complexity).

• Per-app DB instances: more isolated, easier to snapshot/restore independently, but heavier on RAM/CPU, and you end up with lots of small DBs.
  Most home-labbers do “one DB engine per type, multiple databases inside it” unless they have security/isolation concerns.

8. Portainer stacks and backups

You’re right — stacks make upgrades and rollbacks safer. A practical workflow:

1. Define explicit bridge networks in Portainer templates (no auto-assignment).

2. Use labels for Watchtower to update only the containers you’ve marked.

3. Always back up your stack config (docker-compose.yml) and DB volumes before update.

I would advise you use portainer or unraid to mange not both.

Thank you again for your answer and its quality 😃

It must be time consuming. The least i can do is trying to be as clear as you.

1-Portainer

Like all people here i started with only unraid to manage containers and portainer was just to have a quick look.

Then i installed compose plugin but i don't find it very convenient.

So i decided to transition my compose stacks to portainer. For now the majority of single containers remains under unraid control.

I have second thought about portainer: mainly the way it stores compose and env files. It's really a pain when you setup a new stack : if there is an error it changes the directory of the compose file before the next try. So now when i setup a new stack i set it first with a single hello world container and then modify it with web editor.

2-Watchtower

The advantage of watchtower is that it takes dependancies in a stack into account before upgrading.

I also noticed there are lifecycle hooks which permit to launch scripts before or after upgrade. I don't use it for now but it's on my roadmap.

3-Docker Networks

I have to remove and setup again all my docker networks but some are more tricky than others. I will also disable all ipv6 settings for docker. it's just awful to manage and my new isp doesn't even provide ipv6 connectivity.

For example the nexcloud-aio network on which you have very little control or authentik which fire itself a container outside of its network.

It came to my attention that when you publish a port for a container this port is accessible by default on all the host interfaces. So if the host has an interface (or subinterface ) in each vlan you can unwillingly open ports in vlans you don't intend to => i need to review my settings for that.

Thank you for the daemon.json trick i ignored you can put it under /boot/config/docker

{"base":"172.30.0.0/16","size":24} : does that mean that it will create /24 networks inside the /16 pool. In which case one pool would suffice.

I thought this file resides in /etc/docker and need to be created at boot time with a script. If it's done automatically when you put it in /boot/config/docker it's more convenient.

4-Hostnames, DNS

I don't think i can use labels for that. My reverse proxy is caddy and is running as opnsense plugin.

In my lan i already use macvlan with static ip and hostnames in unbound/dnsmasq. It makes it convenient for firewall rules (using an alias).

5-Databases

Until now i have only one mariadb instance which holds several databases.

The others are in separate instances. I have mysql, postgres, mongodb and tsdb.

For some i even have separate instance only for making daily dumps of the database.

By the way i see too many times in compose files database used with the latest tag. It's an awful idea as it may (and certainly will) breaks the database through an update.

I finally managed to adjust some docker settings.

for the default subnet you give me this example in json format:

{

	"default-address-pools":[

	{"base":"172.30.0.0/16","size":24},

	{"base":"172.31.0.0/16","size":24}

	]

	}

I tried creating the daemon.json file in /boot/config/docker and restarting docker but that doesn't work.

When i googled i see the people advice to add DOCKER_OPT="--argument1 --argument2 ..." in /boot/configdocker.cfg but i didn't find howto convert the json into arguments.

in fact there is a much more easy (and secure option).

in the docker.cfg you add DOCKER_OPTS="--config-file=/boot/config/docker/daemon.json"

you can place and name the file as you want as soon as it's accessible to the docker daemon.

The best part is that you can validate the file before restarting the daemon:

dockerd --validate --config-file=/boot/config/docker/daemon.json

and from the docker docs here is the full list of configurable options

 {
  "allow-direct-routing": false,
  "authorization-plugins": [],
  "bip": "",
  "bip6": "",
  "bridge": "",
  "builder": {
    "gc": {
      "enabled": true,
      "defaultKeepStorage": "10GB",
      "policy": [
        { "keepStorage": "10GB", "filter": ["unused-for=2200h"] },
        { "keepStorage": "50GB", "filter": ["unused-for=3300h"] },
        { "keepStorage": "100GB", "all": true }
      ]
    }
  },
  "cgroup-parent": "",
  "containerd": "/run/containerd/containerd.sock",
  "containerd-namespace": "docker",
  "containerd-plugins-namespace": "docker-plugins",
  "data-root": "",
  "debug": true,
  "default-address-pools": [
    {
      "base": "172.30.0.0/16",
      "size": 24
    },
    {
      "base": "172.31.0.0/16",
      "size": 24
    }
  ],
  "default-cgroupns-mode": "private",
  "default-gateway": "",
  "default-gateway-v6": "",
  "default-network-opts": {},
  "default-runtime": "runc",
  "default-shm-size": "64M",
  "default-ulimits": {
    "nofile": {
      "Hard": 64000,
      "Name": "nofile",
      "Soft": 64000
    }
  },
  "dns": [],
  "dns-opts": [],
  "dns-search": [],
  "exec-opts": [],
  "exec-root": "",
  "experimental": false,
  "features": {
    "cdi": true,
    "containerd-snapshotter": true
  },
  "fixed-cidr": "",
  "fixed-cidr-v6": "",
  "group": "",
  "host-gateway-ip": "",
  "hosts": [],
  "proxies": {
    "http-proxy": "http://proxy.example.com:80",
    "https-proxy": "https://proxy.example.com:443",
    "no-proxy": "*.test.example.com,.example.org"
  },
  "icc": false,
  "init": false,
  "init-path": "/usr/libexec/docker-init",
  "insecure-registries": [],
  "ip": "0.0.0.0",
  "ip-forward": false,
  "ip-masq": false,
  "iptables": false,
  "ip6tables": false,
  "ipv6": false,
  "labels": [],
  "live-restore": true,
  "log-driver": "json-file",
  "log-format": "text",
  "log-level": "",
  "log-opts": {
    "cache-disabled": "false",
    "cache-max-file": "5",
    "cache-max-size": "20m",
    "cache-compress": "true",
    "env": "os,customer",
    "labels": "somelabel",
    "max-file": "5",
    "max-size": "10m"
  },
  "max-concurrent-downloads": 3,
  "max-concurrent-uploads": 5,
  "max-download-attempts": 5,
  "mtu": 0,
  "no-new-privileges": false,
  "node-generic-resources": [
    "NVIDIA-GPU=UUID1",
    "NVIDIA-GPU=UUID2"
  ],
  "pidfile": "",
  "raw-logs": false,
  "registry-mirrors": [],
  "runtimes": {
    "cc-runtime": {
      "path": "/usr/bin/cc-runtime"
    },
    "custom": {
      "path": "/usr/local/bin/my-runc-replacement",
      "runtimeArgs": [
        "--debug"
      ]
    }
  },
  "seccomp-profile": "",
  "selinux-enabled": false,
  "shutdown-timeout": 15,
  "storage-driver": "",
  "storage-opts": [],
  "swarm-default-advertise-addr": "",
  "tls": true,
  "tlscacert": "",
  "tlscert": "",
  "tlskey": "",
  "tlsverify": true,
  "userland-proxy": false,
  "userland-proxy-path": "/usr/libexec/docker-proxy",
  "userns-remap": ""
}

Of course you need dig to find the meaning of certain.