I've had an issue rrecently where one single docker (out of all of the ones I need its PiHole which runs most of my things) for some reason wont start when its assigned a specific IP following a recent reboot

docker: Error response from daemon: failed to set up container networking: Address already in use

When I change the IP it boots just fine and has no issues but for some reason I cant allocate it the same IP its had for years now,

I have tried,

• Turning off Docker and turning it On

• All other containers are stopped and nothing is using that IP

• No ping response from the network showing if something else has taken the IP within the network

• docker network inspect -v br0.101 shows nothing else is using the IP's running

The container has been working fine until a recent reboot to update my server and now just this one container wont start on the custom port. All other dockers appear to be working fine,

When I change it to another random IP it comes online however I have alot of things mapped to this IP so I would rather fix the issue than just change the port as something is taking this port or blocking docker from starting using this IP. Is there a way to force it to start knowing the IP is available?

Edit more testing

So did some testing and it seems no docker wants to use the address which I have allocated to Pihole on the network, so something is blocking any container using the specific IP. What would be blocking one specific IP from being used?

Edited June 3Jun 3 by brent3000

I did some more digging and I found this suggestion that it may be a stale address or something just stuck in the system and wont release the IP, the comments seem to be related 'due to a stale IP allocation in Docker's network database.'

/etc/rc.d/rc.docker stop

cp /var/lib/docker/network/files/local-kv.db \
   /var/lib/docker/network/files/local-kv.db.bak

rm /var/lib/docker/network/files/local-kv.db

/etc/rc.d/rc.docker start

I had to also re-create my docker networks following this,

This ended up resolving my issue but I wanted to know what may have been the cause of this or what would be the issue which would have prevented one single container from reloading post reboot but also what would be a good way to avoid this in the future?

Edited June 3Jun 3 by brent3000

So this issue came back again, is there a way to force a docker to run or flush any stale IP's? I'm not keen on always rebuilding my lan details every reboot, not sure if its something with Pihole or something greater,

Ok so something in the system is forcing this IP to be blocked or un-accessible, the container works a treat on legit any other IP, what or how or where do I see this IP being consumed and kill that specific value without needing to re-map all my networks?

None of my other apps seem to have this issue so I'm not sure if its just the way Pihole is setup to run or if there is a way to force it to run as there is no docker containers even running yet it thinks the IP is taken,

I have even had the container offline for over 2 hours while working on some other tasks and still the same error. What validation or checks does this even do to verify the address is already in use?

EDIT : Some more searching and the issue seems to be related to a 'stale' IP in the local-kv.db causing the issue, is there a fix of a KI about this? My current fix after almost killing my server is just to make the ip from .2 to .20 but this isnt a fix as nothing can use the .2 address now it seems (none of the other docker containers can now start on it)

Edited Thursday at 11:26 AM5 days by brent3000