unraid & dockers losing lan and internet connections

[mbc0]

Hi, 

 

I am having this weird situation every few days where my dockers and unraid lose internet connection and some dockers are losing connections (Agent DVR loses contact with 4 of my 6 cameras) Plex loses connection to the internet etc.  Unraid console is unable to ping most of my lan or the internet.

 

I have noticed that all 21 disks are spun up all the time when this happens, the auto spin down stops working.

 

If I stop my docker service I am then able to ping the internet and the cameras that were unavailable whilst the docker service was running and then start the docker service and everything is back up and running as normal.

 

I have attached my diags and could really do with some guidance please!

 

unraid1-diagnostics-20230608-2122.zip

unraid1-diagnostics-20230608-2122.zip

[mbc0]

Last night I removed my 10gbe card from the server and now just using the on-board 100mb just in case it is causing these problems, I cannot see anything useful in the diags unfortunately, hoping someone might see something when they get a chance 🙂

[mbc0]

Ok, 

 

Woke this morning to see it has happened again and it appears it is triggered by cabackup (probably just where it is stopping and starting dockers)

 

I have Sophos XG Home running on a 3rd Unraid Server with a 4 port NIC which is setup to these ip ranges (This has been my setup for years)

192.168.0.* - Local LAN

192.168.1.* - Local WiFi

192.168.8.* - 5G Router

 

I am able to use & ping all these ranges from all devices and all 3 unraid servers until this error occurs.

 

When this issue occurs I am unable to ping the outside world and any other ip addresses that are 192.168.1.* & 192.168.8.* as the destination is unreachable

 

ONLY 192.168.0.* will continue to work. I can even still ping 192.168.0.10 which is my Sophos installation for DSL Broadband but still have no internet access.

 

When this error occurs, all other Unraid servers & devices can see each other and have zero problems.

 

I currently have the issue as I type this and have stopped all containers but left the docker service running and the problem still exists.

 

As soon as I disable the docker service the networks are available again and I can ping everything as expected!

 

I cannot see anything to help me in the diags, really would appreciate some help here 🙂

 

Thank you

[mbc0]

OK, another update!

 

I have disabled all the containers, disabled the docker service and rebooted unraid, on reboot the network is working as expected but as soon as I enabled the docker service with no containers running at all the network problem above re-occured. I stopped the docker service again and everything is again working as expected so at least I now know the docker service is causing the issue, how would I go about resolving this? it there a problem with my routing table? (I really don't understand it)

 

[mbc0]

When I start my docker service, this is what the routing table changes to

 

[Vr2Io]

2 hours ago, mbc0 said:

I currently have the issue as I type this and have stopped all containers but left the docker service running and the problem still exists.

 

As soon as I disable the docker service the networks are available again and I can ping everything as expected!

 

I cannot see anything to help me in the diags, really would appreciate some help here 🙂

Really weird, several post report similar problem. Any wrong docker network setting ?

 

 

2 hours ago, mbc0 said:

When this issue occurs I am unable to ping the outside world and any other ip addresses that are 192.168.1.* & 192.168.8.* as the destination is unreachable

You say have 4 NIC, assume 3 interface for 3 subnet. There must have routing in the table, but your screen miss both. Unless you route those thr gateway 192.168.0.33. ( you bridge eth0 eth1, no eth2 and eth3 )

 

# Generated settings:
IFNAME[0]="br0"
BRNAME[0]="br0"
BRSTP[0]="yes"
BRFD[0]="0"
DESCRIPTION[0]="Onboard"
BRNICS[0]="eth0 eth1"
PROTOCOL[0]="ipv4"
USE_DHCP[0]="no"
IPADDR[0]="192.168.0.33"
NETMASK[0]="255.255.255.0"
GATEWAY[0]="192.168.0.2"
DNS_SERVER1="208.67.222.222"
DNS_SERVER2="208.67.220.220"
USE_DHCP6[0]="yes"
DHCP6_KEEPRESOLV="no"
SYSNICS="1"

Edited June 10, 2023 by Vr2Io

[mbc0]

13 hours ago, Vr2Io said:

You say have 4 NIC, assume 3 interface for 3 subnet. There must have routing in the table, but your screen miss both. Unless you route those thr gateway 192.168.0.33. ( you bridge eth0 eth1, no eth2 and eth3 )

 

Hi, 

The 4port NIC is in my other unraid server (HP Microserver) which basically just runs Sophos XG Firewall, Home Assistand, etc very on resources, but that is why the routing table is not showing that.

 

I have disabled the on-board NIC, re-installed the 10gbe card deleted the network.cfg file and let unraid recreate it on reboot and now everything "seems" to be working as expected!  I have stopped/started the docker service 10+ times and all seems stable so will se how it goes for a few days, I hope this is the end of it!

[mbc0]

Just to update, deleting the network.cfg file and rebooting has resolved my issues

[diarun]

Hello, I have had the same behaviour since about the middle of July.

 

My server disconnects from the network at irregular intervals.  However, I noticed that I have to edit a Docker container to "force" this behaviour. Currently, it doesn't seem to me that unraid hangs without intervention.

At first I thought it was due to the combination of editing containers within unraid and portainer. Or maybe it has to do with the fact that several tools access docker.sock?

 

Whatever. It is annoying and not healthy for the system that the server has to be stalled again and again.

 

@mbc0 Where exactly on the server did you delete the network.cfg file? I would also like to try this suggestion.

[diarun]

12 minutes ago, diarun said:

@mbc0Where exactly on the server did you delete the network.cfg file? I would also like to try this suggestion.

 

Ah, I think I've found it: The file is on the USB flash drive in the config/ folder.

 

What happens when network.cfg is deleted? Doesn't the system bristle at the fact that this file is missing? Is the file created automatically after the reboot?

I just want to make sure that I don't "break" anything else on my system.

 

THX for help.