6.12.13 - massive network latency when docker is enabled - server breaking for me

Does the issue go away if your revert back to the previous release you were running?

2 hours ago, JorgeB said:

Does the issue go away if your revert back to the previous release you were running?

I wanted to try troubleshoot a bit before i took that step, but logically, i see that or deleting my existing docker image are likely the next step. so I will work on that i guess if there is nothing in the diag files i supplied that immediately jump out at people.

 

3 hours ago, JorgeB said:

Does the issue go away if your revert back to the previous release you were running?

I backed up my Docker.img and made a new docker.img, left it empty but enabled it. network latency immediately returned. I disabled docker again, it went away. 
I rolled back to 6.12.10, rebooted, no latency, I enabled the empty docker.img and it returned immedately again. it is lower than it has been, but it still shouldnt be happening. I cannot see why it is happening at all. my network devices for the server are showing next to no usage at idle, and there is no HDD activity while the network is being lagged out to the point of disconnections.

And is 6.12.10 the release you upgraded form?

1 minute ago, JorgeB said:

And is 6.12.10 the release you upgraded form?

no, i was on 6.12.2 i believe, i am not certain how to roll that far back.

 

You could have downgraded from the GUI, but if if you went to a different release that's no longer possible, since AFAIK there aren't any similar reports, I would recommend recreating the flash drive with the minimum config and retesting, to make sure it's not a config issue.

ok, do you have a link to a guide that I can stick to for recreating the flash drive? and will it lose my license for unraid?

 

Backup the current one first and then redo it and just restore the bare minimum, like the key, super.dat and the pools folder for the assignments, also copy the docker user templates folder, if all works you can then reconfigure the server or try restoring a few config files at a time from the backup to see if you can find the culprit.    

I got a new fresh usb c flash drive, i ran the USB creation tool for 6.12.13 on that while i backed up my existing flash drive with unraid on it. I copied over the key, pools, shares files from config and nothing else. I connected the flash drive, selected the order of my drives as they were, and it had to rebuild my parity for the last 11+ hours. while it was doing this, no issues with the network at all. as soon as the parity had finished, all i did was enable docker and the network flooded instantly and started disconnecting devices from my home wired network, i disabled docker and the disconnecting immediately stopped.

I am out of ideas. and now by the look of it, i have an entire unraid server that i will have to setup again if this issue can be diagnosed and solved.

Edited October 19 by Schmackei

See if you can retest with a different NIC, or a different server.

16 minutes ago, JorgeB said:

See if you can retest with a different NIC, or a different server.

I have tested both NIC on the system, with the same result, that was part of the initial isolation test. I have tested with different and new netwrok cables, with the same result. I have tested other devices on the network using the old network cables with out any issue. I have tested with a different switch, and a different router. the only time this issue is happening, is when Docker is enabled on unraid. it didnt exist prior to the update, it can be 100% proven to not exist as an origin from any other device on the network. and I have tested 100% new network components, and cables with the issue being 100% replicatable from the server/docker issue. 

I have, as mentioned above also, changed out the USB flash drive for a new one with a fresh install of the OS. Att his point, it would be spending a substantial amount of money to replace the server in hope that it solves the problem, but that is not viable and simply not going to happen for a problem that didnt exist prior to the OS update. 

So, explain to be clearly, like I am a 5 year old, what you mean by test with a different NIC or a different server. since i have already tester, retested and retested again with different NICs with the same result, and I dont have thousands of dollars at my disposal to troubleshoot with a different server. Maybe you can loan me your server to test it with. *shakes head*

Suggest try don't bridging both NIC at br0 and disable IPv6.

 

Edited October 20 by Vr2Io

1 hour ago, Vr2Io said:

Suggest try don't bridging both NIC at br0 and disable IPv6.

 

There was a time that was not bridged, or bonded. let me go and disable that now. should I also disable bonded too?
I never have IPV6 enabled on my server, or my home network.

Edited October 20 by Schmackei

4 minutes ago, Schmackei said:

should I also disable bonded too?

Yes too, make network as simple as possible.

 

5 minutes ago, Schmackei said:

I never have IPV6 enabled on my server, or my home network.

I overlook DHCP6(0) was "yes", but actually protocol is "ipv4" only.

 

When you enable bridge and bridging both NIC, it may easy cause looping, so whole network will down.

12 minutes ago, Vr2Io said:

When you enable bridge and bridging both NIC, it may easy cause looping, so whole network will down.

 

I have disabled bridging and bonding, left it as one NIC eth0, rebooted, not at bad, but same issue. I then swapped it to eth1, rebooted, same issue again. no resolution here. still only doing it as soon as Docker is enabled.

1 minute ago, Schmackei said:

I have disabled bridging and bonding, left it as one NIC eth0, rebooted, not at bad, but same issue. I then swapped it to eth1, rebooted, same issue again. no resolution here. still only doing it as soon as Docker is enabled.

 

Then what docker have running when docker service start ?

 

12 minutes ago, Vr2Io said:

 

Then what docker have running when docker service start ?

none, they were all turned off. and after building a new flash, there was no dockers containers installed, and i was getting the same issue.

 

19 minutes ago, Schmackei said:

 

none, they were all turned off. and after building a new flash, there was no dockers containers installed, and i was getting the same issue.

 

Note, I use same mobo but it running in Windows not Unraid. You still use 1st release BIOS and currently were 4.x

BTW, would you describe a bit more on this mobo running with Unraid in past ? With docker enable ?

 

There are some strange on your ifconfig, there are an unknow "br-f67f951ba59f" network there ( although down ) and your eth1 have some ipv6 setting too. The right corner is an example of one of my unraid, you found  no such at all.

 

I think the "last" was I can try booting Unraid in same mobo and enable docker service as a ref. for you.

 

Edited October 20 by Vr2Io

Pls note, I never face similar problem on different build even different Unraid OS version. Pls also try only connect one Ethernet and check any different. 

24 minutes ago, Vr2Io said:

Note, I use same mobo but it running in Windows not Unraid. You still use 1st release BIOS and currently were 4.x

BTW, would you describe a bit more on this mobo running with Unraid in past ? With docker enable ?

 

There are some strange on your ifconfig, there are an unknow "br-f67f951ba59f" network there ( although down ) and your eth1 have some ipv6 setting too. The right corner is an example of one of my unraid, you found  no such at all.

 

I think the "last" was I can try booting Unraid in same mobo and enable docker service as a ref. for you.

 

 

here is a new diagnostics for you, that way we arent going back and looking at the network settings that no longer exist and arent relevant to the issue we are still facing.

I am not certain what the unknown network was, the OS was installed years ago and it was a project for me to learn how to navigate around, im sure at some point before 6.10 I manually installed things like wireguard, tunnels, or tailscale and it might be residual from that. but as i said, since the fist Diag file was attached, the network was reset. the flash drive and OS were freshly reset, the docker was freshly reset, and there was no docker containers setup. the issue persisted even after a fresh install of unraid, and only when docker was enabled in the settings. Also it is strange that eth1 was showing ipv6 settings when it was quite very selected as ipv4 only, maybe that is another fun quirk with the OS that needs to be looked at.

I have been re-adding some things as the array does have some very important things i need to access for work through the week, and I would rather not have to go and retreive it from my offsite as it is hours away.

My MOBO revision might be older than yours, no idea why, i have flashed it with the latest bios that is listed. I am not certain what other information you would like from me regarding the mobo, be specific if you want specifics.

jarvis-diagnostics-20241020-1433.zip

I have make test on same mobo in Unraid 6.12.13, I also move two onboard NIC as eth0 ( so have two test ) and start docker service, the USB also fresh. I can't reproduce your problem.