SOLVED-ish - Updating 6.11.5 to 6.12.8 - container custom networks all gone on dropdowns

Any ideas, @limetech?

Okay, I was away on business for several days and just used it to step away from this for a bit.  I'm back and I've really got to ask - seriously?  I have 3 paid Unraid server licenses, I attached diagnostics, and marked this as an Urgent issue, and in 5 days I can't get an acknowledgement and some help?  Love Unraid and @limetech, but really guys, that's crap.  I have a server stuck on 6.11.5, a version you've deprecated CA and functionality for to presumably encourage upgrades, and no one can address an issue where the upgrade blows away multiple networks?

Something went wrong in the creation of the docker networks.

Feb 16 10:53:04 unraid-outer root: Error response from daemon: network dm-ac543d521cd9 is already using parent interface vhost2.60
Feb 16 10:53:08 unraid-outer root: Error response from daemon: network dm-c962d0d87feb is already using parent interface vhost4.70

 

Can you post a screenshot of your docker settings page (advanced mode)

 

And post the result of this command

docker network ls

 

6 hours ago, bonienl said:

Something went wrong in the creation of the docker networks.

Feb 16 10:53:04 unraid-outer root: Error response from daemon: network dm-ac543d521cd9 is already using parent interface vhost2.60
Feb 16 10:53:08 unraid-outer root: Error response from daemon: network dm-c962d0d87feb is already using parent interface vhost4.70

 

Can you post a screenshot of your docker settings page (advanced mode)

 

And post the result of this command

docker network ls

 

 

Yessir!  Give me a few, I'll jump on that right now!

@bonienl

Edited February 22 by BurntOC

Can you also post a screenshot of the docker settings page with Docker stopped.

 

21 minutes ago, bonienl said:

Can you also post a screenshot of the docker settings page with Docker stopped.

 

Here you go

 

This looks okay.

Can you update to 6.12.8 and provide the same information.

Thx

 

Will do.  It will probably be this evening to keep a few of my services up this afternoon.

@bonienl

I just upgraded again and the problem remains.  Downgraded to get my containers back up and I forgot to run a "docker network ls", but I did do a screencap of the docker settings (think it looks the same except 6.12.8 up top), a screencap of me trying to edit the network on on existing container (confirmed that I only get those options on any new containers as well), and I did run a diagnostics.  Hope this helps ID the cause.

unraid-outer-diagnostics-20240223-0947.zip

Edited February 23 by BurntOC

The "docker network ls" command is useful to see what docker has created as networks.

 

These coming days I'll try to replicate your set up and see in more detail what is happening.

 

Some explanation: 6.12 uses a different implementation when bridging is disabled and macvlan is enabled, when upgrading from 6.11 - it should - convert automatically to the new settings without user intervention, it is not clear what exactly happens in your situation.

 

The usual fix is to manually edit the container(s) and set the proper settings.

 

Thanks for looking at it.  I do believe there is some real "bug" here - it's always worked/upgraded fine before but as you said I know there were some changes.  Odd that my other similar one works.  I'd stumblined on some discussion of deleting local-kv.db to reset the docker network stuff but I didn't want to go near doing stuff like that if I can help it - unless y'all advise me to.  We all use stuff running off this server most of the day so it's impactful if I screw it up.

Edited February 25 by BurntOC

Any updates @bonienl? 

Just wanted to add that I see this problem as well; after reboot the custom network is gone from the container settings dropdown list. Only after stopping and starting the docker service from Settings->Docker does it become available.

 

I have for example Swag that uses a static IP on eth0, and it doesn’t get the static IP setting after a server reboot, the network type says “none” and the dropdown only has “bridge” and “host” and “none” as options, even though the container was setup with “custom”. After I stop/start the docker service the container network type is properly updated to “custom” with the correct static IP.

Good to hear.  Can you recreate @bonienl?  It would be nice to confirm before I have to do another production upgrade test and possible rollback.  I'm ready to get off 6.11.5 but stopped dead on this server.

Just doing another weekly check in @bonienl.  I reached out via support and they said you've got this.  I take it at this point you guys have verified there's an error in the upgrade (docker network downgrade, I guess, I dunno anymore) process and are trying to figure out a way to address in 6.12.9 or 6.13?

@bonienl  With another several weeks passing, and 6.12.10 released I thougth I'd circle back again.  Any ideas on how to address this issue?

Sorry for the late reply, but I have some personal issues to deal with first.

 

I can't test at the moment, but there are two ways to deal with your configuration.

 

1. Define eth0 as bonding interface as well (next to bond2). Then upgrade and see if all assignment are correctly processed.

 

2. If the above doesn't work, then (temporary) configure eth2 as regular interface (remove bonding)

 

@BurntOC Please let us know if you make any progress with Bonienl's last post.  We will continue to have the Engineering team look into this.

 

 

 

Will do.  It might be tomorrow before I can test, but I'll try to not let it slide beyond that.

@bonienl I'm really sorry to hear about the other challenges. - understood.  @Adam-M  I attempted both, and neither worked.  In the first scenario, I could see that the appropriate networks were still displayed for each container in the Docker screen , but none were started and attempting to start gave a "execution error".  I attempted to edit a container to select the network again but the only one shown besides the builtin (host, IIRC) was bond0.  I forgot to take diagnostics. 

I downgraded, then tried the second approach.  Same thing re: the containers not starting and execution error.  I forgot to edit to see what they showed, but I'd expect it was similar.  I did take a diagnostic, which I've attached here.  I've had to downgrade back again because of the importance of these containers to a lot of our daily work.

 

unraid-outer-diagnostics-20240416-1758.zip

 

 

I made a post about this issues with macvlan trace issues. There is a problem with the network config when in-place upgrading since version 6.9 and up...

6.9 is when unraid tried to make a full stop switch to ipvlan. and move off macvlan. in doing this as the default was macvlan. unraid broke every docker network...

 

My fix was to rebuild the network. by deleting the network config file from the config folder. this recreates the unraid network settings and removes the fault docker setting that were trying to be applied.

 

once back in go to setting network and reset your gui configs. (TURN OFF BRIDGING!)

 

The devs need to fix the docker menu option for custom to be default bridging off or default docker custom network ipvaln.

 

since 6.9 and this network setting change, the default has been bridging on and custom docker network macvlan...

^This needs to be fix...

 

due to the nature of dockers networks. I don't see why unriad enginer team doesn't make 2 custom docker networks... 

 

Labeling 1 for ipvlan and 1 for macvlan ...

 

 

Edited April 17 by bmartino1

I tested the bonding suggestion, and that worked for me. After a reboot the container got the correct IP, but my network setting maybe different from yours. Next I’ll attempt an upgrade to 6.12.10 and see if that also works with regards to the container network settings surviving.

 

EDIT: Upgraded and the network settings is indeed still sticking, I'll keep monitoring for stability:

 

Edited April 17 by goodyear
Tested to update

Last night I took the painful step of just blowing up the networks altogether. I deleted the network.cfg and the local-kv.db for Docker, then set up eth0, updated to 6.12.10, added the other interfaces and Docker networks back, and then set up my dozens of containers all over again.

 

Three hours later I'm up on a current working state on 6.12.10 - still running macvlan as is my preference. I just hope that I get more stability on 6.12 this time than I ever did when I could update through 6.12.5.  I'llmark this as the solution, but note that there are others who have had other successful workarounds.