[[6.9.2] Docker with multiple connected networks cannot start on reboot / docker daemon restart]

It was fixed for a time, then it came back in 6.12, reported here:

Quote

 

Not sure how to elevate it to the devs, it's a simple fix.

In the mean time I went back to my workaround/hack.

[Unraid OS version 6.12.4-rc18 available]

1 hour ago, ljm42 said:

I'd recommend installing the Fix Common Problems plugin,

 

Already installed.

 

I've run macvlan on this setup since I've been running Unraid and I've only ever seen maybe two or three call traces and they never brought my system down so I never thought much of them, never worried much about them.

 

I had switched recently to ipvlan in anticipation of eventually upgrading to 6.12.x and decided I don't care much for ipvlan.

 

I thought maybe I'd switch back to macvlan and this 'fix' may ward-off call traces since they seem to increase with the kernel in 6.12.x.

 

I can either just try & see or stay on ipvlan I guess.

 

I was just wondering if there was something about *just* using the physical interface vs some layer on top (like bridge, bond, etc.) or if it was *just* bridges that were the issue.

 

Anyway, thanks for your thoughts!

[Unraid OS version 6.12.4-rc18 available]

Greetings.

 

Is it *only* bridges that have the macvlan issue?

 

I have a LACP bond that also acts as a VLAN trunk, from there I have several interfaces defined for various VLANs.

 

'Enable Bridging' is currently set to 'Yes' on that bond and as a result those interfaces are, for example, br2.10, br2.20, etc.

 

I have several docker networks defined across those VLAN interfaces, they show as, for example, 'IPv4 custom network on interface br2.10', 'IPv4 custom network on interface br2.20', etc.

 

If I simply set 'Enable Bridging' to 'No' on that bond, will this macvlan fix work then for all those custom docker networks?

 

(I would assume I'd be looking at interface names like 'bond0.10' and 'bond0.20' then instead of 'br2.10', etc.)

 

[[6.9.2] Docker with multiple connected networks cannot start on reboot / docker daemon restart]

Just wanted to reply & note this is now fixed in the 6.11.1 release:

 

On 10/6/2022 at 5:56 PM, limetech said:

Updated docker to v20.10.18 and improved networking:

• When DHCP is used, wait for IPv4 assignment before proceeding on system startup, this avoids a possible race-condition at boot time when host access to custom networks is enabled.
• Allow user defined networks to be reconnected at docker service start. Now all defined networks will be automatically reconnected.

[[6.10.2] /etc/rc.d/rc.docker ignores additional container networks and can't auto-start affected containers]

Thank you for picking this up & incorporating it, I can get rid of my go hack that replaces the rc script at boot when this comes out.

 

 

Many thanks!!

 

[[6.9.2] Docker with multiple connected networks cannot start on reboot / docker daemon restart]

HA! that was the first thing I tried.

 

In order to use a user script you would have to mark the container to NOT auto start and you'd have to both attach your network as well as start the container in your user script and you'd have to schedule it for first array start.

 

The issue with that approach is it leaves a LOT of holes, for example simply stopping and starting the docker service will destroy & re-create the networks then when you manually start your container (because it's not set to auto-start, remember?) it will fall flat on its face.

 

When I went down this path it turned out to be full of holes and the effort to work around all the holes was far more than the solution I noted above.

 

In order to do this properly you need to intervene AFTER the rc script has destroyed & re-created the docker networks but BEFORE it tries to start up the containers.

 

No amount of go file or user script magic will work, unfortunately.

 

The only place to intervene is that spot in the rc script I noted in my previous post.

 

To get you started do:

 

mkdir /boot/scripts
cp /etc/rc.d/rc.docker /boot/scripts/rc.docker.hack
nano /boot/scripts/rc.docker.hack

 

Then make the changes I described in the nano editor, save it (ctrl-x and answer y when it asks to save). To test it copy it to /tmp (since unraid won't allow anything to execute from the flash drive, even if you chmod it executable):

 

cp /boot/scripts/rc.docker.hack /tmp/ && chmod u+x /tmp/rc.docker.hack

#stop
/tmp/rc.docker.hack stop

#start
/tmp/rc.docker.hack start

#restart
/tmp/rc.docker.hack restart

 

Once you're satisfied it works copy it back to /boot/scripts/:

 

cp /tmp/rc.docker.hack /boot/scripts/

 

(it'll overwrite) and then edit /boot/config/go:

 

nano /boot/config/go

 

And add the line I noted in my previous post, save it like before.

 

*This will copy-in your 'hack' script to /etc/rc.d very early in the boot process (since lines in the go file are ran very early) so YOUR script is ran to start docker, not the original one, at boot as well as any time the docker service is restarted (either by you from the GUI or when the array is stopped and started, etc.).

 

This is the cleanest way to get things working as they should.

 

[Docker post arguments results in 'Execution error No such container']

I'm going back over posts I found while researching my similar issue, hoping my results can help others.

 

I believe we have the same issue (if you are even still having this issue), I found out what is happening:

 

 

Hopefully this helps...

[[6.9.2] Docker with multiple connected networks cannot start on reboot / docker daemon restart]

A few days ago I had this issue as well, I spent a little time researching it.

 

Unraid re-creates the Docker networks on reboot (actually, the service script destroys them on stop then re-creates them on start, so this happens even if you just restart the docker service).

When the networks are re-created they get new network ids.

 

Problem is, the container config still references the old network ids (you can see this if you do a docker container inspect <container>) so if you look in syslog you'll see messages like these:

 

unraid rc.docker: Plex-Media-Server: Error response from daemon: network ea31527bec520a923c5c8a466c0e265b775ba66262c7613d08684c786f5de5b4 not found
unraid rc.docker: Error: failed to start containers: Plex-Media-Server

 

If you try to start it in the GUI it's not very good at relaying the actual error message, you just get a 'No Such Container' error (as you noted).

 

If you look carefully through /etc/rc.d/rc.docker you'll see how Unraid deals with this with respect to the 'default' network assigned to the container (the one you chose in the GUI when you created the container) - it will read the GUI-stored container 'config' from an xml template under /boot/config/plugins/dockerMan/templates-user/my<container>.xml and in there is a <Network> tag with the network name, it'll make note of the container and the network in an array and later in the script it will loop over the array and re-connect the networks to the containers so the network id is current and they will start.

 

Problem is it ignores any additional networks that are attached later (outside the GUI, or even in the GUI but with the 'Post Arguments' trick you so often see in the forums).

 

Sooo... I spent a little time 'massaging' the rc.docker script to also add any additional networks into the same array so they, too, will be re-attached later.

 

If you want to do the same you'll need to edit /etc/rc.d/rc.docker and, around line ~179 you should see a block of code like this:

 

  # get container settings for custom networks to reconnect later
  declare -A NETRESTORE
  CONTAINERS=$(docker container ls -a --format='{{.Names}}'|tr '\n' ' ')
  for CONTAINER in $CONTAINERS; do
    # the file case (due to fat32) might be different so use find to match
    XMLFILE=$(find /boot/config/plugins/dockerMan/templates-user -maxdepth 1 -iname my-${CONTAINER}.xml)
    if [[ -n $XMLFILE ]]; then
      THIS_NETWORK=
      THIS_IP=
      while read_dom; do
        [[ $ENTITY == Network ]] && THIS_NETWORK=$CONTENT
        [[ $ENTITY == MyIP ]] && THIS_IP=${CONTENT// /} && THIS_IP=${THIS_IP//,/;}
      done <$XMLFILE
      # only restore valid networks
      if [[ -n $THIS_NETWORK ]]; then
        THIS_ID=$(docker container inspect "$CONTAINER" --format='{{.ID}}')
        NETRESTORE[$THIS_NETWORK]="$THIS_ID,$THIS_IP ${NETRESTORE[$THIS_NETWORK]}"
      fi
    fi #<-- insert additional code BETWEEN this line
  done #<-- and this line

 

You'll need to insert the following, additional code in-between the last 'fi' and the last 'done' (see my 'insert here' style comments above):

 

    # the loop above is based on the xml template which only defines one network - what about
    # containers where someone has done docker network connect <second network> <container>?
    # for those additional networks we need to cycle through them and add them to the array also
    ALL_CONTAINER_NETWORKS=$(docker container inspect $CONTAINER\
      --format='{{range $key, $value := .NetworkSettings.Networks}}{{$key}},
        {{if $value.IPAMConfig}}
          {{if $value.IPAMConfig.IPv4Address}}{{$value.IPAMConfig.IPv4Address}}{{end}}
          {{if $value.IPAMConfig.IPv6Address}}{{$value.IPAMConfig.IPv6Address}}{{end}}
        {{end}}
      |{{end}}'\
    )
    # an unfortunate side-effect of spreading the command above across multiple lines (for readability) is those newlines
    # sometimes end up in the final string, so take this opportunity to remove extra spaces and newlines from the result
    ALL_CONTAINER_NETWORKS=${ALL_CONTAINER_NETWORKS//[ $'\n']/}
    for CN in ${ALL_CONTAINER_NETWORKS//|/ }; do
      AN_ADDITIONAL_CONTAINER_NETWORK=${CN%,*}
      AN_ADDITIONAL_CONTAINER_IP=${CN#*,}
      if [[ -n $AN_ADDITIONAL_CONTAINER_NETWORK ]] && [[ $AN_ADDITIONAL_CONTAINER_NETWORK != $THIS_NETWORK ]]; then
        echo "container $CONTAINER has an additional network that will be restored: $CN" | logger -t $(basename $0)
        NETRESTORE[$AN_ADDITIONAL_CONTAINER_NETWORK]="$THIS_ID,$AN_ADDITIONAL_CONTAINER_IP ${NETRESTORE[$AN_ADDITIONAL_CONTAINER_NETWORK]}"
      fi
    done

 

Then save it and try to stop/start the docker service & see if it works OK.

 

If so, you'll need a way to make it permanent since /etc is sitting on a ramdisk and will be blown away on the next reboot.

 

The simplest way I found is to store a copy of the modified rc.docker script under /boot/scripts (I created the scripts directory for this) and then add a line to /boot/config/go to replace the one under /etc/rc.d with the modified one, like so:

 

mv /etc/rc.d/rc.docker /etc/rc.d/rc.docker_orig && cp /boot/scripts/rc.docker.hack /etc/rc.d/rc.docker && chmod 755 /etc/rc.d/rc.docker

 

This will run very early on in the boot process, before docker is started.

 

This seems to work well for me, I am on 6.10 at the moment, I make no guarantees that it will work for everyone but feel free to tweak it for your needs if it doesn't!

 

What would be great is if some form of this could make it into the next release so we don't have to make hacks like these!