[Plugin] Docker Compose Manager

[nearcatch]

21 hours ago, hasown said:

I have this function in my profile.sh to update docker-compose without waiting for the plugin to be updated. This function works even if the plugin isn't installed, so if you only use docker-compose from the command-line, this is all you need.

 

New version of this function. Now it checks your local version and only downloads a new one if the github repo version is different or if docker-compose is missing entirely. It also sends an unRAID notification when a download happens, so you can run this function daily using cron or a userscript and get notified when an update happens.

EDIT: new-new version. Now if you pass "check" when calling the function, it only notifies of new versions instead of downloading. Passing anything else or nothing will download a new version if available.

HELP: If anyone knows how to print newlines into an unraid notification without using <br>, please let me know. <br> works fine for dashboard notifications but they look weird in discord notifications using a slack webhook.

 

# notify [-e "event"] [-s "subject"] [-d "description"] [-i "normal|warning|alert"] [-m "message"] [-x] [-t] [-b] [add]
alias notify='/usr/local/emhttp/webGui/scripts/notify'
# dc update
dcupdate() {
  COMPOSE_LOCAL=$(docker compose version 2>/dev/null | cut -d " " -f4)
  COMPOSE_LOCAL=${COMPOSE_LOCAL:-"none"}
  COMPOSE_REPO=$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep 'tag_name' | cut -d '"' -f4)
  echo Current: ${COMPOSE_LOCAL}
  if [ ${COMPOSE_LOCAL} != ${COMPOSE_REPO} ]; then
    dcdownload() {
      echo Repo: ${COMPOSE_REPO}
      curl -L "https://github.com/docker/compose/releases/download/${COMPOSE_REPO}/docker-compose-$(uname -s)-$(uname -m)" --create-dirs -o /usr/local/lib/docker/cli-plugins/docker-compose && chmod +x "${_}"
      echo "Installed: $(docker compose version | cut -d ' ' -f4)"
      notify -e "docker-compose updater" -s "Update Complete" -d "New version: $(docker compose version | cut -d ' ' -f4)<br>Previous version: ${COMPOSE_LOCAL}" -i "normal"
    }
    if [ -n "${1}" ]; then
      if [ "${1}" = "check" ]; then
        echo "Update available: ${COMPOSE_REPO}"
        notify -e "docker-compose updater" -s "Update Available" -d "Repo version: ${COMPOSE_REPO}<br>Local version: ${COMPOSE_LOCAL}" -i "normal"
      else
        dcdownload
      fi
    else
      dcdownload
    fi
  else
    echo Repo: ${COMPOSE_REPO}
    echo "Versions match, no update needed"
  fi
  unset COMPOSE_LOCAL
  unset COMPOSE_REPO
}

 

First run passes "check" as an argument to check for updates without downloading. Second is passing a non "check" argument. Third is no argument.:

 

Notification for check only:

Notification for completed update:

Edited September 13, 2023 by hasown

[primeval_god]

3 hours ago, L0rdRaiden said:

all the compose stack fail even if they dont have auto start enable

The compose stacks without autostart do not 'fail' they are working as intended. The Red Stop Icon indicates that the compose stack is stopped, i.e. its containers exist but none of them are running. This is expected behavior for a running stack with autostart off when docker restarts. It is the same behavior as for non-compose containers on unRAID that are not configured with autostart. 

 

As for your compose stacks that are set to auto start, the problem and recommendation remain the same as last time it was brought up. Compose stacks do not seem to work with the docker networks created by unRAID (via the gui). It appears that those networks are removed and recreated by unRAID when docker is restarted. Compose does not like this as it references networks not by name but by id in its internal live state. The solution is to not use those networks with compose stacks, instead you need to create custom macvlan networks manually on the command line using a docker network command. Once done the "Preserve user defined networks" setting will ensure that the custom networks are not removed between docker restarts. Compose containers should start correctly when attached to those networks. 

 

As for why non-compose containers work fine with the networks unRAID creates, I dont really know. I assume it has something to do with how dockerman works, though it could just be something compose is picky about.

[L0rdRaiden]

12 hours ago, primeval_god said:

The compose stacks without autostart do not 'fail' they are working as intended. The Red Stop Icon indicates that the compose stack is stopped, i.e. its containers exist but none of them are running. This is expected behavior for a running stack with autostart off when docker restarts. It is the same behavior as for non-compose containers on unRAID that are not configured with autostart. 

 

As for your compose stacks that are set to auto start, the problem and recommendation remain the same as last time it was brought up. Compose stacks do not seem to work with the docker networks created by unRAID (via the gui). It appears that those networks are removed and recreated by unRAID when docker is restarted. Compose does not like this as it references networks not by name but by id in its internal live state. The solution is to not use those networks with compose stacks, instead you need to create custom macvlan networks manually on the command line using a docker network command. Once done the "Preserve user defined networks" setting will ensure that the custom networks are not removed between docker restarts. Compose containers should start correctly when attached to those networks. 

 

As for why non-compose containers work fine with the networks unRAID creates, I dont really know. I assume it has something to do with how dockerman works, though it could just be something compose is picky about.

 

Thanks for your detailed answer but regarding the red square all the stack always start with a red square does not matter if they have auto start enable or not.

 

So I don't want to create a different network because the stacks need to comunicate with services in the existing networks that are being used by other containers and VM. I know you can allow connectivity but it will unnecessarily complicate things.

So, considering this "restriction", maybe this question is very stupid but for some unknown reason compose manager is not able to properly auto-start stacks when existing networks (created by unraid) are being used BUT if I manually click in on "compose launch" button, the stack is launched properly and works, so my question is why auto-start does not work and it works prefectly when I manually press the button "compose up"

Is the command different?

Is because autostart tries to start the compose stacks too soon and networks are not ready? if this is the case can we add a delay option? like wait 30 secs after docker starts to start the docker compose stacks?

 

I mean I can not understand why if I do it manually works but the autostart does not work, if they are doing basically the same thing, starting a compose stack.

Edited September 14, 2023 by L0rdRaiden

[primeval_god]

4 hours ago, L0rdRaiden said:

 

Thanks for your detailed answer but regarding the red square all the stack always start with a red square does not matter if they have auto start enable or not.

The containers from any stack that was running will still exist when docker is restarted but will be in the stopped state. For those stacks that are not set to autostart the the containers will stay stopped and the red square is the expected symbol to mark this condition. As for your stacks that are set to autostart, due to the other issue you have, the containers fail to start when compose up is run, leaving the containers in the stopped state and the red square displayed. 

 

4 hours ago, L0rdRaiden said:

So, considering this "restriction", maybe this question is very stupid but for some unknown reason compose manager is not able to properly auto-start stacks when existing networks (created by unraid) are being used BUT if I manually click in on "compose launch" button, the stack is launched properly and works, so my question is why auto-start does not work and it works prefectly when I manually press the button "compose up"

Is the command different?

Please clarify this statement. From your previous posts i was under the impression that clicking "Compose Up" would not result in the stack starting. My understanding was that you first had to click "Compose Down" then "Compose Up", which in effect removes and recreates the entire stack. 

The commands executed by auto-start and the compose up button are the same.

 

4 hours ago, L0rdRaiden said:

So I don't want to create a different network because the stacks need to comunicate with services in the existing networks that are being used by other containers and VM. I know you can allow connectivity but it will unnecessarily complicate things.

My suggestion would be to move the VMs and other containers on to the custom created networks as well. Once created they will be available for use in the rest of the unRAID GUI. Essentially just replace all the unRAID created networks with a manually created equivalent. 

 

[L0rdRaiden]

16 hours ago, primeval_god said:

The containers from any stack that was running will still exist when docker is restarted but will be in the stopped state. For those stacks that are not set to autostart the the containers will stay stopped and the red square is the expected symbol to mark this condition. As for your stacks that are set to autostart, due to the other issue you have, the containers fail to start when compose up is run, leaving the containers in the stopped state and the red square displayed. 

 

Please clarify this statement. From your previous posts i was under the impression that clicking "Compose Up" would not result in the stack starting. My understanding was that you first had to click "Compose Down" then "Compose Up", which in effect removes and recreates the entire stack. 

The commands executed by auto-start and the compose up button are the same.

 

My suggestion would be to move the VMs and other containers on to the custom created networks as well. Once created they will be available for use in the rest of the unRAID GUI. Essentially just replace all the unRAID created networks with a manually created equivalent. 

 

 

I'm now in 6.12.4 and I have applied the fix for the macvlan problems so now my networks are ethx, instead brx. I am using physical NICs for every networks excep for the docker internal networks.

Now the behaviour is a little bit weird sometimes If I run them after the red squeare they just run ok but sometimes they complain about the missing network. I have to do some more test, I think it works when the stack was stoped before reboot and autostart is disable, in this case I can run the stack manually after reboot and it works.

 

I have done a little test and it looks like that it properly works if I create the networks manually, so this is one possible solution but I wouldn't like to mess too much with the Unraid networks "standards".

 

I know is too much to ask but could you add an option (optional) in the settings that on boot or docker service restart, compose manager will do a compose down and the compose up, and not only compose up. I guess this fill fix the problems for the "noobs" without messing with the creation of custom networks in docker.

A setting to delay the startup like Unraid does would be nice as well, I have a lot of containers pending to migrate but I don't want to run them at the same time on restart.

 

In any case you are doing and excellent job, I don't understand why compose doesn't get official support from unraid, that doesn't mean that it has to replace dockerman but to provide compose as a fully suported alternative.

[L0rdRaiden]

@primeval_god please could you add the propose change? stop and start the stacks on unraid/docker restart? it's a small change

This will fix the issue without the need to do custom networks

[primeval_god]

18 hours ago, L0rdRaiden said:

@primeval_god please could you add the propose change? stop and start the stacks on unraid/docker restart? it's a small change

This will fix the issue without the need to do custom networks

I will look into adding an option to bring the stacks down and up on restart. The current behavior will remain the default however as stopping and starting the containers is more inline with the expected behavior for docker containers on unRAID. I dont have a timeline for when i will be able to make the changes.

[Mainfrezzer]

On 9/23/2023 at 2:48 AM, primeval_god said:

I will look into adding an option to bring the stacks down and up on restart. The current behavior will remain the default however as stopping and starting the containers is more inline with the expected behavior for docker containers on unRAID. I dont have a timeline for when i will be able to make the changes.

I do not use the plugin but I've stumbled upon the bug report and investigated a bit and just found the activity here. --force-recreate on autostart should be all that's needed.

Edit: I did open a pull request
Edit 2: seems like its only respected on the first container within the stack, odd. ill probe it a bit more, for the time being i closed it

So bizarre, Aria2 is the first entry in the stack and works flawlessly with the --force-recreate flag, but the other 2 within the stack, nah, they dont give a damn.




Okay found the fix. Setting the flag within the compose.sh upon start up solves the issues, opened the pull request again^^

I do wanna add a video to show it working flawlessly, so here it is. I hope i was able to help you all out.

 

Addendum: Since i dont know when the changes are being merged, i want to provide the fix here for you to use as long as the plugin hasnt been updated yet.
plugins.zip
The changes can be found here
https://github.com/Mainfrezzer/compose_plugin

Edited September 25, 2023 by Mainfrezzer

[L0rdRaiden]

@primeval_god do you plan to apply the proposed fix? I think is the right solution for the problem some of us are experiencing

[primeval_god]

11 hours ago, L0rdRaiden said:

@primeval_god do you plan to apply the proposed fix? I think is the right solution for the problem some of us are experiencing

A new version is available which adds the option to force-recreate during autostart. The option can be selected in the compose settings menu. The default behavior is not to run without the --force-recreate option. 

[L0rdRaiden]

When I create a new docker compose and I use the same name for the container that I was using in dockerman for some reason the links to information related to the original docker with dockerman like donate, more info, support appear as well, it also broke the web UI set with the labels.

I have tried deleting the dockerman templanes but does anyone know what unraid config file do I have to delete or edit so dockerman menu information won't be associated with a container created with compose?

 

 

[L0rdRaiden]

After enable the new setting is still problematic

 

I have disable and enable docker, not a normal reboot in case it matters.

 

The result

 

Examples

 

 

But if I manually do compose down, compose up, (buttons) it works.... any idea?

Does it work only on reboot and not If docker is disable via settings?

[L0rdRaiden]

14 minutes ago, L0rdRaiden said:

After enable the new setting is still problematic

 

I have disable and enable docker, not a normal reboot in case it matters.

 

The result

 

Examples

 

 

But if I manually do compose down, compose up, (buttons) it works.... any idea?

Does it work only on reboot and not If docker is disable via settings?

I tried a normal reboot and it worked, it would be easy to make it work as well when docker is disable/enable via webui?

[primeval_god]

2 hours ago, L0rdRaiden said:

I tried a normal reboot and it worked, it would be easy to make it work as well when docker is disable/enable via webui?

No idea, if I remember correctly (not able to look right now) the auto start is hooked into the unRAID started event. I am not certain if there is a specific event for docker start.

[L0rdRaiden]

9 hours ago, L0rdRaiden said:

When I create a new docker compose and I use the same name for the container that I was using in dockerman for some reason the links to information related to the original docker with dockerman like donate, more info, support appear as well, it also broke the web UI set with the labels.

I have tried deleting the dockerman templanes but does anyone know what unraid config file do I have to delete or edit so dockerman menu information won't be associated with a container created with compose?

 

 

If someone else have this problem, the fix is to delete the template from unraid and restart the server.

[bullmoose20]

i keep getting the following error:

Quote

failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount1320609642/Dockerfile: no such file or directory

version: '2'
services:
  gaseous-server:
    container_name: gaseous-server
    build:
      context: ./
      dockerfile: Dockerfile
    restart: unless-stopped
    networks:
      - gaseous
    depends_on:
      - gsdb
    ports:
      - 5198:80
    volumes:
      - gs:/mnt/user/appdata/gaseous-server
    environment:
      - dbhost=gsdb
      - dbuser=root
      - dbpass=gaseous
      - igdbclientid=(redacted)
      - igdbclientsecret=(redacted)
  gsdb:
    container_name: gsdb
    image: mysql:8
    restart: unless-stopped
    networks:
      - gaseous
    volumes:
      - gsdb:/mnt/user/appdata/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=gaseous
      - MYSQL_USER=gaseous
      - MYSQL_PASSWORD=gaseous
networks:
  gaseous:
    driver: bridge
volumes:
  gs:
  gsdb:

 

[L0rdRaiden]

21 minutes ago, bullmoose20 said:

i keep getting the following error:

version: '2'
services:
  gaseous-server:
    container_name: gaseous-server
    build:
      context: ./
      dockerfile: Dockerfile
    restart: unless-stopped
    networks:
      - gaseous
    depends_on:
      - gsdb
    ports:
      - 5198:80
    volumes:
      - gs:/mnt/user/appdata/gaseous-server
    environment:
      - dbhost=gsdb
      - dbuser=root
      - dbpass=gaseous
      - igdbclientid=(redacted)
      - igdbclientsecret=(redacted)
  gsdb:
    container_name: gsdb
    image: mysql:8
    restart: unless-stopped
    networks:
      - gaseous
    volumes:
      - gsdb:/mnt/user/appdata/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=gaseous
      - MYSQL_USER=gaseous
      - MYSQL_PASSWORD=gaseous
networks:
  gaseous:
    driver: bridge
volumes:
  gs:
  gsdb:

 

are you building you own image? can you use one from docker hub?

[L0rdRaiden]

ultra minor bug

https://docs.docker.com/compose/compose-file/03-compose-file/

compose manager should support alternative file names for compose files, compose.yaml (preferred)

[primeval_god]

3 hours ago, bullmoose20 said:

i keep getting the following error:

version: '2'
services:
  gaseous-server:
    container_name: gaseous-server
    build:
      context: ./
      dockerfile: Dockerfile
    restart: unless-stopped
    networks:
      - gaseous
    depends_on:
      - gsdb
    ports:
      - 5198:80
    volumes:
      - gs:/mnt/user/appdata/gaseous-server
    environment:
      - dbhost=gsdb
      - dbuser=root
      - dbpass=gaseous
      - igdbclientid=(redacted)
      - igdbclientsecret=(redacted)
  gsdb:
    container_name: gsdb
    image: mysql:8
    restart: unless-stopped
    networks:
      - gaseous
    volumes:
      - gsdb:/mnt/user/appdata/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=gaseous
      - MYSQL_USER=gaseous
      - MYSQL_PASSWORD=gaseous
networks:
  gaseous:
    driver: bridge
volumes:
  gs:
  gsdb:

 

Remove the version line. It is no longer recommended for compose files.

 

Also is this a compose file you created yourself or something you pulled from a GitHub repo? If so did you specify a custom location when creating the stack? That is recommended when stacks require additional files beyond the compose file.

Edited October 5, 2023 by primeval_god

[primeval_god]

2 hours ago, L0rdRaiden said:

ultra minor bug

https://docs.docker.com/compose/compose-file/03-compose-file/

compose manager should support alternative file names for compose files, compose.yaml (preferred)

Not a bug, the compose manager plugin is an opinionated management gui. It is intended to only manage compose stacks created via its own interface, which always use the same naming scheme for the compose file and supplementary files. 

[bullmoose20]

4 hours ago, L0rdRaiden said:

are you building you own image? can you use one from docker hub?

chatted with the dev.. my bad.. I had the wrong docker-compose file

[L0rdRaiden]

13 hours ago, primeval_god said:

Not a bug, the compose manager plugin is an opinionated management gui. It is intended to only manage compose stacks created via its own interface, which always use the same naming scheme for the compose file and supplementary files. 

Anyway I just thinking further in case someday I migrate docker compose out of unraid.

Another question is regarding the env file

https://docs.docker.com/compose/compose-file/05-services/#env_file

Are you doing something in the background so the env file is loaded automatically if it in the same folder than the docker-compose.yml?

If I migrate this to a generic docker compose installation will I need to add "env_file: .env" under the services section? or if it is in the same folder is a native behaviour of docker to pick it up automatically?

 

I really appreciate your patience with me  

[L0rdRaiden]

another question

if the variable TZ is defined in the env file as TZ=whatever

since both have the same name, will this work?

or do I need to declare it like this

- TZ=${TZ}

 

 

I think it works but I can find it documented... so...

Edited October 5, 2023 by L0rdRaiden

[enk]

Did someone encounter problems with the compose down command? Yesterday I commanded a running stack to compose down and it blocked my server. Unraid was unaccessable via ssh, until docker itself finally crashed a few hours later. Then the server was accessible again (the docker tab showed the not running docker servce) and I could start a reboot. (I wasn't at home and could not issue an powercycle manually)

[primeval_god]

On 10/5/2023 at 10:15 AM, L0rdRaiden said:

Anyway I just thinking further in case someday I migrate docker compose out of unraid.

Another question is regarding the env file

https://docs.docker.com/compose/compose-file/05-services/#env_file

Are you doing something in the background so the env file is loaded automatically if it in the same folder than the docker-compose.yml?

If I migrate this to a generic docker compose installation will I need to add "env_file: .env" under the services section? or if it is in the same folder is a native behaviour of docker to pick it up automatically?

I am doing very little with the env file actually. Pretty much the editor just creates it in the project folder, and allows you to edit it. If it works (because i am not certain i use it myself) it is picked up because it is in the same folder as the compose file. I probably should be specifying it on the command line with either the --file or --env-file command. Using the "env_file:" option in a compose service i think has a slightly different behavior. Rather than making the variables contained in the file available to use in the compose file I think it directly passes them into the container as -e options. 

 

 

On 10/5/2023 at 2:15 PM, L0rdRaiden said:

- TZ=${TZ}

I think this is the correct format for using the .env file as is. If i understand correctly if you used the "env_file:" option for the container then the TZ from the file would be passed directly to the container, but you may not then be able to use other variables from the .env file for substitution elsewhere in the compose file.