[Plugin] Docker Compose Manager

[Presjar]

On 4/29/2023 at 5:08 AM, L0rdRaiden said:

my script to update compose with user scripts and cron jobs.

 

Any feedback for improvement is welcome, I will improve it.

 

Does this script take into account the /boot/config/plugins/compose.manager/projects/MYPROJECT/docker-compose.override.yml ?

 

In a script I had previous been using, I defined the docker-compose.yml and the docker-Compose.override.yml as separate files when starting the stack outside of the plugin UI.

 

I have also setup your script, my project directory is /mnt/cacheone/appdata/dockerConfiguration/compose 

 

However it doesn't seem to run docker-compose pull on my yml file. It goes straight to the No updates available notification. I'm very sure there are updates available. using the 'Update Stack' UI button results in downloads happening.

I added a few Echos to see what is happening, but I can't see anything obviously wrong.

# Define the root directory to search for projects
projects_dir="/mnt/cacheone/appdata/dockerConfiguration/compose"
echo "Project Directory: $projects_dir"
# Find all subdirectories containing docker-compose files
compose_files=$(find "$projects_dir" -maxdepth 2 -name "docker-compose.y*ml")
echo "Located Compose Files: $compose_files"
# Loop over each file found
for file in $compose_files; do
  echo "I'm in the loop"
  # Get the directory containing the file
  dir="$(dirname "$file")"
echo "dir: $dir"
  # Get the last folder name in the directory path
  project_name="$(basename "$dir")"
echo "Project name: $project_name"
  # Change the current directory to the project directory
  pushd "$dir"

  # Check if there are any updates available for the Docker images
  if [[ $(docker-compose pull 2>&1) =~ 'Downloaded newer image' ]]; then
    # Get the names of the updated Docker images
    updated_images=$(docker-compose images --quiet | xargs docker inspect --format '{{.RepoTags}}' | tr -d '[] ' | sed 's/,/\n/g')

    # Stop any running containers associated with the project
    docker-compose down

    # Pull the latest version of the Docker images
    docker-compose pull

    # Start the containers in detached mode
    docker-compose up -d --remove-orphans

    # Clean up any unused Docker images
    docker image prune -f

    # Output a notification with the names of the updated Docker images
    echo "Updates available for project $project_name"
    echo "Updated Docker images:"
    echo "$updated_images"
    /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Docker Compose Updates" -d "$project_name updated the images $updated_images" -i "normal"

  else
    echo "No updates available for project $project_name"
    /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Docker Compose Updates" -d "$project_name no updates available" -i "normal"

  fi

  # Change back to the original directory
  popd
done

This is my terminal output with the extra echos

Script location: /tmp/user.scripts/tmpScripts/3.Docker-Compose UPDATE/script
Note that closing this window will abort the execution of this script
Project Directory: /mnt/cacheone/appdata/dockerConfiguration/compose
Located Compose Files: /mnt/cacheone/appdata/dockerConfiguration/compose/traefik2_stack/docker-compose.yml
I'm in the loop
dir: /mnt/cacheone/appdata/dockerConfiguration/compose/traefik2_stack
Project name: traefik2_stack
/mnt/cacheone/appdata/dockerConfiguration/compose/traefik2_stack /
No updates available for project traefik2_stack
/

Any ideas where to go next? I really like the idea of your script. Much better than my original attempt which was just downloading all the images even if they are up to date.

 

Edited May 5, 2023 by Presjar

[L0rdRaiden]

@Presjar

I have tried to fix it, not sure if it works or not yet.

 

I have been trying as well to make Watchtower work with no success.

 

#!/bin/bash

# Define the root directory to search for projects
projects_dir="/boot/config/plugins/compose.manager/projects"

# Find all subdirectories containing docker-compose files
compose_files=$(find "$projects_dir" -maxdepth 2 -name "docker-compose.y*ml")

# Loop over each file found
for file in $compose_files; do
  # Get the directory containing the file
  dir="$(dirname "$file")"

  # Get the last folder name in the directory path
  project_name="$(basename "$dir")"

  # Change the current directory to the project directory
  pushd "$dir" > /dev/null

  # Get the image IDs before pulling
  before_pull=$(docker-compose images --quiet)

  # Pull the latest version of the Docker images
  docker-compose pull

  # Get the image IDs after pulling
  after_pull=$(docker-compose images --quiet)

  # Compare the image IDs to check if any images have been updated
  if [[ "$before_pull" != "$after_pull" ]]; then
    # Get the names of the updated Docker images
    updated_images=$(docker-compose images --quiet | xargs docker inspect --format '{{.RepoTags}}' | tr -d '[] ' | sed 's/,/\n/g')

    # Stop any running containers associated with the project
    docker-compose down

    # Start the containers in detached mode
    docker-compose up -d --remove-orphans

    # Clean up any unused Docker images
    docker image prune -f

    # Output a notification with the names of the updated Docker images
    echo "Updates available for project $project_name"
    echo "Updated Docker images:"
    echo "$updated_images"
    /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Docker Compose Updates" -d "$project_name updated the images $updated_images" -i "normal"
  else
    echo "No updates available for project $project_name"
    /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Docker Compose Updates" -d "$project_name no updates available" -i "normal"
  fi

  # Change back to the original directory
  popd > /dev/null
done

 

Edited May 16, 2023 by L0rdRaiden

[L0rdRaiden]

@primeval_god Could you please share the commands you use in the plugin to run compose up/down and update?

 

On the other hand I have this bug where this containers are detected like all of them have an updated while is not true

and when I click on apply update I get this message

 

"Configuration not found. Was this container created using this plugin?"

 

[primeval_god]

4 hours ago, L0rdRaiden said:

@primeval_god Could you please share the commands you use in the plugin to run compose up/down and update?

If you enable the debug option the plugin should print the commands it uses to the system log when a button is clicked.

 

4 hours ago, L0rdRaiden said:

On the other hand I have this bug where this containers are detected like all of them have an updated while is not true

and when I click on apply update I get this message

 

"Configuration not found. Was this container created using this plugin?"

This is expected behavior. At this time dockerman does not correctly display the update status of containers created by other plugins such as compose and the update button it provides cannot be used to update such containers.. 

[L0rdRaiden]

11 hours ago, primeval_god said:

If you enable the debug option the plugin should print the commands it uses to the system log when a button is clicked.

 

This is expected behavior. At this time dockerman does not correctly display the update status of containers created by other plugins such as compose and the update button it provides cannot be used to update such containers.. 

 

Instead of trying to do script like I'm trying can you add the feature to schedule auto updates in docker compose manager?

CPU pinning support would be awesome as well.

[primeval_god]

4 hours ago, L0rdRaiden said:

 

Instead of trying to do script like I'm trying can you add the feature to schedule auto updates in docker compose manager?

For autoupdates i would recommend something like Watchtower.

 

4 hours ago, L0rdRaiden said:

CPU pinning support would be awesome as well.

Should already be possible by adding cpuset into your compose file.

[Avsynthe]

On 5/17/2023 at 11:36 PM, primeval_god said:

For autoupdates i would recommend something like Watchtower.

 

Would CA Watchtower work or should we be installing Watchtower using compose?

[evan326]

plugin: installing: compose.manager.plg
Executing hook script: pre_plugin_checks
plugin: downloading: compose.manager.plg ... done

Executing hook script: pre_plugin_checks

+==============================================================================
| Skipping package compose.manager-package-2023.04.27 (already installed)
+==============================================================================


----------------------------------------------------
Applying WebUI Patches...
----------------------------------------------------

plugin: run failed: '/bin/bash' returned 2
Executing hook script: post_plugin_checks

Hi, I've been using this plugin for months and it's great. After updating to rc2 I'm having issues. Every few times I visit the docker page, the compose manager is gone. It seems sporadic, it is also after most, but not all reboots. I have to go to the app page, it isn't listed under installed apps. After reinstalling it, it works for a while. During installation I get these messages. 

I was going to try and uninstall and reinstall, but it doesn't ever say it's installed in the app page.

Edited May 21, 2023 by evan326

[L0rdRaiden]

On 5/21/2023 at 4:14 AM, evan326 said:

plugin: installing: compose.manager.plg
Executing hook script: pre_plugin_checks
plugin: downloading: compose.manager.plg ... done

Executing hook script: pre_plugin_checks

+==============================================================================
| Skipping package compose.manager-package-2023.04.27 (already installed)
+==============================================================================


----------------------------------------------------
Applying WebUI Patches...
----------------------------------------------------

plugin: run failed: '/bin/bash' returned 2
Executing hook script: post_plugin_checks

Hi, I've been using this plugin for months and it's great. After updating to rc2 I'm having issues. Every few times I visit the docker page, the compose manager is gone. It seems sporadic, it is also after most, but not all reboots. I have to go to the app page, it isn't listed under installed apps. After reinstalling it, it works for a while. During installation I get these messages. 

I was going to try and uninstall and reinstall, but it doesn't ever say it's installed in the app page.

 

I have the same problem, I upgrade to RC, the plugin disappears from the plugin list I installed it again, gives that error, it "works" but appears as not installed everywhere.

[Lightarrow]

3 hours ago, L0rdRaiden said:

I have the same problem, I upgrade to RC, the plugin disappears from the plugin list I installed it again, gives that error, it "works" but appears as not installed everywhere.

I had the same issue and fixed it by going in the flash drive and manually deleting the plugin folder, then reinstalling it. 

[Presjar]

On 5/16/2023 at 5:38 PM, L0rdRaiden said:

@Presjar

I have tried to fix it, not sure if it works or not yet.

 

I have been trying as well to make Watchtower work with no success.

 

On 5/17/2023 at 11:36 PM, primeval_god said:

For autoupdates i would recommend something like Watchtower.

 

Thanks for taking more of a look into it. Seems like the option on Watchtower is encouraged, but a bit silly when there is a button in the plugin that updates. This this can't be easily modified run on a schedule is confusing to me. Using another application to manage updates is a bit overkill.

I know the basic script I have on schedule will download all the docker images again, which happens to also grab any updates. It just wastes a few MB downloading images I already have if no update is available. The compose plugin still works with the containers after running the update this way.

#!/bin/bash
#--------------Original Script
docker compose -f /mnt/cacheone/appdata/dockerConfiguration/compose/traefik2_stack/docker-compose.yml -f /boot/config/plugins/compose.manager/projects/Traefik2_stack/docker-compose.override.yml -p traefik2_stack pull
docker compose -f /mnt/cacheone/appdata/dockerConfiguration/compose/traefik2_stack/docker-compose.yml -f /boot/config/plugins/compose.manager/projects/Traefik2_stack/docker-compose.override.yml -p traefik2_stack up -d --remove-orphans
docker image prune -f

/usr/local/emhttp/webGui/scripts/notify -i normal -e "Docker Image Update Completed" -s "" -d ""

 

Edit:
Realized your script also just grabs all the image files, even if the image is current.  But giving the notification of which images actually change after the pull is a great idea!

I have your script running now up to the pull stage. But I had to identify the docker-compose full path. When I run your script as is I get the error that the docker-compose command is not recognized. But if I use the full path for the docker-compose the script works. It is strange because in the web terminal I can just type docker-compose and it works perfectly.

Edited May 25, 2023 by Presjar

[Matthew_K]

I have been experimenting with the Docker Compose plugin, either purposefully or accidentally, in an attempt to make a group of containers work harmoniously. Check it out:

Throughout the process of setting this up, it's important to consider a few safety measures:

1. Prior to allowing users to edit the stack, remind them to first shut down the stack.

2 .Provide an option to save a working history. Essentially, whenever a Docker Compose configuration is successfully deployed, create a checkpoint or save point.

I encountered an issue where I failed to follow rule 1, and then had to rely on rule 2 to revert the changes I made by using "compose down".
 

Finally, I believe it would be more user-friendly if the Docker Compose interface opened in a new window or tab. Currently, when opened, it expands downward, requiring users to scroll down the page to see what was deployed.

[Stupifier]

On 5/28/2023 at 10:26 PM, Matthew_K said:

Finally, I believe it would be more user-friendly if the Docker Compose interface opened in a new window or tab. Currently, when opened, it expands downward, requiring users to scroll down the page to see what was deployed.

This! It would be excellent to have the Docker Compose Interface be more easily accessible. I hate going to the docker page only to wait forever for all my various non-compose dockers to load on the page...then scroll all the way down to the bottom to do compose stuff.

Something/Anything more streamlined would be appreciated.

Good work on this plugin overall though, I love it.

[primeval_god]

On 6/1/2023 at 2:45 PM, Stupifier said:

This! It would be excellent to have the Docker Compose Interface be more easily accessible. I hate going to the docker page only to wait forever for all my various non-compose dockers to load on the page...then scroll all the way down to the bottom to do compose stuff.

Something/Anything more streamlined would be appreciated.

Good work on this plugin overall though, I love it.

The compose interface is already in a tab. You just have to change the "Page View" setting to tabbed in the unRAID display settings.

Edited June 4, 2023 by primeval_god

[Kilrah]

16 hours ago, primeval_god said:

The compose interface is already in a tab. You just have to change the "Page View" setting to tabbed in the unRAID display settings.

Oof but then everything is tabbed, can't have a tabbed main page

Edited June 5, 2023 by Kilrah

[L0rdRaiden]

Some of my compose containers fail to start at boot, I'm under the impression that this is because some network components or other things of the OS are not ready so early. Anyone is experiencing the same thing? any way to fix it or add a delay?

 

I'm using only the autostart feature that comes with compose manager,

 

Thanks in advance.

[Kilrah]

What network are they using?

[L0rdRaiden]

41 minutes ago, Kilrah said:

What network are they using?

 

This configuration or similar (different names on the internal network)

 

networks:
  br1:
    driver: macvlan
    external: true
  graylog-network:
    internal: true

 

In a clean restart fails some of the compose fail to start, so I have to stop them and start them again.

[L0rdRaiden]

this is how it looks like after restarting the array

 

 

and if I try to start some of them

 

 

 

 

I have to stop them and start again to work.

 

Maybe a delay won't fix it and the issue is another but in any case delay and order features would be nice.

Edited June 21, 2023 by L0rdRaiden

[Kilrah]

On 6/21/2023 at 7:49 PM, L0rdRaiden said:

Maybe a delay won't fix it and the issue is another but in any case delay and order features would be nice.

It won't, the problem is that the brX networks are recreated from scratch on each boot. The stack references the network by ID, but when it's recreated it'll get a new random ID so the network referenced in the stack will never exist. Clicking Compose Up recreates the whole stack so it now references the new network.

 

If you need macvlan/ipvlan for compose stacks you have to make your own network and enable "preserve custom networks" in docker settings. There should be a few posts in this thread about it.

Edited July 13, 2023 by Kilrah

[L0rdRaiden]

43 minutes ago, Kilrah said:

It won't, the problem is that the brX networks are recreated from scratch on each boot. The stack references the network by ID, but when it's recreated it'll get a new random ID so the network referenced in the stack will never exist. Clicking Compose Up recreates the whole stack so it now references the new network.

 

If you need macvlan for compose stacks you have to make your own network and enable "preserve custom networks" in docker settings. There should be a few posts in this thread about it.

 

 

but br1 is a network created by unraid

 

the internal ones are only docker compose, do you mean that those networks are the problematic? in any case might be something that can be solve in the plugin side, it doesn't make sense that if the plugin launches the compose file it doesn't work and If I do it manually pressing the button it works.

[Kilrah]

Using brX is usually only required in special cases where the container needs a specific IP to have access to all ports. From a compose stack it requires the workaround I mentioned, but normally you should not need to use brX and thus not have the problem in the first place. So what's the reason for using brX in this case?

 

  

17 minutes ago, L0rdRaiden said:

it doesn't make sense that if the plugin launches the compose file it doesn't work and If I do it manually pressing the button it works.

Because autostart just starts the existing containers that have been created the last time Compose Up was clicked. The Compose Up button however deletes the existing containers and creates new ones.

Edited June 21, 2023 by Kilrah

[primeval_god]

4 hours ago, Kilrah said:

Because autostart just starts the existing containers that have been created the last time Compose Up was clicked. The Compose Up button however deletes the existing containers and creates new ones.

The composeman autostart runs a compose up on each stack. There should be no difference between the Compose Up button and autostart. Regardless i would bet that your diagnosis of the overall problem is correct.

 

4 hours ago, L0rdRaiden said:

it doesn't make sense that if the plugin launches the compose file it doesn't work and If I do it manually pressing the button it works.

My guess is that as @Kilrah is correct that the issue is the br1 network is recreated thus when compose tries to start the existing containers it fails, regardless whether it is via the button or autostart. If i understand your posts above you are doing compose down first then compose up to restart the stack. This makes sense as compose down followed by compose up removes all containers and then recreates them whereas compose up alone would just start existing containers.

[Kilrah]

Oh right, had to do a down first when I had the issue.

[L0rdRaiden]

On 6/22/2023 at 3:24 AM, primeval_god said:

The composeman autostart runs a compose up on each stack. There should be no difference between the Compose Up button and autostart. Regardless i would bet that your diagnosis of the overall problem is correct.

 

My guess is that as @Kilrah is correct that the issue is the br1 network is recreated thus when compose tries to start the existing containers it fails, regardless whether it is via the button or autostart. If i understand your posts above you are doing compose down first then compose up to restart the stack. This makes sense as compose down followed by compose up removes all containers and then recreates them whereas compose up alone would just start existing containers.

 

Therefore is a problem with the plugin, because compose manager should do a clean stop of the compose files before restart or shutdown, so on next time it will autostart without issues.

 

I need brx for security reasons, I need the containers to be in different vlans/networks and to setup different firewalls rules between hosts. This won't be possible if everything is behind on IP with a layer 3-4 fw.