Presjar Posted May 5, 2023 Share Posted May 5, 2023 (edited) On 4/29/2023 at 5:08 AM, L0rdRaiden said: my script to update compose with user scripts and cron jobs. Any feedback for improvement is welcome, I will improve it. Does this script take into account the /boot/config/plugins/compose.manager/projects/MYPROJECT/docker-compose.override.yml ? In a script I had previous been using, I defined the docker-compose.yml and the docker-Compose.override.yml as separate files when starting the stack outside of the plugin UI. I have also setup your script, my project directory is /mnt/cacheone/appdata/dockerConfiguration/compose However it doesn't seem to run docker-compose pull on my yml file. It goes straight to the No updates available notification. I'm very sure there are updates available. using the 'Update Stack' UI button results in downloads happening. I added a few Echos to see what is happening, but I can't see anything obviously wrong. # Define the root directory to search for projects projects_dir="/mnt/cacheone/appdata/dockerConfiguration/compose" echo "Project Directory: $projects_dir" # Find all subdirectories containing docker-compose files compose_files=$(find "$projects_dir" -maxdepth 2 -name "docker-compose.y*ml") echo "Located Compose Files: $compose_files" # Loop over each file found for file in $compose_files; do echo "I'm in the loop" # Get the directory containing the file dir="$(dirname "$file")" echo "dir: $dir" # Get the last folder name in the directory path project_name="$(basename "$dir")" echo "Project name: $project_name" # Change the current directory to the project directory pushd "$dir" # Check if there are any updates available for the Docker images if [[ $(docker-compose pull 2>&1) =~ 'Downloaded newer image' ]]; then # Get the names of the updated Docker images updated_images=$(docker-compose images --quiet | xargs docker inspect --format '{{.RepoTags}}' | tr -d '[] ' | sed 's/,/\n/g') # Stop any running containers associated with the project docker-compose down # Pull the latest version of the Docker images docker-compose pull # Start the containers in detached mode docker-compose up -d --remove-orphans # Clean up any unused Docker images docker image prune -f # Output a notification with the names of the updated Docker images echo "Updates available for project $project_name" echo "Updated Docker images:" echo "$updated_images" /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Docker Compose Updates" -d "$project_name updated the images $updated_images" -i "normal" else echo "No updates available for project $project_name" /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Docker Compose Updates" -d "$project_name no updates available" -i "normal" fi # Change back to the original directory popd done This is my terminal output with the extra echos Script location: /tmp/user.scripts/tmpScripts/3.Docker-Compose UPDATE/script Note that closing this window will abort the execution of this script Project Directory: /mnt/cacheone/appdata/dockerConfiguration/compose Located Compose Files: /mnt/cacheone/appdata/dockerConfiguration/compose/traefik2_stack/docker-compose.yml I'm in the loop dir: /mnt/cacheone/appdata/dockerConfiguration/compose/traefik2_stack Project name: traefik2_stack /mnt/cacheone/appdata/dockerConfiguration/compose/traefik2_stack / No updates available for project traefik2_stack / Any ideas where to go next? I really like the idea of your script. Much better than my original attempt which was just downloading all the images even if they are up to date. Edited May 5, 2023 by Presjar Quote Link to comment
L0rdRaiden Posted May 16, 2023 Share Posted May 16, 2023 (edited) @Presjar I have tried to fix it, not sure if it works or not yet. I have been trying as well to make Watchtower work with no success. #!/bin/bash # Define the root directory to search for projects projects_dir="/boot/config/plugins/compose.manager/projects" # Find all subdirectories containing docker-compose files compose_files=$(find "$projects_dir" -maxdepth 2 -name "docker-compose.y*ml") # Loop over each file found for file in $compose_files; do # Get the directory containing the file dir="$(dirname "$file")" # Get the last folder name in the directory path project_name="$(basename "$dir")" # Change the current directory to the project directory pushd "$dir" > /dev/null # Get the image IDs before pulling before_pull=$(docker-compose images --quiet) # Pull the latest version of the Docker images docker-compose pull # Get the image IDs after pulling after_pull=$(docker-compose images --quiet) # Compare the image IDs to check if any images have been updated if [[ "$before_pull" != "$after_pull" ]]; then # Get the names of the updated Docker images updated_images=$(docker-compose images --quiet | xargs docker inspect --format '{{.RepoTags}}' | tr -d '[] ' | sed 's/,/\n/g') # Stop any running containers associated with the project docker-compose down # Start the containers in detached mode docker-compose up -d --remove-orphans # Clean up any unused Docker images docker image prune -f # Output a notification with the names of the updated Docker images echo "Updates available for project $project_name" echo "Updated Docker images:" echo "$updated_images" /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Docker Compose Updates" -d "$project_name updated the images $updated_images" -i "normal" else echo "No updates available for project $project_name" /usr/local/emhttp/webGui/scripts/notify -e "Unraid Server Notice" -s "Docker Compose Updates" -d "$project_name no updates available" -i "normal" fi # Change back to the original directory popd > /dev/null done Edited May 16, 2023 by L0rdRaiden Quote Link to comment
L0rdRaiden Posted May 16, 2023 Share Posted May 16, 2023 @primeval_god Could you please share the commands you use in the plugin to run compose up/down and update? On the other hand I have this bug where this containers are detected like all of them have an updated while is not true and when I click on apply update I get this message "Configuration not found. Was this container created using this plugin?" Quote Link to comment
primeval_god Posted May 16, 2023 Author Share Posted May 16, 2023 4 hours ago, L0rdRaiden said: @primeval_god Could you please share the commands you use in the plugin to run compose up/down and update? If you enable the debug option the plugin should print the commands it uses to the system log when a button is clicked. 4 hours ago, L0rdRaiden said: On the other hand I have this bug where this containers are detected like all of them have an updated while is not true and when I click on apply update I get this message "Configuration not found. Was this container created using this plugin?" This is expected behavior. At this time dockerman does not correctly display the update status of containers created by other plugins such as compose and the update button it provides cannot be used to update such containers.. 1 Quote Link to comment
L0rdRaiden Posted May 17, 2023 Share Posted May 17, 2023 11 hours ago, primeval_god said: If you enable the debug option the plugin should print the commands it uses to the system log when a button is clicked. This is expected behavior. At this time dockerman does not correctly display the update status of containers created by other plugins such as compose and the update button it provides cannot be used to update such containers.. Instead of trying to do script like I'm trying can you add the feature to schedule auto updates in docker compose manager? CPU pinning support would be awesome as well. Quote Link to comment
primeval_god Posted May 17, 2023 Author Share Posted May 17, 2023 4 hours ago, L0rdRaiden said: Instead of trying to do script like I'm trying can you add the feature to schedule auto updates in docker compose manager? For autoupdates i would recommend something like Watchtower. 4 hours ago, L0rdRaiden said: CPU pinning support would be awesome as well. Should already be possible by adding cpuset into your compose file. Quote Link to comment
Avsynthe Posted May 20, 2023 Share Posted May 20, 2023 On 5/17/2023 at 11:36 PM, primeval_god said: For autoupdates i would recommend something like Watchtower. Would CA Watchtower work or should we be installing Watchtower using compose? Quote Link to comment
evan326 Posted May 21, 2023 Share Posted May 21, 2023 (edited) plugin: installing: compose.manager.plg Executing hook script: pre_plugin_checks plugin: downloading: compose.manager.plg ... done Executing hook script: pre_plugin_checks +============================================================================== | Skipping package compose.manager-package-2023.04.27 (already installed) +============================================================================== ---------------------------------------------------- Applying WebUI Patches... ---------------------------------------------------- plugin: run failed: '/bin/bash' returned 2 Executing hook script: post_plugin_checks Hi, I've been using this plugin for months and it's great. After updating to rc2 I'm having issues. Every few times I visit the docker page, the compose manager is gone. It seems sporadic, it is also after most, but not all reboots. I have to go to the app page, it isn't listed under installed apps. After reinstalling it, it works for a while. During installation I get these messages. I was going to try and uninstall and reinstall, but it doesn't ever say it's installed in the app page. Edited May 21, 2023 by evan326 Quote Link to comment
L0rdRaiden Posted May 23, 2023 Share Posted May 23, 2023 On 5/21/2023 at 4:14 AM, evan326 said: plugin: installing: compose.manager.plg Executing hook script: pre_plugin_checks plugin: downloading: compose.manager.plg ... done Executing hook script: pre_plugin_checks +============================================================================== | Skipping package compose.manager-package-2023.04.27 (already installed) +============================================================================== ---------------------------------------------------- Applying WebUI Patches... ---------------------------------------------------- plugin: run failed: '/bin/bash' returned 2 Executing hook script: post_plugin_checks Hi, I've been using this plugin for months and it's great. After updating to rc2 I'm having issues. Every few times I visit the docker page, the compose manager is gone. It seems sporadic, it is also after most, but not all reboots. I have to go to the app page, it isn't listed under installed apps. After reinstalling it, it works for a while. During installation I get these messages. I was going to try and uninstall and reinstall, but it doesn't ever say it's installed in the app page. I have the same problem, I upgrade to RC, the plugin disappears from the plugin list I installed it again, gives that error, it "works" but appears as not installed everywhere. Quote Link to comment
Lightarrow Posted May 23, 2023 Share Posted May 23, 2023 3 hours ago, L0rdRaiden said: I have the same problem, I upgrade to RC, the plugin disappears from the plugin list I installed it again, gives that error, it "works" but appears as not installed everywhere. I had the same issue and fixed it by going in the flash drive and manually deleting the plugin folder, then reinstalling it. 1 Quote Link to comment
Presjar Posted May 25, 2023 Share Posted May 25, 2023 (edited) On 5/16/2023 at 5:38 PM, L0rdRaiden said: @Presjar I have tried to fix it, not sure if it works or not yet. I have been trying as well to make Watchtower work with no success. On 5/17/2023 at 11:36 PM, primeval_god said: For autoupdates i would recommend something like Watchtower. Thanks for taking more of a look into it. Seems like the option on Watchtower is encouraged, but a bit silly when there is a button in the plugin that updates. This this can't be easily modified run on a schedule is confusing to me. Using another application to manage updates is a bit overkill. I know the basic script I have on schedule will download all the docker images again, which happens to also grab any updates. It just wastes a few MB downloading images I already have if no update is available. The compose plugin still works with the containers after running the update this way. #!/bin/bash #--------------Original Script docker compose -f /mnt/cacheone/appdata/dockerConfiguration/compose/traefik2_stack/docker-compose.yml -f /boot/config/plugins/compose.manager/projects/Traefik2_stack/docker-compose.override.yml -p traefik2_stack pull docker compose -f /mnt/cacheone/appdata/dockerConfiguration/compose/traefik2_stack/docker-compose.yml -f /boot/config/plugins/compose.manager/projects/Traefik2_stack/docker-compose.override.yml -p traefik2_stack up -d --remove-orphans docker image prune -f /usr/local/emhttp/webGui/scripts/notify -i normal -e "Docker Image Update Completed" -s "" -d "" Edit: Realized your script also just grabs all the image files, even if the image is current. But giving the notification of which images actually change after the pull is a great idea! I have your script running now up to the pull stage. But I had to identify the docker-compose full path. When I run your script as is I get the error that the docker-compose command is not recognized. But if I use the full path for the docker-compose the script works. It is strange because in the web terminal I can just type docker-compose and it works perfectly. Edited May 25, 2023 by Presjar Quote Link to comment
Matthew_K Posted May 29, 2023 Share Posted May 29, 2023 I have been experimenting with the Docker Compose plugin, either purposefully or accidentally, in an attempt to make a group of containers work harmoniously. Check it out: Throughout the process of setting this up, it's important to consider a few safety measures: 1. Prior to allowing users to edit the stack, remind them to first shut down the stack. 2 .Provide an option to save a working history. Essentially, whenever a Docker Compose configuration is successfully deployed, create a checkpoint or save point. I encountered an issue where I failed to follow rule 1, and then had to rely on rule 2 to revert the changes I made by using "compose down". Finally, I believe it would be more user-friendly if the Docker Compose interface opened in a new window or tab. Currently, when opened, it expands downward, requiring users to scroll down the page to see what was deployed. Quote Link to comment
Stupifier Posted June 1, 2023 Share Posted June 1, 2023 On 5/28/2023 at 10:26 PM, Matthew_K said: Finally, I believe it would be more user-friendly if the Docker Compose interface opened in a new window or tab. Currently, when opened, it expands downward, requiring users to scroll down the page to see what was deployed. This! It would be excellent to have the Docker Compose Interface be more easily accessible. I hate going to the docker page only to wait forever for all my various non-compose dockers to load on the page...then scroll all the way down to the bottom to do compose stuff. Something/Anything more streamlined would be appreciated. Good work on this plugin overall though, I love it. Quote Link to comment
primeval_god Posted June 4, 2023 Author Share Posted June 4, 2023 (edited) On 6/1/2023 at 2:45 PM, Stupifier said: This! It would be excellent to have the Docker Compose Interface be more easily accessible. I hate going to the docker page only to wait forever for all my various non-compose dockers to load on the page...then scroll all the way down to the bottom to do compose stuff. Something/Anything more streamlined would be appreciated. Good work on this plugin overall though, I love it. The compose interface is already in a tab. You just have to change the "Page View" setting to tabbed in the unRAID display settings. Edited June 4, 2023 by primeval_god 1 1 Quote Link to comment
Kilrah Posted June 5, 2023 Share Posted June 5, 2023 (edited) 16 hours ago, primeval_god said: The compose interface is already in a tab. You just have to change the "Page View" setting to tabbed in the unRAID display settings. Oof but then everything is tabbed, can't have a tabbed main page Edited June 5, 2023 by Kilrah Quote Link to comment
L0rdRaiden Posted June 21, 2023 Share Posted June 21, 2023 Some of my compose containers fail to start at boot, I'm under the impression that this is because some network components or other things of the OS are not ready so early. Anyone is experiencing the same thing? any way to fix it or add a delay? I'm using only the autostart feature that comes with compose manager, Thanks in advance. Quote Link to comment
Kilrah Posted June 21, 2023 Share Posted June 21, 2023 What network are they using? Quote Link to comment
L0rdRaiden Posted June 21, 2023 Share Posted June 21, 2023 41 minutes ago, Kilrah said: What network are they using? This configuration or similar (different names on the internal network) networks: br1: driver: macvlan external: true graylog-network: internal: true In a clean restart fails some of the compose fail to start, so I have to stop them and start them again. Quote Link to comment
L0rdRaiden Posted June 21, 2023 Share Posted June 21, 2023 (edited) this is how it looks like after restarting the array and if I try to start some of them I have to stop them and start again to work. Maybe a delay won't fix it and the issue is another but in any case delay and order features would be nice. Edited June 21, 2023 by L0rdRaiden Quote Link to comment
Kilrah Posted June 21, 2023 Share Posted June 21, 2023 (edited) On 6/21/2023 at 7:49 PM, L0rdRaiden said: Maybe a delay won't fix it and the issue is another but in any case delay and order features would be nice. It won't, the problem is that the brX networks are recreated from scratch on each boot. The stack references the network by ID, but when it's recreated it'll get a new random ID so the network referenced in the stack will never exist. Clicking Compose Up recreates the whole stack so it now references the new network. If you need macvlan/ipvlan for compose stacks you have to make your own network and enable "preserve custom networks" in docker settings. There should be a few posts in this thread about it. Edited July 13, 2023 by Kilrah Quote Link to comment
L0rdRaiden Posted June 21, 2023 Share Posted June 21, 2023 43 minutes ago, Kilrah said: It won't, the problem is that the brX networks are recreated from scratch on each boot. The stack references the network by ID, but when it's recreated it'll get a new random ID so the network referenced in the stack will never exist. Clicking Compose Up recreates the whole stack so it now references the new network. If you need macvlan for compose stacks you have to make your own network and enable "preserve custom networks" in docker settings. There should be a few posts in this thread about it. but br1 is a network created by unraid the internal ones are only docker compose, do you mean that those networks are the problematic? in any case might be something that can be solve in the plugin side, it doesn't make sense that if the plugin launches the compose file it doesn't work and If I do it manually pressing the button it works. Quote Link to comment
Kilrah Posted June 21, 2023 Share Posted June 21, 2023 (edited) Using brX is usually only required in special cases where the container needs a specific IP to have access to all ports. From a compose stack it requires the workaround I mentioned, but normally you should not need to use brX and thus not have the problem in the first place. So what's the reason for using brX in this case? 17 minutes ago, L0rdRaiden said: it doesn't make sense that if the plugin launches the compose file it doesn't work and If I do it manually pressing the button it works. Because autostart just starts the existing containers that have been created the last time Compose Up was clicked. The Compose Up button however deletes the existing containers and creates new ones. Edited June 21, 2023 by Kilrah Quote Link to comment
primeval_god Posted June 22, 2023 Author Share Posted June 22, 2023 4 hours ago, Kilrah said: Because autostart just starts the existing containers that have been created the last time Compose Up was clicked. The Compose Up button however deletes the existing containers and creates new ones. The composeman autostart runs a compose up on each stack. There should be no difference between the Compose Up button and autostart. Regardless i would bet that your diagnosis of the overall problem is correct. 4 hours ago, L0rdRaiden said: it doesn't make sense that if the plugin launches the compose file it doesn't work and If I do it manually pressing the button it works. My guess is that as @Kilrah is correct that the issue is the br1 network is recreated thus when compose tries to start the existing containers it fails, regardless whether it is via the button or autostart. If i understand your posts above you are doing compose down first then compose up to restart the stack. This makes sense as compose down followed by compose up removes all containers and then recreates them whereas compose up alone would just start existing containers. Quote Link to comment
Kilrah Posted June 22, 2023 Share Posted June 22, 2023 Oh right, had to do a down first when I had the issue. Quote Link to comment
L0rdRaiden Posted June 23, 2023 Share Posted June 23, 2023 On 6/22/2023 at 3:24 AM, primeval_god said: The composeman autostart runs a compose up on each stack. There should be no difference between the Compose Up button and autostart. Regardless i would bet that your diagnosis of the overall problem is correct. My guess is that as @Kilrah is correct that the issue is the br1 network is recreated thus when compose tries to start the existing containers it fails, regardless whether it is via the button or autostart. If i understand your posts above you are doing compose down first then compose up to restart the stack. This makes sense as compose down followed by compose up removes all containers and then recreates them whereas compose up alone would just start existing containers. Therefore is a problem with the plugin, because compose manager should do a clean stop of the compose files before restart or shutdown, so on next time it will autostart without issues. I need brx for security reasons, I need the containers to be in different vlans/networks and to setup different firewalls rules between hosts. This won't be possible if everything is behind on IP with a layer 3-4 fw. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.