[Plugin] Docker Compose Manager

[NasKaya]

On 5/2/2022 at 3:16 PM, primeval_god said:

Interesting, this should not be the case. Using compose pulls images into the local docker repo just like using docker directly. When restarting your system the image should exist locally and compose up should work just fine. What does your compose file look like?

 

This is how my compose file lookes like:

 

version: "2"
services:
  deconz:
    image: deconzcommunity/deconz:latest
    container_name: deconz-community
    restart: always
    volumes:
      - /mnt/user/appdata/deconzcommunity:/opt/deCONZ/
    devices:
      - /dev/serial/by-id/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2139560-if00:/dev/ttyACM0
    ports:
      - 80:80
      - 443:443
      - 6080:6080
    environment:
      - DECONZ_WEB_PORT=80
      - DECONZ_WS_PORT=443
      - DECONZ_VNC_MODE=1
      - DECONZ_NOVNC_PORT=6080
      - DECONZ_UPNP=1
      - DEBUG_INFO=1
      - DEBUG_APS=0
      - DEBUG_ZCL=0
      - DEBUG_ZDP=0
      - DEBUG_OTAU=0
    networks:
      br0:
        ipv4_address: 192.168.12.8
networks:
  br0:
    external: true

[Solverz]

Does the "compose pull" button, just issue the command docker compose pull or is there some way for it to also issue docker compose build? For locally created images?

[primeval_god]

19 hours ago, Solverz said:

Does the "compose pull" button, just issue the command docker compose pull or is there some way for it to also issue docker compose build? For locally created images?

It just issues compose pull for now. At the moment this plugin doesn't really have any considerations for the build aspect of compose.

[bergi9]

I have 2 issues with this plugin so far on my unraid 6.10.0-rc8:

1. Terminal output mode doesn't work as it just shows a 404 page. Basic output works.

bergi-stor nginx: 2022/05/16 22:32:41 [error] 5624#5624: *4467297 "/usr/local/emhttp/dockerterminal/compose_manager_action/index.html" is not found (2: No such file or directory) while sending to client, client: 172.18.0.6, server: , request: "GET /dockerterminal/compose_manager_action/ HTTP/1.1", host: "server.local", referrer: "http://server.local/plugins/compose.manager/php/show_ttyd.php?done=Done"

 

2. docker-compose can't pull private repository even if I'm logged in with docker login (via unraid web-cli )

Edited May 16, 2022 by bergi9
Added logs for the terminal output mode issue

[primeval_god]

On 5/16/2022 at 4:43 PM, bergi9 said:

I have 2 issues with this plugin so far on my unraid 6.10.0-rc8:

1. Terminal output mode doesn't work as it just shows a 404 page. Basic output works.

bergi-stor nginx: 2022/05/16 22:32:41 [error] 5624#5624: *4467297 "/usr/local/emhttp/dockerterminal/compose_manager_action/index.html" is not found (2: No such file or directory) while sending to client, client: 172.18.0.6, server: , request: "GET /dockerterminal/compose_manager_action/ HTTP/1.1", host: "server.local", referrer: "http://server.local/plugins/compose.manager/php/show_ttyd.php?done=Done"

 

Thanks good to know, i havent yet tested against 6.10 as I still run 6.9.

 

On 5/16/2022 at 4:43 PM, bergi9 said:

2. docker-compose can't pull private repository even if I'm logged in with docker login (via unraid web-cli )

I cant find a good reason that this should be the case. Are any errors reported in the compose log? 

[bergi9]

6 hours ago, primeval_god said:

I cant find a good reason that this should be the case. Are any errors reported in the compose log? 

I have a private repository on docker hub. This repository is not accessible to the user if the user is not logged in. So with common docker login and docker pull it works. For compose it says it can't access the repository, basically it does not use the docker logged in user.

Error response from daemon: pull access denied for <redacted/repo>, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

 

[primeval_god]

Does /root/.docker/config.json exist on your system? From what i can see compose should grab credentials from that file, which is where docker login caches them.

 

[bergi9]

16 hours ago, primeval_god said:

Does /root/.docker/config.json exist on your system? From what i can see compose should grab credentials from that file, which is where docker login caches them.

 

Yes, it exists and it's filled.

 

When I set the output mode to terminal, the pull with private repository works. Not for basic output mode.

Edited May 19, 2022 by bergi9
More testing

[OFark]

Seem as of 6.10 the docker compose commands just show a 404 window.

 

Server nginx: 2022/05/19 12:32:38 [error] 14890#14890: *80825 "/usr/local/emhttp/dockerterminal/compose_manager_action/index.html" is not found (2: No such file or directory) while sending to client, client: #####, server: , request: "GET /dockerterminal/compose_manager_action/ HTTP/1.1", host: "#####", referrer: "http://#####/plugins/compose.manager/php/show_ttyd.php?done=Done"

 

Edited May 19, 2022 by OFark

[spychodelics]

Logs after pressing compose, pull, down, up 

 

Unraid 6.10 stable

Compose.Manager 2022.05.14

 

May 19 14:11:28 unraid-server webGUI: Successful login user root from 141.100.90.229
May 19 14:11:36 unraid-server emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/checkall
May 19 14:16:33 unraid-server root: 20787
May 19 14:16:33 unraid-server root: ttyd -R -o -i '/var/tmp/compose_manager_action.sock' '/usr/local/emhttp/plugins/compose.manager/scripts/compose.sh' 'pull' '/boot/config/plugins/compose.manager/projects/Modulhandbuch/compose.yml' 'Modulhandbuch' > /dev/null &
May 19 14:16:33 unraid-server root: '/usr/local/emhttp/plugins/compose.manager/scripts/compose.sh' 'pull' '/boot/config/plugins/compose.manager/projects/Modulhandbuch/compose.yml' 'Modulhandbuch'
May 19 14:16:33 unraid-server root: /plugins/compose.manager/php/show_ttyd.php
May 19 14:16:34 unraid-server nginx: 2022/05/19 14:16:34 [error] 6274#6274: *621299 "/usr/local/emhttp/dockerterminal/compose_manager_action/index.html" is not found (2: No such file or directory) while sending to client, client: 141.100.90.229, server: , request: "GET /dockerterminal/compose_manager_action/ HTTP/1.1", host: "xxx.xxx.xxx.xxx:8080", referrer: "http://xxx.xxx.xxx.xxx:8080/plugins/compose.manager/php/show_ttyd.php?done=Done"
May 19 14:16:41 unraid-server root: 1312
May 19 14:16:41 unraid-server root: ttyd -R -o -i '/var/tmp/compose_manager_action.sock' '/usr/local/emhttp/plugins/compose.manager/scripts/compose.sh' 'down' '/boot/config/plugins/compose.manager/projects/Modulhandbuch/compose.yml' 'Modulhandbuch' > /dev/null &
May 19 14:16:41 unraid-server root: '/usr/local/emhttp/plugins/compose.manager/scripts/compose.sh' 'down' '/boot/config/plugins/compose.manager/projects/Modulhandbuch/compose.yml' 'Modulhandbuch'
May 19 14:16:41 unraid-server root: /plugins/compose.manager/php/show_ttyd.php
May 19 14:16:42 unraid-server nginx: 2022/05/19 14:16:42 [error] 6274#6274: *621299 "/usr/local/emhttp/dockerterminal/compose_manager_action/index.html" is not found (2: No such file or directory) while sending to client, client: 141.100.90.229, server: , request: "GET /dockerterminal/compose_manager_action/ HTTP/1.1", host: "xxx.xxx.xxx.xxx:8080", referrer: "http://xxx.xxx.xxx.xxx:8080/plugins/compose.manager/php/show_ttyd.php?done=Done"
May 19 14:16:46 unraid-server root: 1521
May 19 14:16:46 unraid-server root: ttyd -R -o -i '/var/tmp/compose_manager_action.sock' '/usr/local/emhttp/plugins/compose.manager/scripts/compose.sh' 'up' '/boot/config/plugins/compose.manager/projects/Modulhandbuch/compose.yml' 'Modulhandbuch' > /dev/null &
May 19 14:16:46 unraid-server root: '/usr/local/emhttp/plugins/compose.manager/scripts/compose.sh' 'up' '/boot/config/plugins/compose.manager/projects/Modulhandbuch/compose.yml' 'Modulhandbuch'
May 19 14:16:46 unraid-server root: /plugins/compose.manager/php/show_ttyd.php
May 19 14:16:47 unraid-server nginx: 2022/05/19 14:16:47 [error] 6274#6274: *621299 "/usr/local/emhttp/dockerterminal/compose_manager_action/index.html" is not found (2: No such file or directory) while sending to client, client: 141.100.90.229, server: , request: "GET /dockerterminal/compose_manager_action/ HTTP/1.1", host: "xxx.xxx.xxx.xxx:8080", referrer: "http://xxx.xxx.xxx.xxx:8080/plugins/compose.manager/php/show_ttyd.php?done=Done"

 

Edited May 19, 2022 by spychodelics

[primeval_god]

@OFark @spychodelics

This plugin has not yet been updated to work correctly with 6.10. Until it is, set the terminal output style to "Basic" in the settings page that should restore functionality. I hope to have an update made in the next couple of days.

Edited May 19, 2022 by primeval_god

[OFark]

Um, 

Quote

terminal output style to "Basic"

 Not sure where I can change this? Do you mean Unraid settings, or this plugins settings? (Not that I've found any settings for this plugin) 

 

If you mean Unraid settings, I don't see an output style option.

[primeval_god]

This plugin has a settings page, there should be a listing for compose.manager under the unraid setting tab.

[OFark]

Found it - Settings - User Utilities - Compose.

 

Thanks. That got it working. 

[rcortese]

On 10/5/2021 at 2:49 PM, hasown said:

 

FYI, the current version of Docker Compose supports both v2 and v3. The recommended way to write compose files is to omit the "version" tag at the top entirely. Compose will then read the yml and interpret all v2 and v3 features.

 

I've found conflicting statements regarding the version tag in docker docs.

 

About versions and upgrading states that omitting the version tag makes it a version 1, which is deprecated:

Quote

Version 1 (Deprecated)

Compose files that do not declare a version are considered “version 1”. In those files, all the services are declared at the root of the document.

 

Yet the Compose Specification (also linked at the same doc page above) states that the version tag is deprecated, and the file should behave as @hasown described:

Quote

The Compose file is a YAML file defining version (DEPRECATED), services (REQUIRED), networks, volumes, configs and secrets.

 

From Version top-level element:

Quote

Top-level version property is defined by the specification for backward compatibility but is only informative.

A Compose implementation SHOULD NOT use this version to select an exact schema to validate the Compose file, but prefer the most recent schema at the time it has been designed.

 

Which sounds great! Gonna try to remove it on some of my stacks. v3 has some good features but is getting more and more swarm oriented, and that is not my use case.

 

Anyhow, Unraid 6.10.0 seems to be giving Compose users some love! The changelog states that Docker Labels now make icons and GUI access possible:

Quote

- Docker labels are added to allow people using Docker compose to make use of icons and GUI access.

- Look at the Docker 'run' command output to see exactly what labels are used.

 

It would be a great addition to a Compose oriented plugin populating this new labels thing through a UI. Most of my VMs exist for resource segregation purposes (CPU pinning and RAM limiting), and all apps inside them are containerized.

 

Definately gonna try this plugin as opposed to manually installing compose using the go file. Great find.

 

Edited May 20, 2022 by rcortese
Quotes cleanup

[primeval_god]

24 minutes ago, rcortese said:

It would be a great addition to a Compose oriented plugin populating this new labels thing through a UI.

This is already planned but not yet implemented.

Edited May 19, 2022 by primeval_god

[primeval_god]

@OFark @bergi9 @spychodelics
A new version is available that fixes the terminal style output for 6.10.0. A big thanks to @bergi9 for helping me track down the issue. 

[arifer]

Hi there, I'm loving this plugin and can't wait for the next update with icon support.

 

I looked through the shell scripts in the `/usr/local/emhttp/plugins/compose.manager/event/` directory. I found a few oddities in the scripts that may need some looking at. The first is the hard-coding of the project names as "$dir/Name" which will not match any stack unless they are so incidentally named "Name". The second is the nested ifs that can be reduced, it's easier to read too that way. And the final oddity would be that none of these scripts are actually called anywhere. I'm guessing that this means that the auto-start feature is not in full-force yet.

 

In the spoiler below, I'm providing the diff to the changes to the shell scripts that I think may help with the first 2 problems mentioned. Please feel free to consider using them!

 

Spoiler

For the `started` script:

--- started
+++ started.patched
@@ -3,18 +3,15 @@
 COMPOSE_ROOT=/boot/config/plugins/compose.manager/projects
 COMPOSE_WRAPPER=/usr/local/emhttp/plugins/compose.manager/scripts/compose.sh

-for dir in $COMPOSE_ROOT/*; do
-    if [ -d "$dir" ]; then
-        if [ -f "$dir/compose.yml" ]; then
-            if [ -f "$dir/autostart" ]; then
-                if [ 'true' == "$(cat $dir/autostart)" ]; then
-                    # echo "$dir/compose.yml"
-                    logger "Starting compose stack: $(cat $dir/Name)"
-                    $COMPOSE_WRAPPER up "$dir/compose.yml" "$(cat $dir/Name)" > /dev/null &
-                fi
-            fi
-        fi
+while read AUTOSTART; do
+    STACK_DIR="$(dirname $AUTOSTART)"
+    STACK_NAME="${STACK_DIR#$COMPOSE_ROOT/}"
+    STACK_FILE="${STACK_DIR}/compose.yml"
+
+    if [ "$(cat $AUTOSTART)" == "false" ] && [ -f "${STACK_FILE}" ]; then
+        logger "Starting compose stack: ${STACK_NAME}"
+        $COMPOSE_WRAPPER up "$STACK_FILE" "$STACK_NAME" > /dev/null &
     fi
-done
+done <<< $(find $COMPOSE_ROOT -type f -name autostart)

 wait

 

For the `stopping_docker` script:

--- stopping_docker
+++ stopping_docker.patched
@@ -3,14 +3,12 @@
 COMPOSE_ROOT=/boot/config/plugins/compose.manager/projects
 COMPOSE_WRAPPER=/usr/local/emhttp/plugins/compose.manager/scripts/compose.sh

-for dir in $COMPOSE_ROOT/*; do
-    if [ -d "$dir" ]; then
-        if [ -f "$dir/compose.yml" ]; then
-            # echo "$dir/compose.yml"
-            logger "Stopping compose stack: $(cat $dir/Name)"
-            $COMPOSE_WRAPPER stop "$dir/compose.yml" "$(cat $dir/Name)" > /dev/null &
-        fi
-    fi
-done
+while read STACK_FILE; do
+    STACK_DIR="$(dirname $STACK_FILE)"
+    STACK_NAME="${STACK_DIR#$COMPOSE_ROOT/}"
+
+    logger "Stopping compose stack: ${STACK_NAME}"
+    $COMPOSE_WRAPPER stop "$STACK_FILE" "$STACK_NAME" > /dev/null &
+done <<< $(find $COMPOSE_ROOT -type f -name)

 wait

 

 

[primeval_god]

1 hour ago, arifer said:

And the final oddity would be that none of these scripts are actually called anywhere. I'm guessing that this means that the auto-start feature is not in full-force yet.

Autostart works just fine. The scripts in the event directory are triggered by unRAID system events.

 

1 hour ago, arifer said:

The first is the hard-coding of the project names as "$dir/Name" which will not match any stack unless they are so incidentally named "Name".

This is not the case. The compose.manager plugin stores stack names in a file called "Name" in each stack directory. "$(cat $dir/Name)" reads the stack name.

Edited May 22, 2022 by primeval_god

[arifer]

8 hours ago, primeval_god said:

Autostart works just fine. The scripts in the event directory are triggered by unRAID system events.

 

Ah I see, I never knew about that. Thank you for enlightening me!

 

8 hours ago, primeval_god said:

This is not the case. The compose.manager plugin stores stack names in a file called "Name" in each stack directory. "$(cat $dir/Name)" reads the stack name.

 

I find that all my stacks are written to a file titled `name` in lower case instead of being capitalised. This is also explicitly expressed in the `exec.php` file.

 

Adding a stack:

        file_put_contents("$folder/name",$stackName);

Changing the name of a stack:

        file_put_contents("$compose_root/$script/name",trim($newName));

[arifer]

I've put together a small patch to dockerMan so as to get the icons from compose labels working.

 

Solution:

The `/usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php` on unRAID needs to be changed. The problem about changing this file is that it does not sustain after reboots so it has to be patched every reboot. It's always possible to use the User Scripts plugin to edit the file on reboot at least until LimeTech fixes it fully or improves on it.

 

Manual method
 

Spoiler

Make a copy of the file first, just in case something goes wrong.

cp /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php.old

 

Navigate to  the comment "// read docker label for WebUI & Icon" at line 302 within the getAllInfo function. 4 lines down, change the following line:

if (!is_file($tmp['icon']) || $reload) $tmp['icon'] = $this->getIcon($image,$name);

to

if (!is_file($tmp['icon']) || $reload) $tmp['icon'] = $this->getIcon($image,$name,$tmp['icon']);

 

Next, navigate to the following getIcon function at line 338. Change the line from:

public function getIcon($Repository,$contName) {

to

public function getIcon($Repository,$contName,$tmpIconUrl='') {

 

Lastly, 2 lines down, under the following line:

$imgUrl = $this->getTemplateValue($Repository, 'Icon','all',$contName);

add

if (!$imgUrl) $imgUrl = $tmpIconUrl;

 

Script:

 

 

Spoiler

Either save this as an executable, or add it as a User script and run it on first boot.

 

#!/bin/bash

D_FILE="/usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php"

sed -i "s/getIcon(\$image,\$name)/getIcon(\$image,\$name,\$tmp['icon'])/" $D_FILE
sed -i "s/getIcon(\$Repository,\$contName)/getIcon(\$Repository,\$contName,\$tmpIconUrl='')/" $D_FILE
sed -i "s/\$imgUrl = \$this.*/&\n\t\tif (\!\$imgUrl) \$imgUrl = \$tmpIconUrl;/" $D_FILE

 

Refresh your Docker page and the icon should now show up. If it does not, you may need to clear your cache.

 

A problem with this patch is that the icon image does not change if you change the value of the label "net.unraid.docker.icon" in the compose file. To force an update, you will have to delete the previous icon files. Suppose the service name is "foo-service", 2 icons will be made:

1. /var/lib/docker/unraid/images/foo-service-icon.png
2. /usr/local/emhttp/state/plugins/dynamix.docker.manager/images/foo-service-icon.png

Delete both files and refresh your page to force unRAID to pull the new icons.

 

Reason:

Spoiler

It seems that while web-UI links get read and parsed fine, the compose icon labels are not used all the way. You can see it from the following segment in when preparing the icon.

public function getIcon($Repository,$contName) {
...
	$imgUrl = $this->getTemplateValue($Repository, 'Icon','all',$contName);
	if (!$imgUrl || trim($imgUrl) == "/plugins/dynamix.docker.manager/images/question.png") return '';
...
}

Looking back at getAllInfo, where getIcon is called to pull the icon images, the Docker label for the icon is actually read and stored in a temporary template array. However, this information is not being passed into the getIcon function, which only searches for the icon URL from the XML template files. To bypass this, we can add an optional parameter for getIcon to take in the icon URL from the Docker label and appropriately update the local variable $imgUrl when getIcon realises that the container does not have an XML template.

public function getIcon($Repository,$contName,$tmpIconUrl='') {
	...
	$imgUrl = $this->getTemplateValue($Repository, 'Icon','all',$contName);
	if (!$imgUrl) $imgUrl = $tmpIconUrl;
	...
}

 

A dirty hack, but a working hack.

 

 

Edited May 23, 2022 by arifer
Submitted when incomplete

[Squid]

35 minutes ago, arifer said:

I've put together a small patch to dockerMan so as to get the icons from compose labels working.

 

https://github.com/limetech/webgui

[arifer]

15 minutes ago, Squid said:

https://github.com/limetech/webgui

 

I can't believe that I never bothered to check if there was a repository for this! Will put up a PR on it. Thank you ^^

[emilgil]

I might have missed something here but whenever I use the Compose Pull-button, updated images for that stack is downloaded as expected. I then press Compose Up and all affected containers are restarted. However, the Update Ready flag is not removed from the running containers and I'm left with a number of dangling containers. Is this normal or am I doing something wrong? It doesn't matter if I shut the containers off (Compose Down) before pulling updates, I get the same behavior.

 

Currently running Unraid 6.10.1 with Compose manager plugin 2022-05-27. 

[primeval_god]

6 hours ago, emilgil said:

I might have missed something here but whenever I use the Compose Pull-button, updated images for that stack is downloaded as expected. I then press Compose Up and all affected containers are restarted. However, the Update Ready flag is not removed from the running containers and I'm left with a number of dangling containers. Is this normal or am I doing something wrong? It doesn't matter if I shut the containers off (Compose Down) before pulling updates, I get the same behavior.

This is expected behavior currently. The "update ready" notification in the unraid webui does not work correctly for containers created with compose. Additionally the compose pull functionality does not remove dangling images the way dockerman does. It is unknown if or when these issues will be addressed.