[Plugin] Docker Compose Manager

[rama3124]

Thanks, I'm more than happy to give this a go but fairly new to unraid so not sure how to modify the plugin. Is it a simple process?

[Joly0]

Hey guys, i might have found a bug. Not sure though.

So i was trying to build this project: https://github.com/ZeppelinBot/Zeppelin/tree/b28ca170edc39ec2248801c50bb42c2d2411f5f5 using this documentation https://zeppelin.wiki/setup/operating-systems/linux-docker

It is important to run

git checkout b28ca17

after git clone command.

I renamed the docker-compose.standalone.yml to docker-compose.yml and the .env.example to .env.
I added a docker compose project to the cloned folder and configured the docker-compose.yml to be the newly renamed one. All looked good. When i started the project, i noticed, that somethign was odd and that it was trying to read the 3 other .yml files aswell.

I also noticed, that the migrate container was not buioldign and running. I only got this fixed, after deleting all three additonal .yml files and leaving only the renamed docker-compose.yml behind.
So i assume, there is some kind of bug in the docker-compose plugin, or in docker-compose it-self, that results in loading the additional .yml files (for whatever reason) and tries to run them or so. I am really not sure.

[phil1c]

 

I'm testing switching to compose and I have a couple questions that I haven't been able to address from previous posts:

1. I understand that the way network names are displayed (ie: some very long string of characters) is due to dockerman.  Is there anything in the works to address this other than overriding "network_mode" in the compose.yml?
2. The test container (single container stack) that I've spun up doesn't show port mappings in the Unraid UI either.  Is there a way for me to address this?
3. I have the following webui label added to the compose.yml:

   net.unraid.docker.webui=http://[IP]:[PORT:6080]/vnc.html?resize=remote&host=[IP]&port=[PORT:6080]&autoconnect=1

   However, when I select the container icon and click "WebUI" no new page/tab attempts to open.  Is there something else I must do to make container webui links work?  I have already attempted to restart the entire server hoping it was an issue with webUI URLs "sticking" as I'd seen mentioned in other posts but.  Additionally, if I create a matching container from the Unraid GUI with the same webui url it works without issue.

Any help would be appreciated.

 

My stack is as follows

services:
  makemkv:
    container_name: pluginMakemkvTest
    networks:
      - bond043
    ports:
      - 5900:5900
      - 6080:6080
    privileged: true
    environment:
      - TZ=America/New_York
      - WEBPAGE_TITLE=MakeMKVplugin
      - VNC_PASSWORD=
      - ENABLE_STARTUP_SCRIPTS=yes
      - UMASK=000
      - PUID=99
      - PGID=100
    volumes:
      - /mnt/user:/media:rw
      - /mnt/cache-unprotected/Downloads/:/downloads:rw
      - /mnt/cache-protected-refresh/appdata/dockgeMakemkvTest:/config:rw
    labels:
      - net.unraid.docker.icon=https://raw.githubusercontent.com/binhex/docker-templates/master/binhex/images/makemkv-icon.png
      - net.unraid.docker.webui=http://[IP]:[PORT:6080]/vnc.html?resize=remote&host=[IP]&port=[PORT:6080]&autoconnect=1
    image: binhex/arch-makemkv
networks:
  bond043:
    name: bond0.43
    external: true

 

[mtongnz]

15 hours ago, phil1c said:

 

I'm testing switching to compose and I have a couple questions that I haven't been able to address from previous posts:

1. I understand that the way network names are displayed (ie: some very long string of characters) is due to dockerman.  Is there anything in the works to address this other than overriding "network_mode" in the compose.yml?
2. The test container (single container stack) that I've spun up doesn't show port mappings in the Unraid UI either.  Is there a way for me to address this?
3. I have the following webui label added to the compose.yml:

   net.unraid.docker.webui=http://[IP]:[PORT:6080]/vnc.html?resize=remote&host=[IP]&port=[PORT:6080]&autoconnect=1

   However, when I select the container icon and click "WebUI" no new page/tab attempts to open.  Is there something else I must do to make container webui links work?  I have already attempted to restart the entire server hoping it was an issue with webUI URLs "sticking" as I'd seen mentioned in other posts but.  Additionally, if I create a matching container from the Unraid GUI with the same webui url it works without issue.

Any help would be appreciated.

 

My stack is as follows

services:
  makemkv:
    container_name: pluginMakemkvTest
    networks:
      - bond043
    ports:
      - 5900:5900
      - 6080:6080
    privileged: true
    environment:
      - TZ=America/New_York
      - WEBPAGE_TITLE=MakeMKVplugin
      - VNC_PASSWORD=
      - ENABLE_STARTUP_SCRIPTS=yes
      - UMASK=000
      - PUID=99
      - PGID=100
    volumes:
      - /mnt/user:/media:rw
      - /mnt/cache-unprotected/Downloads/:/downloads:rw
      - /mnt/cache-protected-refresh/appdata/dockgeMakemkvTest:/config:rw
    labels:
      - net.unraid.docker.icon=https://raw.githubusercontent.com/binhex/docker-templates/master/binhex/images/makemkv-icon.png
      - net.unraid.docker.webui=http://[IP]:[PORT:6080]/vnc.html?resize=remote&host=[IP]&port=[PORT:6080]&autoconnect=1
    image: binhex/arch-makemkv
networks:
  bond043:
    name: bond0.43
    external: true

 

Fix for 1 & 2:

https://github.com/dcflachs/compose_plugin/pull/27

3 should work. Maybe try putting quotes around it

[primeval_god]

On 6/29/2024 at 9:01 AM, Joly0 said:

Hey guys, i might have found a bug. Not sure though.

So i was trying to build this project: https://github.com/ZeppelinBot/Zeppelin/tree/b28ca170edc39ec2248801c50bb42c2d2411f5f5 using this documentation https://zeppelin.wiki/setup/operating-systems/linux-docker

It is important to run

git checkout b28ca17

after git clone command.

I renamed the docker-compose.standalone.yml to docker-compose.yml and the .env.example to .env.
I added a docker compose project to the cloned folder and configured the docker-compose.yml to be the newly renamed one. All looked good. When i started the project, i noticed, that somethign was odd and that it was trying to read the 3 other .yml files aswell.

I also noticed, that the migrate container was not buioldign and running. I only got this fixed, after deleting all three additonal .yml files and leaving only the renamed docker-compose.yml behind.
So i assume, there is some kind of bug in the docker-compose plugin, or in docker-compose it-self, that results in loading the additional .yml files (for whatever reason) and tries to run them or so. I am really not sure.

The compose plugin is intended for running simple compose stacks created via its webui not arbitrary compose stacks pulled from github. I dont know off hand how multiple additional compose files in the stack root directory would effect the commands used to start and stop stacks.

[primeval_god]

18 hours ago, mtongnz said:

Fix for 1 & 2:

https://github.com/dcflachs/compose_plugin/pull/27

3 should work. Maybe try putting quotes around it

Actually I think the fix for 1 & 2 is this https://github.com/unraid/webgui/pull/1616

[phil1c]

57 minutes ago, primeval_god said:

Actually I think the fix for 1 & 2 is this https://github.com/unraid/webgui/pull/1616

Ok, so for now I can either manually incorporate the changes made there or wait for Unraid to merge the changes, correct?  Have you used these changes?  I see a comment that it breaks editing existing containers (here) : is this true?

 

19 hours ago, mtongnz said:

Fix for 1 & 2:

https://github.com/dcflachs/compose_plugin/pull/27

3 should work. Maybe try putting quotes around it

Thanks for the suggestion.  I added quotes and yet still nothing. I inspected the rendered HTML of the page to see what link was created and I have the following:

1. When I check on a similar container created using the unraid UI, I see the following:

   function onclick(event) {
     addDockerContainerContext('binhex-makemkv', '096fa2b590ee', '/boot/config/plugins/dockerMan/templates-user/my-binhex-makemkv.xml', 1, 0, 0, false, 'http://10.43.64.31:6080/vnc.html?resize=remote&host=10.43.64.31&port=6080&autoconnect=1', ....

   You can see the link was "rendered" (for lack of a better term) from the webUI URL entered in the container template, same as above.

2. When I check on the compose-stack-created single container, i see the following:

   unction onclick(event) {
     addDockerContainerContext('pluginMakemkvTest', '096fa2b590ee', '', 1, 0, 3, false, 'http://[IP]:[PORT:6080]/vnc.html?resize=remote&host=[IP]&port=[PORT:6080]&autoconnect=1',

   As you can see, in the case of the compose stack container, the "webUI URL" is entered directly as typed into the "UI Labels" field. This does not change if I add quotes, remove them, enter the webUI URL into the UI Labels field, nor if I leave the UI Labels field blank and only leave the

For some reason, the compose created container isn't actually "generating" the webUI from the given template.

 

I'm on 6.12.10 if that makes a difference.

 

 

[primeval_god]

9 hours ago, phil1c said:

This does not change if I add quotes, remove them, enter the webUI URL into the UI Labels field, nor if I leave the UI Labels field blank and only leave the

If it doesnt change when you change the text of the label then its because the webui is using a cached version as described elsewhere in this thread. You can use the "docker inspect" command in a shell to see what the contents of the net.unraid.docker.webui label are. That string is exactly what unRAID Dockerman sees, how it process it and if it caches it are in part of unRAIDs code.

[apandey]

On 6/30/2024 at 1:22 AM, phil1c said:

I have the following webui label added to the compose.yml:

net.unraid.docker.webui=http://[IP]:[PORT:6080]/vnc.html?resize=remote&host=[IP]&port=[PORT:6080]&autoconnect=1

However, when I select the container icon and click "WebUI" no new page/tab attempts to open.  Is there something else I must do to make container webui links work?

I keep my docker compose files in version control and apply them to unraid server using ansible. The reason it doesn't work for you is because the unraid managed labels get overridden in a separate docker-compose.override.yml file sitting next to the docker-compose.yml

 

assuming your stackname/docker-compose.yml has the following:

 

services:
  serviceA:
    image: something:latest
    labels:
      some.non-unraid.label: some-value

 

then create a stackname/docker-compose.override.yml with something like below. I use folder view plugin, and have some unraid labels there too, but you only need to include what you want

 

services:
  serviceA:
    labels:
      folder.view: Network
      net.unraid.docker.managed: 'composeman'
      net.unraid.docker.icon: 'https://url/to/icon.png'
      net.unraid.docker.webui: 'http://[IP]:[PORT:8443]'
      net.unraid.docker.shell: '/bin/bash'

 

you can check the contents of /boot/config/plugins/compose.manager/projects/<stackname>/ to see the structure (I also deploy name and autorun files to fully automate my setup)

Edited July 1 by apandey

[apandey]

On 6/23/2024 at 1:56 PM, mtongnz said:

I put in a PR to fix this but it's not merged yet:

https://github.com/dcflachs/compose_plugin/pull/27

You can manually modify the plugin and test it if you'd like.

I applied this to my server, and it seems to work very well. Any chance this can be merged so we can all benefit going forward?

[phil1c]

14 hours ago, apandey said:

you can check the contents of /boot/config/plugins/compose.manager/projects/<stackname>/ to see the structure (I also deploy name and autorun files to fully automate my setup)

Ok, I found the matching override files as you described above and made the relevant changes, as below:

services:
  makemkv:
    labels:
      net.unraid.docker.managed: 'composeman'
      net.unraid.docker.icon: '${ICON_URL}'
      net.unraid.docker.webui: 'http://[IP]:[PORT:6080]/vnc.html?resize=remote&host=[IP]&port=[PORT:6080]&autoconnect=1'
      net.unraid.docker.shell: 'bash'

 

After making the changes, I followed the instructions here to force the update of the plugins docker.json file to grab the new webui URL.  This worked as I temporarily set the url to google just to make sure the change was occurring and that it was a link (successful: webui link opened a tab to google . com), after which I returned it to the value above.  However, still the url was not "rendered" into an actual link.  The override file is as described above and the docker.json shows the following:

 "pluginMakemkvTest": {
        "running": true,
        ...
        "url": "http://[IP]:[PORT:6080]/vnc.html?resize=remote&host=[IP]&port=[PORT:6080]&autoconnect=1",
        "icon": "/state/plugins/dynamix.docker.manager/images/pluginMakemkvTest-icon.png",
        ...
    },

and 'docker inspect pluginMakemkvTest' shows the following:

"net.unraid.docker.managed": "composeman",
"net.unraid.docker.shell": "bash",
"net.unraid.docker.webui": "http://[IP]:[PORT:6080]/vnc.html?resize=remote&host=[IP]&port=[PORT:6080]&autoconnect=1",

So the value from the override is being correctly passed to the docker manager, it's just not being parsed into a url, it's being taken wholesale as a text string.

 

Out of curiosity, I changed the webui url again (in the override file) to the exact same "webui URL" of a totally unrelated but running container, forced the recreation of the docker.json file, and spun up the compose stack.  Still, the webui URL does not parse:

docker.json

 "pluginMakemkvTest": {
        "running": true,
...
        "url": "http://[IP]:[PORT:8989]",
        "icon": "/state/plugins/dynamix.docker.manager/images/pluginMakemkvTest-icon.png",

docker inspect

"net.unraid.docker.managed": "composeman",
"net.unraid.docker.shell": "bash",
"net.unraid.docker.webui": "http://[IP]:[PORT:8989]",

 

I'm not sure what's relevant here, so I'll add:

1. Looking at the docker.json file, all of my running containers with webui URLs have a properly parsed url.  All of the non-running containers show the raw text used to create the URL.

Running container from docker.json

"librespeed": {
        "running": true,
        ...
        "url": "http://actual.IPv4.address:80",

Non-running container from docker.json

"DiskSpeed": {
        "running": false,
        ...
        "url": "http://[IP]:[PORT:8888]/",

1. The icon URL correctly pulls the url from the .env variables
2. EDIT:  Adding here: when i inspect the rendered HTML of the Docker tab page, I do see the updated URL ("http://[IP]:[PORT:8989]") as the url for the webui link, so the url string is going from override file > docker manager > compose plugin docker.json AND rendered Docker tab webpage.

 

Is there something that is supposed to trigger docker manager to parse the webui URL into an actual link upon starting a container that isn't occurring when using the compose plugin?

 

Happy to do more digging/troubleshooting to help sort this out.

Edited July 2 by phil1c

[phil1c]

14 hours ago, primeval_god said:

If it doesnt change when you change the text of the label then its because the webui is using a cached version as described elsewhere in this thread. You can use the "docker inspect" command in a shell to see what the contents of the net.unraid.docker.webui label are. That string is exactly what unRAID Dockerman sees, how it process it and if it caches it are in part of unRAIDs code.

 

I found this post and followed the instructions to force the recreation of the docker.json file to manually fix this.  You can take a look at my previous post to see the steps I tried already.

[sir_storealot]

I really like this plugin, it is very useful!

 

However, I find I have to constantly increase the zoom level of my browser when editing the compose or environment files. Is it possible to somehow increase the font size in the built-in editor? Currently the font of the editor is significantly smaller than the rest of the Unraid GUI text, which forces me to zoom in each time.

[phil1c]

On 7/1/2024 at 9:43 AM, primeval_god said:

If it doesnt change when you change the text of the label then its because the webui is using a cached version as described elsewhere in this thread. You can use the "docker inspect" command in a shell to see what the contents of the net.unraid.docker.webui label are. That string is exactly what unRAID Dockerman sees, how it process it and if it caches it are in part of unRAIDs code.

Do you have a moment to look at my update posts detailing my further investigation into the issue I'm facing?

[mikegeezy]

I'm trying to get this set of containers up and running:
 

https://github.com/Labelbase/Labelbase/blob/master/install.md

 

So far, so good, but I'm getting a permission error when it tries to start the Django container.  Is that an indication that it needs to be run with elevated privileges, or is there something else going on here?  I don't really like running these as root.  Feel it's not worth the risk and I don't have the time (or knowledge frankly) do look under the hood.

 

Thanks all, great plugin so far!

 

[primeval_god]

16 hours ago, mikegeezy said:

I'm trying to get this set of containers up and running:
 

https://github.com/Labelbase/Labelbase/blob/master/install.md

 

So far, so good, but I'm getting a permission error when it tries to start the Django container.  Is that an indication that it needs to be run with elevated privileges, or is there something else going on here?  I don't really like running these as root.  Feel it's not worth the risk and I don't have the time (or knowledge frankly) do look under the hood.

 

Thanks all, great plugin so far!

 

This thread is for questions about the composeman plugin itself rather than specific compose stacks. As this is pulled from github, (not the envisioned use case of this plugin) I might hazard a guess that some of the file permissions are incorrect as unraid doesn't really have users other than root. How that interplay with permissions withing the container environment I dont know.

[yayitazale]

Hi:

 

I'm having a issue while trying to update a stack, I get a connection close before it pulls all the containers images:

 

 

 

[mikegeezy]

7 hours ago, primeval_god said:

This thread is for questions about the composeman plugin itself rather than specific compose stacks. As this is pulled from github, (not the envisioned use case of this plugin) I might hazard a guess that some of the file permissions are incorrect as unraid doesn't really have users other than root. How that interplay with permissions withing the container environment I dont know.

 

Hey thanks for your response!  Didn't realize I was in "off-label" territory.  You were right in the sense that it was some kind of permission issue.  I had it in the folder /boot/config/plugins/compose.manager/projects/ but was unable to chown on it there for some reason. 

 

I was able to fix it by moving it to /mnt/user/appdata and:

 

chmod -R 0777 /mnt/user/appdata/Labelbase/

chown -R nobody:users /mnt/user/appdata/Labelbase/

 

[primeval_god]

3 hours ago, mikegeezy said:

 

Hey thanks for your response!  Didn't realize I was in "off-label" territory.

Yeah this plugin is really designed for simple, single compose file, single env file stacks created via its webui. Stacks that are pulled from elsewhere can quickly start to get more complex than this plugin was designed for.

[primeval_god]

On 7/12/2024 at 12:33 PM, phil1c said:

Do you have a moment to look at my update posts detailing my further investigation into the issue I'm facing?

As a test try removing the connection to the external bond043 network (compose will create a network for the container). 

[phil1c]

On 7/15/2024 at 9:24 AM, primeval_god said:

As a test try removing the connection to the external bond043 network (compose will create a network for the container). 

Ok.  I commented out all references to the external bond043 network.  I wiped the docker.json and brought the stack up and it worked!  I now have proper WebUI links.  Of course, I'd like to keep my dockers on that separate external network.  How else do you need me to troubleshoot to help determine the root cause?

 

 

For reference, my compose file and override file (which do generate the correct webui URL) look like this now:

docker-compose.yml

services:
  makemkv:
    container_name: pluginMakemkvTest
#    networks:
#      - bond043
    ports:
      - 5910:5900
      - 6090:6080
    privileged: true
    environment:
      - TZ=America/New_York
      - WEBPAGE_TITLE=MakeMKVplugin
      - VNC_PASSWORD=
      - ENABLE_STARTUP_SCRIPTS=yes
      - UMASK=000
      - PUID=99
      - PGID=100
    volumes:
      - /mnt/user:/media:rw
      - /mnt/cache-unprotected/Downloads/:/downloads:rw
      - /mnt/cache-protected-refresh/appdata/dockgeMakemkvTest:/config:rw
    image: binhex/arch-makemkv
#networks:
#  bond043:
#    name: bond0.43
#    external: true

docker-compose.override.yml

services:
  makemkv:
    labels:
      net.unraid.docker.managed: 'composeman'
      net.unraid.docker.icon: '${ICON_URL}'
      net.unraid.docker.webui: 'http://[IP]:[PORT:6080]/vnc.html?resize=remote&host=[IP]&port=[PORT:6080]&autoconnect=1'
      net.unraid.docker.shell: 'bash'

 

[primeval_god]

9 hours ago, phil1c said:

How else do you need me to troubleshoot to help determine the root cause?

It would be interesting to know what kind of network bond0.43 is. Also whether the issue happens if the network (of the same type as bond0.43) is not external (i.e. created when the compose stack is brought up).

 

Unfortunately at this point a fix is likely out of my hands. The issue is something about the way Dockerman tries to extract the IP and PORT information from the container in order to re-write the url. For some reason it is not playing nicely with the way compose is attaching the network to the container. I suspect it may be related to the issue where dockerman doesnt display network names for containers created by compose but i am not certain. regardless any fixes will have to be made in dockerman.

[primeval_god]

@Mihle This thread is for discussing issues with the compose plugin, not for supporting/discussing specific compose stacks. 

[phil1c]

9 hours ago, primeval_god said:

It would be interesting to know what kind of network bond0.43 is. Also whether the issue happens if the network (of the same type as bond0.43) is not external (i.e. created when the compose stack is brought up).

Bond0 is a bonded group of 2x 1gbe network connections using 802.3ad mode.  Bond0.43 is a subinterface for accessing VLAN43 where I segment all dockers and VMs on this machine.  Docker uses macvlan for custom network types.

 

If there is something else you want to know about my network, let me know.  I understand that this may be a bug with Dockerman, but still happy to help satisfy curiosities to determine a root cause.

[primeval_god]

48 minutes ago, phil1c said:

Docker uses macvlan for custom network types.

Custom networks can be macvlan, ipvlan, and bridge type.