[Plugin] Docker Compose Manager

[Kees Fluitman]

hey guys. I suddenly have an issue, where I click anything, compose up, or update stack, it opens the Modal window, and instantly disconnects. So nothing happens. Any ideas? I cant update or run most of my stacks now. I'll do it manually for now, but it's weird.

 

Edited January 9 by Kees Fluitman

[primeval_god]

9 hours ago, Kees Fluitman said:

hey guys. I suddenly have an issue, where I click anything, compose up, or update stack, it opens the Modal window, and instantly disconnects. So nothing happens. Any ideas? I cant update or run most of my stacks now. I'll do it manually for now, but it's weird.

Has anything changed on your system recently? Any changes with your stacks? Anything in your system log?

Edited January 10 by primeval_god

[Kees Fluitman]

6 minutes ago, primeval_god said:

Has anything changed on your system recently? Any changes with your stacks? Anything in your system log?

hey. I've started noticing it didnt work anywhere. Any modal was throwing errors. The web dev console showed 507 responses I believe. That the server didnt have sufficient storage:
 

Quote

507 Insufficient Storage (WebDAV)

I dont know why WebDAV though? Or if they mean memory...anyway...

 

So i figured let's dive deeper. And multiple thing started to act weird. I then looked at my memory allocations, and usage. And it seemed there was no more free memory. At least, not real "free" memory. There was allocated memory for cache/buffer. but even that was starting to lower. At it's peak, memory used was 32GB, the rest cache/buffer (total 47GB).

So i shutdown the system. and restarted. And now it's back to normal (25GB used, 11GB cache, and 8/10GBfree). I do already see a slow incrementation in my Grafana Chart. But i guess that could also be grafana itself at the moment? This eating of my memory started when i started to setup all my monitoring (Grafana, Prometheus, Telegraf, Loki, etc.) to play around. So I'll keep a close eye on that.

Edited January 10 by Kees Fluitman

[primeval_god]

On 1/5/2024 at 3:54 PM, OddMagnet said:

Is the viewing of logs on a container basis bugged?

Not to my knowledge. Is traefik configured to log to a file or to stdout?

[Ed Wardo]

17 hours ago, primeval_god said:

When creating a stack there is an advanced option (in a dropdown at the bottom) for specifying the project location. If you need to add extra files to the compose project (such as a build directory) you should select a project location on a pool or array.

 

Awesome. That'll do the trick!

I didn't use the GUI to create the project. I had just used the terminal to create a new dir in the projects directory.

 

Looking at the results of doing it through the GUI, it seems that it creates 2 text files: indirect and name

 

I would assume you could convert a project by moving the files and then creating these to files after docker-compose down?

[primeval_god]

6 hours ago, Ed Wardo said:

I would assume you could convert a project by moving the files and then creating these to files after docker-compose down?

The best way would be to create a new project with the gui, then copy the files from your old project into the new project directory. The stack must be down for this to work.

 

6 hours ago, Ed Wardo said:

I didn't use the GUI to create the project. I had just used the terminal to create a new dir in the projects directory.

Looking at the results of doing it through the GUI, it seems that it creates 2 text files: indirect and name

The compose manager plugin is really designed for managing simple compose stacks created through its webui (similar to how dockerman is designed to manage containers launched via dockerman). Compose stacks launched via other means, including the command line are not explicitly supported. Additionally compose projects pulled from github (or other such places) are not explicitly supported as they, like random dockerhub images, are not necessarily designed with unRAID in mind. 

All that to say if you are beyond a simple single compose file and an env file, here be dragons.

[mrpops2ko]

On 10/3/2021 at 8:22 PM, primeval_god said:

Compose Manager

Beta Release!

This plugin installs docker compose and compose switch.

Use "docker compose" or "docker-compose" from the command line.

See https://docs.docker.com/compose/cli-command/ for additional details.

Install via Community Applications

 

This plugin now adds a very basic control tab for compose on the docker page. The controls allow the creation of compose yaml files, as well as allowing you to issue compose up and compose down commands from the web ui. The web ui components are based heavily on the user.scripts plugin so a huge shoutout and thanks goes to @Squid.

 

Future Work:

• Add scheduling for compose commands?
• Allow more options for configuring compose commands.

 

hi im not sure if you are accepting feature requests but this would be useful for me. 

 

i've been down a rabbit hole of trying to do something like having a global .env file which i can use to do references and interpolation. it seems thats not possible if you have docker-compose files in directories and try to reference outside of it, even by hardcoding. 

i tried both env_file full/path and env_file ../.env and neither work. 

i'm wondering if we could have a global menu added and what that does, is upon boot, create, start and stop of any containers it will check the values in the global .env file and place them into each containers .env file.

this would then allow easy referencing in compose, so for example instead of having to write out a long /mnt/ssd/cache/appdata/ you could reference that was $DOCKORDIR and just call $DOCKORDIR each time. 

[mtongnz]

On 1/15/2024 at 1:04 PM, mrpops2ko said:

 

hi im not sure if you are accepting feature requests but this would be useful for me. 

 

i've been down a rabbit hole of trying to do something like having a global .env file which i can use to do references and interpolation. it seems thats not possible if you have docker-compose files in directories and try to reference outside of it, even by hardcoding. 

i tried both env_file full/path and env_file ../.env and neither work. 

i'm wondering if we could have a global menu added and what that does, is upon boot, create, start and stop of any containers it will check the values in the global .env file and place them into each containers .env file.

this would then allow easy referencing in compose, so for example instead of having to write out a long /mnt/ssd/cache/appdata/ you could reference that was $DOCKORDIR and just call $DOCKORDIR each time. 

 

@dcflachs has just merged my pull request for this:  https://github.com/dcflachs/compose_plugin/pull/24  Should hopefully have an update available in unRaid very soon.

 

To use it, create a file called envToUse in the project folder (i.e. where the compose.yaml file is) and put the full path to the env file.  I use vscode-server to manage my files so it's pretty easy.  But if there's a bit of interest, I might look at building a UI integration of some sort.

[mrpops2ko]

5 hours ago, mtongnz said:

 

@dcflachs has just merged my pull request for this:  https://github.com/dcflachs/compose_plugin/pull/24  Should hopefully have an update available in unRaid very soon.

 

To use it, create a file called envToUse in the project folder (i.e. where the compose.yaml file is) and put the full path to the env file.  I use vscode-server to manage my files so it's pretty easy.  But if there's a bit of interest, I might look at building a UI integration of some sort.

thanks but can you elaborate more on this because i think it might not be exactly what i wanted now

 

what i wanted was to put the envToUse in /boot/config/plugins/compose.manager/projects/envToUse

 

and then that env file would subsequently be copied / replicated or symlinked to all the sub directories (i.e /boot/config/plugins/compose.manager/projects/plex/.env)

the reason for this, is that the .env has special properties that otherwise addressing them via env_file declaring doesn't do. like my previous example with doing interpolation $DOCKORDIR

your previous comment 'Is there a Github repo for this?  I've modified some of this plugin locally to allow use of a "master" .env with multiple projects and I'd like to share it back.'

seemed to hint at doing exactly this. if i declare an envToUse does it replicate across other projects?

[L0rdRaiden]

Could you please include docker scout cli binaries as part of compose manager?

[primeval_god]

8 hours ago, mtongnz said:

 

@dcflachs has just merged my pull request for this:  https://github.com/dcflachs/compose_plugin/pull/24  Should hopefully have an update available in unRaid very soon.

 

To use it, create a file called envToUse in the project folder (i.e. where the compose.yaml file is) and put the full path to the env file.  I use vscode-server to manage my files so it's pretty easy.  But if there's a bit of interest, I might look at building a UI integration of some sort.

Actually I made some changes beyond yours, the UI integration is done and i changed the name of the envToUse file. There is now a button for each stack called "Settings", selecting that will bring up a popup window with a field where you can change the path to the env file for the stack. There should be no need to manually edit files in the projects folders.

 

@mrpops2ko

With this new functionality you can specify a shared env file for some or all of your stacks. 

Edited January 16 by primeval_god

[primeval_god]

2 hours ago, L0rdRaiden said:

Could you please include docker scout cli binaries as part of compose manager?

I am sorry @L0rdRaiden but at this time I have no plans to add docker scout to the compose manager. I would prefer to keep the contents of this plugin as tightly focused on supporting simple compose stacks as possible. Additionally I already get some flak for not keeping up with the docker compose releases I am not really keen on adding something else i wont keep up to date. 

 

The bits of this plugin that actually pull and package the compose binary are fairly simple if you were interesting in making a plugin for installing docker scout though.

[OddMagnet]

On 1/10/2024 at 2:50 AM, primeval_god said:

Not to my knowledge. Is traefik configured to log to a file or to stdout?

I had my traefik configured to log to a file... 🤦‍♂️

 

New question, are there plans to show extraction progress in the UI as well?
I thought that my compose-up was stuck when pulling Home Assistant (first time, haven't used that before)

After restarting a couple of times I decided to try a manual `docker pull` and figured out that the extraction is just extremely slow.

[primeval_god]

2 hours ago, OddMagnet said:

New question, are there plans to show extraction progress in the UI as well?
I thought that my compose-up was stuck when pulling Home Assistant (first time, haven't used that before)

After restarting a couple of times I decided to try a manual `docker pull` and figured out that the extraction is just extremely slow.

What output style are you using? Check the compose manager setting page.

[OddMagnet]

I'm currently using `Terminal`

[primeval_god]

Ok in that case you have as much information as is available about the compose up process. It is the direct output of the compose utility, no different than if you were to execute compose up via the terminal.

[mrpops2ko]

7 hours ago, primeval_god said:

Actually I made some changes beyond yours, the UI integration is done and i changed the name of the envToUse file. There is now a button for each stack called "Settings", selecting that will bring up a popup window with a field where you can change the path to the env file for the stack. There should be no need to manually edit files in the projects folders.

 

@mrpops2ko

With this new functionality you can specify a shared env file for some or all of your stacks. 

yeah unfortunately this implementation suffers with the exact same issues i mentioned previously in relation to interpolation. 

 

the only real solution to do global envs is to have one which sits outside and has a symlink to it in the project directory called .env

 

the other solutions wont work. this is because docker has some special functions associated with .env rather than anything else, which allows it to pass interpolated variables as well as env variables. 

 

 

[primeval_god]

10 minutes ago, mrpops2ko said:

yeah unfortunately this implementation suffers with the exact same issues i mentioned previously in relation to interpolation. 

 

the only real solution to do global envs is to have one which sits outside and has a symlink to it in the project directory called .env

 

the other solutions wont work. this is because docker has some special functions associated with .env rather than anything else, which allows it to pass interpolated variables as well as env variables. 

Are you certain? Have you tried it? If I understand correctly specifying the env file using the --env-file flag in the compose command (which is how this latest change works) is the equivalent to having the env file in the same folder as the compose file rather than being the equivalent of specifying the env file in the compose file using the env_file section.

Edited January 16 by primeval_god

[mrpops2ko]

2 minutes ago, primeval_god said:

Are you certain? Have you tried it? If I understand correctly specifying the env file using the --env-file flag in the compose command (which is how this latest change works) is the equivalent to having the env file in the same folder as the compose file rather than being the equivalent of specifying the env file in the compose file using the env_file section.

yes tested it just now and it doesn't work. its not the same. .env is special and env_file won't work, nor will -e flag 

 

it has to be in the same folder as the project

[mrpops2ko]

there are effectively 2 options to have proper global .env support, its either have a file that sits at /boot/config/plugins/compose.manager/projects/.env and symlink it to each project folder, so for example /boot/config/plugins/compose.manager/projects/plex/.env would be a symlink of /boot/config/plugins/compose.manager/projects/.env

 

or alternatively on every click of the docker up / docker down / stack update, it would take the values from /boot/config/plugins/compose.manager/projects/.env and append them to that specific projects .env file

 

this would then allow for variable interpolation in the compose files, as most people desire

[OddMagnet]

47 minutes ago, primeval_god said:

Ok in that case you have as much information as is available about the compose up process. It is the direct output of the compose utility, no different than if you were to execute compose up via the terminal.

 

That's weird, when I do a 

docker pull some_image

in the normal terminal I also get extraction progress

[primeval_god]

2 hours ago, OddMagnet said:

That's weird, when I do a 

docker pull some_image

in the normal terminal I also get extraction progress

The cli equivalent is docker compose up, not docker pull. The images are pulled as part of the up process.

Edited January 17 by primeval_god

[primeval_god]

3 hours ago, mrpops2ko said:

yes tested it just now and it doesn't work. its not the same. .env is special and env_file won't work, nor will -e flag 

Can you give me a simple composefile and env file combination illustrating the problem?

[mrpops2ko]

47 minutes ago, primeval_god said:

Can you give me a simple composefile and env file combination illustrating the problem?

take this one for example

 

version: "3.9"
services:
  tautulli:
    image: lscr.io/linuxserver/tautulli:latest
    container_name: tautulli
    env_file:
      - path: /boot/config/plugins/compose.manager/projects/.env
      service: tautulli
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=${TZ}
    ports:
      - 8181:8181
    restart: unless-stopped
    networks:
      br0:
        ipv4_address: 192.168.1.133
    volumes:
      - ${DOCKERDIR}/tautulli/config:/config

networks:
  br0:
    external: true

you can delete the networking, its just using ipvlan but you can see we are declaring variables ${DOCKORDIR} and ${TZ}

now define those values in your .env lets say 

TZ=UTC

DOCKERDIR=/mnt/cache/appdata

watch as you'll be unable to run, or rather when you run it you get an error stating that it cant find those variables so its defaulting to blank ones. This is because .env is special and will accept both environment variables and interpolate. declaring like above wont do that. 

if you wanted to get rid of the error you could on your docker host do `export DOCKERDIR="/mnt/cache/appdata"' but this would then store that in the unraid host which isn't desired. 

now if you just simply copy the .env from /boot/config/plugins/compose.manager/projects/.env and place it in /boot/config/plugins/compose.manager/projects/tautulli/.env (so it resides alongside the docker compose file) then everything will just work because of the things i mentioned. 

[primeval_god]

13 minutes ago, mrpops2ko said:

    env_file:
      - path: /boot/config/plugins/compose.manager/projects/.env
      service: tautulli

 

I dont know what the above section does but it it seems to break things. The following works just fine with a 

version: "3.9"
services:
  tautulli:
    image: lscr.io/linuxserver/tautulli:latest
    container_name: tautulli
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=${TZ}
    ports:
      - 8181:8181
    restart: unless-stopped
    networks:
      br0:
        ipv4_address: 192.168.1.133
    volumes:
      - ${DOCKERDIR}/tautulli/config:/config

with an env file like this 

TZ=UTC
DOCKERDIR=/mnt/cache/appdata

placed in /tmp/env-test-2/