[Plugin] Docker Compose Manager

Kilrah · February 29

I believe a fix in the webgui/dockerman has been already merged to stop tracking updates from non-template containers.

tescher · February 29

Is it possible to test this fix? I'm not totally perfect in Github, but it should be possible to swap a file maybe?

Thanks

L0rdRaiden · March 7

I have a problem with this compose, If I start the server and docker start automatically the proxy won't work, the dockers runs no apparent error appears but it will return a 404.

If I stop and start the compose it will work at any time, just after a reboot, just after an array stop, at any point in time it will return correctly the app behind traefik.

Any idea why is this and how to troubleshoot it?

###############################################################
# Web Proxy DMZ
###############################################################

# Common settings #############################################

x-default: &config
  restart: unless-stopped
  cpuset: 10,22,11,23,8,20,9,21
  security_opt:
    - no-new-privileges:true
x-dns: &dns
  dns:
    - 10.10.50.5
    - 10.10.50.6
x-labels: &labels
  com.centurylinklabs.watchtower.enable: "true"
  net.unraid.docker.managed: "composeman"
  net.unraid.docker.shell: "sh"

# Services ####################################################

services:

## Traefik ####################################################

  traefik:
    container_name: ProxyDMZ-Traefik
    image: traefik:latest
    <<: [*config, *dns]
    depends_on:
    # - wpsocketproxy:
    # - modsecurity
      - crowdsec
    networks:
    # wp-netsocketproxy:
    # wp-netmodsecurity:
      wp-netredis:
      eth2:
        ipv4_address: ${ProxyDMZTraefik_ip}
    ports:
      - 80:80
      - 443:443
      - 8080:8080 # Dashboard port
    volumes:
      - /mnt/services/docker/WebProxyDMZ/Traefik:/etc/traefik/
    environment:
      - TZ
    # - DOCKER_HOST=wpsocketproxy:2375
      - CF_API_EMAIL_FILE
      - CF_DNS_API_TOKEN_FILE
    secrets:
      - CF_API_EMAIL
      - CF_DNS_API_TOKEN
    labels:
      <<: *labels
      net.unraid.docker.icon: "https://raw.githubusercontent.com/ibracorp/unraid-templates/master/icons/traefik.png"
      net.unraid.docker.webui: "https://traefik.domain.com/dashboard/#/"

## CrowdSec ###################################################

  crowdsec:
    image: crowdsecurity/crowdsec
    container_name: ProxyDMZ-CrowdSec
    <<: [*config, *dns]
    depends_on:
      - redis-cs
    networks:
      eth2:
        ipv4_address: ${ProxyDMZCrowdSec_ip}
    ports:
      - 8080:8080
     #- 6060:6060 # PROMETEUS
    environment:
      TZ:
      COLLECTIONS: "crowdsecurity/traefik crowdsecurity/home-assistant crowdsecurity/http-cve crowdsecurity/whitelist-good-actors"
    # GID: "${GID-1000}"
      PGID:
      PUID:
      CUSTOM_HOSTNAME: CrowdSecDMZ
      DISABLE_LOCAL_API: "false" # True Only after successfully registering and validating remote agent below.
    volumes:
      - /mnt/services/docker/WebProxyDMZ/CrowdSec/data:/var/lib/crowdsec/data
      - /mnt/services/docker/WebProxyDMZ/CrowdSec:/etc/crowdsec
      - /mnt/services/docker/WebProxyDMZ/Traefik/logs:/var/log/traefik:ro
      - /mnt/services/docker/HomeAssistant:/var/log/homeassistant:ro    
    labels:
      <<: *labels
      net.unraid.docker.icon: "https://raw.githubusercontent.com/ibracorp/app-logos/main/crowdsec/crowdsec.png"

## CrowdSec - Redis ###########################################

  redis-cs:
    image: redis:alpine
    container_name: ProxyDMZ-CrowdSec-Redis
    <<: *config
    command: [ "sh", "-c", "exec redis-server --requirepass $REDIS_PASSWORD" ]  # redis-cli -a "password" --stat # select 1 # dbsize
    networks:
      wp-netredis:
    volumes:
      - /mnt/services/docker/WebProxyDMZ/Redis:/data
    environment:
      - TZ
    labels:
      <<: *labels
      net.unraid.docker.icon: "https://raw.githubusercontent.com/A75G/docker-templates/master/templates/icons/redis.png"


# Networks ####################################################

networks:
  eth2:
    name: eth2
    external: true
  eth1:
    name: eth1
    external: true
# wp-netsocketproxy:
#   internal: true
  wp-netredis:
    internal: true
# wp-netmodsecurity:
#   internal: true

# Secrets #####################################################

secrets:
  # Traefik - CF_API_EMAIL
  CF_API_EMAIL:
    file: $SECRETSDIR/CF_API_EMAIL
  # Traefik - CF_API_EMAIL
  CF_DNS_API_TOKEN:
    file: $SECRETSDIR/CF_DNS_API_TOKEN

mtongnz · March 8

11 hours ago, L0rdRaiden said:

I have a problem with this compose, If I start the server and docker start automatically the proxy won't work, the dockers runs no apparent error appears but it will return a 404.

If I stop and start the compose it will work at any time, just after a reboot, just after an array stop, at any point in time it will return correctly the app behind traefik.

Any idea why is this and how to troubleshoot it?

###############################################################
# Web Proxy DMZ
###############################################################

# Common settings #############################################

x-default: &config
  restart: unless-stopped
  cpuset: 10,22,11,23,8,20,9,21
  security_opt:
    - no-new-privileges:true
x-dns: &dns
  dns:
    - 10.10.50.5
    - 10.10.50.6
x-labels: &labels
  com.centurylinklabs.watchtower.enable: "true"
  net.unraid.docker.managed: "composeman"
  net.unraid.docker.shell: "sh"

# Services ####################################################

services:

## Traefik ####################################################

  traefik:
    container_name: ProxyDMZ-Traefik
    image: traefik:latest
    <<: [*config, *dns]
    depends_on:
    # - wpsocketproxy:
    # - modsecurity
      - crowdsec
    networks:
    # wp-netsocketproxy:
    # wp-netmodsecurity:
      wp-netredis:
      eth2:
        ipv4_address: ${ProxyDMZTraefik_ip}
    ports:
      - 80:80
      - 443:443
      - 8080:8080 # Dashboard port
    volumes:
      - /mnt/services/docker/WebProxyDMZ/Traefik:/etc/traefik/
    environment:
      - TZ
    # - DOCKER_HOST=wpsocketproxy:2375
      - CF_API_EMAIL_FILE
      - CF_DNS_API_TOKEN_FILE
    secrets:
      - CF_API_EMAIL
      - CF_DNS_API_TOKEN
    labels:
      <<: *labels
      net.unraid.docker.icon: "https://raw.githubusercontent.com/ibracorp/unraid-templates/master/icons/traefik.png"
      net.unraid.docker.webui: "https://traefik.domain.com/dashboard/#/"

## CrowdSec ###################################################

  crowdsec:
    image: crowdsecurity/crowdsec
    container_name: ProxyDMZ-CrowdSec
    <<: [*config, *dns]
    depends_on:
      - redis-cs
    networks:
      eth2:
        ipv4_address: ${ProxyDMZCrowdSec_ip}
    ports:
      - 8080:8080
     #- 6060:6060 # PROMETEUS
    environment:
      TZ:
      COLLECTIONS: "crowdsecurity/traefik crowdsecurity/home-assistant crowdsecurity/http-cve crowdsecurity/whitelist-good-actors"
    # GID: "${GID-1000}"
      PGID:
      PUID:
      CUSTOM_HOSTNAME: CrowdSecDMZ
      DISABLE_LOCAL_API: "false" # True Only after successfully registering and validating remote agent below.
    volumes:
      - /mnt/services/docker/WebProxyDMZ/CrowdSec/data:/var/lib/crowdsec/data
      - /mnt/services/docker/WebProxyDMZ/CrowdSec:/etc/crowdsec
      - /mnt/services/docker/WebProxyDMZ/Traefik/logs:/var/log/traefik:ro
      - /mnt/services/docker/HomeAssistant:/var/log/homeassistant:ro    
    labels:
      <<: *labels
      net.unraid.docker.icon: "https://raw.githubusercontent.com/ibracorp/app-logos/main/crowdsec/crowdsec.png"

## CrowdSec - Redis ###########################################

  redis-cs:
    image: redis:alpine
    container_name: ProxyDMZ-CrowdSec-Redis
    <<: *config
    command: [ "sh", "-c", "exec redis-server --requirepass $REDIS_PASSWORD" ]  # redis-cli -a "password" --stat # select 1 # dbsize
    networks:
      wp-netredis:
    volumes:
      - /mnt/services/docker/WebProxyDMZ/Redis:/data
    environment:
      - TZ
    labels:
      <<: *labels
      net.unraid.docker.icon: "https://raw.githubusercontent.com/A75G/docker-templates/master/templates/icons/redis.png"


# Networks ####################################################

networks:
  eth2:
    name: eth2
    external: true
  eth1:
    name: eth1
    external: true
# wp-netsocketproxy:
#   internal: true
  wp-netredis:
    internal: true
# wp-netmodsecurity:
#   internal: true

# Secrets #####################################################

secrets:
  # Traefik - CF_API_EMAIL
  CF_API_EMAIL:
    file: $SECRETSDIR/CF_API_EMAIL
  # Traefik - CF_API_EMAIL
  CF_DNS_API_TOKEN:
    file: $SECRETSDIR/CF_DNS_API_TOKEN

I had the same issue. Services seem to start in whatever random order unraid decides and this can break things.

I wrote this script to address this:

https://gist.github.com/mtongnz/f3ca13c7ff8472e54bbb7ff8cec265e1

I use the down_up action and run at array start and now it seems that everything works as it should. Male sure you edit line 5 of the script and list all the compose manager projects you'd like to restart - I doubt your project names match mine

As a side note, I also disable all the auto start sliders on the docker page for any compose projects or the containers created by them.

primeval_god · March 8

10 hours ago, mtongnz said:

Services seem to start in whatever random order unraid decides and this can break things.

What services exactly are you referring to?

mtongnz · March 9

10 hours ago, primeval_god said:

What services exactly are you referring to?

Sorry... Should've said docker containers. If they're on auto start, unraid starts them in whatever order it feels like

JonathanM · March 9

11 minutes ago, mtongnz said:

If they're on auto start, unraid starts them in whatever order it feels like

Then something is wrong, it normally starts them from top to bottom, and you can insert a delay after each one if you wish. To change the order you click the green padlock to unlock sorting, to set delays you toggle advanced view. The delay for a container is AFTER that container is started, not before, so there is no way to delay the start of the first container.

Kilrah · March 9

7 hours ago, mtongnz said:

Sorry... Should've said docker containers. If they're on auto start, unraid starts them in whatever order it feels like

Set autostart on the compose stack, not on the containers it creates.

L0rdRaiden · March 9

2 hours ago, Kilrah said:

Set autostart on the compose stack, not on the containers it creates.

that is exactly the problem, if you use the autostart built in docker compose manager, with some stacks it won't work correctly, don't ask me why.

I'm going to try the script and report back

primeval_god · March 9

For the compose plugin there is no way currently to control the order in which stacks start, but that shouldnt matter as you should not have dependencies between stacks. As for containers within a stack I believe compose files have syntax to control which containers depend on others to start.

An issue some people have is with stacks that depend on external networks. Specifically the docker networks that unRAID creates change on every bootup thus containers must be recreated (rather than restarted) on boot. There is an option in the plugin settings to recreate containers on startup, but my recommendation is to just use custom docker networks and ignore the ones unRAID creates.

hrv231 · March 9

Hello,

I'm still having a problem with the plugin, when the docker compose has a mix of bridge networks and macvlan networks.

Docker compose:

services:
  qbit:
    image: linuxserver/qbittorrent:latest
    restart: unless-stopped
    environment:
      - UMASK=000
      - PUID=99
      - PGID=100
      - TZ=America/Los_Angeles
      - WEBUI_PORT=8090
      - TCP_PORT_6881=6881
      - UDP_PORT_6881=6881
    ports:
      - 6881:6881
      - 6881:6881/udp
      - 8090:8090
    volumes:
      - /mnt/user/appdata/qbittorrent:/config:rw,slave
      - /mnt/user/qb-completed:/completed:rw,slave
      - /mnt/user/qb-incomplete:/incomplete:rw,slave
    networks:
      eth0:
        ipv4_address: 10.10.30.90


  jackett:
    image: linuxserver/jackett
   //reduced info
    network_mode: bridge

  radarr:
    image: linuxserver/radarr
    //reduced info
    network_mode: bridge
    depends_on:
      - qbit
      - jackett

  sonarr:
    image: linuxserver/sonarr
    //reduced info
    network_mode: bridge
    depends_on:
      - qbit
      - jackett


networks:
  eth0:
    external: true
    name: eth0

This is what the GUI shows:

As you can see, the docker that is supposed to have the real ip, is not showing that IP, and for some reason unknown to me, it shows the same IP as one of the dockers from the same "docker compose"

Is someone else seeing the same error?

Fraddles · March 9

Not sure if it covers your use case, but for Home Assistant, to expose a single fixed IP and have a private network for containers to communicate, I did it like this;

version: '3.9'
###
networks:
  hass:
  bond0:
    external: true

###
services:
  homeassistant:
    container_name: Home-Assistant
    image: ghcr.io/home-assistant/home-assistant:stable
    networks:
      bond0:
        ipv4_address: 192.168.xx.xx
      hass:
    volumes:
      - /mnt/docker/appdata/HASS/Home-Assistant:/config
    restart: always
##
  whisper:
    container_name: Whisper
    image: rhasspy/wyoming-whisper
    networks:
      - hass
    volumes:
      - /mnt/docker/appdata/HASS/Whisper:/data
    command: --model small-int8 --language en --beam-size 5
    restart: always
##
  piper:
    container_name: Piper
    image: rhasspy/wyoming-piper
    networks:
      - hass
    volumes:
      - /mnt/docker/appdata/HASS/Piper:/data
    command: --voice en-us-ryan-medium
    restart: always

All three containers can communicate via the `hass` network, and the HA container has a fixed IP that is exposed via macvlan.

Although not in this example (and not yet migrated to Unraid), my InfluxDB stack uses a similar setup, with InfluxDB itself on a static macvlan IP, but the Adminer container exposed via port mapping on the host IP. Just add a `ports:` section as usual...

Cheers.

MichelPigassou · March 10

Hey everyone.

If my compose file references an existing network, in Unraid's UI the network is referenced by its ID instead of its name.

First screenshot is the UI (in the list of containers), second screenshot is the output of `docker network ls`.

Content of my docker-compose (top-level):

```

networks:
default:
name: lsio
external: true

```

Looks like a bug, because it's all working. Not sure where to report it, and maybe I'd like to get a confirmation first that is indeed a bug.

apandey · March 11

10 hours ago, MichelPigassou said:

Looks like a bug, because it's all working. Not sure where to report it, and maybe I'd like to get a confirmation first that is indeed a bug.

One other thing you should know is that unraid recreates these networks on reboot, so the ID will change and break the compose managed containers, unless you have the “recreate containers on startup” setting on. If not, you will need to compose down/up after each reboot

Edited March 11 by apandey

primeval_god · March 11

20 hours ago, MichelPigassou said:

Looks like a bug, because it's all working. Not sure where to report it, and maybe I'd like to get a confirmation first that is indeed a bug.

Its a known issue with the way dockerman shows networks.

Kilrah · March 11

On 3/9/2024 at 11:28 AM, L0rdRaiden said:

if you use the autostart built in docker compose manager, with some stacks it won't work correctly, don't ask me why.

If there are dependencies within a stack then they should be set with depends_on.

Kees Fluitman · March 13

Is it possible to auto update stacks in compose.manager?

primeval_god · March 13

1 hour ago, Kees Fluitman said:

Is it possible to auto update stacks in compose.manager?

No. I am not a fan of autoupdates for anything, but especially not container stacks. If you are looking to auto update containers i would suggest looking into watchtower.

Pulteney · March 15

On 3/10/2024 at 7:18 PM, MichelPigassou said:

Hey everyone.

If my compose file references an existing network, in Unraid's UI the network is referenced by its ID instead of its name.

First screenshot is the UI (in the list of containers), second screenshot is the output of `docker network ls`.

Content of my docker-compose (top-level):

```

networks:
default:
name: lsio
external: true

```

Looks like a bug, because it's all working. Not sure where to report it, and maybe I'd like to get a confirmation first that is indeed a bug.

Having the same problem. Reverse proxy seems to work as I can connect to the container with the docker hostname, but it sucks not being able to see which containers use which networks in the Docker Webui.

Is there any solution to this?

Kees Fluitman · March 20

On 3/13/2024 at 7:49 PM, primeval_god said:

No. I am not a fan of autoupdates for anything, but especially not container stacks. If you are looking to auto update containers i would suggest looking into watchtower.

i agree. for most stacks. but sometimes i have single app stacks in the compose manager, which I would like to auto update. (they are less critical). But ill manage just doing it manually weekly i guess.

blueink · March 29

Sorry! Removed my question as this isn’t the right place and I don’t want to clutter up the thread. Thank you for this awesome plugin!

Edited March 29 by blueink
Didn’t want to clutter up the thread

primeval_god · March 29

2 hours ago, blueink said:

I appreciate any insight anyone can provide!

This thread is not really the right place to ask about issues with specific compose stacks, unless the problem is with the compose manager plugin rather than the stack itself. Having lots of people post whole compose files in this thread makes it harder for others to find info about the plugin itself.

I am not really an expert on compose files in general and i dont use that specific application so i dont know how much help I can be. On general observation is that you should remove the version: "3.8" at the top of the file. Compose no longer requires it and actually recommends against calling out a specific compose version in compose files to better support mixing and matching syntax from various version of the compose spec.

Edited March 29 by primeval_god

quack7017 · March 31

On 3/10/2024 at 7:15 AM, hrv231 said:
Hello,

I'm still having a problem with the plugin, when the docker compose has a mix of bridge networks and macvlan networks.

Docker compose:
services:
  qbit:
    image: linuxserver/qbittorrent:latest
    restart: unless-stopped
    environment:
      - UMASK=000
      - PUID=99
      - PGID=100
      - TZ=America/Los_Angeles
      - WEBUI_PORT=8090
      - TCP_PORT_6881=6881
      - UDP_PORT_6881=6881
    ports:
      - 6881:6881
      - 6881:6881/udp
      - 8090:8090
    volumes:
      - /mnt/user/appdata/qbittorrent:/config:rw,slave
      - /mnt/user/qb-completed:/completed:rw,slave
      - /mnt/user/qb-incomplete:/incomplete:rw,slave
    networks:
      eth0:
        ipv4_address: 10.10.30.90


  jackett:
    image: linuxserver/jackett
   //reduced info
    network_mode: bridge

  radarr:
    image: linuxserver/radarr
    //reduced info
    network_mode: bridge
    depends_on:
      - qbit
      - jackett

  sonarr:
    image: linuxserver/sonarr
    //reduced info
    network_mode: bridge
    depends_on:
      - qbit
      - jackett


networks:
  eth0:
    external: true
    name: eth0
This is what the GUI shows:

As you can see, the docker that is supposed to have the real ip, is not showing that IP, and for some reason unknown to me, it shows the same IP as one of the dockers from the same "docker compose"

Is someone else seeing the same error?

Hi there, I have the same issue.

mtongnz · April 6

On 3/21/2024 at 7:37 AM, Kees Fluitman said:

i agree. for most stacks. but sometimes i have single app stacks in the compose manager, which I would like to auto update. (they are less critical). But ill manage just doing it manually weekly i guess.

If you use the User Scripts plugin, I'd recommend my script here:

https://gist.github.com/mtongnz/f3ca13c7ff8472e54bbb7ff8cec265e1

It lets you interact with this Compose Manager plugin via script and also run commands on multiple compose projects in a single command.

Change line 5 (argumentDefault) and you can then schedule the command.

I use it scheduled to "down_up" all my projects on array start - so it runs a down then an up on each stack. This ensures that all my dependencies start in the correct order as I was having issues when I left unRaid to autostart everything.

mtongnz · April 6

On 3/31/2024 at 7:39 PM, quack7017 said:

Hi there, I have the same issue.

I have a feeling this is related to various other network display issues for non-dockerman containers (such as the correct network name not showing). This just got merged:

https://github.com/unraid/webgui/pull/1616#issuecomment-2040820998

It shows multiple networks and the associated IPs.

Ports seem to not correctly show which are internal and which are exposed for compose stacks at present - I'm currently working on a fix.

image.png.cad00897c138d06e1ca198881bbf4939.png

[Plugin] Docker Compose Manager

Recommended Posts

Link to comment

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

primeval_god

mtongnz

sjtuross

Posted Images

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation