Trying to add loki driver support to docker

[Uleepera]

Greetings, 

 

I've been working on adding loki, promtail, and grafana to my server to aggregate logging across the server as well as multiple other devices around my homelab.  The driver was added successfully but now I'm trying to locate daemon.json to add the code referenced at https://grafana.com/docs/loki/latest/clients/docker-driver/configuration/ but its not in that location and I ran a system wide find and still couldn't locate it. 

 

Could someone please point me towards it or is a different file being used?

Edited May 4, 2023 by Uleepera

[Ford Prefect]

just a hunch here...if this fiile is not available, why not simply create it?

As unraid is running in RAM only, you could make it persistant by adding a command in your GO file on the Stick -> create the file, save on the boot stick, then copy it into /etc/docker/ when booting by adding the copy command to the GO file.

[Uleepera]

45 minutes ago, Ford Prefect said:

just a hunch here...if this fiile is not available, why not simply create it?

As unraid is running in RAM only, you could make it persistant by adding a command in your GO file on the Stick -> create the file, save on the boot stick, then copy it into /etc/docker/ when booting by adding the copy command to the GO file.

I read a post elsewhere of someone looking to modify parts of that file who said unRAIDs docker behaves abnormally if you add the file to the normal directory. 

Edited May 4, 2023 by Uleepera

[Ford Prefect]

Well, as I said, I don't know either.

Assess the risk involved....what could go wrong?

• I'd disable auto start of docker containers (avoiding that defective changes persist in their appdata folder)
• Install an extra Docker for testing
• deactivate docker deamon in settings
• create the file (in RAM)
• restart docker deamon
• check if file is still there
• assess if this is working and what side effects are there (as you read somewhere)
  as you change the way doocker deamon does logging, the log entries in each container are possibly different, corrupt or similar?
• if everthing runs fine, edit the go fille - if there are problems, reboot and everything should be as normal as it was before

...use a test system, not the unraid production box to kepp the WAF up

 

 

Edited May 4, 2023 by Ford Prefect

[L0rdRaiden]

@Uleepera please check this, I think you don't need to edit the file

 

And if you don't mind report back I plan to do the same soon.

 

[Uleepera]

On 6/1/2023 at 4:06 PM, L0rdRaiden said:

@Uleepera please check this, I think you don't need to edit the file

 

And if you don't mind report back I plan to do the same soon.

 

Have not had a chance to look at this yet.  I will, my temp fix was to setup a VM and run everything thru a slim Ubuntu install which has worked but isn't optimal. 

[L0rdRaiden]

It's possible with the docker version with

      - /var/run/docker.sock:/var/run/docker.sock

 

 

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /positions/positions.yaml

clients:
  - url: http://10.10.40.251:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /host/log/*log

- job_name: docker
  # use docker.sock to filter containers
  docker_sd_configs:
    - host: unix:///var/run/docker.sock
      refresh_interval: 15s
      #filters:
      #  - name: label
      #    values: ["logging=promtail"]    # use container name to create a loki label
  relabel_configs:
    - source_labels: ['__meta_docker_container_name']
      regex: '/(.*)'
      target_label: 'container'
    - source_labels: ['__meta_docker_container_log_stream']
      target_label: 'logstream'
    - source_labels: ['__meta_docker_container_label_logging_jobname']
      target_label: 'job'

 

  promtail:
    # run as root, update to rootless mode later
    user: "0:0"
    container_name: Mon-Promtail
    image: grafana/promtail:main
    command: -config.file=/etc/promtail/docker-config.yaml
    depends_on:
      - loki
    restart: unless-stopped
    networks:
      mon-netsocketproxy:
      mon-netgrafana:
      br1:
        ipv4_address: 10.10.40.252
    dns: 10.10.50.5
    ports:
      - 9800:9800
    volumes:
      # logs for linux host only
      - /var/log:/host/log
      #- /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /mnt/user/Docker/Monitoring/Promtail/promtail-config.yaml:/etc/promtail/docker-config.yaml
      - /mnt/user/Docker/Monitoring/Promtail/positions:/positions
      - /var/run/docker.sock:/var/run/docker.sock
    labels:
      - "com.centurylinklabs.watchtower.enable=true"

 

[phyzical]

Thanks for that @L0rdRaiden.

 

worked like a charm, no plugin needed

[manofoz]

On 6/6/2023 at 10:27 AM, L0rdRaiden said:

It's possible with the docker version with

      - /var/run/docker.sock:/var/run/docker.sock

 

 

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /positions/positions.yaml

clients:
  - url: http://10.10.40.251:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /host/log/*log

- job_name: docker
  # use docker.sock to filter containers
  docker_sd_configs:
    - host: unix:///var/run/docker.sock
      refresh_interval: 15s
      #filters:
      #  - name: label
      #    values: ["logging=promtail"]    # use container name to create a loki label
  relabel_configs:
    - source_labels: ['__meta_docker_container_name']
      regex: '/(.*)'
      target_label: 'container'
    - source_labels: ['__meta_docker_container_log_stream']
      target_label: 'logstream'
    - source_labels: ['__meta_docker_container_label_logging_jobname']
      target_label: 'job'

 

  promtail:
    # run as root, update to rootless mode later
    user: "0:0"
    container_name: Mon-Promtail
    image: grafana/promtail:main
    command: -config.file=/etc/promtail/docker-config.yaml
    depends_on:
      - loki
    restart: unless-stopped
    networks:
      mon-netsocketproxy:
      mon-netgrafana:
      br1:
        ipv4_address: 10.10.40.252
    dns: 10.10.50.5
    ports:
      - 9800:9800
    volumes:
      # logs for linux host only
      - /var/log:/host/log
      #- /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /mnt/user/Docker/Monitoring/Promtail/promtail-config.yaml:/etc/promtail/docker-config.yaml
      - /mnt/user/Docker/Monitoring/Promtail/positions:/positions
      - /var/run/docker.sock:/var/run/docker.sock
    labels:
      - "com.centurylinklabs.watchtower.enable=true"

 

 

This looks really interesting but I don't have quite enough experience to interprite how to apply this. I'd love to have my logs set up with Loki and Promtail so I'm going to give it a shot. I think it would be valuable experince. 

 

Is the first file a promtail-config.yml file? If so the seems easy enough to adapt. The second looks like an entry for a docker compose file. I tried that with the following that also includes the Loki dependency but Promtail immedialty closes both the console and log windows when I try to open them:

 

version: "3"
services:
  loki:
    image: grafana/loki:main
    ports:
      - 3100:3100
    command: -config.file=/etc/loki/local-config.yaml
  promtail:
    image: grafana/promtail:main
    ports:
      - 9800:9800
    command: -config.file=/etc/promtail/docker-config.yaml
    depends_on:
      - loki
    restart: unless-stopped
    volumes:
      # logs for linux host only
      - /var/log:/host/log
      #- /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /mnt/user/appdata/promtail-config.yaml:/etc/promtail/docker-config.yaml
      - /mnt/user/appdata/promtail/positions:/positions
      - /var/run/docker.sock:/var/run/docker.sock
    labels:
      - "com.centurylinklabs.watchtower.enable=true"

 

I am also missing a local-config.yaml. I am going to try one from here: https://grafana.com/docs/loki/latest/configure/examples/.

 

Thanks! 

 

Edit: I fixed the path error for the promtail config and used the local loki config from the website above and the promtail window doesn't dissapear immediatly anymore. Getting logs for both. 

 

pomtail complaining about: 
 

Quote

level=warn ts=2023-09-26T20:55:42.868024887Z caller=client.go:419 component=client host=192.168.0.200:3100 msg="error sending batch, will retry" status=429 tenant= error="server returned HTTP status 429 Too Many Requests (429): Ingestion rate limit exceeded for user fake (limit: 4194304 bytes/sec) while attempting to ingest '8423' lines totaling '1048557' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased"

 

Seems like a good sign, I will try the root user.

 

Loki has a simular complaint:

 

Quote

level=warn ts=2023-09-26T20:56:35.531113139Z caller=grpc_logging.go:60 method=/logproto.Pusher/Push duration=10.891684ms msg=gRPC err="rpc error: code = Code(429) desc = entry with timestamp 2023-09-26 10:26:42.627270316 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{container=\"binhex-sabnzbd\", logstream=\"stdout\"}' totaling 139B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased',\nentry with timestamp 2023-09-26 10:26:42.627448876 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{container=\"binhex-sabnzbd\", logstream=\"stdout\"}' totaling 58B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased',\nentry with timestamp 2023-09-26 10:26:42.627456959 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{container=\"binhex-sabnzbd\", logstream=\"stdout\"}' totaling 139B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased',\nentry with timestamp 2023-09-26 10:26:42.627593069 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{container=\"binhex-sabnzbd\", logstream=\"stdout\"}' totaling 58B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased',\nentry with timestamp 2023-09-26 10:26:42.627597038 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{container=\"binhex-sabnzbd\", logstream=\"stdout\"}' totaling 139B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased',\nentry with timestamp 2023-09-26 10:26:42.62774611 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{container=\"binhex-sabnzbd\", logstream=\"stdout\"}' totaling 58B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased',\nentry with timestamp 2023-09-26 10:26:42.627752147 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{container=\"binhex-sabnzbd\", logstream=\"stdout\"}' totaling 139B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased',\nentry with timestamp 2023-09-26 10:26:42.627887143 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{container=\"binhex-sabnzbd\", logstream=\"stdout\"}' totaling 58B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased',\nentry with timestamp 2023-09-26 10:26:42.627890474 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{container=\"binhex-sabnzbd\", logstream=\"stdout\"}' totaling 138B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased',\nentry with timestamp 2023-09-26 10:26:42.628039619 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{container=\"binhex-sabnzbd\", logstream=\"stdout\"}' totaling 58B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased',\nuser 'fake', total ignored: 1697 out of 13599 for stream: {container=\"binhex-sabnzbd\", logstream=\"stdout\"}"

 

Edited September 26, 2023 by manofoz

[manofoz]

Well somehow I stumbled through that and got logs to show up in Grafana! 

 

Looks like what ever is logging to STDOUT. I will need try and configure this to show logs that are on the file system. 

Thanks!