The Ops Community ⚙️

Alex
Alex

Posted on

loki && vector

#devops #loki #logs #sre

loki && vector

during our past (experiments)[https://community.ops.io/la3mmchen/loki-grafana-1i9d] with loki we created events in loki via curl. this created just a homeopathic amount of events so lets now bring in a logshipper that can talk to loki.

Looking around we can see at (Loki clients)[https://grafana.com/docs/loki/latest/clients/] a curated list from Grafana. The most prominent solution is (Promtail)[https://grafana.com/docs/loki/latest/clients/promtail/] a logshipper thats part of the Loki project.

vector.dev

At work we use another logshipper because of its versatility in handling inputs and outputs: (vector)[https://vector.dev/], an agent solution from data dog.

so lets try to enrich our existing compose-stack with a vector instance that speaks with loki. for this we first need a configuration for loki and some directory to store content in.

$ mkdir -p vector/logs
$ cat <<EOF >>vector/main.yaml
---
data_dir: /tmp
timezone: local

sources:
  samplelogs:
    type: file
    include:
      - "/logs/*.log"
    exclude: []
    ignore_older_secs: 999999

sinks:
  loki:
    type: loki
    inputs:
      - samplelogs
    endpoint: "http://loki:3100"
    encoding:
      codec: json
    labels:
      forwarder: vector
EOF
Enter fullscreen mode Exit fullscreen mode

Now grep some example logs and move them to ./vector/logs/*log. If you don't mind the content of the data sent to Loki just do:

$ docker info > vector/logs/docker.log
Enter fullscreen mode Exit fullscreen mode

enhance compose stack

enhance the existing stack from the (previous)[https://community.ops.io/la3mmchen/loki-grafana-1i9d] article with the following block and start your docker-compose stack.

(...)
  vector:
    image: "timberio/vector:0.21.X-alpine"
    depends_on:
      - loki
    entrypoint: ["/usr/local/bin/vector", "-c", "/config/main.yaml"]
    volumes:
      - ./vector/main.yaml:/config/main.yaml
      - ./vector/logs/:/logs
Enter fullscreen mode Exit fullscreen mode

You should now see some entry's if you browse logs, lets check fast via logcli:

$ docker-compose run logcli series --analyze-labels '{}'
Creating loki_logcli_run ... done
http://loki:3100/loki/api/v1/series?end=1654024447473483400&match=%7B%7D&start=1654020847473483400
Total Streams:  1
Unique Labels:  1

Label Name  Unique Values  Found In Streams
forwarder   1
Enter fullscreen mode Exit fullscreen mode

this tells us we've got one stream going into loki. this is because we've attached one label to the logs we've send to grafana loki (forwarder) so therefore loki just saves on stream for us.
Lets change this by attaching the filename as label to the streams; change the vector config to in the labels section to:

    labels:
      forwarder: vector
      file: "{{ file }}"
Enter fullscreen mode Exit fullscreen mode

This tells vector to attach another label with the value of the parsed file to the log stream. After you've restarted vector you can send another example file, e.g. docker --help > vector/logs/docker-help.log and we should now see more within our stats:

$ docker-compose run logcli series --analyze-labels '{}'
Creating loki_logcli_run ... done
http://loki:3100/loki/api/v1/series?end=1654027368307655600&match=%7B%7D&start=1654023768307655600
Total Streams:  2
Unique Labels:  2

Label Name  Unique Values  Found In Streams
forwarder   1              2
file        1              1
Enter fullscreen mode Exit fullscreen mode

Feel free to check the available streams in Grafana, it should look somehow like this:

Screenshot from Grafana

follow up

in the next article we will have a look at LogQl that enables you so browse the store information.

appendix 

---
version: "3.9"

services:
  loki:
    image: "grafana/loki:2.5.0"
    ports:
      - "3100:3100"
  logcli:
    image: "grafana/logcli:2.5.0-amd64"
    environment:
      - "LOKI_ADDR=http://loki:3100"
  grafana:
    image: "grafana/grafana:8.5.3"
    depends_on:
      - loki
    ports:
      - "3000:3000"
    volumes:
      - ./grafana-provision:/etc/grafana/provisioning
  vector:
    image: "timberio/vector:0.21.X-alpine"
    depends_on:
      - loki
    entrypoint: ["/usr/local/bin/vector", "-c", "/config/main.yaml"]
    volumes:
      - ./vector/main.yaml:/config/main.yaml
      - ./vector/logs/:/logs
Enter fullscreen mode Exit fullscreen mode

Oldest comments (0)