loki && vector
during our past (experiments)[https://community.ops.io/la3mmchen/loki-grafana-1i9d] with loki we created events in loki via curl. this created just a homeopathic amount of events so lets now bring in a logshipper that can talk to loki.
Looking around we can see at (Loki clients)[https://grafana.com/docs/loki/latest/clients/] a curated list from Grafana. The most prominent solution is (Promtail)[https://grafana.com/docs/loki/latest/clients/promtail/] a logshipper thats part of the Loki project.
vector.dev
At work we use another logshipper because of its versatility in handling inputs and outputs: (vector)[https://vector.dev/], an agent solution from data dog.
so lets try to enrich our existing compose-stack with a vector instance that speaks with loki. for this we first need a configuration for loki and some directory to store content in.
$ mkdir -p vector/logs
$ cat <<EOF >>vector/main.yaml
---
data_dir: /tmp
timezone: local
sources:
samplelogs:
type: file
include:
- "/logs/*.log"
exclude: []
ignore_older_secs: 999999
sinks:
loki:
type: loki
inputs:
- samplelogs
endpoint: "http://loki:3100"
encoding:
codec: json
labels:
forwarder: vector
EOF
Now grep some example logs and move them to ./vector/logs/*log. If you don't mind the content of the data sent to Loki just do:
$ docker info > vector/logs/docker.log
enhance compose stack
enhance the existing stack from the (previous)[https://community.ops.io/la3mmchen/loki-grafana-1i9d] article with the following block and start your docker-compose stack.
(...)
vector:
image: "timberio/vector:0.21.X-alpine"
depends_on:
- loki
entrypoint: ["/usr/local/bin/vector", "-c", "/config/main.yaml"]
volumes:
- ./vector/main.yaml:/config/main.yaml
- ./vector/logs/:/logs
You should now see some entry's if you browse logs, lets check fast via logcli:
$ docker-compose run logcli series --analyze-labels '{}'
Creating loki_logcli_run ... done
http://loki:3100/loki/api/v1/series?end=1654024447473483400&match=%7B%7D&start=1654020847473483400
Total Streams: 1
Unique Labels: 1
Label Name Unique Values Found In Streams
forwarder 1
this tells us we've got one stream going into loki. this is because we've attached one label to the logs we've send to grafana loki (forwarder) so therefore loki just saves on stream for us.
Lets change this by attaching the filename as label to the streams; change the vector config to in the labels section to:
labels:
forwarder: vector
file: "{{ file }}"
This tells vector to attach another label with the value of the parsed file to the log stream. After you've restarted vector you can send another example file, e.g. docker --help > vector/logs/docker-help.log and we should now see more within our stats:
$ docker-compose run logcli series --analyze-labels '{}'
Creating loki_logcli_run ... done
http://loki:3100/loki/api/v1/series?end=1654027368307655600&match=%7B%7D&start=1654023768307655600
Total Streams: 2
Unique Labels: 2
Label Name Unique Values Found In Streams
forwarder 1 2
file 1 1
Feel free to check the available streams in Grafana, it should look somehow like this:
follow up
in the next article we will have a look at LogQl that enables you so browse the store information.
appendix
---
version: "3.9"
services:
loki:
image: "grafana/loki:2.5.0"
ports:
- "3100:3100"
logcli:
image: "grafana/logcli:2.5.0-amd64"
environment:
- "LOKI_ADDR=http://loki:3100"
grafana:
image: "grafana/grafana:8.5.3"
depends_on:
- loki
ports:
- "3000:3000"
volumes:
- ./grafana-provision:/etc/grafana/provisioning
vector:
image: "timberio/vector:0.21.X-alpine"
depends_on:
- loki
entrypoint: ["/usr/local/bin/vector", "-c", "/config/main.yaml"]
volumes:
- ./vector/main.yaml:/config/main.yaml
- ./vector/logs/:/logs
Top comments (0)