The Ops Community ⚙️: Alex

loki && logql

Alex — Thu, 02 Jun 2022 05:05:01 +0000

loki && logql

for the last part of my mini series about grafana loki we will now have a look on the query language that is used to power searches.

setup some real logs

first of all we need some real data and not mocked data as we've used in the previous articles.
for this we add another container to our stack: a nginx webserver that we use to generate some logs.

enhance the stack from the (previous)[https://community.ops.io/la3mmchen/loki-vector-21cf] article. note the change volume for the vector container.

(...)
  vector:
    image: "timberio/vector:0.21.X-alpine"
    depends_on:
      - loki
    entrypoint: ["/usr/local/bin/vector", "-c", "/config/main.yaml"]
    volumes:
      - ./vector/main.yaml:/config/main.yaml
      - ./nginx/logs/:/logs
  web:
    image: "nginx"
    ports:
      - 8080:80
    volumes:
      - ./nginx/logs/access.log:/var/log/nginx/access.log

Before starting the stack create the subdirectory and an empty file with:

$ mkdir -p nginx/logs
$ touch nginx/logs/access.log

With this setup the nginx write its access log to a local folder (nginx/logs/) that is mounted into the vector container to be consumed.
vector should now send every line to loki.

create some requests

lets create one hundred random requests and let vector send them to loki:

$ for i in {1..100}; do curl localhost:8080/$i; done

Nginx will serve a 404 Not Found but this doesn't matter as we've got something in our access log.

browse the logs

Right after creating the request you should see them in Loki. For this we can use logcli but you also can do the queries in the browser.

We first lookup what labels are there and what possible values for the label "file" are.

$ docker-compose run logcli labels
Creating loki_logcli_run ... done
http://loki:3100/loki/api/v1/labels?end=1654114422211392500&start=1654110822211392500
__name__
file
forwarder
$ docker-compose run logcli labels file
Creating loki_logcli_run ... done
http://loki:3100/loki/api/v1/label/file/values?end=1654114440666947800&start=1654110840666947800
/logs/access.log

Afterwards we want to see the log stream for the access log. We do this by executing a query and searching log streams that contain the label file with the value /logs/access.log. In our case this is just one log stream, but in reality there can be multiple log streams.

$ docker-compose run logcli query '{file="/logs/access.log"}'
Creating loki_logcli_run ... done
(...)
{"file":"/logs/access.log","host":"7fdf7d95debe","message":"172.23.0.1 - - [01/Jun/2022:20:12:01 +0000] \"GET /100 HTTP/1.1\" 404 153 \"-\" \"curl/7.79.1\" \"-\"","source_type":"file"}
(...)

You will see some of the requests you've done in the output of the logcli command.

lets search a certain request. We do this by sending the log stream to a filter operator |= that matches equality to a search string.

$ docker-compose run logcli query '{file="/logs/access.log"} |= "/44"'
(...){"file":"/logs/access.log","host":"7fdf7d95debe","message":"172.23.0.1 - - [01/Jun/2022:20:11:59 +0000] \"GET /44 HTTP/1.1\" 404 153 \"-\" \"curl/7.79.1\" \"-\"","source_type":"file"}
(...)

We can also apply regular expressions and combine multiple filter. In this theoretical case we first select all log events in the stream that has a regex match on `/4.*" and the filter out the event(s) that matches "/41".

bash $ docker-compose run logcli query '{file=~"/logs/acccess.*"} |~ "/4.*" |= "/41"' (..) Common labels: {file="/logs/access.log", forwarder="vector"} 2022-06-01T20:11:59Z {} {"file":"/logs/access.log","host":"7fdf7d95debe","message":"172.23.0.1 - - [01/Jun/2022:20:11:59 +0000] \"GET /41 HTTP/1.1\" 404 153 \"-\" \"curl/7.79.1\" \"-\"","source_type":"file"} (..)

And "like prometheus" we can also do regex matches on the label filtering if you've seen in the above example: file=~"/logs/acccess.*

Thats basically a rough introduction into LogQl. Read more at the (official documentation)[https://grafana.com/docs/loki/latest/logql/] and start playing around with it.

loki && vector

Alex — Wed, 01 Jun 2022 03:55:11 +0000

loki && vector

during our past (experiments)[https://community.ops.io/la3mmchen/loki-grafana-1i9d] with loki we created events in loki via curl. this created just a homeopathic amount of events so lets now bring in a logshipper that can talk to loki.

Looking around we can see at (Loki clients)[https://grafana.com/docs/loki/latest/clients/] a curated list from Grafana. The most prominent solution is (Promtail)[https://grafana.com/docs/loki/latest/clients/promtail/] a logshipper thats part of the Loki project.

vector.dev

At work we use another logshipper because of its versatility in handling inputs and outputs: (vector)[https://vector.dev/], an agent solution from data dog.

so lets try to enrich our existing compose-stack with a vector instance that speaks with loki. for this we first need a configuration for loki and some directory to store content in.

$ mkdir -p vector/logs
$ cat <<EOF >>vector/main.yaml
---
data_dir: /tmp
timezone: local

sources:
  samplelogs:
    type: file
    include:
      - "/logs/*.log"
    exclude: []
    ignore_older_secs: 999999

sinks:
  loki:
    type: loki
    inputs:
      - samplelogs
    endpoint: "http://loki:3100"
    encoding:
      codec: json
    labels:
      forwarder: vector
EOF

Now grep some example logs and move them to ./vector/logs/*log. If you don't mind the content of the data sent to Loki just do:

$ docker info > vector/logs/docker.log

enhance compose stack

enhance the existing stack from the (previous)[https://community.ops.io/la3mmchen/loki-grafana-1i9d] article with the following block and start your docker-compose stack.

(...)
  vector:
    image: "timberio/vector:0.21.X-alpine"
    depends_on:
      - loki
    entrypoint: ["/usr/local/bin/vector", "-c", "/config/main.yaml"]
    volumes:
      - ./vector/main.yaml:/config/main.yaml
      - ./vector/logs/:/logs

You should now see some entry's if you browse logs, lets check fast via logcli:

$ docker-compose run logcli series --analyze-labels '{}'
Creating loki_logcli_run ... done
http://loki:3100/loki/api/v1/series?end=1654024447473483400&match=%7B%7D&start=1654020847473483400
Total Streams:  1
Unique Labels:  1

Label Name  Unique Values  Found In Streams
forwarder   1

this tells us we've got one stream going into loki. this is because we've attached one label to the logs we've send to grafana loki (forwarder) so therefore loki just saves on stream for us.
Lets change this by attaching the filename as label to the streams; change the vector config to in the labels section to:

    labels:
      forwarder: vector
      file: "{{ file }}"

This tells vector to attach another label with the value of the parsed file to the log stream. After you've restarted vector you can send another example file, e.g. docker --help > vector/logs/docker-help.log and we should now see more within our stats:

$ docker-compose run logcli series --analyze-labels '{}'
Creating loki_logcli_run ... done
http://loki:3100/loki/api/v1/series?end=1654027368307655600&match=%7B%7D&start=1654023768307655600
Total Streams:  2
Unique Labels:  2

Label Name  Unique Values  Found In Streams
forwarder   1              2
file        1              1

Feel free to check the available streams in Grafana, it should look somehow like this:

follow up

in the next article we will have a look at LogQl that enables you so browse the store information.

appendix

---
version: "3.9"

services:
  loki:
    image: "grafana/loki:2.5.0"
    ports:
      - "3100:3100"
  logcli:
    image: "grafana/logcli:2.5.0-amd64"
    environment:
      - "LOKI_ADDR=http://loki:3100"
  grafana:
    image: "grafana/grafana:8.5.3"
    depends_on:
      - loki
    ports:
      - "3000:3000"
    volumes:
      - ./grafana-provision:/etc/grafana/provisioning
  vector:
    image: "timberio/vector:0.21.X-alpine"
    depends_on:
      - loki
    entrypoint: ["/usr/local/bin/vector", "-c", "/config/main.yaml"]
    volumes:
      - ./vector/main.yaml:/config/main.yaml
      - ./vector/logs/:/logs

loki && grafana

Alex — Tue, 31 May 2022 07:45:51 +0000

loki && grafana

in my previously written articles i explained what loki is and how we can setup loki via docker-compose.

finally: some GUI

this time we add a grafana instance to our docker-compsose stack to talk to our loki instance. and finally we get some fancy user interface.

lets add a grafana instance to our docker-compose stack and provide a loki datasource during startup.

First provide a datasource definition in the folder you've located the docker-compose file:

$ mkdir -p grafana-provision/datasources
$ cat <<EOF >>grafana-provision/datasources/auto.yaml
apiVersion: 1

datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    jsonData:
      maxLines: 1000
EOF

enhance the compose stack with an additional instance:

(...)
  grafana:
    image: "grafana/grafana:8.5.3"
    depends_on:
      - loki
    ports:
      - "3000:3000"
    volumes:
      - ./grafana-provision:/etc/grafana/provisioning

access loki

upon starting the compose stack you should now have a working datasource in the configuration/datasources panel and should be able to find a datasource "loki" in the dropdown in the explore mode. Navigate your browser to http://localhost:3000/datasources/

if you've freshly started the compose stack the explore feature is not quite useful as there is nothing to be explored.

add some data

remembering the commands from the previous article we create some data via curl:

$ curl -S -H "Content-Type: application/json" -XPOST -s http://localhost:3100/loki/api/v1/push --data-raw '{"streams": [{ "stream": { "app": "app1" }, "values": [ [ "1653937064000000000", "random log line" ] ] }]}'
$ curl -S -H "Content-Type: application/json" -XPOST -s http://localhost:3100/loki/api/v1/push --data-raw '{"streams": [{ "stream": { "app": "app2" }, "values": [ [ "1653937064000000000", "other random log line" ] ] }]}'
$ curl -S -H "Content-Type: application/json" -XPOST -s http://localhost:3100/loki/api/v1/push --data-raw '{"streams": [{ "stream": { "app": "app3" }, "values": [ [ "1653937064000000000", "another random log line" ] ] }]}'

hint: you might have to create a new timestamp via TIMESTAMP="date +%s000000000" echo ${TIMESTAMP}

browse logs

if you followed along you should now see the following selectable labels in your browser:

select one label value and hit "show logs" and you should see the previously injected log events.

closing thoughts

thats it basically. in the upcoming post we select a log shipper that speaks with loki. after that we have a look on LogQl, the query language that you've now already used a little bit.

complete compose-stack

---
version: "3.9"

services:
  loki:
    image: "grafana/loki:2.5.0"
    ports:
      - "3100:3100"
  logcli:
    image: "grafana/logcli:2.5.0-amd64"
    environment:
      - "LOKI_ADDR=http://loki:3100"
  grafana:
    image: "grafana/grafana:8.5.3"
    depends_on:
      - loki
    ports:
      - "3000:3000"
    volumes:
      - ./grafana-provision:/etc/grafana/provisioning

loki && curl && logcli

Alex — Mon, 30 May 2022 04:59:28 +0000

With https://community.ops.io/la3mmchen/grafana-loki-28b9 I've shared some words on Loki as an introduction.

Lets follow up with some practical stuff: How can we run Loki an our local notebook with docker-compose?

Loki

As Grafana provides a good enough container images we just use the one from docker hub to get a local instance running.

---
version: "3.9"

services:
  loki:
    image: "grafana/loki:2.5.0"
    ports:
      - "3100:3100"

As Loki follows the unix philosophy "Do One Thing and Do It Well" it's just a backend to send logs to and to query logs from.
Plain speaking: there is no graphical user interface in the Loki server.

We interact with the Loki instance with curl using the http api:

$  curl http://localhost:3100/ready  
ready
$ curl http://localhost:3100/loki/api/v1/labels
{"status":"success","data":["__name__"]}

As the second request shows there are not many labels in the system yet.

Lets change this.

interact with curl

Loki offers a http api to send metrics. The data format is described at http push api, we use the following curl to create a new event stream:

$ curl -S -H "Content-Type: application/json" -XPOST -s http://localhost:3100/loki/api/v1/push --data-raw '{"streams": [{ "stream": { "app": "app1" }, "values": [ [ "1653855518000000000", "random log line" ] ] }]}'
$ curl http://localhost:3100/loki/api/v1/labels
{"status":"success","data":["__name__","app"]}

As you can see in the second curl there is now a new label named "app". We can explore possible values for this label with the following curl.

$ curl http://localhost:3100/loki/api/v1/label/app/values
{"status":"success","data":["app1"]}

And finaly see the stream with this label:

$ curl -G -Ss  http://localhost:3100/loki/api/v1/query_range --data-urlencode 'query={app="app1"}' | jq .
{
  "status": "success",
(...)
        "values": [
          [
            "1653855518000000000",
            "random log line"
          ]
        ]
      }
(...)

(output shorted)

logcli

A more covenient way to interact with Loki from the command line is using logcli

$ LOKI_ADDR=http://localhost:3100 logcli labels
http://localhost:3100/loki/api/v1/labels?end=1653856566774536000&start=1653852966774536000
__name__
app
$ LOKI_ADDR=http://localhost:3100 logcli query '{app=~".+"}'
http://localhost:3100/loki/api/v1/query_range?direction=BACKWARD&end=1653856568404563000&limit=30&query=%7Bapp%3D~%22.%2B%22%7D&start=1653852968404563000
Common labels: {app="app1"}
2022-05-29T22:18:38+02:00 {} random log line
http://localhost:3100/loki/api/v1/query_range?direction=BACKWARD&end=1653855518000000001&limit=30&query=%7Bapp%3D~%22.%2B%22%7D&start=1653852968404563000
Common labels: {app="app1"}

in case you do not want to install logcli locally just add another container to our docker-compose setup from the beginning of this article:

(..)
  logcli:
    image: "grafana/logcli:2.5.0-amd64"
    environment:
      - "LOKI_ADDR=http://loki:3100"

with this container you can just do:

$ docker-compose run logcli labels
Creating loki_logcli_run ... done
(..)
__name__
app

following up

in the next post we will add a grafana instance to our docker-compose that will enable us to click something ;-)

references

code reference

---
version: "3.9"

services:
  loki:
    image: "grafana/loki:2.5.0"
    ports:
      - "3100:3100"
  logcli:
    image: "grafana/logcli:2.5.0-amd64"
    environment:
      - "LOKI_ADDR=http://loki:3100"

Grafana Loki

Alex — Sun, 29 May 2022 17:55:33 +0000

Debugging applications in a distributed system like kubernetes or cloud environments might be hard without a tool at hand that collect logs and let you browse them.

There are quite a few players in this game - e.g. Elasticsearch, Splunk, or DataDog.

Let me introduce one possible option to host a logfile collector because i found no other article that covers it.

Loki

It is fast. and can run with less overhead. Bonus point: it integrates very well in you're probably already existing central dashboarding solution Grafana.

Introduction

"just like prometheus. but for logs."

Consider log events as a stream like prometheus metrics. If you're familiar with prometheus you know the following notation. This represents one metric (up) with two different label values that represents the state of two different applications. Prometheus save timestamps and values for those two streams and let you query the results, e.g. via Grafana.

up{app_name="login-api"}
up{app_name="logout-api"}

Loki also thinks of incoming data as streams and distinguishes them with labels. In the following example you see one example stream with some labels. i removed some labels for brevity.

{"file":"/var/log/pods/ingress_ingress-nginx-controller-5fc96d49ff-986gd_ab5b1165-cbe7-437d-a1a1-8d32473929cc/controller/0.log",(...) request":{"latency":"0.012 s","protocol":"HTTP/1.1","referer":"","remoteIp":"<...>","remoteUser":"","requestMethod":"GET","requestPlain":"GET /ping HTTP/1.1","requestSize":"205","requestTime":"0.015","requestUrl":"<..>","responseSize":"14","status":"200","userAgent":"Blackbox Exporter/0.20.0"},"source_type":"kubernetes_logs","stream":"stdout"}

For incoming events Loki normalizes the volatile parts of the fields and store the occurring timestamp whenever an event matching this stream is entering the system.

This makes it easy to store lot of similar event that for example are produced in kubernetes clusters. Also it does not need lot of disk space for gathering logs.

Why not Elasticsearch?

The still state-of-the-art tooling for collection logs is the elastic stack.

Elasticsearch as search tool sends every single event into the underlying Lucene indices. Without any optimization this makes a beautiful search experience for the user because they can execute full-text searches on the collected events.

This might lead to increased resource usage, though.

Upcoming

I'll compose more articles on this topic.

Following up: how to setup a local Loki instance with docker-compose to be able to test, a selection of log shipper that works with Loki.

The Ops Community ⚙️: Alex

loki && logql

loki && logql

setup some real logs

create some requests

browse the logs

loki && vector

loki && vector

vector.dev

enhance compose stack

follow up

appendix

loki && grafana

loki && grafana

finally: some GUI

access loki

add some data

browse logs

closing thoughts

complete compose-stack

loki && curl && logcli

Loki

interact with curl

logcli

following up

references

links

code reference

Grafana Loki

Loki

Introduction

Why not Elasticsearch?

Upcoming