loki && logql

for the last part of my mini series about grafana loki we will now have a look on the query language that is used to power searches.

setup some real logs

first of all we need some real data and not mocked data as we've used in the previous articles.
for this we add another container to our stack: a nginx webserver that we use to generate some logs.

enhance the stack from the (previous)[https://community.ops.io/la3mmchen/loki-vector-21cf] article. note the change volume for the vector container.

(...)
  vector:
    image: "timberio/vector:0.21.X-alpine"
    depends_on:
      - loki
    entrypoint: ["/usr/local/bin/vector", "-c", "/config/main.yaml"]
    volumes:
      - ./vector/main.yaml:/config/main.yaml
      - ./nginx/logs/:/logs
  web:
    image: "nginx"
    ports:
      - 8080:80
    volumes:
      - ./nginx/logs/access.log:/var/log/nginx/access.log

Before starting the stack create the subdirectory and an empty file with:

$ mkdir -p nginx/logs
$ touch nginx/logs/access.log

With this setup the nginx write its access log to a local folder (nginx/logs/) that is mounted into the vector container to be consumed.
vector should now send every line to loki.

create some requests

lets create one hundred random requests and let vector send them to loki:

$ for i in {1..100}; do curl localhost:8080/$i; done

Nginx will serve a 404 Not Found but this doesn't matter as we've got something in our access log.

browse the logs

Right after creating the request you should see them in Loki. For this we can use logcli but you also can do the queries in the browser.

We first lookup what labels are there and what possible values for the label "file" are.

$ docker-compose run logcli labels
Creating loki_logcli_run ... done
http://loki:3100/loki/api/v1/labels?end=1654114422211392500&start=1654110822211392500
__name__
file
forwarder
$ docker-compose run logcli labels file
Creating loki_logcli_run ... done
http://loki:3100/loki/api/v1/label/file/values?end=1654114440666947800&start=1654110840666947800
/logs/access.log

Afterwards we want to see the log stream for the access log. We do this by executing a query and searching log streams that contain the label file with the value /logs/access.log. In our case this is just one log stream, but in reality there can be multiple log streams.

$ docker-compose run logcli query '{file="/logs/access.log"}'
Creating loki_logcli_run ... done
(...)
{"file":"/logs/access.log","host":"7fdf7d95debe","message":"172.23.0.1 - - [01/Jun/2022:20:12:01 +0000] \"GET /100 HTTP/1.1\" 404 153 \"-\" \"curl/7.79.1\" \"-\"","source_type":"file"}
(...)

You will see some of the requests you've done in the output of the logcli command.

lets search a certain request. We do this by sending the log stream to a filter operator |= that matches equality to a search string.

$ docker-compose run logcli query '{file="/logs/access.log"} |= "/44"'
(...){"file":"/logs/access.log","host":"7fdf7d95debe","message":"172.23.0.1 - - [01/Jun/2022:20:11:59 +0000] \"GET /44 HTTP/1.1\" 404 153 \"-\" \"curl/7.79.1\" \"-\"","source_type":"file"}
(...)

We can also apply regular expressions and combine multiple filter. In this theoretical case we first select all log events in the stream that has a regex match on `/4.*" and the filter out the event(s) that matches "/41".

bash
$ docker-compose run logcli query '{file=~"/logs/acccess.*"} |~ "/4.*" |= "/41"'
(..)
Common labels: {file="/logs/access.log", forwarder="vector"}
2022-06-01T20:11:59Z {} {"file":"/logs/access.log","host":"7fdf7d95debe","message":"172.23.0.1 - - [01/Jun/2022:20:11:59 +0000] \"GET /41 HTTP/1.1\" 404 153 \"-\" \"curl/7.79.1\" \"-\"","source_type":"file"}
(..)


And "like prometheus" we can also do regex matches on the label filtering if you've seen in the above example: file=~"/logs/acccess.*

Thats basically a rough introduction into LogQl. Read more at the (official documentation)[https://grafana.com/docs/loki/latest/logql/] and start playing around with it.