Debugging applications in a distributed system like kubernetes or cloud environments might be hard without a tool at hand that collect logs and let you browse them.

There are quite a few players in this game - e.g. Elasticsearch, Splunk, or DataDog.

Let me introduce one possible option to host a logfile collector because i found no other article that covers it.

Loki

It is fast. and can run with less overhead. Bonus point: it integrates very well in you're probably already existing central dashboarding solution Grafana.

Introduction

"just like prometheus. but for logs."

Consider log events as a stream like prometheus metrics. If you're familiar with prometheus you know the following notation. This represents one metric (up) with two different label values that represents the state of two different applications. Prometheus save timestamps and values for those two streams and let you query the results, e.g. via Grafana.

up{app_name="login-api"}
up{app_name="logout-api"}

Loki also thinks of incoming data as streams and distinguishes them with labels. In the following example you see one example stream with some labels. i removed some labels for brevity.

{"file":"/var/log/pods/ingress_ingress-nginx-controller-5fc96d49ff-986gd_ab5b1165-cbe7-437d-a1a1-8d32473929cc/controller/0.log",(...) request":{"latency":"0.012 s","protocol":"HTTP/1.1","referer":"","remoteIp":"<...>","remoteUser":"","requestMethod":"GET","requestPlain":"GET /ping HTTP/1.1","requestSize":"205","requestTime":"0.015","requestUrl":"<..>","responseSize":"14","status":"200","userAgent":"Blackbox Exporter/0.20.0"},"source_type":"kubernetes_logs","stream":"stdout"}

For incoming events Loki normalizes the volatile parts of the fields and store the occurring timestamp whenever an event matching this stream is entering the system.

This makes it easy to store lot of similar event that for example are produced in kubernetes clusters. Also it does not need lot of disk space for gathering logs.

Why not Elasticsearch?

The still state-of-the-art tooling for collection logs is the elastic stack.

Elasticsearch as search tool sends every single event into the underlying Lucene indices. Without any optimization this makes a beautiful search experience for the user because they can execute full-text searches on the collected events.

This might lead to increased resource usage, though.

Upcoming

I'll compose more articles on this topic.

Following up: how to setup a local Loki instance with docker-compose to be able to test, a selection of log shipper that works with Loki.