Hack Tools

Loki v1.1 releases: horizontally-scalable, highly-available, multi-tenant log aggregation system

By root

Posted on December 6, 2019

Loki: like Prometheus, but for logs.

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Compared to other log aggregation systems, Loki:

does not do full-text indexing on logs. By storing compressed, unstructured logs and only indexing metadata, Loki is simpler to operate and cheaper to run.
indexes and groups log streams using the same labels you’re already using with Prometheus, enabling you to seamlessly switch between metrics and logs using the same labels that you’re already using with Prometheus.
is an especially good fit for storing Kubernetes Pod logs. Metadata such as Pod labels is automatically scraped and indexed.
has native support in Grafana (needs Grafana v6.0).

A Loki-based logging stack consists of 3 components:

promtail is the agent, responsible for gathering logs and sending them to Loki.
loki is the main server, responsible for storing logs and processing queries.
Grafana for querying and displaying the logs.

Loki is like Prometheus, but for logs: we prefer a multidimensional label-based approach to indexing and want a single-binary, easy to operate system with no dependencies. Loki differs from Prometheus by focussing on logs instead of metrics and delivering logs via push, instead of pull.

Changelog v1.1

Broken version info in startup log message:
1095 pstibrany: Makefile changes to allow easy builds with or without vendoring. Also fixes version bug for both cases.
The hashing algorithm used to calculate the hash for a stream was creating hash collisions in some instances.
Please Note this is just one part of the fix and is only in Promtail, the second part for Loki can be tracked in PR1247 which didn’t quite make the cut for 1.1.0 and will be in 1.2.0:
1254 pstibrany: pkg/promtail/client: Handle fingerprint hash collisions
Thank you @putrasattvika for finding and fixing an important bug where logs were some logs were missed in a query shortly after a flush!
1299 putrasattvika: storage: fix missing logs with batched chunk iterator
Thank you @danieldabate for helping to again improve our API to be more Prometheus compatible:
1355 danieldabate: HTTP API: Support duration and float formats for step parameter
LogQL will support duration formats that are not typically handled by Go like [1d] or [1w]
1357 cyriltovena: Supports same duration format in LogQL as Prometheus