Grafana Loki has a microservices-based architecture and is designed to run as a horizontally scalable, distributed system.
The system has multiple components that can run separately and in parallel.
Grafana Loki's design compiles the code for all components into a single binary or Docker image.
The `-target` command-line flag controls which component(s) that binary will behave as.
All data, both in memory and in long-term storage, may be partitioned by a
tenant ID, pulled from the `X-Scope-OrgID` HTTP header in the request when Grafana Loki
is running in multi-tenant mode. When Loki is **not** in multi-tenant mode, the
header is ignored and the tenant ID is set to "fake", which will appear in the
index and in stored chunks.
To get started easily, run Grafana Loki in "single binary" mode with all components running simultaneously in one process, or in "simple scalable deployment" mode, which groups components into read, write, and backend parts.
Grafana Loki is designed to easily redeploy a cluster under a different mode as your needs change, with no configuration changes or minimal configuration changes.
For more information, refer to [Deployment modes]({{< relref "./deployment-modes" >}}) and [Components]({{< relref "./components" >}}).
Loki stores all data in a single object storage backend, such as Amazon Simple Storage Service (S3), Google Cloud Storage (GCS), Azure Blob Storage, among others.
This mode uses an adapter called **index shipper** (or short **shipper**) to store index (TSDB or BoltDB) files the same way we store chunk files in object storage.
This mode of operation became generally available with Loki 2.0 and is fast, cost-effective, and simple. It is where all current and future development lies.
Prior to 2.0, Loki had different storage backends for indexes and chunks. For more information, refer to [Legacy storage]({{< relref "../operations/storage/legacy-storage" >}}).
### Data format
Grafana Loki has two main file types: **index** and **chunks**.
- The [**index**](#index-format) is a table of contents of where to find logs for a specific set of labels.
- The [**chunk**](#chunk-format) is a container for log entries for a specific set of labels.
The diagram above shows the high-level overview of the data that is stored in the chunk and data that is stored in the index.
#### Index format
## Chunk Format
There are two index formats that are currently supported as single store with index shipper:
Time Series Database (or short TSDB) is an [index format](https://github.com/prometheus/prometheus/blob/main/tsdb/docs/format/index.md) originally developed by the maintainers of [Prometheus](https://github.com/prometheus/prometheus) for time series (metric) data.
It is extensible and has many advantages over the deprecated BoltDB index.
New storage features in Loki are solely available when using TSDB.
`ts` is the Unix nanosecond timestamp of the logs, while len is the length in
`ts` is the Unix nanosecond timestamp of the logs, while `len` is the length in
bytes of the log entry.
Symbols store references to the actual strings containing label names and values in the
`structuredMetadata` section of the chunk.
## Storage
### Single Store
## Write path
Loki stores all data in a single object storage backend. This mode of operation became generally available with Loki 2.0 and is fast, cost-effective, and simple, not to mention where all current and future development lies. This mode uses an adapter called [`boltdb_shipper`]({{< relref "../operations/storage/boltdb-shipper" >}}) to store the `index` in object storage (the same way we store `chunks`).
On a high level, the write path in Loki works as follows:
### Deprecated: Multi-store
1. The distributor receives an HTTP POST request with streams and log lines.
1. The distributor hashes each stream contained in the request so it can determine the ingester instance to which it needs to be sent based on the information from the consistent hash ring.
1. The distributor sends each stream to the appropriate ingester and its replicas (based on the configured replication factor).
1. The ingester receives the stream with log lines and creates a chunk or appends to an existing chunk for the stream's data.
A chunk is unique per tenant and per label set.
1. The ingester acknowledges the write.
1. The distributor waits for a majority (quorum) of the ingesters to acknowledge their writes.
1. The distributor responds with a success (2xx status code) in case it received at least a quorum of acknowledged writes.
or with an error (4xx or 5xx status code) in case write operations failed.
The **chunk store** is Loki's long-term data store, designed to support
interactive querying and sustained writing without the need for background
maintenance tasks. It consists of:
Refer to [Components]({{< relref "./components" >}}) for a more detailed description of the components involved in the write path.
- An index for the chunks. This index can be backed by:
Unlike the other core components of Loki, the chunk store is not a separate
service, job, or process, but rather a library embedded in the two services
that need to access Loki data: the [ingester]({{< relref "../../get-started/components#ingester" >}}) and [querier]({{< relref "../../get-started/components#querier" >}}).
{{% /admonition %}}
The chunk store relies on a unified interface to the
"[NoSQL](https://en.wikipedia.org/wiki/NoSQL)" stores (DynamoDB, Bigtable, and
Cassandra) that can be used to back the chunk store index. This interface
assumes that the index is a collection of entries keyed by:
- A **hash key**. This is required for *all* reads and writes.
- A **range key**. This is required for writes and can be omitted for reads,
which can be queried by prefix or range.
The interface works somewhat differently across the supported databases:
- DynamoDB supports range and hash keys natively. Index entries are thus
modelled directly as DynamoDB entries, with the hash key as the distribution
key and the range as the DynamoDB range key.
- For Bigtable and Cassandra, index entries are modelled as individual column
values. The hash key becomes the row key and the range key becomes the column
key.
A set of schemas are used to map the matchers and label sets used on reads and
writes to the chunk store into appropriate operations on the index. Schemas have
been added as Loki has evolved, mainly in an attempt to better load balance
Christian Haudum
2 years ago
committed by