diff --git a/docs/README.md b/docs/README.md
index 82307bac1d..866a449e37 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -1,42 +1,56 @@
+# Loki Documentation
+
Like Prometheus, but for logs!
-Grafana Loki is a set of components, that can be composed into a fully featured
+Grafana Loki is a set of components that can be composed into a fully featured
logging stack.
-It builds around the idea of treating a single log line as-is. This means that
-instead of full-text indexing them, related logs are grouped using the same
-labels as in Prometheus. This is much more efficient and scales better.
-
-## Components
-- **[Loki](loki/README.md)**: The main server component is called Loki. It is
- responsible for permanently storing the logs it is being shipped and it
- executes the LogQL
- queries from clients.
- Loki shares its high-level architecture with Cortex, a highly scalable
- Prometheus backend.
-- **[Promtail](promtail/README.md)**: To ship logs to a central place, an
- agent is required. Promtail
- is deployed to every node that should be monitored and sends the logs to Loki.
- It also does important task of pre-processing the log lines, including
- attaching labels to them for easier querying.
-- *Grafana*: The *Explore* feature of Grafana 6.0+ is the primary place of
- contact between a human and Loki. It is used for discovering and analyzing
- logs.
+Unlike other logging systems, Loki is built around the idea of only indexing
+metadata about your logs: labels (just like Prometheus labels). Log data itself
+is then compressed and stored in chunks in object stores such as S3 or GCS, or
+even locally on the filesystem. A small index and highly compressed chunks
+simplifies the operation and significantly lowers the cost of Loki.
-Alongside these main components, there are some other ones as well:
+## Table of Contents
-- **[LogCLI](logcli.md)**: A command line interface to query logs and labels
- from Loki
-- **[Canary](canary/README.md)**: An audit utility to analyze the log-capturing
- performance of Loki. Ingests data into Loki and immediately reads it back to
- check for latency and loss.
-- **[Docker
- Driver](https://github.com/grafana/loki/tree/master/cmd/docker-driver)**: A
- Docker [log
- driver](https://docs.docker.com/config/containers/logging/configure/) to ship
- logs captured by Docker directly to Loki, without the need of an agent.
-- **[Fluentd
- Plugin](https://github.com/grafana/loki/tree/master/fluentd/fluent-plugin-grafana-loki)**:
- An Fluentd [output plugin](https://docs.fluentd.org/output), to use Fluentd
- for shipping logs into Loki
+1. [Overview](overview/README.md)
+ 1. [Comparison to other Log Systems](overview/comparisons.md)
+2. [Installation](installation/README.md)
+ 1. [Installing with Tanka](installation/tanka.md)
+ 2. [Installing with Helm](installation/helm.md)
+ 3. [Installing Locally](installation/local.md)
+3. [Getting Started](getting-started/README.md)
+ 1. [Grafana](getting-started/grafana.md)
+ 2. [LogCLI](getting-started/logcli.md)
+ 4. [Troubleshooting](getting-started/troubleshooting.md)
+4. [Configuration](configuration/README.md)
+ 1. [Examples](configuration/examples.md)
+5. [Clients](clients/README.md)
+ 1. [Promtail](clients/promtail/README.md)
+ 1. [Installation](clients/promtail/installation.md)
+ 2. [Configuration](clients/promtail/configuration.md)
+ 3. [Scraping](clients/promtail/scraping.md)
+ 4. [Pipelines](clients/promtail/pipelines.md)
+ 5. [Troubleshooting](clients/promtail/troubleshooting.md)
+ 2. [Docker Driver](clients/docker-driver/README.md)
+ 1. [Configuration](clients/docker-driver/configuration.md)
+ 3. [Fluentd](clients/fluentd.md)
+6. [LogQL](logql.md)
+7. [Operations](operations/README.md)
+ 1. [Authentication](operations/authentication.md)
+ 2. [Observability](operations/observability.md)
+ 3. [Scalability](operations/scalability.md)
+ 4. [Storage](operations/storage/README.md)
+ 1. [Table Manager](operations/storage/table-manager.md)
+ 2. [Retention](operations/storage/retention.md)
+ 5. [Multi-tenancy](operations/multi-tenancy.md)
+ 6. [Loki Canary](operations/loki-canary.md)
+8. [HTTP API](api.md)
+9. [Architecture](architecture.md)
+10. [Community](community/README.md)
+ 1. [Governance](community/governance.md)
+ 2. [Getting in Touch](community/getting-in-touch.md)
+ 3. [Contributing to Loki](community/contributing.md)
+11. [Loki Maintainers Guide](./maintaining/README.md)
+ 1. [Releasing Loki](./maintaining/release.md)
diff --git a/docs/api.md b/docs/api.md
new file mode 100644
index 0000000000..c0e6193899
--- /dev/null
+++ b/docs/api.md
@@ -0,0 +1,639 @@
+# Loki's HTTP API
+
+Loki exposes an HTTP API for pushing, querying, and tailing log data.
+Note that [authenticating](operations/authentication.md) against the API is
+out of scope for Loki.
+
+The HTTP API includes the following endpoints:
+
+- [`GET /loki/api/v1/query`](#get-lokiapiv1query)
+- [`GET /loki/api/v1/query_range`](#get-lokiapiv1query_range)
+- [`GET /loki/api/v1/label`](#get-lokiapiv1label)
+- [`GET /loki/api/v1/label//values`](#get-lokiapiv1labelnamevalues)
+- [`GET /loki/api/v1/tail`](#get-lokiapiv1tail)
+- [`POST /loki/api/v1/push`](#post-lokiapiv1push)
+- [`GET /api/prom/tail`](#get-apipromtail)
+- [`GET /api/prom/query`](#get-apipromquery)
+- [`GET /ready`](#get-ready)
+- [`POST /flush`](#post-flush)
+- [`GET /metrics`](#get-metrics)
+
+## Microservices Mode
+
+When deploying Loki in microservices mode, the set of endpoints exposed by each
+component is different.
+
+These endpoints are exposed by all components:
+
+- [`GET /ready`](#get-ready)
+- [`GET /metrics`](#get-metrics)
+
+These endpoints are exposed by just the querier:
+
+- [`GET /loki/api/v1/query`](#get-lokiapiv1query)
+- [`GET /loki/api/v1/query_range`](#get-lokiapiv1query_range)
+- [`GET /loki/api/v1/label`](#get-lokiapiv1label)
+- [`GET /loki/api/v1/label//values`](#get-lokiapiv1labelnamevalues)
+- [`GET /loki/api/v1/tail`](#get-lokiapiv1tail)
+- [`GET /api/prom/tail`](#get-lokiapipromtail)
+- [`GET /api/prom/query`](#get-apipromquery)
+
+While these endpoints are exposed by just the distributor:
+
+- [`POST /loki/api/v1/push`](#post-lokiapiv1push)
+
+And these endpoints are exposed by just the ingester:
+
+- [`POST /flush`](#post-flush)
+
+The API endpoints starting with `/loki/` are [Prometheus API-compatible](https://prometheus.io/docs/prometheus/latest/querying/api/) and the result formats can be used interchangeably.
+
+[Example clients](#example-clients) can be found at the bottom of this document.
+
+## Matrix, Vector, And Streams
+
+Some Loki API endpoints return a result of a matrix, a vector, or a stream:
+
+- Matrix: a table of values where each row represents a different label set
+ and the columns are each sample value for that row over the queried time.
+ Matrix types are only returned when running a query that computes some value.
+
+- Instant Vector: denoted in the type as just `vector`, an Instant Vector
+ represents the latest value of a calculation for a given labelset. Instant
+ Vectors are only returned when doing a query against a single point in
+ time.
+
+- Stream: a Stream is a set of all values (logs) for a given label set over the
+ queried time range. Streams are the only type that will result in log lines
+ being returned.
+
+## `GET /loki/api/v1/query`
+
+`/loki/api/v1/query` allows for doing queries against a single point in time. The URL
+query parameters support the following values:
+
+- `query`: The [LogQL](./logql.md) query to perform
+- `limit`: The max number of entries to return
+- `time`: The evaluation time for the query as a nanosecond Unix epoch. Defaults to now.
+- `direction`: Determines the sort order of logs. Supported values are `forward` or `backward`. Defaults to `backward.`
+
+In microservices mode, `/loki/api/v1/query` is exposed by the querier.
+
+Response:
+
+```
+{
+ "status": "success",
+ "data": {
+ "resultType": "vector" | "streams",
+ "result": [] | []
+ }
+}
+```
+
+Where `` is:
+
+```
+{
+ "metric": {
+
+ },
+ "value": [
+ ,
+
+ ]
+}
+```
+
+And `` is:
+
+```
+{
+ "stream": {
+
+ },
+ "values": [
+ [
+ ,
+
+ ],
+ ...
+ ]
+}
+```
+
+### Examples
+
+```bash
+$ curl -G -s "http://localhost:3100/loki/api/v1/query" --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq
+{
+ "status": "success",
+ "data": {
+ "resultType": "vector",
+ "result": [
+ {
+ "metric": {},
+ "value": [
+ 1559848867745737,
+ "1267.1266666666666"
+ ]
+ },
+ {
+ "metric": {
+ "level": "warn"
+ },
+ "value": [
+ 1559848867745737,
+ "37.77166666666667"
+ ]
+ },
+ {
+ "metric": {
+ "level": "info"
+ },
+ "value": [
+ 1559848867745737,
+ "37.69"
+ ]
+ }
+ ]
+ }
+}
+```
+
+```bash
+$ curl -G -s "http://localhost:3100/loki/api/v1/query" --data-urlencode 'query={job="varlogs"}' | jq
+{
+ "status": "success",
+ "data": {
+ "resultType": "streams",
+ "result": [
+ {
+ "stream": {
+ "filename": "/var/log/myproject.log",
+ "job": "varlogs",
+ "level": "info"
+ },
+ "values": [
+ [
+ "1568234281726420425",
+ "foo"
+ ],
+ [
+ "1568234269716526880",
+ "bar"
+ ]
+ ]
+ }
+ ]
+ }
+}
+```
+
+## `GET /loki/api/v1/query_range`
+
+`/loki/api/v1/query_range` is used to do a query over a range of time and
+accepts the following query parameters in the URL:
+
+- `query`: The [LogQL](./logql.md) query to perform
+- `limit`: The max number of entries to return
+- `start`: The start time for the query as a nanosecond Unix epoch. Defaults to one hour ago.
+- `end`: The start time for the query as a nanosecond Unix epoch. Defaults to now.
+- `step`: Query resolution step width in seconds. Defaults to 1.
+- `direction`: Determines the sort order of logs. Supported values are `forward` or `backward`. Defaults to `backward.`
+
+Requests against this endpoint require Loki to query the index store in order to
+find log streams for particular labels. Because the index store is spread out by
+time, the time span covered by `start` and `end`, if large, may cause additional
+load against the index server and result in a slow query.
+
+In microservices mode, `/loki/api/v1/query_range` is exposed by the querier.
+
+Response:
+
+```
+{
+ "status": "success",
+ "data": {
+ "resultType": "matrix" | "streams",
+ "result": [] | []
+ }
+}
+```
+
+Where `` is:
+
+```
+{
+ "metric": {
+
+ },
+ "values": [
+ ,
+
+ ]
+}
+```
+
+And `` is:
+
+```
+{
+ "stream": {
+
+ },
+ "values": [
+ [
+ ,
+
+ ],
+ ...
+ ]
+}
+```
+
+### Examples
+
+```bash
+$ curl -G -s "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' --data-urlencode 'step=300' | jq
+{
+ "status": "success",
+ "data": {
+ "resultType": "matrix",
+ "result": [
+ {
+ "metric": {
+ "level": "info"
+ },
+ "values": [
+ [
+ 1559848958663735,
+ "137.95"
+ ],
+ [
+ 1559849258663735,
+ "467.115"
+ ],
+ [
+ 1559849558663735,
+ "658.8516666666667"
+ ]
+ ]
+ },
+ {
+ "metric": {
+ "level": "warn"
+ },
+ "values": [
+ [
+ 1559848958663735,
+ "137.27833333333334"
+ ],
+ [
+ 1559849258663735,
+ "467.69"
+ ],
+ [
+ 1559849558663735,
+ "660.6933333333334"
+ ]
+ ]
+ }
+ ]
+ }
+}
+```
+
+```bash
+$ curl -G -s "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={job="varlogs"}' | jq
+{
+ "status": "success",
+ "data": {
+ "resultType": "streams",
+ "result": [
+ {
+ "stream": {
+ "filename": "/var/log/myproject.log",
+ "job": "varlogs",
+ "level": "info"
+ },
+ "values": [
+ {
+ "1569266497240578000",
+ "foo"
+ },
+ {
+ "1569266492548155000",
+ "bar"
+ }
+ ]
+ }
+ ]
+ }
+}
+```
+
+## `GET /loki/api/v1/label`
+
+`/loki/api/v1/label` retrieves the list of known labels within a given time span. It
+accepts the following query parameters in the URL:
+
+- `start`: The start time for the query as a nanosecond Unix epoch. Defaults to 6 hours ago.
+- `end`: The start time for the query as a nanosecond Unix epoch. Defaults to now.
+
+In microservices mode, `/loki/api/v1/label` is exposed by the querier.
+
+Response:
+
+```
+{
+ "values": [
+ ,
+ ...
+ ]
+}
+```
+
+### Examples
+
+```bash
+$ curl -G -s "http://localhost:3100/loki/api/v1/label" | jq
+{
+ "values": [
+ "foo",
+ "bar",
+ "baz"
+ ]
+}
+```
+
+## `GET /loki/api/v1/label//values`
+
+`/loki/api/v1/label//values` retrieves the list of known values for a given
+label within a given time span. It accepts the following query parameters in
+the URL:
+
+- `start`: The start time for the query as a nanosecond Unix epoch. Defaults to 6 hours ago.
+- `end`: The start time for the query as a nanosecond Unix epoch. Defaults to now.
+
+In microservices mode, `/loki/api/v1/label//values` is exposed by the querier.
+
+Response:
+
+```
+{
+ "values": [
+ ,
+ ...
+ ]
+}
+```
+
+### Examples
+
+```bash
+$ curl -G -s "http://localhost:3100/loki/api/v1/label/foo/values" | jq
+{
+ "values": [
+ "cat",
+ "dog",
+ "axolotl"
+ ]
+}
+```
+
+## `GET /loki/api/v1/tail`
+
+`/loki/api/v1/tail` is a WebSocket endpoint that will stream log messages based on
+a query. It accepts the following query parameters in the URL:
+
+- `query`: The [LogQL](./logql.md) query to perform
+- `delay_for`: The number of seconds to delay retrieving logs to let slow
+ loggers catch up. Defaults to 0 and cannot be larger than 5.
+- `limit`: The max number of entries to return
+- `start`: The start time for the query as a nanosecond Unix epoch. Defaults to one hour ago.
+
+In microservices mode, `/loki/api/v1/tail` is exposed by the querier.
+
+Response (streamed):
+
+```
+{
+ "streams": [
+ {
+ "stream": {
+
+ },
+ "values": [
+ [
+ ,
+
+ ]
+ ]
+ }
+ ],
+ "dropped_entries": [
+ {
+ "labels": {
+
+ },
+ "timestamp": ""
+ }
+ ]
+}
+```
+
+## `POST /loki/api/v1/push`
+
+Alias (DEPRECATED): `POST /loki/api/v1/push`
+
+`/loki/api/v1/push` is the endpoint used to send log entries to Loki. The default
+behavior is for the POST body to be a snappy-compressed protobuf messsage:
+
+- [Protobuf definition](/pkg/logproto/logproto.proto)
+- [Go client library](/pkg/promtail/client/client.go)
+
+Alternatively, if the `Content-Type` header is set to `application/json`, a
+JSON post body can be sent in the following format:
+
+```
+{
+ "streams": [
+ {
+ "labels": "",
+ "entries": [
+ {
+ "ts": "",
+ "line": ""
+ }
+ ]
+ }
+ ]
+}
+```
+
+> **NOTE**: logs sent to Loki for every stream must be in timestamp-ascending
+> order, meaning each log line must be more recent than the one last received.
+> If logs do not follow this order, Loki will reject the log with an out of
+> order error.
+
+In microservices mode, `/loki/api/v1/push` is exposed by the distributor.
+
+### Examples
+
+```bash
+$ curl -H "Content-Type: application/json" -XPOST -s "https://localhost:3100/loki/api/v1/push" --data-raw \
+ '{"streams": [{ "labels": "{foo=\"bar\"}", "entries": [{ "ts": "2018-12-18T08:28:06.801064-04:00", "line": "fizzbuzz" }] }]}'
+```
+
+## `GET /api/prom/tail`
+
+> **DEPRECATED**: `/api/prom/tail` is deprecated. Use `/loki/api/v1/tail`
+> instead.
+
+`/api/prom/tail` is a WebSocket endpoint that will stream log messages based on
+a query. It accepts the following query parameters in the URL:
+
+- `query`: The [LogQL](./logql.md) query to perform
+- `delay_for`: The number of seconds to delay retrieving logs to let slow
+ loggers catch up. Defaults to 0 and cannot be larger than 5.
+- `limit`: The max number of entries to return
+- `start`: The start time for the query as a nanosecond Unix epoch. Defaults to one hour ago.
+
+In microservices mode, `/api/prom/tail` is exposed by the querier.
+
+Response (streamed):
+
+```json
+{
+ "streams": [
+ {
+ "labels": "",
+ "entries": [
+ {
+ "ts": "",
+ "line": ""
+ }
+ ]
+ }
+ ],
+ "dropped_entries": [
+ {
+ "Timestamp": "",
+ "Labels": ""
+ }
+ ]
+}
+```
+
+`dropped_entries` will be populated when the tailer could not keep up with the
+amount of traffic in Loki. When present, it indicates that the entries received
+in the streams is not the full amount of logs that are present in Loki. Note
+that the keys in `dropped_entries` will be sent as uppercase `Timestamp`
+and `Labels` instead of `labels` and `ts` like in the entries for the stream.
+
+As the response is streamed, the object defined by the response format above
+will be sent over the WebSocket multiple times.
+
+
+
+## `GET /api/prom/query`
+
+> **WARNING**: `/api/prom/query` is DEPRECATED; use `/loki/api/v1/query_range`
+> instead.
+
+`/api/prom/query` supports doing general queries. The URL query parameters
+support the following values:
+
+- `query`: The [LogQL](./logql.md) query to perform
+- `limit`: The max number of entries to return
+- `start`: The start time for the query as a nanosecond Unix epoch. Defaults to one hour ago.
+- `end`: The start time for the query as a nanosecond Unix epoch. Defaults to now.
+- `direction`: Determines the sort order of logs. Supported values are `forward` or `backward`. Defaults to `backward.`
+- `regexp`: a regex to filter the returned results
+
+In microservices mode, `/api/prom/query` is exposed by the querier.
+
+Note that the larger the time span between `start` and `end` will cause
+additional load on Loki and the index store, resulting in slower queries.
+
+Response:
+
+```
+{
+ "streams": [
+ {
+ "labels": "",
+ "entries": [
+ {
+ "ts": "",
+ "line": ""
+ },
+ ...
+ ],
+ },
+ ...
+ ]
+}
+```
+
+### Examples
+
+```bash
+$ curl -G -s "http://localhost:3100/api/prom/query" --data-urlencode '{foo="bar"}' | jq
+{
+ "streams": [
+ {
+ "labels": "{filename=\"/var/log/myproject.log\", job=\"varlogs\", level=\"info\"}",
+ "entries": [
+ {
+ "ts": "2019-06-06T19:25:41.972739Z",
+ "line": "foo"
+ },
+ {
+ "ts": "2019-06-06T19:25:41.972722Z",
+ "line": "bar"
+ }
+ ]
+ }
+ ]
+}
+```
+
+### Examples
+
+```bash
+$ curl -H "Content-Type: application/json" -XPOST -s "https://localhost:3100/loki/api/v1/push" --data-raw \
+ '{"streams": [{ "labels": "{foo=\"bar\"}", "entries": [{ "ts": "2018-12-18T08:28:06.801064-04:00", "line": "fizzbuzz" }] }]}'
+```
+
+## `GET /ready`
+
+`/ready` returns HTTP 200 when the Loki ingester is ready to accept traffic. If
+running Loki on Kubernetes, `/ready` can be used as a readiness probe.
+
+In microservices mode, the `/ready` endpoint is exposed by all components.
+
+## `POST /flush`
+
+`/flush` triggers a flush of all in-memory chunks held by the ingesters to the
+backing store. Mainly used for local testing.
+
+In microservices mode, the `/flush` endpoint is exposed by the ingester.
+
+## `GET /metrics`
+
+`/metrics` exposes Prometheus metrics. See
+[Observing Loki](operations/observability.md)
+for a list of exported metrics.
+
+In microservices mode, the `/metrics` endpoint is exposed by all components.
+
+## Example Clients
+
+Please note that the Loki API is not stable yet and breaking changes may occur
+when using or writing a third-party client.
+
+- [Promtail](https://github.com/grafana/loki/tree/master/pkg/promtail) (Official, Go)
+- [promtail-client](https://github.com/afiskon/promtail-client) (Go)
+- [push-to-loki.py](https://github.com/sleleko/devops-kb/blob/master/python/push-to-loki.py) (Python 3)
diff --git a/docs/architecture.md b/docs/architecture.md
new file mode 100644
index 0000000000..88c93b8790
--- /dev/null
+++ b/docs/architecture.md
@@ -0,0 +1,306 @@
+# Loki's Architecture
+
+This document will expand on the information detailed in the [Loki
+Overview](overview/README.md).
+
+## Multi Tenancy
+
+All data - both in memory and in long-term storage - is partitioned by a
+tenant ID, pulled from the `X-Scope-OrgID` HTTP header in the request when Loki
+is running in multi-tenant mode. When Loki is **not** in multi-tenant mode, the
+header is ignored and the tenant ID is set to "fake", which will appear in the
+index and in stored chunks.
+
+## Modes of Operation
+
+
+
+Loki has a set of components (defined below in [Components](#components)) which
+are internally referred to as modules. Each component spawns a gRPC server for
+internal traffic and an HTTP/1 server for external API requests. All components
+come with an HTTP/1 server, but most only expose readiness, health, and metrics
+endpoints.
+
+Which component Loki runs is determined by either the `-target` flag at the
+command line or the `target: ` section in Loki's config file. When the
+value of `target` is `all`, Loki will run all of its components in a single
+process. This is referred to as "single process", "single binary", or monolithic
+mode. Monolithic mode is the default deployment of Loki when Loki is installed
+using Helm.
+
+When `target` is _not_ set to `all` (i.e., it is set to `querier`, `ingester`,
+or `distributor`), then Loki is said to be running in "horizontally scalable",
+or microservices, mode.
+
+Each component of Loki, such as the ingesters and distributors, communicate with
+one another over gRPC using the gRPC listen port defined in the Loki config.
+When running components in monolithic mode, this is still true: each component,
+although running in the same process, will connect to each other over the local
+network for inter-component communication.
+
+Single process mode is ideally suited for local development, small workloads,
+and for evaluation purposes. Monolithic mode can be scaled with multiple
+processes with the following limitations:
+
+1. Local index and local storage cannot currently be used when running
+ monolithic mode with more than one replica, as each replica must be able to
+ access the same storage backend, and local storage is not safe for concurrent
+ access.
+2. Individual components cannot be scaled independently, so it is not possible
+ to have more read components than write components.
+
+## Components
+
+### Distributor
+
+The **distributor** service is responsible for handling incoming streams by
+clients. It's the first stop in the write path for log data. Once the
+distributor receives a set of streams, each stream is validated for correctness
+and to ensure that it is within the configured tenant (or global) limits. Valid
+chunks are then split into batches and sent to multiple [ingesters](#ingester)
+in parallel.
+
+#### Hashing
+
+Distributors use consistent hashing in conjunction with a configurable
+replication factor to determine which instances of the ingester service should
+receive a given stream.
+
+A stream is a set of logs associated to a tenant and a unique labelset. The
+stream is hashed using both the tenant ID and the labelset and then the hash is
+used to find the ingesters to send the stream to.
+
+A hash ring stored in [Consul](https://www.consul.io) is used to achieve
+consistent hashing; all [ingesters](#ingester) register themselves into the hash
+ring with a set of tokens they own. Each token is a random unsigned 32-bit
+number. Along with a set of tokens, ingesters register their state into the
+hash ring. The state JOINING, and ACTIVE may all receive write requests, while
+ACTIVE and LEAVING ingesters may receive read requests. When doing a hash
+lookup, distributors only use tokens for ingesters who are in the appropriate
+state for the request.
+
+To do the hash lookup, distributors find the smallest appropriate token whose
+value is larger than the hash of the stream. When the replication factor is
+larger than 1, the next subsequent tokens (clockwise in the ring) that belong to
+different ingesters will also be included in the result.
+
+The effect of this hash set up is that each token that an ingester owns is
+responsible for a range of hashes. If there are three tokens with values 0, 25,
+and 50, then a hash of 3 would be given to the ingester that owns the token 25;
+the ingester owning token 25 is responsible for the hash range of 1-25.
+
+#### Quorum consistency
+
+Since all distributors share access to the same hash ring, write requests can be
+sent to any distributor.
+
+To ensure consistent query results, Loki uses
+[Dynamo-style](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf)
+quorum consistency on reads and writes. This means that the distributor will wait
+for a positive response of at least one half plus one of the ingesters to send
+the sample to before responding to the client that initiated the send.
+
+### Ingester
+
+The **ingester** service is responsible for writing log data to long-term
+storage backends (DynamoDB, S3, Cassandra, etc.) on the write path and returning
+log data for in-memory queries on the read path.
+
+Ingesters contain a _lifecycler_ which manages the lifecycle of an ingester in
+the hash ring. Each ingester has a state of either `PENDING`, `JOINING`,
+`ACTIVE`, `LEAVING`, or `UNHEALTHY`:
+
+1. `PENDING` is an Ingester's state when it is waiting for a handoff from
+ another ingester that is `LEAVING`.
+
+2. `JOINING` is an Ingester's state when it is currently inserting its tokens
+ into the ring and initializing itself. It may receive write requests for
+ tokens it owns.
+
+3. `ACTIVE` is an Ingester's state when it is fully initialized. It may receive
+ both write and read requests for tokens it owns.
+
+4. `LEAVING` is an Ingester's state when it is shutting down. It may receive
+ read requests for data it still has in memory.
+
+5. `UNHEALTHY` is an Ingester's state when it has failed to heartbeat to
+ Consul. `UNHEALHTY` is set by the distributor when it periodically checks the ring.
+
+Each log stream that an ingester receives is built up into a set of many
+"chunks" in memory and flushed to the backing storage backend at a configurable
+interval.
+
+Chunks are compressed and marked as read-only when:
+
+1. The current chunk has reached capacity (a configurable value).
+2. Too much time has passed without the current chunk being updated
+3. A flush occurs.
+
+Whenever a chunk is compressed and marked as read-only, a writable chunk takes
+its place.
+
+If an ingester process crashes or exits abruptly, all the data that has not yet
+been flushed will be lost. Loki is usually configured to replicate multiple
+replicas (usually 3) of each log to mitigate this risk.
+
+When a flush occurs to a persistent storage provider, the chunk is hashed based
+on its tenant, labels, and contents. This means that multiple ingesters with the
+same copy of data will not write the same data to the backing store twice, but
+if any write failed to one of the replicas, multiple differing chunk objects
+will be created in the backing store. See [Querier](#querier) for how data is
+deduplicated.
+
+#### Handoff
+
+By default, when an ingester is shutting down and tries to leave the hash ring,
+it will wait to see if a new ingester tries to enter before flushing and will
+try to initiate a handoff. The handoff will transfer all of the tokens and
+in-memory chunks owned by the leaving ingester to the new ingester.
+
+Before joining the hash ring, ingesters will wait in `PENDING` state for a
+handoff to occur. After a configurable timeout, ingesters in the `PENDING` state
+that have not received a transfer will join the ring normally, inserting a new
+set of tokens.
+
+This process is used to avoid flushing all chunks when shutting down, which is a
+slow process.
+
+### Querier
+
+The **querier** service handles queries using the [LogQL](./logql.md) query
+language, fetching logs both from the ingesters and long-term storage.
+
+Queriers query all ingesters for in-memory data before falling back to
+running the same query against the backend store. Because of the replication
+factor, it is possible that the querier may receive duplicate data. To resolve
+this, the querier internally **deduplicates** data that has the same nanosecond
+timestamp, label set, and log message.
+
+## Chunk Format
+
+```
+ -------------------------------------------------------------------
+ | | |
+ | MagicNumber(4b) | version(1b) |
+ | | |
+ -------------------------------------------------------------------
+ | block-1 bytes | checksum (4b) |
+ -------------------------------------------------------------------
+ | block-2 bytes | checksum (4b) |
+ -------------------------------------------------------------------
+ | block-n bytes | checksum (4b) |
+ -------------------------------------------------------------------
+ | #blocks (uvarint) |
+ -------------------------------------------------------------------
+ | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
+ -------------------------------------------------------------------
+ | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
+ -------------------------------------------------------------------
+ | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
+ -------------------------------------------------------------------
+ | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
+ -------------------------------------------------------------------
+ | checksum(from #blocks) |
+ -------------------------------------------------------------------
+ | #blocks section byte offset |
+ -------------------------------------------------------------------
+```
+
+`mint` and `maxt` describe the minimum and maximum Unix nanosecond timestamp,
+respectively.
+
+### Block Format
+
+A block is comprised of a series of entries, each of which is an individual log
+line.
+
+Note that the bytes of a block are stored compressed using Gzip. The following
+is their form when uncompressed:
+
+```
+ -------------------------------------------------------------------
+ | ts (varint) | len (uvarint) | log-1 bytes |
+ -------------------------------------------------------------------
+ | ts (varint) | len (uvarint) | log-2 bytes |
+ -------------------------------------------------------------------
+ | ts (varint) | len (uvarint) | log-3 bytes |
+ -------------------------------------------------------------------
+ | ts (varint) | len (uvarint) | log-n bytes |
+ -------------------------------------------------------------------
+```
+
+`ts` is the Unix nanosecond timestamp of the logs, while len is the length in
+bytes of the log entry.
+
+## Chunk Store
+
+The **chunk store** is Loki's long-term data store, designed to support
+interactive querying and sustained writing without the need for background
+maintenance tasks. It consists of:
+
+* An index for the chunks. This index can be backed by:
+ * [Amazon DynamoDB](https://aws.amazon.com/dynamodb)
+ * [Google Bigtable](https://cloud.google.com/bigtable)
+ * [Apache Cassandra](https://cassandra.apache.org)
+* A key-value (KV) store for the chunk data itself, which can be:
+ * [Amazon DynamoDB](https://aws.amazon.com/dynamodb)
+ * [Google Bigtable](https://cloud.google.com/bigtable)
+ * [Apache Cassandra](https://cassandra.apache.org)
+ * [Amazon S3](https://aws.amazon.com/s3)
+ * [Google Cloud Storage](https://cloud.google.com/storage/)
+
+> Unlike the other core components of Loki, the chunk store is not a separate
+> service, job, or process, but rather a library embedded in the two services
+> that need to access Loki data: the [ingester](#ingester) and [querier](#querier).
+
+The chunk store relies on a unified interface to the
+"[NoSQL](https://en.wikipedia.org/wiki/NoSQL)" stores (DynamoDB, Bigtable, and
+Cassandra) that can be used to back the chunk store index. This interface
+assumes that the index is a collection of entries keyed by:
+
+* A **hash key**. This is required for *all* reads and writes.
+* A **range key**. This is required for writes and can be omitted for reads,
+which can be queried by prefix or range.
+
+The interface works somewhat differently across the supported databases:
+
+* DynamoDB supports range and hash keys natively. Index entries are thus
+ modelled directly as DynamoDB entries, with the hash key as the distribution
+ key and the range as the DynamoDB range key.
+* For Bigtable and Cassandra, index entries are modelled as individual column
+ values. The hash key becomes the row key and the range key becomes the column
+ key.
+
+A set of schemas are used to map the matchers and label sets used on reads and
+writes to the chunk store into appropriate operations on the index. Schemas have
+been added as Loki has evolved, mainly in an attempt to better load balance
+writes and improve query performance.
+
+> The current schema recommendation is the **v10 schema**.
+
+## Read Path
+
+To summarize, the read path works as follows:
+
+1. The querier receives an HTTP/1 request for data.
+2. The querier passes the query to all ingesters for in-memory data.
+3. The ingesters receive the read request and return data matching the query, if
+ any.
+4. The querier lazily loads data from the backing store and runs the query
+ against it if no ingesters returned data.
+5. The querier iterates over all received data and deduplicates, returning a
+ final set of data over the HTTP/1 connection.
+
+## Write Path
+
+
+
+To summarize, the write path works as follows:
+
+1. The distributor receives an HTTP/1 request to store data for streams.
+2. Each stream is hashed using the hash ring.
+3. The distributor sends each stream to the appropriate ingesters and their
+ replicas (based on the configured replication factor).
+4. Each ingester will create a chunk or append to an existing chunk for the
+ stream's data. A chunk is unique per tenant and per labelset.
+5. The distributor responds with a success code over the HTTP/1 connection.
diff --git a/docs/canary/README.md b/docs/canary/README.md
deleted file mode 100644
index 77e70234d5..0000000000
--- a/docs/canary/README.md
+++ /dev/null
@@ -1,166 +0,0 @@
-# loki-canary
-
-A standalone app to audit the log capturing performance of Loki.
-
-## How it works
-
-
-
-loki-canary writes a log to a file and stores the timestamp in an internal
-array, the contents look something like this:
-
-```nohighlight
-1557935669096040040 ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp
-```
-
-The relevant part is the timestamp, the `p`'s are just filler bytes to make the
-size of the log configurable.
-
-Promtail (or another agent) then reads the log file and ships it to Loki.
-
-Meanwhile loki-canary opens a websocket connection to loki and listens for logs
-it creates
-
-When a log is received on the websocket, the timestamp in the log message is
-compared to the internal array.
-
-If the received log is:
-
- * The next in the array to be received, it is removed from the array and the
- (current time - log timestamp) is recorded in the `response_latency`
- histogram, this is the expected behavior for well behaving logs
- * Not the next in the array received, is is removed from the array, the
- response time is recorded in the `response_latency` histogram, and the
- `out_of_order_entries` counter is incremented
- * Not in the array at all, it is checked against a separate list of received
- logs to either increment the `duplicate_entries` counter or the
- `unexpected_entries` counter.
-
-In the background, loki-canary also runs a timer which iterates through all the
-entries in the internal array, if any are older than the duration specified by
-the `-wait` flag (default 60s), they are removed from the array and the
-`websocket_missing_entries` counter is incremented. Then an additional query is
-made directly to loki for these missing entries to determine if they were
-actually missing or just didn't make it down the websocket. If they are not
-found in the followup query the `missing_entries` counter is incremented.
-
-## Installation
-
-### Binary
-Loki Canary is provided as a pre-compiled binary as part of the
-[Releases](https://github.com/grafana/loki/releases) on GitHub.
-
-### Docker
-Loki Canary is also provided as a Docker container image:
-```bash
-# change tag to the most recent release
-$ docker pull grafana/loki-canary:v0.2.0
-```
-
-### Kubernetes
-To run on Kubernetes, you can do something simple like:
-
-`kubectl run loki-canary --generator=run-pod/v1
---image=grafana/loki-canary:latest --restart=Never --image-pull-policy=IfNotPresent
---labels=name=loki-canary -- -addr=loki:3100`
-
-Or you can do something more complex like deploy it as a daemonset, there is a
-ksonnet setup for this in the `production` folder, you can import it using
-jsonnet-bundler:
-
-```shell
-jb install github.com/grafana/loki-canary/production/ksonnet/loki-canary
-```
-
-Then in your ksonnet environments `main.jsonnet` you'll want something like
-this:
-
-```jsonnet
-local loki_canary = import 'loki-canary/loki-canary.libsonnet';
-
-loki_canary {
- loki_canary_args+:: {
- addr: "loki:3100",
- port: 80,
- labelname: "instance",
- interval: "100ms",
- size: 1024,
- wait: "3m",
- },
- _config+:: {
- namespace: "default",
- }
-}
-
-```
-
-### From Source
-If the other options are not sufficient for your use-case, you can compile
-`loki-canary` yourself:
-
-```bash
-# clone the source tree
-$ git clone https://github.com/grafana/loki
-
-# build the binary
-$ make loki-canary
-
-# (optionally build the container image)
-$ make loki-canary-image
-```
-
-## Configuration
-
-It is required to pass in the Loki address with the `-addr` flag, if your server
-uses TLS, also pass `-tls=true` (this will create a `wss://` instead of `ws://`
-connection)
-
-You should also pass the `-labelname` and `-labelvalue` flags, these are used by
-loki-canary to filter the log stream to only process logs for this instance of
-loki-canary, so they must be unique per each of your loki-canary instances. The
-ksonnet config in this project accomplishes this by passing in the pod name as
-the labelvalue
-
-If you get a high number of `unexpected_entries` you may not be waiting long
-enough and should increase `-wait` from 60s to something larger.
-
-__Be cognizant__ of the relationship between `pruneinterval` and the `interval`.
-For example, with an interval of 10ms (100 logs per second) and a prune interval
-of 60s, you will write 6000 logs per minute, if those logs were not received
-over the websocket, the canary will attempt to query loki directly to see if
-they are completely lost. __However__ the query return is limited to 1000
-results so you will not be able to return all the logs even if they did make it
-to Loki.
-
-__Likewise__, if you lower the `pruneinterval` you risk causing a denial of
-service attack as all your canaries attempt to query for missing logs at
-whatever your `pruneinterval` is defined at.
-
-All options:
-
-```nohighlight
- -addr string
- The Loki server URL:Port, e.g. loki:3100
- -buckets int
- Number of buckets in the response_latency histogram (default 10)
- -interval duration
- Duration between log entries (default 1s)
- -labelname string
- The label name for this instance of loki-canary to use in the log selector (default "name")
- -labelvalue string
- The unique label value for this instance of loki-canary to use in the log selector (default "loki-canary")
- -pass string
- Loki password
- -port int
- Port which loki-canary should expose metrics (default 3500)
- -pruneinterval duration
- Frequency to check sent vs received logs, also the frequency which queries for missing logs will be dispatched to loki (default 1m0s)
- -size int
- Size in bytes of each log line (default 100)
- -tls
- Does the loki connection use TLS?
- -user string
- Loki username
- -wait duration
- Duration to wait for log entries before reporting them lost (default 1m0s)
-```
diff --git a/docs/canary/loki-canary-daemonset.yml b/docs/canary/loki-canary-daemonset.yml
deleted file mode 100644
index 5138b5e9c6..0000000000
--- a/docs/canary/loki-canary-daemonset.yml
+++ /dev/null
@@ -1,42 +0,0 @@
----
-kind: DaemonSet
-apiVersion: extensions/v1beta1
-metadata:
- labels:
- app: loki-canary
- name: loki-canary
- name: loki-canary
-spec:
- template:
- metadata:
- name: loki-canary
- labels:
- app: loki-canary
- spec:
- containers:
- - args:
- - -addr=loki:3100
- image: grafana/loki-canary:latest
- imagePullPolicy: IfNotPresent
- name: loki-canary
- resources: {}
----
-apiVersion: v1
-kind: Service
-metadata:
- name: loki-canary
- labels:
- app: loki-canary
-spec:
- type: ClusterIP
- selector:
- app: loki-canary
- ports:
- - name: metrics
- protocol: TCP
- port: 3500
- targetPort: 3500
-
-
-
-
diff --git a/docs/canary/loki-canary.yml b/docs/canary/loki-canary.yml
deleted file mode 100644
index 0f3741d28f..0000000000
--- a/docs/canary/loki-canary.yml
+++ /dev/null
@@ -1,36 +0,0 @@
----
-apiVersion: v1
-kind: Pod
-metadata:
- labels:
- app: loki-canary
- name: loki-canary
- name: loki-canary
-spec:
- containers:
- - args:
- - -addr=loki:3100
- image: grafana/loki-canary:latest
- imagePullPolicy: IfNotPresent
- name: loki-canary
- resources: {}
----
-apiVersion: v1
-kind: Service
-metadata:
- name: loki-canary
- labels:
- app: loki-canary
-spec:
- type: ClusterIP
- selector:
- app: loki-canary
- ports:
- - name: metrics
- protocol: TCP
- port: 3500
- targetPort: 3500
-
-
-
-
diff --git a/docs/chunks_diagram.png b/docs/chunks_diagram.png
new file mode 100644
index 0000000000..03df98ae3e
Binary files /dev/null and b/docs/chunks_diagram.png differ
diff --git a/docs/clients/README.md b/docs/clients/README.md
new file mode 100644
index 0000000000..413058d6a5
--- /dev/null
+++ b/docs/clients/README.md
@@ -0,0 +1,29 @@
+# Loki Clients
+
+Loki supports the following official clients for sending logs:
+
+1. [Promtail](./promtail/README.md)
+2. [Docker Driver](./docker-driver/README.md)
+3. [Fluentd](./fluentd.md)
+
+## Picking a Client
+
+While all clients can be used simultaneously to cover multiple use cases, which
+client is initially picked to send logs depends on your use case.
+
+Promtail is the client of choice when you're running Kubernetes, as you can
+configure it to automatically scrape logs from pods running on the same node
+that Promtail runs on. Promtail and Prometheus running together in Kubernetes
+enables powerful debugging: if Prometheus and Promtail use the same labels,
+users can use tools like Grafana to switch between metrics and logs based on the
+label set.
+
+Promtail is also the client of choice on bare-metal: since it can be configured
+to tail logs from all files given a host path, it is the easiest way to send
+logs to Loki from plain-text files (e.g., things that log to `/var/log/*.log`).
+
+When using Docker and not Kubernetes, the Docker Logging driver should be used,
+as it automatically adds labels appropriate to the running container.
+
+The Fluentd plugin is ideal when you already have Fluentd deployed and you don't
+need the service discovery capabilities of Promtail.
diff --git a/docs/clients/docker-driver/README.md b/docs/clients/docker-driver/README.md
new file mode 100644
index 0000000000..ad8be5b9be
--- /dev/null
+++ b/docs/clients/docker-driver/README.md
@@ -0,0 +1,54 @@
+# Docker Driver Client
+
+Loki officially supports a Docker plugin that will read logs from Docker
+containers and ship them to Loki. The plugin can be configured to send the logs
+to a private Loki instance or [Grafana Cloud](https://grafana.com/oss/loki).
+
+> Docker plugins are not yet supported on Windows; see the
+> [Docker docs](https://docs.docker.com/engine/extend) for more information.
+
+Documentation on configuring the Loki Docker Driver can be found on the
+[configuration page](./configuration.md).
+
+## Installing
+
+The Docker plugin must be installed on each Docker host that will be running
+containers you want to collect logs from.
+
+Run the following command to install the plugin:
+
+```bash
+docker plugin install grafana/loki-docker-driver:latest --alias loki
+--grant-all-permissions
+```
+
+To check installed plugins, use the `docker plugin ls` command. Plugins that
+have started successfully are listed as enabled:
+
+```
+$ docker plugin ls
+ID NAME DESCRIPTION ENABLED
+ac720b8fcfdb loki Loki Logging Driver true
+```
+
+Once the plugin is installed it can be [configured](./configuration.md).
+
+## Upgrading
+
+The upgrade process involves disabling the existing plugin, upgrading, and then
+re-enabling:
+
+```bash
+docker plugin disable loki
+docker plugin upgrade loki grafana/loki-docker-driver:master
+docker plugin enable loki
+```
+
+## Uninstalling
+
+To cleanly uninstall the plugin, disable and remove it:
+
+```bash
+docker plugin disable loki
+docker plugin rm loki
+```
diff --git a/docs/clients/docker-driver/configuration.md b/docs/clients/docker-driver/configuration.md
new file mode 100644
index 0000000000..e106c4d59e
--- /dev/null
+++ b/docs/clients/docker-driver/configuration.md
@@ -0,0 +1,152 @@
+# Configuring the Docker Driver
+
+The Docker daemon on each machine has a default logging driver and
+each container will use the default driver unless configured otherwise.
+
+## Change the logging driver for a container
+
+The `docker run` command can be configured to use a different logging driver
+than the Docker daemon's default with the `--log-driver` flag. Any options that
+the logging driver supports can be set using the `--log-opt =` flag.
+`--log-opt` can be passed multiple times for each option to be set.
+
+The following command will start Grafana in a container and send logs to Grafana
+Cloud, using a batch size of 400 entries and no more than 5 retries if a send
+fails.
+
+```bash
+docker run --log-driver=loki \
+ --log-opt loki-url="https://:@logs-us-west1.grafana.net/loki/api/v1/push" \
+ --log-opt loki-retries=5 \
+ --log-opt loki-batch-size=400 \
+ grafana/grafana
+```
+
+> **Note**: The Loki logging driver still uses the json-log driver in
+> combination with sending logs to Loki. This is mainly useful to keep the
+> `docker logs` command working. You can adjust file size and rotation
+> using the respective log option `max-size` and `max-file`.
+
+## Change the default logging driver
+
+If you want the Loki logging driver to be the default for all containers,
+change Docker's `daemon.json` file (located in `/etc/docker` on Linux) and set
+the value of `log-driver` to `loki`:
+
+```json
+{
+ "debug": true,
+ "log-driver": "loki"
+}
+```
+
+Options for the logging driver can also be configured with `log-opts` in the
+`daemon.json`:
+
+```json
+{
+ "debug" : true,
+ "log-driver": "loki",
+ "log-opts": {
+ "loki-url": "https://:@logs-us-west1.grafana.net/loki/api/v1/push",
+ "loki-batch-size": "400"
+ }
+}
+```
+
+> **Note**: log-opt configuration options in daemon.json must be provided as
+> strings. Boolean and numeric values (such as the value for loki-batch-size in
+> the example above) must therefore be enclosed in quotes (`"`).
+
+After changing `daemon.json`, restart the Docker daemon for the changes to take
+effect. All containers from that host will then send logs to Loki.
+
+## Configure the logging driver for a Swarm service or Compose
+
+You can also configure the logging driver for a [swarm
+service](https://docs.docker.com/engine/swarm/how-swarm-mode-works/services/)
+directly in your compose file. This also applies for `docker-compose`:
+
+```yaml
+version: "3.7"
+services:
+ logger:
+ image: grafana/grafana
+ logging:
+ driver: loki
+ options:
+ loki-url: "https://:@logs-us-west1.grafana.net/loki/api/v1/push"
+```
+
+You can then deploy your stack using:
+
+```bash
+docker stack deploy my_stack_name --compose-file docker-compose.yaml
+```
+
+Or with `docker-compose`:
+
+```bash
+docker-compose -f docker-compose.yaml up
+```
+
+Once deployed, the Grafana service will send its logs to Loki.
+
+> **Note**: stack name and service name for each swarm service and project name
+> and service name for each compose service are automatically discovered and
+> sent as Loki labels, this way you can filter by them in Grafana.
+
+## Labels
+
+By default, the Docker driver will add the following labels to each log line:
+
+- `filename`: where the log is written to on disk
+- `host`: the hostname where the log has been generated
+- `container_name`: the name of the container generating logs
+- `swarm_stack`, `swarm_service`: added when deploying from Docker Swarm.
+
+Custom labels can be added using the `loki-external-labels`,
+`loki-pipeline-stage-file`, `labels`, `env`, and `env-regex` options. See the
+next section for all supported options.
+
+## Supported log-opt options
+
+The following are all supported options that the Loki logging driver supports:
+
+| Option | Required? | Default Value | Description
+| ------------------------------- | :-------: | :------------------------: | -------------------------------------- |
+| `loki-url` | Yes | | Loki HTTP push endpoint.
+| `loki-external-labels` | No | `container_name={{.Name}}` | Additional label value pairs separated by `,` to send with logs. The value is expanded with the [Docker tag template format](https://docs.docker.com/engine/admin/logging/log_tags/). (e.g.,: `container_name={{.ID}}.{{.Name}},cluster=prod`)
+| `loki-timeout` | No | `10s` | The timeout to use when sending logs to the Loki instance. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
+| `loki-batch-wait` | No | `1s` | The amount of time to wait before sending a log batch complete or not. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
+| `loki-batch-size` | No | `102400` | The maximum size of a log batch to send.
+| `loki-min-backoff` | No | `100ms` | The minimum amount of time to wait before retrying a batch. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
+| `loki-max-backoff` | No | `10s` | The maximum amount of time to wait before retrying a batch. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
+| `loki-retries` | No | `10` | The maximum amount of retries for a log batch.
+| `loki-pipeline-stage-file` | No | | The location of a pipeline stage configuration file. Pipeline stages allows to parse log lines to extract more labels. [See the Promtail documentation for more info.](../promtail/pipelines.md)
+| `loki-tls-ca-file` | No | | Set the path to a custom certificate authority.
+| `loki-tls-cert-file` | No | | Set the path to a client certificate file.
+| `loki-tls-key-file` | No | | Set the path to a client key.
+| `loki-tls-server-name` | No | | Name used to validate the server certificate.
+| `loki-tls-insecure-skip-verify` | No | `false` | Allow to skip tls verification.
+| `loki-proxy-url` | No | | Proxy URL use to connect to Loki.
+| `max-size` | No | -1 | The maximum size of the log before it is rolled. A positive integer plus a modifier representing the unit of measure (k, m, or g). Defaults to -1 (unlimited). This is used by json-log required to keep the `docker log` command working.
+| `max-file` | No | 1 | The maximum number of log files that can be present. If rolling the logs creates excess files, the oldest file is removed. Only effective when max-size is also set. A positive integer. Defaults to 1.
+| `labels` | No | | Comma-separated list of keys of labels, which should be included in message, if these labels are specified for container.
+| `env` | No | | Comma-separated list of keys of environment variables to be included in message if they specified for a container.
+| `env-regex` | No | | A regular expression to match logging-related environment variables. Used for advanced log label options. If there is collision between the label and env keys, the value of the env takes precedence. Both options add additional fields to the labels of a logging message.
+
+## Troubleshooting
+
+Plugin logs can be found as docker daemon log. To enable debug mode refer to the
+[Docker daemon documentation](https://docs.docker.com/config/daemon/).
+
+The standard output (`stdout`) of a plugin is redirected to Docker logs. Such
+entries are prefixed with `plugin=`.
+
+To find out the plugin ID of the Loki logging driver, use `docker plugin ls` and
+look for the `loki` entry.
+
+Depending on your system, location of Docker daemon logging may vary. Refer to
+[Docker documentation for Docker daemon](https://docs.docker.com/config/daemon/)
+log location for your specific platform.
diff --git a/docs/clients/fluentd.md b/docs/clients/fluentd.md
new file mode 100644
index 0000000000..03885155c8
--- /dev/null
+++ b/docs/clients/fluentd.md
@@ -0,0 +1,179 @@
+# Fluentd
+
+Loki has a [Fluentd](https://fluentd.org/) output plugin called
+`fluent-plugin-grafana-loki` that enables shipping logs to a private Loki
+instance or [Grafana Cloud](https://grafana.com/oss/loki).
+
+The plugin offers two line formats and uses protobuf to send compressed data to
+Loki.
+
+Key features:
+
+* `extra_labels`: Labels to be added to every line of a log file, useful for
+ designating environments
+* `label_keys`: Customizable list of keys for stream labels
+* `line_format`: Format to use when flattening the record to a log line (`json`
+ or `key_value`).
+
+## Installation
+
+```bash
+$ gem install fluent-plugin-grafana-loki
+```
+
+## Usage
+
+In your Fluentd configuration, use `@type loki`. Additional configuration is
+optional. Default values look like this:
+
+```
+
+ @type loki
+ url "https://logs-us-west1.grafana.net"
+ username "#{ENV['LOKI_USERNAME']}"
+ password "#{ENV['LOKI_PASSWORD']}"
+ extra_labels {"env":"dev"}
+ flush_interval 10s
+ flush_at_shutdown true
+ buffer_chunk_limit 1m
+
+```
+
+### Multi-worker usage
+
+Loki doesn't currently support out-of-order inserts - if you try to insert a log
+entry with an earlier timestamp after a log entry with with identical labels but
+a later timestamp, the insert will fail with the message
+`HTTP status code: 500, message: rpc error: code = Unknown desc = Entry out of
+order`. Therefore, in order to use this plugin in a multi-worker Fluentd setup,
+you'll need to include the worker ID in the labels sent to Loki.
+
+For example, using
+[fluent-plugin-record-modifier](https://github.com/repeatedly/fluent-plugin-record-modifier):
+
+```
+
+ @type record_modifier
+
+ fluentd_worker "#{worker_id}"
+
+
+
+
+ @type loki
+ # ...
+ label_keys "fluentd_worker"
+ # ...
+
+```
+
+## Docker Image
+
+There is a Docker image `grafana/fluent-plugin-grafana-loki:master` which
+contains default configuration files to git log information
+a host's `/var/log` directory, and from the host's journald. To use it, you can set
+the `LOKI_URL`, `LOKI_USERNAME`, and `LOKI_PASSWORD` environment variables
+(`LOKI_USERNAME` and `LOKI_PASSWORD` can be left blank if Loki is not protected
+behind an authenticating proxy).
+
+An example Docker Swarm Compose configuration looks like:
+
+```yaml
+services:
+ fluentd:
+ image: grafana/fluent-plugin-grafana-loki:master
+ command:
+ - "fluentd"
+ - "-v"
+ - "-p"
+ - "/fluentd/plugins"
+ environment:
+ LOKI_URL: http://loki:3100
+ LOKI_USERNAME:
+ LOKI_PASSWORD:
+ deploy:
+ mode: global
+ configs:
+ - source: loki_config
+ target: /fluentd/etc/loki/loki.conf
+ networks:
+ - loki
+ volumes:
+ - host_logs:/var/log
+ # Needed for journald log ingestion:
+ - /etc/machine-id:/etc/machine-id
+ - /dev/log:/dev/log
+ - /var/run/systemd/journal/:/var/run/systemd/journal/
+ logging:
+ options:
+ tag: infra.monitoring
+```
+
+## Configuration
+
+### Proxy Support
+
+Starting with version 0.8.0, this gem uses `excon`, which supports proxy with
+environment variables - https://github.com/excon/excon#proxy-support
+
+### `url`
+
+The URL of the Loki server to send logs to. When sending data the publish path
+(`/loki/api/v1/push`) will automatically be appended. By default the URL is set to
+`https://logs-us-west1.grafana.net`, the URL of the Grafana Labs [hosted
+Loki](https://grafana.com/loki) service.
+
+### `username` / `password`
+
+Specify a username and password if the Loki server requires authentication.
+If using the Grafana Labs' hosted Loki, the username needs to be set to your
+instanceId and the password should be a grafana.com API Key.
+
+### `tenant`
+
+Loki is a multi-tenant log storage platform and all requests sent must include a
+tenant. For some installations (like Hosted Loki) the tenant will be set
+automatically by an authenticating proxy. Otherwise you can define a tenant to
+be passed through. The tenant can be any string value.
+
+### output format
+
+Loki is intended to index and group log streams using only a small set of
+labels and is not intended for full-text indexing. When sending logs to Loki,
+the majority of log message will be sent as a single log "line".
+
+There are few configurations settings to control the output format:
+
+- `extra_labels`: (default: nil) set of labels to include with every Loki
+ stream. (e.g., `{"env":"dev", "datacenter": "dc1"}`)
+
+- `remove_keys`: (default: nil) comma separated list of record keys to
+ remove. All other keys will be placed into the log line.
+
+- `label_keys`: (default: "job,instance") comma separated list of keys to use as
+ stream labels.
+
+- `line_format`: format to use when flattening the record to a log line. Valid
+ values are `json` or `key_value`. If set to `json` the log line sent to Loki
+ will be the fluentd record (excluding any keys extracted out as labels) dumped
+ as json. If set to `key_value`, the log line will be each item in the record
+ concatenated together (separated by a single space) in the format
+ `=`.
+
+- `drop_single_key`: if set to true, when the set of extracted `label_keys`
+ after dropping with `remove_keys`, the log line sent to Loki will just be
+ the value of the single remaining record key.
+
+### Buffer options
+
+`fluentd-plugin-loki` extends [Fluentd's builtin Output
+plugin](https://docs.fluentd.org/v1.0/articles/output-plugin-overview) and uses
+the `compat_parameters` plugin helper. It adds the following options:
+
+```
+buffer_type memory
+flush_interval 10s
+retry_limit 17
+retry_wait 1.0
+num_threads 1
+```
diff --git a/docs/clients/promtail/README.md b/docs/clients/promtail/README.md
new file mode 100644
index 0000000000..3602a5c9df
--- /dev/null
+++ b/docs/clients/promtail/README.md
@@ -0,0 +1,83 @@
+# Promtail
+
+Promtail is an agent which ships the contents of local logs to a private Loki
+instance or [Grafana Cloud](https://grafana.com/oss/loki). It is usually
+deployed to every machine that has applications needed to be monitored.
+
+It primarily:
+
+1. Discovers targets
+2. Attaches labels to log streams
+3. Pushes them to the Loki instance.
+
+Currently, Promtail can tail logs from two sources: local log files and the
+systemd journal (on AMD64 machines only).
+
+## Log File Discovery
+
+Before Promtail can ship any data from log files to Loki, it needs to find out
+information about its environment. Specifically, this means discovering
+applications emitting log lines to files that need to be monitored.
+
+Promtail borrows the same
+[service discovery mechanism from Prometheus](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config),
+although it currently only supports `static` and `kubernetes` service
+discovery. This limitation is due to the fact that `promtail` is deployed as a
+daemon to every local machine and, as such, does not discover label from other
+machines. `kubernetes` service discovery fetches required labels from the
+Kubernetes API server while `static` usually covers all other use cases.
+
+Just like Prometheus, `promtail` is configured using a `scrape_configs` stanza.
+`relabel_configs` allows for fine-grained control of what to ingest, what to
+drop, and the final metadata to attach to the log line. Refer to the docs for
+[configuring Promtail](configuration.md) for more details.
+
+## Labeling and Parsing
+
+During service discovery, metadata is determined (pod name, filename, etc.) that
+may be attached to the log line as a label for easier identification when
+querying logs in Loki. Through `relabel_configs`, discovered labels can be
+mutated into the desired form.
+
+To allow more sophisticated filtering afterwards, Promtail allows to set labels
+not only from service discovery, but also based on the contents of each log
+line. The `pipeline_stages` can be used to add or update labels, correct the
+timestamp, or re-write log lines entirely. Refer to the documentation for
+[pipelines](pipelines.md) for more details.
+
+## Shipping
+
+Once Promtail has a set of targets (i.e., things to read from, like files) and
+all labels are set correctly, it will start tailing (continuously reading) the
+logs from targets. Once enough data is read into memory or after a configurable
+timeout, it is flushed as a single batch to Loki.
+
+As Promtail reads data from sources (files and systemd journal, if configured),
+it will track the last offset it read in a positions file. By default, the
+positions file is stored at `/var/log/positions.yaml`. The positions file helps
+Promtail continue reading from where it left off in the case of the Promtail
+instance restarting.
+
+## API
+
+Promtail features an embedded web server exposing a web console at `/` and the following API endpoints:
+
+### `GET /ready`
+
+This endpoint returns 200 when Promtail is up and running, and there's at least one working target.
+
+### `GET /metrics`
+
+This endpoint returns Promtail metrics for Prometheus. See
+"[Operations > Observability](../../operations/observability.md)" to get a list
+of exported metrics.
+
+### Promtail web server config
+
+The web server exposed by Promtail can be configured in the Promtail `.yaml` config file:
+
+```yaml
+server:
+ http_listen_host: 127.0.0.1
+ http_listen_port: 9080
+```
diff --git a/docs/clients/promtail/configuration.md b/docs/clients/promtail/configuration.md
new file mode 100644
index 0000000000..2674ea705d
--- /dev/null
+++ b/docs/clients/promtail/configuration.md
@@ -0,0 +1,949 @@
+# Configuring Promtail
+
+Promtail is configured in a YAML file (usually referred to as `config.yaml`)
+which contains information on the Promtail server, where positions are stored,
+and how to scrape logs from files.
+
+* [Configuration File Reference](#configuration-file-reference)
+* [server_config](#server_config)
+* [client_config](#client_config)
+* [position_config](#position_config)
+* [scrape_config](#scrape_config)
+ * [pipeline_stages](#pipeline_stages)
+ * [regex_stage](#regex_stage)
+ * [json_stage](#json_stage)
+ * [template_stage](#template_stage)
+ * [match_stage](#match_stage)
+ * [timestamp_stage](#timestamp_stage)
+ * [output_stage](#output_stage)
+ * [labels_stage](#labels_stage)
+ * [metrics_stage](#metrics_stage)
+ * [metric_counter](#metric_counter)
+ * [metric_gauge](#metric_gauge)
+ * [metric_histogram](#metric_histogram)
+ * [journal_config](#journal_config)
+ * [relabel_config](#relabel_config)
+ * [static_config](#static_config)
+ * [file_sd_config](#file_sd_config)
+ * [kubernetes_sd_config](#kubernetes_sd_config)
+* [target_config](#target_config)
+* [Example Docker Config](#example-docker-config)
+* [Example Journal Config](#example-journal-config)
+
+## Configuration File Reference
+
+To specify which configuration file to load, pass the `-config.file` flag at the
+command line. The file is written in [YAML format](https://en.wikipedia.org/wiki/YAML),
+defined by the schema below. Brackets indicate that a parameter is optional. For
+non-list parameters the value is set to the specified default.
+
+For more detailed information on configuring how to discover and scrape logs from
+targets, see [Scraping](scraping.md). For more information on transforming logs
+from scraped targets, see [Pipelines](pipelines.md).
+
+Generic placeholders are defined as follows:
+
+* ``: a boolean that can take the values `true` or `false`
+* ``: any integer matching the regular expression `[1-9]+[0-9]*`
+* ``: a duration matching the regular expression `[0-9]+(ms|[smhdwy])`
+* ``: a string matching the regular expression `[a-zA-Z_][a-zA-Z0-9_]*`
+* ``: a string of Unicode characters
+* ``: a valid path relative to current working directory or an
+ absolute path.
+* ``: a valid string consisting of a hostname or IP followed by an optional port number
+* ``: a regular string
+* ``: a regular string that is a secret, such as a password
+
+Supported contents and default values of `config.yaml`:
+
+```yaml
+# Configures the server for Promtail.
+[server: ]
+
+# Describes how promtail connects to multiple instances
+# of Loki, sending logs to each.
+clients:
+ - []
+
+# Describes how to save read file offsets to disk
+[positions: ]
+
+scrape_configs:
+ - []
+
+# Configures how tailed targets will be watched.
+[target_config: ]
+```
+
+## server_config
+
+The `server_config` block configures Promtail's behavior as an HTTP server:
+
+```yaml
+# HTTP server listen host
+[http_listen_host: ]
+
+# HTTP server listen port
+[http_listen_port: | default = 80]
+
+# gRPC server listen host
+[grpc_listen_host: ]
+
+# gRPC server listen port
+[grpc_listen_port: | default = 9095]
+
+# Register instrumentation handlers (/metrics, etc.)
+[register_instrumentation: | default = true]
+
+# Timeout for graceful shutdowns
+[graceful_shutdown_timeout: | default = 30s]
+
+# Read timeout for HTTP server
+[http_server_read_timeout: | default = 30s]
+
+# Write timeout for HTTP server
+[http_server_write_timeout: | default = 30s]
+
+# Idle timeout for HTTP server
+[http_server_idle_timeout: | default = 120s]
+
+# Max gRPC message size that can be received
+[grpc_server_max_recv_msg_size: | default = 4194304]
+
+# Max gRPC message size that can be sent
+[grpc_server_max_send_msg_size: | default = 4194304]
+
+# Limit on the number of concurrent streams for gRPC calls (0 = unlimited)
+[grpc_server_max_concurrent_streams: | default = 100]
+
+# Log only messages with the given severity or above. Supported values [debug,
+# info, warn, error]
+[log_level: | default = "info"]
+
+# Base path to server all API routes from (e.g., /v1/).
+[http_path_prefix: ]
+```
+
+## client_config
+
+The `client_config` block configures how Promtail connects to an instance of
+Loki:
+
+```yaml
+# The URL where Loki is listening, denoted in Loki as http_listen_host and
+# http_listen_port. If Loki is running in microservices mode, this is the HTTP
+# URL for the Distributor.
+url:
+
+# Maximum amount of time to wait before sending a batch, even if that
+# batch isn't full.
+[batchwait: | default = 1s]
+
+# Maximum batch size (in bytes) of logs to accumulate before sending
+# the batch to Loki.
+[batchsize: | default = 102400]
+
+# If using basic auth, configures the username and password
+# sent.
+basic_auth:
+ # The username to use for basic auth
+ [username: ]
+
+ # The password to use for basic auth
+ [password: ]
+
+ # The file containing the password for basic auth
+ [password_file: ]
+
+# Bearer token to send to the server.
+[bearer_token: ]
+
+# File containing bearer token to send to the server.
+[bearer_token_file: ]
+
+# HTTP proxy server to use to connect to the server.
+[proxy_url: ]
+
+# If connecting to a TLS server, configures how the TLS
+# authentication handshake will operate.
+tls_config:
+ # The CA file to use to verify the server
+ [ca_file: ]
+
+ # The cert file to send to the server for client auth
+ [cert_file: ]
+
+ # The key file to send to the server for client auth
+ [key_file: ]
+
+ # Validates that the server name in the server's certificate
+ # is this value.
+ [server_name: ]
+
+ # If true, ignores the server certificate being signed by an
+ # unknown CA.
+ [insecure_skip_verify: | default = false]
+
+# Configures how to retry requests to Loki when a request
+# fails.
+backoff_config:
+ # Initial backoff time between retries
+ [minbackoff: | default = 100ms]
+
+ # Maximum backoff time between retries
+ [maxbackoff: | default = 10s]
+
+ # Maximum number of retries to do
+ [maxretries: | default = 10]
+
+# Static labels to add to all logs being sent to Loki.
+# Use map like {"foo": "bar"} to add a label foo with
+# value bar.
+external_labels:
+ [ : ... ]
+
+# Maximum time to wait for a server to respond to a request
+[timeout: | default = 10s]
+```
+
+## position_config
+
+The `position_config` block configures where Promtail will save a file
+indicating how far it has read into a file. It is needed for when Promtail
+is restarted to allow it to continue from where it left off.
+
+```yaml
+# Location of positions file
+[filename: | default = "/var/log/positions.yaml"]
+
+# How often to update the positions file
+[sync_period: | default = 10s]
+```
+
+## scrape_config
+
+The `scrape_config` block configures how Promtail can scrape logs from a series
+of targets using a specified discovery method:
+
+```yaml
+# Name to identify this scrape config in the promtail UI.
+job_name:
+
+# Describes how to parse log lines. Suported values [cri docker raw]
+[entry_parser: | default = "docker"]
+
+# Describes how to transform logs from targets.
+[pipeline_stages: ]
+
+# Describes how to scrape logs from the journal.
+[journal: ]
+
+# Describes how to relabel targets to determine if they should
+# be processed.
+relabel_configs:
+ - []
+
+# Static targets to scrape.
+static_configs:
+ - []
+
+# Files containing targets to scrape.
+file_sd_configs:
+ - []
+
+# Describes how to discover Kubernetes services running on the
+# same host.
+kubernetes_sd_configs:
+ - []
+```
+
+### pipeline_stages
+
+The [pipeline](./pipelines.md) stages (`pipeline_stages`) is used to transform
+log entries and their labels after discovery. It is simply an array of various
+stages, defined below.
+
+The purpose of most stages is to extract fields and values into a temporary
+set of key-value pairs that is passed around from stage to stage.
+
+```yaml
+- [
+
+ |
+ |
+ |
+ |
+ |
+ |
+
+ ]
+```
+
+Example:
+
+```yaml
+pipeline_stages:
+ - regex:
+ expr: "./*"
+ - json:
+ timestamp:
+ source: time
+ format: RFC3339
+ labels:
+ stream:
+ source: json_key_name.json_sub_key_name
+ output:
+```
+
+#### regex_stage
+
+The Regex stage takes a regular expression and extracts captured named groups to
+be used in further stages.
+
+```yaml
+regex:
+ # The RE2 regular expression. Each capture group must be named.
+ expression:
+
+ # Name from extracted data to parse. If empty, uses the log message.
+ [source: ]
+```
+
+#### json_stage
+
+The JSON stage parses a log line as JSON and takes
+[JMESPath](http://jmespath.org/) expressions to extract data from the JSON to be
+used in further stages.
+
+```yaml
+json:
+ # Set of key/value pairs of JMESPath expressions. The key will be
+ # the key in the extracted data while the expression will the value,
+ # evaluated as a JMESPath from the source data.
+ expressions:
+ [ : ... ]
+
+ # Name from extracted data to parse. If empty, uses the log message.
+ [source: ]
+```
+
+#### template_stage
+
+The template stage uses Go's
+[`text/template`](https://golang.org/pkg/text/template) language to manipulate
+values.
+
+```yaml
+template:
+ # Name from extracted data to parse. If key in extract data doesn't exist, an
+ # entry for it will be created.
+ source:
+
+ # Go template string to use. In additional to normal template
+ # functions, ToLower, ToUpper, Replace, Trim, TrimLeft, TrimRight,
+ # TrimPrefix, TrimSuffix, and TrimSpace are available as functions.
+ template:
+```
+
+Example:
+
+```yaml
+template:
+ source: level
+ template: '{{ if eq .Value "WARN" }}{{ Replace .Value "WARN" "OK" -1 }}{{ else }}{{ .Value }}{{ end }}'
+```
+
+#### match_stage
+
+The match stage conditionally executes a set of stages when a log entry matches
+a configurable [LogQL](../../logql.md) stream selector.
+
+```yaml
+match:
+ # LogQL stream selector.
+ selector:
+
+ # Names the pipeline. When defined, creates an additional label in
+ # the pipeline_duration_seconds histogram, where the value is
+ # concatenated with job_name using an underscore.
+ [pipieline_name: ]
+
+ # Nested set of pipeline stages only if the selector
+ # matches the labels of the log entries:
+ stages:
+ - [
+
+ |
+ |
+ |
+ |
+ |
+ |
+
+ ]
+```
+
+#### timestamp_stage
+
+The timestamp stage parses data from the extracted map and overrides the final
+time value of the log that is stored by Loki. If this stage isn't present,
+Promtail will associate the timestamp of the log entry with the time that
+log entry was read.
+
+```yaml
+timestamp:
+ # Name from extracted data to use for the timestamp.
+ source:
+
+ # Determines how to parse the time string. Can use
+ # pre-defined formats by name: [ANSIC UnixDate RubyDate RFC822
+ # RFC822Z RFC850 RFC1123 RFC1123Z RFC3339 RFC3339Nano Unix
+ # UnixMs UnixNs].
+ format:
+
+ # IANA Timezone Database string.
+ [location: ]
+```
+
+##### output_stage
+
+The output stage takes data from the extracted map and sets the contents of the
+log entry that will be stored by Loki.
+
+```yaml
+output:
+ # Name from extracted data to use for the log entry.
+ source:
+```
+
+#### labels_stage
+
+The labels stage takes data from the extracted map and sets additional labels
+on the log entry that will be sent to Loki.
+
+```yaml
+labels:
+ # Key is REQUIRED and the name for the label that will be created.
+ # Value is optional and will be the name from extracted data whose value
+ # will be used for the value of the label. If empty, the value will be
+ # inferred to be the same as the key.
+ [ : [] ... ]
+```
+
+#### metrics_stage
+
+The metrics stage allows for defining metrics from the extracted data.
+
+Created metrics are not pushed to Loki and are instead exposed via Promtail's
+`/metrics` endpoint. Prometheus should be configured to scrape Promtail to be
+able to retrieve the metrics configured by this stage.
+
+
+```yaml
+# A map where the key is the name of the metric and the value is a specific
+# metric type.
+metrics:
+ [: [ | | ] ...]
+```
+
+##### metric_counter
+
+Defines a counter metric whose value only goes up.
+
+```yaml
+# The metric type. Must be Counter.
+type: Counter
+
+# Describes the metric.
+[description: ]
+
+# Key from the extracted data map to use for the mtric,
+# defaulting to the metric's name if not present.
+[source: ]
+
+config:
+ # Filters down source data and only changes the metric
+ # if the targeted value exactly matches the provided string.
+ # If not present, all data will match.
+ [value: ]
+
+ # Must be either "inc" or "add" (case insensitive). If
+ # inc is chosen, the metric value will increase by 1 for each
+ # log line receieved that passed the filter. If add is chosen,
+ # the extracted value most be convertible to a positive float
+ # and its value will be added to the metric.
+ action:
+```
+
+##### metric_gauge
+
+Defines a gauge metric whose value can go up or down.
+
+```yaml
+# The metric type. Must be Gauge.
+type: Gauge
+
+# Describes the metric.
+[description: ]
+
+# Key from the extracted data map to use for the mtric,
+# defaulting to the metric's name if not present.
+[source: ]
+
+config:
+ # Filters down source data and only changes the metric
+ # if the targeted value exactly matches the provided string.
+ # If not present, all data will match.
+ [value: ]
+
+ # Must be either "set", "inc", "dec"," add", or "sub". If
+ # add, set, or sub is chosen, the extracted value must be
+ # convertible to a positive float. inc and dec will increment
+ # or decrement the metric's value by 1 respectively.
+ action:
+```
+
+##### metric_histogram
+
+Defines a histogram metric whose values are bucketed.
+
+```yaml
+# The metric type. Must be Histogram.
+type: Histogram
+
+# Describes the metric.
+[description: ]
+
+# Key from the extracted data map to use for the mtric,
+# defaulting to the metric's name if not present.
+[source: ]
+
+config:
+ # Filters down source data and only changes the metric
+ # if the targeted value exactly matches the provided string.
+ # If not present, all data will match.
+ [value: ]
+
+ # Must be either "inc" or "add" (case insensitive). If
+ # inc is chosen, the metric value will increase by 1 for each
+ # log line receieved that passed the filter. If add is chosen,
+ # the extracted value most be convertible to a positive float
+ # and its value will be added to the metric.
+ action:
+
+ # Holds all the numbers in which to bucket the metric.
+ buckets:
+ -
+```
+
+### journal_config
+
+The `journal_config` block configures reading from the systemd journal from
+Promtail. Requires a build of Promtail that has journal support _enabled_. If
+using the AMD64 Docker image, this is enabled by default.
+
+```yaml
+# The oldest relative time from process start that will be read
+# and sent to Loki.
+[max_age: | default = 7h]
+
+# Label map to add to every log coming out of the journal
+labels:
+ [ : ... ]
+
+# Path to a directory to read entries from. Defaults to system
+# path when empty.
+[path: ]
+```
+
+### relabel_config
+
+Relabeling is a powerful tool to dynamically rewrite the label set of a target
+before it gets scraped. Multiple relabeling steps can be configured per scrape
+configuration. They are applied to the label set of each target in order of
+their appearance in the configuration file.
+
+After relabeling, the `instance` label is set to the value of `__address__` by
+default if it was not set during relabeling. The `__scheme__` and
+`__metrics_path__` labels are set to the scheme and metrics path of the target
+respectively. The `__param_` label is set to the value of the first passed
+URL parameter called ``.
+
+Additional labels prefixed with `__meta_` may be available during the relabeling
+phase. They are set by the service discovery mechanism that provided the target
+and vary between mechanisms.
+
+Labels starting with `__` will be removed from the label set after target
+relabeling is completed.
+
+If a relabeling step needs to store a label value only temporarily (as the
+input to a subsequent relabeling step), use the `__tmp` label name prefix. This
+prefix is guaranteed to never be used by Prometheus itself.
+
+```yaml
+# The source labels select values from existing labels. Their content is concatenated
+# using the configured separator and matched against the configured regular expression
+# for the replace, keep, and drop actions.
+[ source_labels: '[' [, ...] ']' ]
+
+# Separator placed between concatenated source label values.
+[ separator: | default = ; ]
+
+# Label to which the resulting value is written in a replace action.
+# It is mandatory for replace actions. Regex capture groups are available.
+[ target_label: ]
+
+# Regular expression against which the extracted value is matched.
+[ regex: | default = (.*) ]
+
+# Modulus to take of the hash of the source label values.
+[ modulus: ]
+
+# Replacement value against which a regex replace is performed if the
+# regular expression matches. Regex capture groups are available.
+[ replacement: | default = $1 ]
+
+# Action to perform based on regex matching.
+[ action: | default = replace ]
+```
+
+`` is any valid
+[RE2 regular expression](https://github.com/google/re2/wiki/Syntax). It is
+required for the `replace`, `keep`, `drop`, `labelmap`,`labeldrop` and
+`labelkeep` actions. The regex is anchored on both ends. To un-anchor the regex,
+use `.*.*`.
+
+`` determines the relabeling action to take:
+
+* `replace`: Match `regex` against the concatenated `source_labels`. Then, set
+ `target_label` to `replacement`, with match group references
+ (`${1}`, `${2}`, ...) in `replacement` substituted by their value. If `regex`
+ does not match, no replacement takes place.
+* `keep`: Drop targets for which `regex` does not match the concatenated `source_labels`.
+* `drop`: Drop targets for which `regex` matches the concatenated `source_labels`.
+* `hashmod`: Set `target_label` to the `modulus` of a hash of the concatenated `source_labels`.
+* `labelmap`: Match `regex` against all label names. Then copy the values of the matching labels
+ to label names given by `replacement` with match group references
+ (`${1}`, `${2}`, ...) in `replacement` substituted by their value.
+* `labeldrop`: Match `regex` against all label names. Any label that matches will be
+ removed from the set of labels.
+* `labelkeep`: Match `regex` against all label names. Any label that does not match will be
+ removed from the set of labels.
+
+Care must be taken with `labeldrop` and `labelkeep` to ensure that logs are
+still uniquely labeled once the labels are removed.
+
+### static_config
+
+A `static_config` allows specifying a list of targets and a common label set
+for them. It is the canonical way to specify static targets in a scrape
+configuration.
+
+```yaml
+# Configures the discovery to look on the current machine. Must be either
+# localhost or the hostname of the current computer.
+targets:
+ - localhost
+
+# Defines a file to scrape and an optional set of additional labels to apply to
+# all streams defined by the files from __path__.
+labels:
+ # The path to load logs from. Can use glob patterns (e.g., /var/log/*.log).
+ __path__:
+
+ # Additional labels to assign to the logs
+ [ : ... ]
+```
+
+### file_sd_config
+
+File-based service discovery provides a more generic way to configure static
+targets and serves as an interface to plug in custom service discovery
+mechanisms.
+
+It reads a set of files containing a list of zero or more
+``s. Changes to all defined files are detected via disk watches
+and applied immediately. Files may be provided in YAML or JSON format. Only
+changes resulting in well-formed target groups are applied.
+
+The JSON file must contain a list of static configs, using this format:
+
+```yaml
+[
+ {
+ "targets": [ "localhost" ],
+ "labels": {
+ "__path__": "", ...
+ "": "", ...
+ }
+ },
+ ...
+]
+```
+
+As a fallback, the file contents are also re-read periodically at the specified
+refresh interval.
+
+Each target has a meta label `__meta_filepath` during the
+[relabeling phase](#relabel_config). Its value is set to the
+filepath from which the target was extracted.
+
+```yaml
+# Patterns for files from which target groups are extracted.
+files:
+ [ - ... ]
+
+# Refresh interval to re-read the files.
+[ refresh_interval: | default = 5m ]
+```
+
+Where `` may be a path ending in `.json`, `.yml` or `.yaml`.
+The last path segment may contain a single `*` that matches any character
+sequence, e.g. `my/path/tg_*.json`.
+
+### kubernetes_sd_config
+
+Kubernetes SD configurations allow retrieving scrape targets from
+[Kubernetes'](https://kubernetes.io/) REST API and always staying synchronized
+with the cluster state.
+
+One of the following `role` types can be configured to discover targets:
+
+#### `node`
+
+The `node` role discovers one target per cluster node with the address
+defaulting to the Kubelet's HTTP port.
+
+The target address defaults to the first existing address of the Kubernetes
+node object in the address type order of `NodeInternalIP`, `NodeExternalIP`,
+`NodeLegacyHostIP`, and `NodeHostName`.
+
+Available meta labels:
+
+* `__meta_kubernetes_node_name`: The name of the node object.
+* `__meta_kubernetes_node_label_`: Each label from the node object.
+* `__meta_kubernetes_node_labelpresent_`: `true` for each label from the node object.
+* `__meta_kubernetes_node_annotation_`: Each annotation from the node object.
+* `__meta_kubernetes_node_annotationpresent_`: `true` for each annotation from the node object.
+* `__meta_kubernetes_node_address_`: The first address for each node address type, if it exists.
+
+In addition, the `instance` label for the node will be set to the node name
+as retrieved from the API server.
+
+#### `service`
+
+The `service` role discovers a target for each service port of each service.
+This is generally useful for blackbox monitoring of a service.
+The address will be set to the Kubernetes DNS name of the service and respective
+service port.
+
+Available meta labels:
+
+* `__meta_kubernetes_namespace`: The namespace of the service object.
+* `__meta_kubernetes_service_annotation_`: Each annotation from the service object.
+* `__meta_kubernetes_service_annotationpresent_`: "true" for each annotation of the service object.
+* `__meta_kubernetes_service_cluster_ip`: The cluster IP address of the service. (Does not apply to services of type ExternalName)
+* `__meta_kubernetes_service_external_name`: The DNS name of the service. (Applies to services of type ExternalName)
+* `__meta_kubernetes_service_label_`: Each label from the service object.
+* `__meta_kubernetes_service_labelpresent_`: `true` for each label of the service object.
+* `__meta_kubernetes_service_name`: The name of the service object.
+* `__meta_kubernetes_service_port_name`: Name of the service port for the target.
+* `__meta_kubernetes_service_port_protocol`: Protocol of the service port for the target.
+
+#### `pod`
+
+The `pod` role discovers all pods and exposes their containers as targets. For
+each declared port of a container, a single target is generated. If a container
+has no specified ports, a port-free target per container is created for manually
+adding a port via relabeling.
+
+Available meta labels:
+
+* `__meta_kubernetes_namespace`: The namespace of the pod object.
+* `__meta_kubernetes_pod_name`: The name of the pod object.
+* `__meta_kubernetes_pod_ip`: The pod IP of the pod object.
+* `__meta_kubernetes_pod_label_`: Each label from the pod object.
+* `__meta_kubernetes_pod_labelpresent_`: `true`for each label from the pod object.
+* `__meta_kubernetes_pod_annotation_`: Each annotation from the pod object.
+* `__meta_kubernetes_pod_annotationpresent_`: `true` for each annotation from the pod object.
+* `__meta_kubernetes_pod_container_init`: `true` if the container is an [InitContainer](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/)
+* `__meta_kubernetes_pod_container_name`: Name of the container the target address points to.
+* `__meta_kubernetes_pod_container_port_name`: Name of the container port.
+* `__meta_kubernetes_pod_container_port_number`: Number of the container port.
+* `__meta_kubernetes_pod_container_port_protocol`: Protocol of the container port.
+* `__meta_kubernetes_pod_ready`: Set to `true` or `false` for the pod's ready state.
+* `__meta_kubernetes_pod_phase`: Set to `Pending`, `Running`, `Succeeded`, `Failed` or `Unknown`
+ in the [lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase).
+* `__meta_kubernetes_pod_node_name`: The name of the node the pod is scheduled onto.
+* `__meta_kubernetes_pod_host_ip`: The current host IP of the pod object.
+* `__meta_kubernetes_pod_uid`: The UID of the pod object.
+* `__meta_kubernetes_pod_controller_kind`: Object kind of the pod controller.
+* `__meta_kubernetes_pod_controller_name`: Name of the pod controller.
+
+#### `endpoints`
+
+The `endpoints` role discovers targets from listed endpoints of a service. For
+each endpoint address one target is discovered per port. If the endpoint is
+backed by a pod, all additional container ports of the pod, not bound to an
+endpoint port, are discovered as targets as well.
+
+Available meta labels:
+
+* `__meta_kubernetes_namespace`: The namespace of the endpoints object.
+* `__meta_kubernetes_endpoints_name`: The names of the endpoints object.
+* For all targets discovered directly from the endpoints list (those not additionally inferred
+ from underlying pods), the following labels are attached:
+ * `__meta_kubernetes_endpoint_hostname`: Hostname of the endpoint.
+ * `__meta_kubernetes_endpoint_node_name`: Name of the node hosting the endpoint.
+ * `__meta_kubernetes_endpoint_ready`: Set to `true` or `false` for the endpoint's ready state.
+ * `__meta_kubernetes_endpoint_port_name`: Name of the endpoint port.
+ * `__meta_kubernetes_endpoint_port_protocol`: Protocol of the endpoint port.
+ * `__meta_kubernetes_endpoint_address_target_kind`: Kind of the endpoint address target.
+ * `__meta_kubernetes_endpoint_address_target_name`: Name of the endpoint address target.
+* If the endpoints belong to a service, all labels of the `role: service` discovery are attached.
+* For all targets backed by a pod, all labels of the `role: pod` discovery are attached.
+
+#### `ingress`
+
+The `ingress` role discovers a target for each path of each ingress.
+This is generally useful for blackbox monitoring of an ingress.
+The address will be set to the host specified in the ingress spec.
+
+Available meta labels:
+
+* `__meta_kubernetes_namespace`: The namespace of the ingress object.
+* `__meta_kubernetes_ingress_name`: The name of the ingress object.
+* `__meta_kubernetes_ingress_label_`: Each label from the ingress object.
+* `__meta_kubernetes_ingress_labelpresent_`: `true` for each label from the ingress object.
+* `__meta_kubernetes_ingress_annotation_`: Each annotation from the ingress object.
+* `__meta_kubernetes_ingress_annotationpresent_`: `true` for each annotation from the ingress object.
+* `__meta_kubernetes_ingress_scheme`: Protocol scheme of ingress, `https` if TLS
+ config is set. Defaults to `http`.
+* `__meta_kubernetes_ingress_path`: Path from ingress spec. Defaults to `/`.
+
+See below for the configuration options for Kubernetes discovery:
+
+```yaml
+# The information to access the Kubernetes API.
+
+# The API server addresses. If left empty, Prometheus is assumed to run inside
+# of the cluster and will discover API servers automatically and use the pod's
+# CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/.
+[ api_server: ]
+
+# The Kubernetes role of entities that should be discovered.
+role:
+
+# Optional authentication information used to authenticate to the API server.
+# Note that `basic_auth`, `bearer_token` and `bearer_token_file` options are
+# mutually exclusive.
+# password and password_file are mutually exclusive.
+
+# Optional HTTP basic authentication information.
+basic_auth:
+ [ username: ]
+ [ password: ]
+ [ password_file: ]
+
+# Optional bearer token authentication information.
+[ bearer_token: ]
+
+# Optional bearer token file authentication information.
+[ bearer_token_file: ]
+
+# Optional proxy URL.
+[ proxy_url: ]
+
+# TLS configuration.
+tls_config:
+ [ ]
+
+# Optional namespace discovery. If omitted, all namespaces are used.
+namespaces:
+ names:
+ [ - ]
+```
+
+Where `` must be `endpoints`, `service`, `pod`, `node`, or
+`ingress`.
+
+See
+[this example Prometheus configuration file](/documentation/examples/prometheus-kubernetes.yml)
+for a detailed example of configuring Prometheus for Kubernetes.
+
+You may wish to check out the 3rd party
+[Prometheus Operator](https://github.com/coreos/prometheus-operator),
+which automates the Prometheus setup on top of Kubernetes.
+
+## target_config
+
+The `target_config` block controls the behavior of reading files from discovered
+targets.
+
+```yaml
+# Period to resync directories being watched and files being tailed to discover
+# new ones or stop watching removed ones.
+sync_period: "10s"
+```
+
+## Example Docker Config
+
+```yaml
+server:
+ http_listen_port: 9080
+ grpc_listen_port: 0
+
+positions:
+ filename: /tmp/positions.yaml
+
+client:
+ url: http://ip_or_hostname_where_Loki_run:3100/loki/api/v1/push
+
+scrape_configs:
+ - job_name: system
+ pipeline_stages:
+ - docker:
+ static_configs:
+ - targets:
+ - localhost
+ labels:
+ job: varlogs
+ host: yourhost
+ __path__: /var/log/*.log
+
+ - job_name: someone_service
+ pipeline_stages:
+ - docker:
+ static_configs:
+ - targets:
+ - localhost
+ labels:
+ job: someone_service
+ host: yourhost
+ __path__: /srv/log/someone_service/*.log
+```
+
+## Example Journal Config
+
+```yaml
+server:
+ http_listen_port: 9080
+ grpc_listen_port: 0
+
+positions:
+ filename: /tmp/positions.yaml
+
+clients:
+ - url: http://ip_or_hostname_where_loki_runns:3100/loki/api/v1/push
+
+scrape_configs:
+ - job_name: journal
+ journal:
+ max_age: 12h
+ path: /var/log/journal
+ labels:
+ job: systemd-journal
+ relabel_configs:
+ - source_labels: ['__journal__systemd_unit']
+ target_label: 'unit'
+```
diff --git a/docs/promtail/deployment.md b/docs/clients/promtail/installation.md
similarity index 63%
rename from docs/promtail/deployment.md
rename to docs/clients/promtail/installation.md
index 4a95805edf..9e00e284e8 100644
--- a/docs/promtail/deployment.md
+++ b/docs/clients/promtail/installation.md
@@ -1,41 +1,63 @@
-# Installation
-Promtail is distributed in binary and in container form.
+# Installing Promtail
-Once it is installed, you have basically two options for operating it:
-Either as a daemon sitting on every node, or as a sidecar for the application.
+Promtail is distributed as a [binary](#binary), [Docker container](#docker), and
+[Helm chart](#helm).
-This usually only depends on the configuration though.
## Binary
-Every release includes binaries:
+
+Every Loki release includes binaries for Promtail:
```bash
-# download a binary (adapt app, os and arch as needed)
-# installs v0.2.0. Go to the releases page for up to date URLs
-$ curl -fSL -o "/usr/local/bin/promtail.gz" "https://github.com/grafana/promtail/releases/download/v0.2.0/promtail-linux-amd64.gz"
+# download a binary (modify app, os, and arch as needed)
+# Installs v0.3.0. Go to the releases page for the latest version
+$ curl -fSL -o "/usr/local/bin/promtail.gz" "https://github.com/grafana/loki/releases/download/v0.3.0/promtail_linux_amd64.gz"
$ gunzip "/usr/local/bin/promtail.gz"
# make sure it is executable
$ chmod a+x "/usr/local/bin/promtail"
```
-Binaries for macOS and Windows are also provided at the [releases page](https://github.com/grafana/loki/releases).
+Binaries for macOS and Windows are also provided at the
+[Releases page](https://github.com/grafana/loki/releases).
## Docker
+
+```bash
+# modify tag to most recent version
+$ docker pull grafana/promtail:v0.3.0
+```
+
+## Helm
+
+Make sure that Helm is
+[installed](https://helm.sh/docs/using_helm/#installing-helm) and
+[deployed](https://helm.sh/docs/using_helm/#installing-tiller) to your cluster.
+Then you can add Loki's chart repository to Helm:
+
+```bash
+$ helm repo add loki https://grafana.github.io/loki/charts
+```
+
+And the chart repository can be updated by running:
+
```bash
-# adapt tag to most recent version
-$ docker pull grafana/promtail:v0.2.0
+$ helm repo update
+```
+
+Finally, Promtail can be deployed with:
+
+```bash
+$ helm upgrade --install promtail loki/promtail --set "loki.serviceName=loki"
```
## Kubernetes
-On Kubernetes, you will use the Docker container above. However, you have too
-choose whether you want to run in daemon mode (`DaemonSet`) or sidecar mode
-(`Pod container`) in before.
-### Daemonset method (Recommended)
-A `DaemonSet` will deploy `promtail` on every node within the Kubernetes cluster.
+### DaemonSet (Recommended)
+
+A `DaemonSet` will deploy `promtail` on every node within a Kubernetes cluster.
-This deployment is great to collect the logs of all containers within the
-cluster. It is the best solution for a single tenant.
+The DaemonSet deployment is great to collect the logs of all containers within a
+cluster. It's the best solution for a single-tenant model.
```yaml
---Daemonset.yaml
@@ -111,10 +133,11 @@ roleRef:
apiGroup: rbac.authorization.k8s.io
```
-### Sidecar Method
-This method will deploy `promtail` as a sidecar container within a pod.
-In a multi-tenant environment, this enables teams to aggregate logs
-for specific pods and deployments for example for all pods in a namespace.
+### Sidecar
+
+The Sidecar method deploys `promtail` as a sidecar container for a specific pod.
+In a multi-tenant environment, this enables teams to aggregate logs for specific
+pods and deployments.
```yaml
---Deployment.yaml
@@ -146,5 +169,4 @@ spec:
mountPath: /etc/promtail
...
...
-
```
diff --git a/docs/clients/promtail/pipelines.md b/docs/clients/promtail/pipelines.md
new file mode 100644
index 0000000000..d46da9c647
--- /dev/null
+++ b/docs/clients/promtail/pipelines.md
@@ -0,0 +1,208 @@
+# Pipelines
+
+A detailed look at how to setup Promtail to process your log lines, including
+extracting metrics and labels.
+
+## Pipeline
+
+A pipeline is used to transform a single log line, its labels, and its
+timestamp. A pipeline is comprised of a set of **stages**. There are 4 types of
+stages:
+
+1. **Parsing stages** parse the current log line and extract data out of it. The
+ extracted data is then available for use by other stages.
+2. **Transform stages** transform extracted data from previous stages.
+3. **Action stages** take extracted data from previous stages and do something
+ with them. Actions can:
+ 1. Add or modify existing labels to the log line
+ 2. Change the timestamp of the log line
+ 3. Change the content of the log line
+ 4. Create a metric based on the extracted data
+4. **Filtering stages** optionally apply a subset of stages based on some
+ condition.
+
+Typical pipelines will start with a parsing stage (such as a
+[regex](./stages/regex.md) or [json](./stages/json.md) stage) to extract data
+from the log line. Then, a series of action stages will be present to do
+something with that extract data. The most common action stage will be a
+[labels](./stages/labels.md) stage to turn extracted data into a label.
+
+A common stage will also be the [match](./stages/match.md) stage to selectively
+apply stages based on the current labels.
+
+Note that pipelines can not currently be used to deduplicate logs; Loki will
+receive the same log line multiple times if, for example:
+
+1. Two scrape configs read from the same file
+2. Duplicate log lines in a file are sent through a pipeline. Deduplication is
+ not done.
+
+However, Loki will perform some deduplication at query time for logs that have
+the exact same nanosecond timestamp, labels, and log contents.
+
+This documented example gives a good glimpse of what you can achieve with a
+pipeline:
+
+```yaml
+scrape_configs:
+- job_name: kubernetes-pods-name
+ kubernetes_sd_configs: ....
+ pipeline_stages:
+
+ # This stage is only going to run if the scraped target has a label
+ # of "name" with value "promtail".
+ - match:
+ selector: '{name="promtail"}'
+ stages:
+ # The regex stage parses out a level, timestamp, and component. At the end
+ # of the stage, the values for level, timestamp, and component are only
+ # set internally for the pipeline. Future stages can use these values and
+ # decide what to do with them.
+ - regex:
+ expression: '.*level=(?P[a-zA-Z]+).*ts=(?P[T\d-:.Z]*).*component=(?P[a-zA-Z]+)'
+
+ # The labels stage takes the level and component entries from the previous
+ # regex stage and promotes them to a label. For example, level=error may
+ # be a label added by this stage.
+ - labels:
+ level:
+ component:
+
+ # Finally, the timestamp stage takes the timestamp extracted from the
+ # regex stage and promotes it to be the new timestamp of the log entry,
+ # parsing it as an RFC3339Nano-formatted value.
+ - timestamp:
+ format: RFC3339Nano
+ source: timestamp
+
+ # This stage is only going to run if the scraped target has a label of
+ # "name" with a value of "nginx".
+ - match:
+ selector: '{name="nginx"}'
+ stages:
+ # This regex stage extracts a new output by matching against some
+ # values and capturing the rest.
+ - regex:
+ expression: \w{1,3}.\w{1,3}.\w{1,3}.\w{1,3}(?P.*)
+
+ # The output stage changes the content of the captured log line by
+ # setting it to the value of output from the regex stage.
+ - output:
+ source: output
+
+ # This stage is only going to run if the scraped target has a label of
+ # "name" with a value of "jaeger-agent".
+ - match:
+ selector: '{name="jaeger-agent"}'
+ stages:
+ # The JSON stage reads the log line as a JSON string and extracts
+ # the "level" field from the object for use in further stages.
+ - json:
+ expressions:
+ level: level
+
+ # The labels stage pulls the value from "level" that was extracted
+ # from the previous stage and promotes it to a label.
+ - labels:
+ level:
+- job_name: kubernetes-pods-app
+ kubernetes_sd_configs: ....
+ pipeline_stages:
+ # This stage will only run if the scraped target has a label of "app"
+ # with a name of *either* grafana or prometheus.
+ - match:
+ selector: '{app=~"grafana|prometheus"}'
+ stages:
+ # The regex stage will extract a level and component for use in further
+ # stages, allowing the level to be defined as either lvl= or
+ # level= and the component to be defined as either
+ # logger= or component=
+ - regex:
+ expression: ".*(lvl|level)=(?P[a-zA-Z]+).*(logger|component)=(?P[a-zA-Z]+)"
+
+ # The labels stage then promotes the level and component extracted from
+ # the regex stage to labels.
+ - labels:
+ level:
+ component:
+
+ # This stage will only run if the scraped target has a label of "app"
+ # and a value of "some-app".
+ - match:
+ selector: '{app="some-app"}'
+ stages:
+ # The regex stage tries to extract a Go panic by looking for panic:
+ # in the log message.
+ - regex:
+ expression: ".*(?Ppanic: .*)"
+
+ # The metrics stage is going to increment a panic_total metric counter
+ # which promtail exposes. The counter is only incremented when panic
+ # was extracted from the regex stage.
+ - metrics:
+ - panic_total:
+ type: Counter
+ description: "total count of panic"
+ source: panic
+ config:
+ action: inc
+```
+
+### Data Accessible to Stages
+
+The following sections further describe the types that are accessible to each
+stage (although not all may be used):
+
+##### Label Set
+
+The current set of labels for the log line. Initialized to be the set of labels
+that were scraped along with the log line. The label set is only modified by an
+action stage, but filtering stages read from it.
+
+The final label set will be index by Loki and can be used for queries.
+
+##### Extracted Map
+
+A collection of key-value pairs extracted during a parsing stage. Subsequent
+stages operate on the extracted map, either transforming them or taking action
+with them. At the end of a pipeline, the extracted map is discarded; for a
+parsing stage to be useful, it must always be paired with at least one action
+stage.
+
+##### Log Timestamp
+
+The current timestamp for the log line. Action stages can modify this value.
+If left unset, it defaults to the time when the log was scraped.
+
+The final value for the timestamp is sent to Loki.
+
+##### Log Line
+
+The current log line, represented as text. Initialized to be the text that
+promtail scraped. Action stages can modify this value.
+
+The final value for the log line is sent to Loki as the text content for the
+given log entry.
+
+## Stages
+
+Parsing stages:
+
+ * [regex](./stages/regex.md): Extract data using a regular expression.
+ * [json](./stages/json.md): Extract data by parsing the log line as JSON.
+
+Transform stages:
+
+ * [template](./stages/template.md): Use Go templates to modify extracted data.
+
+Action stages:
+
+ * [timestamp](./stages/timestamp.md): Set the timestamp value for the log entry.
+ * [output](./stages/output.md): Set the log line text.
+ * [labels](./stages/labels.md): Update the label set for the log entry.
+ * [metrics](./stages/metrics.md): Calculate metrics based on extracted data.
+
+Filtering stages:
+
+ * [match](./stages/match.md): Conditionally run stages based on the label set.
+
diff --git a/docs/clients/promtail/scraping.md b/docs/clients/promtail/scraping.md
new file mode 100644
index 0000000000..cc1d4d23d7
--- /dev/null
+++ b/docs/clients/promtail/scraping.md
@@ -0,0 +1,143 @@
+# Promtail Scraping (Service Discovery)
+
+## File Target Discovery
+
+Promtail discovers locations of log files and extract labels from them through
+the `scrape_configs` section in the config YAML. The syntax is identical to what
+[Prometheus uses](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config).
+
+`scrape_configs` contains one or more entries which are executed for each
+discovered target (i.e., each container in each new pod running in the
+instance):
+
+```
+scrape_configs:
+ - job_name: local
+ static_configs:
+ - ...
+
+ - job_name: kubernetes
+ kubernetes_sd_config:
+ - ...
+```
+
+If more than one scrape config section matches your logs, you will get duplicate
+entries as the logs are sent in different streams likely with slightly
+different labels.
+
+There are different types of labels present in Promtail:
+
+* Labels starting with `__` (two underscores) are internal labels. They usually
+ come from dynamic sources like service discovery. Once relabeling is done,
+ they are removed from the label set. To persist internal labels so they're
+ sent to Loki, rename them so they don't start with `__`. See
+ [Relabeling](#relabeling) for more information.
+
+* Labels starting with `__meta_kubernetes_pod_label_*` are "meta labels" which
+ are generated based on your Kubernetes pod's labels.
+
+ For example, if your Kubernetes pod has a label `name` set to `foobar`, then
+ the `scrape_configs` section will receive an internal label
+ `__meta_kubernetes_pod_label_name` with a value set to `foobar`.
+
+* Other labels starting with `__meta_kubernetes_*` exist based on other
+ Kubernetes metadata, such as the namespace of the pod
+ (`__meta_kubernetes_namespace`) or the name of the container inside the pod
+ (`__meta_kubernetes_pod_container_name`). Refer to
+ [the Prometheus docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config)
+ for the full list of Kubernetes meta labels.
+
+* The `__path__` label is a special label which Promtail uses after discovery to
+ figure out where the file to read is located. Wildcards are allowed.
+
+* The label `filename` is added for every file found in `__path__` to ensure the
+ uniqueness of the streams. It is set to the absolute path of the file the line
+ was read from.
+
+### Kubernetes Discovery
+
+Note that while Promtail can utilize the Kubernetes API to discover pods as
+targets, it can only read log files from pods that are running on the same node
+as the one Promtail is running on. Promtail looks for a `__host__` label on
+each target and validates that it is set to the same hostname as Promtail's
+(using either `$HOSTNAME` or the hostname reported by the kernel if the
+environment variable is not set).
+
+This means that any time Kubernetes service discovery is used, there must be a
+`relabel_config` that creates the intermediate label `__host__` from
+`__meta_kubernetes_pod_node_name`:
+
+```yaml
+relabel_configs:
+ - source_labels: ['__meta_kubernetes_pod_node_name']
+ target_label: '__host__'
+```
+
+See [Relabeling](#relabeling) for more information.
+
+## Relabeling
+
+Each `scrape_configs` entry can contain a `relabel_configs` stanza.
+`relabel_configs` is a list of operations to transform the labels from discovery
+into another form.
+
+A single entry in `relabel_configs` can also reject targets by doing an `action:
+drop` if a label value matches a specified regex. When a target is dropped, the
+owning `scrape_config` will not process logs from that particular source.
+Other `scrape_configs` without the drop action reading from the same target
+may still use and forward logs from it to Loki.
+
+A common use case of `relabel_configs` is to transform an internal label such
+as `__meta_kubernetes_*` into an intermediate internal label such as
+`__service__`. The intermediate internal label may then be dropped based on
+value or transformed to a final external label, such as `__job__`.
+
+### Examples
+
+* Drop the target if a label (`__service__` in the example) is empty:
+```yaml
+ - action: drop
+ regex: ^$
+ source_labels:
+ - __service__
+```
+* Drop the target if any of the `source_labels` contain a value:
+```yaml
+ - action: drop
+ regex: .+
+ separator: ''
+ source_labels:
+ - __meta_kubernetes_pod_label_name
+ - __meta_kubernetes_pod_label_app
+```
+* Persist an internal label by renaming it so it will be sent to Loki:
+```yaml
+ - action: replace
+ source_labels:
+ - __meta_kubernetes_namespace
+ target_label: namespace
+```
+* Persist all Kubernetes pod labels by mapping them, like by mapping
+ `__meta_kube__meta_kubernetes_pod_label_foo` to `foo`.
+```yaml
+ - action: labelmap
+ regex: __meta_kubernetes_pod_label_(.+)
+```
+
+Additional reading:
+
+ * [Julien Pivotto's slides from PromConf Munich, 2017](https://www.slideshare.net/roidelapluie/taking-advantage-of-prometheus-relabeling-109483749)
+
+## HTTP client options
+
+Promtail uses the Prometheus HTTP client implementation for all calls to Loki.
+Therefore it can be configured using the `clients` stanza, where one or more
+connections to Loki can be established:
+
+```yaml
+clients:
+ - [ ]
+```
+
+Refer to [`client_config`](./configuration.md#client_config) from the Promtail
+Configuration reference for all available options.
diff --git a/docs/clients/promtail/stages/README.md b/docs/clients/promtail/stages/README.md
new file mode 100644
index 0000000000..a1dda74de1
--- /dev/null
+++ b/docs/clients/promtail/stages/README.md
@@ -0,0 +1,25 @@
+# Stages
+
+This section is a collection of all stages Promtail supports in a
+[Pipeline](../ppipelines.md).
+
+Parsing stages:
+
+ * [regex](./regex.md): Extract data using a regular expression.
+ * [json](./json.md): Extract data by parsing the log line as JSON.
+
+Transform stages:
+
+ * [template](./template.md): Use Go templates to modify extracted data.
+
+Action stages:
+
+ * [timestamp](./timestamp.md): Set the timestamp value for the log entry.
+ * [output](./output.md): Set the log line text.
+ * [labels](./labels.md): Update the label set for the log entry.
+ * [metrics](./metrics.md): Calculate metrics based on extracted data.
+
+Filtering stages:
+
+ * [match](./match.md): Conditionally run stages based on the label set.
+
diff --git a/docs/clients/promtail/stages/json.md b/docs/clients/promtail/stages/json.md
new file mode 100644
index 0000000000..2f115ab25d
--- /dev/null
+++ b/docs/clients/promtail/stages/json.md
@@ -0,0 +1,91 @@
+# `json` stage
+
+The `json` stage is a parsing stage that reads the log line as JSON and accepts
+[JMESPath](http://jmespath.org/) expressions to extract data.
+
+## Schema
+
+```yaml
+json:
+ # Set of key/value pairs of JMESPath expressions. The key will be
+ # the key in the extracted data while the expression will the value,
+ # evaluated as a JMESPath from the source data.
+ expressions:
+ [ : ... ]
+
+ # Name from extracted data to parse. If empty, uses the log message.
+ [source: ]
+```
+
+This stage uses the Go JSON unmarshaler, which means non-string types like
+numbers or booleans will be unmarshaled into those types. The extracted data
+can hold non-string values and this stage does not do any type conversions;
+downstream stages will need to perform correct type conversion of these values
+as necessary. Please refer to the [the `template` stage](./template.md) for how
+to do this.
+
+If the value extracted is a complex type, such as an array or a JSON object, it
+will be converted back into a JSON string before being inserted into the
+extracted data.
+
+## Examples
+
+### Using log line
+
+For the given pipeline:
+
+```yaml
+- json:
+ expressions:
+ output: log
+ stream: stream
+ timestamp: time
+```
+
+Given the following log line:
+
+```
+{"log":"log message\n","stream":"stderr","time":"2019-04-30T02:12:41.8443515Z"}
+```
+
+The following key-value pairs would be created in the set of extracted data:
+
+- `output`: `log message\n`
+- `stream`: `stderr`
+- `timestamp`: `2019-04-30T02:12:41.8443515`
+
+### Using extracted data
+
+For the given pipeline:
+
+```yaml
+- json:
+ expressions:
+ output: log
+ stream: stream
+ timestamp: time
+ extra:
+- json:
+ expressions:
+ user:
+ source: extra
+```
+
+And the given log line:
+
+```
+{"log":"log message\n","stream":"stderr","time":"2019-04-30T02:12:41.8443515Z","extra":"{\"user\":\"marco\"}"}
+```
+
+The first stage would create the following key-value pairs in the set of
+extracted data:
+
+- `output`: `log message\n`
+- `stream`: `stderr`
+- `timestamp`: `2019-04-30T02:12:41.8443515`
+- `extra`: `{"user": "marco"}`
+
+The second stage will parse the value of `extra` from the extracted data as JSON
+and append the following key-value pairs to the set of extracted data:
+
+- `user`: `marco`
diff --git a/docs/clients/promtail/stages/labels.md b/docs/clients/promtail/stages/labels.md
new file mode 100644
index 0000000000..171d818019
--- /dev/null
+++ b/docs/clients/promtail/stages/labels.md
@@ -0,0 +1,37 @@
+# `labels` stage
+
+The labels stage is an action stage that takes data from the extracted map and
+modifies the label set that is sent to Loki with the log entry.
+
+## Schema
+
+```yaml
+labels:
+ # Key is REQUIRED and the name for the label that will be created.
+ # Value is optional and will be the name from extracted data whose value
+ # will be used for the value of the label. If empty, the value will be
+ # inferred to be the same as the key.
+ [ : [] ... ]
+```
+
+### Examples
+
+For the given pipeline:
+
+```yaml
+- json:
+ expressions:
+ stream: stream
+- labels:
+ stream:
+```
+
+Given the following log line:
+
+```
+{"log":"log message\n","stream":"stderr","time":"2019-04-30T02:12:41.8443515Z"}
+```
+
+The first stage would extract `stream` into the extracted map with a value of
+`stderr`. The labels stage would turn that key-value pair into a label, so the
+log line sent to Loki would include the label `stream` with a value of `stderr`.
diff --git a/docs/clients/promtail/stages/match.md b/docs/clients/promtail/stages/match.md
new file mode 100644
index 0000000000..5a931b01c2
--- /dev/null
+++ b/docs/clients/promtail/stages/match.md
@@ -0,0 +1,85 @@
+# `match` stage
+
+The match stage is a filtering stage that conditionally applies a set of stages
+when a log entry matches a configurable [LogQL](../../../logql.md) stream
+selector.
+
+## Schema
+
+```yaml
+match:
+ # LogQL stream selector.
+ selector:
+
+ # Names the pipeline. When defined, creates an additional label in
+ # the pipeline_duration_seconds histogram, where the value is
+ # concatenated with job_name using an underscore.
+ [pipieline_name: ]
+
+ # Nested set of pipeline stages only if the selector
+ # matches the labels of the log entries:
+ stages:
+ - [
+
+ |
+ |
+ |
+ |
+ |
+ |
+
+ ]
+```
+
+Refer to the [Promtail Configuration Reference](../configuration.md) for the
+schema on the various other stages referenced here.
+
+### Example
+
+For the given pipeline:
+
+```yaml
+pipeline_stages:
+- json:
+ expressions:
+ app:
+- labels:
+ app:
+- match:
+ selector: "{app=\"loki\"}"
+ stages:
+ - json:
+ expressions:
+ msg: message
+- match:
+ pipeline_name: "app2"
+ selector: "{app=\"pokey\"}"
+ stages:
+ - json:
+ expressions:
+ msg: msg
+- output:
+ source: msg
+```
+
+And the given log line:
+
+```
+{ "time":"2012-11-01T22:08:41+00:00", "app":"loki", "component": ["parser","type"], "level" : "WARN", "message" : "app1 log line" }
+```
+
+The first stage will add `app` with a value of `loki` into the extracted map,
+while the second stage will add `app` as a label (again with the value of `loki`).
+
+The third stage uses LogQL to only execute the nested stages when there is a
+label of `app` whose value is `loki`. This matches in our case; the nested
+`json` stage then adds `msg` into the extracted map with a value of `app1 log
+line`.
+
+The fourth stage uses LogQL to only executed the nested stages when there is a
+label of `app` whose value is `pokey`. This does **not** match in our case, so
+the nested `json` stage is not ran.
+
+The final `output` stage changes the contents of the log line to be the value of
+`msg` from the extracted map. In this case, the log line is changed to `app1 log
+line`.
diff --git a/docs/clients/promtail/stages/metrics.md b/docs/clients/promtail/stages/metrics.md
new file mode 100644
index 0000000000..b1302c9103
--- /dev/null
+++ b/docs/clients/promtail/stages/metrics.md
@@ -0,0 +1,215 @@
+# `metrics` stage
+
+The `metrics` stage is an action stage that allows for defining and updating
+metrics based on data from the extracted map. Note that created metrics are not
+pushed to Loki and are instead exposed via Promtail's `/metrics` endpoint.
+Prometheus should be configured to scrape Promtail to be able to retrieve the
+metrics configured by this stage.
+
+## Schema
+
+```yaml
+# A map where the key is the name of the metric and the value is a specific
+# metric type.
+metrics:
+ [: [ | | ] ...]
+```
+
+### metric_counter
+
+Defines a counter metric whose value only goes up.
+
+```yaml
+# The metric type. Must be Counter.
+type: Counter
+
+# Describes the metric.
+[description: ]
+
+# Key from the extracted data map to use for the mtric,
+# defaulting to the metric's name if not present.
+[source: ]
+
+config:
+ # Filters down source data and only changes the metric
+ # if the targeted value exactly matches the provided string.
+ # If not present, all data will match.
+ [value: ]
+
+ # Must be either "inc" or "add" (case insensitive). If
+ # inc is chosen, the metric value will increase by 1 for each
+ # log line receieved that passed the filter. If add is chosen,
+ # the extracted value most be convertible to a positive float
+ # and its value will be added to the metric.
+ action:
+```
+
+### metric_gauge
+
+Defines a gauge metric whose value can go up or down.
+
+```yaml
+# The metric type. Must be Gauge.
+type: Gauge
+
+# Describes the metric.
+[description: ]
+
+# Key from the extracted data map to use for the mtric,
+# defaulting to the metric's name if not present.
+[source: ]
+
+config:
+ # Filters down source data and only changes the metric
+ # if the targeted value exactly matches the provided string.
+ # If not present, all data will match.
+ [value: ]
+
+ # Must be either "set", "inc", "dec"," add", or "sub". If
+ # add, set, or sub is chosen, the extracted value must be
+ # convertible to a positive float. inc and dec will increment
+ # or decrement the metric's value by 1 respectively.
+ action: 