- [Operations](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/operations/_index.md) for important aspects of running Loki.
- [Promtail](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/clients/promtail/_index.md) is an agent which can tail your log files and push them to Loki.
- [Pipelines](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/clients/promtail/pipelines.md) for detailed log processing pipeline documentation
- [Docker Logging Driver](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/clients/docker-driver/_index.md) is a docker plugin to send logs directly to Loki from Docker containers.
- [LogCLI](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/getting-started/logcli.md) on how to query your logs without Grafana.
- [Loki Canary](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/operations/loki-canary.md) for monitoring your Loki installation for missing logs.
- [Troubleshooting](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/getting-started/troubleshooting.md) for help around frequent error messages.
- [Loki in Grafana](https://github.com/grafana/loki/tree/v1.6.0/docs/sources/getting-started/grafana.md) for how to set up a Loki datasource in Grafana and query your logs.
Commonly used sections:
- [API documentation](https://grafana.com/docs/loki/latest/api/) for alternative ways of getting logs into Loki.
- [Operations](https://grafana.com/docs/loki/latest/operations/) for important aspects of running Loki.
- [Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) is an agent which can tail your log files and push them to Loki.
- [Pipelines](https://grafana.com/docs/loki/latest/clients/promtail/pipelines/) for detailed log processing pipeline documentation
- [Docker Logging Driver](https://grafana.com/docs/loki/latest/clients/docker-driver/) is a docker plugin to send logs directly to Loki from Docker containers.
- [LogCLI](https://grafana.com/docs/loki/latest/getting-started/logcli/) on how to query your logs without Grafana.
- [Loki Canary](https://grafana.com/docs/loki/latest/operations/loki-canary/) for monitoring your Loki installation for missing logs.
- [Troubleshooting](https://grafana.com/docs/loki/latest/getting-started/troubleshooting/) for help around frequent error messages.
- [Loki in Grafana](https://grafana.com/docs/loki/latest/getting-started/grafana/) for how to set up a Loki datasource in Grafana and query your logs.
When you contribute to documentation, it is a good practice to build the docs on your local machine to make sure your changes appear as you expect. This README explains the process for doing that.
#### Requirements
Docker >= 2.1.0.3
### Build the doc site
1. In the command line, make sure you are in the docs folder: `cd docs`.
2. Run `make docs`. This launches a preview of the docs website at `http://localhost:3002/docs/loki/latest/` which will refresh automatically when changes to content in the `sources` directory are made.
---
### Content guidelines
Edit content in the `sources` directory.
### Edit the side menu
The side menu is built automatically from the folder structure. Ordering is done with the `weight` front matter param.
### Add images
Images are sourced in this repo alonside content. They will sync to the website repo just like markdown files.
---
## Deploy changes to grafana.com
When a PR is merged to master with changes in the `docs/sources` directory, those changes are automatically synched to the grafana/website repo and published to the staging site.
Generally, someone from marketing will publish to production each day, so as long as the sync is successful your docs edits will be published.
Please see the [Documentation Site](https://grafana.com/docs/loki/latest/) The files in this directory are used to generate it but consequently the links don't work in Github.
@ -121,4 +121,4 @@ Starting in version 1.6.0 Loki and Promtail have flags which will dump the entir
`-print-config-stderr` is nice when running loki directly e.g. `./loki ` as you can get a quick output of the entire Loki config.
`-log-config-reverse-order` is the flag we run Loki with in all our environments, the config entries are reversed so that the order of configs reads correctly top to bottom when viewed in Grafana's Explore.
`-log-config-reverse-order` is the flag we run Loki with in all our environments, the config entries are reversed so that the order of configs reads correctly top to bottom when viewed in Grafana's Explore.
@ -265,7 +265,7 @@ That's it, save the config and you can `reboot` the machine (or simply restart t
Let's head back to Grafana and verify that your Promtail logs are available in Grafana by using the [LogQL](../../../logql/) query `{unit="promtail.service"}` in Explore. Finally make sure to checkout [live tailing][live tailing] to see logs appearing as they are ingested in Loki.
[promtail]: ../../promtail/README.md
[promtail]: ../../promtail/README
[aws cli]: https://aws.amazon.com/cli/
[create an vpc]: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-subnets-commands-example.html
[create an ssh key]: https://confluence.atlassian.com/bitbucketserver/creating-ssh-keys-776639788.html
@ -276,18 +276,18 @@ Let's head back to Grafana and verify that your Promtail logs are available in G
@ -99,7 +99,7 @@ Once deployed, the Grafana service will send its logs to Loki.
## Labels
Loki can received a set of labels along with log line. These labels are used to index log entries and query back logs using [LogQL stream selector](../../logql/_index.md#log-stream-selector).
Loki can received a set of labels along with log line. These labels are used to index log entries and query back logs using [LogQL stream selector](../../logql#log-stream-selector).
By default, the Docker driver will add the following labels to each log line:
@ -21,7 +21,7 @@ docker run -v /var/log:/var/log \
You can run Fluent Bit as a [Daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) to collect all your Kubernetes workload logs.
To do so you can use our [Fluent Bit helm chart](../../production/helm/fluent-bit/README.md):
To do so you can use our [Fluent Bit helm chart](../../production/helm/fluent-bit/README):
> Make sure [tiller](https://helm.sh/docs/install/) is installed correctly in your cluster
You can use fluent-bit Loki Docker image as a Firelens log router in AWS ECS.
For more information about this see our [AWS documentation](../aws/ecs/_index.md)
For more information about this see our [AWS documentation](../aws/ecs)
### Local
First you need to follow those [instructions](https://github.com/grafana/loki/blob/master/cmd/fluent-bit/README.md) to build the plugin dynamic library.
First you need to follow those [instructions](https://github.com/grafana/loki/blob/master/cmd/fluent-bit/README) to build the plugin dynamic library.
The assuming you have Fluent Bit installed in your `$PATH` you can run the plugin using:
@ -92,7 +92,7 @@ You can also adapt your plugins.conf, removing the need to change the command li
### Labels
Labels are used to [query logs](../../logql/_index.md) `{container_name="nginx", cluster="us-west1"}`, they are usually metadata about the workload producing the log stream (`instance`, `container_name`, `region`, `cluster`, `level`). In Loki labels are indexed consequently you should be cautious when choosing them (high cardinality label values can have performance drastic impact).
Labels are used to [query logs](../../logql) `{container_name="nginx", cluster="us-west1"}`, they are usually metadata about the workload producing the log stream (`instance`, `container_name`, `region`, `cluster`, `level`). In Loki labels are indexed consequently you should be cautious when choosing them (high cardinality label values can have performance drastic impact).
You can use `Labels`, `RemoveKeys` , `LabelKeys` and `LabelMapPath` to how the output plugin will perform labels extraction.
@ -103,7 +103,7 @@ If set to true, it will add all Kubernetes labels to Loki labels automatically a
### LabelMapPath
When using the `Parser` and `Filter` plugins Fluent Bit can extract and add data to the current record/log data. While Loki labels are key value pair, record data can be nested structures.
You can pass a json file that defines how to extract [labels](../../docs/sources/overview/_index.md#overview-of-loki) from each record. Each json key from the file will be matched with the log record to find label values. Values from the configuration are used as label names.
You can pass a json file that defines how to extract [labels](../../docs/sources/overview#overview-of-loki) from each record. Each json key from the file will be matched with the log record to find label values. Values from the configuration are used as label names.
Loki includes an [AWS SAM](https://aws.amazon.com/serverless/sam/) package template for shipping Cloudwatch logs to Loki via a set of promtails [here](../../../tools/lambda-promtail/). This is done via an intermediary [lambda function](https://aws.amazon.com/lambda/) which processes cloudwatch events and propagates them to a promtail instance (or set of instances behind a load balancer) via the push-api [scrape config](docs/clients/promtail/configuration#loki_push_api_config).
Loki includes an [AWS SAM](https://aws.amazon.com/serverless/sam/) package template for shipping Cloudwatch logs to Loki via a set of promtails [here](../../../tools/lambda-promtail/). This is done via an intermediary [lambda function](https://aws.amazon.com/lambda/) which processes cloudwatch events and propagates them to a promtail instance (or set of instances behind a load balancer) via the push-api [scrape config](../promtail/configuration#loki_push_api_config).
@ -91,7 +91,7 @@ ttl can be configured using `cache_ttl` config.
### Write Deduplication disabled
Loki does write deduplication of chunks and index using Chunks and WriteDedupe cache respectively, configured with [ChunkStoreConfig](../../configuration/README.md#chunk_store_config).
Loki does write deduplication of chunks and index using Chunks and WriteDedupe cache respectively, configured with [ChunkStoreConfig](../../configuration/README#chunk_store_config).
The problem with write deduplication when using `boltdb-shipper` though is ingesters only keep uploading boltdb files periodically to make them available to all the other services which means there would be a brief period where some of the services would not have received updated index yet.
The problem due to that is if an ingester which first wrote the chunks and index goes down and all the other ingesters which were part of replication scheme skipped writing those chunks and index due to deduplication, we would end up missing those logs from query responses since only the ingester which had the index went down.
This problem would be faced even during rollouts which is quite common.
@ -57,7 +57,7 @@ You may use any subsitutable services, such as those that implement the S3 API l
### Cassandra
Cassandra can also be utilized for the index store and asides from the experimental [boltdb-shipper](./storage/boltdb-shipper.md), it's the only non-cloud offering that can be used for the index that's horizontally scalable and has configurable replication. It's a good candidate when you already run Cassandra, are running on-prem, or do not wish to use a managed cloud offering.
Cassandra can also be utilized for the index store and asides from the experimental [boltdb-shipper](./storage/boltdb-shipper), it's the only non-cloud offering that can be used for the index that's horizontally scalable and has configurable replication. It's a good candidate when you already run Cassandra, are running on-prem, or do not wish to use a managed cloud offering.
### BigTable
@ -73,7 +73,7 @@ DynamoDB is susceptible to rate limiting, particularly due to overconsuming what
### BoltDB
BoltDB is an embedded database on disk. It is not replicated and thus cannot be used for high availability or clustered Loki deployments, but is commonly paired with a `filesystem` chunk store for proof of concept deployments, trying out Loki, and development. There is also an experimental mode, the [boltdb-shipper](./operations/storage/boltdb-shipper.md), which aims to support clustered deployments using `boltdb` as an index.
BoltDB is an embedded database on disk. It is not replicated and thus cannot be used for high availability or clustered Loki deployments, but is commonly paired with a `filesystem` chunk store for proof of concept deployments, trying out Loki, and development. There is also an experimental mode, the [boltdb-shipper](./operations/storage/boltdb-shipper), which aims to support clustered deployments using `boltdb` as an index.
## Period Configs
@ -113,7 +113,7 @@ table_manager:
retention_period: 2520h
```
For more information, see the table manager [doc](./operations/storage/table-manager.md).
For more information, see the table manager [doc](./operations/storage/table-manager).
### Provisioning
@ -132,13 +132,13 @@ table_manager:
inactive_read_throughput: <int> | Default = 300
```
Note, there are a few other DynamoDB provisioning options including DynamoDB autoscaling and on-demand capacity. See the [docs](./configuration/README.md#provision_config) for more information.
Note, there are a few other DynamoDB provisioning options including DynamoDB autoscaling and on-demand capacity. See the [docs](./configuration/README#provision_config) for more information.
## Upgrading Schemas
When a new schema is released and you want to gain the advantages it provides, you can! Loki can transparently query & merge data from across schema boundaries so there is no disruption of service and upgrading is easy.
First, you'll want to create a new [period_config](./configuration/README.md#period_config) entry in your [schema_config](./configuration/README.md#schema_config). The important thing to remember here is to set this at some point in the _future_ and then roll out the config file changes to Loki. This allows the table manager to create the required table in advance of writes and will ensure that existing data isn't queried as if it adheres to the new schema.
First, you'll want to create a new [period_config](./configuration/README.md#period_config) entry in your [schema_config](./configuration/README#schema_config). The important thing to remember here is to set this at some point in the _future_ and then roll out the config file changes to Loki. This allows the table manager to create the required table in advance of writes and will ensure that existing data isn't queried as if it adheres to the new schema.
As an example, let's say it's 2020-07-14 and we want to start using the `v11` schema on the 20th:
```yaml
@ -168,7 +168,7 @@ With the exception of the `filesystem` chunk store, Loki will not delete old chu
We're interested in adding targeted deletion in future Loki releases (think tenant or stream level granularity) and may include other strategies as well.
For more information, see the configuration [docs](./operations/storage/retention.md).
For more information, see the configuration [docs](./operations/storage/retention).
Owen Diehl
6 years ago
committed by