From 7fc25743eb37dc6e495652610b45b9ab5f55ca23 Mon Sep 17 00:00:00 2001 From: Susana Ferreira Date: Thu, 19 Jan 2023 22:10:42 +0100 Subject: [PATCH] Docs fix: add missing sections on index.template file (#8216) **What this PR does / why we need it**: PR https://github.com/grafana/loki/pull/7916 which introduced the documentation automation tool for Loki's configuration values, incorrectly removed the last sections of the documentation. This PR introduces them accordingly. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --- docs/sources/configuration/_index.md | 86 ++++++++++++++++++++++- docs/sources/configuration/index.template | 86 ++++++++++++++++++++++- 2 files changed, 170 insertions(+), 2 deletions(-) diff --git a/docs/sources/configuration/_index.md b/docs/sources/configuration/_index.md index a59bf4dedf..e1a9b0e660 100644 --- a/docs/sources/configuration/_index.md +++ b/docs/sources/configuration/_index.md @@ -4078,4 +4078,88 @@ Named store from this example can be used by setting object_store to store-1 in [gcs: ] [swift: ] -``` \ No newline at end of file +``` + +## Runtime Configuration file + +Loki has a concept of "runtime config" file, which is simply a file that is reloaded while Loki is running. It is used by some Loki components to allow operator to change some aspects of Loki configuration without restarting it. File is specified by using `-runtime-config.file=` flag and reload period (which defaults to 10 seconds) can be changed by `-runtime-config.reload-period=` flag. Previously this mechanism was only used by limits overrides, and flags were called `-limits.per-user-override-config=` and `-limits.per-user-override-period=10s` respectively. These are still used, if `-runtime-config.file=` is not specified. + +At the moment, two components use runtime configuration: limits and multi KV store. + +Options for runtime configuration reload can also be configured via YAML: + +```yaml +# Configuration file to periodically check and reload. +[file: : default = empty] + +# How often to check the file. +[period: : default 10s] +``` + +Example runtime configuration file: + +```yaml +overrides: + tenant1: + ingestion_rate_mb: 10 + max_streams_per_user: 100000 + max_chunks_per_query: 100000 + tenant2: + max_streams_per_user: 1000000 + max_chunks_per_query: 1000000 + +multi_kv_config: + mirror-enabled: false + primary: consul +``` + +## Accept out-of-order writes + +Since the beginning of Loki, log entries had to be written to Loki in order +by time. +This limitation has been lifted. +Out-of-order writes are enabled globally by default, but can be disabled/enabled +on a cluster or per-tenant basis. + +- To disable out-of-order writes for all tenants, +place in the `limits_config` section: + + ``` + limits_config: + unordered_writes: false + ``` + +- To disable out-of-order writes for specific tenants, +configure a runtime configuration file: + + ``` + runtime_config: overrides.yaml + ``` + + In the `overrides.yaml` file, add `unordered_writes` for each tenant + permitted to have out-of-order writes: + + ``` + overrides: + "tenantA": + unordered_writes: false + ``` + +How far into the past accepted out-of-order log entries may be +is configurable with `max_chunk_age`. +`max_chunk_age` defaults to 2 hour. +Loki calculates the earliest time that out-of-order entries may have +and be accepted with + +``` +time_of_most_recent_line - (max_chunk_age/2) +``` + +Log entries with timestamps that are after this earliest time are accepted. +Log entries further back in time return an out-of-order error. + +For example, if `max_chunk_age` is 2 hours +and the stream `{foo="bar"}` has one entry at `8:00`, +Loki will accept data for that stream as far back in time as `7:00`. +If another log line is written at `10:00`, +Loki will accept data for that stream as far back in time as `9:00`. diff --git a/docs/sources/configuration/index.template b/docs/sources/configuration/index.template index 5ceb47fef2..2dcadf8642 100644 --- a/docs/sources/configuration/index.template +++ b/docs/sources/configuration/index.template @@ -95,4 +95,88 @@ Pass the `-config.expand-env` flag at the command line to enable this way of set ### Supported contents and default values of `loki.yaml` -{{ .ConfigFile }} \ No newline at end of file +{{ .ConfigFile }} + +## Runtime Configuration file + +Loki has a concept of "runtime config" file, which is simply a file that is reloaded while Loki is running. It is used by some Loki components to allow operator to change some aspects of Loki configuration without restarting it. File is specified by using `-runtime-config.file=` flag and reload period (which defaults to 10 seconds) can be changed by `-runtime-config.reload-period=` flag. Previously this mechanism was only used by limits overrides, and flags were called `-limits.per-user-override-config=` and `-limits.per-user-override-period=10s` respectively. These are still used, if `-runtime-config.file=` is not specified. + +At the moment, two components use runtime configuration: limits and multi KV store. + +Options for runtime configuration reload can also be configured via YAML: + +```yaml +# Configuration file to periodically check and reload. +[file: : default = empty] + +# How often to check the file. +[period: : default 10s] +``` + +Example runtime configuration file: + +```yaml +overrides: + tenant1: + ingestion_rate_mb: 10 + max_streams_per_user: 100000 + max_chunks_per_query: 100000 + tenant2: + max_streams_per_user: 1000000 + max_chunks_per_query: 1000000 + +multi_kv_config: + mirror-enabled: false + primary: consul +``` + +## Accept out-of-order writes + +Since the beginning of Loki, log entries had to be written to Loki in order +by time. +This limitation has been lifted. +Out-of-order writes are enabled globally by default, but can be disabled/enabled +on a cluster or per-tenant basis. + +- To disable out-of-order writes for all tenants, +place in the `limits_config` section: + + ``` + limits_config: + unordered_writes: false + ``` + +- To disable out-of-order writes for specific tenants, +configure a runtime configuration file: + + ``` + runtime_config: overrides.yaml + ``` + + In the `overrides.yaml` file, add `unordered_writes` for each tenant + permitted to have out-of-order writes: + + ``` + overrides: + "tenantA": + unordered_writes: false + ``` + +How far into the past accepted out-of-order log entries may be +is configurable with `max_chunk_age`. +`max_chunk_age` defaults to 2 hour. +Loki calculates the earliest time that out-of-order entries may have +and be accepted with + +``` +time_of_most_recent_line - (max_chunk_age/2) +``` + +Log entries with timestamps that are after this earliest time are accepted. +Log entries further back in time return an out-of-order error. + +For example, if `max_chunk_age` is 2 hours +and the stream `{foo="bar"}` has one entry at `8:00`, +Loki will accept data for that stream as far back in time as `7:00`. +If another log line is written at `10:00`, +Loki will accept data for that stream as far back in time as `9:00`.