33 Commits (dff72d2a52094fb2a831b5930cbfc67759b0978d)

Author	SHA1	Message	Date
Ed Welch	0f7e620809	docs: re-arrange upgrade guide (#12504) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
J Stickler	3634ff4d64	docs: move Storage page under Configuration (#12478)	2 years ago
Ed Welch	3ece2ea6c4	refactor!: remove async cache writeback on chunk fetcher (#12456) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	bc26198c00	feat: enable caching of index stats results, volume results, series results and label results by default (#12452) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	7607f04484	feat!: enable structure metadata by default. (#12453) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	24f4e45165	feat: Enable automatic stream sharding by default (#12451) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	d39dc09ca6	refactor!: refactor how we do defaults for runtime overrides (#12448) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Salva Corts	ef75ba35d6	Deprecate and flip -legacy-read-mode to false by default (#11665) ... **What this PR does / why we need it**: This PR addresses a pending TODO to flip the value of the `-legacy-read-mode` to false by default. We also deprecate it as we plan to remove it after Loki 3.0.	2 years ago
J Stickler	0dc8c75331	Update admonitions to add shortcode (#11622) ... **What this PR does / why we need it**: Completing the work that was started in #9409 but not completed because there were multiple variations on how to format notes in the docs source files. We rolled out different colors for admonitions recently, which highlighted that there were still a few notes not using the admonition shortcode. I searched the docs for the following words: * tip * caution * important * warning * note So hopefully I've caught all of the admonitions that weren't using the shortcode.	2 years ago
Ashwanth	32e9ee8209	chore: follow-up to #11151 and #11025 (#11225) ... **What this PR does / why we need it**: - remove usage of `enforce_metric_name` in ksonnet - https://github.com/grafana/loki/pull/11151 - make `cortex_ruler_client_request_duration_seconds` prefix configurable - updates upgrade guide to remove reference to metrics that loki does not export **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	2 years ago
Ashwanth	766f27645d	config: remove flags & YAML settings deprecated in 2.9.x (#11151) ... **What this PR does / why we need it**: - remove `-ruler.evaluation-delay-duration` - remove `boltdb.shipper.compactor.deletion-mode` - remove compactor flags with prefix`-boltdb.shipper.compactor.*` - remove `validation.enforce-metric-name` - remove `log.use-buffered`, `log.use-sync` **Special notes for your reviewer**: `log.use-buffered`, `log.use-sync` are not mentioned in the upgrade guide since they were hidden fields **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [X] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Michel Hollands	380f902c30	Change default of metrics.namespace (#11110) ... **What this PR does / why we need it**: Change the default for a subset of metrics. After this all Loki metrics should start with "loki_". This changes the following metrics: cortex_distributor_ingester_clients cortex_dns_failures_total cortex_dns_lookups_total cortex_dns_provider_results cortex_frontend_query_range_duration_seconds_bucket cortex_frontend_query_range_duration_seconds_count cortex_frontend_query_range_duration_seconds_sum cortex_ingester_flush_queue_length cortex_kv_request_duration_seconds_bucket cortex_kv_request_duration_seconds_count cortex_kv_request_duration_seconds_sum cortex_member_consul_heartbeats_total cortex_prometheus_last_evaluation_samples cortex_prometheus_notifications_alertmanagers_discovered cortex_prometheus_notifications_dropped_total cortex_prometheus_notifications_errors_total cortex_prometheus_notifications_latency_seconds cortex_prometheus_notifications_latency_seconds_count cortex_prometheus_notifications_latency_seconds_sum cortex_prometheus_notifications_queue_capacity cortex_prometheus_notifications_queue_length cortex_prometheus_notifications_sent_total cortex_prometheus_rule_evaluation_duration_seconds cortex_prometheus_rule_evaluation_duration_seconds_count cortex_prometheus_rule_evaluation_duration_seconds_sum cortex_prometheus_rule_evaluation_failures_total cortex_prometheus_rule_evaluations_total cortex_prometheus_rule_group_duration_seconds cortex_prometheus_rule_group_duration_seconds_count cortex_prometheus_rule_group_duration_seconds_sum cortex_prometheus_rule_group_interval_seconds cortex_prometheus_rule_group_iterations_missed_total cortex_prometheus_rule_group_iterations_total cortex_prometheus_rule_group_last_duration_seconds cortex_prometheus_rule_group_last_evaluation_timestamp_seconds cortex_prometheus_rule_group_rules cortex_query_frontend_connected_schedulers cortex_query_frontend_queries_in_progress cortex_query_frontend_retries_bucket cortex_query_frontend_retries_count cortex_query_frontend_retries_sum cortex_query_scheduler_connected_frontend_clients cortex_query_scheduler_connected_querier_clients cortex_query_scheduler_inflight_requests cortex_query_scheduler_inflight_requests_count cortex_query_scheduler_inflight_requests_sum cortex_query_scheduler_queue_duration_seconds_bucket cortex_query_scheduler_queue_duration_seconds_count cortex_query_scheduler_queue_duration_seconds_sum cortex_query_scheduler_queue_length cortex_query_scheduler_running cortex_quota_cgroup_cpu_max cortex_quota_cgroup_cpu_period cortex_quota_cpu_count cortex_quota_gomaxprocs cortex_ring_member_heartbeats_total cortex_ring_member_tokens_owned cortex_ring_member_tokens_to_own cortex_ring_members cortex_ring_oldest_member_timestamp cortex_ring_tokens_total cortex_ruler_client_request_duration_seconds_bucket cortex_ruler_client_request_duration_seconds_count cortex_ruler_client_request_duration_seconds_sum cortex_ruler_clients cortex_ruler_config_last_reload_successful cortex_ruler_config_last_reload_successful_seconds cortex_ruler_config_updates_total cortex_ruler_managers_total cortex_ruler_ring_check_errors_total cortex_ruler_sync_rules_total **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [X] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0) --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Ashwanth	a118e994d8	Remove shared_store and shared_store_key_prefix from shipper and compactor (#10840) ... **What this PR does / why we need it**: #### Removes `shared_store` and `shared_store_key_prefix` from index shipper and compactor configs and their corresponding CLI flags. - `-tsdb.shipper.shared-store` - `-boltdb.shipper.shared-store` - `-tsdb.shipper.shared-store.key-prefix` - `-boltdb.shipper.shared-store.key-prefix` - `-boltdb.shipper.compactor.shared-store` - `-boltdb.shipper.compactor.shared-store.key-prefix` `shared_store` has been a confusing option allowing users to easily misconfigure Loki. Going forward `object_store` setting in the [period_config](https://grafana.com/docs/loki/latest/configure/#period_config) (which already configured the store for chunks) will be used to configure store for the index. And the newly added `path_prefix` option under the `index` key in `period_config` will configure the path under which index tables are stored. This change enforces chunks and index files for a given period reside together in the same storage bucket. More details in the upgrade guide. --- `-compactor.delete-request-store` has to be **explicitly configured** going forward. Without setting this, loki wouldn't know which object store to use for storing delete requests. Path prefix for storing deletes is decided by `-compactor.delete-request-store.key-prefix` which defaults to `index/`. **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Kaviraj Kanagaraj	a550b767d3	config: Remove already deprecated `store.max-look-back-period`. (#11038) ... This is already been deprecated longback. Decided to remove in next versions of Loki Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
J Stickler	34915c7bd7	Spell out Kubernetes in docs (#10922) ... Developers tend to abbreviate "Kubernetes" to just K8, but as a best practice it should be spelled out in the documentation.	2 years ago
Ashwanth	8bb615c2cc	config: loki better defaults (#10793) ... **What this PR does / why we need it**: Updates some of the configuration defaults to provide a better experience for users out of the box. I tried to add notes for non-trivial changes explaining the reason. | configuration | new default | old default | notes | | ------------------------------------------------------ | ----------- | ----------- | -------- | `compactor.delete-max-interval` | 24h | 0 | splits the delete requests into intervals no longer than `delete_max_interval` | | `distributor.max-line-size` | 256KB | 0 | - | | `ingester.sync-period` | 1h | 0 | ensures that the chunk cuts for a given stream are synchronized across the ingesters in the replication set. Helps with deduplicating chunks. | | `ingester.sync-min-utilization` | 0.1 | 0 | ^ | | `frontend.max-querier-bytes-read` | 150GB | 0 | - | | `frontend.max-cache-freshness` | 10m | 1m | avoid caching results for data that is still in churn | | `frontend.max-stats-cache-freshness` | 10m | 0 | avoid caching results for data that is still in churn | | `memcached.batchsize` | 256 | 1024 | - | | `memcached.parallelism` | 10 | 100 | - | | `querier.tsdb-max-query-parallelism` | 128 | 512 | avoid over parallelising queries for smaller installations | | `querier.split-queries-by-interval` | 1h | 30m | avoid over parallelising queries for smaller installations | | `querier.compress-http-responses` | true | false | compress response if the request accepts gzip encoding | | `query-scheduler.max-outstanding-requests-per-tenant` | 32000 | 100 | current default is too small, opening a big dashboard could hit this limit | | `validation.max-label-names-per-series` | 15 | 30 | reducing this to avoid blowing up the series count and index size. Also enforces users to think about what labels are being added. | **Special notes for your reviewer**: regarding `TestQueryTSDB_WithCachedPostings`: removed the assert on get calls as this is hard to get right (splits, querySize mw making stats calls) validating additions to the cache and misses is more important here I think. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Ashwanth	0be191309c	config: better defaults for querier max concurrency (#10785) ... **What this PR does / why we need it**: To simply loki configuration, this pr removes `querier.worker-parallelism` as it does not offer additional value to already existing `querier.max-concurrent` Also updates the default value of `querier.max-concurrent` to 4. This is a better default for users that are just starting out. Users can consider increasing it as they scale up their loki installation. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Christian Haudum	54e46446fa	Do not transfer off chunks on ingester shutdown (#10709) ... This PR removes the ability of an ingester in LEAVING state prior to shutdown to transfer chunks off to another ingester in the ring in PENDING state. The **Write Ahead Log** (WAL) supersedes the chunk transfer feature. See upgrade notes for information how to replace the removed setting. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Salva Corts <salva.corts@grafana.com>	2 years ago
Ashwanth	3038170aee	Add deprecation node for write dedupe cache (#10736) ... **What this PR does / why we need it**: Adds a deprecation node for write dedupe cache. This cannot be fully removed yet as legacy indexes rely on this. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Ashwanth	fcdd55b6c9	embedded cache: update metric namespace and remove duplicate metrics (#10693) ... **What this PR does / why we need it**: Removes the following embedded cache metrics since `loki_cache_request_duration_seconds` (which instruments requests made to all the caches) [already covers them](6a07988491/pkg/storage/chunk/cache/cache.go (L110)): - `querier_cache_added_total` - `querier_cache_gets_total` - `querier_cache_misses_total` Updates the namespace and subsystem for the following metrics - `querier_cache_added_new_total` is renamed to `loki_embeddedcache_added_new_total` - `querier_cache_evicted_total` is renamed to `loki_embeddedcache_evicted_total` - `querier_cache_entries` is renamed to `loki_embeddedcache_entries` - `querier_cache_memory_bytes` is renamed to `loki_embeddedcache_memory_bytes` Removes already deprecated metric `querier_cache_stale_gets_total` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Ashwanth	2902f986a4	cache: Enable embedded cache if no other cache is configured. (#10620) ... **What this PR does / why we need it**: FIFO cache is replaced by embedded-cache in https://github.com/grafana/loki/pull/6821 This PR enables embedded cache for query range results, chunks cache, index_stats_results and volume_results if no other cache is explicitly configured (redis, memcache). **Special notes for your reviewer**: Also removes a deprecated flag `cache-split-interval` **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	e03637151b	Remove legacy ingester shutdown handler (#10655) ... The handler was deprecated for a while and is removed in favour or `/ingester/shutdown?flush=true`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Christian Haudum	ebef2fe1cd	Remove backup index writer from TSDB configuration (#10534) ... The setting `use_boltdb_shipper_as_backup` (`-tsdb.shipper.use-boltdb-shipper-as-backup`) was a remnant from the development of the TSDB storage. It was used to allow writing to both TSDB and BoltDB when TSDB was still highly experimental. Since TSDB is now stable and the recommended index type, the setting has become irrelevant and therefore was removed. The previous default value `false` is applied. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	4d96393935	Reflect 2.9 upgrade notes on main (#10536) ... Cleanup upgrade notes from 2.9 release on main branch. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Danny Kopping	2c84959901	Distributor: add ingester append timeouts error (#10456) ... **What this PR does / why we need it**: Failing to send samples to ingesters because the request exceeded its timeout is a very clear signal that ingesters are unable to keep up with demand. In an incident today we saw that ingesters' push latencies were increased sharply by an expensive regex query which was starving other goroutines of time on CPU. This new alert `loki_distributor_ingester_append_timeouts_total` will give us a high-signal metric which we can use for alerting.	2 years ago
Shantanu Alshi	f1bbdc51dc	Remove split_queries_by_interval and forward_headers_list (#10395) ... …range config Removed split_queries_by_interval and forward_headers_list from query_range config The fields were deprecated and hence going to be removed in the next major version **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Shantanu Alshi	145319e5f5	Remove experimental enable ruler api toggle (#10380) ... Remove `-experimental.ruler.enable-api` in favour of `-ruler.enable-api`. The former was already deprecated for some time.	2 years ago
Shantanu Alshi	92ba055ae5	Remove ruler.wal-cleaer.period (#10378) ... The config with typo is now replaced by `-ruler.wal-cleaner.period`	2 years ago
Shantanu Alshi	325c332f3e	Remove deprecated s3 sse toggle (#10377) ... This PR removes a legacy and already deprecated boolean setting `-s3.sse-encryption` in favor of the more fine-grain `-s3.sse.*` settings.	2 years ago
Ashwanth	470cbbf118	querier: remove query_timeout and engine timeout from querier conf (#10302) ... **What this PR does / why we need it**: Removes `query_timeout` and `engine:timeout` from querier's config section Both these configs have been deprecated in 2.7.0 release in favor of `query_timeout` in `limits_config` which also allows applying per-tenant overrides at runtime. CLI flag `-querier.engine.timeout` is also removed **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Ashwanth	b1cd5da607	upgrade doc: suggest disabling transfers (#8648) ... **What this PR does / why we need it**: Explicitly mention that WAL and `ingester.max-transfer-retries` are [incompatible](https://github.com/grafana/loki/blob/ashwanth/disable-transfers-upgrade-doc/pkg/ingester/ingester.go#L146) in 2.4.0 upgrade notes. Given that WAL is enabled by default, suggest users to unset `ingester.max-transfer-retries` in case they explicitly configured it. **Which issue(s) this PR fixes**: Fixes #8625 **Special notes for your reviewer**: **Checklist** - [X ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
J Stickler	d0545bff2d	[docs] Create top level Send data section, part 2 (#10247) ... Part of the database information architecture Epic # 8710 Which issue(s) this PR fixes: Second half of work for issue # 8741 that was started in PR #10192 When closed, fixes #8741 Move the following files under Send data: ./sources/clients/promtail/_index.md ./sources/clients/promtail/configuration.md ./sources/clients/promtail/gcplog-cloud.md ./sources/clients/promtail/installation.md ./sources/clients/promtail/logrotation/_index.md ./sources/clients/promtail/pipelines.md ./sources/clients/promtail/scraping.md ./sources/clients/promtail/stages/_index.md ./sources/clients/promtail/stages/cri.md ./sources/clients/promtail/stages/decolorize.md ./sources/clients/promtail/stages/docker.md ./sources/clients/promtail/stages/drop.md ./sources/clients/promtail/stages/json.md ./sources/clients/promtail/stages/labelallow.md ./sources/clients/promtail/stages/labeldrop.md ./sources/clients/promtail/stages/labels.md ./sources/clients/promtail/stages/limit.md ./sources/clients/promtail/stages/logfmt.md ./sources/clients/promtail/stages/match.md ./sources/clients/promtail/stages/metrics.md ./sources/clients/promtail/stages/multiline.md ./sources/clients/promtail/stages/output.md ./sources/clients/promtail/stages/pack.md ./sources/clients/promtail/stages/regex.md ./sources/clients/promtail/stages/replace.md ./sources/clients/promtail/stages/static_labels.md ./sources/clients/promtail/stages/template.md ./sources/clients/promtail/stages/tenant.md ./sources/clients/promtail/stages/timestamp.md ./sources/clients/promtail/troubleshooting/_index.md This PR also - Revises the Clients landing page to clarify supported clients. - Updates the metadata (descriptions, weights) - Adds aliases to redirect from the old URLs. - Updates cross-references broken by the move/renaming. - A few other small fixes (headings, typos, etc.) **Special notes for your reviewer**: Please review the updates to the Clients landing page (now called Send Data) as I've made some extensive edits to try to clarify recommended/supported clients. The file is docs/sources/send-data/_index.md --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
J Stickler	aae92b9840	[Docs] Create top-level Setup section (#9974) ... **Which issue(s) this PR fixes**: Fixes #8712 **Special notes for your reviewer**: Move the following files under a new Setup section (with three subsections: Install, Migrate, and Upgrade). - ./sources/installation/sizing/index.md - ./sources/installation/_index.md - ./sources/installation/docker.md - ./sources/installation/helm/_index.md - ./sources/installation/helm/concepts.md - ./sources/installation/helm/configure-storage/index.md - ./sources/installation/helm/install-monolithic/index.md - ./sources/installation/helm/install-scalable/index.md - ./sources/installation/helm/migrate-from-distributed/index.md - ./sources/installation/helm/migrate-to-three-scalable-targets/index.md - ./sources/installation/helm/monitor-and-alert/index.md - ./sources/installation/helm/reference.md - ./sources/installation/helm/upgrade-from-2.x/index.md - ./sources/installation/install-from-source.md - ./sources/installation/istio.md - ./sources/installation/local.md - ./sources/installation/tanka.md Updates cross-references broken by the move, Updates file metadata (descriptions, weights) and adds aliases for redirects to avoid 404s. Probably fixes a couple of typos too.	2 years ago
Vladyslav Diachenko	28e9e15639	[jsonnet] removed deprecated policy.v1beta1 (#9978) ... **What this PR does / why we need it**: in k8s version 1.25 some k8s APIs are deprecated, such as policy.v1beta1 which is moved to policy.v1 **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
J Stickler	7797a98074	TOC Sync: Move configuration files under Configure (#9879) ... **Which issue(s) this PR fixes**: Fixes #8713 **Special notes for your reviewer**: Replaces PR [#8817](https://github.com/grafana/loki/pull/8817), which was stale and badly out of date. Part of the Database products table of contents alignment project. Moves configuration files from Configuration to new Configure heading in TOC. Updates front matter for the topic. Fixes links broken by the move. Updates the MakeFile to point to the new Configure DOCS_SOURCES_PATH.	2 years ago
Ashwanth	f9fc1df4c0	logfmt: add --strict flag to enable strict parsing, perform nostrict parsing by default (#9626) ... **What this PR does / why we need it**: Introduces `--strict` flag for logfmt parser to enable strict parsing. With strict parsing enabled, logfmt parser stops scanning the log line and returns early with an error if it encounters any poorly formatted key=value pair. This pr updates the parser to perform non-strict parsing by default where it skips over the invalid tokens. This offers the flexibility to parse semi-structured log lines. Also adds `--keep-empty` flag to retain standalone keys(with empty value) as labels with value set to empty string **Special notes for your reviewer**: **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Christian Haudum	ce81895241	Add shuffle sharding to index gateway (#9710) ... This PR introduces shuffle sharding of the index files across index gateways where the shuffle sharding key is the tenant ID. Instead of having a global replication factor on the index gateways which affects all tenants the same, whether they are small or large tenants, the shuffle sharding allows to specify a shard factor per tenant (on top of the replication factor). This a) reduces the amount of indexes that are downloaded on individual index gateways in case of a high replication factor (which is currently the de-factor standard to deal with high load) and b) allows to scale tenants individually. The shuffle sharding algorithm is part of dskit's ring implementation and is not part of this commit. If you have set a high RF for the index gateways, it is required to reduce the RF and instead set the shard factor for the tenants. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 years ago
J Stickler	11e2375e6d	Removing 'please' from the docs. (#9658) ... **Which issue(s) this PR fixes**: Fixes #8922 **Special notes for your reviewer**: This is to conform with our Style Guide. https://developers.google.com/style/tone#politeness-and-use-of-please	3 years ago
Jack Baldry	8391e557b1	Update doc-validator to v3.0.0 (#9666) ... ## 3.0.0 ### Added - Structured output for use with [`reviewdog`](https://github.com/reviewdog/reviewdog). You can achieve the original error output by piping the output to the following `jq` expression: `jq -r '"ERROR: \(.location.path):\(.location.range.start.line):\(.location.range.start.column): \(.message)"'`. - Suggestions for simple link fixes. In GitHub, [`reviewdog`](https://github.com/reviewdog/reviewdog) comments these suggestions for convenient replacement. - Support for anchors referring to repeated headings. In the case that a page has multiple headings that share the same text, the renderer appends a zero indexed, numbered suffix to the identifier. For example, when there are two headings that are both "Heading text", the first anchor identifier is `heading-text` and the second is `heading-text-1`. - Error when running `doc-validator` on no files. **Notes for reviewer:** This also updates the `make-docs` procedure. Please ensure you can run `make docs` from the `docs/` directory before approving and merging this PR. --------- Signed-off-by: Jack Baldry <jack.baldry@grafana.com>	3 years ago
Ashwanth	52cd0a39b8	compactor: multi-store support (#7447) ... **What this PR does / why we need it**: This PR adds multi-store support for compactors. Since loki allows users to configure mutiple stores using schema_config, compactor should be able to operate on multiple object stores that contain index. Currently, it can perform compaction on indexes in a single store. To maintain backward compatibility: if `boltdb.shipper.compactor.shared-store` is set, compactor will only operate on that store, else compactor will be initialized to operate on all the object store indexes (`boltdb, tsdb`) defined in the schema config. This PR also adds a new config option to define where delete requests are to be stored - `boltdb.shipper.compactor.delete-request-store`. If it's not set, `boltdb.shipper.compactor.shared-store` is used for storing them, this is to ensure no config changes are required by the users when upgrading. Refer to [docs/sources/upgrading/_index.md](a85d01c9cd/docs/sources/upgrading/_index.md) for more details. **Which issue(s) this PR fixes:** Fixes https://github.com/grafana/loki/issues/7276 **Checklist** - [X] Reviewed the `CONTRIBUTING.md` guide - [X] Documentation added - [X] Tests updated - [X] `CHANGELOG.md` updated - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	3 years ago
Michel Hollands	abd6131bba	Update prepare shutdown (#9175) ... **What this PR does / why we need it**: This updates the PrepareShutdown method so it supports GET and DELETE methods as well. This makes it similar to Mimir: https://github.com/grafana/mimir/pull/4718. The status is now stored in a local file. A new config setting had to be added for this file as there is no obvious place to store it. **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Michel Hollands <michel.hollands@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	3 years ago
René Scheibe	e050d5c494	Docs: Update Loki GitHub URLs (#9080) ... * use `main` instead of `master` branch * fix outdated URLs	3 years ago
Ashwanth	5cef03d0ea	index-shipper: add support for multiple stores (#7754) ... Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> **What this PR does / why we need it**: Currently loki initializes a single instance of index-shipper to [handle all the table ranges](ff7b462973/pkg/storage/factory.go (L188)) (from across periods) for a given index type `boltdb-shipper, tsdb`. Since index-shipper only has the object client handle to the store defined by `shared_store_type`, it limits the index uploads to a single store. Setting `shared_store_type` to a different store at a later point in time would mean losing access to the indexes stored in the previously configured store. With this PR, we initialize a separate index-shipper & table manager for each period if `shared_store_type` is not explicity configured. This offers the flexibility to store index in multiple stores (across providers). **Note**: - usage of `shared_store_type` in this commit text refers to one of these config options depending on the index in use: `-boltdb.shipper.shared-store`, `-tsdb.shipper.shared-store` - `shared_store_type` used to default to the `object_store` from the latest `period_config` if not explicitly configured. This PR removes these defaults in favor of supporting index uploads to multiple stores. **Which issue(s) this PR fixes**: Fixes #7276 **Special notes for your reviewer**: All the instances of downloads table manager operate on the same cacheDir. But it shouldn't be a problem as the tableRanges do not overlap across periods. **Checklist** - [X] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Trevor Whitney	e82885da50	Change loki version in docs for 2.8 release (#8985) ... Changes version references in Loki docs for the upcoming release Also adds a script for diffing metrics when doing a release.	3 years ago
Erik Sjölund	2cccc1bc68	Docs: Fix typos and grammar (#8966) ... **What this PR does / why we need it**: Fixes typos and grammar. Most of them were occurrences of `it's` that should be `its`. **Which issue(s) this PR fixes**: None **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Erik Sjölund <erik.sjolund@gmail.com>	3 years ago
Athira Varghese	ffb961c439	feat(storage): add support for IBM cloud object storage as storage client (#8826) ... **What this PR does / why we need it**: Add support for IBM cloud object storage as storage client **Which issue(s) this PR fixes**: Fixes NA **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated --------- Signed-off-by: Shahul <shahulsonhal@gmail.com> Co-authored-by: Aman Kumar Singh <amankrsingh2110@gmail.com> Co-authored-by: Suruthi-G-K <shruthi.suruthi@gmail.com> Co-authored-by: tareqmamari <tariq.mamari@de.ibm.com> Co-authored-by: shahulsonhal <shahulsonhal@gmail.com> Co-authored-by: Aditya C S <aditya.gnu@gmail.com> Co-authored-by: Tareq Al-Maamari <tariq.mamari@gmail.com>	3 years ago
René Scheibe	4c4c7e3010	Promtail: Fix examples how to build it (#8898) ... The build flags have to be provided before the package paths. Otherwise the `build` command fails with this error: > malformed import path "--tags=promtail_journal_enabled": leading dash See `go help build`: > usage: go build [-o output] [build flags] [packages]	3 years ago
Travis Patterson	9d20bed26a	Short circuit parsers (#8724) ... In many cases we only need a few labels from a line. Because most of our parsers parse lines incrementally, we can stop parsing a line after we have all the labels we want from it. This pr uses `ParserHints` to keep track of the number of extracted labels. It also provides a way for parsers to know when they should stop parsing. Notes: - parsers had inconsistent ordering between the `ShouldExtract` call and adding the `_extracted` label to duplicates. This PR makes appending `_extracted` always happen before `ShouldExtract` to keep counts of what is extracted compared to expected labels consistent. Next Steps: - When the user specifies a query with a grouping containing the `_extracted` label but there is no duplicate between the passed line and labels, short circuiting will not work. I'll address this in a follow up PR. Benchmarks: To try and show a balanced view of what this buys us, this pr picks a label our of the middle of a line. In a best case this might be much better. In the worst case, we have to parse the whole line. ``` benchstat short_circuit_old.txt short_circuit_new.txt name old time/op new time/op delta KeyExtraction/json-8 456ns ± 3% 256ns ± 2% -43.84% (p=0.000 n=9+10) KeyExtraction/logfmt-8 347ns ± 4% 171ns ± 2% -50.86% (p=0.000 n=10+10) KeyExtraction/logfmt-expression-8 552ns ± 2% 368ns ± 2% -33.23% (p=0.000 n=9+10) name old alloc/op new alloc/op delta KeyExtraction/json-8 5.00B ± 0% 5.00B ± 0% ~ (all equal) KeyExtraction/logfmt-8 5.00B ± 0% 5.00B ± 0% ~ (all equal) KeyExtraction/logfmt-expression-8 16.0B ± 0% 16.0B ± 0% ~ (all equal) name old allocs/op new allocs/op delta KeyExtraction/json-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) KeyExtraction/logfmt-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) KeyExtraction/logfmt-expression-8 2.00 ± 0% 2.00 ± 0% ~ (all equal) ``` --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago
Ed Welch	a4eb536fb2	Loki: remove `subqueries` from metrics.go logging and replace it with separate split and shard counters (#8761) ... **What this PR does / why we need it**: Currently the `metrics.go` log line emitted after every query includes a metric called "subqueries". This currently tracks the number of queries created by the split_by_time operations done in Loki, but does not include any counts for subqueries created as a result of sharding. It becomes difficult to make a single subqueries counter that gives useful information to determine how much a query is split by time and sharded by a shard factor, especially now that sharding in TSDB indexes is dynamic. This PR removes and deprecates the `subqueries` stat and instead creates a `splits` and `shards` statistic which records how much a query was split_by_time and the total number of shards created as well. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Ed Welch	eadb9d2c00	Loki: Change the behavior of the 0 value for retention_period to disable retention. (#8753) ... **What this PR does / why we need it**: Issue #8359 exposed a very unfortunate inconsistency in a Loki config where the zero value did not disable a feature. By convention in Loki the zero value for a config typically "disables" it. This PR changes the retention_period to honor the zero value of `0` or `0s` to disable retention. **Which issue(s) this PR fixes**: Fixes #8359 **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	3 years ago