131 Commits (11e02cccea3df55ffd85ca48c5d3f65d290a956a)

Author	SHA1	Message	Date
Roel Arents	5fb1cdb5dd	azure: add connection-string (#10187) ... **What this PR does / why we need it**: Adds a connection-string option to the azure blob storage config. So the [Azurite](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite) emulator can be used in local test environments (my use case). But also to support the use of a SAS token. See the (future) addition of this option to the Thanos object storage client: https://github.com/thanos-io/objstore/pull/51 **Which issue(s) this PR fixes**: n/a **Special notes for your reviewer**: https://github.com/thanos-io/objstore/pull/51 isn't merged yet. But I'm already creating this PR to see what others think. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Salva Corts	5f7bde74ac	Add limits (#10552) ... **What this PR does / why we need it**: This PR introduces two new limits for structured metadata: - `max_structured_metadata_size`: Defaults to 64 KB. Limits the maximum size of the structured metadata attached to a log line. - `max_structured_metadata_entries_count`: Default to 128. Limits the maximum number of structured metadata entries attached to a log line. We discussed adding other limits like `max_structured_metadata_label_name_length` and `max_structured_metadata_label_value_length` but discarded this idea for now.	2 years ago
Christian Haudum	ebef2fe1cd	Remove backup index writer from TSDB configuration (#10534) ... The setting `use_boltdb_shipper_as_backup` (`-tsdb.shipper.use-boltdb-shipper-as-backup`) was a remnant from the development of the TSDB storage. It was used to allow writing to both TSDB and BoltDB when TSDB was still highly experimental. Since TSDB is now stable and the recommended index type, the setting has become irrelevant and therefore was removed. The previous default value `false` is applied. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Jack Baldry	5408ec1c9c	Fix alias to redirect `/docs/loki/<LOKI_VERSION>/configuration/` to `/docs/loki/<LOKI_VERSION>/configure/` (#10503) ... Tested locally with `make docs` Signed-off-by: Jack Baldry <jack.baldry@grafana.com> Signed-off-by: Jack Baldry <jack.baldry@grafana.com>	2 years ago
Owen Diehl	2cc80c59c4	per tenant gate for structured metadata feature (#10402) ... Adds a per tenant configuration to block access to the upcoming structured-metadata push API extension. --------- Co-authored-by: Salva Corts <salva.corts@grafana.com>	2 years ago
Nick Pillitteri	a41aee83a4	Update to latest dskit version for SpanLogger changes (#10423) ... **What this PR does / why we need it**: In particular, dskit has changed the SpanLogger struct such that all fields are unexported. This updates Loki to work with the new version while preserving the existing Loki behavior. **Which issue(s) this PR fixes**: N/A **Special notes for your reviewer**: Required for Mimir vendor update job in `backend-enterprise` because Reasons. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	2 years ago
Trevor Whitney	a05744a385	Enable results cache for volume queries (#10403) ... This PR enables the results cache for volume queries to the `index/volume` and `index/volume_range` endpoints	2 years ago
Shantanu Alshi	f1bbdc51dc	Remove split_queries_by_interval and forward_headers_list (#10395) ... …range config Removed split_queries_by_interval and forward_headers_list from query_range config The fields were deprecated and hence going to be removed in the next major version **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Danny Kopping	e3ab2c22de	Fix CLI flags for congestion control (#10394) ... **What this PR does / why we need it**: Adds `congestion-control` prefixes to new flags. Currently to enable congestion control via a flag, one must use `-store.enabled` which is obviously misleading. --------- Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Shantanu Alshi	92ba055ae5	Remove ruler.wal-cleaer.period (#10378) ... The config with typo is now replaced by `-ruler.wal-cleaner.period`	2 years ago
Shantanu Alshi	325c332f3e	Remove deprecated s3 sse toggle (#10377) ... This PR removes a legacy and already deprecated boolean setting `-s3.sse-encryption` in favor of the more fine-grain `-s3.sse.*` settings.	2 years ago
Ashwanth	dff3622eaa	deprecate old index types and non-object stores (#10341) ... **What this PR does / why we need it**: Deprecates old index types: `aws, aws-dynamo, gcp, gcp-columnkey, bigtable, bigtable-hashed, cassandra, grpc`. Only `tsdb` and `boltdb-shipper` are supported. Deprecates non-object stores. The next major release might not allow the usage of these as `object_store` in `schema_config`: `aws-dynamo, gcp, gcp-columnkey, bigtable, bigtable-hashed, cassandra, grpc.` Deprecates table manager which is not needed once when using `boltdb-shipper` or `tsdb` index type **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Ashwanth	18482c67d6	compactor: deprecate -boltdb.shipper.compactor. prefix in favor of -compactor. (#10344) ... **What this PR does / why we need it**: Compactor config use `-boltdb.shipper.compactor.` as prefix which is misleading as the config options apply to tsdb index as well. This pr introduces `-compactor.` prefix which can now be used to configure compactor configuration options and deprecates `-boltdb.shipper.compactor.` **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Kaviraj Kanagaraj	760c255431	doc(schema): Start recommending `v12` schema instead of `v11` (#10355) ... **What this PR does / why we need it**: `v12` is stable schema with more efficient/stable chunks and index formats. Start recommending it instead of old `v11`. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Ashwanth	cc6634972f	config: deprecates unordered-writes and few unused configs (#10324) ... **What this PR does / why we need it**: Deprecates the following configs: out of order writes will be the default and the only format with the next major release - ingester.unordered-writes these need not be configurable, were added for testing - log.use-buffered - log.use-sync unused - frontend.forward-headers-list **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Ashwanth	470cbbf118	querier: remove query_timeout and engine timeout from querier conf (#10302) ... **What this PR does / why we need it**: Removes `query_timeout` and `engine:timeout` from querier's config section Both these configs have been deprecated in 2.7.0 release in favor of `query_timeout` in `limits_config` which also allows applying per-tenant overrides at runtime. CLI flag `-querier.engine.timeout` is also removed **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Christian Haudum	e3b25fb36c	Deprecate ruler setting with misleading name (#10322) ... The setting defaults to `0s` and is not supposed to be changed, therefore it should be removed in a future version (e.g. 3.0) of Loki. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
loanshark	43e542ee64	Storage: Remove `SignatureVersionv2` from the AWS S3 API request authentication. (#10295) ... AWS S3 stopped accepting requests signed using SigV2 in all regions on June 24, 2019, any requests signed using SigV2 made after this time will fail. * https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingAWSSDK.html#UsingAWSSDK-sig2-deprecation * https://aws.amazon.com/blogs/aws/amazon-s3-update-sigv2-deprecation-period-extended-modified/	2 years ago
Danny Kopping	beed298549	Dynamic client-side throttling to avoid object storage rate-limits (GCS only) (#10140) ... **What this PR does / why we need it**: Across the various cloud providers' object storage services, there are different rate-limits implemented. Rate-limits can be imposed under multiple conditions, such as server-side scale up (ramping up from low volume to high, "local" limit), reaching some defined upper limit ("absolute" limit), etc. We cannot know apriori when these rate-limits will be imposed, so we can't set up a client-side limiter to only allow a certain number of requests through per second. Additionally, that would require global coordination between queriers - which is difficult. With the above constraints, I have instead taken inspiration from TCP's [congestion control algorithms](https://en.wikipedia.org/wiki/TCP_congestion_control). This PR implements [AIMD](https://en.wikipedia.org/wiki/Additive_increase/multiplicative_decrease) (Additive Increase, Multiplicative Decrease), which is used in the congestion _avoidance_ phase of congestion control. The default window size (object store requests per second) is 2000; in other words, we skip the "slow start" phase. The controller uses the Go [`rate.Limiter`](https://pkg.go.dev/golang.org/x/time/rate), which implements the token-bucket algorithm. To put it simply: - every successful request widens the window (per second client rate-limit) - every rate-limited response reduces the window size by a backoff factor (0.5 by default, so it will halve) - when the limit has been reached, the querier will be delayed from making further requests until tokens are available	2 years ago
J Stickler	d0545bff2d	[docs] Create top level Send data section, part 2 (#10247) ... Part of the database information architecture Epic # 8710 Which issue(s) this PR fixes: Second half of work for issue # 8741 that was started in PR #10192 When closed, fixes #8741 Move the following files under Send data: ./sources/clients/promtail/_index.md ./sources/clients/promtail/configuration.md ./sources/clients/promtail/gcplog-cloud.md ./sources/clients/promtail/installation.md ./sources/clients/promtail/logrotation/_index.md ./sources/clients/promtail/pipelines.md ./sources/clients/promtail/scraping.md ./sources/clients/promtail/stages/_index.md ./sources/clients/promtail/stages/cri.md ./sources/clients/promtail/stages/decolorize.md ./sources/clients/promtail/stages/docker.md ./sources/clients/promtail/stages/drop.md ./sources/clients/promtail/stages/json.md ./sources/clients/promtail/stages/labelallow.md ./sources/clients/promtail/stages/labeldrop.md ./sources/clients/promtail/stages/labels.md ./sources/clients/promtail/stages/limit.md ./sources/clients/promtail/stages/logfmt.md ./sources/clients/promtail/stages/match.md ./sources/clients/promtail/stages/metrics.md ./sources/clients/promtail/stages/multiline.md ./sources/clients/promtail/stages/output.md ./sources/clients/promtail/stages/pack.md ./sources/clients/promtail/stages/regex.md ./sources/clients/promtail/stages/replace.md ./sources/clients/promtail/stages/static_labels.md ./sources/clients/promtail/stages/template.md ./sources/clients/promtail/stages/tenant.md ./sources/clients/promtail/stages/timestamp.md ./sources/clients/promtail/troubleshooting/_index.md This PR also - Revises the Clients landing page to clarify supported clients. - Updates the metadata (descriptions, weights) - Adds aliases to redirect from the old URLs. - Updates cross-references broken by the move/renaming. - A few other small fixes (headings, typos, etc.) **Special notes for your reviewer**: Please review the updates to the Clients landing page (now called Send Data) as I've made some extensive edits to try to clarify recommended/supported clients. The file is docs/sources/send-data/_index.md --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Charles Korn	611481b6cf	Upgrade dskit (#10249)	2 years ago
Charles Korn	8e6a5433ac	Upgrade dskit and remove dependency on weaveworks/common (#10220) ... **What this PR does / why we need it**: This PR upgrades `github.com/grafana/dskit` removes Loki's dependency on `github.com/weaveworks/common`. The changes in dskit, apart from the migration (https://github.com/grafana/dskit/pull/356), are: * https://github.com/grafana/dskit/pull/347 * https://github.com/grafana/dskit/pull/349, which required small changes in `pkg/canary/reader/reader.go` and `pkg/canary/writer/push.go` * https://github.com/grafana/dskit/pull/352 * https://github.com/grafana/dskit/pull/354 **Which issue(s) this PR fixes**: (none) **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Dylan Guedes	1221658fc5	Introduces cache to TSDB postings (#9621) ... **What this PR does / why we need it**: Introduces a new interface for fetching TSDB postings and implements a cached TSDB postings. The new postings interface is named `PostingsReader` and contains two implementations: a simple one, which is identical to the former Postings behavior, and a new one, that caches postings using the existing `index_queries_cache_config`. By default, the simple/former implementation is used. The new cached implementation can be configured in the following way: ```yaml storage_config: tsdb_shipper: cache_postings: true ``` In the future, we'll remove the `cache_postings` flag and change the behavior to cache postings by default.	2 years ago
Mohamed-Amine Bouqsimi	2fcde18207	Fix query_timeout warning messages (#10083) ... **What this PR does / why we need it**: This PR fixes the warning messages that are logged in relation to the timeouts migration. The messages reference the per-tenant config, but the config in question in the `AdjustForTimeoutsMigration` function is the global `limits_config`. This is a source of confusion for users. **Which issue(s) this PR fixes**: Fixes #9801	2 years ago
Danny Kopping	8fa4134ef9	Update dskit (#10091)	2 years ago
Christian Haudum	ab3d58a265	Add initial delay before starting flush loop (#9995) ... **What this PR does / why we need it**: Delay the first flush operation by up to 0.8x the flush time period. This will ensure that multiple ingesters started at the same time do not flush at the same time. Flushing at the same time can cause concurrently writing the same chunk to object storage, which in AWS S3 leads to being rate limited. Additionally, add a +/- 1% jitter to each flush check interval to prevent flushing at the same time for reasons mentioned above. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
J Stickler	cc2aeac860	Break up and move Best Practices topic. (#9912) ... I realized that I should have moved the Best Practices topic as part of #8711, since most of the content is best practices for using labels. This PR: * Splits the file into two parts (Best practices for labels and best practices for configuration) * Moves the two files to Get Started and Configure, respectively. * Updates page weights for both sections as needed.	2 years ago
Vladyslav Diachenko	be1a9c4ec5	Revert "updated to the latest version of dskit" (#9953) ... Reverts grafana/loki#9920	2 years ago
Vladyslav Diachenko	3243676f4c	updated to the latest version of dskit (#9920) ... **What this PR does / why we need it**: Updated Loki with the latest version of dskit **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Travis Patterson	cb2b3ca9ea	Add limit to the series volume endpoint (#9833) ... To protect index gateways, this PR adds a per-tenant limit to the `series_volume` endpoint so the accumulation of series can't grow without bound.	2 years ago
J Stickler	7797a98074	TOC Sync: Move configuration files under Configure (#9879) ... **Which issue(s) this PR fixes**: Fixes #8713 **Special notes for your reviewer**: Replaces PR [#8817](https://github.com/grafana/loki/pull/8817), which was stale and badly out of date. Part of the Database products table of contents alignment project. Moves configuration files from Configuration to new Configure heading in TOC. Updates front matter for the topic. Fixes links broken by the move. Updates the MakeFile to point to the new Configure DOCS_SOURCES_PATH.	2 years ago