469 Commits (965f19a3b95f529b51b28aadfaddaf33be7f50d4)

Author	SHA1	Message	Date
lpugoy	965f19a3b9	lambda-promtail: Add support for WAF logs in S3 (#10416) ... **What this PR does / why we need it**: Adds support for WAF logs in S3 in lambda-promtail **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Shantanu Alshi	97d52d948d	Upgrade thanos objstore (#10451)	2 years ago
Danny Kopping	2c84959901	Distributor: add ingester append timeouts error (#10456) ... **What this PR does / why we need it**: Failing to send samples to ingesters because the request exceeded its timeout is a very clear signal that ingesters are unable to keep up with demand. In an incident today we saw that ingesters' push latencies were increased sharply by an expensive regex query which was starving other goroutines of time on CPU. This new alert `loki_distributor_ingester_append_timeouts_total` will give us a high-signal metric which we can use for alerting.	2 years ago
Periklis Tsirakidis	e5c425eef6	Allow using the forget button when access via the internal server (#10221)	2 years ago
Vladyslav Diachenko	00a3c5b026	renamed `non indexed labels` feature to `structured metadata` (#10432) ... Loki team has made a decision to rename the feature because the new name is more precise. --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Salva Corts <salva.corts@grafana.com>	2 years ago
Shantanu Alshi	3e80e7ed9d	Fix changelog entry (#10406) ... Changes the changelog entry as per suggestion in #10395	2 years ago
Shantanu Alshi	f1bbdc51dc	Remove split_queries_by_interval and forward_headers_list (#10395) ... …range config Removed split_queries_by_interval and forward_headers_list from query_range config The fields were deprecated and hence going to be removed in the next major version **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Bryan Boreham	a32cbeb37a	indexshipper/storage: fix race conditions (#10314) ... **What this PR does / why we need it**: There were many race conditions reported if you run `go test -race` on this package. Google's 'singleflight' package is much neater and makes all the race warnings go away. **Which issue(s) this PR fixes**: Relates to #8586 **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - NA Documentation added - NA Tests updated - [x] `CHANGELOG.md` updated - NA If the change is worth mentioning in the release notes, add `add-to-release-notes` label - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - NA For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Shantanu Alshi	145319e5f5	Remove experimental enable ruler api toggle (#10380) ... Remove `-experimental.ruler.enable-api` in favour of `-ruler.enable-api`. The former was already deprecated for some time.	2 years ago
Shantanu Alshi	92ba055ae5	Remove ruler.wal-cleaer.period (#10378) ... The config with typo is now replaced by `-ruler.wal-cleaner.period`	2 years ago
Shantanu Alshi	325c332f3e	Remove deprecated s3 sse toggle (#10377) ... This PR removes a legacy and already deprecated boolean setting `-s3.sse-encryption` in favor of the more fine-grain `-s3.sse.*` settings.	2 years ago
Salva Corts	44e1c88aca	Add changelog for non-indexed labels (#10379) ... **What this PR does / why we need it**: We recently implemented a new feature to support attaching and querying non-indexed labels attached to each log line. This PR updates the changelog with entries referencing the main PRs of this feature. --------- Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	2 years ago
Karsten Jeschkies	f8658fdc38	Enable sharding for avg_over_time. (#10373) ... **What this PR does / why we need it**: The range aggregation `avg_over_time` is simply shardable if there's no label reduction. If there is one it can be express as ``` sum by (method) ( sum_over_time( {container="app"} | json | unwrap bytes [$__interval]) ) / sum by (method) ( count_over_time( {container="app"} | json [$__interval] ) ) ``` Note that `sum by`, `sum_over_time` and `count_over_time` are shardable once more. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Ashwanth	dff3622eaa	deprecate old index types and non-object stores (#10341) ... **What this PR does / why we need it**: Deprecates old index types: `aws, aws-dynamo, gcp, gcp-columnkey, bigtable, bigtable-hashed, cassandra, grpc`. Only `tsdb` and `boltdb-shipper` are supported. Deprecates non-object stores. The next major release might not allow the usage of these as `object_store` in `schema_config`: `aws-dynamo, gcp, gcp-columnkey, bigtable, bigtable-hashed, cassandra, grpc.` Deprecates table manager which is not needed once when using `boltdb-shipper` or `tsdb` index type **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Shantanu Alsi	586b9042c6	Upgrade Thanos objstore, dskit and other modules (#10366) ... **What this PR does / why we need it**: Upgrades various common modules including dskit, thanos objstore, prometheus, etc. Fixes errors/ changes resulting due to module upgrades. This is required in order to eventually upgrade BEL dependency in GEL to point to master. **Which issue(s) this PR fixes**: None **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Ashwanth	18482c67d6	compactor: deprecate -boltdb.shipper.compactor. prefix in favor of -compactor. (#10344) ... **What this PR does / why we need it**: Compactor config use `-boltdb.shipper.compactor.` as prefix which is misleading as the config options apply to tsdb index as well. This pr introduces `-compactor.` prefix which can now be used to configure compactor configuration options and deprecates `-boltdb.shipper.compactor.` **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Kaviraj Kanagaraj	c030217e1b	Revert "LogQL: Introduce `distinct` (#8662)" (#10356) ... **What this PR does / why we need it**: This reverts commit 38b298c3fe. We are removing this PR #8662 because [known issue tracked here](https://github.com/grafana/loki/issues/9594) and some issues we facing internally with query splitting and sharding work(hard to parallelize, merging two `distinct` sets can have duplicates, hard to make it work with metric queries, etc). We decided best to remove it before make it to public release. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Ashwanth	cc6634972f	config: deprecates unordered-writes and few unused configs (#10324) ... **What this PR does / why we need it**: Deprecates the following configs: out of order writes will be the default and the only format with the next major release - ingester.unordered-writes these need not be configurable, were added for testing - log.use-buffered - log.use-sync unused - frontend.forward-headers-list **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Ashwanth	470cbbf118	querier: remove query_timeout and engine timeout from querier conf (#10302) ... **What this PR does / why we need it**: Removes `query_timeout` and `engine:timeout` from querier's config section Both these configs have been deprecated in 2.7.0 release in favor of `query_timeout` in `limits_config` which also allows applying per-tenant overrides at runtime. CLI flag `-querier.engine.timeout` is also removed **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [X] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Christian Haudum	e3b25fb36c	Deprecate ruler setting with misleading name (#10322) ... The setting defaults to `0s` and is not supposed to be changed, therefore it should be removed in a future version (e.g. 3.0) of Loki. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Bryan Boreham	cf353bb9a0	Series Index Store: fix race in GetSeries (#10310)	2 years ago
Bryan Boreham	f89a51fde8	Tracing: elide small traces for Stats call (#10308) ... **What this PR does / why we need it**: At work I see a lot of traces getting data dropped due to being tens of megabytes in size. Looking at what was captured, I see many many repeats of these four spans: <img width="1070" alt="image" src="https://github.com/grafana/loki/assets/8125524/4268e664-6910-49bb-8f68-69b139a5a66b"> Events on the parent trace are cheaper and, for small operations like this, just as useful. Where there was an event line I added a `function` field matching the previous span name. For `storeEntry.Stats()` I didn't add an event line as it doesn't seem to do very much. But please correct me; I'm not hugely familiar with the Loki codebase. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - NA Documentation added - NA Tests updated - [x] `CHANGELOG.md` updated - NA If the change is worth mentioning in the release notes, add `add-to-release-notes` label - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - NA For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
loanshark	43e542ee64	Storage: Remove `SignatureVersionv2` from the AWS S3 API request authentication. (#10295) ... AWS S3 stopped accepting requests signed using SigV2 in all regions on June 24, 2019, any requests signed using SigV2 made after this time will fail. * https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingAWSSDK.html#UsingAWSSDK-sig2-deprecation * https://aws.amazon.com/blogs/aws/amazon-s3-update-sigv2-deprecation-period-extended-modified/	2 years ago
William Dumont	a1c1152b79	Promtail: Add additional fields in cloudflare config (#10301) ... **What this PR does / why we need it**: In this PR the cloudflare config is enhanced by adding a custom fields type and by having a new property "additional_types" to give the customer the ability to specify the fields that he wants. This functionality has been added in the grafana agent and therefore should also be added in promtail. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/agent/issues/4860 **Special notes for your reviewer**: **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md`	2 years ago
Danny Kopping	beed298549	Dynamic client-side throttling to avoid object storage rate-limits (GCS only) (#10140) ... **What this PR does / why we need it**: Across the various cloud providers' object storage services, there are different rate-limits implemented. Rate-limits can be imposed under multiple conditions, such as server-side scale up (ramping up from low volume to high, "local" limit), reaching some defined upper limit ("absolute" limit), etc. We cannot know apriori when these rate-limits will be imposed, so we can't set up a client-side limiter to only allow a certain number of requests through per second. Additionally, that would require global coordination between queriers - which is difficult. With the above constraints, I have instead taken inspiration from TCP's [congestion control algorithms](https://en.wikipedia.org/wiki/TCP_congestion_control). This PR implements [AIMD](https://en.wikipedia.org/wiki/Additive_increase/multiplicative_decrease) (Additive Increase, Multiplicative Decrease), which is used in the congestion _avoidance_ phase of congestion control. The default window size (object store requests per second) is 2000; in other words, we skip the "slow start" phase. The controller uses the Go [`rate.Limiter`](https://pkg.go.dev/golang.org/x/time/rate), which implements the token-bucket algorithm. To put it simply: - every successful request widens the window (per second client rate-limit) - every rate-limited response reduces the window size by a backoff factor (0.5 by default, so it will halve) - when the limit has been reached, the querier will be delayed from making further requests until tokens are available	2 years ago
Danny Kopping	a18ca92375	Track effectiveness of hedged requests (#10281) ... **What this PR does / why we need it**: We currently have request hedging implemented in many object store clients. Right now we track how many hedged requests we make, and how many are rate-limited, but nothing actually tells us _how effective_ the hedged requests are. In other words, we don't know how many hedged requests were issued and _won_ (completed before the original).	2 years ago
Vardhaman Surana	e3b7bcb67d	Ruler: add limit parameter in rulegroup (#10109) ... **What this PR does / why we need it**: This PR adds the support for group level **limit** parameter in ruler which was introduced in prometheus by PR: https://github.com/prometheus/prometheus/pull/9260 **Which issue(s) this PR fixes**: Fixes #4439	2 years ago
Shantanu Alsi	b57d260ddc	Upgrade alpine 3.16.5 -> 3.16.7 (#10188) ... **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #10181 Fixes Open SSL vulnerabilities [CVE-2023-1255](https://security.alpinelinux.org/vuln/CVE-2023-1255), [CVE-2023-2650](https://security.alpinelinux.org/vuln/CVE-2023-2650), [CVE-2023-2975](https://security.alpinelinux.org/vuln/CVE-2023-2975), [CVE-2023-3446](https://security.alpinelinux.org/vuln/CVE-2023-3446), [CVE-2023-3817](https://security.alpinelinux.org/vuln/CVE-2023-3817) Alpine Release Notes: [3.16.7](https://www.alpinelinux.org/posts/Alpine-3.15.10-3.16.7-3.17.5-3.18.3-released.html) [3.16.6](https://www.alpinelinux.org/posts/Alpine-3.15.9-3.16.6-3.17.4-3.18.2-released.html) **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Dylan Guedes	1221658fc5	Introduces cache to TSDB postings (#9621) ... **What this PR does / why we need it**: Introduces a new interface for fetching TSDB postings and implements a cached TSDB postings. The new postings interface is named `PostingsReader` and contains two implementations: a simple one, which is identical to the former Postings behavior, and a new one, that caches postings using the existing `index_queries_cache_config`. By default, the simple/former implementation is used. The new cached implementation can be configured in the following way: ```yaml storage_config: tsdb_shipper: cache_postings: true ``` In the future, we'll remove the `cache_postings` flag and change the behavior to cache postings by default.	2 years ago
Mohamed-Amine Bouqsimi	5a7d6789b2	Fix missing alibabacloud ruler storage type (#10026) ... **What this PR does / why we need it**: This PR fixes the missing `alibabacloud` storage type for the ruler. **Which issue(s) this PR fixes**: Fixes #9892 **Special notes for your reviewer**: @chaudum **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Salva Corts <salva.corts@grafana.com>	2 years ago
Francois Gouteroux	2880b5055a	fix: parse expression error message based on record or alert (#10065)	2 years ago
Bruno FERNANDO	a8a349690b	feat(promtail): retrieve BotTags field from cloudflare (#10010) ... **What this PR does / why we need it**: Hey folks 👋 Little contribution here to add a useful log field for cloudflare users. Indeed I add the [BotTags field](https://developers.cloudflare.com/bots/concepts/cloudflare-bot-tags/) which is pretty useful when debugging some specific traffic handled by cloudflare Don't hesitate to reach me if you have any questions Cheers 😉 **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Michel Hollands	9ee1d42720	Update changelog to include 2.7.6 (#10042) ... **What this PR does / why we need it**: Add the details of the 2.7.6 branch to the changelog **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [X] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	2 years ago
Michel Hollands	44ae10eaeb	Add 2.8.3 release to changelog (#10014) ... **What this PR does / why we need it**: Add the PRs of the 2.8.3 release to the changelog --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	2 years ago
Christian Haudum	ab3d58a265	Add initial delay before starting flush loop (#9995) ... **What this PR does / why we need it**: Delay the first flush operation by up to 0.8x the flush time period. This will ensure that multiple ingesters started at the same time do not flush at the same time. Flushing at the same time can cause concurrently writing the same chunk to object storage, which in AWS S3 leads to being rate limited. Additionally, add a +/- 1% jitter to each flush check interval to prevent flushing at the same time for reasons mentioned above. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
J Stickler	aae92b9840	[Docs] Create top-level Setup section (#9974) ... **Which issue(s) this PR fixes**: Fixes #8712 **Special notes for your reviewer**: Move the following files under a new Setup section (with three subsections: Install, Migrate, and Upgrade). - ./sources/installation/sizing/index.md - ./sources/installation/_index.md - ./sources/installation/docker.md - ./sources/installation/helm/_index.md - ./sources/installation/helm/concepts.md - ./sources/installation/helm/configure-storage/index.md - ./sources/installation/helm/install-monolithic/index.md - ./sources/installation/helm/install-scalable/index.md - ./sources/installation/helm/migrate-from-distributed/index.md - ./sources/installation/helm/migrate-to-three-scalable-targets/index.md - ./sources/installation/helm/monitor-and-alert/index.md - ./sources/installation/helm/reference.md - ./sources/installation/helm/upgrade-from-2.x/index.md - ./sources/installation/install-from-source.md - ./sources/installation/istio.md - ./sources/installation/local.md - ./sources/installation/tanka.md Updates cross-references broken by the move, Updates file metadata (descriptions, weights) and adds aliases for redirects to avoid 404s. Probably fixes a couple of typos too.	2 years ago
Vladyslav Diachenko	28e9e15639	[jsonnet] removed deprecated policy.v1beta1 (#9978) ... **What this PR does / why we need it**: in k8s version 1.25 some k8s APIs are deprecated, such as policy.v1beta1 which is moved to policy.v1 **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Travis Patterson	23e59fa4a9	reset pipeline caches when tailing rather than making a new pipeline (#9949) ... Recent query optimizations made pipeline creation more expensive with the assumption that it occurred relatively infrequently. To avoid growing pipline caches without bound, integers recreate pipelines on every push. The combination of these things means that tailing high-volume streams can cause ingester failure. This fix adds the ability to clear pipeline caches so we don't need to create a new one each time.	2 years ago
Travis Patterson	6dc722bc9f	correctly handle unpack when reordering stages (#9936) ... The optimization to reorder stages doesn't take into account the `unpack` stage. When line filters come after unpack stages, they shouldn't be moved.	2 years ago
Christian Rohmann	d04dc31f92	Bump tail dependency to address issue in retry logic (#9915) ... **What this PR does / why we need it**: This bumps the grafana/tail version used, mostly to address an issue with the retry logic (https://github.com/grafana/tail/pull/16) **Which issue(s) this PR fixes**: Fixes #9864 #9877 #9869, maybe others **Special notes for your reviewer**: I did NOT add a note about this dependency bump in the changelog. Let me know if you want me to add that ... or kindly add that line yourself. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Dylan Guedes	bc0069756e	Queriers: Stop emitting spans for every objectstorage call (#9857) ... **What this PR does / why we need it**: Stop emitting spans for every call to our object storages (like S3 or Azure). They are causing every chunk fetching to emit a span. Request metrics will still be reported just fine. **Which issue(s) this PR fixes**: N/A	2 years ago
Ashwanth	de937e72cd	promtail: apply defaults to HTTP client config (#9863) ... **What this PR does / why we need it**: Apply defaults to HTTPClientConfig and validate it. similar to how it's done [here](e9644d8186/vendor/github.com/prometheus/prometheus/config/config.go (L524)) in prometheus. promtail [configuration page](https://grafana.com/docs/loki/latest/clients/promtail/configuration/) incorrectly claims that `follow_redirects` is true, but it is being set to false given that the defaults are not being applied currently **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Karsten Jeschkies	b35bbd80d6	Support content negotiation between query frontend and querier. (#9813) ... **What this PR does / why we need it**: Currently, the querier sends results to the query frontend in JSON which is then decoded to Protobuf. It is more efficient to send the results as Protobuf. This will also allow to extend the results with custom data structures. The change is backwards compatible through content negotiation. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
manohar-koukuntla	999259a4d3	jsonnet: Add TSDB shipper configuration flag (#9790) ... This commit adds the TSDB equivalent of `use_boltdb_shipper` flag to be able to configure `tsdb_shipper` section. Fixes #9789	2 years ago
Ashwanth	f9fc1df4c0	logfmt: add --strict flag to enable strict parsing, perform nostrict parsing by default (#9626) ... **What this PR does / why we need it**: Introduces `--strict` flag for logfmt parser to enable strict parsing. With strict parsing enabled, logfmt parser stops scanning the log line and returns early with an error if it encounters any poorly formatted key=value pair. This pr updates the parser to perform non-strict parsing by default where it skips over the invalid tokens. This offers the flexibility to parse semi-structured log lines. Also adds `--keep-empty` flag to retain standalone keys(with empty value) as labels with value set to empty string **Special notes for your reviewer**: **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [X] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Christian Haudum	a65c99d9bf	Add per-tenant request counter metric to index gateway server (#9797) ... This commit add a counter metric `loki_index_gateway_requests_total` with labels `operation`, `tenant`, `status` for gRPC requests that are served by the index gateway. **What for?** The per-tenant RPS on the index gateway is used to derive the per-tenant shard factor. **Why tracking on the server?** Unlike tracking index gateway RPS on the client side, tracking on the server side does not yield that many series, even in multi-tenant installations with a lot of tenants, because the amount of index gateway instances is relatively small compared to the amount of queriers and frontends. **Special notes for your reviewer**: The previous approach of tracking requests on the client https://github.com/grafana/loki/pull/9781 has been abandoned. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
René Scheibe	954df433e9	Promtail: Fix file descriptor leak (#8987) ... This reduces the amount of open file descriptors per followed journal file from 3 to 2. It also releases file descriptors for removed files (journal vacuuming). The `defaultJournalEntryFunc` is only called once upon startup. It creates a `Journal` instance to only read a single entry to determine the age of the entry referenced in the positions file. Right after, this instance is not required anymore. The actually used `Journal` instance used to follow the journal is created inside `journalTargetWithReader`.	2 years ago
Sandeep Sukhani	3e1f2fc273	caching: do not try to fill the gap in log results cache when the new query interval does not overlap the cached query interval (#9757) ... **What this PR does / why we need it**: Currently, when we find a relevant cached negative response for a logs query, we do the following: * If the cached query completely covers the new query: * return back an empty response. * else: * fill the gaps on either/both sides of the cached query. The problem with filling the gaps is that when the cached query does not overlap at all with the new query, we have to extend the query beyond what the query requests for. However, with the logs query, we have a limit on the number of lines we can send back in the response. So, this could result in the query response having logs which were not requested by the query, which then get filtered out by the [response extractor](b78d3f0552/pkg/querier/queryrange/log_result_cache.go (L299)), unexpectedly resulting in an empty response. For example, if the query was cached for start=15, end=20 and we get a `backwards` query for start=5, end=10. To fill the gap, the query would be executed for start=5, end=15. Now, if we have logs more than the query `limit` in the range 10-15, we would filter out all the data in the response extractor and send back an empty response to the user. This PR fixes the issue by doing the following changes when handling cache hit: * If the cached query completely covers the new query: * return back an empty response[_existing_]. * else if the cached query does not overlap with the new query: * do the new query as requested. * If the new query results in an empty response and has a higher interval than the cached query: * update the cache * else: * query the data for missing intervals on both/either side[_existing_] * update the cache with extended intervals if the new queries resulted in an empty response[_existing_] **Special notes for your reviewer**: We could do further improvements in the handling of queries not overlapping with cached query by selectively extending the queries based on query direction and cached query lying before/after the new query. For example, if the new query is doing `backwards` query and the `cachedQuery.End` < `newQuery.Start`, it should be okay to extend the query and do `cachedQuery.End` to `newQuery.End` to fill the cache since query would first fill the most relevant data before hitting the limits. I did not want to complicate the fix so went without implementing this approach. We can revisit later if we feel we need to improve our caching. **Checklist** - [x] Tests updated - [x] `CHANGELOG.md` updated --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago
Ashwanth	10e55a6359	store: fix incorrect usage of period config in table range (#9754) ... **What this PR does / why we need it**: We are incorrectly taking reference to a loop variable in store init, I overlooked this part as the reference was being taken in the call to `GetIndexTableNumberRange` As a result of this, `TableRange` of each period (which is used by index-shipper) points to the latest `periodConfig`. This in itself is not a problem if the index prefix doesn't change as we use the `periodConfig` only to generate the `tableName`. But if the index prefix changes across periods, we end up losing access to the indexes using the old prefix. **Special notes for your reviewer**: **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Susana Ferreira	35465d0297	Fix instant query summary split stats (#9773) ... **What this PR does / why we need it**: Fix instant query summary statistic's `splits` corresponding to the number of subqueries a query is split into based on `split_queries_by_interval`. * Update rangemapper with a statistics structure to include the number of split queries a query is mapped into. * In the `split_by_range` middleware once the mapped query is returned update the middleware statistics with the number of split queries. This value will then be merged with the statistics of the Loki response. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago