2985 Commits (e0ca67dd4563e41c57b2f1409ef235b76b2a1a6e)

Author	SHA1	Message	Date
Christian Haudum	e0ca67dd45	fix(query engine): Include lines with ts equal to end timestamp of the query range when executing range aggregations (#13448) ... **Background** When performing range vector aggregations, such as `count_over_time({env="dev"}[1h])`, the query range is divided into multiple steps at which the aggregation operation (e.g. counting the log lines) is evaluated. Each step starts at `current step - step interval` and ends at `current step`, as depicted in the following chart. The select range for the logs is extended by the `step interval` into the past, in order to select logs for calculating the first step.  However, the select range for logs is `start` inclusive and `end` exclusive (written as `[start, end)`), but the evaluation of the steps for the range aggregation is `start` exclusive and `end` inclusive (written as `(start, end]`). This leads to the problem that the very first timestamp at the beginning of the select range and the very last timestamp at the end of the select range are not included in the range aggregation. The "missing" last timestamp is not a problem, because a) in an instant query it is not supposed to be included anyway because of the `[start, end)` inclusivity of the query range and b) in a range query the last point of the previous step will be part of the next step evaluation. **Issue** The missing first timestamp, however, gets problematic when executing an instant query and the log timestamps are exactly at the start of the query range. This can happen when the query is split in the query frontend into multiple smaller time ranges, e.g. `1h`, `30m`, ... Since the sub queries are executed independently on the queriers, all logs that have a timestamp exactly a multiple of the split interval, e.g. 00:00, 01:00, 02:00, ... for a 1h interval, are dismissed and therefore missing in the query result over the full time range of the original query. **Fix** In order to avoid the missing logs that have a timestamp a multiple of the split interval in instant queries, we need to adjust the query range for logs to also include the `end` timestamp (written as `[start, end]`). This is done by adding a "leap nanosecond" to the `end` timestamp of the log select range. This ensures that the included `end` timestamp of the step evaluation is also included in the log selection. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Cyril Tovena	790135bd7d	feat: RF1 improves replication stategy to support 1 replica (#13469)	11 months ago
Cyril Tovena	6ea83b45b5	fix: Properly reset wal segment writer (#13468)	11 months ago
Owen Diehl	4220737a52	feat(exporter): include boolean values in limit exporter (#13466)	11 months ago
Cyril Tovena	e08b4a7f88	feat: Pattern ingesters add a limiter for high eviction rate (#13464)	11 months ago
Christian Haudum	845359d4ab	test: Add range aggregation test for instant queries (#13447) ... This PR adds test coverage for the `streamRangeVectorIterator`, which is used by instant queries to evaluate the single step of the query. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	11 months ago
Christian Haudum	5f4b8fc9e4	fix: Fix panic in BloomStore initialisation (#13457) ... The initialisation of the `BloomStore` service failed with a panic if the index gateways were configured to use `ring` mode. This happened, because the bloom store requires to wire up the configuration for the shipper store, see https://github.com/grafana/loki/blob/1a508b0aa/pkg/loki/modules.go#L864-L867 At this time, the index gateway ring manager wasn't initialized yet, leading to a `nil` pointer dereference. This PR fixes the issue by adding the index gateway ring as dependency of the bloom store so it gets initialised first. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Cyril Tovena	7c86e651ac	fix: Fixes pattern pruning stability (#13429)	11 months ago
Salva Corts	2affa4862e	refactor(blooms): Improve task progress logging in builder (#13394)	11 months ago
Yandi Lee	304db100b3	fix(gateway client): Fix nil pointer dereference panic when using ruler in ring mode (#13436) ... Fixes gateway client return with nil pointer panic when using ring mode for ruler service. `clients.dnsProvider` will only be init when using `simple mode` here: 1834065704/pkg/indexgateway/client.go (L184) Service like `ruler` using `ring mode` will run into panic `nil pointer reference` during restarting or exiting:	11 months ago
Christian Haudum	5ebd24dfb2	chore: Convert iterators in pattern module to `v2.Iterator` interface (#13407) ... This is first follow up to #13273 It changes the iterator from the pattern module to use the `v2.CloseIterator` to avoid re-implementing the iterator logic. With this change, the function `Error()` on the pattern iterator is renamed to `Err()` to conform with the `v2.Iterator` interface. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Ashwanth	221491c123	fix(log results cache): include pipeline wrapper disabled in cache key (#13328)	11 months ago
Christian Haudum	ce71f1cf95	feat(api)!: Fail log queries when executed on instant query endpoint (#13421) ... ### What this PR does / why we need it **Background** A log selector expression is a LogQL expression that returns logs, in contrast to a sample expressions, which returns metrics (samples). The simplest form of log selector expressions are label matchers, e.g. `{env="prod"}`. **Change** This PR changes the behaviour of Loki so that the instant query endpoint `/api/v1/query` does not allow sending a log selector expression as query any more. Instead, it returns a status code 400 (Bad Request) with the error message **"log queries are not supported as an instant query type, please change you query to a range query type"**. **Why** Previously this API endpoint allowed these types of log queries, but returned inconsistent results, which where a major cause for confusion. Returning a concise error helps the user understand that they likely selected the wrong query type in Grafana when executing the query. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
benclive	6da364e94f	chore: Set memberlist func for ingester-rf1 init (#13424)	11 months ago
Cyril Tovena	1fa5127797	fix: Remove query size limit for detected fields (#13423)	11 months ago
Cyril Tovena	0076bbdb42	chore: Refactor storage interface for rf1 (#13415)	11 months ago
benclive	0f9088bc89	chore: use read-only index store for ingester RF1 (#13419)	11 months ago
Cyril Tovena	bc01e6fd64	feat: Drain uses different tokenizer based on log format (#13384)	11 months ago
Christian Haudum	69b805d8ff	test: Fix flaky congestion test (#13410) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Salva Corts	47f6ea53fc	feat(loki): Do not enforce max-query-bytes-read and max-querier-bytes-read in limited tripperware (#13406)	11 months ago
Cyril Tovena	1e33ae1546	chore: Improve performance of structured metadata (#13404)	11 months ago
benclive	973aa2d7ba	feat: Use prefixed WAL storage path in Object Storage (#13377)	11 months ago
Vladyslav Diachenko	33bd390f19	chore: adjusted SegmentWriter to the reader conversion (#13342) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: George Robinson <george.robinson@grafana.com> Co-authored-by: Ben Clive <ben.clive@grafana.com>	11 months ago
George Robinson	7f35179cd3	feat: Ingester RF-1 (#13365)	11 months ago
benclive	e612dd3dfe	feat: exclude in pattern tokens (#13397)	11 months ago
benclive	8c8454b9db	feat: exclude and from creating new tokens in patterns (#13395)	11 months ago
benclive	a1efafdad2	perf: Limit tokens per pattern to 128 (#13376)	11 months ago
Salva Corts	11e19763d0	fix(blooms): Delete outdated metas during planning (#13363)	11 months ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273) ... Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	3ac231728e	fix(blooms): Suppress error from resolving server addresses for blocks (#13385) ... Instead of returning the error, `FilterChunks` returns the full, unfiltered list of chunks. This makes sure that queries do not fail if no bloom gateway instances are available. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
benclive	bf1d6e36f7	perf: Gather aggregate per-line and per-tenant metrics for Drain patterns (#13368)	11 months ago
Owen Diehl	04bc3a423c	fix(blooms): ensure tokenizer cache is reset between series (#13370)	11 months ago
Salva Corts	0602b90072	refactor(blooms): Better task naming (#13361)	11 months ago
Salva Corts	23f17f8947	refactor(blooms): Fix and improve build metrics (#13360) ... Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Ashwanth	eb8a363066	fix(sharding): use without() grouping when merging `avg_over_time` shard results (#12176)	11 months ago
Ashwanth	a457c5d171	fix(retry): fix retries when using protobuf encoding (#13316)	11 months ago
JordanRushing	e28c15f56c	fix: support multi-zone ingesters when converting global to local limits for streams in limiter.go (#13321) ... Signed-off-by: JordanRushing <rushing.jordan@gmail.com>	11 months ago
benclive	5ef18cff92	perf: Prune unused pattern branches (#13329)	11 months ago
Salva Corts	fcb9b283ba	fix(blooms): Remove unused arg (#13343)	11 months ago
Vladyslav Diachenko	19c050926e	feat: convert WalSegmentWriter to io.ReadSeeker (#13340) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	11 months ago
Salva Corts	d0f56eeb0a	fix(blooms): Minor fixes and improvements for testing in dev (#13341)	11 months ago
Vladyslav Diachenko	5ae5b31b1f	fix: stream ownership check (#13314) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	11 months ago
Dylan Guedes	4eb45cc588	refactor: Introduce context cause to our code (#13224) ... Introduce the usage of Go new `WithCause` functions to make our context cancellation errors more descriptive.	11 months ago
Owen Diehl	f661593cae	refactor(blooms): more easily testable day partitioning (#13331)	11 months ago
Christian Haudum	3b0502ddd3	chore(blooms): Various minor code cleanups (#13332) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Paul Rogers	40ee766724	feat: Collect duplicate log line metrics (#13084)	11 months ago
Salva Corts	8ebce009f3	refactor(blooms): Implement retry in builder (#13306)	11 months ago
Salva Corts	0c289a83df	refactor(blooms): Do not use sleeps on integration test (#13308)	11 months ago
Christian Haudum	b1adfce5d0	chore(mempool): Wait for buffer instead of returning error (#13298) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	66f97b2aec	fix(blooms): Fix panic in bloom gateway (#13303) ... The slice of block queriers can contain `nil` values, which causes nip pointer dereference when calling `Close()` on them. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago