3001 Commits (f6a94d303444dbf22cf1198f549c9cde070f1bdc)

Author	SHA1	Message	Date
Trevor Whitney	f6a94d3034	fix: detected fields incorrect type bug (#13515)	11 months ago
Salva Corts	e506995e59	fix(blooms): Remove backoff from notify planner (#13506)	11 months ago
Ned Andreev	ac284ca00e	feat: improve placeholder replacement of byte sizes (#13508)	11 months ago
George Robinson	07c3c761ab	chore: Improve test coverage of WAL Manager (#13498)	11 months ago
George Robinson	b92b07d552	chore: Remove unused metrics from Ingester-RF1 (#13499)	11 months ago
Salva Corts	c263a681f8	fix: add logging to empty bloom (#13502)	11 months ago
Owen Diehl	bfa69556af	fix(blooms): skip empty blooms on reads (#13500) ... Read path safeguard to prevent filtering chunks associated with empty blooms while we work on finding the underlying cause of empty blooms in compaction.	11 months ago
Christian Haudum	652ad2455c	perf(blooms): Avoid tiny string allocations for insert cache (#13487) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Trevor Whitney	2c053ee00c	feat: downsample aggregated metrics (#13449)	11 months ago
George Robinson	583f7f3388	chore: Remove un-used errors from WAL Manager (#13496)	11 months ago
George Robinson	aa55c690f9	chore: Small fixes to comments and ErrFull in manager.go (#13494)	11 months ago
George Robinson	bfe97d724f	feat: Add metrics to WAL Manager (#13490)	11 months ago
George Robinson	8f1d12f656	feat: Use WAL Manager (#13491)	11 months ago
Karsten Jeschkies	08615bf751	fix: Return empty vector instead of nil for empty evaluator. (#13485)	11 months ago
Salva Corts	2cc901ad69	refactor(blooms): Apply retention in planner (#13484)	11 months ago
George Robinson	15c8b45d26	feat: WAL Manager (#13428)	11 months ago
Christian Haudum	e0ca67dd45	fix(query engine): Include lines with ts equal to end timestamp of the query range when executing range aggregations (#13448) ... **Background** When performing range vector aggregations, such as `count_over_time({env="dev"}[1h])`, the query range is divided into multiple steps at which the aggregation operation (e.g. counting the log lines) is evaluated. Each step starts at `current step - step interval` and ends at `current step`, as depicted in the following chart. The select range for the logs is extended by the `step interval` into the past, in order to select logs for calculating the first step.  However, the select range for logs is `start` inclusive and `end` exclusive (written as `[start, end)`), but the evaluation of the steps for the range aggregation is `start` exclusive and `end` inclusive (written as `(start, end]`). This leads to the problem that the very first timestamp at the beginning of the select range and the very last timestamp at the end of the select range are not included in the range aggregation. The "missing" last timestamp is not a problem, because a) in an instant query it is not supposed to be included anyway because of the `[start, end)` inclusivity of the query range and b) in a range query the last point of the previous step will be part of the next step evaluation. **Issue** The missing first timestamp, however, gets problematic when executing an instant query and the log timestamps are exactly at the start of the query range. This can happen when the query is split in the query frontend into multiple smaller time ranges, e.g. `1h`, `30m`, ... Since the sub queries are executed independently on the queriers, all logs that have a timestamp exactly a multiple of the split interval, e.g. 00:00, 01:00, 02:00, ... for a 1h interval, are dismissed and therefore missing in the query result over the full time range of the original query. **Fix** In order to avoid the missing logs that have a timestamp a multiple of the split interval in instant queries, we need to adjust the query range for logs to also include the `end` timestamp (written as `[start, end]`). This is done by adding a "leap nanosecond" to the `end` timestamp of the log select range. This ensures that the included `end` timestamp of the step evaluation is also included in the log selection. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Cyril Tovena	790135bd7d	feat: RF1 improves replication stategy to support 1 replica (#13469)	11 months ago
Cyril Tovena	6ea83b45b5	fix: Properly reset wal segment writer (#13468)	11 months ago
Owen Diehl	4220737a52	feat(exporter): include boolean values in limit exporter (#13466)	11 months ago
Cyril Tovena	e08b4a7f88	feat: Pattern ingesters add a limiter for high eviction rate (#13464)	11 months ago
Christian Haudum	845359d4ab	test: Add range aggregation test for instant queries (#13447) ... This PR adds test coverage for the `streamRangeVectorIterator`, which is used by instant queries to evaluate the single step of the query. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	11 months ago
Christian Haudum	5f4b8fc9e4	fix: Fix panic in BloomStore initialisation (#13457) ... The initialisation of the `BloomStore` service failed with a panic if the index gateways were configured to use `ring` mode. This happened, because the bloom store requires to wire up the configuration for the shipper store, see https://github.com/grafana/loki/blob/1a508b0aa/pkg/loki/modules.go#L864-L867 At this time, the index gateway ring manager wasn't initialized yet, leading to a `nil` pointer dereference. This PR fixes the issue by adding the index gateway ring as dependency of the bloom store so it gets initialised first. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Cyril Tovena	7c86e651ac	fix: Fixes pattern pruning stability (#13429)	11 months ago
Salva Corts	2affa4862e	refactor(blooms): Improve task progress logging in builder (#13394)	11 months ago
Yandi Lee	304db100b3	fix(gateway client): Fix nil pointer dereference panic when using ruler in ring mode (#13436) ... Fixes gateway client return with nil pointer panic when using ring mode for ruler service. `clients.dnsProvider` will only be init when using `simple mode` here: 1834065704/pkg/indexgateway/client.go (L184) Service like `ruler` using `ring mode` will run into panic `nil pointer reference` during restarting or exiting:	11 months ago
Christian Haudum	5ebd24dfb2	chore: Convert iterators in pattern module to `v2.Iterator` interface (#13407) ... This is first follow up to #13273 It changes the iterator from the pattern module to use the `v2.CloseIterator` to avoid re-implementing the iterator logic. With this change, the function `Error()` on the pattern iterator is renamed to `Err()` to conform with the `v2.Iterator` interface. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Ashwanth	221491c123	fix(log results cache): include pipeline wrapper disabled in cache key (#13328)	11 months ago
Christian Haudum	ce71f1cf95	feat(api)!: Fail log queries when executed on instant query endpoint (#13421) ... ### What this PR does / why we need it **Background** A log selector expression is a LogQL expression that returns logs, in contrast to a sample expressions, which returns metrics (samples). The simplest form of log selector expressions are label matchers, e.g. `{env="prod"}`. **Change** This PR changes the behaviour of Loki so that the instant query endpoint `/api/v1/query` does not allow sending a log selector expression as query any more. Instead, it returns a status code 400 (Bad Request) with the error message **"log queries are not supported as an instant query type, please change you query to a range query type"**. **Why** Previously this API endpoint allowed these types of log queries, but returned inconsistent results, which where a major cause for confusion. Returning a concise error helps the user understand that they likely selected the wrong query type in Grafana when executing the query. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
benclive	6da364e94f	chore: Set memberlist func for ingester-rf1 init (#13424)	11 months ago
Cyril Tovena	1fa5127797	fix: Remove query size limit for detected fields (#13423)	11 months ago
Cyril Tovena	0076bbdb42	chore: Refactor storage interface for rf1 (#13415)	11 months ago
benclive	0f9088bc89	chore: use read-only index store for ingester RF1 (#13419)	11 months ago
Cyril Tovena	bc01e6fd64	feat: Drain uses different tokenizer based on log format (#13384)	11 months ago
Christian Haudum	69b805d8ff	test: Fix flaky congestion test (#13410) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Salva Corts	47f6ea53fc	feat(loki): Do not enforce max-query-bytes-read and max-querier-bytes-read in limited tripperware (#13406)	11 months ago
Cyril Tovena	1e33ae1546	chore: Improve performance of structured metadata (#13404)	11 months ago
benclive	973aa2d7ba	feat: Use prefixed WAL storage path in Object Storage (#13377)	11 months ago
Vladyslav Diachenko	33bd390f19	chore: adjusted SegmentWriter to the reader conversion (#13342) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: George Robinson <george.robinson@grafana.com> Co-authored-by: Ben Clive <ben.clive@grafana.com>	11 months ago
George Robinson	7f35179cd3	feat: Ingester RF-1 (#13365)	11 months ago
benclive	e612dd3dfe	feat: exclude in pattern tokens (#13397)	11 months ago
benclive	8c8454b9db	feat: exclude and from creating new tokens in patterns (#13395)	11 months ago
benclive	a1efafdad2	perf: Limit tokens per pattern to 128 (#13376)	11 months ago
Salva Corts	11e19763d0	fix(blooms): Delete outdated metas during planning (#13363)	11 months ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273) ... Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	3ac231728e	fix(blooms): Suppress error from resolving server addresses for blocks (#13385) ... Instead of returning the error, `FilterChunks` returns the full, unfiltered list of chunks. This makes sure that queries do not fail if no bloom gateway instances are available. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
benclive	bf1d6e36f7	perf: Gather aggregate per-line and per-tenant metrics for Drain patterns (#13368)	11 months ago
Owen Diehl	04bc3a423c	fix(blooms): ensure tokenizer cache is reset between series (#13370)	11 months ago
Salva Corts	0602b90072	refactor(blooms): Better task naming (#13361)	11 months ago
Salva Corts	23f17f8947	refactor(blooms): Fix and improve build metrics (#13360) ... Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago