510 Commits (3be5a4576fd0f0dca321e017a637f7a3159c00e5)

Author	SHA1	Message	Date
jackyin	6dd6b65139	fix: ast left cycular reference result in oom (#13501) ... Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	10 months ago
Kaviraj Kanagaraj	94bd3a042a	chore(lint-error): remove unused `stepCount` in `engine.go` (#13709) ... Signed-off-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	10 months ago
Callum Styan	29a37d5dcd	fix: special case the return values from a sharded first/last_over_time query (#13578) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	10 months ago
Ashwanth	65697676e6	chore: do not shard log queries with empty filter (#13568)	10 months ago
Karsten Jeschkies	08615bf751	fix: Return empty vector instead of nil for empty evaluator. (#13485)	11 months ago
Christian Haudum	e0ca67dd45	fix(query engine): Include lines with ts equal to end timestamp of the query range when executing range aggregations (#13448) ... **Background** When performing range vector aggregations, such as `count_over_time({env="dev"}[1h])`, the query range is divided into multiple steps at which the aggregation operation (e.g. counting the log lines) is evaluated. Each step starts at `current step - step interval` and ends at `current step`, as depicted in the following chart. The select range for the logs is extended by the `step interval` into the past, in order to select logs for calculating the first step.  However, the select range for logs is `start` inclusive and `end` exclusive (written as `[start, end)`), but the evaluation of the steps for the range aggregation is `start` exclusive and `end` inclusive (written as `(start, end]`). This leads to the problem that the very first timestamp at the beginning of the select range and the very last timestamp at the end of the select range are not included in the range aggregation. The "missing" last timestamp is not a problem, because a) in an instant query it is not supposed to be included anyway because of the `[start, end)` inclusivity of the query range and b) in a range query the last point of the previous step will be part of the next step evaluation. **Issue** The missing first timestamp, however, gets problematic when executing an instant query and the log timestamps are exactly at the start of the query range. This can happen when the query is split in the query frontend into multiple smaller time ranges, e.g. `1h`, `30m`, ... Since the sub queries are executed independently on the queriers, all logs that have a timestamp exactly a multiple of the split interval, e.g. 00:00, 01:00, 02:00, ... for a 1h interval, are dismissed and therefore missing in the query result over the full time range of the original query. **Fix** In order to avoid the missing logs that have a timestamp a multiple of the split interval in instant queries, we need to adjust the query range for logs to also include the `end` timestamp (written as `[start, end]`). This is done by adding a "leap nanosecond" to the `end` timestamp of the log select range. This ensures that the included `end` timestamp of the step evaluation is also included in the log selection. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	845359d4ab	test: Add range aggregation test for instant queries (#13447) ... This PR adds test coverage for the `streamRangeVectorIterator`, which is used by instant queries to evaluate the single step of the query. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	11 months ago
Christian Haudum	d8cc1cea83	chore: Make dedicated iterator package (#13273) ... Iterators of various types are widely used throughout the Loki code base. With the recent code additions of the bloom filters, a new set of utility functions for iterators emerged in `github.com/grafana/loki/v3/pkg/storage/bloom/v1`. The package defines interfaces for various common types, but also provides implementations to create and compose new iterators that implement these interfaces. This new package uses Go Generics. - However, at the current state, there are multiple iterator interfaces and implementations, which adds cognitive overhead in understanding how they work and which ones should be used. The idea is to unify them into a single iterator lib for Loki. - As a first step towards a single iterator library for the Loki code base, this PR moves the utilities from the `pkg/storage/bloom/v1` package to the `pkg/iter/v2` package. - Second, it changes the existing `EntryIterator` and `SampleIterator` iterators to "inherit" `v2.CloseIterator` in order to expose the same function names. - And lastly, naming conventions of iterator interfaces and structs are unified. The basic iterator interface (defined in `pkg/iter/v2/interface.go`) looks like so: ```go // Usage: // // for it.Next() { // curr := it.At() // // do something // } // if it.Err() != nil { // // do something // } type Iterator[T any] interface { Next() bool Err() error At() T } ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Ashwanth	eb8a363066	fix(sharding): use without() grouping when merging `avg_over_time` shard results (#12176)	11 months ago
Dylan Guedes	4eb45cc588	refactor: Introduce context cause to our code (#13224) ... Introduce the usage of Go new `WithCause` functions to make our context cancellation errors more descriptive.	11 months ago
Dylan Guedes	c1fada9af0	refactor: Remove unnecessary spanlogger usage (#13255) ... Historically, we've been using `spanlogger` to enrich our spans. That isn't the right usage for it (as of now), as they'll be emitting log lines. Our current usage is causing: - Noisy/important log lines less visible - Unnecessary logging volume - Spread of the bad practice. With this PR I'm moving away from `spanlogger` all places where the log message is too bad or nonexistent. That's because these cases are indicating we don't care about that log line at all, and only about the data injected in the span.	11 months ago
Trevor Whitney	467eb1bb1b	feat: collect and serve pre-aggregated bytes and counts (#13020) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	11 months ago
Sven Grossmann	9981e9e40d	fix(orFilters): fix multiple or filters would get wrong filtertype (#13169)	12 months ago
benclive	6df81db978	feat: Support negative numbers in LogQL (#13091)	12 months ago
benclive	f6f8babf83	feat: Add utf8 support to Pattern Lexer to support utf8 chars (#13085)	1 year ago
Karsten Jeschkies	6e4555010e	fix: Introduce feature flag for [last|first]_over_time sharding. (#13067)	1 year ago
Kaviraj Kanagaraj	88e545fc95	feat(cache): Add `Cache-Control: no-cache` support for Loki instant queries. (#12896) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	1 year ago
Karsten Jeschkies	f66172eed1	feat: improve performance of `first_over_time` and `last_over_time` queries by sharding them (#11605) ... Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Owen Diehl	5a643c7115	chore: reduces span footprint + double recording (#12864) ... Trying to remove some cruft from traces.	1 year ago
Owen Diehl	bc78d13d9b	feat(blooms): compute chunks once (#12664) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Shantanu Alshi	587a6d20e9	feat: Detected labels from store (#12441)	1 year ago
Owen Diehl	9c25985b97	feat(blooms): record time spent resolving shards (#12636)	1 year ago
Travis Patterson	11e768726f	fix(query sharding): Generalize avg -> sum/count sharding using existing binop mapper (#12599)	1 year ago
Cyril Tovena	48dae4417c	feat: improve syntax parser for pattern (#12489)	1 year ago
Kaviraj Kanagaraj	e9b6ce9de4	fix(logql): Linefilter behaviour with chained `or` (#12463) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	1 year ago
Travis Patterson	cf04ec1320	feat: Enable warnings in Loki query responses (#12425)	1 year ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Anton Kolesnikov	36c703dae8	feat: add pattern match line filter (#12398)	1 year ago
Shantanu Alshi	5190dda64b	feat(detected_labels): Initial skeleton for the API (#12390) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	1 year ago
Ashwanth	c8c8477c93	chore(stats): add pipeline_wrapper_filtered_lines (#12399)	1 year ago
Owen Diehl	6e1547fe62	fix(blooms): logql shard embeds tsdb (#12396)	1 year ago
Owen Diehl	3922d3864a	fix(tsdb): correctly use bit prefix calculation in tsdb shard matching (#12394)	1 year ago
Anton Kolesnikov	cc941fe42a	chore: refactor line filter MatchType (#12388)	1 year ago
Trevor Whitney	5b4bfa54b6	feat(detected_fields): initial plumbing of new endpoint (#12385)	1 year ago
Travis Patterson	2b8db8b691	fix: attempt non-string label filtering when errors are present (#12378)	1 year ago
Owen Diehl	a36483b731	feat(blooms): bloom integration in query planning (#12208) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Travis Patterson	04398a1562	fix: log state of 'disable pipeline wrappers' (#12345)	1 year ago
Callum Styan	ab17a50b90	fix: don't shard `quantile_over_time` if it's not the top level aggregation for a query (#12326) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Callum Styan	611cb01b70	fix: quantile sharding; fix bug where sharding would always happen (#12283) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Travis Patterson	d175184fe5	chore: remove shards from wrapper signature (#12322)	1 year ago
Travis Patterson	1d99b06514	fix: explicitly serialize and deserialize noop label filters (#12124)	1 year ago
Ashwanth	3b5ed8c3ec	chore: pass shard count to injected pipelines and extractors (#12213)	1 year ago
BiKangNing	75e031d8e8	chore: fix function names in doc strings (#12313) ... Signed-off-by: depthlending <bikangning@outlook.com>	1 year ago
Christian Haudum	863acc0152	chore(blooms): Make block query concurrency configurable (#12292) ... A concurrency of 10 per worker seems to be too high for a reasonable 8GiB memory limit on bloom gateways. So this PR makes the concurrency configurable and uses a default of 4. Memory usage of loading block pages into memory therefore being: ``` worker_concurrency x block_query_concurrency x max_block_page_size x 2 ``` With current defaults: ``` 4 x 4 x 32MiB x 2 = 1GiB ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Travis Patterson	9e66e05e68	chore: make error labels injectable via 'Add' (#12251)	1 year ago
Bryan Boreham	9621d5c5ee	test: Fix race condition in LogQL test (#12247)	1 year ago
Salva Corts	a8aa3f68cb	feat: Bloom filter regexes (#12096)	1 year ago
Danny Kopping	609fdb3214	feat(querier/query-frontend): track and log congestion control latency (#12058) ... Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	1 year ago
Karsten Jeschkies	6578a00481	feat: Support usage trackers for received and discarded bytes. (#11840)	1 year ago
Salva Corts	4fa5148eb5	refactor: Pass query plan down to bloom gateway (#12037)	1 year ago