560 Commits (8976709f24dc9baa603d030fa23484382f7ba767)

Author	SHA1	Message	Date
Eng Zer Jun	982b16b404	refactor: replace min/max helpers with built-in min/max (#16438) ... Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	11 months ago
Ed Welch	558a0839b2	chore: log X-Query-Tags in the frontend executing query log message (#16407)	11 months ago
Trevor Whitney	5b16c0be7d	refactor: allow sample expressions to have multiple extractors (#16224)	11 months ago
Trevor Whitney	417d0a5252	chore: add multi-variant query support to LogQL (#16196)	11 months ago
benclive	6d691ac38d	fix(approx_topk): Map approx_topk operation in all cases (#16131)	11 months ago
Christian Haudum	1187a73667	chore: LogQL syntax and AST simplification and cosistency (#16028) ... This PR aims for making the grammar file (syntax.y) for the LogQL syntax more readable and the AST and its expressions more consistent in naming. You could say these are cosmetics, and yes they are, but if they can help developers better and faster understand the code it's a win. Personally I found the grammar yacc file intimidating until I understood that it was way more complicated that it needed to be. I hope it enables future additions to the syntax to be easier to implement. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Christian Haudum	bebd1532b7	ci: Fix `gochecksumtype` linter error (#16096) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Christian Haudum	3888866616	fix(engine): Fix `Walk()` function implementation on various `Expr` implementations (#16033) ... The `Walk(f WalkFn)` implementation expects to first visit the current node and then invoke `Walk(f)` on all its children if they are not `nil`. This also fixes the `checkIntervalLimit(syntax.SampleExpr, time.Duration)` function, which did not visit the expression if it was a `*ConcatSampleExpr`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	12 months ago
Karsten Jeschkies	83339cbde0	fix: Adjust with offset in last, first and quantile over time queries. (#15915) ... **What this PR does / why we need it**: The query sharding for `last_over_time`, `first_over_time` and `quantile_over_time` would not adjust start and end by the passed `offset` for the underlying range query. This change introduces tests and fixes the adjustment.	12 months ago
Sven Grossmann	9182addea6	fix: prevent fast ascii comparison if char is not letter (#15774)	1 year ago
Karsten Jeschkies	bef20431cb	perf(approx_topk): Reduce memory usage of HyperLogLog in approx_topk. (#15559) ... The count min sketch data structure backing the new approx_topk aggregation uses HyperLogLog (HLL) to track the actual cardinality of the aggregated vector. We were using the sparse version of the HLL. However, that resulted in memory and allocation overhead.	1 year ago
Christian Haudum	7033091bc9	feat: Detect fields based on per-tenant configuration and put them into structured metadata at ingest time (#15188) ... This PR introduces a new feature that allows for extraction of "fields" into structured metadata at ingest time. Fields can either be regular labels, structured metadata keys, or keys from `logfmt` or `json` formatted log lines. The fields are defined in a per-tenant configuration as `map[string][]string`, where the key is the target key of the structured metadata, and the value is the list of source fields in given order and the order given above. Example configuration: ```yaml limits_config: discover_generic_fields: fields: trace_id: - "trace_id" - "TRACE_ID" - "traceID" - "TraceID" org_id: - "org_id" - "tenant_id" - "user_id" ``` While parsing of log lines comes with a certain penalty at ingest time (increased latency and CPU usage on distributors), the idea is to extract certain fields once to avoid parsing the log lines every single time at query time. This is mainly useful in combination with bloom filters. **JSONpath support** Should the value of the config map support jsonpath expression, such as ``` limits_config: discover_generic_fields: fields: ticket_id: - "message.ticket.id" ``` Where the log line looks like this: ```json {"timestamp": 1733128051000, "message": {"ticket": {"id": "2024-d95f87018cdb1f10"}}} ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ed Welch	a3808b6b1d	fix: structured metadata not always passed down for processing deletes (#15526) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Karsten Jeschkies	04994ca821	perf: Improve `approx_topk` performance by reducing allocations. (#15450) ... **What this PR does / why we need it**: The metrics slice should keep a constant amount of memory by removing the smallest item when the maximum of labels is reached. ``` › benchstat before.log after.log goos: linux goarch: amd64 pkg: github.com/grafana/loki/v3/pkg/logql cpu: AMD Ryzen 7 3700X 8-Core Processor │ before.log │ after.log │ │ sec/op │ sec/op vs base │ _HeapCountMinSketchVectorAdd-16 839.0m ± 3% 418.9m ± 2% -50.07% (p=0.000 n=10) │ before.log │ after.log │ │ B/op │ B/op vs base │ _HeapCountMinSketchVectorAdd-16 72.58Mi ± 0% 12.10Mi ± 0% -83.33% (p=0.000 n=10) │ before.log │ after.log │ │ allocs/op │ allocs/op vs base │ _HeapCountMinSketchVectorAdd-16 4073.9k ± 0% 116.9k ± 0% -97.13% (p=0.000 n=10) ``` **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] Title matches the required conventional commits format, see [here](https://www.conventionalcommits.org/en/v1.0.0/) - **Note** that Promtail is considered to be feature complete, and future development for logs collection will be in [Grafana Alloy](https://github.com/grafana/alloy). As such, `feat` PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	1 year ago
Cyril Tovena	2150fbcf52	feat: line_format bypass template execution if possible (#15411) ... Co-authored-by: Ed Welch <ed@edjusted.com>	1 year ago
Trevor Whitney	17f197206d	fix: allow comparison against zero bytes (#15217) ... Co-authored-by: George Robinson <george.robinson@grafana.com>	1 year ago
Cyril Tovena	079779e8ba	chore: Improve containsLower performance using quick rejection (#15076)	1 year ago
Paul Rogers	f2c2a22bdc	chore: Preparation for incoming static code analysis CI check (#15164) ... Co-authored-by: Danny Cooper <danny.cooper@grafana.com>	1 year ago
Eng Zer Jun	062d1e9ca4	refactor: replace experimental `maps` and `slices` with stdlib (#15051) ... Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	1 year ago
Callum Styan	1ea49e31ea	chore: remove minor dead code that was missed in a PR review (#15094) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Shantanu Alshi	2ae1ead16a	chore(metric_extraction): Optimize labels result (#15068)	1 year ago
Cyril Tovena	90c5dbf0d3	chore: Revert syntax module (#14831)	1 year ago
Cyril Tovena	28c9645d10	chore: Introduce logql/syntax go module (#14790)	1 year ago
Karsten Jeschkies	7b53f20f70	feat: Introduce shardable probabilistic topk for instant queries. (#14243) ... Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Christian Haudum	0760095a3e	chore: Add bool to metrics.go log line whether query used bloom filters (#14645) ... Better observability of which queries use bloom filters for chunk filtering. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ashwanth	82fb2f0ae2	fix(label_format): renamed label should use ParsedLabel category (#14515) ... Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	1 year ago
Vladyslav Diachenko	08b1a9080b	fix(logql): updated JSONExpressionParser not to unescape extracted values if it is JSON object. (#14499) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Trevor Whitney	b5462b6639	chore: remove copied loop vars (#14383)	1 year ago
Christian Haudum	833bf0def6	fix(logql): Fix panic in json parsing when using empty array index (#14393) ... This commit fixes a panic that occurs with the following json parser expression: ``` | json keys[""] ``` when the log line is the following: ```json {"keys": ["a", "b", "c"]} ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Callum Styan	4d6992982d	feat: add query user and query source to "executing query" log lines (#14320) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Ravishankar	2d4792a54f	fix: allow rename of structuremetadata labels (#13955) ... Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	04f87ca933	chore: Lint in prep for golangci compatible with Go 1.23 (#14289)	1 year ago
Christian Haudum	14b468b236	chore: Add `has_labelfilter_before_parser` to metrics.go log line (#14244) ... The value of this field indicates that the query could potentially benefit from bloom filters on structured metadata, if the queried field is a structured metadata field. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ashwanth	900751c3bb	fix: report correct status code for metric and log queries in metrics.go (#12102)	1 year ago
Karsten Jeschkies	48e4a346b8	chore: Remove unused quantile merge step evaluator. (#14160)	1 year ago
benclive	78b275bf10	fix: Propagate query stats from quantile & topk queries (#13831) ... Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Sven Grossmann	63e84b476a	fix(aggregated-metrics): correctly create logfmt string (#14124)	1 year ago
Robert Fratto	53cfef3bca	chore(storage/bloom/v1): add ExtractTestableLabelMatchers (#14119) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	1 year ago
Paul Rogers	f226b598a4	chore: Final batch of linting cleanup for Go 1.23 and newer golangci (#14120)	1 year ago
Kason Braley	c5083c7f1f	perf(logql): Micro-optimizations for IP filter (#14072) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	1 year ago
Ravishankar	55e374e85e	fix: skipping label if it contains special symbol (#14068) ... Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	1 year ago
Paul Rogers	18cef217fc	chore: Linting updates in prep for Go 1.23 (#14071)	1 year ago
Paul Rogers	02205f480e	chore: Remove call to deprecated rand.Seed() (#14041)	1 year ago
benclive	a0c75987a2	fix: Propagate headers/warnings/stats from quantile downstreams (#13881)	1 year ago
Travis Patterson	3bf7fa9f15	fix: sanatize structured metadata at query time (#13983)	1 year ago
Ashwanth	ae938d06d9	fix: record datasample queries are limited query type (#13930)	1 year ago
Johan Lindvall	40f4f14791	fix: Improve execution speed for queries with label filters (#13922) ... Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	1 year ago
Trevor Whitney	c6ab6b31e8	chore: remove initial metric aggregation experiment (#13729)	1 year ago
Ned Andreev	5ef83a741b	fix: panic when parsing and extracting JSON key values (#13790)	1 year ago
Marco Pracucci	b88583da7d	feat: upgrade prometheus (#13671) ... Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago