581 Commits (bfa2a1df65e233ce99fccd01df2130f440ebd166)

Author	SHA1	Message	Date
Christian Haudum	6b4430afd0	chore(engine): Collect pipeline records into Loki result (#17447) ... This PR adds the missing piece between existing HTTP handler and the new execution engine, which converts the Apache Arrow records from the vectorized, columnar, pull-based iterator pipeline into the existing (row-based) Loki response format. Since the engine only supports logs queries at the moment, the transformation logic does not take possible metric query results into account. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Ed Welch	87922da28d	fix: improve structured metadata label normalization performance (#17332)	9 months ago
Karsten Jeschkies	a864f8b986	refactor(stringlabels): Support stringlabels in log.StreamPipeline (#17216)	9 months ago
Shantanu Alshi	f70dd18f82	chore(deps): Use updated prometheus/otlptranslate (#17227)	9 months ago
benclive	9e68fb01f4	feat(dataobj): Add query stats collection to the dataobj readers (#17128)	9 months ago
Christian Haudum	cfe3675295	fix(ci): Fix deprecation errors from staticcheck linter (#17210) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Robert Fratto	7e71ded446	chore(logql/bench): store proper timestamps in generated chunks (#17132)	9 months ago
Robert Fratto	bd4331363b	chore: align benchmark results between chunks and dataobjs (#17127)	9 months ago
Robert Fratto	88beefb02a	fix(logql): Fix inconsistency with parsed field short circuiting (#17104) ... PR #8724 originally changed the behaviour of LogQL so that the first extracted field takes precedence over any later extracted field with the same name. However, this PR had two subtle issues due to the caching that parse stages perform for key lookups: * The precedence logic did not always apply to duplicate fields within the same parse stage (notably for logfmt). * The caching behaviour could incorrectly cause stages to permanently ignore fields for future log lines where that stage should otherwise perform extraction. Additionally, once structured metadata was introduced, it was incorrectly being flagged as an "extracted field." Combined with the caching behaviour described above, this means that parsed fields only take precedence over structured metadata if the first log line encountered by the engine doesn't have that field also set as structured metadata. This can be demonstrated with the following scenario: 1. A log line from a stream that uses structured metadata for `trace_id` is encountered first. This (incorrectly) causes the trace_id field to be ignored in all parse stages. 2. All later log lines from streams without structured metadata can no longer extract or filter on a parsed `trace_id` for the lifetime of the query, due to the caching from step 1. The fix is two-fold: * Decouple skipping previously extracted fields from the logic which caches interned keys. * Only mark parsed labels as extracted fields (previously all labels). Because the check for whether a field is already extracted is now called more times, the list of extracted fields has been updated to a map to allow for faster checks.	9 months ago
benclive	8e421186df	chore: Collect basic stats from dataobj queries (#17111)	9 months ago
Trevor Whitney	940a067a50	chore: apply series limit per variant for multi-variant queries (#16887)	9 months ago
Robert Fratto	300c1e859e	chore(logql/bench): Fix bug in bench data generation and update naming conventions (#17051) ... This PR fixes a bug found in generating test data for chunks: MemChunks weren't being closed properly before flushing, which caused the last block of every chunk to be omitted from the flush. As a result, benchmark tests against chunks were inaccurate, causing many metric queries to appear 5-10x faster with chunks than they actually are. Additionally, this PR expands on #17006 with another naming convention change: the type of query is now included in the benchmark name to allow developers to only run benchmarks of a specific query type (metric, log). PR #17006 neglected to reflect the naming convention change in the benchmark TUI, which stopped it from working. This is now fixed to match the latest naming convention introduced in this PR, and the TUI will now work again.	9 months ago
Christian Haudum	1705f98a9f	chore(engine): Wire up new execution engine (#17032) ... This PR connects the already existing components of the new query engine and wires them up in the querier API. The feature flag `-querier.engine.enable-v2-engine` controls whether supported queries should be executed using the new engine. Note that this implementation currently does not execute any queries. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Robert Fratto	5b787c0f5b	chore(logql/bench): use benchstat naming conventions for benchmarks (#17006)	9 months ago
Christian Haudum	0a3230f457	chore(ast): Update WalkFn signature to return bool (#16989) ... The boolean return value indicates whether walking/traversing should be continued with child nodes. This PR does not change any existing traversal behaviour. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Trevor Whitney	507d269832	feat: return the jsonpath of json detected fields (#16861)	10 months ago
Robert Fratto	9be95a3e8c	chore(dataobj): reduce memory overhead of merging logs stripes (#16833)	10 months ago
Trevor Whitney	23800f6100	chore: convert multi-variant queries from global to per-tenant config (#16718)	10 months ago
Cyril Tovena	1b57297a46	feat(logql): Add LogQL Benchmark Suite (#16434)	10 months ago
Paul Rogers	27431b7e7e	chore: Linting update for new golangci (#16572)	10 months ago
Trevor Whitney	40d7b979a4	chore: implement multi-variant queries (#16426)	10 months ago
Eng Zer Jun	982b16b404	refactor: replace min/max helpers with built-in min/max (#16438) ... Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	10 months ago
Ed Welch	558a0839b2	chore: log X-Query-Tags in the frontend executing query log message (#16407)	11 months ago
Trevor Whitney	5b16c0be7d	refactor: allow sample expressions to have multiple extractors (#16224)	11 months ago
Trevor Whitney	417d0a5252	chore: add multi-variant query support to LogQL (#16196)	11 months ago
benclive	6d691ac38d	fix(approx_topk): Map approx_topk operation in all cases (#16131)	11 months ago
Christian Haudum	1187a73667	chore: LogQL syntax and AST simplification and cosistency (#16028) ... This PR aims for making the grammar file (syntax.y) for the LogQL syntax more readable and the AST and its expressions more consistent in naming. You could say these are cosmetics, and yes they are, but if they can help developers better and faster understand the code it's a win. Personally I found the grammar yacc file intimidating until I understood that it was way more complicated that it needed to be. I hope it enables future additions to the syntax to be easier to implement. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	bebd1532b7	ci: Fix `gochecksumtype` linter error (#16096) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Christian Haudum	3888866616	fix(engine): Fix `Walk()` function implementation on various `Expr` implementations (#16033) ... The `Walk(f WalkFn)` implementation expects to first visit the current node and then invoke `Walk(f)` on all its children if they are not `nil`. This also fixes the `checkIntervalLimit(syntax.SampleExpr, time.Duration)` function, which did not visit the expression if it was a `*ConcatSampleExpr`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Karsten Jeschkies	83339cbde0	fix: Adjust with offset in last, first and quantile over time queries. (#15915) ... **What this PR does / why we need it**: The query sharding for `last_over_time`, `first_over_time` and `quantile_over_time` would not adjust start and end by the passed `offset` for the underlying range query. This change introduces tests and fixes the adjustment.	11 months ago
Sven Grossmann	9182addea6	fix: prevent fast ascii comparison if char is not letter (#15774)	12 months ago
Karsten Jeschkies	bef20431cb	perf(approx_topk): Reduce memory usage of HyperLogLog in approx_topk. (#15559) ... The count min sketch data structure backing the new approx_topk aggregation uses HyperLogLog (HLL) to track the actual cardinality of the aggregated vector. We were using the sparse version of the HLL. However, that resulted in memory and allocation overhead.	1 year ago
Christian Haudum	7033091bc9	feat: Detect fields based on per-tenant configuration and put them into structured metadata at ingest time (#15188) ... This PR introduces a new feature that allows for extraction of "fields" into structured metadata at ingest time. Fields can either be regular labels, structured metadata keys, or keys from `logfmt` or `json` formatted log lines. The fields are defined in a per-tenant configuration as `map[string][]string`, where the key is the target key of the structured metadata, and the value is the list of source fields in given order and the order given above. Example configuration: ```yaml limits_config: discover_generic_fields: fields: trace_id: - "trace_id" - "TRACE_ID" - "traceID" - "TraceID" org_id: - "org_id" - "tenant_id" - "user_id" ``` While parsing of log lines comes with a certain penalty at ingest time (increased latency and CPU usage on distributors), the idea is to extract certain fields once to avoid parsing the log lines every single time at query time. This is mainly useful in combination with bloom filters. **JSONpath support** Should the value of the config map support jsonpath expression, such as ``` limits_config: discover_generic_fields: fields: ticket_id: - "message.ticket.id" ``` Where the log line looks like this: ```json {"timestamp": 1733128051000, "message": {"ticket": {"id": "2024-d95f87018cdb1f10"}}} ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ed Welch	a3808b6b1d	fix: structured metadata not always passed down for processing deletes (#15526) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Karsten Jeschkies	04994ca821	perf: Improve `approx_topk` performance by reducing allocations. (#15450) ... **What this PR does / why we need it**: The metrics slice should keep a constant amount of memory by removing the smallest item when the maximum of labels is reached. ``` › benchstat before.log after.log goos: linux goarch: amd64 pkg: github.com/grafana/loki/v3/pkg/logql cpu: AMD Ryzen 7 3700X 8-Core Processor │ before.log │ after.log │ │ sec/op │ sec/op vs base │ _HeapCountMinSketchVectorAdd-16 839.0m ± 3% 418.9m ± 2% -50.07% (p=0.000 n=10) │ before.log │ after.log │ │ B/op │ B/op vs base │ _HeapCountMinSketchVectorAdd-16 72.58Mi ± 0% 12.10Mi ± 0% -83.33% (p=0.000 n=10) │ before.log │ after.log │ │ allocs/op │ allocs/op vs base │ _HeapCountMinSketchVectorAdd-16 4073.9k ± 0% 116.9k ± 0% -97.13% (p=0.000 n=10) ``` **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] Title matches the required conventional commits format, see [here](https://www.conventionalcommits.org/en/v1.0.0/) - **Note** that Promtail is considered to be feature complete, and future development for logs collection will be in [Grafana Alloy](https://github.com/grafana/alloy). As such, `feat` PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	1 year ago
Cyril Tovena	2150fbcf52	feat: line_format bypass template execution if possible (#15411) ... Co-authored-by: Ed Welch <ed@edjusted.com>	1 year ago
Trevor Whitney	17f197206d	fix: allow comparison against zero bytes (#15217) ... Co-authored-by: George Robinson <george.robinson@grafana.com>	1 year ago
Cyril Tovena	079779e8ba	chore: Improve containsLower performance using quick rejection (#15076)	1 year ago
Paul Rogers	f2c2a22bdc	chore: Preparation for incoming static code analysis CI check (#15164) ... Co-authored-by: Danny Cooper <danny.cooper@grafana.com>	1 year ago
Eng Zer Jun	062d1e9ca4	refactor: replace experimental `maps` and `slices` with stdlib (#15051) ... Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	1 year ago
Callum Styan	1ea49e31ea	chore: remove minor dead code that was missed in a PR review (#15094) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Shantanu Alshi	2ae1ead16a	chore(metric_extraction): Optimize labels result (#15068)	1 year ago
Cyril Tovena	90c5dbf0d3	chore: Revert syntax module (#14831)	1 year ago
Cyril Tovena	28c9645d10	chore: Introduce logql/syntax go module (#14790)	1 year ago
Karsten Jeschkies	7b53f20f70	feat: Introduce shardable probabilistic topk for instant queries. (#14243) ... Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Christian Haudum	0760095a3e	chore: Add bool to metrics.go log line whether query used bloom filters (#14645) ... Better observability of which queries use bloom filters for chunk filtering. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ashwanth	82fb2f0ae2	fix(label_format): renamed label should use ParsedLabel category (#14515) ... Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	1 year ago
Vladyslav Diachenko	08b1a9080b	fix(logql): updated JSONExpressionParser not to unescape extracted values if it is JSON object. (#14499) ... Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Trevor Whitney	b5462b6639	chore: remove copied loop vars (#14383)	1 year ago
Christian Haudum	833bf0def6	fix(logql): Fix panic in json parsing when using empty array index (#14393) ... This commit fixes a panic that occurs with the following json parser expression: ``` | json keys[""] ``` when the log line is the following: ```json {"keys": ["a", "b", "c"]} ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago