620 Commits (aa66baec978cd5c24702c19494784cf33bf8b513)

Author	SHA1	Message	Date
Robert Fratto	ba4bd5dc95	chore: introduce efficient byte range reader in new engine (#19024) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	8 months ago
benclive	0b9b4c1ee5	chore: Support multi-tenancy in indexobj builder (#18880)	8 months ago
Ashwanth	827b1f2df4	chore(engine): add support for range metric queries (#18892)	9 months ago
Robert Fratto	f311846aa3	chore(logql/bench): align data directory with production layout (#18889)	9 months ago
Ashwanth	88c8b44f2d	chore(dataobj): read non-projected predicate columns (#18797)	9 months ago
benclive	e57877e208	chore: Refactor metastore Updater (breaking change) (#18821)	9 months ago
Robert Fratto	2ca8e58b0e	chore(dataobj): add ability to buffer pending sections to disk (#18780) ... Signed-off-by: Robert Fratto <robertfratto@gmail.com>	9 months ago
Periklis Tsirakidis	0ec355c349	chore(dataobj): Use storage prefix for metastore index lookups (#18555)	9 months ago
Robert Fratto	d4b027ae0c	chore(dataobj): return Object on Builder.Flush (#18622)	9 months ago
dannyc-grafana	ecdd613a0c	chore: Autofix to make nosec comments also readable by semgrep (#18664)	9 months ago
Robert Fratto	9129d8cdd0	chore(engine): use columnar logs.Reader for DataObjScan (#18577)	10 months ago
Ashwanth	3421e31a1f	chore(bench): add instant metric queries (#18583) ... Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Christian Haudum	db6c6d974b	chore: Remove `slicelabels` build tag from Makefiles (#18518) ... Since the codebase now consistently uses the new `stringlabels` labels API after merging #18490 the build tags in the Makefile can safely be removed again. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
benclive	077e4196ef	chore: Add indexes to LogQL benchmarks (#18500)	10 months ago
Robert Fratto	86c57c8616	chore: consistently use new labels.Labels API (#18490)	10 months ago
Robert Fratto	dfa6bc3267	chore(ci): add workflow for manually running LogQL correctness tests (#18486)	10 months ago
benclive	517c317c97	chore: Integrate indexes with the new query engine (#18427)	10 months ago
Ashwanth	05d569cb4e	chore(bench_test): skip tests if new engine does not support feature (#18488)	10 months ago
Sandeep Sukhani	d379de501f	fix: while validating logql expression, detect and validate expression in label_replace (#18470)	10 months ago
Periklis Tsirakidis	bcf24e4cfb	chore(dataobj): Use indexpointer sections in metastore reads and writes (#18448)	10 months ago
Christian Haudum	d9f7d785ff	chore(engine): Sort entries in dataobj sections by timestamp descending (#18367) ... Sorting sections by `timestamp DESC` first and `streamID ASC` second allows for more efficient reading (by section) for BACKWARD log queries. - This change breaks FORWARD log queries. Which is ok for the time being. We will add support later again when columnar reading of the dataobj is introduced. - This change renders existing dataobj in storage incompatible. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Christian Haudum	d0e059d0de	chore: Report allocations in logql benchmarks (#18399) ... Quite self-explanatory. Reporting allocs allows for comparing allocation count/size across different benchmark runs. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Trevor Whitney	c0c2265283	feat: return metric query partial results for drilldown (#18286)	10 months ago
Christian Haudum	b1fc685397	chore: Extend instrumentation of dataobject uploader (#18292) ... These changes provide better visibility into the dataobj uploader. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	10 months ago
Ashwanth	5f3300a084	chore(bench_test): correctly integrate new query engine + misc changes (#18172)	10 months ago
Christian Haudum	b316d52612	chore(engine): Use shard information from query frontend in the new query engine (#17792) ... **What this PR does / why we need it**: This PR introduces the sharding to the new query engine to be able to test the engine using the exsiting Loki architecture with query frontend, query scheduler, and queriers. **Note, this is only an interim solution used for validating and testing.** By design, the first phase of the query engine implementation does only local execution of queries without sharding or time-based splitting. However, for testing the results of this first phase, we can utilise dual-ingestion (writing both chunks/tsdb and data objects in a way that sharding and splitting is performed in the query frontend using the existing middlewares. On the queriers themselves, the sub-queries are parsed and planned using the new logical and physical planner. If the query can successfully be planned, it will be executed by the new engine, otherwise it falls back to the old engine. **Shard size considerations** While performing time-splitting in the frontend works for data objects as well, sharding by information from TSDB is not directly mappable to data objects. The default target shard size in TSDB is 600MB (decompressed), whereas target size of data objects is 1GB compressed or roughly 10-15GB uncompressed. However, individual logs sections of a data object have a target size of 128MB, which is roughly 0.9-1.2GB. That is 1.5-2x larger than the TSDB target shard size. So when using the sharding calculation from TSDB, it would over-shard for data object sections, which is likely acceptable for testing and good enough for proving that local execution with the new engine works. **How does sharding with data objects in this PR work?** The query frontend passes down the calculated shards as part of the query parameters of the serialised sub-request. The logical planner on the querier stores the Shard annotation on the `MAKETABLE` alongside the stream selector. This is then used by the physical planner to filter out only the relevant sections of the resolved data objects from the metastore lookup. During exeuction, only readers for the relevant sections are initialised when performing the `DataObjScan`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	11 months ago
Ed Welch	970ea61143	chore: Revert "chore: enforce max-query-series when returning shard results" (#18156)	11 months ago
Ed Welch	9addc62660	chore: enforce max-query-series when returning shard results (#17991)	11 months ago
Christian Haudum	153bbbcf8a	fix(jsonparser): Fix possible JSON log line corruption caused by `json` parser on query path (#18056) ... This PR fixes two bugs in the json parser: 1. A bug that caused the original log line to be mutated by the json parser when the extracted field was a nested field and collided with an already existing label name, such as in line `{"app":{"id": 123, "name": "bar"}}` and label `app_name="foo"`. 2. A bug that caused a wrong json path for the extracted field in the same conditions as 1, such as `[]string{"app", "name_extracted"}` for `app_name`, instead of `[]string{"app", "name"}`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Karsten Jeschkies	c2573fea5a	refactor(stringlabels): Support stringlabels in util, engine and logql. (#17998) ... This is a next step of support Prometheus `stringlabels` implementation in Loki. It adds support in the `util`, `engine` and `logql` packages. The tests should compile with build tag `stringlabels`.	11 months ago
Christian Haudum	632ab581f6	chore(ci): Fix linter issues (#17973) ... Use named field in struct literals, which is enforced by the linter. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	11 months ago
Charles Korn	545fb1597c	refactor(stringlabels): Support stringlabels in `logql/log` package (#17838) ... This is another step towards supporting Prometheus' stringlabels implementation in Loki. It adds support in the `logql/log` package. Part of https://github.com/grafana/loki/issues/17122 The tests should now compile and pass with `-tags stringlabels`. --- Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	11 months ago
Oleg Zaytsev	c8a15f451c	refactor: Use OTel tracing library (#17859) ... Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	11 months ago
Trevor Whitney	3d2286a840	chore: implement multi-variant extractor that can return multiple samples (#17149)	12 months ago
benclive	7ca7c3bf51	fix: Implement approx-topk function on querier (#17816)	12 months ago
Robert Fratto	a237444f0d	refactor(dataobj): invert dependency between dataobj and sections (#17762) ... Originally, the dataobj package was a higher-level API around sections. This design caused it to become a bottleneck: * Implementing any new public behaviour for a section required bubbling it up to the dataobj API for it to be exposed, making it tedious to add new sections or update existing ones. * The `dataobj.Builder` pattern was focused on constructing dataobjs for storing log data, which will cause friction as we build objects around other use cases. This PR builds on top of the foundation laid out by #17704 and #17708, fully inverting the dependency between dataobj and sections: * The `dataobj` package has no knowledge of what sections exist, and can now be used for writing and reading generic sections. Section packages now create higher-level APIs around the abstractions provided by `dataobj`. * Section packages are now public, and callers interact directly with these packages for writing and reading section-specific data. * All logic for a section (encoding, decoding, buffering, reading) is now fully self-contained inside the section package. Previously, the implementation of each section was spread across three packages (`pkg/dataobj/internal/encoding`, `pkg/dataobj/internal/sections/SECTION`, `pkg/dataobj`). * Cutting a section is now a decision made by the caller rather than the section implementation. Previously, the logs section builder would create multiple sections. For the most part, this change is a no-op, with two exceptions: 1. Section cutting is now performed by the caller; however, this shouldn't result in any issues. 2. Removing the high-level `dataobj.Stream` and `dataobj.Record` types will temporarily reduce the allocation gains from #16988. I will address this after this PR is merged.	12 months ago
Owen Diehl	ccb65f9670	feat(engine, dataobj): logql bench wiring for new engine [non-fork] (#17627) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	1 year ago
Trevor Whitney	577d501839	refactor: extractors can return multiple samples (#17064) ... Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	1 year ago
Karsten Jeschkies	bf6f130cdf	refactor(stringlabels): Support stringlabels in chunkenc (#17263)	1 year ago
Christian Haudum	6b4430afd0	chore(engine): Collect pipeline records into Loki result (#17447) ... This PR adds the missing piece between existing HTTP handler and the new execution engine, which converts the Apache Arrow records from the vectorized, columnar, pull-based iterator pipeline into the existing (row-based) Loki response format. Since the engine only supports logs queries at the moment, the transformation logic does not take possible metric query results into account. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ed Welch	87922da28d	fix: improve structured metadata label normalization performance (#17332)	1 year ago
Karsten Jeschkies	a864f8b986	refactor(stringlabels): Support stringlabels in log.StreamPipeline (#17216)	1 year ago
Shantanu Alshi	f70dd18f82	chore(deps): Use updated prometheus/otlptranslate (#17227)	1 year ago
benclive	9e68fb01f4	feat(dataobj): Add query stats collection to the dataobj readers (#17128)	1 year ago
Christian Haudum	cfe3675295	fix(ci): Fix deprecation errors from staticcheck linter (#17210) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Robert Fratto	7e71ded446	chore(logql/bench): store proper timestamps in generated chunks (#17132)	1 year ago
Robert Fratto	bd4331363b	chore: align benchmark results between chunks and dataobjs (#17127)	1 year ago
Robert Fratto	88beefb02a	fix(logql): Fix inconsistency with parsed field short circuiting (#17104) ... PR #8724 originally changed the behaviour of LogQL so that the first extracted field takes precedence over any later extracted field with the same name. However, this PR had two subtle issues due to the caching that parse stages perform for key lookups: * The precedence logic did not always apply to duplicate fields within the same parse stage (notably for logfmt). * The caching behaviour could incorrectly cause stages to permanently ignore fields for future log lines where that stage should otherwise perform extraction. Additionally, once structured metadata was introduced, it was incorrectly being flagged as an "extracted field." Combined with the caching behaviour described above, this means that parsed fields only take precedence over structured metadata if the first log line encountered by the engine doesn't have that field also set as structured metadata. This can be demonstrated with the following scenario: 1. A log line from a stream that uses structured metadata for `trace_id` is encountered first. This (incorrectly) causes the trace_id field to be ignored in all parse stages. 2. All later log lines from streams without structured metadata can no longer extract or filter on a parsed `trace_id` for the lifetime of the query, due to the caching from step 1. The fix is two-fold: * Decouple skipping previously extracted fields from the logic which caches interned keys. * Only mark parsed labels as extracted fields (previously all labels). Because the check for whether a field is already extracted is now called more times, the list of extracted fields has been updated to a map to allow for faster checks.	1 year ago
benclive	8e421186df	chore: Collect basic stats from dataobj queries (#17111)	1 year ago
Trevor Whitney	940a067a50	chore: apply series limit per variant for multi-variant queries (#16887)	1 year ago