apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
benclive	17285207f1	chore: Reduce buffer size for bloom lookups (#19926 )	2 months ago
Christian Haudum	1cc97513d7	chore(metastore): Cleanup and reduce log volume (#19889 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
Christian Haudum	97fc8b3a50	chore(metastore): Simplify public API of Metastore (#19886 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 months ago
benclive	c31e25762e	chore: Fix metadata bloom check with multiple predicates (#19389 )	3 months ago
benclive	bd5a065487	chore: Two small fixes for new engine (#19241 )	4 months ago
Christian Haudum	457ec0956d	chore(dataobj-index): Handle special cases in the section lookup of the metastore (#19203 ) There are three edge cases that this PR improves: * No table (toc) paths could be resolved: return early with empty result * No index paths could be resolved: return early with empty result * No relevant sections could be resolved: return with empty result instead of with an error Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
Sophie Waldman	98b411a649	chore(linter): Fix issues flagged by updated golangci-lint version (#19206 )	4 months ago
benclive	941864cec9	chore: Init objects in metastore in parallel (#19175 )	4 months ago
benclive	dfb60dcaf0	fix: Metastore timerange filter (#19171 )	4 months ago
benclive	3148a77966	chore: improve metastore Section calls (#19157 )	4 months ago
Christian Haudum	4c81ebc7c3	chore(dataobj): Add option to set target row limit for pages (#19128 ) This PR introduces an option for the column builder to limit the number of rows in a page. It can be set for logs object and index objects separately using `-dataobj-consumer.max-page-rows` and `-dataobj-index-builder.max-page-rows` respectively. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 months ago
benclive	b786095aea	chore: Read multi-tenant objects in query-engine (#19104 )	4 months ago
benclive	5d428b2ac4	chore: Fix panic for queries with unsupported predicate types (#19086 )	4 months ago
George Robinson	f60f818bdd	feat: build multi-tenant data objects (#19034 )	4 months ago
benclive	2d30c6a5a4	chore: Implement building multi-tenant indexes from multi-tenant objects (#18959 )	4 months ago
George Robinson	3bab0ff5fe	fix: backoff cannot be canceled via ctx (#19041 )	5 months ago
benclive	0b9b4c1ee5	chore: Support multi-tenancy in indexobj builder (#18880 )	5 months ago
benclive	d93a44141f	chore: Make metastore event synchronous (#18928 )	5 months ago
benclive	e57877e208	chore: Refactor metastore Updater (breaking change) (#18821 )	5 months ago
Robert Fratto	2ca8e58b0e	chore(dataobj): add ability to buffer pending sections to disk (#18780 ) Signed-off-by: Robert Fratto <robertfratto@gmail.com>	5 months ago
Periklis Tsirakidis	0ec355c349	chore(dataobj): Use storage prefix for metastore index lookups (#18555 )	5 months ago
Robert Fratto	1c06a8fc11	chore(dataobj): fix compilation error following #18622 (#18738 )	5 months ago
Robert Fratto	d4b027ae0c	chore(dataobj): return Object on Builder.Flush (#18622 )	5 months ago
Ashwanth	3faec029bf	chore(metastore): include traceID in metastore logs (#18660 )	5 months ago
benclive	613116e371	chore: Fix empty matchers returning all entries in Metastore Section lookup (#18642 )	5 months ago
Robert Fratto	517b59f087	chore: update github.com/thanos-io/objstore fork (#18620 )	6 months ago
benclive	077e4196ef	chore: Add indexes to LogQL benchmarks (#18500 )	6 months ago
Robert Fratto	86c57c8616	chore: consistently use new labels.Labels API (#18490 )	6 months ago
benclive	517c317c97	chore: Integrate indexes with the new query engine (#18427 )	6 months ago
Periklis Tsirakidis	bcf24e4cfb	chore(dataobj): Use indexpointer sections in metastore reads and writes (#18448 )	6 months ago
Christian Haudum	b1fc685397	chore: Extend instrumentation of dataobject uploader (#18292 ) These changes provide better visibility into the dataobj uploader. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Christian Haudum	b316d52612	chore(engine): Use shard information from query frontend in the new query engine (#17792 ) What this PR does / why we need it: This PR introduces the sharding to the new query engine to be able to test the engine using the exsiting Loki architecture with query frontend, query scheduler, and queriers. Note, this is only an interim solution used for validating and testing. By design, the first phase of the query engine implementation does only local execution of queries without sharding or time-based splitting. However, for testing the results of this first phase, we can utilise dual-ingestion (writing both chunks/tsdb and data objects in a way that sharding and splitting is performed in the query frontend using the existing middlewares. On the queriers themselves, the sub-queries are parsed and planned using the new logical and physical planner. If the query can successfully be planned, it will be executed by the new engine, otherwise it falls back to the old engine. Shard size considerations While performing time-splitting in the frontend works for data objects as well, sharding by information from TSDB is not directly mappable to data objects. The default target shard size in TSDB is 600MB (decompressed), whereas target size of data objects is 1GB compressed or roughly 10-15GB uncompressed. However, individual logs sections of a data object have a target size of 128MB, which is roughly 0.9-1.2GB. That is 1.5-2x larger than the TSDB target shard size. So when using the sharding calculation from TSDB, it would over-shard for data object sections, which is likely acceptable for testing and good enough for proving that local execution with the new engine works. How does sharding with data objects in this PR work? The query frontend passes down the calculated shards as part of the query parameters of the serialised sub-request. The logical planner on the querier stores the Shard annotation on the `MAKETABLE` alongside the stream selector. This is then used by the physical planner to filter out only the relevant sections of the resolved data objects from the metastore lookup. During exeuction, only readers for the relevant sections are initialised when performing the `DataObjScan`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	7 months ago
benclive	e6fe98c7c3	chore: Emit metastore event to kafka after object build (#17842 )	8 months ago
Robert Fratto	a237444f0d	refactor(dataobj): invert dependency between dataobj and sections (#17762 ) Originally, the dataobj package was a higher-level API around sections. This design caused it to become a bottleneck: * Implementing any new public behaviour for a section required bubbling it up to the dataobj API for it to be exposed, making it tedious to add new sections or update existing ones. * The `dataobj.Builder` pattern was focused on constructing dataobjs for storing log data, which will cause friction as we build objects around other use cases. This PR builds on top of the foundation laid out by #17704 and #17708, fully inverting the dependency between dataobj and sections: * The `dataobj` package has no knowledge of what sections exist, and can now be used for writing and reading generic sections. Section packages now create higher-level APIs around the abstractions provided by `dataobj`. * Section packages are now public, and callers interact directly with these packages for writing and reading section-specific data. * All logic for a section (encoding, decoding, buffering, reading) is now fully self-contained inside the section package. Previously, the implementation of each section was spread across three packages (`pkg/dataobj/internal/encoding`, `pkg/dataobj/internal/sections/SECTION`, `pkg/dataobj`). * Cutting a section is now a decision made by the caller rather than the section implementation. Previously, the logs section builder would create multiple sections. For the most part, this change is a no-op, with two exceptions: 1. Section cutting is now performed by the caller; however, this shouldn't result in any issues. 2. Removing the high-level `dataobj.Stream` and `dataobj.Record` types will temporarily reduce the allocation gains from #16988. I will address this after this PR is merged.	8 months ago
aarogoss	ab65ac5e8e	feat: refactors kafka/client to improve metric collection (#17308 )	9 months ago
Christian Haudum	e297c51d20	chore(metastore): Use constants for label names (#17048 ) This PR contains improvements for: 1. Make label names contants 2. Use `strconv.FormatInt` for encoding timestamp into label value (equivalent to `strconv.ParseInt` when decoding) 3. Use `labels.New` to construct labels, rather then string formatting Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Christian Haudum	ba61aa88de	chore(engine): Integrate metastore into physical query planning (#17016 ) This PR integrates the data object lookup using the metastore into the query planning. To achieve that, I extended the `Metastore` interface with an additional function that not only returns the data object paths, but also the stream IDs of streams that match the given label matchers. ```go type Metastore interface { ... // StreamsIDs returns object store paths and stream IDs for all matching objects for the given matchers between [start,end] StreamIDs(ctx context.Context, start, end time.Time, matchers ...*labels.Matcher) ([]string, [][]int64, error) ... } ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Robert Fratto	9be95a3e8c	chore(dataobj): reduce memory overhead of merging logs stripes (#16833 )	10 months ago
aarogoss	a57a80ea32	feat(metastore): Implement Labels() and Values() on ObjectMetastore (#16734 )	10 months ago
benclive	91bab5e7f2	chore(metastore): Fix stream read condition (#16631 )	10 months ago
benclive	475d25f459	feat(metastore): Add initial interface for Metastore (#16604 )	10 months ago
Cyril Tovena	4bc95c0c68	feat(dataobj): Add methods for querying data objects metadata (#16190 )	11 months ago
Cyril Tovena	5aa9e47d1c	fix(dataobj): Fixes timerange matching for dataobj files (#16222 )	11 months ago
Cyril Tovena	fa790e1ac8	feat(metastore): Implement listing data object for a given tenant and timerange (#16180 )	11 months ago
Owen Diehl	d9892b4481	fix(metastore): empty object corrections (#16147 )	11 months ago
Cyril Tovena	1fa952dbf2	refactor(dataobj): Consolidate configuration and add querier support (#16144 )	11 months ago
benclive	b7f6bb359b	chore(dataobj): Add iterator for metastores (#16114 )	11 months ago
benclive	7f32de6cf2	chore(dataobj): Refactor processor & builder to separate concerns (#16055 )	11 months ago
benclive	ca4c025ad0	chore(dataobj): Create initial dataobj builder (#16011 ) Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	11 months ago

49 Commits (9d0d1084e39fdfd4a58d701b13ff755f4e5f36eb)