2817 Commits (a0f358fcc8295d93ee38b67738e8d90045c50dab)

Author	SHA1	Message	Date
Christian Haudum	a0f358fcc8	fix(blooms): Fully deduplicate chunks from FilterChunkRef responses (#12807) ... This PR aims for full de-duplication of chunks and series from filter requests from the index gateway to the bloom gateway. Whenever we merge/de-duplicate slices, the inputs need to be sorted. It appears that the Removals (chunks) from the v1.Output are not guaranteed to be sorted. When comparing ShortRefs, both From, Through, and Checksum need to be used. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	1665e853a0	fix(blooms): dont break iterator conventions (#12808) ... Followup to #12806 which exposes skipped pages more explicitly than as an error. * refactors skip logic for bloom pages that are too large * s/Seek/LoadOffset/ for LazyBloomIter * removes unused code	1 year ago
Owen Diehl	bc78d13d9b	feat(blooms): compute chunks once (#12664) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Owen Diehl	eac56224b8	feat(blooms): limit bloom size during creation (#12796)	1 year ago
Jonas L. B	c0113db4e8	fix(promtail): Handle docker logs when a log is split in multiple frames (#12374)	1 year ago
Salva Corts	76ba24e3d8	fix(blooms): Reset error on LazyBloomIter.Seek (#12806)	1 year ago
Ned Andreev	40479029d7	feat: Generic logline placeholder replacement and tokenization (#12799) ... This code allows us to preprocess generic logs and replace highly variable dynamic data (timestamps, IPs, numbers, UUIDs, hex values, bytesizes and durations) with static placeholders for easier pattern extraction and more efficient and user-friendly matching by the Drain algorithm. Additionally, there is logic that splits generic log lines into discrete tokens that can be used with Drain for better results than just naively splitting the logs on every whitespace. The tokenization here handles quote counting and emits quoted strings as a part of the same token. On the other side, it also handles likely JSON logs without any white spaces in them better, by trying to split `{"key":value}` pairs (without actually parsing the JSON). All of this is done without using regular expressions and without actually parsing the log lines in any specific format. That's why it works very efficiently in terms of CPU usage and allocations, and should handle all log formats and unformatted logs equally well.	1 year ago
Michel Hollands	33e82ec133	fix: remove unused parameter causing lint error (#12801) ... Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	1 year ago
benclive	7b8533e435	feat: Add step param to Patterns Query API (#12703)	1 year ago
Christian Haudum	3bf2d1fea0	fix(blooms): Deduplicate filtered series and chunks (#12791) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	4cc3a73172	test: Add mutex and setter/getter for addresses in test (#12793)	1 year ago
Owen Diehl	da8ee7e7f2	chore(blooms): Instrument source bytes added to blooms directly (#12782)	1 year ago
Trevor Whitney	af09f53a33	ci: release workflows get build image from makefile (#12778)	1 year ago
Owen Diehl	fb9b0e80a7	chore(blooms): adds more instrumentation to block building (#12779)	1 year ago
Christian Haudum	ecefb49508	fix(blooms): Fix a regression introduced with #12774 (#12776) ... This PR fixes a regression introduced with #12774 where series where wrongly identified as "skipped" even though there were part of blocks. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	c36b1142c7	fix(blooms): Correctly return unfiltered chunks for series that are not mapped to any block (#12774) ... This PR fixes a conceptual mistake in the code of resolving blocks on the index gateways. Currently, a series does not resolve to any block is discarded instead of being kept for the response. This change adds the chunks of the skipped series to the bloom querier response. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Trevor Whitney	ae180d6e07	fix(nix): lambda-promtail vendor hash (#12763)	1 year ago
Trevor Whitney	210ea93a69	feat: add detected-fields command to logcli (#12739)	1 year ago
Shantanu Alshi	587a6d20e9	feat: Detected labels from store (#12441)	1 year ago
Paul Rogers	c178cc62df	test: Data race updates for memchunk tests (#12752)	1 year ago
dependabot[bot]	ff2063bcaf	chore(deps): bump golang.org/x/net from 0.17.0 to 0.23.0 in /pkg/push (#12694) ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Periklis Tsirakidis <periklis@redhat.com>	1 year ago
Christian Haudum	5540c92d50	perf(blooms): Resolve bloom blocks on index gateway and shard by block address (#12720) ... This pull request changes how data is sharded across bloom gateways. Currently, chunks are grouped by series and shared by the fingerprint of the series across the available bloom gateways using jumphash algorithm. This however, leads to over-querying bloom blocks, because bloom blocks have consecutive fingerprint ranges, whereas sharding keys are evenly distributed across keyspace. This PR changes the sharding in the way that bloom blocks for series are already resolved on the index gateways and that their address is used for sharding data. This has the advantage that the grouped series can be mapped to the correct bloom blocks on the client side. Sending the block ref along with the grouped series to the bloom gateway allows for efficient querying of the data because each bloom gateway therefore owns exactly 1/nth of the blocks. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Trevor Whitney	f68d1f7faf	fix: make detected fields work for both json and proto (#12682)	1 year ago
Ed Welch	4c563f7823	docs: unhide compactor GRPC config (#12587) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: DylanGuedes <djmgguedes@gmail.com>	1 year ago
benclive	5ae33d19ff	test: Add new tests for Drain pattern matching (#12655)	1 year ago
Dylan Guedes	402d1d7c48	perf: TSDB: Add fast-path to `inversePostingsForMatcher` (#12679)	1 year ago
Cyril Tovena	19bfef48cb	feat: Add pattern ingester support in SSD mode (#12685)	1 year ago
Func	4a05964d55	fix: Use to the proper config names in warning messages (#12114) ... Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Seiya	0eba448fc7	fix: crrect initialization of a few slices (#12674) ... Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Michel Hollands	6dce98870d	fix: Fix the lokitool imports (#12673) ... Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	1 year ago
Michel Hollands	7b7d3d4cd2	feat: add lokitool (#12166) ... Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	1 year ago
洪阿南	0d1ebebd3a	fix: Add a missing `continue` in fuse which may cause incorrect bloom test result (#12650)	1 year ago
Trevor Whitney	6c33809015	feat: split detected fields queries (#12491)	1 year ago
Owen Diehl	9c25985b97	feat(blooms): record time spent resolving shards (#12636)	1 year ago
Salva Corts	0ee2a6126a	fix(blooms): Fix findGaps when ownership goes to MaxUInt64 and that is covered by existing meta (#12558)	1 year ago
Christian Haudum	2f24b670cb	chore: Restructure `indexgateway` component package (#12630) ... This PR moves code for the index gateway server and client components into the `pkg/indexgateway` directory, so it that the index gateway has its own top-level package like other components, such as ingester, distributor, querier, ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Sandeep Sukhani	0831802a99	fix: fix setting of info log level when trying to detect level from log lines (#12635)	1 year ago
Cyril Tovena	397aa56e15	feat: Enable log volume endpoint by default (#12628)	1 year ago
Paul Rogers	13d45bc512	test: Fix deadlock in table_manager relating to Stop/SyncTables (#12597)	1 year ago
Travis Patterson	11e768726f	fix(query sharding): Generalize avg -> sum/count sharding using existing binop mapper (#12599)	1 year ago
Travis Patterson	c4ac8cc009	feat: add warnings to metadata context directly (#12579)	1 year ago
Paul Rogers	d34f1627cc	test: 2 data race fixes (#12584)	1 year ago
Cyril Tovena	48dae4417c	feat: improve syntax parser for pattern (#12489)	1 year ago
Salva Corts	a32e54053f	chore(blooms): Remove noisy log line in index gateways (#12555)	1 year ago
Sandeep Sukhani	ec888ec8a5	fix: make the tsdb filenames correctly reproducible from the identifier (#12536)	1 year ago
Callum Styan	ec81991f4d	feat: include the stream we failed to create in the stream limit error message (#12437) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Sandeep Sukhani	be501c2564	fix: use nanosecond precision for timestamp in compacted tsdb index file names (#12502)	1 year ago
Ed Welch	670041368a	chore: add more validation at startup to make sure we have a proper schema, and necessary directories are created (#12496) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: Dylan Guedes <djmgguedes@gmail.com>	1 year ago
Owen Diehl	712a2b6967	fix(blooms): fixup bloom testing (#12494)	1 year ago
Kaviraj Kanagaraj	e9b6ce9de4	fix(logql): Linefilter behaviour with chained `or` (#12463) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	1 year ago