2985 Commits (e0ca67dd4563e41c57b2f1409ef235b76b2a1a6e)

Author	SHA1	Message	Date
Christian Haudum	738c274a58	fix(blooms): Fix `partitionSeriesByDay` function (#12900) ... The function `partitionSeriesByDay` could yield empty when chunks were outside of the requested time range. This would result in requests for a time range that does not contain any chunks for a series to filter. These requests would return errors, because a subsequent `partitionSeriesByDay` for would remove that day resulting in no task. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	1951322154	chore(blooms): records more bloom iteration stats (#12889)	1 year ago
Dylan Guedes	7b6f0577c3	feat: Querier: Split gRPC client into two. (#12726) ... **What this PR does / why we need it**: Split the gRPC client used by the querier into two, one for the communication with the scheduler, the other for communicating with the query-frontend. - This change is retrocompatible: you don't have to change anything to keep existing behavior. - To configure the custom scheduler grpc client, you can use the new `query_scheduler_grpc_client` config or the new CLI flag `querier.scheduler-grpc-client` - If you'd like to configure your frontend grpc client using a better named section, you can use the new `query_frontend_grpc_client` instead of the old `grpc_client_config`. Just make sure you don't use both at the same time, it will result in an error. This work is necessary for configuring custom behavior between `querier<->scheduler` vs `querier<->frontend`. A use case is configuring mTLS when a different certificate is used by queriers, schedulers and frontends. You can only configure a single `server_name` with our current setup, making it impossible.	1 year ago
Shantanu Alshi	6904a6520d	fix: Fix log level detection (#12651)	1 year ago
Owen Diehl	93aaf29e68	fix(spans): corrects early-close for a few spans (#12887)	1 year ago
Owen Diehl	c1d56c11d6	chore(instrumentation): reintroduce span propagation across scheduler boundaries (#12880)	1 year ago
Kaviraj Kanagaraj	b6049f6792	fix: `codec` not initialized in downstream roundtripper (#12873) ... Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	1 year ago
Sven Grossmann	2b3ae48d9b	feat(detectedFields): add parser to response (#12872)	1 year ago
Owen Diehl	8b34751e17	chore(blooms): additional spans for bloom read path (#12866) ... Adds some timing information to pre-existing spans to help better understand bloom read path latency responsibility	1 year ago
Owen Diehl	5a643c7115	chore: reduces span footprint + double recording (#12864) ... Trying to remove some cruft from traces.	1 year ago
Owen Diehl	4c9b22f110	feat(blooms): ignore individual bloom-gw failures (#12863)	1 year ago
Christian Haudum	fcd544c2d9	feat(blooms): Add in-memory LRU cache for meta files (#12862) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
benclive	fd2301fd62	fix: Ensure Drain patterns are valid for LogQL pattern match filter (#12815)	1 year ago
Michel Hollands	599a3002d2	docs: update the lokitool docs (#12805) ... Signed-off-by: Michel Hollands <michel.hollands@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	1 year ago
Christian Haudum	37c88220b3	fix(blooms): Handle not found metas gracefully (#12853) ... There is a time window between between listing metas and fetching them from object storage which could lead to a race condition that the meta is not found in object storage, because it was deleted and superseded by a newer meta. This can happen when querying recent bloom data, that is still subject to updates, and results in an error like this: ``` rpc error: code = Unknown desc = failed to get meta file bloom/tsdb_index_19843/XXXX/metas/18fbdc8500000000-1921d15dffffffff-270affee.json: storage: object doesn't exist (Trace ID: 4fe28d32cfa3e3df9495c3a5d4a683fb) ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	48bbf983d0	chore(blooms): Remove ID field from task struct (#12851) ... We've seen a few cases where creating the ULID failed for unknown reasons, and the ID is not really used. It was only useful early on in the development for debugging. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	7bbd8b5087	feat(blooms): ingester aware bounded impl (#12840)	1 year ago
Christian Haudum	667076d935	fix(blooms): Do not fail requests when fetching metas from cache fails (#12838) ... The bloom shipper uses metas to resolve available blocks. Metas are fetched from cache, and if not available from object storage. If fetching metas from cache fails, e.g. timeout, the request should not fail, but proceed as if no metas were available. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	a0f358fcc8	fix(blooms): Fully deduplicate chunks from FilterChunkRef responses (#12807) ... This PR aims for full de-duplication of chunks and series from filter requests from the index gateway to the bloom gateway. Whenever we merge/de-duplicate slices, the inputs need to be sorted. It appears that the Removals (chunks) from the v1.Output are not guaranteed to be sorted. When comparing ShortRefs, both From, Through, and Checksum need to be used. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	1665e853a0	fix(blooms): dont break iterator conventions (#12808) ... Followup to #12806 which exposes skipped pages more explicitly than as an error. * refactors skip logic for bloom pages that are too large * s/Seek/LoadOffset/ for LazyBloomIter * removes unused code	1 year ago
Owen Diehl	bc78d13d9b	feat(blooms): compute chunks once (#12664) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Owen Diehl	eac56224b8	feat(blooms): limit bloom size during creation (#12796)	1 year ago
Jonas L. B	c0113db4e8	fix(promtail): Handle docker logs when a log is split in multiple frames (#12374)	1 year ago
Salva Corts	76ba24e3d8	fix(blooms): Reset error on LazyBloomIter.Seek (#12806)	1 year ago
Ned Andreev	40479029d7	feat: Generic logline placeholder replacement and tokenization (#12799) ... This code allows us to preprocess generic logs and replace highly variable dynamic data (timestamps, IPs, numbers, UUIDs, hex values, bytesizes and durations) with static placeholders for easier pattern extraction and more efficient and user-friendly matching by the Drain algorithm. Additionally, there is logic that splits generic log lines into discrete tokens that can be used with Drain for better results than just naively splitting the logs on every whitespace. The tokenization here handles quote counting and emits quoted strings as a part of the same token. On the other side, it also handles likely JSON logs without any white spaces in them better, by trying to split `{"key":value}` pairs (without actually parsing the JSON). All of this is done without using regular expressions and without actually parsing the log lines in any specific format. That's why it works very efficiently in terms of CPU usage and allocations, and should handle all log formats and unformatted logs equally well.	1 year ago
Michel Hollands	33e82ec133	fix: remove unused parameter causing lint error (#12801) ... Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	1 year ago
benclive	7b8533e435	feat: Add step param to Patterns Query API (#12703)	1 year ago
Christian Haudum	3bf2d1fea0	fix(blooms): Deduplicate filtered series and chunks (#12791) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	4cc3a73172	test: Add mutex and setter/getter for addresses in test (#12793)	1 year ago
Owen Diehl	da8ee7e7f2	chore(blooms): Instrument source bytes added to blooms directly (#12782)	1 year ago
Trevor Whitney	af09f53a33	ci: release workflows get build image from makefile (#12778)	1 year ago
Owen Diehl	fb9b0e80a7	chore(blooms): adds more instrumentation to block building (#12779)	1 year ago
Christian Haudum	ecefb49508	fix(blooms): Fix a regression introduced with #12774 (#12776) ... This PR fixes a regression introduced with #12774 where series where wrongly identified as "skipped" even though there were part of blocks. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	c36b1142c7	fix(blooms): Correctly return unfiltered chunks for series that are not mapped to any block (#12774) ... This PR fixes a conceptual mistake in the code of resolving blocks on the index gateways. Currently, a series does not resolve to any block is discarded instead of being kept for the response. This change adds the chunks of the skipped series to the bloom querier response. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Trevor Whitney	ae180d6e07	fix(nix): lambda-promtail vendor hash (#12763)	1 year ago
Trevor Whitney	210ea93a69	feat: add detected-fields command to logcli (#12739)	1 year ago
Shantanu Alshi	587a6d20e9	feat: Detected labels from store (#12441)	1 year ago
Paul Rogers	c178cc62df	test: Data race updates for memchunk tests (#12752)	1 year ago
dependabot[bot]	ff2063bcaf	chore(deps): bump golang.org/x/net from 0.17.0 to 0.23.0 in /pkg/push (#12694) ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Periklis Tsirakidis <periklis@redhat.com>	1 year ago
Christian Haudum	5540c92d50	perf(blooms): Resolve bloom blocks on index gateway and shard by block address (#12720) ... This pull request changes how data is sharded across bloom gateways. Currently, chunks are grouped by series and shared by the fingerprint of the series across the available bloom gateways using jumphash algorithm. This however, leads to over-querying bloom blocks, because bloom blocks have consecutive fingerprint ranges, whereas sharding keys are evenly distributed across keyspace. This PR changes the sharding in the way that bloom blocks for series are already resolved on the index gateways and that their address is used for sharding data. This has the advantage that the grouped series can be mapped to the correct bloom blocks on the client side. Sending the block ref along with the grouped series to the bloom gateway allows for efficient querying of the data because each bloom gateway therefore owns exactly 1/nth of the blocks. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Trevor Whitney	f68d1f7faf	fix: make detected fields work for both json and proto (#12682)	1 year ago
Ed Welch	4c563f7823	docs: unhide compactor GRPC config (#12587) ... Signed-off-by: Edward Welch <edward.welch@grafana.com> Co-authored-by: DylanGuedes <djmgguedes@gmail.com>	1 year ago
benclive	5ae33d19ff	test: Add new tests for Drain pattern matching (#12655)	1 year ago
Dylan Guedes	402d1d7c48	perf: TSDB: Add fast-path to `inversePostingsForMatcher` (#12679)	1 year ago
Cyril Tovena	19bfef48cb	feat: Add pattern ingester support in SSD mode (#12685)	1 year ago
Func	4a05964d55	fix: Use to the proper config names in warning messages (#12114) ... Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Seiya	0eba448fc7	fix: crrect initialization of a few slices (#12674) ... Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Michel Hollands	6dce98870d	fix: Fix the lokitool imports (#12673) ... Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	1 year ago
Michel Hollands	7b7d3d4cd2	feat: add lokitool (#12166) ... Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	1 year ago
洪阿南	0d1ebebd3a	fix: Add a missing `continue` in fuse which may cause incorrect bloom test result (#12650)	1 year ago