82 Commits (05d569cb4e9fbf90c73ebe8ffd39d3df53e2b3fa)

Author	SHA1	Message	Date
Sandeep Sukhani	255fce20d8	fix: Fix handling of duplicate delete requests (#18460)	9 months ago
Sandeep Sukhani	bd293206cd	feat(compactor HS): Wire things up and add some metrics (#18421)	9 months ago
Christian Haudum	d446d47cf8	chore(engine): Prefetch inputs of SortMerge operator (#18395) ... This PR changes how inputs of the SortMerge operator are consumed. The inputs are wrapped with a `prefetchWrapper` which consumes the next batch (`arrow.Record`) in the background while the current batch is processed by the parent node. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Sandeep Sukhani	1ea8d48fc4	chore: expose tables manager interface from compactor (#18344)	10 months ago
Sandeep Sukhani	364660748d	feat(compactor HS): refactor interfaces and build a link between delete requests manager and compactor for HS of compactor (#18326) ... Signed-off-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	10 months ago
Sandeep Sukhani	ba3c67076a	feat(compactor HS): add support for applying storage updates after finishing processing of deletion manifest (#18294)	10 months ago
Sandeep Sukhani	8007e22db5	feat(compactor HS): store index updates per processed segment in the manifest to the object storage (#18212)	10 months ago
Paul Rogers	589652112f	chore: lint fix for queue.go (#18181)	10 months ago
Sandeep Sukhani	d05c4bc500	feat(compactor HS): add support for worker for processing of jobs from the compactor's job queue (#18165)	10 months ago
Ashwanth	ba9b93ba42	chore(engine): add optimisations to pushdown and project groupby labels (#18101)	10 months ago
Sandeep Sukhani	385e3175cf	feat(compactor HS): use streaming RPCs for interacting with job queue over grpc (#18125)	10 months ago
renovate[bot]	8b265ede15	fix(deps): update module github.com/prometheus/common to v0.64.0 (main) (#16750) ... Signed-off-by: Paul Rogers <129207811+paul1r@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com> Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	10 months ago
Sandeep Sukhani	58f4fca65a	feat(compactor HS): add a job runner for processing of deletion jobs (#18058)	11 months ago
Charles Korn	545fb1597c	refactor(stringlabels): Support stringlabels in `logql/log` package (#17838) ... This is another step towards supporting Prometheus' stringlabels implementation in Loki. It adds support in the `logql/log` package. Part of https://github.com/grafana/loki/issues/17122 The tests should now compile and pass with `-tags stringlabels`. --- Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	11 months ago
George Robinson	5394a4cae1	fix: fix lint failure in retention.go (#17894)	11 months ago
Sandeep Sukhani	267f4dc0ec	feat(compactor HS): add job queue and deletion job builder (#17843)	11 months ago
Sandeep Sukhani	e8a94efcad	fix: set delete request batch to empty when we load to fail requests (#17810) ... Signed-off-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	11 months ago
Sandeep Sukhani	fdf3637ccf	chore: improve duplicate series detection while iterating table for retention/deletion (#17682)	12 months ago
Sandeep Sukhani	a5e95bacee	fix: move the check for duplicate series up the call stack to fix an issue when retention is enabled (#17663)	12 months ago
Sandeep Sukhani	47961f802d	feat: Compactor deletion manifest builder (#17474)	12 months ago
Karsten Jeschkies	bf6f130cdf	refactor(stringlabels): Support stringlabels in chunkenc (#17263)	12 months ago
Sandeep Sukhani	32c50888b0	fix: concurrent map access for processedSeries in delete requests manager (#17469)	1 year ago
Footur	ba8830b080	docs: Document the default value for the working_directory for compactor (#17075) ... Signed-off-by: Footur <3769085+Footur@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	1 year ago
Christian Haudum	525b41155d	chore(engine): Add expression evaluator to evaluate physical expressions against a given record batch (#17257) ... The evaluator takes a physcial expressions and an arrow.Record as input and return a ColumnVector. This PR only implements the evaluation of the LiteralExpr and ColumnExpr. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Karsten Jeschkies	8142f123f5	refactor(stringlabels): Support stringlabels in filter.Func (#17233)	1 year ago
Karsten Jeschkies	a864f8b986	refactor(stringlabels): Support stringlabels in log.StreamPipeline (#17216)	1 year ago
Sandeep Sukhani	288ec8c64d	fix: fix timeout and series progress marker for same requests with different shards (#17125)	1 year ago
Sandeep Sukhani	db59d0e9b2	fix: wire up deletion series progress tracking (#17099)	1 year ago
Sandeep Sukhani	e3e1f096ea	feat: store details of processed streams while processing delete requests (#16825)	1 year ago
Paul Rogers	24c260e3e4	chore(deps): Update dskit (#16782)	1 year ago
Sandeep Sukhani	3277321e21	feat: when using sqlite for storing delete requests, use the stored completion time to reduce the requests considered for querytime filtering (#16600)	1 year ago
Paul Rogers	27431b7e7e	chore: Linting update for new golangci (#16572)	1 year ago
Dylan Guedes	ea13730e63	feat: Snapshopt stream overrides values on each request (#16523) ... **What this PR does / why we need it**: Caches a snapshot of resolved policies and retention temporarily to guarantee consistent results on the same request. We evaluate retention and policies in a map when emitting some metrics so having policies or retention inconsistent along the chain for the same push request can cause us to miscalculate things or worse, interact with a nil map.	1 year ago
Sandeep Sukhani	5b33e65bcd	feat: add support for using sqlite for storing delete requests (#16437)	1 year ago
Eng Zer Jun	982b16b404	refactor: replace min/max helpers with built-in min/max (#16438) ... Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	1 year ago
Cyril Tovena	dbf2befc1d	feat: Distributed Operational UI (#16097)	1 year ago
Sandeep Sukhani	5e6d0b5f12	fix: remove unused variable in delete requests store code (#16241) ... Signed-off-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	1 year ago
Sandeep Sukhani	bd0ca9387d	chore: refactor delete requests store interface to prepare for replacing boltdb with sqlite for storing delete requests (#16181)	1 year ago
Sandeep Sukhani	89c60542a1	chore: avoid doing some unnecessary work while listing or merging sharded delete requests (#16125)	1 year ago
Dylan Guedes	5c8e832260	feat: Introduce policy stream mapping (#15982) ... **What this PR does / why we need it**: Introduces the idea of policies to Loki, which are recognizable based on the given stream selectors. This is an improved version of https://github.com/grafana/loki/pull/15561 and built on top of https://github.com/grafana/loki/pull/15875. A policy mapping can be configured the following way: ```yaml 12345: policy_stream_mapping: policy6: - selector: `{env="prod"}` priority: 2 - selector: `{env=~"prod|staging"}` priority: 1 - selector: `{team="finance"}` priority: 4 policy7: - selector: `{env=~"prod|dev"}` priority: 3 ``` With that configuration, pushes to tenant `12345` with the labels `{env="prod", team="finance"}` would be assigned to policy6 because the third mapping for policy6 matches these labels and has higher priority than any other matching.	1 year ago
Sandeep Sukhani	c99771efb8	chore: upload delete requests table only when there are updates since last upload (#16008)	1 year ago
Cyril Tovena	4df8c60b6b	chore(querier): Refactor the store and querier interface. (#15969)	1 year ago
Dylan Guedes	0bc23db95d	refactor: Add retention hours to `discarded` metrics (#15875) ... **What this PR does / why we need it**: We have no distinction between the type of data that gets discarded. On this PR I'm adding `retention_hours` to our discarded metrics so that we can better look at what data was discarded.	1 year ago
Sandeep Sukhani	07ccf7c561	chore: merge back delete request when we are done processing all its splits (#15968)	1 year ago
Sandeep Sukhani	81940c86d5	feat: dedupe duplicate delete requests while loading them for processing in compactor (#15852)	1 year ago
Sandeep Sukhani	00d4fcb712	fix: do not update delete requests tracking metrics for users whose delete requests are not supposed to be processed (#15855)	1 year ago
Sandeep Sukhani	905caa11cf	chore: align delete request splits to shard interval (#15790)	1 year ago
Paul Rogers	f2c2a22bdc	chore: Preparation for incoming static code analysis CI check (#15164) ... Co-authored-by: Danny Cooper <danny.cooper@grafana.com>	1 year ago
Trevor Whitney	b5462b6639	chore: remove copied loop vars (#14383)	2 years ago
Christian Haudum	f52f8ad540	chore: Clarify compression package (#14257) ... This PR renames "encoding" to "codec" in the compression package to remove the cognitive dissonance. It also removes the `Enc` prefix for codec identifiers, so that they adhere Go's best practice of naming conventions, e.g. `compression.EncGZIP` becomes `compression.GZIP` when used in a different package. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago