91 Commits (RN-35)

Author	SHA1	Message	Date
dannyc-grafana	ecdd613a0c	chore: Autofix to make nosec comments also readable by semgrep (#18664)	5 months ago
Sandeep Sukhani	9281ab3959	docs(compactor HS): add documentation for horizontally scalable compactor (#18654)	5 months ago
Sandeep Sukhani	d5a959e37b	chore: use boltdb for tracking series processing progress in delete requests manager (#18629)	5 months ago
Saikat	cdb190be96	fix: ensure to return delete requests only for the requested tanent (#18589)	6 months ago
Sandeep Sukhani	5bf3f74ab2	feat(compactor HS): tune compactor worker config and add experimental feature warning to the docs (#18539)	6 months ago
Sandeep Sukhani	8054076873	fix: do not try to merge the already consolidated delete requests while listing them (#18544)	6 months ago
Sandeep Sukhani	000f90d74b	feat(compactor HS): Fixes and changes for getting horizontally scalable compactor to work well (#18526)	6 months ago
Ashwanth	df03f186b1	chore: fix failing lint on main (#18511)	6 months ago
Robert Fratto	86c57c8616	chore: consistently use new labels.Labels API (#18490)	6 months ago
Sandeep Sukhani	255fce20d8	fix: Fix handling of duplicate delete requests (#18460)	6 months ago
Sandeep Sukhani	bd293206cd	feat(compactor HS): Wire things up and add some metrics (#18421)	6 months ago
Christian Haudum	d446d47cf8	chore(engine): Prefetch inputs of SortMerge operator (#18395) ... This PR changes how inputs of the SortMerge operator are consumed. The inputs are wrapped with a `prefetchWrapper` which consumes the next batch (`arrow.Record`) in the background while the current batch is processed by the parent node. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Sandeep Sukhani	1ea8d48fc4	chore: expose tables manager interface from compactor (#18344)	6 months ago
Sandeep Sukhani	364660748d	feat(compactor HS): refactor interfaces and build a link between delete requests manager and compactor for HS of compactor (#18326) ... Signed-off-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com> Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	6 months ago
Sandeep Sukhani	ba3c67076a	feat(compactor HS): add support for applying storage updates after finishing processing of deletion manifest (#18294)	6 months ago
Sandeep Sukhani	8007e22db5	feat(compactor HS): store index updates per processed segment in the manifest to the object storage (#18212)	7 months ago
Paul Rogers	589652112f	chore: lint fix for queue.go (#18181)	7 months ago
Sandeep Sukhani	d05c4bc500	feat(compactor HS): add support for worker for processing of jobs from the compactor's job queue (#18165)	7 months ago
Ashwanth	ba9b93ba42	chore(engine): add optimisations to pushdown and project groupby labels (#18101)	7 months ago
Sandeep Sukhani	385e3175cf	feat(compactor HS): use streaming RPCs for interacting with job queue over grpc (#18125)	7 months ago
renovate[bot]	8b265ede15	fix(deps): update module github.com/prometheus/common to v0.64.0 (main) (#16750) ... Signed-off-by: Paul Rogers <129207811+paul1r@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com> Co-authored-by: Paul Rogers <129207811+paul1r@users.noreply.github.com>	7 months ago
Sandeep Sukhani	58f4fca65a	feat(compactor HS): add a job runner for processing of deletion jobs (#18058)	7 months ago
Charles Korn	545fb1597c	refactor(stringlabels): Support stringlabels in `logql/log` package (#17838) ... This is another step towards supporting Prometheus' stringlabels implementation in Loki. It adds support in the `logql/log` package. Part of https://github.com/grafana/loki/issues/17122 The tests should now compile and pass with `-tags stringlabels`. --- Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	7 months ago
George Robinson	5394a4cae1	fix: fix lint failure in retention.go (#17894)	7 months ago
Sandeep Sukhani	267f4dc0ec	feat(compactor HS): add job queue and deletion job builder (#17843)	7 months ago
Sandeep Sukhani	e8a94efcad	fix: set delete request batch to empty when we load to fail requests (#17810) ... Signed-off-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	8 months ago
Sandeep Sukhani	fdf3637ccf	chore: improve duplicate series detection while iterating table for retention/deletion (#17682)	8 months ago
Sandeep Sukhani	a5e95bacee	fix: move the check for duplicate series up the call stack to fix an issue when retention is enabled (#17663)	8 months ago
Sandeep Sukhani	47961f802d	feat: Compactor deletion manifest builder (#17474)	8 months ago
Karsten Jeschkies	bf6f130cdf	refactor(stringlabels): Support stringlabels in chunkenc (#17263)	8 months ago
Sandeep Sukhani	32c50888b0	fix: concurrent map access for processedSeries in delete requests manager (#17469)	9 months ago
Footur	ba8830b080	docs: Document the default value for the working_directory for compactor (#17075) ... Signed-off-by: Footur <3769085+Footur@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	9 months ago
Christian Haudum	525b41155d	chore(engine): Add expression evaluator to evaluate physical expressions against a given record batch (#17257) ... The evaluator takes a physcial expressions and an arrow.Record as input and return a ColumnVector. This PR only implements the evaluation of the LiteralExpr and ColumnExpr. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	9 months ago
Karsten Jeschkies	8142f123f5	refactor(stringlabels): Support stringlabels in filter.Func (#17233)	9 months ago
Karsten Jeschkies	a864f8b986	refactor(stringlabels): Support stringlabels in log.StreamPipeline (#17216)	9 months ago
Sandeep Sukhani	288ec8c64d	fix: fix timeout and series progress marker for same requests with different shards (#17125)	9 months ago
Sandeep Sukhani	db59d0e9b2	fix: wire up deletion series progress tracking (#17099)	9 months ago
Sandeep Sukhani	e3e1f096ea	feat: store details of processed streams while processing delete requests (#16825)	10 months ago
Paul Rogers	24c260e3e4	chore(deps): Update dskit (#16782)	10 months ago
Sandeep Sukhani	3277321e21	feat: when using sqlite for storing delete requests, use the stored completion time to reduce the requests considered for querytime filtering (#16600)	10 months ago
Paul Rogers	27431b7e7e	chore: Linting update for new golangci (#16572)	10 months ago
Dylan Guedes	ea13730e63	feat: Snapshopt stream overrides values on each request (#16523) ... **What this PR does / why we need it**: Caches a snapshot of resolved policies and retention temporarily to guarantee consistent results on the same request. We evaluate retention and policies in a map when emitting some metrics so having policies or retention inconsistent along the chain for the same push request can cause us to miscalculate things or worse, interact with a nil map.	10 months ago
Sandeep Sukhani	5b33e65bcd	feat: add support for using sqlite for storing delete requests (#16437)	10 months ago
Eng Zer Jun	982b16b404	refactor: replace min/max helpers with built-in min/max (#16438) ... Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	11 months ago
Cyril Tovena	dbf2befc1d	feat: Distributed Operational UI (#16097)	11 months ago
Sandeep Sukhani	5e6d0b5f12	fix: remove unused variable in delete requests store code (#16241) ... Signed-off-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	11 months ago
Sandeep Sukhani	bd0ca9387d	chore: refactor delete requests store interface to prepare for replacing boltdb with sqlite for storing delete requests (#16181)	11 months ago
Sandeep Sukhani	89c60542a1	chore: avoid doing some unnecessary work while listing or merging sharded delete requests (#16125)	11 months ago
Dylan Guedes	5c8e832260	feat: Introduce policy stream mapping (#15982) ... **What this PR does / why we need it**: Introduces the idea of policies to Loki, which are recognizable based on the given stream selectors. This is an improved version of https://github.com/grafana/loki/pull/15561 and built on top of https://github.com/grafana/loki/pull/15875. A policy mapping can be configured the following way: ```yaml 12345: policy_stream_mapping: policy6: - selector: `{env="prod"}` priority: 2 - selector: `{env=~"prod|staging"}` priority: 1 - selector: `{team="finance"}` priority: 4 policy7: - selector: `{env=~"prod|dev"}` priority: 3 ``` With that configuration, pushes to tenant `12345` with the labels `{env="prod", team="finance"}` would be assigned to policy6 because the third mapping for policy6 matches these labels and has higher priority than any other matching.	11 months ago
Sandeep Sukhani	c99771efb8	chore: upload delete requests table only when there are updates since last upload (#16008)	11 months ago