apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Kaviraj Kanagaraj	fac5997b18	feat: Support split align and caching for instant metric query results (#11814 ) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	1 year ago
Salva Corts	b7cb85f92b	Tail queue (#11930 )	1 year ago
Christian Haudum	38c09a09bd	Bloom gateway: Fix metric for fetched blocks (#11988 ) This PR fixes the metric for fetched blocks Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	85f7baaeda	Blooms/integration fixes (#11979 )	1 year ago
Christian Haudum	9f86473b0a	Various bloom gateway instrumentation changes (#11983 ) See individual commit messages for details. * Improve help text for chunk removals metric * End counter metric with _total * Fix incorrect observation of failed tasks * Observe chunk removals from block querier * Move chunks/series metrics into bloom querier * Observe total/filtered series and total/filtered chunks in bloom gateway Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	fb728a6781	(chore) Cleanup duplicate functions/strucs in bloomgateway package (#11978 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	ef40136715	Use TSDB index prefix on blooms directory (#11977 )	1 year ago
Owen Diehl	5d1798f16d	Blooms/misc fixes (#11974 )	1 year ago
Christian Haudum	31711d5558	(chore) Bloom gateway: Improve fingerprint partitioning in client (#11971 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	f158b5bc3e	correctly set block iter when no overlapping blocks are found (#11973 )	1 year ago
Owen Diehl	87ae2efe41	Pull/11950 ammendments (#11972 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	543aaab055	(Blooms) Add metrics to compactor (#11966 )	1 year ago
Owen Diehl	443720f47b	tsdb parsing handles uint (#11969 )	1 year ago
JordanRushing	cad4b8e749	Reduce Distributor auto-forget unhealthy cycles from 10->2 (#11935 ) Signed-off-by: JordanRushing <rushing.jordan@gmail.com>	1 year ago
Christian Haudum	5967ceeb43	Replace min/max token with `TokenRange` in bloom ring utilities (#11960 ) This PR replaces min/max token fields with a `TokenRange` field that uses the `Range[uint32]` type. The `Range[uint32]` uses similar semantics as the `FingerprintBounds` we use for fingerprint ranges. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	2177037ee3	(chore): Simplify implementation of instance sort iterator (#11959 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	d0fae5cd69	[blooms] Remove unnecessary token sorting (#11958 )	1 year ago
Salva Corts	6434df7345	Bloom compactor shuffle-sharding (#11947 )	1 year ago
Owen Diehl	bd12e16339	[Blooms] Use correct table address function (#11955 )	1 year ago
Owen Diehl	3c06b360ec	meta & removal support for bloom compaction (#11941 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Ashwanth	9e7725b31b	feat(metadata): introduce a separate split interval for recent query window (#11897 )	1 year ago
Justin Burnham	1b4d23f9b7	Ruler: Disable x-scope-orgid header append in remote write (#11819 ) Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	1 year ago
Christian Haudum	1c43991ddc	Remove `bloomcompactor.DayTable` in favour of `config.DayTime` (#11917 ) Both structs shared the same semantics. This PR moves additional functionality from the `DayTable` to the `DayTime` struct. To get the table name of a day (ordinal number of day since unix epoch), call `DayTime.Table()`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	b2e4cc39a8	Add mutex for t.stopped to prevent data races (#11933 ) What this PR does / why we need it: This addresses the data race present on the `t.stopped` variable in `tail.go`. ``` ================== WARNING: DATA RACE Write at 0x00c00098b198 by goroutine 568: github.com/grafana/loki/pkg/querier.(Tailer).close() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail.go:272 +0x104 github.com/grafana/loki/pkg/querier.TestTailer.func7.2() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail_test.go:169 +0x34 runtime.deferreturn() /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/panic.go:477 +0x34 testing.tRunner() /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0 testing.(T).Run.func1() /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40 Previous read at 0x00c00098b198 by goroutine 569: github.com/grafana/loki/pkg/querier.(Tailer).loop() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail.go:88 +0x13c github.com/grafana/loki/pkg/querier.newTailer.func1() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail.go:342 +0x34 Goroutine 568 (running) created at: testing.(T).Run() /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x5e8 github.com/grafana/loki/pkg/querier.TestTailer() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail_test.go:158 +0x10dc testing.tRunner() /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0 testing.(T).Run.func1() /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40 Goroutine 569 (running) created at: github.com/grafana/loki/pkg/querier.newTailer() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail.go:342 +0x300 github.com/grafana/loki/pkg/querier.TestTailer.func7() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/tail_test.go:168 +0x138 testing.tRunner() /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0 testing.(T).Run.func1() /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40 ================== ``` Which issue(s) this PR fixes: Relates to: https://github.com/grafana/loki/issues/8586 Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Paul Rogers	472496feb7	Parse JSON String arrays properly so string elements can be retrieved (#11921 ) What this PR does / why we need it: This PR imports the newly forked grafana/jsonparser over the buger/jsonparser module. The latter has seemingly been abandoned. PR 10690 introduces a fix to the jsonparser module, which has been incorporated into the grafana fork of the module. The PR is designed to fix accessing string array elements from within a JSON structure. For example, with the following JSON: `{"log":{"message":{"content":{"misses":["a","b","c","d"]}}}}` The Loki code, before this PR, when searching for `json misses = "log.message.content.misses[0]" ` will result in an "Unknown value type error". After this PR is merged, the result will assign `a` to the `misses` variable. Which issue(s) this PR fixes: Fixes #[9179](https://github.com/grafana/loki/issues/9179) https://github.com/grafana/loki/pull/10690 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Christian Haudum	eb8464a641	Bloom compactor: Load blocks lazily in batches (#11919 ) To avoid loading possibly lots of blocks upfront, this PR introduces lazy loading of blocks in batches using an iterator that loads blocks on demand. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	0bb2574040	makes batchedLoader generic + removes unnecessary interfaces & adapters (#11924 ) While reviewing https://github.com/grafana/loki/pull/11919, I figured it'd be nice to make `batchedLoader` generic so we can reuse it's logic. This let me test it easier and remove a lot of now-unnecessary adapter code (interfaces, types)	1 year ago
Owen Diehl	7560067439	Bloom/running (#11918 ) Includes a few small fixes in order to get the compactor running, including * Wires up i/o from `v1` pkg and the `bloomshipper` pkg in order to archive+write blocks remotely after building * config error handling * fix an error iterating chunks * handles gzipping when loading tsdbs from storage * bypasses internal cache in `BloomTSDBStore`'s internal `indexshipper/storage.Client` to get up to date results, which is easier for prototyping * `ClosableReadSeekerAdapter` --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Signed-off-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	c277158f51	Cleanup data race associated with workerID var (#11922 ) What this PR does / why we need it: A data race existed with the workerID variable, as it could be modified by multiple goroutines. Relates to: https://github.com/grafana/loki/issues/8586 -- Before fix: ``` go test -count=1 -race ./pkg/querier/worker ================== WARNING: DATA RACE Read at 0x00c000494108 by goroutine 229: github.com/grafana/loki/pkg/querier/worker.(processorManager).concurrency.func1() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/worker/processor_manager.go:81 +0x118 Previous write at 0x00c000494108 by goroutine 222: github.com/grafana/loki/pkg/querier/worker.(processorManager).concurrency() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/worker/processor_manager.go:70 +0x108 github.com/grafana/loki/pkg/querier/worker.(querierWorker).resetConcurrency() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/worker/worker.go:267 +0x10c github.com/grafana/loki/pkg/querier/worker.(querierWorker).AddressAdded() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/worker/worker.go:219 +0x868 github.com/grafana/loki/pkg/querier/worker.TestResetConcurrency.func1() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/worker/worker_test.go:64 +0x1c8 testing.tRunner() /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0 testing.(T).Run.func1() /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40 Goroutine 229 (running) created at: github.com/grafana/loki/pkg/querier/worker.(processorManager).concurrency() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/worker/processor_manager.go:75 +0xcc github.com/grafana/loki/pkg/querier/worker.(querierWorker).resetConcurrency() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/worker/worker.go:267 +0x10c github.com/grafana/loki/pkg/querier/worker.(querierWorker).AddressAdded() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/worker/worker.go:219 +0x868 github.com/grafana/loki/pkg/querier/worker.TestResetConcurrency.func1() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/worker/worker_test.go:64 +0x1c8 testing.tRunner() /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0 testing.(T).Run.func1() /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40 Goroutine 222 (running) created at: testing.(T).Run() /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x5e8 github.com/grafana/loki/pkg/querier/worker.TestResetConcurrency() /Users/progers/dev/src/github.com/grafana/loki/pkg/querier/worker/worker_test.go:52 +0x1b0 testing.tRunner() /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x1b0 testing.(T).Run.func1() /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x40 ================== --- FAIL: TestResetConcurrency (0.02s) --- FAIL: TestResetConcurrency/concurrency_is_correct_when_numTargets_does_not_divide_evenly_into_maxConcurrent (0.01s) testing.go:1465: race detected during execution of test testing.go:1465: race detected during execution of test FAIL FAIL github.com/grafana/loki/pkg/querier/worker 4.626s FAIL ``` -- After fix: ``` go clean -testcache go test -count=1 -race ./pkg/querier/worker ok github.com/grafana/loki/pkg/querier/worker 6.034s ``` Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist* - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Salva Corts	7a95cb8663	Configurable bloom tokenizer and block settings (#11889 ) What this PR does / why we need it: This PR makes the max block size configurable and wires up the already existing nGramLen and nGramSkip settings. Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Salva Corts	681bb57971	Limit bloom block size (#11878 ) What this PR does / why we need it: This PR limits the size of the blocks created by the compactor. The block builder keeps adding series' blooms to a block until the limit is exceeded, meaning that blocks may grow beyond the configured maximum. This is needed so we avoid having tiny blocks which had space for small blooms but later a bigger blooms didn't fit. Blocks are built lazily: the generator returns an iterator that builds one block at a time. Special notes for your reviewer: The maximum size is currently set to 50 MBs. We will make this configurable on a followup PR. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	57619b76a4	Bloom gateway: Change Summary metrics to Histogram metrics (#11913 ) Even though the main information from these metrics are the `sum` and the `count`, histograms offer the ability to aggregate them. Follow up to https://github.com/grafana/loki/pull/11907 Ref: https://prometheus.io/docs/practices/histograms/#quantiles Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	d58031e853	Blooms/bloomcompactor wiring (#11914 ) * Moves `router` struct into the `Compactor` struct * Implements period aware `TSDBStore` * Integrates everything together, hooks into `modules.go` * Removes some now unused compactor code * Light refactoring to make types align where appropriate	1 year ago
Christian Haudum	3efd201bff	Fix test utility function `MkBasicSeriesWithBlooms` (#11909 ) The utility correctly populated the blooms, but returned incorrect values for `keys`, which are the indexed keys of a bloom. The values need to be copied when appending to the result slice. Additionally, the source for the tokenizer changed to be avoid duplicate keys that are present in all blooms. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	c1d5ec14fd	Update metrics of bloomgateway worker (#11907 ) What this PR does / why we need it: Update metrics for bloomgatway worker to improve observability of the query path. ``` dequeueDuration : Summary processDuration : Summary metasFetched : Summary blocksFetched : Summary tasksDequeued : Counter tasksProcessed : Counter blockQueryLatency : Histogram ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Karsten Jeschkies	ddaa497554	Prettify concat and downstream expressions. (#11877 ) What this PR does / why we need it: Prettifying `ConcatSampleExpr` et al will simplify debugging the shard mapper. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Christian Haudum	60551dacc4	Remove unused sharding code from bloom gateway (#11906 ) The sharding by fingerprint range is done on the client side, not on the server side. The sharding related code was not used in the gateway. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	52167c4d32	logs query & hashedquery separately to minimize stacktraces (#11901 ) Rather than call all our values together, this PR precomputes a few in `metrics.go` so any issues will result in smaller stack traces.	1 year ago
Owen Diehl	fe53eb6817	fix type errors after introduction of typedcache into bloomshipper (#11902 ) Recent changes causing compilation failures -- this fixes type signatures after refactoring. Also removes test coverage comparisons because they prevent us from fixing `main`	1 year ago
Owen Diehl	3a1bd41ef5	[Blooms] Routing & wiring for bloom building (#11894 ) A few changes wiring up bloom building logic: * A `Router` as the controlling mechanism for iterating tenants+periods, queueing work for parallel computation, determining which fingerprint ranges and tenants are owned by a replica, and waiting for it's completion * Updates the `SimplerBloomController` to accept ownership ranges, periods, and tenants as arguments for use by the `Router` * Builds a `BloomTSDBStore` struct around the `indexshipper/storage.Client` interface in order to iterate relevant TSDBs/etc	1 year ago
Christian Haudum	3b122e4be1	Bloom gateway: Wire up cache config with implementation (#11898 ) Until now, the cache configuration in the bloom gateway did not initialise the cache. This PR wires the config with the actual implementation and passes them to the bloom store. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	9128036145	Bloom gateway: Integrate processor into worker (#11895 ) This PR replaces the iteration of the bloom gateway worker with the code encapsulated in the previously established processor. Since the processor only relies on the BloomStore, rather than on the BloomShipper, the PR also removes unused code from the shipper. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Karsten Jeschkies	483ee56139	Inject downstream accumulator. (#11863 ) What this PR does / why we need it: We inject the downstream accumulator into the downstreamer instead of constructing it on the first arrival of the first result. Since we know the query type before executing the first query we can pass the correct accumulator. This will allow us to define special `topk` or `sum by` accumulators in the future. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	ba0538c2a5	[Blooms] tsdb iterator implementation for bloom construction (#11886 ) Builds a lazy iterator over a tsdb file for bloom construction. This is just the library utility, not the integration.	1 year ago
Christian Haudum	c4ac168a8c	BloomStore: Change signature of `FetchBlocks()` to return `[]CloseableBlockQuerier` (#11884 ) What this PR does / why we need it: In order to encapsulate the reference counting on the `BlockDirectory`, the `BloomStore` now returns a slice of `CloseableBlockQuerier` instead of the directory itself. The caller is responsible for closing the returned querier in order to release the reader resource and decrement the ref counter. The PR also renames `ClosableBlockQuerier` to `CloseableBlockQuerier`. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	7bbbf232d0	Extend BloomStore with `Client(model.Time)` function (#11881 ) What this PR does / why we need it: For certain operations, we want direct access to the `BloomClient`. Since every schema period has its own client, the `Client()` function accepts a `model.Timestamp` for which the client should be returned. The function may return an error, if no client for the given time could be found. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	69d152b618	(chore) BloomStore: Clean up `FetchBlocks()` (#11876 ) What this PR does / why we need it: * Removes `FetchBlocksWithQueue` and replace it with `FetchBlocks` * Operate on single `BlockRef` when loading a block directory from cache/fs/storage. * Ensure order of responses from the FetchBlocks request --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	af66ecec6a	[Blooms] bloomshipper meta integration (#11874 ) Builds on top of https://github.com/grafana/loki/pull/11872, standardizing on the `bloomshipper` types `Meta` and `MetaRef`.	1 year ago
Owen Diehl	3a92834de0	[Blooms] Stateless archiving (#11872 ) Moves the archival code to an interface based approach, removing the explicit fs dependency. We can now implement with in-memory solutions. Also refactors to use a tar-file iterator and marks tests as parallelizable.	1 year ago
Christian Haudum	73edf7a943	(chore) Bloomshipper: Separate store and client (#11865 ) What this PR does / why we need it: This PR removes the `StoreAndClient` interface that was accepted by the `BloomShipper`. Since the `BloomStore` had to not only implement the `Store` interface, but also the `Client` interface, it caused re-implementation of the same methods in different ways. Now the shipper solely relies on the `Store` interface. See individual commit messages for more context. Tests have been rewritten from scratch and placed in their own respective test files for store and client. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago

1 2 3 4 5 ...

2604 Commits (fac5997b18e3fb07f92c20f4fa429213574e49cf)