12 Commits (39a52974af551e80c54dff573643b59cfc62c72f)

Author	SHA1	Message	Date
Christian Haudum	a9345d0fb6	chore(blooms): Clean up settings and make reasonable defaults (#12483) ... Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	ff08d2fcc2	chore(blooms): Remove bloom gateway ring (#12484) ... Remove ring in favour of a client side, DNS discovery of the bloom gateway servers combined with consistent hashing using the jumphash algorithm (#12470). Instead of configuring the ring, the bloom gateway client needs to be configured with the `-bloom-gateway-client.addresses` flag, which takes a list of comma separated, DNS-discovery-style strings. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Christian Haudum	863acc0152	chore(blooms): Make block query concurrency configurable (#12292) ... A concurrency of 10 per worker seems to be too high for a reasonable 8GiB memory limit on bloom gateways. So this PR makes the concurrency configurable and uses a default of 4. Memory usage of loading block pages into memory therefore being: ``` worker_concurrency x block_query_concurrency x max_block_page_size x 2 ``` With current defaults: ``` 4 x 4 x 32MiB x 2 = 1GiB ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	63c88489d0	refactor: Add RF and Tokens to Loki ring Cfg and allow overwriting docs. (#12142) ... Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Christian Haudum	a0fce39169	feat(bloom-gw): Make num multiplexing tasks configurable, and other adjustments (#12169) ... * We want to be able to change the amount of dequeued items per iterations to test different configurations. * Ignore block pages that are magnitues bigger than the target page size. * Increase download queue capacity to avoid blocking when enqueueing Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	f7710ad7aa	feat: (blooms) Configurable number of tokens (#12125)	1 year ago
Salva Corts	a0b462d366	Bloom-Gateway cache (#11380) ... **What this PR does / why we need it**: This PR adds caching to the bloom-gateway client. It uses the result cache from https://github.com/grafana/loki/pull/11343. Here's how we: - Merge responses: group all chunks by FP and remove duplicated chunk checksums. - Extract responses based on time span: For all chunks in each FP, add to the extracted response only the chunks that overlaps with the desired start and end time.	1 year ago
Salva Corts	67ecf5007b	Make Blooms-Gateway queue settings configurable (#11321) ... **What this PR does / why we need it**: This PR adds three new settings to the Bloom-Gateway config: - `worker_concurrency`: replaces the `numWorkers` constant. Named after the scheduler's `scheduler_worker_concurrency` config. - `max_outstanding_per_tenant`: replaces the `maxTasksPerTenant` constant. Named after the frontend's `max_outstanding_per_tenant` config. - `pending_tasks_initial_capacity`: replaces the pendingTasksInitialCap constant. **Note to the reviewers:** - I think in the future `max_outstanding_per_tenant` should be configurable per-tenant. This would require refactoring the queue pkg and the frontend/scheduler, so I think it doesn't make sense it in this PR. - I moved the `Limits` struct definition to `config.go` so have all the adjustable settings in the same place.	2 years ago
Christian Haudum	e5b006ce0a	Enable/disable bloom gateway per tenant (#11203) ... This commits adds a per-tenant setting to enable/disable the use of the bloom gateway component for filtering chunks. As an operator, you want to control what tenant can use the filtering of chunks based on search string, because the creation of the bloom filters requires a significant amount of resources. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	2abb472e7e	Unify ring managers across components (#10931) ... **What this PR does / why we need it**: This PR creates a re-usable `RingManager` implementation that is then used by * Query Scheduler ring * Index Gateway ring * Bloom Gateway ring * Bloom Compactor ring **Which issue(s) this PR fixes**: Every component had its own implementation of the ring manager which only really differed in the log messages containing the component name. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	b49b3ce6e1	Add basic structure of bloom gateways (#10782) ... ### Summary This pull requests adds the basic structure for the new bloom gateway component. - Adds new `bloom-gateway` target that runs with multiple instances joined by a ring - Adds a querier and client component on the index gateway to filter chunk refs - Adds the gRPC protobuf definitions for commication between index gateways and bloom gateways - Adds a store component used on the bloom gateways to query binary bloom files ``` Querier Query Frontend | | ................................... service boundary | | +----+------+ | indexgateway.Gateway** | bloomgateway.BloomQuerier | bloomgateway.GatewayClient | logproto.BloomGatewayClient | ................................... service boundary | bloomgateway.Gateway | bloomshipper.Store | bloomshipper.Shipper | bloomshipper.BloomFileClient** | ObjectClient** | ................................... service boundary | object storage ** not part of this PR ``` This PR still contains a lot of TODOs and possibilities for optimisations, which will be addressed in subsequent pull requests. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago