apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Christian Haudum	1c43991ddc	Remove `bloomcompactor.DayTable` in favour of `config.DayTime` (#11917 ) Both structs shared the same semantics. This PR moves additional functionality from the `DayTable` to the `DayTime` struct. To get the table name of a day (ordinal number of day since unix epoch), call `DayTime.Table()`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	57619b76a4	Bloom gateway: Change Summary metrics to Histogram metrics (#11913 ) Even though the main information from these metrics are the `sum` and the `count`, histograms offer the ability to aggregate them. Follow up to https://github.com/grafana/loki/pull/11907 Ref: https://prometheus.io/docs/practices/histograms/#quantiles Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	3efd201bff	Fix test utility function `MkBasicSeriesWithBlooms` (#11909 ) The utility correctly populated the blooms, but returned incorrect values for `keys`, which are the indexed keys of a bloom. The values need to be copied when appending to the result slice. Additionally, the source for the tokenizer changed to be avoid duplicate keys that are present in all blooms. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	c1d5ec14fd	Update metrics of bloomgateway worker (#11907 ) What this PR does / why we need it: Update metrics for bloomgatway worker to improve observability of the query path. ``` dequeueDuration : Summary processDuration : Summary metasFetched : Summary blocksFetched : Summary tasksDequeued : Counter tasksProcessed : Counter blockQueryLatency : Histogram ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	60551dacc4	Remove unused sharding code from bloom gateway (#11906 ) The sharding by fingerprint range is done on the client side, not on the server side. The sharding related code was not used in the gateway. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	3b122e4be1	Bloom gateway: Wire up cache config with implementation (#11898 ) Until now, the cache configuration in the bloom gateway did not initialise the cache. This PR wires the config with the actual implementation and passes them to the bloom store. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	9128036145	Bloom gateway: Integrate processor into worker (#11895 ) This PR replaces the iteration of the bloom gateway worker with the code encapsulated in the previously established processor. Since the processor only relies on the BloomStore, rather than on the BloomShipper, the PR also removes unused code from the shipper. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	c4ac168a8c	BloomStore: Change signature of `FetchBlocks()` to return `[]CloseableBlockQuerier` (#11884 ) What this PR does / why we need it: In order to encapsulate the reference counting on the `BlockDirectory`, the `BloomStore` now returns a slice of `CloseableBlockQuerier` instead of the directory itself. The caller is responsible for closing the returned querier in order to release the reader resource and decrement the ref counter. The PR also renames `ClosableBlockQuerier` to `CloseableBlockQuerier`. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	7bbbf232d0	Extend BloomStore with `Client(model.Time)` function (#11881 ) What this PR does / why we need it: For certain operations, we want direct access to the `BloomClient`. Since every schema period has its own client, the `Client()` function accepts a `model.Timestamp` for which the client should be returned. The function may return an error, if no client for the given time could be found. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	69d152b618	(chore) BloomStore: Clean up `FetchBlocks()` (#11876 ) What this PR does / why we need it: * Removes `FetchBlocksWithQueue` and replace it with `FetchBlocks` * Operate on single `BlockRef` when loading a block directory from cache/fs/storage. * Ensure order of responses from the FetchBlocks request --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	f97fb093f3	(chore) Bloom store: Rewrite block downloading/caching in bloom store fetcher (#11857 ) What this PR does / why we need it: 1. The `BloomStore` interface gets a new function `FetchBlocks()`, which accepts `[]BlockRef` and returns `[]BlockDirectory`. The fetcher implements a new function `FetchBlocks()` which returns a list of block directories. A block directory represents a local file path that contains the extracted files of a bloom block. It also holds a counter of active readers that access the directory. This is used for safely deleting the directory in case it needs to be removed from disk (eg. max disk/cache size reached). The fetcher resolves the block directory from three places: 1. Cache: The in-memory cache that holds actively accessed directories and keeps track of used disk size. 2. Filesystem: In case the cache was emptied (e.g. when restarting the process) but the block directory is present on disk, it can be re-assambled into a bloom directory that can be put to cache. 3. Storage: If the directory is not present locally, the block archive is downloaded and extracted and the block directory is put to cache. 2. The `{Meta,Block}Client` interfaces are unified: Both clients define the same set of operations: Get{Meta,Block} Get{Meta,Block}s Put{Meta,Block} Delete{Meta,Block}s 3. The `blockDownloader` is replaced by a simple queue powered by a channel. The queue is implemented using generics, so it can be re-used for different request/response types. 4. Code related to "block caching" moved into a separate file `cache.go` and unused code is removed. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Owen Diehl	d413c28fa3	Blooms: Align compactor to shipper (#11855 ) Does some alignment work between the `bloomcompactor` and the `bloomshipper` pkgs. Notably: * Uses `bloomshipper.BlockRef` everywhere (removes old bloomshipper struct * Integrates `v1.FingerprintBounds` in `Ref` struct * `Location` interface to distinguish local paths vs paths in object storage for certain types (`{Meta,Bloom}Ref`s) * Introduces `KeyResolver` interface to generate locations from these structs * Integrates `KeyResolver` into our bloom store. In the future, this will allow us to change key structures across schema boundaries when we want to change\|improve them. * Removes `BlockPath` from `BlockRef` in favor of the new resolving functionality. This is also beneficial because it lets us _calculate_ locations from the pure Ref objects, rather than tie some arbitrary state to them which can change (or not be populated by accident).	1 year ago
Owen Diehl	098eef7bd4	Blooms/bounds integration (#11848 ) Replaces `Ref`'s min/max fingerprint fields with `v1.FingerprintBounds` and improves a bunch of checks that use it. Builds on top of https://github.com/grafana/loki/pull/11847	1 year ago
Christian Haudum	1191f88190	(chore) Bloom shipper: Extend `Interval` struct with utility functions (#11841 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	4411649a0e	(chore) Bloom shipper: Replace `Keyspace` struct with `v1.FingerprintBounds` (#11839 ) The latter struct has more utility functions to compare and operate on bounds. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	509f6f6b66	Bloom/controller wiring (#11831 ) Functionality & testware for building bloom block plans.	1 year ago
Christian Haudum	6902130c9a	Bloom shipper: Restructure bloom store (#11828 ) What this PR does / why we need it: This PR changes the structure of how the bloom client * The `BloomStore` is the top component interface which implements the `Store` and the `Client` interfaces. The store holds a store entry for each schema period. * The `bloomStoreEntry` implements the `Store` and `Client` interfaces. It holds a bloom client for a single schema period. Additionally, the store entry also exposes a fetcher for metas (and in the future for blocks), which is responsible for getting data from cache or storage (if not available in cache). * The `BloomClient` implement the `Client` interface. The bloom client uses an object client for the schema period of the bloom client. * The `Fetcher` can fetch and cache metas. This structure is very similar to what we use for the chunk store. Note Before implementing `FetchBlocks()` in the `BloomStore`, I want to implement the new `FingerprintBounds` type also in the shipper and bloom gateway code to be consistent across components, as well as to make use of the new utilities exposed by them. The store implementation probably needs some specific test cases around using the correct store entry for different schema period configs. The code however is mostly taken from the chunk store implementation. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Vladyslav Diachenko	821c0b1ae0	DequeueMany fix (#11797 ) What this PR does / why we need it: Previously we relied only on the index of the tenant's queue to read the requests from the same tenant's queue. However, as long as the queue is aggressively used by the consumers in parallel, there are a few edge cases when knowing the index of last used tenant's queue is not enough to guarantee that we dequeue items from exactly the same tenant's queue. Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Christian Haudum	ecab33a932	Bloom Gateway: Extract processing logic from bloom worker into separate struct (#11812 ) What this PR does / why we need it: The processor executes a set of tasks. It cleanly separates I/O from logic and therefore is more testable than the current worker implementation. `9602214abf/pkg/bloomgateway/processor.go (L33-L36)` Note: The processor is not used yet in the worker. Certain parts must be refactored when #11810 is merged. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	e0e143a88f	Bloom Gateway: Make tasks cancelable (#11792 ) This PR refactors the bloom gateway workers so that tasks that have been enqueued by requests do not end up locking the results channel and therefore the worker, in case the request was cancelled (`context cancelled`) or timed out (`context deadline exceeded`). It also handles errors from the shipper in a way that they are returned to the waiting request asap so it can return and does not need to wait for all tasks to finish. This PR also fixes the worker shutdown in a way that it now gracefully stops and continues to work off the remaining tasks from the queue. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	c01a823f1f	(chore) Bloom Gateway: Avoid unnecessary merging of tasks prior to passing them to the FusedQuerier (#11809 ) What this PR does / why we need it: The `requestIterator` (`v1.Iterator[v1.Request]`) can be built directly from the task and does not require any unnecessary merging of tasks, since this is done by the `v1.FusedQuerier` already. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	9f3fd24958	Bloom Shipper: Fix bug in block boundary check (#11784 ) What this PR does / why we need it: The check `isOutsideRange()` returned incorrect results, because the end timestamp of the search interval is not inclusive. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	6871f3c555	Bloom Gateway: Partition requests into multiple tasks before enqueuing (#11768 ) What this PR does / why we need it: This PR changes at which point a request is split into multiple tasks in case the chunks boundaries of the request overlap multiple days. This solves the problem that the request handler needs to know for how many responses from the block querier needs to wait. Previously, a single task that spanned across multiple days was executed multiple times (once for each matching day). --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	600e390b16	Extend `TestPartitionFingerprintRange` test case (#11750 ) What this PR does / why we need it: The test cases now also cover partially out of bounds blocks as well as inconsecutive block ranges. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	5acc29f2f6	Bloom Gateway: Fix panic in chunk removal after receiving query results (#11770 ) What this PR does / why we need it: While removing not matching chunks from a fingerprint, the for loop over the chunks removes items, and therefore the index of the current iteration needs to be adjusted to accommodate for the removal, otherwise it may panic with `out of bounds`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	de3f1786a1	Bloom Gateway: Remove unnecessary wrapper around v1.Request (#11739 ) What this PR does / why we need it: The wrapper `FilterRequest` wraps `v1.Request` but the additional field with the error channel is never used. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	7c60390364	Remove unnecessary wrapper around bloom shipper (#11724 ) What this PR does / why we need it: The bloom store was initially thought as a translation layer between the bloom gateway and the bloom shipper to be able to simplify the API of the shipper. However, it turned out that the store calls were only a 1-to-1 mapping of the shipper calls and therefore adding unnecessary complexity and call stack. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	bdcb69540f	Bloom gateway: Add metrics for store operations and chunk ref counts (#11677 ) For better observability of the bloom gateway, this PR adds two additional metrics that expose the amount of chunk refs pre and post filtering. This can be used to calculate the filter ratio of the gateways. The PR also adds a metric that observes the latency of the actual processing time of bloom filters within the worker. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	6bcac00c70	Add e2e tests for bloom filtering (#11645 ) What? This PR adds an end-to-end integration test for creating bloom filters and using them in the gateway to filter chunks. Why? This test helped to identify bugs in the bloom shipper code that would have taken a very long time to discover in other ways, such as deploying to a dev environment or running a long-running docker compose setup. Notes The following commits of this PR are actual changes to the compactor/gateway code to make e2e test work: * `2dcc761ba0` * `84052ddb34` * `3ba44f83c8` The bloom gateway code path for processing blocks was cleaned up, because it still contained the unused "sequention processing" path of blocks, which was initially kept to verify the callback based processing works alike: * `f55d79b84f` --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Paul Rogers	6c4699d8f7	Remove call to set default resolver (#11580 ) What this PR does / why we need it: Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Christian Haudum	0a0b7c83ea	Bloomshipper: Use `model.Time` in `MetaRef` and `BlockRef` (#11566 ) The `Ref` of `MetaRef` and `BlockRef` do have start and end timestamps, which are of type `int64`. This however, can lead to the problem that different components might use different precisions (`s`, `ms`, `ns`) for these timestamps. This PR changes the current implementation of the bloom shipper to use `ms` precision timestamps (using `model.Time`) instead of `s`, to match the behaviour that we have to encode ChunkRefs to object store keys. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	55fa0d19e4	Fix panic in bloom gateway client (#11537 ) The bloom gateway client panicked, because the ring, from which it gets the subring for the tenant was never passed as argument in the constructor, even though it was part of the client struct. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	62176339c4	Bloom querying: Propagate line filters from querier to bloom filter (#11492 ) In order to query the bloom filters with the search strings from the user query, we need to propagate the line filters through the different implementations of the store, namely through the GetChunks function. In order to avoid converting data structures, the line filter from the syntax tree is used as custom protobuf type, because it is passed from querier to index gateway and from index gateway to bloom gateway. So far, the line filters request have not been tokenized to n-grams. This PR adds the ability to read the n-gram length from the block schema at query time and tokenizes the line filters from the input just ahead of searching the block. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	c4f5a57bc8	Bloom Compactor: Optimize check for fingerprint ownership (#11389 ) Calling `c.sharding.OwnsFingerprint(tenant, uint64(fingerprint))` for each Series of a TSDB index is very expensive, because it not only creates the tenant's sub-ring but also needs to check the fingerprint against it. Instead, we can pre-calculate the current instance's token ranges and check if the (uint32 converted) fingerprint is contained within these ranges. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	a0b462d366	Bloom-Gateway cache (#11380 ) What this PR does / why we need it: This PR adds caching to the bloom-gateway client. It uses the result cache from https://github.com/grafana/loki/pull/11343. Here's how we: - Merge responses: group all chunks by FP and remove duplicated chunk checksums. - Extract responses based on time span: For all chunks in each FP, add to the extracted response only the chunks that overlaps with the desired start and end time.	2 years ago
Christian Haudum	79693d79ef	Fix `getFromThrough()` function for ShortRefs (#11386 ) What this PR does / why we need it: Since chunks don't necessarily have the same duration, we cannot assume that the last item of a slice sorted by `From` time also has the greatest `Through` time. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	19b0160821	Bloom gateway client: Use binary search to determin membership of fingerprints to server instances (#11336 ) Instead of iterating over all fingerprints and determine their respective instance from the ring individually, this new approach takes calculates the token ranges from the instances from the ring and matches and partitions the fingerprints based on that using a binary search to determine membership to a token range. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	0e433f304e	Bloom Gateway: process blocks immediately when they are available (#11360 ) At current point in time, the worker calls `w.store.GetBlockQueriersForBlockRefs()` to obtain a list of block queriers sorted by fingerprint in ascending order. However, the shipper used by the store to fetch blocks by its refs, calls a callback function when a requested block is available. Instead of waiting for all callbacks to be called and then returning a single list of block queriers, the store now exposes a new function `ForEach()` that also accepts a callback that is passed to the underlying shipper. This allows to process blocks in the worker as soon as they become available. Some block may be served from cache, some may need to be downloaded. That way, we reduce waiting time. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ashwanth	d9f3bf30b1	config: adds `frontend.max-query-capacity` to tune per-tenant query capacity (#11284 ) What this PR does / why we need it: Adds a new config `frontend.max-query-capacity` that allows users to configure what portion of the the available querier replicas can be used by a tenant. `max_query_capacity` is the corresponding YAML option that can be configured in limits or runtime overrides. For example, setting this to 0.5 would allow a tenant to use half of the available queriers. This complements the existing `frontend.max-queriers-per-tenant`. When both are configured, the smaller value of the resulting querier replica count is considered: ``` min(frontend.max-queriers-per-tenant, ceil(querier_replicas * frontend.max-query-capacity)) ``` All queriers will handle requests for a tenant if neither limits are applied. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: noticed that we don't pass down the shuffle sharding limits for frontend (only using it with schedulers) `26f097162a/pkg/loki/modules.go (L895)` but the [docs](`26f097162a/pkg/validation/limits.go (L276)`) mention that`frontend.max-queriers-per-tenant` applies to frontend as well. ``` This option only works with queriers connecting to the query-frontend / query-scheduler, not when using downstream URL. ``` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Salva Corts	71ddec7550	Bloom-Gateway: Use pools to reduce allocations (#11324 ) What this PR does / why we need it: This PR addresses some TODOs in the Bloom-Gateway component to reduce allocations by pooling. Note for reviewers: - I'm not really sure about the max size for the preallocated pools. Used 16K for all of them but that's probably too much for pools such as the `responsesPool`, `addressesPool`... Feel free to leave a comment in the PR with an estimate on the size you think it would make sense to set as the max.	2 years ago
Salva Corts	67ecf5007b	Make Blooms-Gateway queue settings configurable (#11321 ) What this PR does / why we need it: This PR adds three new settings to the Bloom-Gateway config: - `worker_concurrency`: replaces the `numWorkers` constant. Named after the scheduler's `scheduler_worker_concurrency` config. - `max_outstanding_per_tenant`: replaces the `maxTasksPerTenant` constant. Named after the frontend's `max_outstanding_per_tenant` config. - `pending_tasks_initial_capacity`: replaces the pendingTasksInitialCap constant. Note to the reviewers: - I think in the future `max_outstanding_per_tenant` should be configurable per-tenant. This would require refactoring the queue pkg and the frontend/scheduler, so I think it doesn't make sense it in this PR. - I moved the `Limits` struct definition to `config.go` so have all the adjustable settings in the same place.	2 years ago
Christian Haudum	d62d4e37d1	Bloom Gateway: Implement chunk filtering using workers that multiplex requests (#11181 ) This change adds an internal request queue to the bloom gateway. Instead of executing every single request individually, which involves resolving bloom blocks, downloading them if needed and executing the chunk filtering, requests are now enqueued to the internal, per-tenant queue. The queue implements the same shuffle sharding mechanism as the queue in the query scheduler component. Workers then dequeue a batch of requests for a single tenant and multiplex them into a single processing task for each day. This has the big advantage that the chunks of multiple requests can be processed in a single sequential scan through a set a bloom blocks, without needing to skip back and forth within the binary stream of the block. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Vladyslav Diachenko	75cfe59596	bloom blocks downloading queue (#11201 ) implemented bloom blocks downloading queue to control the concurrency of downloading the blocks from the storage Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Christian Haudum	e5b006ce0a	Enable/disable bloom gateway per tenant (#11203 ) This commits adds a per-tenant setting to enable/disable the use of the bloom gateway component for filtering chunks. As an operator, you want to control what tenant can use the filtering of chunks based on search string, because the creation of the bloom filters requires a significant amount of resources. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	4248825ad2	Bloom-compactor Sharding (#11154 ) What this PR does / why we need it: This PR adds tenant and fingerprint (FP) sharding to bloom compactors. Note that the bloom-compactor doesn't yet perform any compaction, but iterates through all tables, tenants, and series checking if the compactor owns the tenant and the series (by the series FP). Actual compaction will be implemented with https://github.com/grafana/loki/pull/11115. A new structure `Job` is added which will carry around all the context for a compaction job such as the tenant ID, the table name, and the series FP. The sharding strategy has two methods: - `OwnsTenant(tenant string)`: Checks if the compactor shard owns the tenant. - `OwnsJob(job Job)`: Checks (again) if the compactor owns the job's tenant. Then, it checks if the compactor owns the job's fingerprint by looking inside the tenant subring. We add a new per-tenant limit: `bloom_compactor_shard_size`. If it's 0, the tenant can use all compactors (i.e. `OwnsTenant` will always return `true`), otherwise, only `bloom_compactor_shard_size` out of the total number of compactors will own the tenant. A given job's FP will be owned by exactly one compactor within the tenant shard. Special notes for your reviewer: - Added a bunch of metrics in `metrics.go` - Added a test for the sharding strategy	2 years ago
Michel Hollands	e93f5bfd93	Add dashboard changes for migrating from cortex_ metrics to loki_ metrics (#11082 ) What this PR does / why we need it: - add missing namespace param for ruler manager metrics - update the loki mixin to use the renamed metrics so both are supported - created the compiled mixin - make the prefix of the cortex_distributor_ingester_clients metrics configurable Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	2 years ago
Karsten Jeschkies	4ed3b117d2	Use newest build image 0.31.2 with golangci-lint update. (#11118 ) What this PR does / why we need it: This just updates the Loki build imaage changed in #11114. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Michel Hollands	8628b15624	Use metrics namespace for more metrics (#11025 ) What this PR does / why we need it: Use the metrics namespace setting instead of hardcoding to `cortex`. This is a follow up to (and based on) https://github.com/grafana/loki/pull/11014. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR --> --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Christian Haudum	3f28e6362d	Fix flaky bloomgateway test case (#11001 ) The order of active users is non-deterministic, so we need to check for matching elements, rather than for equal slice. Failed test run: https://drone.grafana.net/grafana/loki/29579/3/6 Passed test run: https://drone.grafana.net/grafana/loki/29577/3/6 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Dylan Guedes	9d9b05ad40	Bloomgateway: Use RUnlock to unlock RLock instead (#10984 ) What this PR does / why we need it: Change `Len()` method to use `RUnlock()` instead, since it was locked using `RLock()`.	2 years ago

1 2

54 Commits (1c43991ddcbb801a6a6d7a535062c7c615b0423a)