apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Christian Haudum	3ac231728e	fix(blooms): Suppress error from resolving server addresses for blocks (#13385 ) Instead of returning the error, `FilterChunks` returns the full, unfiltered list of chunks. This makes sure that queries do not fail if no bloom gateway instances are available. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	3b0502ddd3	chore(blooms): Various minor code cleanups (#13332 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	a0f358fcc8	fix(blooms): Fully deduplicate chunks from FilterChunkRef responses (#12807 ) This PR aims for full de-duplication of chunks and series from filter requests from the index gateway to the bloom gateway. Whenever we merge/de-duplicate slices, the inputs need to be sorted. It appears that the Removals (chunks) from the v1.Output are not guaranteed to be sorted. When comparing ShortRefs, both From, Through, and Checksum need to be used. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	3bf2d1fea0	fix(blooms): Deduplicate filtered series and chunks (#12791 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	5540c92d50	perf(blooms): Resolve bloom blocks on index gateway and shard by block address (#12720 ) This pull request changes how data is sharded across bloom gateways. Currently, chunks are grouped by series and shared by the fingerprint of the series across the available bloom gateways using jumphash algorithm. This however, leads to over-querying bloom blocks, because bloom blocks have consecutive fingerprint ranges, whereas sharding keys are evenly distributed across keyspace. This PR changes the sharding in the way that bloom blocks for series are already resolved on the index gateways and that their address is used for sharding data. This has the advantage that the grouped series can be mapped to the correct bloom blocks on the client side. Sending the block ref along with the grouped series to the bloom gateway allows for efficient querying of the data because each bloom gateway therefore owns exactly 1/nth of the blocks. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	58c8c61905	perf(blooms): Use jumpmap for client side sharding of the filter requests (#12470 ) What this PR does / why we need it: Use the same client-side sharding mechanism for bloom gateway filter requests that we use for client side sharding in the memcached client. This uses DNS discovery to resolve a list of addresses uses as buckets for the jumphash algorithm to shard GroupedChunkRefs across bloom gateway servers. The ring component is still part of the bloom gateway, but it is not used any more. Special notes for your reviewer: https://arxiv.org/pdf/1406.2294.pdf Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Owen Diehl	a6f3b93f7e	chore(blooms): bloom_client records cache locality score (#12458 ) This is a helpful way to measure how effectively we distribute filter queries to bloom-gws. Should mostly converge to 1 once we start using jump-hash instead of the ring for these, but it'll give us a good way to measure that improvement.	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Salva Corts	4fa5148eb5	refactor: Pass query plan down to bloom gateway (#12037 )	2 years ago
Christian Haudum	1f56da206d	refactor: Simplify fingerprint partitioning in bloom gateway client (#12018 ) Use the recently introduced `tokenRangesForInstance()` function to determine the the ranges for each replication set. For this purpose, the function moves into the `bloomutils` packages. This PR eliminates the structs `addrsWithBounds` and `instanceWithFingerprints` (which had somewhat similar usage) and unifies them into `rsWithRanges` that holds both the fingerprint ranges and the assigned GroupedChunkRefs. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	71fa802c86	Map ring token keyspace (uint32) into fingerprint keyspace (uint64) (#11975 ) In order to compare the keyspace from the bloom gateway's ring with the fingerprints, they need to use the same keyspace. The ring, however, uses tokens within a `uint32` keyspace, whereas the fingerprints use a `uint64` keyspace. Therefore the ranges from the ring tokens need to be mapped into the `uint64` keyspace. This is done bit-shifting the Min value 32 bits to the left and the Max value 32 bits to the left plus adding `(1<<32)-1` to "fill" the remaining values up to the next Min value. The structs used to combined ring instance information and token/fingerprint ranges are yet to be simplified. However, this is not goal of this PR. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Christian Haudum	31711d5558	(chore) Bloom gateway: Improve fingerprint partitioning in client (#11971 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	5967ceeb43	Replace min/max token with `TokenRange` in bloom ring utilities (#11960 ) This PR replaces min/max token fields with a `TokenRange` field that uses the `Range[uint32]` type. The `Range[uint32]` uses similar semantics as the `FingerprintBounds` we use for fingerprint ranges. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	60551dacc4	Remove unused sharding code from bloom gateway (#11906 ) The sharding by fingerprint range is done on the client side, not on the server side. The sharding related code was not used in the gateway. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Paul Rogers	6c4699d8f7	Remove call to set default resolver (#11580 ) What this PR does / why we need it: Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Christian Haudum	55fa0d19e4	Fix panic in bloom gateway client (#11537 ) The bloom gateway client panicked, because the ring, from which it gets the subring for the tenant was never passed as argument in the constructor, even though it was part of the client struct. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	c4f5a57bc8	Bloom Compactor: Optimize check for fingerprint ownership (#11389 ) Calling `c.sharding.OwnsFingerprint(tenant, uint64(fingerprint))` for each Series of a TSDB index is very expensive, because it not only creates the tenant's sub-ring but also needs to check the fingerprint against it. Instead, we can pre-calculate the current instance's token ranges and check if the (uint32 converted) fingerprint is contained within these ranges. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	a0b462d366	Bloom-Gateway cache (#11380 ) What this PR does / why we need it: This PR adds caching to the bloom-gateway client. It uses the result cache from https://github.com/grafana/loki/pull/11343. Here's how we: - Merge responses: group all chunks by FP and remove duplicated chunk checksums. - Extract responses based on time span: For all chunks in each FP, add to the extracted response only the chunks that overlaps with the desired start and end time.	2 years ago
Christian Haudum	19b0160821	Bloom gateway client: Use binary search to determin membership of fingerprints to server instances (#11336 ) Instead of iterating over all fingerprints and determine their respective instance from the ring individually, this new approach takes calculates the token ranges from the instances from the ring and matches and partitions the fingerprints based on that using a binary search to determine membership to a token range. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Michel Hollands	e93f5bfd93	Add dashboard changes for migrating from cortex_ metrics to loki_ metrics (#11082 ) What this PR does / why we need it: - add missing namespace param for ruler manager metrics - update the loki mixin to use the renamed metrics so both are supported - created the compiled mixin - make the prefix of the cortex_distributor_ingester_clients metrics configurable Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	2 years ago
Christian Haudum	d84026cab6	Adjust bloom gateway request structure (#10926 ) What this PR does / why we need it: Since we already group chunkRefs by fingerprint on the client side to determine the correct bloom gateway to send the request to, we can keep the grouping for the gRPC request. This will not only optimize the request size, but also helps to reduce complexity of querying blooms on the bloom gateway itself. There is still conversion between `[]logproto.ShortRef` and `v1.ChunkRefs` required, which could be removed by sending `v1.ChunkRefs` over the wire. Special notes for your reviewer*: Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	b49b3ce6e1	Add basic structure of bloom gateways (#10782 ) ### Summary This pull requests adds the basic structure for the new bloom gateway component. - Adds new `bloom-gateway` target that runs with multiple instances joined by a ring - Adds a querier and client component on the index gateway to filter chunk refs - Adds the gRPC protobuf definitions for commication between index gateways and bloom gateways - Adds a store component used on the bloom gateways to query binary bloom files ``` Querier Query Frontend \| \| ................................... service boundary \| \| +----+------+ \| indexgateway.Gateway \| bloomgateway.BloomQuerier \| bloomgateway.GatewayClient \| logproto.BloomGatewayClient \| ................................... service boundary \| bloomgateway.Gateway \| bloomshipper.Store \| bloomshipper.Shipper \| bloomshipper.BloomFileClient \| ObjectClient \| ................................... service boundary \| object storage not part of this PR ``` This PR still contains a lot of TODOs and possibilities for optimisations, which will be addressed in subsequent pull requests. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago

22 Commits (0c39e1529fc0bf1d7c3eff5435f3045a70dda776)