apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Karsten Jeschkies	483ee56139	Inject downstream accumulator. (#11863 ) What this PR does / why we need it: We inject the downstream accumulator into the downstreamer instead of constructing it on the first arrival of the first result. Since we know the query type before executing the first query we can pass the correct accumulator. This will allow us to define special `topk` or `sum by` accumulators in the future. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	ba0538c2a5	[Blooms] tsdb iterator implementation for bloom construction (#11886 ) Builds a lazy iterator over a tsdb file for bloom construction. This is just the library utility, not the integration.	1 year ago
Christian Haudum	c4ac168a8c	BloomStore: Change signature of `FetchBlocks()` to return `[]CloseableBlockQuerier` (#11884 ) What this PR does / why we need it: In order to encapsulate the reference counting on the `BlockDirectory`, the `BloomStore` now returns a slice of `CloseableBlockQuerier` instead of the directory itself. The caller is responsible for closing the returned querier in order to release the reader resource and decrement the ref counter. The PR also renames `ClosableBlockQuerier` to `CloseableBlockQuerier`. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	7bbbf232d0	Extend BloomStore with `Client(model.Time)` function (#11881 ) What this PR does / why we need it: For certain operations, we want direct access to the `BloomClient`. Since every schema period has its own client, the `Client()` function accepts a `model.Timestamp` for which the client should be returned. The function may return an error, if no client for the given time could be found. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	69d152b618	(chore) BloomStore: Clean up `FetchBlocks()` (#11876 ) What this PR does / why we need it: * Removes `FetchBlocksWithQueue` and replace it with `FetchBlocks` * Operate on single `BlockRef` when loading a block directory from cache/fs/storage. * Ensure order of responses from the FetchBlocks request --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	af66ecec6a	[Blooms] bloomshipper meta integration (#11874 ) Builds on top of https://github.com/grafana/loki/pull/11872, standardizing on the `bloomshipper` types `Meta` and `MetaRef`.	1 year ago
Owen Diehl	3a92834de0	[Blooms] Stateless archiving (#11872 ) Moves the archival code to an interface based approach, removing the explicit fs dependency. We can now implement with in-memory solutions. Also refactors to use a tar-file iterator and marks tests as parallelizable.	1 year ago
Christian Haudum	73edf7a943	(chore) Bloomshipper: Separate store and client (#11865 ) What this PR does / why we need it: This PR removes the `StoreAndClient` interface that was accepted by the `BloomShipper`. Since the `BloomStore` had to not only implement the `Store` interface, but also the `Client` interface, it caused re-implementation of the same methods in different ways. Now the shipper solely relies on the `Store` interface. See individual commit messages for more context. Tests have been rewritten from scratch and placed in their own respective test files for store and client. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Callum Styan	2e3fa3b861	quantile sharding: switch to ddsketch sparse constructor and remove quantile vector pooling in Join function (#11844 ) This should reduce our memory usage for our quantile sketches by about half, at least in our current situation we'll OOM after ~160-170 minutes instead of ~90 😢 --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	1 year ago
Christian Haudum	f97fb093f3	(chore) Bloom store: Rewrite block downloading/caching in bloom store fetcher (#11857 ) What this PR does / why we need it: 1. The `BloomStore` interface gets a new function `FetchBlocks()`, which accepts `[]BlockRef` and returns `[]BlockDirectory`. The fetcher implements a new function `FetchBlocks()` which returns a list of block directories. A block directory represents a local file path that contains the extracted files of a bloom block. It also holds a counter of active readers that access the directory. This is used for safely deleting the directory in case it needs to be removed from disk (eg. max disk/cache size reached). The fetcher resolves the block directory from three places: 1. Cache: The in-memory cache that holds actively accessed directories and keeps track of used disk size. 2. Filesystem: In case the cache was emptied (e.g. when restarting the process) but the block directory is present on disk, it can be re-assambled into a bloom directory that can be put to cache. 3. Storage: If the directory is not present locally, the block archive is downloaded and extracted and the block directory is put to cache. 2. The `{Meta,Block}Client` interfaces are unified: Both clients define the same set of operations: Get{Meta,Block} Get{Meta,Block}s Put{Meta,Block} Delete{Meta,Block}s 3. The `blockDownloader` is replaced by a simple queue powered by a channel. The queue is implemented using generics, so it can be re-used for different request/response types. 4. Code related to "block caching" moved into a separate file `cache.go` and unused code is removed. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Dylan Guedes	9b7e48aca9	Ingesters: Revert "Add metrics.go log line when reading from an ingester (#11571 )" (#11818 ) What this PR does / why we need it: This reverts commit `f04d0db434` Although it is useful, the amount of logging it generates is too massive. In the future we might revisit it but with a different approach.	1 year ago
Salva Corts	0358b6d968	Correctly check chunks in bloom block querier. (#11862 ) What this PR does / why we need it: This PR fixes the block querier to correctly test blooms for chunks. Before this PR, for each chunk we were checking the same set of search terms against the bloom. Therefore, we were not filtering chunks, but only series. This PR uses the chunk ref as a prefix for the search term. Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Ashwanth	69919a303b	ruler: add filtering support for prometheus rules api (#11817 ) What this PR does / why we need it: Adds `rule_name`, `rule_group`, `file` and `type` query parameters for filtering the response of `/prometheus/api/v1/rules` endpoint. Replicates mimir's functionality: https://github.com/grafana/mimir/pull/5291 - all of them are optional. - `type` paremeter accepts either `alert` or `record` - `rule_name`, `rule_group`, `file` can accept multiple values and they filter the response accordingly. There is a minor change in behavior: `/prometheus/api/v1/rules` endpoint will no longer return empty rule groups which is inline with both [prometheus](https://github.com/prometheus/prometheus/pull/12270/files#diff-315f251cdd7e93fcec1e7e9505744da1d1828f30d2b61d1f4ce963fa26bf1909R1403) and [mimir](https://github.com/grafana/mimir/pull/5291/files#diff-e5424c21c0e827bd1c9d3f669ed605897696bdc27993bc8bfd7113eba787b49dR1120). This is not a breaking change since rule groups with [no rules fail validation](`27fbd62505/pkg/ruler/base/manager.go (L295)`) and cannot be created. Which issue(s) this PR fixes: Fixes https://github.com/grafana/loki/issues/9295 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Sandeep Sukhani	c350641a64	otlp: add docs for per tenant otlp config (#11849 ) What this PR does / why we need it: In PR #11143 we added support for per tenant otlp config. This PR adds the relevant documentation to explain how the config looks and works. Checklist - [x] Documentation added --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	1 year ago
Owen Diehl	0f34d9155e	Blooms/block metadata (#11859 ) A few updates to the bloom library: * Uses `FingerprintBounds` in series headers * Encodes `BlockOptions` in the series file so we can later read the target page & block sizes the block was generated with in addition to the schema. * Introduces `BlockMetadata` struct and loads it correctly from blocks. This struct will be used to convert to the `BlockRef`s from the `bloomshipper` pkg and used in the bloom compactor + bloom gateway * Integrates checksums better into block building and XORs the headers metadata from each file (blooms, series) together to generate a final checksum for the block (a combination of both files).	1 year ago
Owen Diehl	d413c28fa3	Blooms: Align compactor to shipper (#11855 ) Does some alignment work between the `bloomcompactor` and the `bloomshipper` pkgs. Notably: * Uses `bloomshipper.BlockRef` everywhere (removes old bloomshipper struct * Integrates `v1.FingerprintBounds` in `Ref` struct * `Location` interface to distinguish local paths vs paths in object storage for certain types (`{Meta,Bloom}Ref`s) * Introduces `KeyResolver` interface to generate locations from these structs * Integrates `KeyResolver` into our bloom store. In the future, this will allow us to change key structures across schema boundaries when we want to change\|improve them. * Removes `BlockPath` from `BlockRef` in favor of the new resolving functionality. This is also beneficial because it lets us _calculate_ locations from the pure Ref objects, rather than tie some arbitrary state to them which can change (or not be populated by accident).	1 year ago
Owen Diehl	098eef7bd4	Blooms/bounds integration (#11848 ) Replaces `Ref`'s min/max fingerprint fields with `v1.FingerprintBounds` and improves a bunch of checks that use it. Builds on top of https://github.com/grafana/loki/pull/11847	1 year ago
Salva Corts	c335cd200e	Extract shard annotation into new FingerprintFilter interface (#11834 ) What this PR does / why we need it: This PR extracts the `Match` and `Bounds` methods of TSDB's `ShardAnnotation` into a new interface `FingerprintFilter`. This will allow us to query the index for any fingerprint bounds, not just power of 2 shard factors. We now use this in the bloom compactor by calling the index `ForSeries` ([here][1]) passing a fingerprint filter with the bounds of the FP range owned by the compactor. Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) [1]: `de4f56e42d/pkg/bloomcompactor/bloomcompactor.go (L408)`	1 year ago
Christian Haudum	1191f88190	(chore) Bloom shipper: Extend `Interval` struct with utility functions (#11841 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	928e3f34f8	Blooms: Prepare compactor refactor (#11847 ) Prepares `bloomcompactor` pkg for integration with new logic, primarily removes/thins out existing code.	1 year ago
Christian Haudum	4411649a0e	(chore) Bloom shipper: Replace `Keyspace` struct with `v1.FingerprintBounds` (#11839 ) The latter struct has more utility functions to compare and operate on bounds. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	509f6f6b66	Bloom/controller wiring (#11831 ) Functionality & testware for building bloom block plans.	1 year ago
Meng Ye	45e5b427b1	docs: fix row_shards doc (#11795 ) for v10 or greater, default is 16 refer to `1002ba00df/pkg/storage/config/schema_config.go (L336)` What this PR does / why we need it: Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	1 year ago
Sandeep Sukhani	7a04da852a	otlp: fix otlp service.name check when no resource attributes are defined (#11837 ) What this PR does / why we need it: When no resource attributes are defined, check for `service.name` resource attributes fail with `nil pointer dereference`. This PR fixes the issue by checking the bool returned by the `Get` method on resource attributes. Checklist - [x] Tests updated	1 year ago
Christian Haudum	6902130c9a	Bloom shipper: Restructure bloom store (#11828 ) What this PR does / why we need it: This PR changes the structure of how the bloom client * The `BloomStore` is the top component interface which implements the `Store` and the `Client` interfaces. The store holds a store entry for each schema period. * The `bloomStoreEntry` implements the `Store` and `Client` interfaces. It holds a bloom client for a single schema period. Additionally, the store entry also exposes a fetcher for metas (and in the future for blocks), which is responsible for getting data from cache or storage (if not available in cache). * The `BloomClient` implement the `Client` interface. The bloom client uses an object client for the schema period of the bloom client. * The `Fetcher` can fetch and cache metas. This structure is very similar to what we use for the chunk store. Note Before implementing `FetchBlocks()` in the `BloomStore`, I want to implement the new `FingerprintBounds` type also in the shipper and bloom gateway code to be consistent across components, as well as to make use of the new utilities exposed by them. The store implementation probably needs some specific test cases around using the correct store entry for different schema period configs. The code however is mostly taken from the chunk store implementation. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Dylan Guedes	3b7ac359bc	Spans: Stop emitting `GetChunks` and `IndexClient.GetChunkRefs` (#11825 ) What this PR does / why we need it: Modify our code to not emit the `GetChunks` and the `IndexClient.GetChunkRefs` spans. We don't worse our observability by getting rid of them because their data is redundant with parent spans. Example: - If an specific index-gateway or ingester struggles, the parent span `/logproto.Querier/GetChunkIDs` will report it just fine - If an specific query causes trouble the `query.Exec` span will show the query name, the `start` and `end`, etc. Also, its children will show the slower parts of the query (in case we have a weak link) Which issue(s) this PR fixes: N/A	1 year ago
Owen Diehl	a56b56c8b0	[Bloom] Keyspace aware bloom creation, library utilities (#11810 ) This PR continues work refactoring the bloom compactor pkg and adds a bunch of utilities for keyspace manipulation/mgmt. A lot of the bloom block logic is contained here as a skeleton. This code is not complete, but is fine to merge as we don't use this yet and should be easier to review before it gets too large.	1 year ago
Vladyslav Diachenko	821c0b1ae0	DequeueMany fix (#11797 ) What this PR does / why we need it: Previously we relied only on the index of the tenant's queue to read the requests from the same tenant's queue. However, as long as the queue is aggressively used by the consumers in parallel, there are a few edge cases when knowing the index of last used tenant's queue is not enough to guarantee that we dequeue items from exactly the same tenant's queue. Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Christian Haudum	ecab33a932	Bloom Gateway: Extract processing logic from bloom worker into separate struct (#11812 ) What this PR does / why we need it: The processor executes a set of tasks. It cleanly separates I/O from logic and therefore is more testable than the current worker implementation. `9602214abf/pkg/bloomgateway/processor.go (L33-L36)` Note: The processor is not used yet in the worker. Certain parts must be refactored when #11810 is merged. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	e0e143a88f	Bloom Gateway: Make tasks cancelable (#11792 ) This PR refactors the bloom gateway workers so that tasks that have been enqueued by requests do not end up locking the results channel and therefore the worker, in case the request was cancelled (`context cancelled`) or timed out (`context deadline exceeded`). It also handles errors from the shipper in a way that they are returned to the waiting request asap so it can return and does not need to wait for all tasks to finish. This PR also fixes the worker shutdown in a way that it now gracefully stops and continues to work off the remaining tasks from the queue. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	c01a823f1f	(chore) Bloom Gateway: Avoid unnecessary merging of tasks prior to passing them to the FusedQuerier (#11809 ) What this PR does / why we need it: The `requestIterator` (`v1.Iterator[v1.Request]`) can be built directly from the task and does not require any unnecessary merging of tasks, since this is done by the `v1.FusedQuerier` already. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	de4f56e42d	[Blooms] Refactoring bloom compactor to isolate state from logic (#11793 ) This PR refactors a bunch of code to separate state+I/O related complexity from logic so we can test and extend it more easily. It also adds more logic for the bloom generator to use.	1 year ago
Salva Corts	d97ac48681	Add timing metrics for compaction (#11785 ) What this PR does / why we need it: This PR modifies the metrics of the bloom compactor to: - Add new histogram metrics to track the time spent to process compaction cycles, tenants and jobs. - Replace `_succeeded` and `_failed` metrics with a new `_completed` metric with a status label. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Trevor Whitney	406b10c675	accidentally merged this from ##11333 (#11788 ) What this PR does / why we need it: I did not intend to merge this change, reverting to what it was before merging #11333	1 year ago
Trevor Whitney	a064f5d670	feat: add release please config (#11333 ) What this PR does / why we need it: The new release process is being developed in another repo for now as it enables much faster feedback loops. We are at the stage where we are pulling the Loki repo and building images, so it's time to move the `release-please` config into Loki, and configure it for the various packages we will be releasing. This PR adds the two basic configs that `release-please` requires. This PR does not change anything about the release process, it just enables us to keep iterating out of band until we're ready to move over. Which issue(s) this PR fixes: Re #11332	1 year ago
Owen Diehl	191466523e	[Blooms] Fix panic when accessing page outside of page bounds (#11787 ) Move variable declaration to only access the desired page when it's valid. Also removes an unused (duplicate) test fn.	1 year ago
Christian Haudum	9f3fd24958	Bloom Shipper: Fix bug in block boundary check (#11784 ) What this PR does / why we need it: The check `isOutsideRange()` returned incorrect results, because the end timestamp of the search interval is not inclusive. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Michel Hollands	2a63d24729	Fix the ObjectExists method on FSObjectClient (#11747 ) What this PR does / why we need it: This method wasn't working. A test was added as well. Special notes for your reviewer: Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [X] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	1 year ago
Christian Haudum	6871f3c555	Bloom Gateway: Partition requests into multiple tasks before enqueuing (#11768 ) What this PR does / why we need it: This PR changes at which point a request is split into multiple tasks in case the chunks boundaries of the request overlap multiple days. This solves the problem that the request handler needs to know for how many responses from the block querier needs to wait. Previously, a single task that spanned across multiple days was executed multiple times (once for each matching day). --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Ashwanth	5c8fd520d3	fix(background cache): increment queue size exactly once (#11776 ) What this PR does / why we need it: While looking at the metrics from one of our cells, we noticed that the queue size is set to a very high value even when the length is pretty much 0. Background cache is incrementing the queue size twice for each enqueued key. This pr removes the additional increment call. `TestBackgroundSizeLimit` might be flaky because of the [writeBackLoop](`8eb09c78c8/pkg/storage/chunk/cache/background.go (L193)`) which dequeues from the channel and reduces the queue size concurrently. To make the test predictable, I have set the `WriteBackGoroutines` to 0. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Christian Haudum	600e390b16	Extend `TestPartitionFingerprintRange` test case (#11750 ) What this PR does / why we need it: The test cases now also cover partially out of bounds blocks as well as inconsecutive block ranges. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	5acc29f2f6	Bloom Gateway: Fix panic in chunk removal after receiving query results (#11770 ) What this PR does / why we need it: While removing not matching chunks from a fingerprint, the for loop over the chunks removes items, and therefore the index of the current iteration needs to be adjusted to accommodate for the removal, otherwise it may panic with `out of bounds`. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Travis Patterson	2c9482d8a2	reset the basemap when the labelsbuilder is reset (#11771 ) This PR fixes an issue where the `label_format` stage sometimes doesn't populate labels when it should. The label `baz` exists on all log lines but is only populated on the first line. ![image](https://github.com/grafana/loki/assets/1413241/86585a94-937f-4e07-9657-b94699ce8586) The issue is this call to [IntoMap](`06bb20914b/pkg/logql/log/fmt.go (L403)`). It tries to avoid copying the map multiple times [here](`06bb20914b/pkg/logql/log/labels.go (L482)`) but the `baseMap` variable isn't reset when the labels builder is reset. As a result, calls to to `IntoMap` after the builder has been reset mean that `!b.hasDel() && !b.hasAdd() && !b.HasErr()` evaluates to `true` but `b.baseMap == nil` also evaluates to `true` so no items are copied. This PR makes it so that `b.baseMap` is reset along with the rest of the builder's state in `Reset`	1 year ago
Owen Diehl	06bb20914b	[Blooms] Specs out BloomGenerator (#11755 ) This is my attempt at making more testable implementations of the work done by the `bloomcompactor` pkg. It's largely inspired by what already exists there, but without depending on things like object storage, chunk clients, etc, it's much easier to locally prove behavior. I'm planning to build on top of this, continuing to integrate work already done in this pkg.	1 year ago
Poyzan	6d66ea382c	[Fix] OR statements not being evaluated as part of nested line filters (#11735 ) What this PR does / why we need it: When nester line filters were being evaluated, the `OR` statements in right expressions were omitted and only `LineFilters` were returned. Resulting in only first value being returned. It can be reproduced in stringer unit test failing with, ``` Error Trace: /Users/poyzannur/workspace/loki/pkg/logql/syntax/ast_test.go:535 Error: Not equal: expected: "{app=\"foo\"} \|= \"foo\" or \"bar\" \|= \"baz\" or \"bal\"" actual : "{app=\"foo\"} \|= \"foo\" or \"bar\" \|= \"baz\"" ``` We now return the `OR` expression as part of nested line filters. Thanks a million to @ashwanthgoli for help with debugging, and extra unit test. Which issue(s) this PR fixes: Fixes https://github.com/grafana/support-escalations/issues/9042 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Sandeep Sukhani	e625d7f1ca	otel: add support for per tenant configuration for mapping otlp data to loki format (#11143 ) What this PR does / why we need it: In OTEL, we pick select Resource Attributes to identify the streams and store all the other attributes as Structured Metadata, as explained here. The problem however is that the list of Resource Attributes that are picked as Stream labels are hardcoded, and there is no way to drop unwanted data. This PR adds support for configuring how data is mapped from OTEL to Loki format per tenant. It also adds support for dropping unwanted data. We decided to make the config look similar to Prometheus's relabling config to make it familiar. Special notes for your reviewer: Opening a draft PR to get some initial feedback. I will add documentation in a separate PR. Checklist - [x] Tests updated - [x] `CHANGELOG.md` updated	1 year ago
Christian Haudum	de3f1786a1	Bloom Gateway: Remove unnecessary wrapper around v1.Request (#11739 ) What this PR does / why we need it: The wrapper `FilterRequest` wraps `v1.Request` but the additional field with the error channel is never used. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Owen Diehl	fda991cdc7	[blooms][testware] Adds testing for the merge builder, sourcing multiple blocks (#11725 ) Adds a few tests to aid in debugging behavior & add some more coverage. Notably tests the `MergeBuilder` sourcing blocks with varying degrees of duplicate data.	1 year ago
Juraj Michálek	0ed536cd7a	Add query-frontend option to select request headers in query logs (#11499 ) What this PR does / why we need it: Adding feature present in mimir, specifically https://github.com/grafana/mimir/pull/5030. Adds a config option to the query-frontend to specify a list of request headers to include in query logs. For example, setting -frontend.log-query-request-headers="X-Grafana-Org-Id" and sending a query with X-Grafana-Org-Id:1 results in query log lines that include header_x_grafana_org_id=1. Which issue(s) this PR fixes: Fixes #11422	1 year ago
Christian Haudum	7c60390364	Remove unnecessary wrapper around bloom shipper (#11724 ) What this PR does / why we need it: The bloom store was initially thought as a translation layer between the bloom gateway and the bloom shipper to be able to simplify the API of the shipper. However, it turned out that the store calls were only a 1-to-1 mapping of the shipper calls and therefore adding unnecessary complexity and call stack. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago

... 3 4 5 6 7 ...

2762 Commits (1f653f02ed40d8d21e3fad01c0aaf5a98b21faf7)