apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Cyril Tovena	1a4436c417	feat(ingester): implement partition shuffle sharding for ingester (#14304 )	1 year ago
Christian Haudum	f52f8ad540	chore: Clarify compression package (#14257 ) This PR renames "encoding" to "codec" in the compression package to remove the cognitive dissonance. It also removes the `Enc` prefix for codec identifiers, so that they adhere Go's best practice of naming conventions, e.g. `compression.EncGZIP` becomes `compression.GZIP` when used in a different package. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	2840d48b92	chore: Clean up unused bloom filter related code (#14183 ) This PR tries to eliminate dead code in the bloom package, mainly related to line tokenization. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Christian Haudum	963779064b	chore: Move compression utilities into separate package (#14167 ) Compression tooling has been part of the `chunkenc` (chunk encoding) package in the past for legacy reasons. Since more components use this now, it's easier to keep it in a separate package. This also eliminates the confusion around "encoding", since this has been incorrectly used synonymously with "compression" in the past. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Robert Fratto <robertfratto@gmail.com>	1 year ago
Trevor Whitney	4f962ef7af	fix: enable service detection for otlp endoint (#14036 ) Co-authored-by: Sven Grossmann <Svennergr@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	1 year ago
Christian Haudum	b75eacc288	chore(blooms)!: Remove bloom compactor component (#13969 ) What this PR does / why we need it: This commit removes the code related to the bloom compactor which is superseded by the bloom planner and builders. A handful of CLI arguments of per-tenant settings changed their prefix from `-bloom-compactor.` to `-bloom-build.`. Other settings for the compactor component itself, also prefixed with `-bloom-compactor.` were removed. See upgrade docs for further information. Special notes for your reviewer*: ✔️ Part of https://github.com/grafana/loki/issues/13957 📔 Documentation update https://github.com/grafana/loki/pull/13965 ⚠️ Integrating the bloom planner and builder into the `backend` target is done in a follow-up PR. --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Salva Corts	b5ac6a0258	feat: Limit to block ingestion until configured date (#13958 )	1 year ago
Owen Diehl	4220737a52	feat(exporter): include boolean values in limit exporter (#13466 )	2 years ago
Salva Corts	47f6ea53fc	feat(loki): Do not enforce max-query-bytes-read and max-querier-bytes-read in limited tripperware (#13406 )	2 years ago
Salva Corts	f7887269f8	refactor(blooms): Limit task retries in bloom planner (#13139 )	2 years ago
Salva Corts	cd64d6ddbd	refactor(blooms): Implement retry mechanisms in planner (#13064 )	2 years ago
Vladyslav Diachenko	1111595179	feat: new stream count limiter (#13006 ) Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	2 years ago
Salva Corts	efd8f5dc1b	refactor(blooms): Add queue to bloom planner and enqueue tasks (#13005 )	2 years ago
Salva Corts	3195036435	refactor(bloom planner): Compute gaps and build tasks from metas and TSDBs (#12994 )	2 years ago
Sandeep Sukhani	7cc9a9386a	fix: fix parsing of default per tenant otlp config (#12836 )	2 years ago
Dylan Guedes	afd9e36306	chore: Call `shardstreams.Config` by value instead of by reference (#12915 ) Use `shardstreams.Config` by value instead of by reference to fix docs generation. Our `docs-generator` tool relies on the struct address/references to assume that flags are present. Using this config by value fixes it.	2 years ago
Shantanu Alshi	6904a6520d	fix: Fix log level detection (#12651 )	2 years ago
Owen Diehl	7bbd8b5087	feat(blooms): ingester aware bounded impl (#12840 )	2 years ago
Owen Diehl	bc78d13d9b	feat(blooms): compute chunks once (#12664 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Owen Diehl	eac56224b8	feat(blooms): limit bloom size during creation (#12796 )	2 years ago
Func	4a05964d55	fix: Use to the proper config names in warning messages (#12114 ) Co-authored-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Cyril Tovena	397aa56e15	feat: Enable log volume endpoint by default (#12628 )	2 years ago
Callum Styan	ec81991f4d	feat: include the stream we failed to create in the stream limit error message (#12437 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Christian Haudum	a9345d0fb6	chore(blooms): Clean up settings and make reasonable defaults (#12483 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Cyril Tovena	9e6e6d22e0	feat: Detect level as structured metadata by default (#12479 )	2 years ago
Ed Welch	7607f04484	feat!: enable structure metadata by default. (#12453 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433 ) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Sandeep Sukhani	b66c343383	feat: add support for discovering and adding log levels as structured metadata (#12428 )	2 years ago
Sandeep Sukhani	288c0061bf	feat: discovery of name of services emitting the logs (#12392 )	2 years ago
crazeteam	3202e34729	chore: fix misorganized annotations (#12351 ) Signed-off-by: crazeteam <lilujing@outlook.com>	2 years ago
Owen Diehl	a36483b731	feat(blooms): bloom integration in query planning (#12208 ) Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	2 years ago
Christian Haudum	eb03974086	feat(blooms): Make bloom block compression configurable (#12293 ) Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Sandeep Sukhani	8b9d8b0226	feat: Make list of otel blessed attributes configurable (#12180 )	2 years ago
Owen Diehl	8a8756aa7a	chore(blooms): honors bloom_compactor_enabled for tenants (#12190 )	2 years ago
Christian Haudum	38e2d7be4a	chore(blooms): Various bloom gateway fixes/improvements (#12120 ) * Response order of `FetchBlocks` is deterministic The response keeps the order of the refs from the request. * Update reference documentation * Skip not found blocks * Unify error messages with object key in bloom client * Change default value for bloom gateway shard size to 0 * Use only 16 tokens per ring instance * Add debug log line to each request in the bloom gateway client Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Kaviraj Kanagaraj	fac5997b18	feat: Support split align and caching for instant metric query results (#11814 ) Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Owen Diehl	85f7baaeda	Blooms/integration fixes (#11979 )	2 years ago
Owen Diehl	5d1798f16d	Blooms/misc fixes (#11974 )	2 years ago
Ashwanth	9e7725b31b	feat(metadata): introduce a separate split interval for recent query window (#11897 )	2 years ago
Salva Corts	7a95cb8663	Configurable bloom tokenizer and block settings (#11889 ) What this PR does / why we need it: This PR makes the max block size configurable and wires up the already existing nGramLen and nGramSkip settings. Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Sandeep Sukhani	e625d7f1ca	otel: add support for per tenant configuration for mapping otlp data to loki format (#11143 ) What this PR does / why we need it: In OTEL, we pick select Resource Attributes to identify the streams and store all the other attributes as Structured Metadata, as explained here. The problem however is that the list of Resource Attributes that are picked as Stream labels are hardcoded, and there is no way to drop unwanted data. This PR adds support for configuring how data is mapped from OTEL to Loki format per tenant. It also adds support for dropping unwanted data. We decided to make the config look similar to Prometheus's relabling config to make it familiar. Special notes for your reviewer: Opening a draft PR to get some initial feedback. I will add documentation in a separate PR. Checklist - [x] Tests updated - [x] `CHANGELOG.md` updated	2 years ago
Ashwanth	51899b521b	feat(metadata cache): adds max_metadata_cache_freshness (#11682 ) What this PR does / why we need it: Adds `max_metadata_cache_freshness` to limit the metadata requests that get cached. When configured, only metadata requests with end time before `now - max_metadata_cache_freshness` are cacheable. _reason for setting the default to 24h?_ metric results cache can [extract samples for the desired time range from an extent](`b6e64e1ef1/pkg/querier/queryrange/queryrangebase/results_cache.go (L78)`) since the samples are associated with a timestamp. But the same is not true for metadata caching, it is not possible to extract a subset of labels/series from a cached extent. As a result, we could return inaccurate results, more that what was requested. for ex: returning results from an entire 1h extent for a 5m query Setting `max_metadata_cache_freshness` to 24h should help us avoid caching recent data. For anything older, we would report cached metadata results at a granularity controlled by `split_metadata_queries_by_interval` Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Vladyslav Diachenko	a5aa8b315d	[bloom-compactor] downloading chunks in batches (#11649 ) What this PR does / why we need it: Added chunks batches iterator to download chunks in batches instead of downloading all of them at once. Otherwise, when the stream contains a lot of chunks, it can lead to OOM. Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Danny Kopping	bcd03150c9	Query-frontend: customisable query splitting for queries overlapping `query_ingester_within` window (#11535 ) What this PR does / why we need it: The config option `query_ingesters_within` defines the window during which logs _could_ be present on ingesters, and as such queriers will send queries to ingesters instead. `split_queries_by_interval` is defined to split queries into subqueries for increased parallelism. Aggressive query splitting within the `query_ingesters_within` window can result in overloading ingesters with unnecessarily large numbers of subqueries, which perversely can impact writes. `query_ingesters_within` is set to 3h by default. In Grafana Cloud Logs we set `split_queries_by_interval` as low as 15m (defaults to 1h), which would result in result in 3*60/15=12 requests. Every querier queries every ingester during this window, so that's 12 requests _per ingester per query_ which has the `query_ingesters_within` window in its time range _(i.e. a query from now to now-7d would include the `query_ingesters_within` window as well, now-3h to now-7d would not)_. However, we _do_ want to split queries so an ingester won't have to handle a query for a full `query_ingesters_within` window - this could involve a large amount of data. To account for this, this PR introduces a new option `split_ingester_queries_by_interval` on the query-frontend; this setting is disabled by default. ![image](https://github.com/grafana/loki/assets/373762/2e671bd8-9e8d-4bf3-addf-bebcfc25e8d7)	2 years ago
Poyzan	599eed7c52	[bloom-compactor] remove BloomCompactorMinTableAge check (#11546 ) What this PR does / why we need it: `BloomCompactorMinTableAge` was added with the idea that we will skip most recent index files from ingesters that are not compacted into TSDB indexes yet. The default value is an hour. This blocks bloom-compacter processing TSBD of the day, because end timestamp is always in the last 15 mins. We can either reduce it something lower than the frequency of TSDB indexes being built (< 15mins). Here I choose to remove it altogether, assuming uncompacted indexes from ingesters will not be processed as a table as there won't be a schema for that table with the [check here](https://github.com/grafana/loki/blob/main/pkg/bloomcompactor/bloomcompactor.go#L268-L272).	2 years ago
Kaviraj Kanagaraj	ce574485cf	feat(caching): Support caching `/series` and `/labels` query results (#11539 ) What this PR does / why we need it: Add support for caching metadata queries (both series and labels). caching happens after splitting similar to other types of queries. This pr adds the following configs to enable them. ``` cache_series_results: true\|false (default false) cache_label_results: true\|false (default false) ``` And the cache backend for them can be configured using `series_results_cache` and `label_results_cache` blocks under the `query_range` section. Currently the split interval for metadata queries is fixed and defaults to 24h, this pr makes it configurable by introducing `split_metadata_queries_by_interval` Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Salva Corts	a0b462d366	Bloom-Gateway cache (#11380 ) What this PR does / why we need it: This PR adds caching to the bloom-gateway client. It uses the result cache from https://github.com/grafana/loki/pull/11343. Here's how we: - Merge responses: group all chunks by FP and remove duplicated chunk checksums. - Extract responses based on time span: For all chunks in each FP, add to the extracted response only the chunks that overlaps with the desired start and end time.	2 years ago
Ashwanth	d9f3bf30b1	config: adds `frontend.max-query-capacity` to tune per-tenant query capacity (#11284 ) What this PR does / why we need it: Adds a new config `frontend.max-query-capacity` that allows users to configure what portion of the the available querier replicas can be used by a tenant. `max_query_capacity` is the corresponding YAML option that can be configured in limits or runtime overrides. For example, setting this to 0.5 would allow a tenant to use half of the available queriers. This complements the existing `frontend.max-queriers-per-tenant`. When both are configured, the smaller value of the resulting querier replica count is considered: ``` min(frontend.max-queriers-per-tenant, ceil(querier_replicas * frontend.max-query-capacity)) ``` All queriers will handle requests for a tenant if neither limits are applied. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: noticed that we don't pass down the shuffle sharding limits for frontend (only using it with schedulers) `26f097162a/pkg/loki/modules.go (L895)` but the [docs](`26f097162a/pkg/validation/limits.go (L276)`) mention that`frontend.max-queriers-per-tenant` applies to frontend as well. ``` This option only works with queriers connecting to the query-frontend / query-scheduler, not when using downstream URL. ``` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Vladyslav Diachenko	75cfe59596	bloom blocks downloading queue (#11201 ) implemented bloom blocks downloading queue to control the concurrency of downloading the blocks from the storage Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Salva Corts	7200c4b95c	Per-tenant n-gram length and skip factor, and bloom false-positive rate (#11290 ) What this PR does / why we need it: This PR adds three per tenant configs: - `bloom_ngram_length`: Configures the n-gram length. - `bloom_ngram_skip`: Configures the n-gram skip factor. - `bloom_false_positive_rate`: Configures the target false-positive rate of the scalable bloom filters. Since the n-gram length and skip factor are now configurable, these values are written into the block metadata so queriers can use the n-gram settings used when creating the blocks to build n-grams compatible with the block.	2 years ago

1 2 3

144 Commits (39b57ec4eac3cbdc718aacae32ab8ff4e989709b)