apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
renovate[bot]	10fe48b815	chore(deps): update alpine docker tag to v3.18.4 (main) (#11167 ) [![Mend Renovate logo banner](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: \| Package \| Type \| Update \| Change \| \|---\|---\|---\|---\| \| alpine \| final \| patch \| `3.18.3` -> `3.18.4` \| \| alpine \| final \| minor \| `3.16.7` -> `3.18.4` \| --- ### Configuration 📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied. ♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 Ignore: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/grafana/loki). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40Ni4wIiwidXBkYXRlZEluVmVyIjoiMzcuNTkuOCIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>	2 years ago
Vladyslav Diachenko	c8093e1ee3	Revert "Compression of bloom blocks (#11267 )" (#11307 ) This reverts commit `af177034` (https://github.com/grafana/loki/pull/11267) because the compression/uncompression must be done outside of the client context: https://raintank-corp.slack.com/archives/C056KUQFBFT/p1700494151990649 Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Periklis Tsirakidis	6b944f7c79	operator: Fix custom CA for object-store in ruler component (#11288 ) Co-authored-by: Robert Jacob <xperimental@solidproject.de>	2 years ago
Salva Corts	7200c4b95c	Per-tenant n-gram length and skip factor, and bloom false-positive rate (#11290 ) What this PR does / why we need it: This PR adds three per tenant configs: - `bloom_ngram_length`: Configures the n-gram length. - `bloom_ngram_skip`: Configures the n-gram skip factor. - `bloom_false_positive_rate`: Configures the target false-positive rate of the scalable bloom filters. Since the n-gram length and skip factor are now configurable, these values are written into the block metadata so queriers can use the n-gram settings used when creating the blocks to build n-grams compatible with the block.	2 years ago
Periklis Tsirakidis	21a07773ba	operator: Add automatic stream sharding support (#11091 ) Co-authored-by: Robert Jacob <xperimental@solidproject.de>	2 years ago
Jan Wozniak	a971dcb7ad	Document caveats for 2.x to 3.x loki helm chart upgrades (#11199 ) What this PR does / why we need it: https://github.com/grafana/loki/issues/9427 discusses a few notable gotchas and differences between `v2.x` and `v3.x` helm charts. In this PR, I would like to propose aggregation of the information from the mentioned issue and appending that to the upgrade guide. Which issue(s) this PR fixes: see also: https://github.com/grafana/loki/issues/9427 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` --------- Signed-off-by: Jan Wozniak <wozniak.jan@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
J Stickler	10a21cfe9b	[Docs] Update Helm installation to include backend (#11191 ) What this PR does / why we need it: Updates the Helm Charts documentation to include `backend` target Fixes broken links Which issue(s) this PR fixes: Fixes #11189 Special notes for your reviewer: I walked through this with @monodot earlier today. Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Poyzan	1d90a8b956	Fix bloom compactor startup duplicate metric registration (#11300 ) What this PR does / why we need it: Follow up from https://github.com/grafana/loki/pull/11285/, where Loki doesn't start up due to duplicate metric registration. ``` 2023-11-22 22:43:32 /src/loki/vendor/github.com/prometheus/client_golang/prometheus/promauto/auto.go:276 +0x178 2023-11-22 22:43:32 github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper/downloads.newMetrics({0x2a62040, 0x4001baa660}) 2023-11-22 22:43:32 /src/loki/pkg/storage/stores/shipper/indexshipper/downloads/metrics.go:21 +0xa4 2023-11-22 22:43:32 github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper/downloads.NewTableManager({{0x4000cdae58, 0x18}, 0x45d964b800, 0x4e94914f0000, 0x0, {0xffff4f32e908, 0x40009c6750}}, 0x24192e0, {0x2a89f50?, 0x40004e7320}, ...) 2023-11-22 22:43:32 /src/loki/pkg/storage/stores/shipper/indexshipper/downloads/table_manager.go:92 +0xa8 2023-11-22 22:43:32 github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper.(indexShipper).init(0x4001b9a480, {0x231b106?, 0x0?}, {0x2a80bf8?, 0x40018b9340?}, {0xffff4f32e908, 0x40009c6750}, 0x0?, {0x0?, 0x0?, ...}, ...) 2023-11-22 22:43:32 /src/loki/pkg/storage/stores/shipper/indexshipper/shipper.go:190 +0x28c 2023-11-22 22:43:32 github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper.NewIndexShipper({_, _}, {{0x4000685900, 0x19}, {0x4000cdae58, 0x18}, 0x4e94914f0000, 0x45d964b800, 0x0, {{0x2313cd3, ...}, ...}, ...}, ...) 2023-11-22 22:43:32 /src/loki/pkg/storage/stores/shipper/indexshipper/shipper.go:155 +0x194 2023-11-22 22:43:32 github.com/grafana/loki/pkg/bloomcompactor.New({{{{0x232535d, 0xa}, {0x232789e, 0xb}, {{...}, {...}, {...}, _}, {_, _}}, ...}, ...}, ...) 2023-11-22 22:43:32 /src/loki/pkg/bloomcompactor/bloomcompactor.go:142 +0x6d4 2023-11-22 22:43:32 github.com/grafana/loki/pkg/loki.(Loki).initBloomCompactor(0x40006dc000) 2023-11-22 22:43:32 /src/loki/pkg/loki/modules.go:1404 +0x358 2023-11-22 22:43:32 github.com/grafana/dskit/modules.(Manager).initModule(0x4000b48318, {0xffffe15d6f7d, 0x4}, 0x403b88?, 0x2313cb3?) 2023-11-22 22:43:32 /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136 +0x1a4 2023-11-22 22:43:32 github.com/grafana/dskit/modules.(Manager).InitModuleServices(0x4000b1a718?, {0x400071c520, 0x1, 0x4000946ec0?}) 2023-11-22 22:43:32 /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108 +0xb4 2023-11-22 22:43:32 github.com/grafana/loki/pkg/loki.(Loki).Run(0x40006dc000, {0x0?, {0x4?, 0x3?, 0x3ff40c0?}}) 2023-11-22 22:43:32 /src/loki/pkg/loki/loki.go:416 +0x74 2023-11-22 22:43:32 main.main() 2023-11-22 22:43:32 /src/loki/cmd/loki/main.go:114 +0xcdc 2023-11-22 22:43:33 panic: a previously registered descriptor with the same fully-qualified name as Desc{fqName: "loki_tsdb_shipper_query_time_table_download_duration_seconds", help: "Time (in seconds) spent in downloading of files per table at query time", constLabels: {}, variableLabels: [{table <nil>}]} has different label names or a different help string ``` Plus a sneaky fix to our loki config, removing a deprecated field. See deprecation clean up here: https://github.com/grafana/loki/pull/11038/files Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist* - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Shashwat Pathak	f3b32023ee	Update _index.md (#11265 ) Replacing the invalid (404) promtail-ec2.yaml with the valid one. What this PR does / why we need it: This PR does - updates the invalid url of the promtail-ec2.yaml with the valid one. Why it is needed - [While implementing this doc](https://grafana.com/docs/loki/latest/send-data/promtail/cloud/ec2/), the curl won't work. Which issue(s) this PR fixes: Fixes #<issue number> No issue has been raised for this Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Bilal Khan	13b58afdd9	made some typo changes in the caching.md file (#11214 ) What this PR does / why we need it: There were some typo mistakes that I fixed in the caching.md file and it was mentioned in the issue that I have given below. Which issue(s) this PR fixes: Fixes [Issue #9073](https://github.com/grafana/loki/issues/9073) Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) Signed-off-by: Bilal Khan <bilalkhanrecovered@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Vitor Bruno de Oliveira Barth	146b1bb10b	Helm: Update MinIO Helm Chart version to 4.0.15 (#11188 ) What this PR does / why we need it: It bumps the version for the MinIO Helm Chart to v4.0.15. The current MinIO version (4.0.12) has a bug where it throws warnings for incorrect types. ``` coalesce.go:237: warning: skipped value for loki.minio.additionalLabels: Not a table. coalesce.go:237: warning: skipped value for loki.minio.additionalAnnotations: Not a table. ``` Which issue(s) this PR fixes: None Special notes for your reviewer: Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [X] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Christian Haudum	b56f36fe6b	Fix bloom compactor startup when using unsupported index type (#11285 ) This PR fixes the check in the bloom compactor startup to ignore all other index types than TSDB. When updating an existing Loki installation that uses BoltDB Shipper in the schema config, you get the following error: ``` invalid tsdb path: /data/boltdb-shipper-cache/XXXXXXXXX_index_19682/write-0-1639160260677624223-1700595900 ││ create index shipper ││ github.com/grafana/loki/pkg/bloomcompactor.New ││ /src/loki/pkg/bloomcompactor/bloomcompactor.go:159 ││ github.com/grafana/loki/pkg/loki.(Loki).initBloomCompactor ││ /src/loki/pkg/loki/modules.go:1406 ││ github.com/grafana/dskit/modules.(Manager).initModule ││ /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136 ││ github.com/grafana/dskit/modules.(Manager).InitModuleServices ││ /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108 ││ github.com/grafana/loki/pkg/loki.(Loki).Run ``` --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Bilal Khan	02b074d889	Added a new paragraph in the contribution guide about an error because it occurred to me and may occur to others also. (#11131 ) What this PR does / why we need it: This PR contains a paragraph in which there is a remember guide when running the `make docs` command. For some users, it may give an error because they have not added the `/tmp` path into the Docker settings. Without this setting, running the `http://localhost:3002/docs/loki/latest/` URL won't work. That's why I added this guide so it becomes easy for others also. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Bilal Khan <bilalkhanrecovered@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com> Co-authored-by: Jack Baldry <jack.baldry@grafana.com>	2 years ago
Paul Rogers	af177034ed	Compression of bloom blocks (#11267 ) What this PR does / why we need it: Compress bloom blocks when writing to object storage, and uncompress when reading from object storage. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Karsten Jeschkies	e6940691c9	Set query plan when copying LokiRequest (#11291 ) What this PR does / why we need it: The recent change https://github.com/grafana/loki/pull/11246 requires that `LokiRequest.Plan` is always set. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Sandeep Sukhani	e28f7f3363	update dskit to latest version (#11287 ) What this PR does / why we need it: Update dskit to latest version	2 years ago
Karsten Jeschkies	5b97fcfd93	Send query plan to querier. (#11246 ) What this PR does / why we need it: Following https://github.com/grafana/loki/pull/11123 and in order to enable https://github.com/grafana/loki/pull/10417 the query frontend should send the serialized LogQL AST instead of the query string to the queriers. This enables the frontend to change the AST and inject expressions that are not expressible in LogQL. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Sandeep Sukhani	162bbb11be	do not run retention loop when retention is not enabled (#11280 ) What this PR does / why we need it: Do not run retention loop when retention is not enabled	2 years ago
Owen Diehl	6e93d150d8	utilities for reducing ngram allocations in token iterator construction (#11276 ) Previously, the `calculatePrefix` function was used to generate the token buffer used, but this wouldn't necessarily have enough space to append the ngram afterwards without reallocating+copying the token buffer. This PR ensures we allocate enough space for both prefix and ngram in the token buffer.	2 years ago
Quentin Bisson	4455cd9d7d	[helm] Fix tracing configuration (#11186 ) What this PR does / why we need it: This PR allows user to enable tracing in the new SSD setup and fixes incorrect documentation because it is currently impossible to enable tracing in this chart (cf. `766f27645d/cmd/loki/main.go (L81)`) Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: QuentinBisson <quentin@giantswarm.io>	2 years ago
Sandeep Sukhani	529084935c	use nanosecond precision for timestamp in compacted boltdb-shipper index file names (#11277 ) What this PR does / why we need it: In PR #9884, we split the compaction and retention loop to run them concurrently. Although we make sure we do not work on the same index from compaction and retention loop, there is a chance that one could run immediately after the other and finish quickly enough to build the index with the same name as the previous one because in `boltdb-shipper` index, we use epoch with `Seconds` precision while building the name of the compacted index file. Since compaction uploads the new file first and then deletes the old file, if the index is built with the same name, we end up uploading the file and deleting it afterwards. This PR fixes the issue by using ns precision for the timestamp in the filenames. Special notes for your reviewer: This is not a problem for TSDB since we also add a checksum to the filenames of the index during compaction.	2 years ago
Pavaningithub	658f4f1f26	Fix Typo in the _index.md (#11242 ) What this PR does / why we need it: Fixing the small typo in the provided example [here](https://grafana.com/docs/loki/latest/setup/install/helm/configure-storage/#configure-storage). This PR just updates the doc. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Bilal Khan	49d5761d90	Fixed the grammatical mistake in an _index.md file (#11226 ) What this PR does / why we need it: In this PR, I fixed the grammatical mistake in an _index.md file. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) Signed-off-by: Bilal Khan <bilalkhanrecovered@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Paul Rogers	5f4a719bb6	tokenizer v1 cleanup (#11272 ) What this PR does / why we need it: Removes all usage of v1 tokenizers, renames v2 to v1 since we never released this in a production way. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Karsten Jeschkies	8d34f857bc	Propagate trace ID with HTTP gRPC request. (#11251 ) The changes in https://github.com/grafana/loki/pull/10688 did not propage the trace ID from the context. `Frontend.RoundTripGRPC` would inject the trace ID into the request. That's not done in `Frontend.Do`. This changes extends the `codec.EncodeRequest` to inject the trace ID there. This is more inline with other metadata.	2 years ago
Sandeep Sukhani	6a62b8cf42	compaction: Separate metrics for tracking retention and compaction (#11263 ) What this PR does / why we need it: In PR #9884, we separated the retention loop from compaction to avoid blocking compaction for too long due to some intensive delete requests. Currently, we track retention and compaction using the same metrics. This PR adds separate metrics for monitoring retention operation. I have also updated the Retention dashboard to use the new metrics.	2 years ago
Joao Marcal	5535267f2c	operator: Remove outdated BoltDB dashboards (#11022 ) Co-authored-by: Periklis Tsirakidis <periklis@redhat.com>	2 years ago
Joao Marcal	0c10f28a46	operator: Adds new value v13 to schema (#10932 ) Co-authored-by: Periklis Tsirakidis <periklis@redhat.com>	2 years ago
Kaviraj Kanagaraj	30d0030bf9	inflight-logging: Add extra metadata to inflight requests logging (#11243 ) What this PR does / why we need it: logging: Add extra metadata to inflight requests This adds extra metadata (similar to what we have in `metrics.go`) but for queries in in-flight (both started and retrying) Changes: Adds following data 1. Query Hash 2. Start and End time 3. Start and End delta 4. Length of the query 5. Moved the helper util to `queryutil` package because of cyclic dependencies with `logql` package. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Find the screenshots of log entries looks like (both in `retry.go` and `roundtrip.go`) ![Screenshot 2023-11-16 at 13 01 32](https://github.com/grafana/loki/assets/3735252/177e97ed-6ee8-41dd-b088-2e4f49562ba0) ![Screenshot 2023-11-16 at 13 02 15](https://github.com/grafana/loki/assets/3735252/fb328a37-dbe3-483e-b083-f21327858029) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Sandeep Sukhani	c716e498ad	compactor: do not block compation when retention is taking too long (#9884 ) What this PR does / why we need it: Currently, we perform compaction and apply retention in the same loop. Although we have a flag for not applying retention every time we perform compaction, we still see compaction getting blocked when processing some intensive delete requests(processed while applying retention). This PR separates out the compaction and retention to run in a separate loop. I have added a table-locking feature to avoid compaction and retention from processing the same tables at a time. However, compaction and retention would treat locked tables differently, as explained below: * When compaction sees a table is locked: It would skip the table and move on to the following table. However, before skipping, it would check if the table has any uncompacted files and increment the newly added counter called `loki_compactor_skipped_compacting_locked_tables_total` to track how often we are skipping compacting tables which have uncompacted files. * When retention sees a table is locked: It would wait for the lock to be released since we can't skip any tables while processing delete requests. Special notes for your reviewer: * The check for tables with uncompacted files looks for count > 1 because initially, we did not support per tenant index in `boltdb-shipper`, so a table can have a single compacted multi-tenant index file. In a rare case where we have a single file which was supposed to be compacted away, it is okay to have a single uncompacted file for a while. The aim here is to not block compaction for too long in a large cell with too many uncompacted files. * Retention only works on the compacted index, so we first compact down any uncompacted files while applying retention. Checklist - [x] Tests updated --------- Co-authored-by: Ashwanth <iamashwanth@gmail.com>	2 years ago
Owen Diehl	58eaad9947	boundscheck & partitioning fingerprints between blocks (#11237 ) Adds a few utilities for comparing fingerprints to blocks that cover a specific fingerprint range. Will likely need to be refactored with more comprehensive types, but the logic still applies.	2 years ago
Owen Diehl	6db9d778a1	inverts the logic when testing a block against a list of chunks (#11248 ) returns the list of chunks in the bloom which failed the test so we can merge these results across blocks	2 years ago
Owen Diehl	15d98fa7ca	iterable tokenizer w/ comparison benching (#11255 ) Rebuilds tokenizers based on an iterable approach. This ends up being significantly faster -- benchmarks below. This PR doesn't remove the old tokenizers, just adds the new ones. Revamping tokenizer consumers & removing the old variants can be done in a separate PR. Roughly, raw tokenizers complete in 66% of the time and chunkID prefixed tokenizers in ~55-60% of the time. ``` pkg: github.com/grafana/loki/pkg/storage/bloom/v1 BenchmarkTokens/three/v1-10 134811 8648 ns/op 0 B/op 0 allocs/op BenchmarkTokens/three/v2-10 179797 5728 ns/op 0 B/op 0 allocs/op BenchmarkTokens/threeSkip1/v1-10 215822 5260 ns/op 0 B/op 0 allocs/op BenchmarkTokens/threeSkip1/v2-10 343784 3514 ns/op 0 B/op 0 allocs/op BenchmarkTokens/threeChunk/v1-10 82394 14559 ns/op 0 B/op 0 allocs/op BenchmarkTokens/threeChunk/v2-10 146973 8138 ns/op 128 B/op 1 allocs/op BenchmarkTokens/threeSkip1Chunk/v1-10 152475 7878 ns/op 0 B/op 0 allocs/op BenchmarkTokens/threeSkip1Chunk/v2-10 251373 4776 ns/op 128 B/op 1 allocs/op PASS ok github.com/grafana/loki/pkg/storage/bloom/v1 10.206s ```	2 years ago
Chris Knutson	5395c36314	ksonnet: generate tsdb_shipper storage_config even if using_boltdb_shipper is false (#11195 ) What this PR does / why we need it: When configuring tsdb shipper without using bolt db shipper, the ksonnet library fails to generate the tsdb_shipper storage config.	2 years ago
Periklis Tsirakidis	8d0fc171cc	operator: Update dependencies and dev tools (#11232 )	2 years ago
Dylan Guedes	4309add1b2	Fix per-pod panel unit (from 'ms' to 's') (#11245 ) What this PR does / why we need it: Change per-pod latency panel unit from 'ms' to 's' (the metric is in the 's' unit)	2 years ago
Trevor Whitney	5c5948367a	Do not run snyk pr comment workflow on forks (#11240 ) Only run the snyk pr comment workflow on PRs from branches, not on forks. We can't run this `on: pull_request_target` because in needs access to the `SNYK_TOKEN` secret, and when run `on: pull_request`, forks don't have permissions to comment on the PR (because they don't get the `GITHUB_TOKEN` secret.	2 years ago
Angus Dippenaar	258a8b3108	Update nix configuration (#8452 ) Fix nix configuration (mainly failing test), and break up various binaries (ie. logcli, promtail, loki, etc.) into their own pakcages. --------- Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	2 years ago
keyolk	8328345a8b	lambda-promtail: fix IAM policy for clouddwatch log stream (#10909 )	2 years ago
Poyzan	d22c1fd39d	[bloom-compactor] Add configs to enable compactor per tenant (#11235 ) What this PR does / why we need it: We want to control bloom compaction per tenant basis. Adding configs to enable/disable bloom compactor. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Poyzan	3f0f8fa5f5	[bloom-compactor] Move meta.json creation at the end of compaction cycle (#11234 ) What this PR does / why we need it: This is a follow up from https://github.com/grafana/loki/pull/11115 Instead of a creating a meta file per bloom creation, create a meta file per compaction cycle. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer:	2 years ago
Karsten Jeschkies	c6f421233b	Increase test coverage for AST serialization to >94%. (#11230 ) What this PR does / why we need it: This is a follow up to https://github.com/grafana/loki/pull/11123 and fixes a few bugs discovered by increasing the test coverage. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Karsten Jeschkies	3a7b5d246b	Serialize query AST to JSON. (#11123 ) What this PR does / why we need it: This introduces the visitor pattern to serialize the LogQL AST to JSON. We've chose this pattern because it will be more flexible one the AST is encoded into Protobuf and an actual queryplan. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Ashwanth	32e9ee8209	chore: follow-up to #11151 and #11025 (#11225 ) What this PR does / why we need it: - remove usage of `enforce_metric_name` in ksonnet - https://github.com/grafana/loki/pull/11151 - make `cortex_ruler_client_request_duration_seconds` prefix configurable - updates upgrade guide to remove reference to metrics that loki does not export Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Karsten Jeschkies	117f108642	Return error if span context was not found (#11207 ) What this PR does / why we need it: This is a follow up to https://github.com/grafana/loki/pull/10956. `lokigrpc.GetParentSpanForRequest` would ignore `opentracing.ErrSpanContextNotFound`. However, that should be up to the caller to decide whether this error should be ignored or not. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	2 years ago
Owen Diehl	265018b729	populate empty resonse for IndexStatsResponse and VolumeResponse (#11209 ) Fixes a panic when the underlying response struct is accessed but not populated after an out-of-bounds request is short-circuited via `NewEmptyResponse` [here](https://github.com/grafana/loki/blob/main/pkg/querier/queryrange/limits.go#L155-L164). This PR ensures the embedded types are not nil in accordance with the other variants in this function.	2 years ago
Christian Haudum	e5b006ce0a	Enable/disable bloom gateway per tenant (#11203 ) This commits adds a per-tenant setting to enable/disable the use of the bloom gateway component for filtering chunks. As an operator, you want to control what tenant can use the filtering of chunks based on search string, because the creation of the bloom filters requires a significant amount of resources. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Karsten Jeschkies	05c4b77ed0	Define QueryResponse and QueryRequest protobufs. (#10956 ) What this PR does / why we need it: This will be a follow up to https://github.com/grafana/loki/pull/10688 finally switching over to protos. Hang on tight! Everything is behind the feature flag `-frontend.encoding=protobuf` which will disable the transcoding to HTTP and HTTPgRPC. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Danny Kopping <dannykopping@gmail.com>	2 years ago
Salva Corts	54edb2143f	Followups #11154 (#11204 ) Addressing comments from #11154 after it was merged	2 years ago
Salva Corts	4248825ad2	Bloom-compactor Sharding (#11154 ) What this PR does / why we need it: This PR adds tenant and fingerprint (FP) sharding to bloom compactors. Note that the bloom-compactor doesn't yet perform any compaction, but iterates through all tables, tenants, and series checking if the compactor owns the tenant and the series (by the series FP). Actual compaction will be implemented with https://github.com/grafana/loki/pull/11115. A new structure `Job` is added which will carry around all the context for a compaction job such as the tenant ID, the table name, and the series FP. The sharding strategy has two methods: - `OwnsTenant(tenant string)`: Checks if the compactor shard owns the tenant. - `OwnsJob(job Job)`: Checks (again) if the compactor owns the job's tenant. Then, it checks if the compactor owns the job's fingerprint by looking inside the tenant subring. We add a new per-tenant limit: `bloom_compactor_shard_size`. If it's 0, the tenant can use all compactors (i.e. `OwnsTenant` will always return `true`), otherwise, only `bloom_compactor_shard_size` out of the total number of compactors will own the tenant. A given job's FP will be owned by exactly one compactor within the tenant shard. Special notes for your reviewer: - Added a bunch of metrics in `metrics.go` - Added a test for the sharding strategy	2 years ago

1 2 3 4 5 ...

5960 Commits (c573defcbbd0928cc0f718e51ef1d15373a3252c) All Branches Search

5960 Commits (c573defcbbd0928cc0f718e51ef1d15373a3252c)

All Branches