apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Poyzan	e56e4c92bc	Run compaction over existing bloom blocks (#11486 ) What this PR does / why we need it: Upon changes to TSDB indexes, the existing blooms needs to be amended with newly added chunks, or new series should be added to blocks. This PR compares the existing meta information with the most up to date TSDB information in the job. In case of no changes to indexes, compaction is skipped; if there's any block needs amending, it does that and merges all blocks in fp range; if there's no previous compaction, it compacts from scratch. Special notes for your reviewer: This logic does not support cutting blocks upon topology changes to bloom-compactors. It can create blocks with series outside of the fp range of the compactor. Cutting blocks will be addressed in a follow-up PR.	1 year ago
Karsten Jeschkies	9aae25008d	Optimize series response format by using repeated fileds. (#11498 ) What this PR does / why we need it: The Protobuf map type is encodied as a repeated field of map entries. Decoding them to a slice is much faster than decoding them into a map. Since Loki is not using the fast key check for a map we can use the slice decoding. This change also allows us to decode the JSON directly into the right protobuf struct. This doulbes the JSON decoding speed and reduces the memory pressure by ~40%. ``` › go test -bench=. -run=^$ -count=10 ./pkg/querier/queryrange > before.log › go test -bench=. -run=^$ -count=10 ./pkg/querier/queryrange > after.log › benchstat before.log after.log goos: darwin goarch: arm64 pkg: github.com/grafana/loki/pkg/querier/queryrange │ before.log │ after.log │ │ sec/op │ sec/op vs base │ ResponseMerge/mergeStreams_unlimited-10 32.36m ± 0% 32.63m ± 2% ~ (p=0.393 n=10) ResponseMerge/mergeOrderedNonOverlappingStreams_unlimited-10 1.050m ± 1% 1.080m ± 3% +2.84% (p=0.005 n=10) ResponseMerge/mergeStreams_limited-10 33.02m ± 0% 32.60m ± 1% -1.29% (p=0.004 n=10) ResponseMerge/mergeOrderedNonOverlappingStreams_limited-10 15.11m ± 0% 15.07m ± 0% ~ (p=0.075 n=10) _CodecDecodeLogs-10 4.395m ± 1% 4.364m ± 0% -0.72% (p=0.005 n=10) _CodecDecodeSamples-10 16.97m ± 0% 16.84m ± 2% -0.77% (p=0.023 n=10) _CodecDecodeSeries/application/vnd.google.protobuf-10 745.8µ ± 8% 736.8µ ± 12% ~ (p=0.739 n=10) _CodecDecodeSeries/application/json;_charset=utf-8-10 15.37m ± 1% 10.60m ± 0% -31.03% (p=0.000 n=10) _MergeResponses-10 1186.9m ± 2% 149.8m ± 1% -87.38% (p=0.000 n=10) _UnwrapSeries-10 9.399m ± 1% 4.049m ± 0% -56.92% (p=0.000 n=10) _DecodeMergeEncodeCycle-10 666.0m ± 3% 194.6m ± 0% -70.79% (p=0.000 n=10) geomean 18.87m 12.51m -33.70% │ before.log │ after.log │ │ B/op │ B/op vs base │ _CodecDecodeLogs-10 3.649Mi ± 0% 3.649Mi ± 0% ~ (p=0.364 n=10) _CodecDecodeSamples-10 18.12Mi ± 0% 18.12Mi ± 0% ~ (p=0.926 n=10) _CodecDecodeSeries/application/vnd.google.protobuf-10 7.647Mi ± 0% 7.647Mi ± 0% ~ (p=0.587 n=10) _CodecDecodeSeries/application/json;_charset=utf-8-10 27.94Mi ± 0% 16.99Mi ± 0% -39.18% (p=0.000 n=10) _MergeResponses-10 2.362Mi ± 0% 2.408Mi ± 0% +1.98% (p=0.000 n=10) _UnwrapSeries-10 19.495Mi ± 0% 8.595Mi ± 0% -55.91% (p=0.000 n=10) _DecodeMergeEncodeCycle-10 772.3Mi ± 0% 772.3Mi ± 0% ~ (p=0.912 n=10) geomean 17.50Mi 14.54Mi -16.91% │ before.log │ after.log │ │ allocs/op │ allocs/op vs base │ _CodecDecodeLogs-10 41.10k ± 0% 41.10k ± 0% ~ (p=1.000 n=10) ¹ _CodecDecodeSamples-10 411.9k ± 0% 411.9k ± 0% ~ (p=1.000 n=10) _CodecDecodeSeries/application/vnd.google.protobuf-10 32.00 ± 0% 32.00 ± 0% ~ (p=1.000 n=10) ¹ _CodecDecodeSeries/application/json;_charset=utf-8-10 304.2k ± 0% 298.1k ± 0% -2.01% (p=0.000 n=10) _MergeResponses-10 100.1k ± 0% 100.1k ± 0% -0.00% (p=0.002 n=10) _UnwrapSeries-10 201.1k ± 0% 198.0k ± 0% -1.54% (p=0.000 n=10) _DecodeMergeEncodeCycle-10 203.1k ± 0% 203.1k ± 0% ~ (p=0.621 n=10) geomean 48.95k 48.70k -0.51% ¹ all samples are equal ``` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Periklis Tsirakidis	4c506532f2	operator: Apply linter on all test files (#11501 )	1 year ago
Karsten Jeschkies	27d21067e1	Clone filterer of label filter (#11514 )	1 year ago
Bayan Taani	8931f4a978	operator: Add warning for old schema configuration (#11158 ) Co-authored-by: Robert Jacob <rojacob@redhat.com> Co-authored-by: Periklis Tsirakidis <periklis@nefeli.eu>	1 year ago
na--	724a841f0f	Add log duplication (tee-ing) in the Distributor (#11496 ) This allows us to inject a Tee in the Distributor that can duplicate all of the incoming log stream, before they are replicated.	1 year ago
Callum Styan	e93243ff5b	reduce cost of pipeline stages by pooling maps we create for labels (#11484 ) If we look at profiling of ingesters in some of our production deployments during which we have high ingester load from querying we see that we spend roughly 15-20% of our CPU time on GC related work. Looking at different `runtime` memory related calls, it looks like ~50% of all CPU time during Pipeline stage processing is the creation of the map for labels that is then passed to text template parsing. This change will make the pipeline processing itself ~15% faster but hopefully will also reduce (very slightly) the impact of GC in heavy query load situations. ``` goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/logql/log cpu: AMD Ryzen 9 5950X 16-Core Processor │ base │ mapPool │ │ sec/op │ sec/op vs base │ _Pipeline/pipeline_bytes-32 7.749µ ± ∞ ¹ 6.464µ ± ∞ ¹ ~ (p=1.000 n=1) ² _Pipeline/pipeline_string-32 7.691µ ± ∞ ¹ 6.549µ ± ∞ ¹ ~ (p=1.000 n=1) ² _Pipeline/line_extractor_bytes-32 7.922µ ± ∞ ¹ 6.882µ ± ∞ ¹ ~ (p=1.000 n=1) ² _Pipeline/line_extractor_string-32 7.900µ ± ∞ ¹ 6.878µ ± ∞ ¹ ~ (p=1.000 n=1) ² _Pipeline/label_extractor_bytes-32 7.952µ ± ∞ ¹ 6.976µ ± ∞ ¹ ~ (p=1.000 n=1) ² _Pipeline/label_extractor_string-32 7.940µ ± ∞ ¹ 7.102µ ± ∞ ¹ ~ (p=1.000 n=1) ² geomean 7.858µ 6.805µ -13.41% ¹ need >= 6 samples for confidence interval at level 0.95 ² need >= 4 samples to detect a difference at alpha level 0.05 │ base │ mapPool │ │ B/op │ B/op vs base │ _Pipeline/pipeline_bytes-32 4.963Ki ± ∞ ¹ 1.413Ki ± ∞ ¹ ~ (p=1.000 n=1) ² _Pipeline/pipeline_string-32 5.025Ki ± ∞ ¹ 1.476Ki ± ∞ ¹ ~ (p=1.000 n=1) ² _Pipeline/line_extractor_bytes-32 5.030Ki ± ∞ ¹ 1.478Ki ± ∞ ¹ ~ (p=1.000 n=1) ² _Pipeline/line_extractor_string-32 5.028Ki ± ∞ ¹ 1.478Ki ± ∞ ¹ ~ (p=1.000 n=1) ² _Pipeline/label_extractor_bytes-32 5.027Ki ± ∞ ¹ 1.478Ki ± ∞ ¹ ~ (p=1.000 n=1) ² _Pipeline/label_extractor_string-32 5.027Ki ± ∞ ¹ 1.478Ki ± ∞ ¹ ~ (p=1.000 n=1) ² geomean 5.017Ki 1.466Ki -70.77% ¹ need >= 6 samples for confidence interval at level 0.95 ² need >= 4 samples to detect a difference at alpha level 0.05 │ base │ mapPool │ │ allocs/op │ allocs/op vs base │ _Pipeline/pipeline_bytes-32 37.00 ± ∞ ¹ 35.00 ± ∞ ¹ ~ (p=1.000 n=1) ² _Pipeline/pipeline_string-32 38.00 ± ∞ ¹ 36.00 ± ∞ ¹ ~ (p=1.000 n=1) ² _Pipeline/line_extractor_bytes-32 37.00 ± ∞ ¹ 35.00 ± ∞ ¹ ~ (p=1.000 n=1) ² _Pipeline/line_extractor_string-32 37.00 ± ∞ ¹ 35.00 ± ∞ ¹ ~ (p=1.000 n=1) ² _Pipeline/label_extractor_bytes-32 37.00 ± ∞ ¹ 35.00 ± ∞ ¹ ~ (p=1.000 n=1) ² _Pipeline/label_extractor_string-32 37.00 ± ∞ ¹ 35.00 ± ∞ ¹ ~ (p=1.000 n=1) ² geomean 37.16 35.16 -5.38% ¹ need >= 6 samples for confidence interval at level 0.95 ² need >= 4 samples to detect a difference at alpha level 0.05 ``` --------- Signed-off-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Travis Patterson	be71a80b15	Implement hooks to instrument query pipelines (#11493 ) This PR implements interfaces to inject new pipelines/extractors into the query path of queriers and ingesters.	1 year ago
Andres Perez	b51b7d7b55	Helm: Use `/ingester/shutdown` endpoint in write pod for `preStop` hook (#11490 ) What this PR does / why we need it: Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Ryan Lonergan	a192fc2a1e	Helm: add support for setting an s3 backoff_config via the loki helm chart (#11458 ) What this PR does / why we need it: Currently the Loki helm chart does not support defining s3 backoff config settings. This PR adds support to the loki.commonStorageConfig helper to add backoff config options when they're specified in the helm values Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	1 year ago
Karsten Jeschkies	d048e93f1a	Always set matcher in label filter deserialization. (#11497 ) What this PR does / why we need it: Originally we would not set the matcher of a label filter if the name or value was empty. However , `post != ""` is a valid filter expression with an empty value. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Christian Haudum	797720d3b2	Add jsonnet lib for deploying Bloom Compactor and Bloom Gateway (#11397 ) This PR adds the deployment of the bloom-compactor and the bloom-gateway to the Loki jsonnet lib. Both components are deployed as StatefulSets, using disks backed by PersistentVolumeClaims. Additionally, the bloom-gateways can also be configured with local SSDs when available. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
na--	71e9fcaab7	Refactor Distributor.Push() to be able to Tee log streams (#11488 ) This allows us to duplicate (i.e. "tee") some of the log streams from the `Distributor` by introducing an optional interface in a follow-up PR.	1 year ago
Matthew Penner	a91f3f11b3	frontend: Use `net.JoinHostPort` to support IPv6 addresses (#10650 ) Signed-off-by: Matthew Penner <me@matthewp.io> Co-authored-by: Periklis Tsirakidis <periklis@redhat.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>	1 year ago
Christian Haudum	cd74ddf4b4	Implement default DFS traversal visitor for LogQL expressions (#11489 ) This can be used as a default visitor for use cases where you only want to visit specific expression without the needing to implement the traversal logic of each and every expression, see implementation in test case. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Michel Hollands	37a0ed60a0	Support GET for /ingester/shutdown (#11477 ) Which issue(s) this PR fixes: Fixes #11431 Checklist - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [X] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Michel Hollands <michel.hollands@gmail.com>	1 year ago
Karsten Jeschkies	8822ae660d	Use correct error write function (#11487 ) What this PR does / why we need it: This fixes the last known regression for `protobuf` encoding. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Trevor Whitney	60340639cc	Add trivy filesystem scanning (#11462 ) What this PR does / why we need it: Adds trivy filesystem scanning to our Makefile and PR pipelines. This will scan the `go.mod` file for vulnerabilities in dependencies, which are sometimes missed by just scanning the image. PS @dannykopping this also adds the changes to the vulnerabilities output we talked about that should make it easier to identify a) where the vulnerability is (ie. docker image / os dep vs go.mod dep), and b) what version the dependency needs to be updated to in order to fix it.	1 year ago
Paul Rogers	3da8a29b50	Chore: Spelling fix for linter (#11482 ) What this PR does / why we need it: Just a minor spelling fix in a comment, no functional change. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
J Stickler	6b4d27c955	fix image links in getting started (#11478 ) Fixes two image links as images aren't showing up from previous PR (#11350). https://grafana.com/docs/loki/next/get-started/quick-start/ should display a graphic and a screenshot of Grafana Explore, but only the captions are visible.	1 year ago
Paul Rogers	23780f0cd4	Compress bloom data before uploading it to object storage (#11460 ) What this PR does / why we need it: Currently, the block_downloader.go file is preparing to use the UnTarGz function when retrieving a bloom block from object storage. This PR is to put the corresponding compression in for the upload to object storage. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Kaviraj Kanagaraj	ed1fccfa12	Revert "bugfix(memcached): Make `memcached` batch fetch truely context aware" (#11479 ) Reverts grafana/loki#11363 This PR was causing some issue. Spend some time testing on some internal cell. Still couldn't track down the issue yet. Reverting to make `main` stable for now.	1 year ago
Vladyslav Diachenko	9b691904ef	BloomShipper: add cache for downloaded blocks (#11394 ) adapted embeddedcache.go to store downloaded bloom blocks on local filesystem. What this PR does / why we need it: This cache will be used by bloom-gateway to not download the blocks each time. In the cache we store the objects(`cachedBlock` struct) that contain: 1. `blockDirectory`: the `path` to the directory where the block was extracted 2. `activeQueriers`: thread-safe counter of active block users. This field is important because we do not want the directory to be removed while it's in use. 3. rest fields are needed for service needs When the downloadingWorker receives this entity, it increases the counter `activeQueriers` and creates blockQuerier wrapper that has `Close` function that will decrease the counter once blockQuerier is not needed anymore. When the cache entry is being removed from the cache, the cache calls the function `removeDirectoryAsync` which asynchronously removes the block's folder. This function checks every `Xms` if there are still active block queriers and once `activeQueriers` count is `0` the folder will be removed. Also, there is a timeout in this function, and once the timeout is reached, the folder will be force removed. Special notes for your reviewer: * If the cache is disabled, then the blocks will be downloaded each time the block is requested. * If the cache is used, the folder will be deleted by embeddedCache when it reaches memory size limit or items count limit. Otherwise, the block's folder will be deleted when `Close` function is called. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Karsten Jeschkies	b18078827e	Map rpc Status back to user error (#11449 ) What this PR does / why we need it: Errors forwarded from the querier to the frontend would be reported with the wrong error code and message. This change fixes this. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Joao Marcal	9a3bcba174	operator: Adds structured metadata dashboards (#11473 )	1 year ago
Zirko	f0ec743ba0	Helm: add cilium networkpolicies (#11425 ) What this PR does / why we need it: This PR adds `ciliumnetworkpolicies` that are equivalent to the standard `networkpolicies` already present in the templates. As `cilium` usage as a CNI is rising and the usage of `ciliumnetworkpolicies` is more and more widespread, having the possibility to deploy those directly from a setting in the values would be a time-saving option for a lot of deployments. Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: QuantumEnigmaa <thibaud@giantswarm.io>	1 year ago
Periklis Tsirakidis	bfc2312427	operator: Update Loki operand to v2.9.3 (#11448 )	1 year ago
J Stickler	4614950064	Update Getting Started topic and graphics (#11350 ) What this PR does / why we need it: Which issue(s) this PR fixes: Final PR to complete fixes #9228 Special notes for your reviewer: Updates Getting Started topic to be a general overview, links to Grafana topics, and Agent configuration file. Adds new Quick Start topic for running Loki Docker compose file locally that includes working sample queries. Adds three new graphics Removes two unused graphics	1 year ago
sossickd	f406f528ee	Helm: Added topologySpreadConstraints to read deployment component (#11434 ) What this PR does / why we need it: Add missing topologySpreadConstraints for the read deployment component. Must have been missed in this [PR](https://github.com/grafana/loki/pull/11086) Helm diff: ```console helm diff upgrade loki ./loki -f ~/values-test.yaml -n logging --allow-unreleased logging, loki-read, Deployment (apps) has changed: # Source: loki/templates/read/deployment-read.yaml apiVersion: apps/v1 kind: Deployment ....... + topologySpreadConstraints: + - labelSelector: + matchLabels: + app.kubernetes.io/component: read + app.kubernetes.io/instance: loki + app.kubernetes.io/name: loki + maxSkew: 1 + nodeAffinityPolicy: Honor + nodeTaintsPolicy: Honor + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: ScheduleAnyway ``` Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	1 year ago
J Stickler	49a160c5d0	alphabetize headings and add mini-TOC (#11444 ) What this PR does / why we need it: Replaces @Alex3k PR #9058 which has gone quite stale (file has moved, new content has been added). Adds a mini table of contents at the beginning of the page with links to headings. Rearranges the headings so that they are in alphabetical order. Special notes for your reviewer: Content below line 27 is unchanged and previously published, it has only been moved in the file.	1 year ago
J Stickler	7c4882a99f	Add note about use of object storage (#11443 ) PR #9031 from @Alex3k has gone quite stale as the topic has been revised since it was submitted. This PR attempts to get at least part of Alex's update into the documentation so that we can close his original PR.	1 year ago
Christian Haudum	5f4285963c	Avoid reading block files into memory (#11447 ) Bloom blocks in the wild likely exceed the memory resources of a compactor. Therefore we need to make sure that we don't copy block data into memory, but rather write to and read from disk. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	1 year ago
Kaviraj Kanagaraj	3555001aee	bugfix(memcached): Make `memcached` batch fetch truely context aware (#11363 ) What this PR does / why we need it: Currently, `Fetch` method of memcached is not truely respecting the `ctx` passed to it. It get lost in `fetchKeysBatched` method https://github.com/grafana/loki/blob/489ac8d529ec39352/pkg/storage/chunk/cache/memcached.go#L171-L226 So even if client cancels the context, memcached would still be fetching keys. This PR make it respect context cancelation. Changes: 1. Add select case to check for `ctx.Done()` before submitting work to `inputCh`. 2. Given, now we have two exit point to stop the mecached actor one via old `Stop()` method and other via context cancel, I Added aditional flag `stopped` to make sure we are not closing channels that are already closed. 3. Updated tests to lock this behaviour. Which issue(s) this PR fixes: Fixes NA Special notes for your reviewer: This bug played a role in some incident recently on our Loki SaaS. cc @slim-bean Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	1 year ago
Grot (@grafanabot)	48b60505f6	[CI/CD] Update yaml file `./production/helm/loki/Chart.yaml` (+1 other) (#11442 ) Here is a summary of the updates contained in this PR: * Update attribute `$.appVersion` in yaml file `./production/helm/loki/Chart.yaml` to the following value: `2.9.3` * Bump version of Helm Chart Add changelog entry to `./production/helm/loki/CHANGELOG.md` Re-generate docs Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	1 year ago
Danny Kopping	c7447e2e62	Shortcut callback execution if there are 1 or fewer elements (#11446 ) What this PR does / why we need it: In the case where we have zero elements, we can exit early; with 1 element there's no need for concurrency. This will provide a small optimisation and also make our profiles/execution traces easier to read. We have a fairly significant amount of requests which don't require concurrency.	1 year ago
J Stickler	cdf1f77bb5	Fix typo (#11445 ) What this PR does / why we need it: Fixes small typo from #9874, which has merge conflicts.	1 year ago
Christian Haudum	c4f5a57bc8	Bloom Compactor: Optimize check for fingerprint ownership (#11389 ) Calling `c.sharding.OwnsFingerprint(tenant, uint64(fingerprint))` for each Series of a TSDB index is very expensive, because it not only creates the tenant's sub-ring but also needs to check the fingerprint against it. Instead, we can pre-calculate the current instance's token ranges and check if the (uint32 converted) fingerprint is contained within these ranges. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Periklis Tsirakidis	c573defcbb	operator: Fix storing authentication credentials in the Loki ConfigMap (#11357 ) Co-authored-by: Robert Jacob <xperimental@solidproject.de> Co-authored-by: Robert Jacob <rojacob@redhat.com>	1 year ago
Karsten Jeschkies	8dde7b9d49	Support tail requests with protobuf encoding. (#11426 ) What this PR does / why we need it: The `TailRequest` would use `bytes plan` instead of `Plan plan`. The former is not well supported in gogoprotobuf. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Karsten Jeschkies	f67fff3eb2	Approximate `quantile_over_time` (#10417 ) What this PR does / why we need it: This change shards `quantile_over_time` queries using t-digest or DDSketch approximations. It can be enabled with `querier.shard_aggregations=quantile_over_time`. Outstanding - [x] Replace generic return type of `StepEvaluator` with interface `StepResult`. - [x] Send mapped query with quantile sketch expression from frontend to querier over the wire. - [x] Serialize sketches. See https://github.com/influxdata/tdigest/issues/34 - [x] Add feature flag. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
J Stickler	5e3496739c	Update notes to use shortcode (#11408 ) What this PR does / why we need it: Updates Notes to use the Hugo admonition shortcode. Which issue(s) this PR fixes: Replaces PR #9409 which has gone stale.	1 year ago
Zirko	ee008cff93	Helm: remove ruler.enabled condition in networkpolicies which lead to linting errors (#11305 ) What this PR does / why we need it: This PR removes the `if .Values.ruler.enabled` condition from the `loki-egress-alertmanager` networkpolicy as this was causing linting issues (i.e nil pointer) whenever enabling the networkpolicies since the `ruler` field is not present in the values. From what I know, the ruler is now part of the backend, thus I also replaced the `rulerSelectorLabels` by the `backendSelectorLabels`. Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: QuantumEnigmaa <thibaud@giantswarm.io> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	1 year ago
Sandeep Sukhani	62a2217d83	compactor: fix metric name for a compactor (#11412 ) What this PR does / why we need it: I added a new metric but prefixed it by mistake with `loki_boltdb_shipper` instead of `loki_compactor`. This PR fixes it.	1 year ago
Karsten Jeschkies	51567addc9	Wrap VolumeResponse. (#11402 ) What this PR does / why we need it: This fixes a regression in `frontend.encoding=protobuf`: ``` body rpc error: code = Internal desc = invalid response format, got (queryrange.VolumeResponse) ``` Checklist* - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
J Stickler	f6b5cde1ea	Add alt text to images (#11400 ) Adding alt text to images that were flagged in this week's 404 and build error report.	1 year ago
J Stickler	bc9a00695e	Update structured metadata note (#11399 ) Update the note on the Structured Metadata page now that it is available on Cloud.	1 year ago
na--	e14b3474c1	tests: Disable some `clients/cmd/fluent-bit` tests when `CGO_ENABLED=0` (#11406 ) What this PR does / why we need it: `CGO_ENABLED=0 go test ./...` fails with compile-time errors because the tests are assuming a `logger` variable is defined. However, that variable is defined (and initialized in an `init()`) here, in a file that imports `"C"` and thus requires CGO to be enabled: `5b8d0e666d/clients/cmd/fluent-bit/out_grafana_loki.go (L3-L34)`	1 year ago
Karsten Jeschkies	5b8d0e666d	Remove querier wait time metric. (#11233 ) What this PR does / why we need it: We would like to know how long a querier worker is idle to understand if workstealing would have an impact. The original metric was too noisy and its cardinality was too high. Instead, we are going to log the wait time. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Danny Kopping <dannykopping@gmail.com>	1 year ago
Elliot Dobson	0945b186ff	lambda-promtail: cloudwatch: add '__aws_log_type' label (#11335 ) What this PR does / why we need it: Adds the `__aws_log_type` label to AWS CloudWatch logs in `lambda-promtail`. Which issue(s) this PR fixes: N/A Special notes for your reviewer: AWS S3 & Kinesis log types already have this label. The `lambda-promtail` documentation [here](https://github.com/grafana/loki/blob/main/docs/sources/send-data/lambda-promtail/_index.md?plain=1#L154) suggests that CloudWatch logs has this label added as well, but in practice it does not AFAICT. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Periklis Tsirakidis	e590963471	operator: Add infra annotations for OpenShift based deployments (#11393 )	1 year ago

... 3 4 5 6 7 ...

6197 Commits (1b4d23f9b754458a311ff82fd6b1d52134bcc5e7) All Branches Search

6197 Commits (1b4d23f9b754458a311ff82fd6b1d52134bcc5e7)

All Branches