apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Paul Rogers	3da8a29b50	Chore: Spelling fix for linter (#11482 ) What this PR does / why we need it: Just a minor spelling fix in a comment, no functional change. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
J Stickler	6b4d27c955	fix image links in getting started (#11478 ) Fixes two image links as images aren't showing up from previous PR (#11350). https://grafana.com/docs/loki/next/get-started/quick-start/ should display a graphic and a screenshot of Grafana Explore, but only the captions are visible.	1 year ago
Paul Rogers	23780f0cd4	Compress bloom data before uploading it to object storage (#11460 ) What this PR does / why we need it: Currently, the block_downloader.go file is preparing to use the UnTarGz function when retrieving a bloom block from object storage. This PR is to put the corresponding compression in for the upload to object storage. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Kaviraj Kanagaraj	ed1fccfa12	Revert "bugfix(memcached): Make `memcached` batch fetch truely context aware" (#11479 ) Reverts grafana/loki#11363 This PR was causing some issue. Spend some time testing on some internal cell. Still couldn't track down the issue yet. Reverting to make `main` stable for now.	1 year ago
Vladyslav Diachenko	9b691904ef	BloomShipper: add cache for downloaded blocks (#11394 ) adapted embeddedcache.go to store downloaded bloom blocks on local filesystem. What this PR does / why we need it: This cache will be used by bloom-gateway to not download the blocks each time. In the cache we store the objects(`cachedBlock` struct) that contain: 1. `blockDirectory`: the `path` to the directory where the block was extracted 2. `activeQueriers`: thread-safe counter of active block users. This field is important because we do not want the directory to be removed while it's in use. 3. rest fields are needed for service needs When the downloadingWorker receives this entity, it increases the counter `activeQueriers` and creates blockQuerier wrapper that has `Close` function that will decrease the counter once blockQuerier is not needed anymore. When the cache entry is being removed from the cache, the cache calls the function `removeDirectoryAsync` which asynchronously removes the block's folder. This function checks every `Xms` if there are still active block queriers and once `activeQueriers` count is `0` the folder will be removed. Also, there is a timeout in this function, and once the timeout is reached, the folder will be force removed. Special notes for your reviewer: * If the cache is disabled, then the blocks will be downloaded each time the block is requested. * If the cache is used, the folder will be deleted by embeddedCache when it reaches memory size limit or items count limit. Otherwise, the block's folder will be deleted when `Close` function is called. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	1 year ago
Karsten Jeschkies	b18078827e	Map rpc Status back to user error (#11449 ) What this PR does / why we need it: Errors forwarded from the querier to the frontend would be reported with the wrong error code and message. This change fixes this. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Joao Marcal	9a3bcba174	operator: Adds structured metadata dashboards (#11473 )	1 year ago
Zirko	f0ec743ba0	Helm: add cilium networkpolicies (#11425 ) What this PR does / why we need it: This PR adds `ciliumnetworkpolicies` that are equivalent to the standard `networkpolicies` already present in the templates. As `cilium` usage as a CNI is rising and the usage of `ciliumnetworkpolicies` is more and more widespread, having the possibility to deploy those directly from a setting in the values would be a time-saving option for a lot of deployments. Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: QuantumEnigmaa <thibaud@giantswarm.io>	1 year ago
Periklis Tsirakidis	bfc2312427	operator: Update Loki operand to v2.9.3 (#11448 )	1 year ago
J Stickler	4614950064	Update Getting Started topic and graphics (#11350 ) What this PR does / why we need it: Which issue(s) this PR fixes: Final PR to complete fixes #9228 Special notes for your reviewer: Updates Getting Started topic to be a general overview, links to Grafana topics, and Agent configuration file. Adds new Quick Start topic for running Loki Docker compose file locally that includes working sample queries. Adds three new graphics Removes two unused graphics	1 year ago
sossickd	f406f528ee	Helm: Added topologySpreadConstraints to read deployment component (#11434 ) What this PR does / why we need it: Add missing topologySpreadConstraints for the read deployment component. Must have been missed in this [PR](https://github.com/grafana/loki/pull/11086) Helm diff: ```console helm diff upgrade loki ./loki -f ~/values-test.yaml -n logging --allow-unreleased logging, loki-read, Deployment (apps) has changed: # Source: loki/templates/read/deployment-read.yaml apiVersion: apps/v1 kind: Deployment ....... + topologySpreadConstraints: + - labelSelector: + matchLabels: + app.kubernetes.io/component: read + app.kubernetes.io/instance: loki + app.kubernetes.io/name: loki + maxSkew: 1 + nodeAffinityPolicy: Honor + nodeTaintsPolicy: Honor + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: ScheduleAnyway ``` Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	1 year ago
J Stickler	49a160c5d0	alphabetize headings and add mini-TOC (#11444 ) What this PR does / why we need it: Replaces @Alex3k PR #9058 which has gone quite stale (file has moved, new content has been added). Adds a mini table of contents at the beginning of the page with links to headings. Rearranges the headings so that they are in alphabetical order. Special notes for your reviewer: Content below line 27 is unchanged and previously published, it has only been moved in the file.	1 year ago
J Stickler	7c4882a99f	Add note about use of object storage (#11443 ) PR #9031 from @Alex3k has gone quite stale as the topic has been revised since it was submitted. This PR attempts to get at least part of Alex's update into the documentation so that we can close his original PR.	1 year ago
Christian Haudum	5f4285963c	Avoid reading block files into memory (#11447 ) Bloom blocks in the wild likely exceed the memory resources of a compactor. Therefore we need to make sure that we don't copy block data into memory, but rather write to and read from disk. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Paul Rogers <paul.rogers@grafana.com>	1 year ago
Kaviraj Kanagaraj	3555001aee	bugfix(memcached): Make `memcached` batch fetch truely context aware (#11363 ) What this PR does / why we need it: Currently, `Fetch` method of memcached is not truely respecting the `ctx` passed to it. It get lost in `fetchKeysBatched` method https://github.com/grafana/loki/blob/489ac8d529ec39352/pkg/storage/chunk/cache/memcached.go#L171-L226 So even if client cancels the context, memcached would still be fetching keys. This PR make it respect context cancelation. Changes: 1. Add select case to check for `ctx.Done()` before submitting work to `inputCh`. 2. Given, now we have two exit point to stop the mecached actor one via old `Stop()` method and other via context cancel, I Added aditional flag `stopped` to make sure we are not closing channels that are already closed. 3. Updated tests to lock this behaviour. Which issue(s) this PR fixes: Fixes NA Special notes for your reviewer: This bug played a role in some incident recently on our Loki SaaS. cc @slim-bean Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	1 year ago
Grot (@grafanabot)	48b60505f6	[CI/CD] Update yaml file `./production/helm/loki/Chart.yaml` (+1 other) (#11442 ) Here is a summary of the updates contained in this PR: * Update attribute `$.appVersion` in yaml file `./production/helm/loki/Chart.yaml` to the following value: `2.9.3` * Bump version of Helm Chart Add changelog entry to `./production/helm/loki/CHANGELOG.md` Re-generate docs Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	1 year ago
Danny Kopping	c7447e2e62	Shortcut callback execution if there are 1 or fewer elements (#11446 ) What this PR does / why we need it: In the case where we have zero elements, we can exit early; with 1 element there's no need for concurrency. This will provide a small optimisation and also make our profiles/execution traces easier to read. We have a fairly significant amount of requests which don't require concurrency.	1 year ago
J Stickler	cdf1f77bb5	Fix typo (#11445 ) What this PR does / why we need it: Fixes small typo from #9874, which has merge conflicts.	1 year ago
Christian Haudum	c4f5a57bc8	Bloom Compactor: Optimize check for fingerprint ownership (#11389 ) Calling `c.sharding.OwnsFingerprint(tenant, uint64(fingerprint))` for each Series of a TSDB index is very expensive, because it not only creates the tenant's sub-ring but also needs to check the fingerprint against it. Instead, we can pre-calculate the current instance's token ranges and check if the (uint32 converted) fingerprint is contained within these ranges. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Periklis Tsirakidis	c573defcbb	operator: Fix storing authentication credentials in the Loki ConfigMap (#11357 ) Co-authored-by: Robert Jacob <xperimental@solidproject.de> Co-authored-by: Robert Jacob <rojacob@redhat.com>	1 year ago
Karsten Jeschkies	8dde7b9d49	Support tail requests with protobuf encoding. (#11426 ) What this PR does / why we need it: The `TailRequest` would use `bytes plan` instead of `Plan plan`. The former is not well supported in gogoprotobuf. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Karsten Jeschkies	f67fff3eb2	Approximate `quantile_over_time` (#10417 ) What this PR does / why we need it: This change shards `quantile_over_time` queries using t-digest or DDSketch approximations. It can be enabled with `querier.shard_aggregations=quantile_over_time`. Outstanding - [x] Replace generic return type of `StepEvaluator` with interface `StepResult`. - [x] Send mapped query with quantile sketch expression from frontend to querier over the wire. - [x] Serialize sketches. See https://github.com/influxdata/tdigest/issues/34 - [x] Add feature flag. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
J Stickler	5e3496739c	Update notes to use shortcode (#11408 ) What this PR does / why we need it: Updates Notes to use the Hugo admonition shortcode. Which issue(s) this PR fixes: Replaces PR #9409 which has gone stale.	1 year ago
Zirko	ee008cff93	Helm: remove ruler.enabled condition in networkpolicies which lead to linting errors (#11305 ) What this PR does / why we need it: This PR removes the `if .Values.ruler.enabled` condition from the `loki-egress-alertmanager` networkpolicy as this was causing linting issues (i.e nil pointer) whenever enabling the networkpolicies since the `ruler` field is not present in the values. From what I know, the ruler is now part of the backend, thus I also replaced the `rulerSelectorLabels` by the `backendSelectorLabels`. Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Signed-off-by: QuantumEnigmaa <thibaud@giantswarm.io> Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	1 year ago
Sandeep Sukhani	62a2217d83	compactor: fix metric name for a compactor (#11412 ) What this PR does / why we need it: I added a new metric but prefixed it by mistake with `loki_boltdb_shipper` instead of `loki_compactor`. This PR fixes it.	1 year ago
Karsten Jeschkies	51567addc9	Wrap VolumeResponse. (#11402 ) What this PR does / why we need it: This fixes a regression in `frontend.encoding=protobuf`: ``` body rpc error: code = Internal desc = invalid response format, got (queryrange.VolumeResponse) ``` Checklist* - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
J Stickler	f6b5cde1ea	Add alt text to images (#11400 ) Adding alt text to images that were flagged in this week's 404 and build error report.	1 year ago
J Stickler	bc9a00695e	Update structured metadata note (#11399 ) Update the note on the Structured Metadata page now that it is available on Cloud.	1 year ago
na--	e14b3474c1	tests: Disable some `clients/cmd/fluent-bit` tests when `CGO_ENABLED=0` (#11406 ) What this PR does / why we need it: `CGO_ENABLED=0 go test ./...` fails with compile-time errors because the tests are assuming a `logger` variable is defined. However, that variable is defined (and initialized in an `init()`) here, in a file that imports `"C"` and thus requires CGO to be enabled: `5b8d0e666d/clients/cmd/fluent-bit/out_grafana_loki.go (L3-L34)`	1 year ago
Karsten Jeschkies	5b8d0e666d	Remove querier wait time metric. (#11233 ) What this PR does / why we need it: We would like to know how long a querier worker is idle to understand if workstealing would have an impact. The original metric was too noisy and its cardinality was too high. Instead, we are going to log the wait time. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Danny Kopping <dannykopping@gmail.com>	1 year ago
Elliot Dobson	0945b186ff	lambda-promtail: cloudwatch: add '__aws_log_type' label (#11335 ) What this PR does / why we need it: Adds the `__aws_log_type` label to AWS CloudWatch logs in `lambda-promtail`. Which issue(s) this PR fixes: N/A Special notes for your reviewer: AWS S3 & Kinesis log types already have this label. The `lambda-promtail` documentation [here](https://github.com/grafana/loki/blob/main/docs/sources/send-data/lambda-promtail/_index.md?plain=1#L154) suggests that CloudWatch logs has this label added as well, but in practice it does not AFAICT. Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`)	1 year ago
Periklis Tsirakidis	e590963471	operator: Add infra annotations for OpenShift based deployments (#11393 )	1 year ago
Periklis Tsirakidis	d683aedb1b	operator: Add tools to test on Azure (#11340 ) )	1 year ago
Salva Corts	a0b462d366	Bloom-Gateway cache (#11380 ) What this PR does / why we need it: This PR adds caching to the bloom-gateway client. It uses the result cache from https://github.com/grafana/loki/pull/11343. Here's how we: - Merge responses: group all chunks by FP and remove duplicated chunk checksums. - Extract responses based on time span: For all chunks in each FP, add to the extracted response only the chunks that overlaps with the desired start and end time.	1 year ago
Christian Haudum	79693d79ef	Fix `getFromThrough()` function for ShortRefs (#11386 ) What this PR does / why we need it: Since chunks don't necessarily have the same duration, we cannot assume that the last item of a slice sorted by `From` time also has the greatest `Through` time. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	1 year ago
Gregor Zeitlinger	abb63b21b9	Update template_functions.md (#9683 ) My previous fix (part of #9672) was wrong - reverting the change Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
Pangidoan Butar	7f79db908b	Update log configuration (#11211 ) What this PR does / why we need it: Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>	2 years ago
akevdmeer	18778cd548	De-duplicate common prefixes as returned for individual buckets (#11317 ) With multiple buckets configured on a single AWS/S3 store, each object is stored on one of the buckets by hashing its object key in `bucketFromKey()`. That is sufficient for `GetObject()` and `PutObject()`. `List()` needs to list each bucket and combine the results. It appends all object keys into a `storageObjects` slice. That works because each key uniquely exists in only one of the buckets. But that is not the case for the common prefixes; a common prefix may exist in multiple buckets. This PR removes duplicates from the common prefixes as gathered from the multiple buckets in `List()`. Without this fix, we find that a repeated common prefix leads to a table being compacted multiple times concurrently which is not safe. E.g. this error results when the compactor concurrently tries to compact a table and a 'losing' execution finds that its clean-up has already been done: `level=error caller=compactor.go:128 table-name=loki_index_tsdb_19683 msg="failed to remove downloaded index file" path=/var/loki/compactor/loki_index_tsdb_19683/1700667008-loki-write-0-1700660468255353690.tsdb err="remove /var/loki/compactor/loki_index_tsdb_19683/1700667008-loki-write-0-1700660468255353690.tsdb: no such file or directory" `	2 years ago
Christian Haudum	856b57336c	Pass `--build-arg` to docker-driver build command (#11379 ) This fixes an issue with the build system when the Go version is updated in the build image and go.mod relies on the new features of the Go version. See failed CI run https://drone.grafana.net/grafana/loki/31777/28/2 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Christian Haudum	19b0160821	Bloom gateway client: Use binary search to determin membership of fingerprints to server instances (#11336 ) Instead of iterating over all fingerprints and determine their respective instance from the ring individually, this new approach takes calculates the token ranges from the instances from the ring and matches and partitions the fingerprints based on that using a binary search to determine membership to a token range. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Salva Corts	489ac8d529	Extract results cache into new pkg (#11343 ) What this PR does / why we need it: This extracts the results cache from `queryrangebase` into its own pkg so we can reuse it in other components such as the bloom-gateway without having to import `queryrangebase`. - Most of the logic inside `pkg/querier/queryrange/queryrangebase/results_cache.go` now lives in `pkg/storage/chunk/cache/results_cache/cache.go`. - Some of the tests in `pkg/querier/queryrange/queryrangebase/results_cache.go` are moved into pkg/storage/chunk/cache/results_cache/cache_test.go. - Note that here we don't have access to the types we use in `queryrangebase` so we created a new set of mock request/response types to test with.	2 years ago
Christian Haudum	0e433f304e	Bloom Gateway: process blocks immediately when they are available (#11360 ) At current point in time, the worker calls `w.store.GetBlockQueriersForBlockRefs()` to obtain a list of block queriers sorted by fingerprint in ascending order. However, the shipper used by the store to fetch blocks by its refs, calls a callback function when a requested block is available. Instead of waiting for all callbacks to be called and then returning a single list of block queriers, the store now exposes a new function `ForEach()` that also accepts a callback that is passed to the underlying shipper. This allows to process blocks in the worker as soon as they become available. Some block may be served from cache, some may need to be downloaded. That way, we reduce waiting time. --------- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Paul Rogers	9538499e60	Add error return to PopulateSeriesWithBloom (#11359 ) What this PR does / why we need it: Adds an error return value to PopulateSeriesWithBloom Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Ashwanth	d9f3bf30b1	config: adds `frontend.max-query-capacity` to tune per-tenant query capacity (#11284 ) What this PR does / why we need it: Adds a new config `frontend.max-query-capacity` that allows users to configure what portion of the the available querier replicas can be used by a tenant. `max_query_capacity` is the corresponding YAML option that can be configured in limits or runtime overrides. For example, setting this to 0.5 would allow a tenant to use half of the available queriers. This complements the existing `frontend.max-queriers-per-tenant`. When both are configured, the smaller value of the resulting querier replica count is considered: ``` min(frontend.max-queriers-per-tenant, ceil(querier_replicas * frontend.max-query-capacity)) ``` All queriers will handle requests for a tenant if neither limits are applied. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: noticed that we don't pass down the shuffle sharding limits for frontend (only using it with schedulers) `26f097162a/pkg/loki/modules.go (L895)` but the [docs](`26f097162a/pkg/validation/limits.go (L276)`) mention that`frontend.max-queriers-per-tenant` applies to frontend as well. ``` This option only works with queriers connecting to the query-frontend / query-scheduler, not when using downstream URL. ``` Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Paul Rogers	ceda046229	Bloom tokenization performance tweaks (#11338 ) What this PR does / why we need it: The main work of this PR is to improve the performance of the reassemble() function. By changing the assignment of the 'cur' variable to not do addition, and incrementing the 'pos' variable separately, the benchmarks improved as follows: BenchmarkTokens/threeSkip1Chunk/v2-10 88864 13487 ns/op BenchmarkTokens/threeSkip1ChunkAt2/v2At2-10 98607 12256 ns/op (This was validated with multiple runs, on multiple size inputs) The remaining changes in the PR are just to add some benchmarking for PopulateSeriesWithBloom, in case we need to do other comparisons for future improvements.	2 years ago
Poyzan	11ef833ed3	Compact multiple series into a bloom block (#11295 ) What this PR does / why we need it: The existing `Job` abstraction has 1 stream fingerprint and compactor was making a 1bloom and 1 block at the end of compaction. This PR changes the `Job` struct to have a list of stream fingerprints. At the end of compaction n streams are iterated over to create n blooms that are appended to 1 block. Stream fingerprints are added to the Job after checking if they are in owned by that compactor's ring, so it is still benefiting from sharding strategy. Resulting block will only contain series blooms that belong to 1 tsdb table. relates to https://github.com/grafana/loki-private/issues/856 Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Logic to control block size will be added in a follow up PR.	2 years ago
someStrangerFromTheAbyss	0b9e573140	Helm: Add extraContainers to write pods (#11319 ) Adds the extraContainers to the write pod: Which issue(s) this PR fixes: This fix will allow my team to add a pods that makes modification to the incoming logs before sending them to loki with minimal network transfer. Special notes for your reviewer: NONE Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [x] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) - [x] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](`0d4416a4b0`) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Dmitriy Okladin	a106b061cb	jsonnet: add memberlist_ring_enabled check condition for consul_config_map (#11312 ) The jsonnet lib should not generate a ConfigMap for consul if you are using memberlist. Fixes #11311	2 years ago
Rens Groothuijsen	cf84b2e9b8	fluent-bit: Redact basic auth credentials from fluent-bit logging (#11308 ) This redacts the basic auth credentials from the configured URL that is displayed when fluent-bit is started, if these are included. Go's built-in redaction only hides the password, turning a url like `http://user:pass@example.com` into `http://user:xxxxx@example.com`. Fixes #10514	2 years ago
Christian Haudum	fe4d9bea68	Remove worker callback function that is only used in tests (#11356 ) The queue already exposes the functionality to track connected consumers (workers), so there is no need to have callback that is only used in tests. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago

1 2 3 4 5 ...

5979 Commits (3da8a29b50c49205a62ec3084622d5c0a9995660) All Branches Search

5979 Commits (3da8a29b50c49205a62ec3084622d5c0a9995660)

All Branches