apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Bryan Boreham	cf353bb9a0	Series Index Store: fix race in GetSeries (#10310 )	2 years ago
Danny Kopping	9734c4b832	Minor fixes to congestion control mechanism (#10319 ) What this PR does / why we need it: Minor logging / metric improvements to #10140	2 years ago
Kaviraj Kanagaraj	416320dc5f	Revert "Stopped String() from doubly escaping regex characters" (#10318 ) Reverts grafana/loki#10277 This currently breaks and cause panics for query with regex label matcher. Reverting it till we find the fix.	2 years ago
Bryan Boreham	f89a51fde8	Tracing: elide small traces for Stats call (#10308 ) What this PR does / why we need it: At work I see a lot of traces getting data dropped due to being tens of megabytes in size. Looking at what was captured, I see many many repeats of these four spans: <img width="1070" alt="image" src="https://github.com/grafana/loki/assets/8125524/4268e664-6910-49bb-8f68-69b139a5a66b"> Events on the parent trace are cheaper and, for small operations like this, just as useful. Where there was an event line I added a `function` field matching the previous span name. For `storeEntry.Stats()` I didn't add an event line as it doesn't seem to do very much. But please correct me; I'm not hugely familiar with the Loki codebase. Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - NA Documentation added - NA Tests updated - [x] `CHANGELOG.md` updated - NA If the change is worth mentioning in the release notes, add `add-to-release-notes` label - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - NA For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Bryan Boreham	ecd371db87	indexshipper/compactor: fix two race conditions (#10313 ) What this PR does / why we need it: We must call wg.Add before starting the goroutine. And protect the data in mockCompactorClient. And another fix in shipper/index/compactor. Which issue(s) this PR fixes: Part of #8586 Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - NA Documentation added - NA Tests updated - NA `CHANGELOG.md` updated - NA If the change is worth mentioning in the release notes, add `add-to-release-notes` label - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - NA For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
loanshark	43e542ee64	Storage: Remove `SignatureVersionv2` from the AWS S3 API request authentication. (#10295 ) AWS S3 stopped accepting requests signed using SigV2 in all regions on June 24, 2019, any requests signed using SigV2 made after this time will fail. * https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingAWSSDK.html#UsingAWSSDK-sig2-deprecation * https://aws.amazon.com/blogs/aws/amazon-s3-update-sigv2-deprecation-period-extended-modified/	2 years ago
Owen Diehl	f53498c4aa	allows filter-only matchers in series endpoint (#10269 ) We already allow empty matchers `{}` as a special case in series queries to return all series. This PR also allows specifying only filter-style matchers. This is specifically helpful when a tenant is trying to understand all streams which do _not_ have a corresponding label. ``` $ logcli series '{does_not_exist=""}' --analyze-labels --org-id="fake" 2023/08/16 10:13:32 http://localhost:3100/loki/api/v1/series?end=1692206012889997000&match=%7Bdoes_not_exist%3D%22%22%7D&start=1692202412889997000 Total Streams: 6 Unique Labels: 2 ``` Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Arve Knudsen	a60e0167a8	Upgrade to latest dskit main (#10300 ) What this PR does / why we need it: Upgrade to latest dskit main branch, which mainly has changes to its `log` package, which requires corresponding fixing up in Loki.	2 years ago
Karsten Jeschkies	b1a948e28b	Record waiting time for queriers to receive requests. (#10253 ) What this PR does / why we need it: We want to investigate whether work stealing among queriers would be useful. So we are recording the wait time of queriers for new requests in their queue. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Danny Kopping	a30629e36e	Azure Blob Storage is no longer experimental (#10299 ) What this PR does / why we need it: Azure Blob Storage is being used in production by many organisations, including Grafana Labs. It should no longer be marked experimental. Which issue(s) this PR fixes: Fixes #10298 Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
akhilanarayanan	4013c1f8b5	Stopped String() from doubly escaping regex characters (#10277 ) What this PR does / why we need it: When queries have regex, characters that needed escaping were escaped twice. Now they're only singly escaped. This is good because it reduces the character count so users can submit queries closer to the 5120 byte limit. Which issue(s) this PR fixes: Sort of addresses #10153 Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Co-authored-by: Akhila Narayanan <akhilanarayanan@Akhilas-MacBook-Air.local>	2 years ago
Danny Kopping	beed298549	Dynamic client-side throttling to avoid object storage rate-limits (GCS only) (#10140 ) What this PR does / why we need it: Across the various cloud providers' object storage services, there are different rate-limits implemented. Rate-limits can be imposed under multiple conditions, such as server-side scale up (ramping up from low volume to high, "local" limit), reaching some defined upper limit ("absolute" limit), etc. We cannot know apriori when these rate-limits will be imposed, so we can't set up a client-side limiter to only allow a certain number of requests through per second. Additionally, that would require global coordination between queriers - which is difficult. With the above constraints, I have instead taken inspiration from TCP's [congestion control algorithms](https://en.wikipedia.org/wiki/TCP_congestion_control). This PR implements [AIMD](https://en.wikipedia.org/wiki/Additive_increase/multiplicative_decrease) (Additive Increase, Multiplicative Decrease), which is used in the congestion _avoidance_ phase of congestion control. The default window size (object store requests per second) is 2000; in other words, we skip the "slow start" phase. The controller uses the Go [`rate.Limiter`](https://pkg.go.dev/golang.org/x/time/rate), which implements the token-bucket algorithm. To put it simply: - every successful request widens the window (per second client rate-limit) - every rate-limited response reduces the window size by a backoff factor (0.5 by default, so it will halve) - when the limit has been reached, the querier will be delayed from making further requests until tokens are available	2 years ago
Mohamed-Amine Bouqsimi	176e9b7292	Add filter on rules' labels to API query (#9128 )	2 years ago
Danny Kopping	a18ca92375	Track effectiveness of hedged requests (#10281 ) What this PR does / why we need it: We currently have request hedging implemented in many object store clients. Right now we track how many hedged requests we make, and how many are rate-limited, but nothing actually tells us _how effective_ the hedged requests are. In other words, we don't know how many hedged requests were issued and _won_ (completed before the original).	2 years ago
Owen Diehl	9097f1ff42	Sharding optimizations (#10101 ) A few bugfixes and more sharding optimizations * fix bug on `<aggr> by\|without ()` groupings which removed the grouping while downstreaming * shardable implementations for `max+min`, operation specific merge strategies which enable many more types of sharded requests, even when label-reduction is performed at edge.	2 years ago
Trevor Whitney	4d2a94ebd3	Return intersecting labels when aggregating volume by label (#10248 ) This PR changes the behavior the `index/volume` endpoint to return intersecting labels when requesting volume aggregating by label. Previously it only returned labels that matched selectors in the query. The old behavior is still possible using the `targetLabels` argument.	2 years ago
Charles Korn	611481b6cf	Upgrade dskit (#10249 )	2 years ago
Salva Corts	287f29bc46	Update chunk and head fmt to v4 (non-indexed labels) (#10242 ) What this PR does / why we need it: This PR updates the default chunk and heads format to v4 to support writing and reading non-indexed labels. --------- Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	2 years ago
Charles Korn	8e6a5433ac	Upgrade dskit and remove dependency on weaveworks/common (#10220 ) What this PR does / why we need it: This PR upgrades `github.com/grafana/dskit` removes Loki's dependency on `github.com/weaveworks/common`. The changes in dskit, apart from the migration (https://github.com/grafana/dskit/pull/356), are: * https://github.com/grafana/dskit/pull/347 * https://github.com/grafana/dskit/pull/349, which required small changes in `pkg/canary/reader/reader.go` and `pkg/canary/writer/push.go` * https://github.com/grafana/dskit/pull/352 * https://github.com/grafana/dskit/pull/354 Which issue(s) this PR fixes: (none) Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Dylan Guedes	0e729e032c	Configuration: Allow customization of default runtime values (#10193 ) What this PR does / why we need it: Adds a new section to our runtime configuration that will allow customization of default runtime values. For instance, our default `log_push_request` is `false`. Previously, there was no way of setting it to `true`, so you'd have to set it for every existing tenant. Now you can by making usage of the new `default` section.	2 years ago
Karsten Jeschkies	a3741d1068	Include response type in merging error message and merge already merged views. (#10218 )	2 years ago
Vardhaman Surana	e3b7bcb67d	Ruler: add limit parameter in rulegroup (#10109 ) What this PR does / why we need it: This PR adds the support for group level limit parameter in ruler which was introduced in prometheus by PR: https://github.com/prometheus/prometheus/pull/9260 Which issue(s) this PR fixes: Fixes #4439	2 years ago
Ed Welch	8ca1d1e0a4	Loki: simplify the FetchChunks method and optimize the experimental l2 cache a little more (#10160 ) What this PR does / why we need it: This started as a simple PR to optimize the new L2 chunks cache code such that we don't look up chunks in L1 that we know won't be there, however, the complexity of the FetchChunks method taking both a slice of requested Chunks as well as a slice of requested Keys made this really challenging. Looking at how we use this method, I saw no reason for there to be a requirement to pass both a list of chunks and keys, we only ever want the Chunks and everywhere we used this we were generating the list of keys external to this function from the list of chunks being passed in. The second change was to the implementation of how we process the cache results to look for found vs missing keys. The current implementation required the chunks to be sorted by external key so that it could iterate through them, I tested out a different version of this which uses a map instead of sorting and iterating. The map performance was slightly better in terms of reduced allocations, not enough to make a meaningful difference, but I think this code is easier to understand and reason about, and it removes any requirement for sorting of the passed in Chunks by External key. Benchmarks: Previous: ``` 100 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 3445 334642 ns/op 478074 B/op 7536 allocs/op BenchmarkFetch-32 2947 344439 ns/op 478534 B/op 7537 allocs/op BenchmarkFetch-32 3489 339239 ns/op 478092 B/op 7536 allocs/op BenchmarkFetch-32 3523 340550 ns/op 478003 B/op 7536 allocs/op BenchmarkFetch-32 2935 346810 ns/op 478590 B/op 7537 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 23.948s 1000 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 363 3225519 ns/op 5305481 B/op 78133 allocs/op BenchmarkFetch-32 358 3308456 ns/op 5308299 B/op 78143 allocs/op BenchmarkFetch-32 336 3379067 ns/op 5322534 B/op 78191 allocs/op BenchmarkFetch-32 357 3330118 ns/op 5308854 B/op 78145 allocs/op BenchmarkFetch-32 349 3390415 ns/op 5313813 B/op 78162 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 25.607s 10,000 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 3 334471200 ns/op 639282104 B/op 2630192 allocs/op BenchmarkFetch-32 2 503434741 ns/op 922833480 B/op 3412973 allocs/op BenchmarkFetch-32 1 1064668769 ns/op 1771502096 B/op 5741173 allocs/op BenchmarkFetch-32 1 1027755910 ns/op 1771419128 B/op 5749049 allocs/op BenchmarkFetch-32 1 1027270578 ns/op 1774711872 B/op 5752726 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 23.251s ``` Now ``` 100 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 3856 300569 ns/op 462932 B/op 6214 allocs/op BenchmarkFetch-32 4016 305387 ns/op 462803 B/op 6214 allocs/op BenchmarkFetch-32 3382 303078 ns/op 463201 B/op 6215 allocs/op BenchmarkFetch-32 3319 303367 ns/op 463308 B/op 6215 allocs/op BenchmarkFetch-32 3289 356986 ns/op 463341 B/op 6215 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 22.879s 10,000 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 393 2641470 ns/op 5193282 B/op 65956 allocs/op BenchmarkFetch-32 433 2684287 ns/op 5174832 B/op 65894 allocs/op BenchmarkFetch-32 427 2778092 ns/op 5177334 B/op 65902 allocs/op BenchmarkFetch-32 418 2714370 ns/op 5181193 B/op 65915 allocs/op BenchmarkFetch-32 415 2822163 ns/op 5183436 B/op 65920 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 25.053s 10,000 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 3 362233430 ns/op 638097202 B/op 2510070 allocs/op BenchmarkFetch-32 2 611639304 ns/op 921128060 B/op 3292828 allocs/op BenchmarkFetch-32 1 1022180867 ns/op 1770465496 B/op 5621431 allocs/op BenchmarkFetch-32 1 1253448115 ns/op 1771278456 B/op 5630224 allocs/op BenchmarkFetch-32 1 1189564672 ns/op 1774053064 B/op 5628113 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 23.960s ``` Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Sandeep Sukhani	0d580e4501	serialize empty head block as well while checkpointing to retain the head block format (#10182 ) What this PR does / why we need it: When the head block is empty, we do not serialize it while checkpointing. This causes a mismatch in chunk and head block format which ideally should be on the same version. The problem happens when we rollback from v4 chunk and block format to v3 and the head is empty in checkpoint which then [gets set to the default version](`ce91076dd4/pkg/ingester/checkpoint.go (L106)`) i.e v3 while the checkpointed chunk would be v4. Checklist - [x] Tests updated --------- Co-authored-by: Salva Corts <salva.corts@grafana.com>	2 years ago
Charles Korn	bde65667f7	Upgrade dskit to use packages migrated from weaveworks/common (#10164 ) What this PR does / why we need it: This PR upgrades dskit and replaces use of packages from weaveworks/common with their migrated equivalents in dskit. See https://github.com/grafana/dskit/pull/342 for more details. Note that Loki still uses some packages from weaveworks/common that I haven't migrated (`aws` and `test`) - I'll migrate these separately. If this PR needs to be rebuilt, I used `rewrite.sh` ([source](https://gist.github.com/charleskorn/48efe62a09d6d70f3de30327003df5c5#file-rewrite-sh)) to generate most of these changes. Which issue(s) this PR fixes: (none) Special notes for your reviewer: (none) Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	2 years ago
Salva Corts	2754176239	Fixes for non-indexed metadata (#10167 ) What this PR does / why we need it: This PR fixes a bug while recovering from the WAL. Before this PR, when reading a head from the WAL, we'd attempt to create a chunk with the head fmt of the WAL head. If we change the chunk fmt to v4, we'd fail to rebound chunks since v4 may be incompatible with the head fmt from the WAL. Instead, in this PR we will always use the Default head fmt for the new chunk (which will change when we change the default chunk fmt to v4). Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Dylan Guedes	1221658fc5	Introduces cache to TSDB postings (#9621 ) What this PR does / why we need it: Introduces a new interface for fetching TSDB postings and implements a cached TSDB postings. The new postings interface is named `PostingsReader` and contains two implementations: a simple one, which is identical to the former Postings behavior, and a new one, that caches postings using the existing `index_queries_cache_config`. By default, the simple/former implementation is used. The new cached implementation can be configured in the following way: ```yaml storage_config: tsdb_shipper: cache_postings: true ``` In the future, we'll remove the `cache_postings` flag and change the behavior to cache postings by default.	2 years ago
Mohamed-Amine Bouqsimi	2fcde18207	Fix query_timeout warning messages (#10083 ) What this PR does / why we need it: This PR fixes the warning messages that are logged in relation to the timeouts migration. The messages reference the per-tenant config, but the config in question in the `AdjustForTimeoutsMigration` function is the global `limits_config`. This is a source of confusion for users. Which issue(s) this PR fixes: Fixes #9801	2 years ago
Salva Corts	48660d5cd5	Fix bugs in non-indexed-labels (#10142 ) What this PR does / why we need it: This PR fixes some bugs discovered while setting the default chunk format yo _V4_ and the default head format to _unordered with non-indexed labels_. Note that this PR doesn't change the defaults, but fixes some bugs that would prevent us from changing them in the future. The bugs it fixes are: - Index out-of-range panic when symbols' labels are empty and we do a Lookup. Test added. - When applying retention, new chunks wouldn't contain non-indexed labels (caused by https://github.com/grafana/loki/pull/10090). Test added. - New chunks would fail to serialize non-indexed labels when applying retention because the head format would be the [dummy one (ordered)][1] [(used here via `c.headFmt`)][2]. Cannot test unless we change the default chunk/head format but tested manually. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) [1]: `457f2e6ec0/pkg/chunkenc/memchunk.go (L375)` [2]: `24fd566833/pkg/chunkenc/memchunk.go (L1088-L1095)`	2 years ago
Dylan Guedes	9130c1a40d	Include stream name on error message (#10131 ) What this PR does / why we need it: Modify the "errTooFarBehind" error to include the stream name if it will be emitted by the write failures manager. This will make the behavior for the write failures more consistent without breaking the existing behavior.	2 years ago
Karsten Jeschkies	fbcaa1d5d8	Lazily decode series protobuf. (#10071 ) What this PR does / why we need it: The protobuf decoding for series responses runs into a memory issue for many series. The response is only merged and passed through the front end. It is more efficient to decode the protobuf encoded series lazily. And `decode`, `merge`, `encode` benchmark shows the benefit of transcoding the protobuf series message into a JSON response directly. This change will not impact production code since it only applies to protobuf encoding messaging between the querier and query frontend that must be explicitly enabled. ``` › go test -v -run=^$ -bench "Benchmark_DecodeMergeEncodeCycle" -memprofile memory_base.prof -count=10 ./pkg/querier/queryrange > before.txt › go test -v -run=^$ -bench "Benchmark_DecodeMergeEncodeCycle" -memprofile memory_base.prof -count=10 ./pkg/querier/queryrange > before.txt › benchstat before.txt after.txt before.txt:5: missing iteration count after.txt:5: missing iteration count goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/querier/queryrange cpu: AMD Ryzen 7 3700X 8-Core Processor │ before.txt │ after.txt │ │ sec/op │ sec/op vs base │ _DecodeMergeEncodeCycle-16 2537.7m ± 2% 934.2m ± 1% -63.19% (p=0.000 n=10) │ before.txt │ after.txt │ │ B/op │ B/op vs base │ _DecodeMergeEncodeCycle-16 1723.4Mi ± 0% 641.1Mi ± 0% -62.80% (p=0.000 n=10) │ before.txt │ after.txt │ │ allocs/op │ allocs/op vs base │ _DecodeMergeEncodeCycle-16 20240.6k ± 0% 203.0k ± 0% -99.00% (p=0.000 n=10) ``` Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	2 years ago
Arve Knudsen	7014e5ce7f	log: Use dskit/log.BufferedLogger instead of LineBufferedLogger (#10005 ) What this PR does / why we need it: `pkg/util/log.LineBufferedLogger` was ported to `dskit/log.BufferedLogger` in https://github.com/grafana/dskit/pull/338, so this utility can be shared among projects. This PR modifies Loki to use the dskit implementation.	2 years ago
Eng Zer Jun	e455a11b49	Ruler: compare header with `strings.EqualFold` (#10103 )	2 years ago
Travis Patterson	77b04b7963	Logging improvements for volume requests (#10099 ) I noticed we were logging volume requests as metric requests in `metrics.go`. This PR logs volume requests on their own with relevant data.	2 years ago
Ed Welch	5c29b7439f	Loki: change the prefix for the l2 chunks cache (#10097 ) What this PR does / why we need it: Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	e97e53f1b3	Loki: add an expirmental l2 chunks cache (#10061 ) What this PR does / why we need it: As we play around with our caching strategies, there has surfaced a potential benefit to having 2 layers of chunks cache with a `duration from now` handoff. i.e. chunks older than _x_ days get sent to an L2 chunks cache. Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Danny Kopping	8fa4134ef9	Update dskit (#10091 )	3 years ago
Salva Corts	24fd566833	Do not send back non-indexed labels for each entry in query response (#10090 ) What this PR does / why we need it: Since the non-indexed labels are part of the returned labels results, we don't need to send them also along with each returned entry. This PR will: - Reduce the size of the response payload - Solve a problem in Grafana only showing the last log line Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
Salva Corts	cbb272da00	Add `_extracted` to non-indexed labels with same name as stream labels (#10082 ) What this PR does / why we need it: In https://github.com/grafana/loki/pull/9702 we added support for returning non-indexed labels in the labels results. The problem is that non-indexed labels may overwrite stream labels if both are named the same way. This PR fixes this by adding an `_extracted` suffix if the non-indexed label is already present in the stream labels. Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago
Trevor Whitney	39be4b5e6b	Only aggregate labels or series based on requested (#10076 ) This improves the logic when calculating volumes by only aggregating series or labels based on which was requested. Previously we aggregated both, and then picked the correct set for the response, causing unnecessary additional memory consumption.	3 years ago
Mohamed-Amine Bouqsimi	5a7d6789b2	Fix missing alibabacloud ruler storage type (#10026 ) What this PR does / why we need it: This PR fixes the missing `alibabacloud` storage type for the ruler. Which issue(s) this PR fixes: Fixes #9892 Special notes for your reviewer: @chaudum Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) --------- Co-authored-by: Salva Corts <salva.corts@grafana.com>	3 years ago
Francois Gouteroux	2880b5055a	fix: parse expression error message based on record or alert (#10065 )	3 years ago
Trevor Whitney	1584443216	logcli: fix volume routes, add targetLabels and aggregateByLabels (#10059 ) This PR fixes `logcli` to use the new endpoints for volume, which are now `volume` and `volume_range` instead of `series_volume` and `series_volume_range`. This PR also adds support for the additional parameters `targetLabels` and `aggregateBy`.	3 years ago
Sampras Lopes	30c84cdadb	fix(compactor): fix Content-Type for `GET /loki/api/v1/delete` api (#10018 ) What this PR does / why we need it: Setting the content type for `GET /loki/api/v1/delete` to application/json since we are returning json objects Which issue(s) this PR fixes: Does not have an active issue Special notes for your reviewer: Checklist - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	3 years ago
Sandeep Sukhani	9b554bba8d	implementing string interning to optimize resource usage for storing and processing non-indexed labels (#10044 ) What this PR does / why we need it: In PR #9700, we added support for storing non-indexed labels in chunks. This PR optimizes resource usage for storing and processing non-indexed labels by doing string interning. We will store deduped label names and values as a list in chunks in the newly added non-indexed labels section. The labels would then be referenced in blocks by their index(called symbols). Additionally, I have started the convention of writing lengths of sections with their offsets within chunks, making it easier to introduce new sections. The section offsets and lengths would be stored at the end of the chunk, similar to [TOC](https://ganeshvernekar.com/blog/prometheus-tsdb-persistent-block-and-its-index/#a-toc) in TSDB. Checklist - [x] Tests updated	3 years ago
Dylan Guedes	122313caf7	Remove redundant spans (#10055 ) What this PR does / why we need it: - Stop using `instrument.CollectedRequest` on all Index calls since it emits spans that are redundant with existing ones and aren't improving Loki's monitoring while making it more expensive to run - Add events to `QuerySample` to improve its monitoring so we can better track what is holding down its performance - Remove redundant `Ingester.GetChunkRef` span	3 years ago
Ed Welch	23a189990c	Loki: Tweak how we handle X-Query-Tags header (#10058 ) What this PR does / why we need it: Adjust the regex for X-Query-Tags to allow @ and . so that email addresses can be displayed properly. Also changed the ToLower to only apply to the key not the value Which issue(s) this PR fixes: Fixes #<issue number> Special notes for your reviewer: Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`) Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Travis Patterson	bfd196aa66	sort sort log volumes by size (#10045 ) To save some frontend work, this PR sorts all the log volumes by volume rather than name. For range queries, only the first value is used.	3 years ago
Trevor Whitney	8aaf5c10a8	add stats, volume, and volume_range commands to logcli (#9966 ) Add `stats`, `volume`, and `volume_range` commands to `logcli`. Does not implement the file client for now. I think it would be cool if in the future the file client could read from a downloaded TSDB index file for these commands.	3 years ago
Salva Corts	a0e9f1aeb5	Make naming consistent for non-Indexed labels (#10024 ) What this PR does / why we need it: We started using _metadata_ to refer to _non-indexed labels_. This PR replaces all the _metadata_ usages and renames them to _non-indexed labels_. Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago

1 2 3 4 5 ...

2242 Commits (ca19e8d7941bfac86355b2853ef2ea110f32075e)