634 Commits (9b7a634e92f8b2dd0d4d8f06f9bf985b20c35b7f)

Author	SHA1	Message	Date
Grot (@grafanabot)	9b7a634e92	[release-2.9.x] Return intersecting labels when aggregating volume by label (#10329) ... Backport 4d2a94ebd3 from #10248 --- This PR changes the behavior the `index/volume` endpoint to return intersecting labels when requesting volume aggregating by label. Previously it only returned labels that matched selectors in the query. The old behavior is still possible using the `targetLabels` argument. Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	2 years ago
Danny Kopping	1761625971	Track effectiveness of hedged requests (#10281) ... **What this PR does / why we need it**: We currently have request hedging implemented in many object store clients. Right now we track how many hedged requests we make, and how many are rate-limited, but nothing actually tells us _how effective_ the hedged requests are. In other words, we don't know how many hedged requests were issued and _won_ (completed before the original).	2 years ago
Grot (@grafanabot)	191c252d7c	[k163] Update chunk and head fmt to v4 (non-indexed labels) (#10245) ... Backport 287f29bc46 from #10242 --- **What this PR does / why we need it**: This PR updates the default chunk and heads format to v4 to support writing and reading non-indexed labels. Co-authored-by: Salva Corts <salva.corts@grafana.com>	2 years ago
Ed Welch	8ca1d1e0a4	Loki: simplify the FetchChunks method and optimize the experimental l2 cache a little more (#10160) ... **What this PR does / why we need it**: This started as a simple PR to optimize the new L2 chunks cache code such that we don't look up chunks in L1 that we know won't be there, however, the complexity of the FetchChunks method taking both a slice of requested Chunks as well as a slice of requested Keys made this really challenging. Looking at how we use this method, I saw no reason for there to be a requirement to pass both a list of chunks and keys, we only ever want the Chunks and everywhere we used this we were generating the list of keys external to this function from the list of chunks being passed in. The second change was to the implementation of how we process the cache results to look for found vs missing keys. The current implementation required the chunks to be sorted by external key so that it could iterate through them, I tested out a different version of this which uses a map instead of sorting and iterating. The map performance was slightly better in terms of reduced allocations, not enough to make a meaningful difference, but I think this code is easier to understand and reason about, and it removes any requirement for sorting of the passed in Chunks by External key. Benchmarks: Previous: ``` 100 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 3445 334642 ns/op 478074 B/op 7536 allocs/op BenchmarkFetch-32 2947 344439 ns/op 478534 B/op 7537 allocs/op BenchmarkFetch-32 3489 339239 ns/op 478092 B/op 7536 allocs/op BenchmarkFetch-32 3523 340550 ns/op 478003 B/op 7536 allocs/op BenchmarkFetch-32 2935 346810 ns/op 478590 B/op 7537 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 23.948s 1000 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 363 3225519 ns/op 5305481 B/op 78133 allocs/op BenchmarkFetch-32 358 3308456 ns/op 5308299 B/op 78143 allocs/op BenchmarkFetch-32 336 3379067 ns/op 5322534 B/op 78191 allocs/op BenchmarkFetch-32 357 3330118 ns/op 5308854 B/op 78145 allocs/op BenchmarkFetch-32 349 3390415 ns/op 5313813 B/op 78162 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 25.607s 10,000 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 3 334471200 ns/op 639282104 B/op 2630192 allocs/op BenchmarkFetch-32 2 503434741 ns/op 922833480 B/op 3412973 allocs/op BenchmarkFetch-32 1 1064668769 ns/op 1771502096 B/op 5741173 allocs/op BenchmarkFetch-32 1 1027755910 ns/op 1771419128 B/op 5749049 allocs/op BenchmarkFetch-32 1 1027270578 ns/op 1774711872 B/op 5752726 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 23.251s ``` Now ``` 100 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 3856 300569 ns/op 462932 B/op 6214 allocs/op BenchmarkFetch-32 4016 305387 ns/op 462803 B/op 6214 allocs/op BenchmarkFetch-32 3382 303078 ns/op 463201 B/op 6215 allocs/op BenchmarkFetch-32 3319 303367 ns/op 463308 B/op 6215 allocs/op BenchmarkFetch-32 3289 356986 ns/op 463341 B/op 6215 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 22.879s 10,000 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 393 2641470 ns/op 5193282 B/op 65956 allocs/op BenchmarkFetch-32 433 2684287 ns/op 5174832 B/op 65894 allocs/op BenchmarkFetch-32 427 2778092 ns/op 5177334 B/op 65902 allocs/op BenchmarkFetch-32 418 2714370 ns/op 5181193 B/op 65915 allocs/op BenchmarkFetch-32 415 2822163 ns/op 5183436 B/op 65920 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 25.053s 10,000 chunks ❯ go test -bench=Fetch -count=5 goos: linuxbench=Fetch -count=5 ─╯ goarch: amd64 pkg: github.com/grafana/loki/pkg/storage/chunk/fetcher cpu: AMD Ryzen 9 5950X 16-Core Processor BenchmarkFetch-32 3 362233430 ns/op 638097202 B/op 2510070 allocs/op BenchmarkFetch-32 2 611639304 ns/op 921128060 B/op 3292828 allocs/op BenchmarkFetch-32 1 1022180867 ns/op 1770465496 B/op 5621431 allocs/op BenchmarkFetch-32 1 1253448115 ns/op 1771278456 B/op 5630224 allocs/op BenchmarkFetch-32 1 1189564672 ns/op 1774053064 B/op 5628113 allocs/op PASS ok github.com/grafana/loki/pkg/storage/chunk/fetcher 23.960s ``` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Charles Korn	bde65667f7	Upgrade dskit to use packages migrated from weaveworks/common (#10164) ... **What this PR does / why we need it**: This PR upgrades dskit and replaces use of packages from weaveworks/common with their migrated equivalents in dskit. See https://github.com/grafana/dskit/pull/342 for more details. Note that Loki still uses some packages from weaveworks/common that I haven't migrated (`aws` and `test`) - I'll migrate these separately. If this PR needs to be rebuilt, I used `rewrite.sh` ([source](https://gist.github.com/charleskorn/48efe62a09d6d70f3de30327003df5c5#file-rewrite-sh)) to generate most of these changes. **Which issue(s) this PR fixes**: (none) **Special notes for your reviewer**: (none) **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	2 years ago
Dylan Guedes	1221658fc5	Introduces cache to TSDB postings (#9621) ... **What this PR does / why we need it**: Introduces a new interface for fetching TSDB postings and implements a cached TSDB postings. The new postings interface is named `PostingsReader` and contains two implementations: a simple one, which is identical to the former Postings behavior, and a new one, that caches postings using the existing `index_queries_cache_config`. By default, the simple/former implementation is used. The new cached implementation can be configured in the following way: ```yaml storage_config: tsdb_shipper: cache_postings: true ``` In the future, we'll remove the `cache_postings` flag and change the behavior to cache postings by default.	2 years ago
Salva Corts	48660d5cd5	Fix bugs in non-indexed-labels (#10142) ... **What this PR does / why we need it**: This PR fixes some bugs discovered while setting the default chunk format yo _V4_ and the default head format to _unordered with non-indexed labels_. Note that this PR doesn't change the defaults, but fixes some bugs that would prevent us from changing them in the future. The bugs it fixes are: - Index out-of-range panic when symbols' labels are empty and we do a Lookup. **Test added**. - When applying retention, new chunks wouldn't contain non-indexed labels (caused by https://github.com/grafana/loki/pull/10090). **Test added**. - New chunks would fail to serialize non-indexed labels when applying retention because the head format would be the [dummy one (ordered)][1] [(used here via `c.headFmt`)][2]. **Cannot test unless we change the default chunk/head format** but tested manually. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) [1]: 457f2e6ec0/pkg/chunkenc/memchunk.go (L375) [2]: 24fd566833/pkg/chunkenc/memchunk.go (L1088-L1095)	2 years ago
Ed Welch	5c29b7439f	Loki: change the prefix for the l2 chunks cache (#10097) ... **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	e97e53f1b3	Loki: add an expirmental l2 chunks cache (#10061) ... **What this PR does / why we need it**: As we play around with our caching strategies, there has surfaced a potential benefit to having 2 layers of chunks cache with a `duration from now` handoff. i.e. chunks older than _x_ days get sent to an L2 chunks cache. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Danny Kopping	8fa4134ef9	Update dskit (#10091)	2 years ago
Trevor Whitney	39be4b5e6b	Only aggregate labels or series based on requested (#10076) ... This improves the logic when calculating volumes by only aggregating series or labels based on which was requested. Previously we aggregated both, and then picked the correct set for the response, causing unnecessary additional memory consumption.	2 years ago
Sampras Lopes	30c84cdadb	fix(compactor): fix Content-Type for `GET /loki/api/v1/delete` api (#10018) ... **What this PR does / why we need it**: Setting the content type for `GET /loki/api/v1/delete` to application/json since we are returning json objects **Which issue(s) this PR fixes**: Does not have an active issue **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Dylan Guedes	122313caf7	Remove redundant spans (#10055) ... **What this PR does / why we need it**: - Stop using `instrument.CollectedRequest` on all Index calls since it emits spans that are redundant with existing ones and aren't improving Loki's monitoring while making it more expensive to run - Add events to `QuerySample` to improve its monitoring so we can better track what is holding down its performance - Remove redundant `Ingester.GetChunkRef` span	2 years ago
Trevor Whitney	a12311c5c8	Allow volume to be aggregated by label (#9988) ... Add a `aggregateBy` query parameter to the `series_volume` endpoint, and rename that endpoint to just `volume`. This allows users to get volumes aggregated into top level labels, rather label+value pairs. --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago
Salva Corts	ce91076dd4	Add non-indexed metadata to chunks (#9700) ... **What this PR does / why we need it**: In https://github.com/grafana/loki/pull/9694 we support adding metadata labels for each entry in the push payload. In this PR we take that metadata and add it to the entries in the chunk. Supporting serialization and deserialization of those metadata labels. --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Vladyslav Diachenko	be1a9c4ec5	Revert "updated to the latest version of dskit" (#9953) ... Reverts grafana/loki#9920	2 years ago
Trevor Whitney	9e22a89bcb	Call FetchChunks in parallel from series index store (#9935) ... This PR changes the invocation of `FetchChunks` in the series index store to happen in parallel. It seems like this was the original intention due to the `// bound concurrency` comment when allocating the `groups` slice. This came out of an investigation of some slow queries, and the traces showed many serial calls to `FetchChunks`.	2 years ago
Christian Haudum	db38b7bccc	Remove unused fallback in index gateway client store (#9916) ... The fallback was once used by boltdb-shipper, but since boltdb-shipper changed to not use the index gateway, the fallback was also never used. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Vladyslav Diachenko	3243676f4c	updated to the latest version of dskit (#9920) ... **What this PR does / why we need it**: Updated Loki with the latest version of dskit **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Trevor Whitney	9e19ff006c	Add targetLabels to SeriesVolume requests (#9878) ... Adds optional `targetLabels` parameter to `series_volume` and `series_volume_range` requests that controls how volumes are aggregated. When provided, volumes are aggregated into the intersections of the provided `targetLabels` only.	2 years ago
Travis Patterson	cb2b3ca9ea	Add limit to the series volume endpoint (#9833) ... To protect index gateways, this PR adds a per-tenant limit to the `series_volume` endpoint so the accumulation of series can't grow without bound.	2 years ago
Dylan Guedes	bc0069756e	Queriers: Stop emitting spans for every objectstorage call (#9857) ... **What this PR does / why we need it**: Stop emitting spans for every call to our object storages (like S3 or Azure). They are causing every chunk fetching to emit a span. Request metrics will still be reported just fine. **Which issue(s) this PR fixes**: N/A	2 years ago
Sandeep Sukhani	7cbfdb5d6d	cache: fix a mistake in returning cached error in the index list cache (#9853) ... **What this PR does / why we need it**: I did a mistake in returning a cached error in the code pushed in PR #9798. Fixing it in the PR with some tests. **Checklist** - [x] Tests updated	2 years ago
Sandeep Sukhani	e089b4b60d	address feedback on pr 9798 (#9842) ... **What this PR does / why we need it**: PR #9798 got merged by mistake without me addressing the feedback. There is some critical feedback which needs to be addressed, particularly https://github.com/grafana/loki/pull/9798#discussion_r1243526743.	2 years ago
Sandeep Sukhani	2ed3c15c22	index: allow refreshing index file names cache for specific tables (#9798) ... **What this PR does / why we need it**: We have a cache for index files available in the object storage, which helps reduce the number of list calls we make to the object storage. Currently, we only support refreshing the whole index cache, which includes all the files for all tables and users. However, in a large cell with many tables and tenants, it is less likely that all the tables are being referenced. Also, most iterative operations would only need a fresh listing of just a few tables at a time so it is wasteful to always refresh the whole index cache. This PR adds support for refreshing the index cache for only specific tables. I am doing it as part of some compactor improvements to run compaction and retention in separate goroutines to avoid blocking compaction when some expensive delete requests are being processed. To make those changes, I needed a way to cost-effectively refresh the index files in the table being picked up for processing. **Checklist** - [x] Tests updated	2 years ago
Trevor Whitney	d4cfbaac4f	Implement series volume range queries (#9812) ... This PR adds a `series_volume_range` endpoint which allows for series volume queries over time with a specified step, returning timeseries data in the form of a Prometheus matrix response. The existing `series_volume` endpoint still returns Prometheus vector responses, and hardcodes the step to 0.	2 years ago
Christian Haudum	a65c99d9bf	Add per-tenant request counter metric to index gateway server (#9797) ... This commit add a counter metric `loki_index_gateway_requests_total` with labels `operation`, `tenant`, `status` for gRPC requests that are served by the index gateway. **What for?** The per-tenant RPS on the index gateway is used to derive the per-tenant shard factor. **Why tracking on the server?** Unlike tracking index gateway RPS on the client side, tracking on the server side does not yield that many series, even in multi-tenant installations with a lot of tenants, because the amount of index gateway instances is relatively small compared to the amount of queriers and frontends. **Special notes for your reviewer**: The previous approach of tracking requests on the client https://github.com/grafana/loki/pull/9781 has been abandoned. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Travis Patterson	8ca035ffbf	Log Volume: Do the prometheus-format conversion in dedicated middleware (#9776) ... This PR moves the Prometheus-format conversion to it's own middleware. This keeps it all in one place and consolidates logic that was in many places. It also sets us up to make range queries as a next step.	2 years ago
Sandeep Sukhani	3e1f2fc273	caching: do not try to fill the gap in log results cache when the new query interval does not overlap the cached query interval (#9757) ... **What this PR does / why we need it**: Currently, when we find a relevant cached negative response for a logs query, we do the following: * If the cached query completely covers the new query: * return back an empty response. * else: * fill the gaps on either/both sides of the cached query. The problem with filling the gaps is that when the cached query does not overlap at all with the new query, we have to extend the query beyond what the query requests for. However, with the logs query, we have a limit on the number of lines we can send back in the response. So, this could result in the query response having logs which were not requested by the query, which then get filtered out by the [response extractor](b78d3f0552/pkg/querier/queryrange/log_result_cache.go (L299)), unexpectedly resulting in an empty response. For example, if the query was cached for start=15, end=20 and we get a `backwards` query for start=5, end=10. To fill the gap, the query would be executed for start=5, end=15. Now, if we have logs more than the query `limit` in the range 10-15, we would filter out all the data in the response extractor and send back an empty response to the user. This PR fixes the issue by doing the following changes when handling cache hit: * If the cached query completely covers the new query: * return back an empty response[_existing_]. * else if the cached query does not overlap with the new query: * do the new query as requested. * If the new query results in an empty response and has a higher interval than the cached query: * update the cache * else: * query the data for missing intervals on both/either side[_existing_] * update the cache with extended intervals if the new queries resulted in an empty response[_existing_] **Special notes for your reviewer**: We could do further improvements in the handling of queries not overlapping with cached query by selectively extending the queries based on query direction and cached query lying before/after the new query. For example, if the new query is doing `backwards` query and the `cachedQuery.End` < `newQuery.Start`, it should be okay to extend the query and do `cachedQuery.End` to `newQuery.End` to fill the cache since query would first fill the most relevant data before hitting the limits. I did not want to complicate the fix so went without implementing this approach. We can revisit later if we feel we need to improve our caching. **Checklist** - [x] Tests updated - [x] `CHANGELOG.md` updated --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago
Ashwanth	10e55a6359	store: fix incorrect usage of period config in table range (#9754) ... **What this PR does / why we need it**: We are incorrectly taking reference to a loop variable in store init, I overlooked this part as the reference was being taken in the call to `GetIndexTableNumberRange` As a result of this, `TableRange` of each period (which is used by index-shipper) points to the latest `periodConfig`. This in itself is not a problem if the index prefix doesn't change as we use the `periodConfig` only to generate the `tableName`. But if the index prefix changes across periods, we end up losing access to the indexes using the old prefix. **Special notes for your reviewer**: **Checklist** - [X] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com>	2 years ago
Travis Patterson	4da0f63789	Remove unused Value field (#9774) ... We didn't end up needing the `Value` field because we can express everything we need to as selectors	2 years ago
Trevor Whitney	dbc3040739	Convert SeriesVolume response to prometheus-style (#9703) ... Changes the response type of the label volume stats endpoint to return volumes as prometheus-style timeseries metrics. It currently only supports instant queries, but is a necessary step to eventually supporting range queries.	2 years ago
Trevor Whitney	d74c67a9d0	Add version to TSDB index builder struct (#9566) ... Add version as a property to the TSDB Index Builder, forcing users of the builder to specify a version for each builder instance. This is the first of many PRs that will add the TSDB index version to the schema version config.	2 years ago
Christian Haudum	ce81895241	Add shuffle sharding to index gateway (#9710) ... This PR introduces shuffle sharding of the index files across index gateways where the shuffle sharding key is the tenant ID. Instead of having a global replication factor on the index gateways which affects all tenants the same, whether they are small or large tenants, the shuffle sharding allows to specify a shard factor per tenant (on top of the replication factor). This a) reduces the amount of indexes that are downloaded on individual index gateways in case of a high replication factor (which is currently the de-factor standard to deal with high load) and b) allows to scale tenants individually. The shuffle sharding algorithm is part of dskit's ring implementation and is not part of this commit. If you have set a high RF for the index gateways, it is required to reduce the RF and instead set the shard factor for the tenants. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Dylan Guedes	24e9efb907	Loki: Report event for `GetChunkRef` instead of SpanLogging (#9748) ... **What this PR does / why we need it**: Modify our calls to `index_client.GetChunkRef` to report invocations through a span event instead of through span logging. This way we won't log a new log line for every GetChunkRef call. **Which issue(s) this PR fixes**: N/A	2 years ago
Dylan Guedes	c6e18a2efb	Loki: Humanize downloaded file size (bytes format) (#9746) ... **What this PR does / why we need it**: Humanize reported size of downloaded file. **Which issue(s) this PR fixes**: N/A	2 years ago
Bryan Boreham	695ea32226	stores/series tests: use abstractions over Labels (#9613) ... **What this PR does / why we need it**: Stop assuming `labels.Labels` is a slice. **Special notes for your reviewer:** This PR only touches tests. Upstream Prometheus is using a different implementation of `labels.Labels`, so I plan to create a series of PRs which make Loki compatible with that. #9532 was the first. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - NA For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Kaviraj Kanagaraj	4c0e01b93f	revert(9476): Bringback async store like before for the queriers (#9715) ... This seems to cause some issue with `querier`, not getting chunks from ingesters correctly via `async_store`. I narrowed down to this PR. But hard `revert` seems not possible anymore. So created this manually. https://github.com/grafana/loki/pull/9476 Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Travis Patterson	db97058a84	Series volume endpoint (#9704) ... This changes the `label_volume` endpoint to the `series_volume` endpoint. The new endpoint still returns volumes but now it does it for the requested streams defined by the selector names passed rather than individual labels. All relevant non-requested labels are aggregated into the returned results ex: Assume we have the following streams: ``` {cluster="prod", team="A", component="foo"} {cluster="prod", team="B", component="foo"} {cluster="dev", team="A", component="foo"} {cluster="dev", team="B", component="foo"} ``` - requesting `{cluster="prod"}` returns one result for all streams containing `{cluster="prod"}` - requesting `{cluster=~".+"}` returns two results for the streams containing `{cluster="prod"}` and `{cluster="dev"}` - requesting `{cluster=~".+", team=".+"}` returns four results for the streams containing: ``` {cluster="prod", team="A"} {cluster="prod", team="B"} {cluster="dev", team="A"} {cluster="dev", team="B"} ``` --------- Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	2 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601) ... **What this PR does / why we need it**: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	2 years ago
Travis Patterson	065bee7e72	Label Volume Endpoint (#9588) ... For a given set of matchers, returns the top N associated label/value pairs by volume. A query for `{cluster=prod}` will return ``` cluster=prod: size (total logs matching this matcher) . . . nth-label=nth-value ``` This is to service use cases where users want to understand where their log volume has come from by label without making multiple requests to the stats endpoint. Note: This PR is a monster but it's mostly plumbing. I've pointed out the most interesting bits that actually get the volumes from ingesters/indexs	2 years ago
Kaviraj Kanagaraj	57f9452231	Revert 9217 chaudum/tsdb chunkrefs pool (#9685) ... Revert #9217 (potential bug in query result) --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	2 years ago
Salva Corts	c6fbff26e1	Add config to avoid caching stats for recent data (#9537) ... **What this PR does / why we need it**: When we query the stats for recent data, we query both the ingesters and the index gateways for the stats. ebdb2b1800/pkg/storage/async_store.go (L112-L114) ebdb2b1800/pkg/storage/async_store.go (L126-L127) Then we merge all the responses, which means summing up all the stats ebdb2b1800/pkg/storage/async_store.go (L157-L158) ebdb2b1800/pkg/storage/stores/index/stats/stats.go (L23-L26) Because we have a replication factor of 3, this means that we will get the stats from the ingesters repeated up to 3 times, hence inflating the stats. In the stats cache, we store the stats for a given matcher set for the whole day, then we extract the stats from the cache by the factor of time from the request that is stored in the cache: 336283acad/pkg/querier/queryrange/index_stats_cache.go (L33) 336283acad/pkg/querier/queryrange/index_stats_cache.go (L40) Inflated stats for recent data will be cached, so subsequent stats extracted from the cache will be inflated regardless of the time. This PR adds a new per-tenant limit `max_stats_cache_freshness` to not cache requests with an end time that falls within Now minus this duration. Here's a scenario illustrating this. The graphs below show the bytes stats queried in the sharding middleware. We are running a log filter query that won't match any log, every 5 seconds with a length of 3h.  As can be seen, after enabling the stats cache and configuring`do_not_cache_request_within` to not cache stats for requests within 30m, the bytes stats used in the sharding middleware stopped increasing. In both cases the stats cache hit ration was 100%.  **Special notes for your reviewer**: - Blocked by https://github.com/grafana/loki/pull/9535 - Note that this PR doesn't fix the root issue of inflated stats form the ingesters, but rather buys us some time to work on that. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Ashwanth	98d1307c7d	config: ensure storage config defaults apply to named stores (#9650) ... **What this PR does / why we need it**: Since named store config does not register any flags, storage configs defined under it do not get the defaults. For example [aws_storage_config](https://grafana.com/docs/loki/latest/configuration/#aws_storage_config) sets the default `storage_class` to `STANDARD`, but the same doesn't get applied by default when using named stores. This PR ensures that named storage configs are always assigned default values when they are unmarshalled by implementing `yaml.Unmarshaler` interface	2 years ago
Danny Kopping	1db560fdc1	Adding background cache (en|de)queue counters (#9665) ... **What this PR does / why we need it**: The background writeback cache exposes gauge metric currently for the current queue size. Gauges can be useful, but they are susceptible to sample errors because they only represent the point in time as the time of the scrape. Exposing counters for the bytes (en|de)queued to/from the cache will be more useful because they can be aggregated. Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	2 years ago
Salva Corts	1694ad0f9b	Stats cache can be configured independently (#9535) ... **What this PR does / why we need it**: Before this PR, the index stats cache would use the same config as the query results cache. This was a limitation since: 1. We would not be able to point to a different cache for storing the index stats if needed. 2. We would not be able to add specific settings for this cache, without adding it to the results cache. In this PR, we refactor the index stats cache config to be independently configurable. Note that if it's not configured, it will try to use the results cache settings. **Which issue(s) this PR fixes**: This is needed for: - https://github.com/grafana/loki/pull/9537 - https://github.com/grafana/loki/pull/9536 **Special notes for your reviewer**: - This PR also refactors all the tripperwares in rountrip.go to reuse the same stats tripperware instead of each one creating their own. - Configuring a new cache in rountrip.go is a requirement for https://github.com/grafana/loki/pull/9536 so the stats summary can distinguish before the stats cache and the results cache. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Salva Corts	87a659a6db	Add span events for index stats and result cache (#9552) ... **What this PR does / why we need it**: This PR adds events to the traces to have some extra observability for how we compute the index stats. We also add some trace events to the results cache.      **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Danny Kopping	a248e3e970	Querier: configurable writeback queue bytes size (#9604) ... **What this PR does / why we need it**: The background writeback process is used to write chunks that have been fetched from the store to cache. Chunks have a target size, but not all chunks will necessarily be this size, so it is tricky to find the correct value for `writeback_buffer` which specifies how many chunks will be held in the queue at any given time. For example: setting `writeback_buffer=10000` (the default) with all chunks being the default target size (`chunk_target_size=1.5MiB`), in the worst case could result in 15GiB of memory being used for this buffer. Practically speaking, the queue never grows so deep, but if the cache Loki is writing to becomes overwhelmed this could happen. Setting a value too low would result in many chunks not being written back to cache, which defeats the purpose. This PR introduces a configurable _byte size_ limit to the writeback cache, which defaults to 1GB in size. With this setting `writeback_size_limit` in place, we get the best of both worlds: a deep queue which can hold many small chunks or a shallow queue which can hold a few large ones without using too much memory so as to overwhelm the querier. If either the `writeback_buffer` or `writeback_size_limit` reaches capacity, any new chunks which are attempted to be added to the queue will be dropped. Two new metrics are added by this PR: - `loki_cache_background_queue_bytes`: gauge, measures the size of the queue in bytes - `loki_cache_dropped_background_writes_bytes_total`: counter, measures the volume of bytes dropped	2 years ago
Travis Patterson	fd0efe0799	Regex line/label filters return original regex when `String` is called (#9584) ... To optimize regex line and label filters, Loki rewrites them to be non-greedy. This can cause extra characters to appear in the regex when `String()` is called. In addition to confusing logging, this can also cause internal panics when a previously-parsable query becomes too long. This PR stores the original regex in the label/line filter and uses that in calls to `String`. This PR also removes regex optimizations from label selectors. Making the optimization at parse time caused the original regex to be lost.	2 years ago
Christian Haudum	b4d0a2cd68	Force using a buffer from the ChunkRefsPool (#9217) ... **What this PR does / why we need it**: Re-using a buffer from the `ChunkRefsPool` avoids unnecessary allocations in TSDB's `GetChunkRefs` call. **Special notes for your reviewer**: Benchmarks ``` $ gotest -v -count=1 ./pkg/storage/stores/tsdb/ -test.run="^$" -test.bench=Bench -test.benchtime=10000x ``` ```console $ benchstat old.txt new.txt name old time/op new time/op delta TenantHeads/10-8 244µs ±58% 55µs ± 4% -77.49% (p=0.000 n=10+9) TenantHeads/100-8 1.75ms ± 9% 0.58ms ±26% -66.89% (p=0.000 n=9+10) TenantHeads/1000-8 16.5ms ± 6% 6.4ms ± 2% -60.89% (p=0.000 n=9+10) IndexClient_Stats-8 25.0µs ±10% 25.1µs ±13% ~ (p=0.863 n=9+9) TSDBIndex_GetChunkRefs-8 3.34ms ±12% 1.46ms ± 2% -56.41% (p=0.000 n=10+9) name old alloc/op new alloc/op delta IndexClient_Stats-8 2.37kB ±12% 2.37kB ±12% ~ (p=0.923 n=10+10) TSDBIndex_GetChunkRefs-8 6.40MB ± 0% 1.40MB ± 0% -78.08% (p=0.000 n=9+10) name old allocs/op new allocs/op delta IndexClient_Stats-8 59.0 ± 0% 59.0 ± 0% ~ (all equal) TSDBIndex_GetChunkRefs-8 42.0 ± 0% 27.0 ± 0% -35.71% (p=0.000 n=10+10) ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago