2217 Commits (0fc934e4ea3ed76b995fd488a5fec3fab5a82886)

Author	SHA1	Message	Date
Salva Corts	2754176239	Fixes for non-indexed metadata (#10167) ... **What this PR does / why we need it**: This PR fixes a bug while recovering from the WAL. Before this PR, when reading a head from the WAL, we'd attempt to create a chunk with the head fmt of the WAL head. If we change the chunk fmt to v4, we'd fail to rebound chunks since v4 may be incompatible with the head fmt from the WAL. Instead, in this PR we will always use the Default head fmt for the new chunk (which will change when we change the default chunk fmt to v4). **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Dylan Guedes	1221658fc5	Introduces cache to TSDB postings (#9621) ... **What this PR does / why we need it**: Introduces a new interface for fetching TSDB postings and implements a cached TSDB postings. The new postings interface is named `PostingsReader` and contains two implementations: a simple one, which is identical to the former Postings behavior, and a new one, that caches postings using the existing `index_queries_cache_config`. By default, the simple/former implementation is used. The new cached implementation can be configured in the following way: ```yaml storage_config: tsdb_shipper: cache_postings: true ``` In the future, we'll remove the `cache_postings` flag and change the behavior to cache postings by default.	2 years ago
Mohamed-Amine Bouqsimi	2fcde18207	Fix query_timeout warning messages (#10083) ... **What this PR does / why we need it**: This PR fixes the warning messages that are logged in relation to the timeouts migration. The messages reference the per-tenant config, but the config in question in the `AdjustForTimeoutsMigration` function is the global `limits_config`. This is a source of confusion for users. **Which issue(s) this PR fixes**: Fixes #9801	2 years ago
Salva Corts	48660d5cd5	Fix bugs in non-indexed-labels (#10142) ... **What this PR does / why we need it**: This PR fixes some bugs discovered while setting the default chunk format yo _V4_ and the default head format to _unordered with non-indexed labels_. Note that this PR doesn't change the defaults, but fixes some bugs that would prevent us from changing them in the future. The bugs it fixes are: - Index out-of-range panic when symbols' labels are empty and we do a Lookup. **Test added**. - When applying retention, new chunks wouldn't contain non-indexed labels (caused by https://github.com/grafana/loki/pull/10090). **Test added**. - New chunks would fail to serialize non-indexed labels when applying retention because the head format would be the [dummy one (ordered)][1] [(used here via `c.headFmt`)][2]. **Cannot test unless we change the default chunk/head format** but tested manually. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) [1]: 457f2e6ec0/pkg/chunkenc/memchunk.go (L375) [2]: 24fd566833/pkg/chunkenc/memchunk.go (L1088-L1095)	2 years ago
Dylan Guedes	9130c1a40d	Include stream name on error message (#10131) ... **What this PR does / why we need it**: Modify the "errTooFarBehind" error to include the stream name if it will be emitted by the write failures manager. This will make the behavior for the write failures more consistent without breaking the existing behavior.	2 years ago
Karsten Jeschkies	fbcaa1d5d8	Lazily decode series protobuf. (#10071) ... **What this PR does / why we need it**: The protobuf decoding for series responses runs into a memory issue for many series. The response is only merged and passed through the front end. It is more efficient to decode the protobuf encoded series lazily. And `decode`, `merge`, `encode` benchmark shows the benefit of transcoding the protobuf series message into a JSON response directly. This change will not impact production code since it only applies to protobuf encoding messaging between the querier and query frontend that must be explicitly enabled. ``` › go test -v -run=^$ -bench "Benchmark_DecodeMergeEncodeCycle" -memprofile memory_base.prof -count=10 ./pkg/querier/queryrange > before.txt › go test -v -run=^$ -bench "Benchmark_DecodeMergeEncodeCycle" -memprofile memory_base.prof -count=10 ./pkg/querier/queryrange > before.txt › benchstat before.txt after.txt before.txt:5: missing iteration count after.txt:5: missing iteration count goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/querier/queryrange cpu: AMD Ryzen 7 3700X 8-Core Processor │ before.txt │ after.txt │ │ sec/op │ sec/op vs base │ _DecodeMergeEncodeCycle-16 2537.7m ± 2% 934.2m ± 1% -63.19% (p=0.000 n=10) │ before.txt │ after.txt │ │ B/op │ B/op vs base │ _DecodeMergeEncodeCycle-16 1723.4Mi ± 0% 641.1Mi ± 0% -62.80% (p=0.000 n=10) │ before.txt │ after.txt │ │ allocs/op │ allocs/op vs base │ _DecodeMergeEncodeCycle-16 20240.6k ± 0% 203.0k ± 0% -99.00% (p=0.000 n=10) ``` **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Arve Knudsen	7014e5ce7f	log: Use dskit/log.BufferedLogger instead of LineBufferedLogger (#10005) ... **What this PR does / why we need it**: `pkg/util/log.LineBufferedLogger` was ported to `dskit/log.BufferedLogger` in https://github.com/grafana/dskit/pull/338, so this utility can be shared among projects. This PR modifies Loki to use the dskit implementation.	2 years ago
Eng Zer Jun	e455a11b49	Ruler: compare header with `strings.EqualFold` (#10103)	2 years ago
Travis Patterson	77b04b7963	Logging improvements for volume requests (#10099) ... I noticed we were logging volume requests as metric requests in `metrics.go`. This PR logs volume requests on their own with relevant data.	2 years ago
Ed Welch	5c29b7439f	Loki: change the prefix for the l2 chunks cache (#10097) ... **What this PR does / why we need it**: **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Ed Welch	e97e53f1b3	Loki: add an expirmental l2 chunks cache (#10061) ... **What this PR does / why we need it**: As we play around with our caching strategies, there has surfaced a potential benefit to having 2 layers of chunks cache with a `duration from now` handoff. i.e. chunks older than _x_ days get sent to an L2 chunks cache. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Danny Kopping	8fa4134ef9	Update dskit (#10091)	2 years ago
Salva Corts	24fd566833	Do not send back non-indexed labels for each entry in query response (#10090) ... **What this PR does / why we need it**: Since the non-indexed labels are part of the returned labels results, we don't need to send them also along with each returned entry. This PR will: - Reduce the size of the response payload - Solve a problem in Grafana only showing the last log line **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Salva Corts	cbb272da00	Add `_extracted` to non-indexed labels with same name as stream labels (#10082) ... **What this PR does / why we need it**: In https://github.com/grafana/loki/pull/9702 we added support for returning non-indexed labels in the labels results. The problem is that non-indexed labels may overwrite stream labels if both are named the same way. This PR fixes this by adding an `_extracted` suffix if the non-indexed label is already present in the stream labels. **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Trevor Whitney	39be4b5e6b	Only aggregate labels or series based on requested (#10076) ... This improves the logic when calculating volumes by only aggregating series or labels based on which was requested. Previously we aggregated both, and then picked the correct set for the response, causing unnecessary additional memory consumption.	2 years ago
Mohamed-Amine Bouqsimi	5a7d6789b2	Fix missing alibabacloud ruler storage type (#10026) ... **What this PR does / why we need it**: This PR fixes the missing `alibabacloud` storage type for the ruler. **Which issue(s) this PR fixes**: Fixes #9892 **Special notes for your reviewer**: @chaudum **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [x] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Salva Corts <salva.corts@grafana.com>	2 years ago
Francois Gouteroux	2880b5055a	fix: parse expression error message based on record or alert (#10065)	2 years ago
Trevor Whitney	1584443216	logcli: fix volume routes, add targetLabels and aggregateByLabels (#10059) ... This PR fixes `logcli` to use the new endpoints for volume, which are now `volume` and `volume_range` instead of `series_volume` and `series_volume_range`. This PR also adds support for the additional parameters `targetLabels` and `aggregateBy`.	2 years ago
Sampras Lopes	30c84cdadb	fix(compactor): fix Content-Type for `GET /loki/api/v1/delete` api (#10018) ... **What this PR does / why we need it**: Setting the content type for `GET /loki/api/v1/delete` to application/json since we are returning json objects **Which issue(s) this PR fixes**: Does not have an active issue **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>	2 years ago
Sandeep Sukhani	9b554bba8d	implementing string interning to optimize resource usage for storing and processing non-indexed labels (#10044) ... **What this PR does / why we need it**: In PR #9700, we added support for storing non-indexed labels in chunks. This PR optimizes resource usage for storing and processing non-indexed labels by doing string interning. We will store deduped label names and values as a list in chunks in the newly added non-indexed labels section. The labels would then be referenced in blocks by their index(called symbols). Additionally, I have started the convention of writing lengths of sections with their offsets within chunks, making it easier to introduce new sections. The section offsets and lengths would be stored at the end of the chunk, similar to [TOC](https://ganeshvernekar.com/blog/prometheus-tsdb-persistent-block-and-its-index/#a-toc) in TSDB. **Checklist** - [x] Tests updated	2 years ago
Dylan Guedes	122313caf7	Remove redundant spans (#10055) ... **What this PR does / why we need it**: - Stop using `instrument.CollectedRequest` on all Index calls since it emits spans that are redundant with existing ones and aren't improving Loki's monitoring while making it more expensive to run - Add events to `QuerySample` to improve its monitoring so we can better track what is holding down its performance - Remove redundant `Ingester.GetChunkRef` span	2 years ago
Ed Welch	23a189990c	Loki: Tweak how we handle X-Query-Tags header (#10058) ... **What this PR does / why we need it**: Adjust the regex for X-Query-Tags to allow @ and . so that email addresses can be displayed properly. Also changed the ToLower to only apply to the key not the value **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Travis Patterson	bfd196aa66	sort sort log volumes by size (#10045) ... To save some frontend work, this PR sorts all the log volumes by volume rather than name. For range queries, only the first value is used.	2 years ago
Trevor Whitney	8aaf5c10a8	add stats, volume, and volume_range commands to logcli (#9966) ... Add `stats`, `volume`, and `volume_range` commands to `logcli`. Does not implement the file client for now. I think it would be cool if in the future the file client could read from a downloaded TSDB index file for these commands.	2 years ago
Salva Corts	a0e9f1aeb5	Make naming consistent for non-Indexed labels (#10024) ... **What this PR does / why we need it**: We started using _metadata_ to refer to _non-indexed labels_. This PR replaces all the _metadata_ usages and renames them to _non-indexed labels_. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Salva Corts	1d04cd5e31	Metadata to labels result and filtering support (#9702) ... **What this PR does / why we need it**: In #9700, we support encoding and decoding metadata for each entry into the chunks. This PR adds support for returning metadata labels for matching entries in a query to the returned LabelResults. It also supports filtering out logs by metadata labels. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	2 years ago
Christian Haudum	ab3d58a265	Add initial delay before starting flush loop (#9995) ... **What this PR does / why we need it**: Delay the first flush operation by up to 0.8x the flush time period. This will ensure that multiple ingesters started at the same time do not flush at the same time. Flushing at the same time can cause concurrently writing the same chunk to object storage, which in AWS S3 leads to being rate limited. Additionally, add a +/- 1% jitter to each flush check interval to prevent flushing at the same time for reasons mentioned above. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Trevor Whitney	a12311c5c8	Allow volume to be aggregated by label (#9988) ... Add a `aggregateBy` query parameter to the `series_volume` endpoint, and rename that endpoint to just `volume`. This allows users to get volumes aggregated into top level labels, rather label+value pairs. --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago
Dylan Guedes	d1be9d54ed	Change "tenant" to "org_id" (#10017) ... **What this PR does / why we need it**: Change our writefailures module to use "org_id" instead of "tenant" to be more consistent with the rest of our system.	2 years ago
Salva Corts	7e248e1e90	Support non-indexed labels in WAL (#10011) ... **What this PR does / why we need it**: In https://github.com/grafana/loki/pull/9700, we support encoding and decoding non-indexed labels for each entry into the chunks. This PR adds support for writing/reading the non-indexed labels to/from the WAL. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Salva Corts	3f161f5c1a	Improve observability for non-indexed labels usage (#9993) ... **What this PR does / why we need it**: In https://github.com/grafana/loki/pull/9700, we support encoding and decoding metadata for each entry into the chunks. In this PR we: - Update the bytes processed stats to account for the bytes from those non-indexed labels - Add new stats for bytes processed for those non-indexed labels - Add new ingestion metrics to track ingested non-indexed bytes **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Karsten Jeschkies	efac690328	Use series hash to identify uniques in merge. (#9985) ... **What this PR does / why we need it**: A run of the protobuf encoding for the querier showed a spike in memory usage in the `codec.MergeResponse` method. The routine was allocating a lot of memory for the `series.String()` method which is then hashed. Instead, we can use a `uint64` hash. ``` › go test -v -run=^$ -bench "Benchmark_MergeResponses$" -count=10 ./pkg/querier/queryrange ... │ before.txt │ after.txt │ │ sec/op │ sec/op vs base │ _MergeResponses-16 4.563 ± 1% 1.197 ± 1% -73.77% (p=0.000 n=10) │ before.txt │ after.txt │ │ B/op │ B/op vs base │ _MergeResponses-16 6578.715Mi ± 0% 2.362Mi ± 0% -99.96% (p=0.000 n=10) │ before.txt │ after.txt │ │ allocs/op │ allocs/op vs base │ _MergeResponses-16 40500.1k ± 0% 100.1k ± 0% -99.75% (p=0.000 n=10) ``` **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Trevor Whitney	ae4cd27d83	add sharded to querystats bytes processed (#9934) ... Add a sharded label to the `loki_logql_querystats_bytes_processed_per_second` metric that we use for determining our `LokiQueryRangeReadPerformanceBudgetBurn` metric so we can exclude slow queries with <= 1 shard from burning our SLO budget.	2 years ago
Ed Welch	fa6f9c638a	Loki: add 'post_filter_lines' stat for tracking how many lines are match a queries filter expression(s) (#9983) ... **What this PR does / why we need it**: To better understand query behavior and labeling strategies of users, logging the number of log lines "post filtering" can be very useful, we already log the total_lines processed in a query so this will allow us to see how many lines in the label selector matched what the query was looking for. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Salva Corts	78aad0b805	Fix buffered iterator (#9976) ... **What this PR does / why we need it**: In https://github.com/grafana/loki/pull/9700, we added support for writing non-indexed labels from entries into chunks. This PR introduced two bugs: - When the buffered iterator is closed, the buffer to read metadata labels is put back into the pool but not set to nil. Subsequent uses of the iterator may write on top of the buffer that was put back on the pool. This may lead to inconsistent/incorrect results. - Inside the buffered iterator, we were not correctly handling EOFs while reading the number of labels and each label length. This was due to the `lastAttemp` variable not being reset. This PR adds a new test for these two bugs and also fixes them. **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/pull/9700 **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)	2 years ago
Ed Welch	6d2b3e8b76	Loki: Fix a merge error when getting rules from sharded rulers using the `/prometheus/api/v1/rules` endpoint (#9973) ... **What this PR does / why we need it**: The `/prometheus/api/v1/rules` api endpoint will return all the rules currently loaded and being evaluated by the ruler. Previously when we merged the rule groups from all rulers in a sharded ruler setup, we only used the rule group name as the key in the map used for merging the results. This introduced a bug when multiple rule groups have the same name but in different namespaces, leading to some rules being reported in the incorrect namespace. This is just a visualization bug, the rules are in the correct space and operating as defined, however Grafana uses this API in conjunction with `/loki/api/v1/rules` to track the state of a rule being added or removed. The `/loki/api/v1/rules` endpoint returns rules as they exist in storage. When someone has multiple rule groups with the same name in different namespaces this can lead to the Grafana UI showing rules in a perpetual state of creating and deleting as it's possible for rules to be returned from `/prometheus/api/v1/rules` in the wrong group. Adding the namespace to the key used in the map for merging rule results prevents this issue. **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Signed-off-by: Edward Welch <edward.welch@grafana.com>	2 years ago
Salva Corts	ce91076dd4	Add non-indexed metadata to chunks (#9700) ... **What this PR does / why we need it**: In https://github.com/grafana/loki/pull/9694 we support adding metadata labels for each entry in the push payload. In this PR we take that metadata and add it to the entries in the chunk. Supporting serialization and deserialization of those metadata labels. --------- Signed-off-by: Vladyslav Diachenko <vlad.diachenko@grafana.com> Co-authored-by: Vladyslav Diachenko <vlad.diachenko@grafana.com>	2 years ago
Salva Corts	88c17ed7ab	Include entry metadata bytes in metrics tracking ingestion stats (#9808) ... **What this PR does / why we need it**: In https://github.com/grafana/loki/pull/9694 we added support for adding metadata labels to each entry in a push payload. In this PR we update the following metrics to take into account the bytes needed to store those labels: - `loki_distributor_bytes_received_total` - `distributor_bytes_received` **Which issue(s) this PR fixes**: Fixes #<issue number> **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	2 years ago
Salva Corts	15af77ba83	Remove Prometheus dependency for push pkg (#9937) ... **What this PR does / why we need it**: In https://github.com/grafana/loki/pull/9694, we modified the `push` pkg to import Prometheus as a dependency to use the `labels.Labels` type for the entries' non-indexed labels. Having Prometheus as a dependency is problematic since any project importing the `push` pkg will need to import Prometheus as a result. In fact, this is one of the reasons why the `push` pkg was extracted from Loki in the first place (https://github.com/grafana/loki/pull/8259). This PR removes the dependency of Prometheus from the `push` pkg by copying some bits of the implementation for Prometheus' `labels.Labels`. We copy: - The Labels struct definition - The JSON Marshaling and Unmarshaling methods for the labels. We need this so labels are encoded as maps instead of an array of objects. --- **Notes for reviewers:** - To implement the JSON Marshaling and Unmarshaling methods the `push` pkg now depends on `golang.org/x/exp`. I think it should be fine for projects importing the `push` pkg to also depend on `golang.org/x/exp`, right?	2 years ago
Vladyslav Diachenko	be1a9c4ec5	Revert "updated to the latest version of dskit" (#9953) ... Reverts grafana/loki#9920	2 years ago
Travis Patterson	23e59fa4a9	reset pipeline caches when tailing rather than making a new pipeline (#9949) ... Recent query optimizations made pipeline creation more expensive with the assumption that it occurred relatively infrequently. To avoid growing pipline caches without bound, integers recreate pipelines on every push. The combination of these things means that tailing high-volume streams can cause ingester failure. This fix adds the ability to clear pipeline caches so we don't need to create a new one each time.	2 years ago
Travis Patterson	6dc722bc9f	correctly handle unpack when reordering stages (#9936) ... The optimization to reorder stages doesn't take into account the `unpack` stage. When line filters come after unpack stages, they shouldn't be moved.	2 years ago
Trevor Whitney	9e22a89bcb	Call FetchChunks in parallel from series index store (#9935) ... This PR changes the invocation of `FetchChunks` in the series index store to happen in parallel. It seems like this was the original intention due to the `// bound concurrency` comment when allocating the `groups` slice. This came out of an investigation of some slow queries, and the traces showed many serial calls to `FetchChunks`.	2 years ago
Dylan Guedes	9ed7fa9df4	WriteFailures: Add metrics to increase monitoring (#9904) ... **What this PR does / why we need it**: Add metrics for our write failures manager. This way we can easily monitor how much logging we're doing and how much we're discarding. **Which issue(s) this PR fixes**: N/A	2 years ago
Karsten Jeschkies	a09cb07e98	Define protobufs for topk and cms. (#9933) ... **What this PR does / why we need it**: This change introduces Protobuf models for the sketch data structures which will be used to shard topk queries. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	2 years ago
Christian Haudum	db38b7bccc	Remove unused fallback in index gateway client store (#9916) ... The fallback was once used by boltdb-shipper, but since boltdb-shipper changed to not use the index gateway, the fallback was also never used. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago
Callum Styan	4d3b15ae89	Implement probabilistic topk via count min sketch and min heap (#9882) ... This PR currently implements the following structures: - **Count Min Sketch**: http://dimacs.rutgers.edu/~graham/pubs/papers/cmencyc.pdf - **sketch-bf**: https://ietresearch.onlinelibrary.wiley.com/doi/epdf/10.1049/ell2.12482 improvements to the heap usage for Count Min Sketch - HeavyKeeper: https://www.usenix.org/system/files/conference/atc18/atc18-gong.pdf I also read and evaluated the following: - topkapi: https://www.cs.rice.edu/~as143/Papers/topkapi.pdf - space saving algorithm: https://www.cse.ust.hk/~raywong/comp5331/References/EfficientComputationOfFrequentAndTop-kElementsInDataStreams.pdf **Description**: As part of our work to ensure every query type is parallelizable we started exploring probabilistic data structures as a way to reduce memory consumption and allow merging of partial results after sharding of queries. In this PR I have implemented a few sketch structures as part of investigating structures that are applicable to topk queries. Unfortunately while many papers made claims of improved performance over the original count min sketch paper, topkapi and heavy keeper either didn't behave as expected or were not mergeable while keeping the same degree of accuracy. Count min sketch when using conservative updates (which vastly reduces error over counting, especially for low flows), also can't keep the same degree of accuracy, but it is still more accurate after merging than heavy keeper. I've also applied some of the principles outlined in the sketchbf paper related to reducing the # of heap operations required when observing events into the sketch, which sped up processing in my benchmark of 100k unique streams each with 1-2k randomized # of occurrences by nearly 3x. --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	2 years ago
Karsten Jeschkies	c2c5249676	Remove goroutines from limiter middleware. (#9923) ... **What this PR does / why we need it**: The `limitedRoundTripper` would spawn thousands if not millions of goroutines. However, they were mostly idle waiting for downstream calls. This would congest the Go scheduler. A semaphore is more suitable to control concurrency as the [example](https://pkg.go.dev/golang.org/x/sync/semaphore#example-package-WorkerPool) in the documentation illustrates.	2 years ago
Salva Corts	aae13c376d	Add metadata to push payload (#9694) ... **What this PR does / why we need it**: We are adding support for attaching labels to each log line. This is one of the series of the PRs broken up to make it easier to review changes. This PR updates the push payload to send labels with each log entry optionally. The log labels are supposed to be in the same format as the stream labels. Just to put it out, here is how it would look for proto and json push payload with same data: **proto(`endpoint`: `(/loki/api/v1/push|/api/prom/push)`, `Content-Type`: `application/x-protobuf`)**(payload built using [push.Stream](4cd1246b88/pkg/push/types.go (L12))): ``` push.Stream{ Entries: []logproto.Entry{ { Timestamp: time.Unix(0, 1688515200000000000), Line: "log line", Labels: `{foo="bar"}`, }, }, Labels: `{app="test"}`, } ``` **v1(`endpoint`: `/loki/api/v1/push`, `Content-Type`: `application/json`)**: ```json { "streams": [{ "stream": { "app": "test" }, "values": [ ["1688515200000000000", "log line", { "foo": "bar" }] ] }] } ``` **legacy-json(`/api/prom/push`, `Content-Type`: `application/json`)**: ```json { "streams": [{ "labels": "{app=\"test\"}", "entries": [{ "ts": "2023-07-05T00:00:00.000000000Z", "line": "log line", "labels": "{foo=\"bar\"}" }] }] } ``` **Which issue(s) this PR fixes**: **Special notes for your reviewer**: We may need to add more thoughtful tests. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Co-authored-by: Sandeep Sukhani <sandeep.d.sukhani@gmail.com>	2 years ago
Christian Haudum	90fc5109dd	Remove unused package pkg/prom1 (#9927) ... Removes old, unused code. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	2 years ago