33 Commits (23b5fc2c9b1a77b8776eac70279018956a458fc6)

Author	SHA1	Message	Date
Trevor Whitney	7983f94b15	feat: detected field values (#14350)	8 months ago
Owen Diehl	bc78d13d9b	feat(blooms): compute chunks once (#12664) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Shantanu Alshi	587a6d20e9	feat: Detected labels from store (#12441)	1 year ago
Trevor Whitney	6c33809015	feat: split detected fields queries (#12491)	1 year ago
Shantanu Alshi	9994f56b53	fix: Fix multi_tenant querier for detected labels (#12447)	1 year ago
Trevor Whitney	a53a0cc792	feat: return real data in detected fields endpoint (#12421) ... parse log lines in the querier to return data about detected fields. this is not sustainable long term, but a short term solution to validate the data we want for the frontend.	1 year ago
Ed Welch	4c88be0ef2	chore: update loki modules for 3.0 release (#12433) ... Signed-off-by: Edward Welch <edward.welch@grafana.com>	1 year ago
Shantanu Alshi	5190dda64b	feat(detected_labels): Initial skeleton for the API (#12390) ... Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>	1 year ago
Owen Diehl	a36483b731	feat(blooms): bloom integration in query planning (#12208) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	1 year ago
Karsten Jeschkies	9aae25008d	Optimize series response format by using repeated fileds. (#11498) ... **What this PR does / why we need it**: The Protobuf map type is encodied as a repeated field of map entries. Decoding them to a slice is much faster than decoding them into a map. Since Loki is not using the fast key check for a map we can use the slice decoding. This change also allows us to decode the JSON directly into the right protobuf struct. This doulbes the JSON decoding speed and reduces the memory pressure by ~40%. ``` › go test -bench=. -run=^$ -count=10 ./pkg/querier/queryrange > before.log › go test -bench=. -run=^$ -count=10 ./pkg/querier/queryrange > after.log › benchstat before.log after.log goos: darwin goarch: arm64 pkg: github.com/grafana/loki/pkg/querier/queryrange │ before.log │ after.log │ │ sec/op │ sec/op vs base │ ResponseMerge/mergeStreams_unlimited-10 32.36m ± 0% 32.63m ± 2% ~ (p=0.393 n=10) ResponseMerge/mergeOrderedNonOverlappingStreams_unlimited-10 1.050m ± 1% 1.080m ± 3% +2.84% (p=0.005 n=10) ResponseMerge/mergeStreams_limited-10 33.02m ± 0% 32.60m ± 1% -1.29% (p=0.004 n=10) ResponseMerge/mergeOrderedNonOverlappingStreams_limited-10 15.11m ± 0% 15.07m ± 0% ~ (p=0.075 n=10) _CodecDecodeLogs-10 4.395m ± 1% 4.364m ± 0% -0.72% (p=0.005 n=10) _CodecDecodeSamples-10 16.97m ± 0% 16.84m ± 2% -0.77% (p=0.023 n=10) _CodecDecodeSeries/application/vnd.google.protobuf-10 745.8µ ± 8% 736.8µ ± 12% ~ (p=0.739 n=10) _CodecDecodeSeries/application/json;_charset=utf-8-10 15.37m ± 1% 10.60m ± 0% -31.03% (p=0.000 n=10) _MergeResponses-10 1186.9m ± 2% 149.8m ± 1% -87.38% (p=0.000 n=10) _UnwrapSeries-10 9.399m ± 1% 4.049m ± 0% -56.92% (p=0.000 n=10) _DecodeMergeEncodeCycle-10 666.0m ± 3% 194.6m ± 0% -70.79% (p=0.000 n=10) geomean 18.87m 12.51m -33.70% │ before.log │ after.log │ │ B/op │ B/op vs base │ _CodecDecodeLogs-10 3.649Mi ± 0% 3.649Mi ± 0% ~ (p=0.364 n=10) _CodecDecodeSamples-10 18.12Mi ± 0% 18.12Mi ± 0% ~ (p=0.926 n=10) _CodecDecodeSeries/application/vnd.google.protobuf-10 7.647Mi ± 0% 7.647Mi ± 0% ~ (p=0.587 n=10) _CodecDecodeSeries/application/json;_charset=utf-8-10 27.94Mi ± 0% 16.99Mi ± 0% -39.18% (p=0.000 n=10) _MergeResponses-10 2.362Mi ± 0% 2.408Mi ± 0% +1.98% (p=0.000 n=10) _UnwrapSeries-10 19.495Mi ± 0% 8.595Mi ± 0% -55.91% (p=0.000 n=10) _DecodeMergeEncodeCycle-10 772.3Mi ± 0% 772.3Mi ± 0% ~ (p=0.912 n=10) geomean 17.50Mi 14.54Mi -16.91% │ before.log │ after.log │ │ allocs/op │ allocs/op vs base │ _CodecDecodeLogs-10 41.10k ± 0% 41.10k ± 0% ~ (p=1.000 n=10) ¹ _CodecDecodeSamples-10 411.9k ± 0% 411.9k ± 0% ~ (p=1.000 n=10) _CodecDecodeSeries/application/vnd.google.protobuf-10 32.00 ± 0% 32.00 ± 0% ~ (p=1.000 n=10) ¹ _CodecDecodeSeries/application/json;_charset=utf-8-10 304.2k ± 0% 298.1k ± 0% -2.01% (p=0.000 n=10) _MergeResponses-10 100.1k ± 0% 100.1k ± 0% -0.00% (p=0.002 n=10) _UnwrapSeries-10 201.1k ± 0% 198.0k ± 0% -1.54% (p=0.000 n=10) _DecodeMergeEncodeCycle-10 203.1k ± 0% 203.1k ± 0% ~ (p=0.621 n=10) geomean 48.95k 48.70k -0.51% ¹ all samples are equal ``` **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	1 year ago
Karsten Jeschkies	f67fff3eb2	Approximate `quantile_over_time` (#10417) ... **What this PR does / why we need it**: This change shards `quantile_over_time` queries using t-digest or DDSketch approximations. It can be enabled with `querier.shard_aggregations=quantile_over_time`. Outstanding - [x] Replace generic return type of `StepEvaluator` with interface `StepResult`. - [x] Send mapped query with quantile sketch expression from frontend to querier over the wire. - [x] Serialize sketches. See https://github.com/influxdata/tdigest/issues/34 - [x] Add feature flag. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [x] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [x] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>	1 year ago
Karsten Jeschkies	1c56aa91ef	Check switches on syntax.Expr for exhaustiveness. (#11113) ... **What this PR does / why we need it**: In preparation to serlialize the LogQL AST as JSON we are declaring the `syntax.Expr` as a "sum type" and test all switch statements on it to be exhaustive. **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](0d4416a4b0)	2 years ago
Charles Korn	bde65667f7	Upgrade dskit to use packages migrated from weaveworks/common (#10164) ... **What this PR does / why we need it**: This PR upgrades dskit and replaces use of packages from weaveworks/common with their migrated equivalents in dskit. See https://github.com/grafana/dskit/pull/342 for more details. Note that Loki still uses some packages from weaveworks/common that I haven't migrated (`aws` and `test`) - I'll migrate these separately. If this PR needs to be rebuilt, I used `rewrite.sh` ([source](https://gist.github.com/charleskorn/48efe62a09d6d70f3de30327003df5c5#file-rewrite-sh)) to generate most of these changes. **Which issue(s) this PR fixes**: (none) **Special notes for your reviewer**: (none) **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [n/a] Documentation added - [n/a] Tests updated - [n/a] `CHANGELOG.md` updated - [n/a] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [n/a] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [n/a] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec) Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>	2 years ago
Trevor Whitney	a12311c5c8	Allow volume to be aggregated by label (#9988) ... Add a `aggregateBy` query parameter to the `series_volume` endpoint, and rename that endpoint to just `volume`. This allows users to get volumes aggregated into top level labels, rather label+value pairs. --------- Co-authored-by: Travis Patterson <travis.patterson@grafana.com>	2 years ago
Travis Patterson	db97058a84	Series volume endpoint (#9704) ... This changes the `label_volume` endpoint to the `series_volume` endpoint. The new endpoint still returns volumes but now it does it for the requested streams defined by the selector names passed rather than individual labels. All relevant non-requested labels are aggregated into the returned results ex: Assume we have the following streams: ``` {cluster="prod", team="A", component="foo"} {cluster="prod", team="B", component="foo"} {cluster="dev", team="A", component="foo"} {cluster="dev", team="B", component="foo"} ``` - requesting `{cluster="prod"}` returns one result for all streams containing `{cluster="prod"}` - requesting `{cluster=~".+"}` returns two results for the streams containing `{cluster="prod"}` and `{cluster="dev"}` - requesting `{cluster=~".+", team=".+"}` returns four results for the streams containing: ``` {cluster="prod", team="A"} {cluster="prod", team="B"} {cluster="dev", team="A"} {cluster="dev", team="B"} ``` --------- Co-authored-by: Trevor Whitney <trevorjwhitney@gmail.com>	2 years ago
Trevor Whitney	4a56445686	Upgrade `golangci-lint` and fix linting errors (#9601) ... **What this PR does / why we need it**: Upgrade `golangci-lint` and fixes all the errors. The upgrade includes some stricter linting.	2 years ago
Travis Patterson	065bee7e72	Label Volume Endpoint (#9588) ... For a given set of matchers, returns the top N associated label/value pairs by volume. A query for `{cluster=prod}` will return ``` cluster=prod: size (total logs matching this matcher) . . . nth-label=nth-value ``` This is to service use cases where users want to understand where their log volume has come from by label without making multiple requests to the stats endpoint. Note: This PR is a monster but it's mostly plumbing. I've pointed out the most interesting bits that actually get the volumes from ingesters/indexs	2 years ago
Peter Štibraný	90a1d4593e	Update Prometheus dependency (#9205)	2 years ago
李国忠	322783e3d8	LogQL: [optimization] syntax: Replace "panic" in "/pkg/logql/syntax" with "error" (#7208)	2 years ago
Robert Fratto	2208ea7147	Update Prometheus dependency to latest release (v2.39.1) (#7425)	3 years ago
Jesus Vazquez	db9dbe8e1d	Bump prometheus dependency (#6403) ... * Bump prometheus dependency We had to add a replace of prometheus in go mod because thanos is importing an older version and then replacing it. * Undo textparser.New variable mismatch * modtimevfs path update * Fix labels withlabels and withoutlabels usages WithoutLabels and WithLabels methods have dissappeared. Now labels.NewBuilder has to be used. * Fix RuleManager interface for Update() Update now accepts a new param called ruleGroupPostProcessFunc which is an optional function to be executed after rule group processing. In this case I've set it to nil in the usages i've found. * NewQueryRange now accepts QueryOpts The NewQueryRange function now accepts QueryOptions so I had to update all the calls. I've set the opts to nil since we're not using them. * gofmt	3 years ago
Owen Diehl	3f4a663dbe	Tsdb/index sampling endpoint (#6347) ... * index sampling endpoint * fix bug in withTenantLabelMatcher * show zero values in stats * handles new errcase * fixes comments	3 years ago
Karsten Jeschkies	c7cbeab8e2	Include tenant ID label in label values. (#5908)	3 years ago
Karsten Jeschkies	7c78b26f80	Enable tenant ID selection for multi-tenant queries. (#5821) ... Closes #5820 Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	3 years ago
JordanRushing	52f9df415d	Update MultiTenantQuerier `Series()` and `Label()` to use `user.InjectOrgID()` instead of `user.InjectUserID()` (#5735)	3 years ago
JordanRushing	1cfff09117	Implement `Series` and `Label` for MultiTenantQuerier (#5566) ... **What this PR does / why we need it**: re: #5490, #5517 This PR continues implementing the `Querier` interface by adding `Label` and `Series` methods for `MultiTenantQuerier`. Two new functions are also added to `pkg/logproto`: `MergeSeriesResponses` and `MergeLabelResponses` which consolidate multi-tenant responses into a single response. **Which issue(s) this PR fixes**: N/A **Special notes for your reviewer**: N/A **Checklist** - [ ] Documentation added - [x] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes. Signed-off-by: JordanRushing <rushing.jordan@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>	3 years ago
Karsten Jeschkies	46956d0bf6	Use dskit/tenant intead of loki/pkg/tenant. (#5724)	3 years ago
Karsten Jeschkies	705baa39f4	Speed up re-labelling in multi-tenant queries. (#5663)	3 years ago
Karsten Jeschkies	3d5a3936d1	Support limits in multi-tenant queries. (#5626) ... **What this PR does / why we need it**: This changes adds support for the the following limits in multi-tenant queries: - `MaxQuerySeries` - `QuerySplitDuration` - `MaxEntriesLimitPerQuery` - `MinShardingLookback` - `MaxQueryLength` - `MaxQueryParallelism` - `MaxQueriersPerUser` - `MaxCacheFreshness` It always uses the more restricted limit. That means the smallest `MaxQuerySeries` but biggest `QuerySplitDuration`. The split duration is special. Zero means disabled. Also, the global default resolver is the multi tenant resolver when multi-tenant queries are enabled. This will result HTTP 400 errors in case the user sends multiple tenants to endpoints that do not support it. **Which issue(s) this PR fixes**: **Special notes for your reviewer**: **Checklist** - [ ] Documentation added - [x] Tests updated - [x] Add an entry in the `CHANGELOG.md` about the changes. Co-authored-by: JordanRushing <rushing.jordan@gmail.com>	3 years ago
Karsten Jeschkies	1deacceca9	Test multi-tenant context propagation. (#5620) ... **What this PR does / why we need it**: The original change injected the user id when the org id was required to fetch all data. **Which issue(s) this PR fixes**: Fixes grafana/loki-private#230 **Special notes for your reviewer**: The changes can be tested as follows 1. Change working directory to `tools/dev/loki-boltdb-storage-s3`. 2. Call `./compose-up.sh` to boot the cluster. 3. Query the logs with ``` curl -v -X GET -G "http://localhost:8007/loki/api/v1/query_range" -H "X-Scope-OrgID: 1|2" --data-urlencode 'query={compose_service=~"log-gen.*"}' | jq '.data.result[].stream' ``` It will should two streams with `__tenant_id: 1` and `__tenant_id_: 2` as part of the labels. **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	3 years ago
Cyril Tovena	79f0e349f5	Refactor LogQL syntax into its own package (#5539) ... * Refactor LogQL syntax into its own package Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Merge upstream Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/loghttp/series.go Co-authored-by: Susana Ferreira <ssncferreira@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>	3 years ago
Karsten Jeschkies	9e16ffbd3b	Support `SelectLogs` and `SelectSamples` in multi-tenant querier. (#5517) ... **What this PR does / why we need it**: This patch adds some functionality to the `MultiTenantQuerier` stub. `SelectLogs` and `SelectSamples` fan out into multiple calls, one for each tenant. The returned iterators are then merged back together and the proper tenant label is added. **Which issue(s) this PR fixes**: Fixes grafana/loki-private#232 **Special notes for your reviewer**: **Checklist** - [ ] Documentation added - [x] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	3 years ago
Karsten Jeschkies	ae39676cce	Stub multi tenant querier. (#5490) ... **What this PR does / why we need it**: - Introduce Queries interface and Querier API. - Configure a multi-tenant querier that just passes requests through. **Which issue(s) this PR fixes**: Closes #231 **Special notes for your reviewer**: **Checklist** - [x] Documentation added - [ ] Tests updated - [ ] Add an entry in the `CHANGELOG.md` about the changes.	3 years ago