Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Karsten Jeschkies fbcaa1d5d8 Lazily decode series protobuf. (#10071) ... **What this PR does / why we need it**: The protobuf decoding for series responses runs into a memory issue for many series. The response is only merged and passed through the front end. It is more efficient to decode the protobuf encoded series lazily. And `decode`, `merge`, `encode` benchmark shows the benefit of transcoding the protobuf series message into a JSON response directly. This change will not impact production code since it only applies to protobuf encoding messaging between the querier and query frontend that must be explicitly enabled. ``` › go test -v -run=^$ -bench "Benchmark_DecodeMergeEncodeCycle" -memprofile memory_base.prof -count=10 ./pkg/querier/queryrange > before.txt › go test -v -run=^$ -bench "Benchmark_DecodeMergeEncodeCycle" -memprofile memory_base.prof -count=10 ./pkg/querier/queryrange > before.txt › benchstat before.txt after.txt before.txt:5: missing iteration count after.txt:5: missing iteration count goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/querier/queryrange cpu: AMD Ryzen 7 3700X 8-Core Processor │ before.txt │ after.txt │ │ sec/op │ sec/op vs base │ _DecodeMergeEncodeCycle-16 2537.7m ± 2% 934.2m ± 1% -63.19% (p=0.000 n=10) │ before.txt │ after.txt │ │ B/op │ B/op vs base │ _DecodeMergeEncodeCycle-16 1723.4Mi ± 0% 641.1Mi ± 0% -62.80% (p=0.000 n=10) │ before.txt │ after.txt │ │ allocs/op │ allocs/op vs base │ _DecodeMergeEncodeCycle-16 20240.6k ± 0% 203.0k ± 0% -99.00% (p=0.000 n=10) ``` **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)		2 years ago
..
LICENSE_APACHE2	Re-license to AGPLv3 (#3630)	4 years ago
alias.go	Add metadata to push payload (#9694)	2 years ago
compat.go	Implement series volume range queries (#9812)	2 years ago
compat_test.go	Remove some unused code (#9611)	2 years ago
extensions.go	Lazily decode series protobuf. (#10071)	2 years ago
indexgateway.pb.go	Allow volume to be aggregated by label (#9988)	2 years ago
indexgateway.proto	Allow volume to be aggregated by label (#9988)	2 years ago
logproto.pb.go	Allow volume to be aggregated by label (#9988)	2 years ago
logproto.proto	Allow volume to be aggregated by label (#9988)	2 years ago
metrics.pb.go	Update Prometheus dependency to latest release (v2.40.4) (#7826)	3 years ago
metrics.proto	Update Prometheus dependency to latest release (v2.40.4) (#7826)	3 years ago
sketch.pb.go	Define protobufs for topk and cms. (#9933)	2 years ago
sketch.proto	Define protobufs for topk and cms. (#9933)	2 years ago
timeseries.go	Add metadata to push payload (#9694)	2 years ago
timeseries_test.go	Implement `LegacySample`, `LegacyLabelPair` proto messages (#5301)	3 years ago
types.go	Logproto: Extract push.proto from logproto package to the separate module (#8259)	2 years ago