apitech/loki - loki - Gitea: Git with a cup of tea

Author SHA1 Message Date

Author	SHA1	Message	Date
Karsten Jeschkies	fbcaa1d5d8	Lazily decode series protobuf. (#10071 ) What this PR does / why we need it: The protobuf decoding for series responses runs into a memory issue for many series. The response is only merged and passed through the front end. It is more efficient to decode the protobuf encoded series lazily. And `decode`, `merge`, `encode` benchmark shows the benefit of transcoding the protobuf series message into a JSON response directly. This change will not impact production code since it only applies to protobuf encoding messaging between the querier and query frontend that must be explicitly enabled. ``` › go test -v -run=^$ -bench "Benchmark_DecodeMergeEncodeCycle" -memprofile memory_base.prof -count=10 ./pkg/querier/queryrange > before.txt › go test -v -run=^$ -bench "Benchmark_DecodeMergeEncodeCycle" -memprofile memory_base.prof -count=10 ./pkg/querier/queryrange > before.txt › benchstat before.txt after.txt before.txt:5: missing iteration count after.txt:5: missing iteration count goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/querier/queryrange cpu: AMD Ryzen 7 3700X 8-Core Processor │ before.txt │ after.txt │ │ sec/op │ sec/op vs base │ _DecodeMergeEncodeCycle-16 2537.7m ± 2% 934.2m ± 1% -63.19% (p=0.000 n=10) │ before.txt │ after.txt │ │ B/op │ B/op vs base │ _DecodeMergeEncodeCycle-16 1723.4Mi ± 0% 641.1Mi ± 0% -62.80% (p=0.000 n=10) │ before.txt │ after.txt │ │ allocs/op │ allocs/op vs base │ _DecodeMergeEncodeCycle-16 20240.6k ± 0% 203.0k ± 0% -99.00% (p=0.000 n=10) ``` Checklist - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (required) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](`d10549e3ec`)	3 years ago

Karsten Jeschkies

Lazily decode series protobuf. (#10071 )

**What this PR does / why we need it**:
The protobuf decoding for series responses runs into a memory issue for
many series. The response is only merged and passed through the front
end. It is more efficient to decode the protobuf encoded series lazily.
And `decode`, `merge`, `encode` benchmark shows the benefit of
transcoding the protobuf series message into a JSON response directly.

This change will not impact production code since it only applies to
protobuf encoding messaging between the querier and query frontend that
must be explicitly enabled.

```
› go test -v -run=^$ -bench "Benchmark_DecodeMergeEncodeCycle" -memprofile memory_base.prof -count=10 ./pkg/querier/queryrange > before.txt
› go test -v -run=^$ -bench "Benchmark_DecodeMergeEncodeCycle" -memprofile memory_base.prof -count=10 ./pkg/querier/queryrange > before.txt
› benchstat before.txt after.txt
before.txt:5: missing iteration count
after.txt:5: missing iteration count
goos: linux
goarch: amd64
pkg: github.com/grafana/loki/pkg/querier/queryrange
cpu: AMD Ryzen 7 3700X 8-Core Processor             
                           │  before.txt  │              after.txt              │
                           │    sec/op    │   sec/op     vs base                │
_DecodeMergeEncodeCycle-16   2537.7m ± 2%   934.2m ± 1%  -63.19% (p=0.000 n=10)

                           │  before.txt   │              after.txt               │
                           │     B/op      │     B/op      vs base                │
_DecodeMergeEncodeCycle-16   1723.4Mi ± 0%   641.1Mi ± 0%  -62.80% (p=0.000 n=10)

                           │  before.txt   │              after.txt              │
                           │   allocs/op   │  allocs/op   vs base                │
_DecodeMergeEncodeCycle-16   20240.6k ± 0%   203.0k ± 0%  -99.00% (p=0.000 n=10)

```

**Checklist**
- [ ] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [ ] Documentation added
- [x] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/setup/upgrade/_index.md`
- [ ] For Helm chart changes bump the Helm chart version in
`production/helm/loki/Chart.yaml` and update
`production/helm/loki/CHANGELOG.md` and
`production/helm/loki/README.md`. [Example
PR](d10549e3ec)

3 years ago

1 Commits (89985399817dde28fe5b94de990133d904bc3e5b)