Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Karsten Jeschkies fbcaa1d5d8 Lazily decode series protobuf. (#10071) ... **What this PR does / why we need it**: The protobuf decoding for series responses runs into a memory issue for many series. The response is only merged and passed through the front end. It is more efficient to decode the protobuf encoded series lazily. And `decode`, `merge`, `encode` benchmark shows the benefit of transcoding the protobuf series message into a JSON response directly. This change will not impact production code since it only applies to protobuf encoding messaging between the querier and query frontend that must be explicitly enabled. ``` › go test -v -run=^$ -bench "Benchmark_DecodeMergeEncodeCycle" -memprofile memory_base.prof -count=10 ./pkg/querier/queryrange > before.txt › go test -v -run=^$ -bench "Benchmark_DecodeMergeEncodeCycle" -memprofile memory_base.prof -count=10 ./pkg/querier/queryrange > before.txt › benchstat before.txt after.txt before.txt:5: missing iteration count after.txt:5: missing iteration count goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/querier/queryrange cpu: AMD Ryzen 7 3700X 8-Core Processor │ before.txt │ after.txt │ │ sec/op │ sec/op vs base │ _DecodeMergeEncodeCycle-16 2537.7m ± 2% 934.2m ± 1% -63.19% (p=0.000 n=10) │ before.txt │ after.txt │ │ B/op │ B/op vs base │ _DecodeMergeEncodeCycle-16 1723.4Mi ± 0% 641.1Mi ± 0% -62.80% (p=0.000 n=10) │ before.txt │ after.txt │ │ allocs/op │ allocs/op vs base │ _DecodeMergeEncodeCycle-16 20240.6k ± 0% 203.0k ± 0% -99.00% (p=0.000 n=10) ``` **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)		3 years ago
..
astmapper	Upgrade `golangci-lint` and fix linting errors (#9601)	3 years ago
queryrange	Lazily decode series protobuf. (#10071)	3 years ago
series	Remove some unused code (#9611)	3 years ago
stats	Update Prometheus dependency to latest release (v2.40.4) (#7826)	3 years ago
worker	Update dskit (#10091)	3 years ago
http.go	Allow volume to be aggregated by label (#9988)	3 years ago
http_test.go	Allow volume to be aggregated by label (#9988)	3 years ago
ingester_querier.go	Allow volume to be aggregated by label (#9988)	3 years ago
ingester_querier_test.go	Allow volume to be aggregated by label (#9988)	3 years ago
metrics.go	Add tailing metrics (#6059)	4 years ago
multi_tenant_querier.go	Allow volume to be aggregated by label (#9988)	3 years ago
multi_tenant_querier_test.go	Allow volume to be aggregated by label (#9988)	3 years ago
querier.go	Allow volume to be aggregated by label (#9988)	3 years ago
querier_mock_test.go	Allow volume to be aggregated by label (#9988)	3 years ago
querier_test.go	Allow volume to be aggregated by label (#9988)	3 years ago
tail.go	Iterators: re-implement mergeEntryIterator using loser.Tree for performance (#8637)	3 years ago
tail_mock_test.go	Improve protobuf serialization (#2031)	6 years ago
tail_test.go	Iterators: re-implement mergeEntryIterator using loser.Tree for performance (#8637)	3 years ago
testutils.go	Removing duplicated limits file and refactored references (#8210)	3 years ago
worker_service.go	loki: enable PathPrefix for queryHandler endpoints (#8406)	3 years ago
worker_service_test.go	Support content negotiation between query frontend and querier. (#9813)	3 years ago