Like Prometheus, but for logs.

prometheushacktoberfestgrafanacloudnativeloggingloki

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

Karsten Jeschkies fbcaa1d5d8 Lazily decode series protobuf. (#10071) ... **What this PR does / why we need it**: The protobuf decoding for series responses runs into a memory issue for many series. The response is only merged and passed through the front end. It is more efficient to decode the protobuf encoded series lazily. And `decode`, `merge`, `encode` benchmark shows the benefit of transcoding the protobuf series message into a JSON response directly. This change will not impact production code since it only applies to protobuf encoding messaging between the querier and query frontend that must be explicitly enabled. ``` › go test -v -run=^$ -bench "Benchmark_DecodeMergeEncodeCycle" -memprofile memory_base.prof -count=10 ./pkg/querier/queryrange > before.txt › go test -v -run=^$ -bench "Benchmark_DecodeMergeEncodeCycle" -memprofile memory_base.prof -count=10 ./pkg/querier/queryrange > before.txt › benchstat before.txt after.txt before.txt:5: missing iteration count after.txt:5: missing iteration count goos: linux goarch: amd64 pkg: github.com/grafana/loki/pkg/querier/queryrange cpu: AMD Ryzen 7 3700X 8-Core Processor │ before.txt │ after.txt │ │ sec/op │ sec/op vs base │ _DecodeMergeEncodeCycle-16 2537.7m ± 2% 934.2m ± 1% -63.19% (p=0.000 n=10) │ before.txt │ after.txt │ │ B/op │ B/op vs base │ _DecodeMergeEncodeCycle-16 1723.4Mi ± 0% 641.1Mi ± 0% -62.80% (p=0.000 n=10) │ before.txt │ after.txt │ │ allocs/op │ allocs/op vs base │ _DecodeMergeEncodeCycle-16 20240.6k ± 0% 203.0k ± 0% -99.00% (p=0.000 n=10) ``` **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e3ec)		2 years ago
..
queryrangebase	Support content negotiation between query frontend and querier. (#9813)	2 years ago
benchmarkutils_test.go	Loki Query Frontend (#1442)	6 years ago
codec.go	Lazily decode series protobuf. (#10071)	2 years ago
codec_test.go	Lazily decode series protobuf. (#10071)	2 years ago
downstreamer.go	Define protobufs for topk and cms. (#9933)	2 years ago
downstreamer_test.go	Update Prometheus dependency (#9205)	3 years ago
extensions.go	Define protobufs for topk and cms. (#9933)	2 years ago
index_stats_cache.go	Add config to avoid caching stats for recent data (#9537)	3 years ago
index_stats_cache_test.go	Support content negotiation between query frontend and querier. (#9813)	2 years ago
limits.go	Remove goroutines from limiter middleware. (#9923)	2 years ago
limits_test.go	Support content negotiation between query frontend and querier. (#9813)	2 years ago
log_result_cache.go	caching: do not try to fill the gap in log results cache when the new query interval does not overlap the cached query interval (#9757)	3 years ago
log_result_cache_test.go	caching: do not try to fill the gap in log results cache when the new query interval does not overlap the cached query interval (#9757)	3 years ago
marshal.go	Allow volume to be aggregated by label (#9988)	2 years ago
metrics.go	adds result cache key version comparison metrics (#7323)	3 years ago
ordering.go	Loki Query Frontend (#1442)	6 years ago
prometheus.go	Cache index stats requests (#8972)	3 years ago
prometheus_test.go	Improve observability for non-indexed labels usage (#9993)	2 years ago
queryrange.pb.go	Define protobufs for topk and cms. (#9933)	2 years ago
queryrange.proto	Define protobufs for topk and cms. (#9933)	2 years ago
querysharding.go	Add back cache stats for index stats requests (#9816)	3 years ago
querysharding_test.go	Support content negotiation between query frontend and querier. (#9813)	2 years ago
roundtrip.go	Logging improvements for volume requests (#10099)	2 years ago
roundtrip_test.go	Allow volume to be aggregated by label (#9988)	2 years ago
series_volume.go	Logging improvements for volume requests (#10099)	2 years ago
series_volume_test.go	sort sort log volumes by size (#10045)	2 years ago
shard_resolver.go	uses lowercase standard for logging and adds total bytes to shard res… (#9828)	3 years ago
shard_resolver_test.go	adds tsdb-max-bytes-per-shard limit (#9811)	3 years ago
split_by_interval.go	Allow volume to be aggregated by label (#9988)	2 years ago
split_by_interval_test.go	Allow volume to be aggregated by label (#9988)	2 years ago
split_by_range.go	Fix instant query summary split stats (#9773)	3 years ago
split_by_range_test.go	Pass engine opts down to middlewares (#9130)	3 years ago
stats.go	Logging improvements for volume requests (#10099)	2 years ago
stats_test.go	Get basic statistics from series volume requests (#9832)	2 years ago
views.go	Lazily decode series protobuf. (#10071)	2 years ago
views_test.go	Lazily decode series protobuf. (#10071)	2 years ago