Like Prometheus, but for logs.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
loki/pkg/querier/queryrange/benchmarkutils_test.go

90 lines
2.0 KiB

Loki Query Frontend (#1442) * Adds frontend to Loki. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improves tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes sneaky bug in entries sorting. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes the split by interval. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Tweak jsonnet deployments and add a way to lint/fmt. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * fix timezone issue. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve tests and rollback change in loghttp package. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes a flaky test that might run one more goroutine. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes windows build. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Improve tracing in the split by interval. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add test stream to proto conversion. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes flappy retry test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Remove err shadowing in stopQueryFrontend as it was confusing. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Refactor grpc message size in the libsonnet config file. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Don't check auth header for GRPC TransferChunks. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Query frontend (#3) * frontend codec merging optimizations * codec benchmarks * removes unused bounds code in queryrange ordering * [wip] splitby uses channels instead of sub batching intervals * splitBy channel limit test * single allocation for merging entries from a single stream * skip merging loki responses when limit is already hit * removes checks for unlimited queries in queryrange * removes splitByInterval{,.interval} spans * removes interval_batch_size from jsonnet lib * moves benchmark utils to own file * renames markers -> entries * priority queue comments * Removes unused logRequest. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Sets the cache interval to the same split interval. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Missing import libsonnet for the frontend. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Frontend should not be a cluster IP. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
5 years ago
package queryrange
import (
"sort"
"github.com/grafana/loki/pkg/logproto"
)
type entry struct {
entry logproto.Entry
labels string
}
type byDirection struct {
direction logproto.Direction
entries []entry
}
func (a byDirection) Len() int { return len(a.entries) }
func (a byDirection) Swap(i, j int) { a.entries[i], a.entries[j] = a.entries[j], a.entries[i] }
func (a byDirection) Less(i, j int) bool {
e1, e2 := a.entries[i], a.entries[j]
if a.direction == logproto.BACKWARD {
switch {
case e1.entry.Timestamp.UnixNano() < e2.entry.Timestamp.UnixNano():
return false
case e1.entry.Timestamp.UnixNano() > e2.entry.Timestamp.UnixNano():
return true
default:
return e1.labels > e2.labels
}
}
switch {
case e1.entry.Timestamp.UnixNano() < e2.entry.Timestamp.UnixNano():
return true
case e1.entry.Timestamp.UnixNano() > e2.entry.Timestamp.UnixNano():
return false
default:
return e1.labels < e2.labels
}
}
func mergeStreams(resps []*LokiResponse, limit uint32, direction logproto.Direction) []logproto.Stream {
output := byDirection{
direction: direction,
entries: []entry{},
}
for _, resp := range resps {
for _, stream := range resp.Data.Result {
for _, e := range stream.Entries {
output.entries = append(output.entries, entry{
entry: e,
labels: stream.Labels,
})
}
}
}
sort.Sort(output)
// limit result
if len(output.entries) >= int(limit) {
output.entries = output.entries[:limit]
}
resultDict := map[string]*logproto.Stream{}
for _, e := range output.entries {
stream, ok := resultDict[e.labels]
if !ok {
stream = &logproto.Stream{
Labels: e.labels,
Entries: []logproto.Entry{},
}
resultDict[e.labels] = stream
}
stream.Entries = append(stream.Entries, e.entry)
}
keys := make([]string, 0, len(resultDict))
for key := range resultDict {
keys = append(keys, key)
}
sort.Strings(keys)
result := make([]logproto.Stream, 0, len(resultDict))
for _, key := range keys {
result = append(result, *resultDict[key])
}
return result
}