Like Prometheus, but for logs.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
loki/pkg/querier/queryrange/split_by_range_test.go

533 lines
21 KiB

Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
package queryrange
import (
"context"
"fmt"
"testing"
"time"
"github.com/go-kit/log"
"github.com/grafana/dskit/user"
"github.com/stretchr/testify/assert"
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
"github.com/stretchr/testify/require"
"github.com/grafana/loki/v3/pkg/loghttp"
"github.com/grafana/loki/v3/pkg/logproto"
"github.com/grafana/loki/v3/pkg/logql/syntax"
"github.com/grafana/loki/v3/pkg/querier/plan"
"github.com/grafana/loki/v3/pkg/querier/queryrange/queryrangebase"
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
)
func Test_RangeVectorSplitAlign(t *testing.T) {
var (
twelve34 = time.Date(1970, 1, 1, 12, 34, 0, 0, time.UTC) // 1970 12:34:00 UTC
twelve = time.Date(1970, 1, 1, 12, 00, 0, 0, time.UTC) // 1970 12:00:00 UTC
eleven = twelve.Add(-1 * time.Hour) // 1970 11:00:00 UTC
ten = eleven.Add(-1 * time.Hour) // 1970 10:00:00 UTC
)
for _, tc := range []struct {
name string
in queryrangebase.Request
subQueries []queryrangebase.RequestResponse
expected queryrangebase.Response
splitByInterval time.Duration
}{
{
name: "sum_splitBy_aligned_with_query_time",
splitByInterval: 1 * time.Minute,
in: &LokiInstantRequest{
Query: `sum(bytes_over_time({app="foo"}[3m]))`,
TimeTs: time.Unix(180, 0),
Path: "/loki/api/v1/query",
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum(bytes_over_time({app="foo"}[3m]))`),
},
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponseWithQueryTime(`sum(bytes_over_time({app="foo"}[1m]))`, 1, time.Unix(60, 0)),
subQueryRequestResponseWithQueryTime(`sum(bytes_over_time({app="foo"}[1m]))`, 2, time.Unix(120, 0)),
subQueryRequestResponseWithQueryTime(`sum(bytes_over_time({app="foo"}[1m]))`, 3, time.Unix(180, 0)),
},
expected: expectedMergedResponseWithTime(1+2+3, time.Unix(180, 0)), // original `TimeTs` of the query.
},
{
name: "sum_splitBy_not_aligned_query_time",
splitByInterval: 1 * time.Hour,
in: &LokiInstantRequest{
Query: `sum(bytes_over_time({app="foo"}[3h]))`,
TimeTs: twelve34,
Path: "/loki/api/v1/query",
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum(bytes_over_time({app="foo"}[3h]))`),
},
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponseWithQueryTime(`sum(bytes_over_time({app="foo"}[34m]))`, 1, twelve34),
subQueryRequestResponseWithQueryTime(`sum(bytes_over_time({app="foo"}[1h]))`, 2, twelve),
subQueryRequestResponseWithQueryTime(`sum(bytes_over_time({app="foo"}[1h]))`, 3, eleven),
subQueryRequestResponseWithQueryTime(`sum(bytes_over_time({app="foo"}[26m]))`, 4, ten),
},
expected: expectedMergedResponseWithTime(1+2+3+4, twelve34), // original `TimeTs` of the query.
},
{
name: "sum_aggregation_splitBy_aligned_with_query_time",
splitByInterval: 1 * time.Minute,
in: &LokiInstantRequest{
Query: `sum by (bar) (bytes_over_time({app="foo"}[3m]))`,
TimeTs: time.Unix(180, 0),
Path: "/loki/api/v1/query",
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum by (bar) (bytes_over_time({app="foo"}[3m]))`),
},
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponseWithQueryTime(`sum by (bar)(bytes_over_time({app="foo"}[1m]))`, 10, time.Unix(60, 0)),
subQueryRequestResponseWithQueryTime(`sum by (bar)(bytes_over_time({app="foo"}[1m]))`, 20, time.Unix(120, 0)),
subQueryRequestResponseWithQueryTime(`sum by (bar)(bytes_over_time({app="foo"}[1m]))`, 30, time.Unix(180, 0)),
},
expected: expectedMergedResponseWithTime(10+20+30, time.Unix(180, 0)),
},
{
name: "sum_aggregation_splitBy_not_aligned_with_query_time",
splitByInterval: 1 * time.Hour,
in: &LokiInstantRequest{
Query: `sum by (bar) (bytes_over_time({app="foo"}[3h]))`,
TimeTs: twelve34,
Path: "/loki/api/v1/query",
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum by (bar) (bytes_over_time({app="foo"}[3h]))`),
},
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponseWithQueryTime(`sum by (bar)(bytes_over_time({app="foo"}[34m]))`, 10, twelve34), // 12:34:00
subQueryRequestResponseWithQueryTime(`sum by (bar)(bytes_over_time({app="foo"}[1h]))`, 20, twelve), // 12:00:00 aligned
subQueryRequestResponseWithQueryTime(`sum by (bar)(bytes_over_time({app="foo"}[1h]))`, 30, eleven), // 11:00:00 aligned
subQueryRequestResponseWithQueryTime(`sum by (bar)(bytes_over_time({app="foo"}[26m]))`, 40, ten), // 10:00:00
},
expected: expectedMergedResponseWithTime(10+20+30+40, twelve34),
},
{
name: "count_over_time_aligned_with_query_time",
splitByInterval: 1 * time.Minute,
in: &LokiInstantRequest{
Query: `sum(count_over_time({app="foo"}[3m]))`,
TimeTs: time.Unix(180, 0),
Path: "/loki/api/v1/query",
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum(count_over_time({app="foo"}[3m]))`),
},
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponseWithQueryTime(`sum(count_over_time({app="foo"}[1m]))`, 1, time.Unix(60, 0)),
subQueryRequestResponseWithQueryTime(`sum(count_over_time({app="foo"}[1m]))`, 1, time.Unix(120, 0)),
subQueryRequestResponseWithQueryTime(`sum(count_over_time({app="foo"}[1m]))`, 1, time.Unix(180, 0)),
},
expected: expectedMergedResponseWithTime(1+1+1, time.Unix(180, 0)),
},
{
name: "count_over_time_not_aligned_with_query_time",
splitByInterval: 1 * time.Hour,
in: &LokiInstantRequest{
Query: `sum(count_over_time({app="foo"}[3h]))`,
TimeTs: twelve34,
Path: "/loki/api/v1/query",
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum(count_over_time({app="foo"}[3h]))`),
},
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponseWithQueryTime(`sum(count_over_time({app="foo"}[34m]))`, 1, twelve34),
subQueryRequestResponseWithQueryTime(`sum(count_over_time({app="foo"}[1h]))`, 1, twelve),
subQueryRequestResponseWithQueryTime(`sum(count_over_time({app="foo"}[1h]))`, 1, eleven),
subQueryRequestResponseWithQueryTime(`sum(count_over_time({app="foo"}[26m]))`, 1, ten),
},
expected: expectedMergedResponseWithTime(1+1+1+1, twelve34),
},
{
name: "sum_agg_count_over_time_align_with_query_time",
splitByInterval: 1 * time.Minute,
in: &LokiInstantRequest{
Query: `sum by (bar) (count_over_time({app="foo"}[3m]))`,
TimeTs: time.Unix(180, 0),
Path: "/loki/api/v1/query",
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum by (bar) (count_over_time({app="foo"}[3m]))`),
},
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponseWithQueryTime(`sum by (bar)(count_over_time({app="foo"}[1m]))`, 0, time.Unix(60, 0)),
subQueryRequestResponseWithQueryTime(`sum by (bar)(count_over_time({app="foo"}[1m]))`, 0, time.Unix(120, 0)),
subQueryRequestResponseWithQueryTime(`sum by (bar)(count_over_time({app="foo"}[1m]))`, 0, time.Unix(180, 0)),
},
expected: expectedMergedResponseWithTime(0+0+0, time.Unix(180, 0)),
},
{
name: "sum_agg_count_over_time_not_align_with_query_time",
splitByInterval: 1 * time.Hour,
in: &LokiInstantRequest{
Query: `sum by (bar) (count_over_time({app="foo"}[3h]))`,
TimeTs: twelve34,
Path: "/loki/api/v1/query",
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum by (bar) (count_over_time({app="foo"}[3h]))`),
},
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponseWithQueryTime(`sum by (bar)(count_over_time({app="foo"}[34m]))`, 0, twelve34),
subQueryRequestResponseWithQueryTime(`sum by (bar)(count_over_time({app="foo"}[1h]))`, 0, twelve),
subQueryRequestResponseWithQueryTime(`sum by (bar)(count_over_time({app="foo"}[1h]))`, 0, eleven),
subQueryRequestResponseWithQueryTime(`sum by (bar)(count_over_time({app="foo"}[26m]))`, 0, ten),
},
expected: expectedMergedResponseWithTime(0+0+0+0, twelve34),
},
{
name: "sum_over_time_aligned_with_query_time",
splitByInterval: 1 * time.Minute,
in: &LokiInstantRequest{
Query: `sum(sum_over_time({app="foo"} | unwrap bar [3m]))`,
TimeTs: time.Unix(180, 0),
Path: "/loki/api/v1/query",
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum(sum_over_time({app="foo"} | unwrap bar [3m]))`),
},
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponseWithQueryTime(`sum(sum_over_time({app="foo"} | unwrap bar[1m]))`, 1, time.Unix(60, 0)),
subQueryRequestResponseWithQueryTime(`sum(sum_over_time({app="foo"} | unwrap bar[1m]))`, 2, time.Unix(120, 0)),
subQueryRequestResponseWithQueryTime(`sum(sum_over_time({app="foo"} | unwrap bar[1m]))`, 3, time.Unix(180, 0)),
},
expected: expectedMergedResponseWithTime(1+2+3, time.Unix(180, 0)),
},
{
name: "sum_over_time_not_aligned_with_query_time",
splitByInterval: 1 * time.Hour,
in: &LokiInstantRequest{
Query: `sum(sum_over_time({app="foo"} | unwrap bar [3h]))`,
TimeTs: twelve34,
Path: "/loki/api/v1/query",
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum(sum_over_time({app="foo"} | unwrap bar [3h]))`),
},
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponseWithQueryTime(`sum(sum_over_time({app="foo"} | unwrap bar[34m]))`, 1, twelve34),
subQueryRequestResponseWithQueryTime(`sum(sum_over_time({app="foo"} | unwrap bar[1h]))`, 2, twelve),
subQueryRequestResponseWithQueryTime(`sum(sum_over_time({app="foo"} | unwrap bar[1h]))`, 3, eleven),
subQueryRequestResponseWithQueryTime(`sum(sum_over_time({app="foo"} | unwrap bar[26m]))`, 4, ten),
},
expected: expectedMergedResponseWithTime(1+2+3+4, twelve34),
},
{
name: "sum_agg_sum_over_time_aligned_with_query_time",
splitByInterval: 1 * time.Minute,
in: &LokiInstantRequest{
Query: `sum by (bar) (sum_over_time({app="foo"} | unwrap bar [3m]))`,
TimeTs: time.Unix(180, 0),
Path: "/loki/api/v1/query",
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum by (bar) (sum_over_time({app="foo"} | unwrap bar [3m]))`),
},
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponseWithQueryTime(`sum by (bar)(sum_over_time({app="foo"} | unwrap bar[1m]))`, 1, time.Unix(60, 0)),
subQueryRequestResponseWithQueryTime(`sum by (bar)(sum_over_time({app="foo"} | unwrap bar[1m]))`, 2, time.Unix(120, 0)),
subQueryRequestResponseWithQueryTime(`sum by (bar)(sum_over_time({app="foo"} | unwrap bar[1m]))`, 3, time.Unix(180, 0)),
},
expected: expectedMergedResponseWithTime(1+2+3, time.Unix(180, 0)),
},
{
name: "sum_agg_sum_over_time_not_aligned_with_query_time",
splitByInterval: 1 * time.Hour,
in: &LokiInstantRequest{
Query: `sum by (bar) (sum_over_time({app="foo"} | unwrap bar [3h]))`,
TimeTs: twelve34,
Path: "/loki/api/v1/query",
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum by (bar) (sum_over_time({app="foo"} | unwrap bar [3h]))`),
},
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponseWithQueryTime(`sum by (bar)(sum_over_time({app="foo"} | unwrap bar[34m]))`, 1, twelve34),
subQueryRequestResponseWithQueryTime(`sum by (bar)(sum_over_time({app="foo"} | unwrap bar[1h]))`, 2, twelve),
subQueryRequestResponseWithQueryTime(`sum by (bar)(sum_over_time({app="foo"} | unwrap bar[1h]))`, 3, eleven),
subQueryRequestResponseWithQueryTime(`sum by (bar)(sum_over_time({app="foo"} | unwrap bar[26m]))`, 4, ten),
},
expected: expectedMergedResponseWithTime(1+2+3+4, twelve34),
},
} {
tc := tc
t.Run(tc.name, func(t *testing.T) {
srm := NewSplitByRangeMiddleware(log.NewNopLogger(), testEngineOpts, fakeLimits{
maxSeries: 10000,
queryTimeout: time.Second,
instantMetricSplitDuration: map[string]time.Duration{
"tenant": tc.splitByInterval,
},
}, true, nilShardingMetrics) // enable splitAlign
ctx := user.InjectOrgID(context.TODO(), "tenant")
byTimeTs := make(map[int64]queryrangebase.RequestResponse)
for _, v := range tc.subQueries {
key := v.Request.(*LokiInstantRequest).TimeTs.UnixNano()
byTimeTs[key] = v
}
resp, err := srm.Wrap(queryrangebase.HandlerFunc(
func(ctx context.Context, req queryrangebase.Request) (queryrangebase.Response, error) {
// req should match with one of the subqueries.
ts := req.(*LokiInstantRequest).TimeTs
subq, ok := byTimeTs[ts.UnixNano()]
if !ok { // every req **should** match with one of the subqueries
return nil, fmt.Errorf("subquery request '%s-%s' not found", req.GetQuery(), ts)
}
// Assert subquery request
assert.Equal(t, subq.Request.GetQuery(), req.GetQuery())
assert.Equal(t, subq.Request, req)
return subq.Response, nil
})).Do(ctx, tc.in)
require.NoError(t, err)
assert.Equal(t, tc.expected, resp.(*LokiPromResponse).Response)
})
}
}
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
func Test_RangeVectorSplit(t *testing.T) {
Pass engine opts down to middlewares (#9130) **What this PR does / why we need it**: The following middlewares in the query frontend uses a downstream engine: - `NewQuerySizeLimiterMiddleware` and `NewQuerierSizeLimiterMiddleware` - `NewQueryShardMiddleware` - `NewSplitByRangeMiddleware` These were all creating the downstream engine as follows: ```go logql.NewDownstreamEngine(logql.EngineOpts{LogExecutingQuery: false}, DownstreamHandler{next: next, limits: limits}, limits, logger), ``` As can be seen, the [engine options configured in Loki][1] were not being used at all. In the case of `NewQuerySizeLimiterMiddleware`, `NewQuerierSizeLimiterMiddleware` and `NewQueryShardMiddleware`, the downstream engine was created to get the `MaxLookBackPeriod`. When creating a new Downstream Engine as above, the `MaxLookBackPeriod` [would always be the default][2] (30 seconds). This PR fixes this by passing down the engine config to these middlewares, so this config is used to create the new downstream engines. **Which issue(s) this PR fixes**: Adresses some pending tasks from https://github.com/grafana/loki/pull/8670#issuecomment-1507031976. **Special notes for your reviewer**: **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` [1]: https://github.com/grafana/loki/blob/1bcf683513b5e11fb1dcc5625761b9760cfbd958/pkg/querier/querier.go#L52 [2]: https://github.com/grafana/loki/blob/edc6b0bff76714ff584c20d7ff0235461a4f4a88/pkg/logql/engine.go#L136-L140
3 years ago
srm := NewSplitByRangeMiddleware(log.NewNopLogger(), testEngineOpts, fakeLimits{
maxSeries: 10000,
queryTimeout: time.Second,
instantMetricSplitDuration: map[string]time.Duration{
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
"tenant": time.Minute,
},
}, false, nilShardingMetrics)
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
ctx := user.InjectOrgID(context.TODO(), "tenant")
for _, tc := range []struct {
in queryrangebase.Request
subQueries []queryrangebase.RequestResponse
expected queryrangebase.Response
}{
{
in: &LokiInstantRequest{
Query: `sum(bytes_over_time({app="foo"}[3m]))`,
TimeTs: time.Unix(1, 0),
Path: "/loki/api/v1/query",
Send query plan to querier. (#11246) **What this PR does / why we need it**: Following https://github.com/grafana/loki/pull/11123 and in order to enable https://github.com/grafana/loki/pull/10417 the query frontend should send the serialized LogQL AST instead of the query string to the queriers. This enables the frontend to change the AST and inject expressions that are not expressible in LogQL. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](https://github.com/grafana/loki/commit/d10549e3ece02120974929894ee333d07755d213) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](https://github.com/grafana/loki/pull/10840/commits/0d4416a4b03739583349934b96f272fb4f685d15) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>
2 years ago
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum(bytes_over_time({app="foo"}[3m]))`),
},
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponse(`sum(bytes_over_time({app="foo"}[1m]))`, 1),
subQueryRequestResponse(`sum(bytes_over_time({app="foo"}[1m] offset 1m0s))`, 2),
subQueryRequestResponse(`sum(bytes_over_time({app="foo"}[1m] offset 2m0s))`, 3),
},
expected: expectedMergedResponse(1 + 2 + 3),
},
{
in: &LokiInstantRequest{
Query: `sum by (bar) (bytes_over_time({app="foo"}[3m]))`,
TimeTs: time.Unix(1, 0),
Path: "/loki/api/v1/query",
Send query plan to querier. (#11246) **What this PR does / why we need it**: Following https://github.com/grafana/loki/pull/11123 and in order to enable https://github.com/grafana/loki/pull/10417 the query frontend should send the serialized LogQL AST instead of the query string to the queriers. This enables the frontend to change the AST and inject expressions that are not expressible in LogQL. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](https://github.com/grafana/loki/commit/d10549e3ece02120974929894ee333d07755d213) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](https://github.com/grafana/loki/pull/10840/commits/0d4416a4b03739583349934b96f272fb4f685d15) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>
2 years ago
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum by (bar) (bytes_over_time({app="foo"}[3m]))`),
},
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponse(`sum by (bar)(bytes_over_time({app="foo"}[1m]))`, 10),
subQueryRequestResponse(`sum by (bar)(bytes_over_time({app="foo"}[1m] offset 1m0s))`, 20),
subQueryRequestResponse(`sum by (bar)(bytes_over_time({app="foo"}[1m] offset 2m0s))`, 30),
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
},
expected: expectedMergedResponse(10 + 20 + 30),
},
{
in: &LokiInstantRequest{
Query: `sum(count_over_time({app="foo"}[3m]))`,
TimeTs: time.Unix(1, 0),
Path: "/loki/api/v1/query",
Send query plan to querier. (#11246) **What this PR does / why we need it**: Following https://github.com/grafana/loki/pull/11123 and in order to enable https://github.com/grafana/loki/pull/10417 the query frontend should send the serialized LogQL AST instead of the query string to the queriers. This enables the frontend to change the AST and inject expressions that are not expressible in LogQL. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](https://github.com/grafana/loki/commit/d10549e3ece02120974929894ee333d07755d213) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](https://github.com/grafana/loki/pull/10840/commits/0d4416a4b03739583349934b96f272fb4f685d15) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>
2 years ago
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum(count_over_time({app="foo"}[3m]))`),
},
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponse(`sum(count_over_time({app="foo"}[1m]))`, 1),
subQueryRequestResponse(`sum(count_over_time({app="foo"}[1m] offset 1m0s))`, 1),
subQueryRequestResponse(`sum(count_over_time({app="foo"}[1m] offset 2m0s))`, 1),
},
expected: expectedMergedResponse(1 + 1 + 1),
},
{
in: &LokiInstantRequest{
Query: `sum by (bar) (count_over_time({app="foo"}[3m]))`,
TimeTs: time.Unix(1, 0),
Path: "/loki/api/v1/query",
Send query plan to querier. (#11246) **What this PR does / why we need it**: Following https://github.com/grafana/loki/pull/11123 and in order to enable https://github.com/grafana/loki/pull/10417 the query frontend should send the serialized LogQL AST instead of the query string to the queriers. This enables the frontend to change the AST and inject expressions that are not expressible in LogQL. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](https://github.com/grafana/loki/commit/d10549e3ece02120974929894ee333d07755d213) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](https://github.com/grafana/loki/pull/10840/commits/0d4416a4b03739583349934b96f272fb4f685d15) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>
2 years ago
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum by (bar) (count_over_time({app="foo"}[3m]))`),
},
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponse(`sum by (bar)(count_over_time({app="foo"}[1m]))`, 0),
subQueryRequestResponse(`sum by (bar)(count_over_time({app="foo"}[1m] offset 1m0s))`, 0),
subQueryRequestResponse(`sum by (bar)(count_over_time({app="foo"}[1m] offset 2m0s))`, 0),
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
},
expected: expectedMergedResponse(0 + 0 + 0),
},
{
in: &LokiInstantRequest{
Query: `sum(sum_over_time({app="foo"} | unwrap bar [3m]))`,
TimeTs: time.Unix(1, 0),
Path: "/loki/api/v1/query",
Send query plan to querier. (#11246) **What this PR does / why we need it**: Following https://github.com/grafana/loki/pull/11123 and in order to enable https://github.com/grafana/loki/pull/10417 the query frontend should send the serialized LogQL AST instead of the query string to the queriers. This enables the frontend to change the AST and inject expressions that are not expressible in LogQL. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](https://github.com/grafana/loki/commit/d10549e3ece02120974929894ee333d07755d213) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](https://github.com/grafana/loki/pull/10840/commits/0d4416a4b03739583349934b96f272fb4f685d15) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>
2 years ago
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum(sum_over_time({app="foo"} | unwrap bar [3m]))`),
},
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponse(`sum(sum_over_time({app="foo"} | unwrap bar[1m]))`, 1),
subQueryRequestResponse(`sum(sum_over_time({app="foo"} | unwrap bar[1m] offset 1m0s))`, 2),
subQueryRequestResponse(`sum(sum_over_time({app="foo"} | unwrap bar[1m] offset 2m0s))`, 3),
},
expected: expectedMergedResponse(1 + 2 + 3),
},
{
in: &LokiInstantRequest{
Query: `sum by (bar) (sum_over_time({app="foo"} | unwrap bar [3m]))`,
TimeTs: time.Unix(1, 0),
Path: "/loki/api/v1/query",
Send query plan to querier. (#11246) **What this PR does / why we need it**: Following https://github.com/grafana/loki/pull/11123 and in order to enable https://github.com/grafana/loki/pull/10417 the query frontend should send the serialized LogQL AST instead of the query string to the queriers. This enables the frontend to change the AST and inject expressions that are not expressible in LogQL. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](https://github.com/grafana/loki/commit/d10549e3ece02120974929894ee333d07755d213) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](https://github.com/grafana/loki/pull/10840/commits/0d4416a4b03739583349934b96f272fb4f685d15) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>
2 years ago
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(`sum by (bar) (sum_over_time({app="foo"} | unwrap bar [3m]))`),
},
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
},
subQueries: []queryrangebase.RequestResponse{
subQueryRequestResponse(`sum by (bar)(sum_over_time({app="foo"} | unwrap bar[1m]))`, 1),
subQueryRequestResponse(`sum by (bar)(sum_over_time({app="foo"} | unwrap bar[1m] offset 1m0s))`, 2),
subQueryRequestResponse(`sum by (bar)(sum_over_time({app="foo"} | unwrap bar[1m] offset 2m0s))`, 3),
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
},
expected: expectedMergedResponse(1 + 2 + 3),
},
} {
tc := tc
t.Run(tc.in.GetQuery(), func(t *testing.T) {
resp, err := srm.Wrap(queryrangebase.HandlerFunc(
func(ctx context.Context, req queryrangebase.Request) (queryrangebase.Response, error) {
// Assert subquery request
for _, reqResp := range tc.subQueries {
if req.GetQuery() == reqResp.Request.GetQuery() {
require.Equal(t, reqResp.Request, req)
// return the test data subquery response
return reqResp.Response, nil
}
}
return nil, fmt.Errorf("subquery request '" + req.GetQuery() + "' not found")
})).Do(ctx, tc.in)
require.NoError(t, err)
require.Equal(t, tc.expected, resp.(*LokiPromResponse).Response)
})
}
}
// subQueryRequestResponse returns a RequestResponse containing the expected subQuery instant request
// and a response containing a sample value returned from the following wrapper
func subQueryRequestResponseWithQueryTime(expectedSubQuery string, sampleValue float64, exec time.Time) queryrangebase.RequestResponse {
return queryrangebase.RequestResponse{
Request: &LokiInstantRequest{
Query: expectedSubQuery,
TimeTs: exec,
Path: "/loki/api/v1/query",
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(expectedSubQuery),
},
},
Response: &LokiPromResponse{
Response: &queryrangebase.PrometheusResponse{
Status: loghttp.QueryStatusSuccess,
Data: queryrangebase.PrometheusData{
ResultType: loghttp.ResultTypeVector,
Result: []queryrangebase.SampleStream{
{
Labels: []logproto.LabelAdapter{
{Name: "app", Value: "foo"},
},
Samples: []logproto.LegacySample{
{TimestampMs: 1000, Value: sampleValue},
},
},
},
},
},
},
}
}
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
// subQueryRequestResponse returns a RequestResponse containing the expected subQuery instant request
// and a response containing a sample value returned from the following wrapper
func subQueryRequestResponse(expectedSubQuery string, sampleValue float64) queryrangebase.RequestResponse {
return queryrangebase.RequestResponse{
Request: &LokiInstantRequest{
Query: expectedSubQuery,
TimeTs: time.Unix(1, 0),
Path: "/loki/api/v1/query",
Send query plan to querier. (#11246) **What this PR does / why we need it**: Following https://github.com/grafana/loki/pull/11123 and in order to enable https://github.com/grafana/loki/pull/10417 the query frontend should send the serialized LogQL AST instead of the query string to the queriers. This enables the frontend to change the AST and inject expressions that are not expressible in LogQL. **Checklist** - [ ] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [x] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](https://github.com/grafana/loki/commit/d10549e3ece02120974929894ee333d07755d213) - [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. [Example PR](https://github.com/grafana/loki/pull/10840/commits/0d4416a4b03739583349934b96f272fb4f685d15) --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Callum Styan <callumstyan@gmail.com>
2 years ago
Plan: &plan.QueryPlan{
AST: syntax.MustParseExpr(expectedSubQuery),
},
Split by range of instant queries (#5662) * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
4 years ago
},
Response: &LokiPromResponse{
Response: &queryrangebase.PrometheusResponse{
Status: loghttp.QueryStatusSuccess,
Data: queryrangebase.PrometheusData{
ResultType: loghttp.ResultTypeVector,
Result: []queryrangebase.SampleStream{
{
Labels: []logproto.LabelAdapter{
{Name: "app", Value: "foo"},
},
Samples: []logproto.LegacySample{
{TimestampMs: 1000, Value: sampleValue},
},
},
},
},
},
},
}
}
// expectedMergedResponse returns the expected middleware Prometheus response with the samples
// as the expectedSampleValue
func expectedMergedResponse(expectedSampleValue float64) *queryrangebase.PrometheusResponse {
return &queryrangebase.PrometheusResponse{
Status: loghttp.QueryStatusSuccess,
Data: queryrangebase.PrometheusData{
ResultType: loghttp.ResultTypeVector,
Result: []queryrangebase.SampleStream{
{
Labels: []logproto.LabelAdapter{},
Samples: []logproto.LegacySample{
{TimestampMs: 1000, Value: expectedSampleValue},
},
},
},
},
}
}
func expectedMergedResponseWithTime(expectedSampleValue float64, exec time.Time) *queryrangebase.PrometheusResponse {
return &queryrangebase.PrometheusResponse{
Status: loghttp.QueryStatusSuccess,
Data: queryrangebase.PrometheusData{
ResultType: loghttp.ResultTypeVector,
Result: []queryrangebase.SampleStream{
{
Labels: []logproto.LabelAdapter{},
Samples: []logproto.LegacySample{
{TimestampMs: exec.UnixMilli(), Value: expectedSampleValue},
},
},
},
},
}
}