The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
grafana/pkg/tsdb/influxdb/influxql/parser_bench_test.go

59 lines
2.2 KiB

InfluxDB: Implement InfluxQL json streaming parser (#76934) * Have the first iteration * Prepare bench testing * rename the test files * Remove unnecessary test file * Introduce influxqlStreamingParser feature flag * Apply streaming parser feature flag * Add new tests * More tests * return executedQueryString only in first frame * add frame meta and config * Update golden json files * Support tags/labels * more tests * more tests * Don't change original response_parser.go * provide context * create util package * don't pass the row * update converter with formatted frameName * add executedQueryString info only to first frame * update golden files * rename * update test file * use pointer values * update testdata * update parsing * update converter for null values * prepare converter for table response * clean up * return timeField in fields * handle no time column responses * better nil field handling * refactor the code * add table tests * fix config for table * table response format * fix value * if there is no time column set name * linting * refactoring * handle the status code * add tracing * Update pkg/tsdb/influxdb/influxql/converter/converter_test.go Co-authored-by: İnanç Gümüş <m@inanc.io> * fix import * update test data * sanity * sanity * linting * simplicity * return empty rsp * rename to prevent confusion * nullableJson field type for null values * better handling null values * remove duplicate test file * fix healthcheck * use util for pointer * move bench test to root * provide fake feature manager * add more tests * partial fix for null values in table response format * handle partial null fields * comments for easy testing * move frameName allocation in readSeries * one less append operation * performance improvement by making string conversion once pkg: github.com/grafana/grafana/pkg/tsdb/influxdb/influxql │ stream2.txt │ stream3.txt │ │ sec/op │ sec/op vs base │ ParseJson-10 314.4m ± 1% 303.9m ± 1% -3.34% (p=0.000 n=10) │ stream2.txt │ stream3.txt │ │ B/op │ B/op vs base │ ParseJson-10 425.2Mi ± 0% 382.7Mi ± 0% -10.00% (p=0.000 n=10) │ stream2.txt │ stream3.txt │ │ allocs/op │ allocs/op vs base │ ParseJson-10 7.224M ± 0% 6.689M ± 0% -7.41% (p=0.000 n=10) * add comment lines --------- Co-authored-by: İnanç Gümüş <m@inanc.io>
2 years ago
package influxql
import (
_ "embed"
"fmt"
"io"
"os"
"strings"
"testing"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/stretchr/testify/require"
"github.com/grafana/grafana/pkg/tsdb/influxdb/influxql/buffered"
"github.com/grafana/grafana/pkg/tsdb/influxdb/influxql/querydata"
"github.com/grafana/grafana/pkg/tsdb/influxdb/models"
)
// TEST_MODE=buffered RES_FORMAT=time_series go test -benchmem -run=^$ -memprofile buffered_mem.out -count=10 -bench ^BenchmarkParseJson github.com/grafana/grafana/pkg/tsdb/influxdb/influxql | tee buffered.txt
// TEST_MODE=stream RES_FORMAT=time_series go test -benchmem -run=^$ -memprofile stream_mem.out -count=10 -bench ^BenchmarkParseJson github.com/grafana/grafana/pkg/tsdb/influxdb/influxql | tee stream.txt
// TEST_MODE=buffered RES_FORMAT=table go test -benchmem -run=^$ -memprofile buffered_table_mem.out -cpuprofile buffered_table_cpu.out -count=10 -bench ^BenchmarkParseJson github.com/grafana/grafana/pkg/tsdb/influxdb/influxql | tee buffered_table.txt
// TEST_MODE=stream RES_FORMAT=table go test -benchmem -run=^$ -memprofile stream_table_mem.out -cpuprofile stream_table_cpu.out -count=10 -bench ^BenchmarkParseJson github.com/grafana/grafana/pkg/tsdb/influxdb/influxql | tee stream_table.txt
InfluxDB: Implement InfluxQL json streaming parser (#76934) * Have the first iteration * Prepare bench testing * rename the test files * Remove unnecessary test file * Introduce influxqlStreamingParser feature flag * Apply streaming parser feature flag * Add new tests * More tests * return executedQueryString only in first frame * add frame meta and config * Update golden json files * Support tags/labels * more tests * more tests * Don't change original response_parser.go * provide context * create util package * don't pass the row * update converter with formatted frameName * add executedQueryString info only to first frame * update golden files * rename * update test file * use pointer values * update testdata * update parsing * update converter for null values * prepare converter for table response * clean up * return timeField in fields * handle no time column responses * better nil field handling * refactor the code * add table tests * fix config for table * table response format * fix value * if there is no time column set name * linting * refactoring * handle the status code * add tracing * Update pkg/tsdb/influxdb/influxql/converter/converter_test.go Co-authored-by: İnanç Gümüş <m@inanc.io> * fix import * update test data * sanity * sanity * linting * simplicity * return empty rsp * rename to prevent confusion * nullableJson field type for null values * better handling null values * remove duplicate test file * fix healthcheck * use util for pointer * move bench test to root * provide fake feature manager * add more tests * partial fix for null values in table response format * handle partial null fields * comments for easy testing * move frameName allocation in readSeries * one less append operation * performance improvement by making string conversion once pkg: github.com/grafana/grafana/pkg/tsdb/influxdb/influxql │ stream2.txt │ stream3.txt │ │ sec/op │ sec/op vs base │ ParseJson-10 314.4m ± 1% 303.9m ± 1% -3.34% (p=0.000 n=10) │ stream2.txt │ stream3.txt │ │ B/op │ B/op vs base │ ParseJson-10 425.2Mi ± 0% 382.7Mi ± 0% -10.00% (p=0.000 n=10) │ stream2.txt │ stream3.txt │ │ allocs/op │ allocs/op vs base │ ParseJson-10 7.224M ± 0% 6.689M ± 0% -7.41% (p=0.000 n=10) * add comment lines --------- Co-authored-by: İnanç Gümüş <m@inanc.io>
2 years ago
// go tool pprof -http=localhost:9999 memprofile.out
// benchstat buffered.txt stream.txt
// benchstat buffered_table.txt stream_table.txt
InfluxDB: Implement InfluxQL json streaming parser (#76934) * Have the first iteration * Prepare bench testing * rename the test files * Remove unnecessary test file * Introduce influxqlStreamingParser feature flag * Apply streaming parser feature flag * Add new tests * More tests * return executedQueryString only in first frame * add frame meta and config * Update golden json files * Support tags/labels * more tests * more tests * Don't change original response_parser.go * provide context * create util package * don't pass the row * update converter with formatted frameName * add executedQueryString info only to first frame * update golden files * rename * update test file * use pointer values * update testdata * update parsing * update converter for null values * prepare converter for table response * clean up * return timeField in fields * handle no time column responses * better nil field handling * refactor the code * add table tests * fix config for table * table response format * fix value * if there is no time column set name * linting * refactoring * handle the status code * add tracing * Update pkg/tsdb/influxdb/influxql/converter/converter_test.go Co-authored-by: İnanç Gümüş <m@inanc.io> * fix import * update test data * sanity * sanity * linting * simplicity * return empty rsp * rename to prevent confusion * nullableJson field type for null values * better handling null values * remove duplicate test file * fix healthcheck * use util for pointer * move bench test to root * provide fake feature manager * add more tests * partial fix for null values in table response format * handle partial null fields * comments for easy testing * move frameName allocation in readSeries * one less append operation * performance improvement by making string conversion once pkg: github.com/grafana/grafana/pkg/tsdb/influxdb/influxql │ stream2.txt │ stream3.txt │ │ sec/op │ sec/op vs base │ ParseJson-10 314.4m ± 1% 303.9m ± 1% -3.34% (p=0.000 n=10) │ stream2.txt │ stream3.txt │ │ B/op │ B/op vs base │ ParseJson-10 425.2Mi ± 0% 382.7Mi ± 0% -10.00% (p=0.000 n=10) │ stream2.txt │ stream3.txt │ │ allocs/op │ allocs/op vs base │ ParseJson-10 7.224M ± 0% 6.689M ± 0% -7.41% (p=0.000 n=10) * add comment lines --------- Co-authored-by: İnanç Gümüş <m@inanc.io>
2 years ago
func BenchmarkParseJson(b *testing.B) {
filePath := "testdata/many_columns.json"
bytes, err := os.ReadFile(filePath)
if err != nil {
panic(fmt.Sprintf("cannot read the file: %s", filePath))
}
testMode := os.Getenv("TEST_MODE")
resFormat := os.Getenv("RES_FORMAT")
InfluxDB: Implement InfluxQL json streaming parser (#76934) * Have the first iteration * Prepare bench testing * rename the test files * Remove unnecessary test file * Introduce influxqlStreamingParser feature flag * Apply streaming parser feature flag * Add new tests * More tests * return executedQueryString only in first frame * add frame meta and config * Update golden json files * Support tags/labels * more tests * more tests * Don't change original response_parser.go * provide context * create util package * don't pass the row * update converter with formatted frameName * add executedQueryString info only to first frame * update golden files * rename * update test file * use pointer values * update testdata * update parsing * update converter for null values * prepare converter for table response * clean up * return timeField in fields * handle no time column responses * better nil field handling * refactor the code * add table tests * fix config for table * table response format * fix value * if there is no time column set name * linting * refactoring * handle the status code * add tracing * Update pkg/tsdb/influxdb/influxql/converter/converter_test.go Co-authored-by: İnanç Gümüş <m@inanc.io> * fix import * update test data * sanity * sanity * linting * simplicity * return empty rsp * rename to prevent confusion * nullableJson field type for null values * better handling null values * remove duplicate test file * fix healthcheck * use util for pointer * move bench test to root * provide fake feature manager * add more tests * partial fix for null values in table response format * handle partial null fields * comments for easy testing * move frameName allocation in readSeries * one less append operation * performance improvement by making string conversion once pkg: github.com/grafana/grafana/pkg/tsdb/influxdb/influxql │ stream2.txt │ stream3.txt │ │ sec/op │ sec/op vs base │ ParseJson-10 314.4m ± 1% 303.9m ± 1% -3.34% (p=0.000 n=10) │ stream2.txt │ stream3.txt │ │ B/op │ B/op vs base │ ParseJson-10 425.2Mi ± 0% 382.7Mi ± 0% -10.00% (p=0.000 n=10) │ stream2.txt │ stream3.txt │ │ allocs/op │ allocs/op vs base │ ParseJson-10 7.224M ± 0% 6.689M ± 0% -7.41% (p=0.000 n=10) * add comment lines --------- Co-authored-by: İnanç Gümüş <m@inanc.io>
2 years ago
if testMode == "" {
testMode = "stream"
}
query := &models.Query{
RawQuery: "Test raw query",
UseRawQuery: true,
ResultFormat: resFormat,
InfluxDB: Implement InfluxQL json streaming parser (#76934) * Have the first iteration * Prepare bench testing * rename the test files * Remove unnecessary test file * Introduce influxqlStreamingParser feature flag * Apply streaming parser feature flag * Add new tests * More tests * return executedQueryString only in first frame * add frame meta and config * Update golden json files * Support tags/labels * more tests * more tests * Don't change original response_parser.go * provide context * create util package * don't pass the row * update converter with formatted frameName * add executedQueryString info only to first frame * update golden files * rename * update test file * use pointer values * update testdata * update parsing * update converter for null values * prepare converter for table response * clean up * return timeField in fields * handle no time column responses * better nil field handling * refactor the code * add table tests * fix config for table * table response format * fix value * if there is no time column set name * linting * refactoring * handle the status code * add tracing * Update pkg/tsdb/influxdb/influxql/converter/converter_test.go Co-authored-by: İnanç Gümüş <m@inanc.io> * fix import * update test data * sanity * sanity * linting * simplicity * return empty rsp * rename to prevent confusion * nullableJson field type for null values * better handling null values * remove duplicate test file * fix healthcheck * use util for pointer * move bench test to root * provide fake feature manager * add more tests * partial fix for null values in table response format * handle partial null fields * comments for easy testing * move frameName allocation in readSeries * one less append operation * performance improvement by making string conversion once pkg: github.com/grafana/grafana/pkg/tsdb/influxdb/influxql │ stream2.txt │ stream3.txt │ │ sec/op │ sec/op vs base │ ParseJson-10 314.4m ± 1% 303.9m ± 1% -3.34% (p=0.000 n=10) │ stream2.txt │ stream3.txt │ │ B/op │ B/op vs base │ ParseJson-10 425.2Mi ± 0% 382.7Mi ± 0% -10.00% (p=0.000 n=10) │ stream2.txt │ stream3.txt │ │ allocs/op │ allocs/op vs base │ ParseJson-10 7.224M ± 0% 6.689M ± 0% -7.41% (p=0.000 n=10) * add comment lines --------- Co-authored-by: İnanç Gümüş <m@inanc.io>
2 years ago
}
b.ResetTimer()
for n := 0; n < b.N; n++ {
buf := io.NopCloser(strings.NewReader(string(bytes)))
var result *backend.DataResponse
switch testMode {
case "buffered":
result = buffered.ResponseParse(buf, 200, query)
case "stream":
result = querydata.ResponseParse(buf, 200, query)
}
require.NotNil(b, result.Frames)
require.NoError(b, result.Error)
}
}