The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
grafana/public/app/plugins/datasource/loki/metricTimeSplitting.ts

53 lines
2.0 KiB

Loki Query Splitting: Split queries into sub-queries with smaller time interval (#62767) * Range splitting: range splitting function * Range splitting: experiment with 1 hour splits * Range splitting: reorganize code * Range splitting: improve code readability and meaning * Range splitting: add partition limit to prevent infinite loops * Range splitting: add error handling * Range splitting: disable for logs queries * Range splitting: support any arbitrary time splitting + respect original from/to in the partition * Chore: remove console logs * Chore: delete unused import * Range splitting: actually send requests in sequence * Range splitting: do not split when > 1 query * Range splitting: combine frames * Chore: rename function * split in reverse * polished reversing * keep reference to the right frame in the result * Range splitting: change request state to Streaming * Range splitting: fix moving only 1 unit of time instead of the provided one * Chore: change default parameter to timeShift = 1 * Range splitting: do not split for range queqries * Range splitting: add initial support for log queries * Range splitting: do not use MutableDataFrame It has bad performance and it's not required * Chore: remove unused export * Query Splitting: move to module * loki: split: fix off-by-one error (#62966) loki: split: fix off-by-one loop * Range splitting: disable for logs volume queries * Range splitting: combine any number of fields, not just hardcoded 2 * Range splitting: optimize frame-combining function * Range splitting: further optimize * Range splitting: combine frame length * Range splitting: combine stats * Range splitting: combine stats without assuming the same order * Query splitting: catch and raise errors * Range splitting: create feature flag * Range splitting: implement feature flag * Range splitting: add unit test for datasource query * Range splitting: add basic test for runPartitionedQuery * Range splitting: add unit test for resultLimitReached * Range splitting: test frame merging * Chore: fix unit test --------- Co-authored-by: Sven Grossmann <svennergr@gmail.com> Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>
2 years ago
// every timestamp in this file is a number which contains an unix-timestamp-in-millisecond format,
// like returned by `new Date().getTime()`. this is needed because the "math"
// has to be done on integer numbers.
// we are trying to be compatible with
// https://github.com/grafana/loki/blob/089ec1b05f5ec15a8851d0e8230153e0eeb4dcec/pkg/querier/queryrange/split_by_interval.go#L327-L336
function expandTimeRange(startTime: number, endTime: number, step: number): [number, number] {
// startTime is decreased to the closes multiple-of-step, if necessary
const newStartTime = startTime - (startTime % step);
// endTime is increased to the closed multiple-of-step, if necessary
let newEndTime = endTime;
const endStepMod = endTime % step;
if (endStepMod !== 0) {
newEndTime += step - endStepMod;
}
return [newStartTime, newEndTime];
}
export function splitTimeRange(
Loki Query Splitting: Split queries into sub-queries with smaller time interval (#62767) * Range splitting: range splitting function * Range splitting: experiment with 1 hour splits * Range splitting: reorganize code * Range splitting: improve code readability and meaning * Range splitting: add partition limit to prevent infinite loops * Range splitting: add error handling * Range splitting: disable for logs queries * Range splitting: support any arbitrary time splitting + respect original from/to in the partition * Chore: remove console logs * Chore: delete unused import * Range splitting: actually send requests in sequence * Range splitting: do not split when > 1 query * Range splitting: combine frames * Chore: rename function * split in reverse * polished reversing * keep reference to the right frame in the result * Range splitting: change request state to Streaming * Range splitting: fix moving only 1 unit of time instead of the provided one * Chore: change default parameter to timeShift = 1 * Range splitting: do not split for range queqries * Range splitting: add initial support for log queries * Range splitting: do not use MutableDataFrame It has bad performance and it's not required * Chore: remove unused export * Query Splitting: move to module * loki: split: fix off-by-one error (#62966) loki: split: fix off-by-one loop * Range splitting: disable for logs volume queries * Range splitting: combine any number of fields, not just hardcoded 2 * Range splitting: optimize frame-combining function * Range splitting: further optimize * Range splitting: combine frame length * Range splitting: combine stats * Range splitting: combine stats without assuming the same order * Query splitting: catch and raise errors * Range splitting: create feature flag * Range splitting: implement feature flag * Range splitting: add unit test for datasource query * Range splitting: add basic test for runPartitionedQuery * Range splitting: add unit test for resultLimitReached * Range splitting: test frame merging * Chore: fix unit test --------- Co-authored-by: Sven Grossmann <svennergr@gmail.com> Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>
2 years ago
startTime: number,
endTime: number,
step: number,
idealRangeDuration: number
): Array<[number, number]> {
Loki Query Splitting: Split queries into sub-queries with smaller time interval (#62767) * Range splitting: range splitting function * Range splitting: experiment with 1 hour splits * Range splitting: reorganize code * Range splitting: improve code readability and meaning * Range splitting: add partition limit to prevent infinite loops * Range splitting: add error handling * Range splitting: disable for logs queries * Range splitting: support any arbitrary time splitting + respect original from/to in the partition * Chore: remove console logs * Chore: delete unused import * Range splitting: actually send requests in sequence * Range splitting: do not split when > 1 query * Range splitting: combine frames * Chore: rename function * split in reverse * polished reversing * keep reference to the right frame in the result * Range splitting: change request state to Streaming * Range splitting: fix moving only 1 unit of time instead of the provided one * Chore: change default parameter to timeShift = 1 * Range splitting: do not split for range queqries * Range splitting: add initial support for log queries * Range splitting: do not use MutableDataFrame It has bad performance and it's not required * Chore: remove unused export * Query Splitting: move to module * loki: split: fix off-by-one error (#62966) loki: split: fix off-by-one loop * Range splitting: disable for logs volume queries * Range splitting: combine any number of fields, not just hardcoded 2 * Range splitting: optimize frame-combining function * Range splitting: further optimize * Range splitting: combine frame length * Range splitting: combine stats * Range splitting: combine stats without assuming the same order * Query splitting: catch and raise errors * Range splitting: create feature flag * Range splitting: implement feature flag * Range splitting: add unit test for datasource query * Range splitting: add basic test for runPartitionedQuery * Range splitting: add unit test for resultLimitReached * Range splitting: test frame merging * Chore: fix unit test --------- Co-authored-by: Sven Grossmann <svennergr@gmail.com> Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>
2 years ago
if (idealRangeDuration < step) {
// we cannot create chunks smaller than `step`
return [[startTime, endTime]];
Loki Query Splitting: Split queries into sub-queries with smaller time interval (#62767) * Range splitting: range splitting function * Range splitting: experiment with 1 hour splits * Range splitting: reorganize code * Range splitting: improve code readability and meaning * Range splitting: add partition limit to prevent infinite loops * Range splitting: add error handling * Range splitting: disable for logs queries * Range splitting: support any arbitrary time splitting + respect original from/to in the partition * Chore: remove console logs * Chore: delete unused import * Range splitting: actually send requests in sequence * Range splitting: do not split when > 1 query * Range splitting: combine frames * Chore: rename function * split in reverse * polished reversing * keep reference to the right frame in the result * Range splitting: change request state to Streaming * Range splitting: fix moving only 1 unit of time instead of the provided one * Chore: change default parameter to timeShift = 1 * Range splitting: do not split for range queqries * Range splitting: add initial support for log queries * Range splitting: do not use MutableDataFrame It has bad performance and it's not required * Chore: remove unused export * Query Splitting: move to module * loki: split: fix off-by-one error (#62966) loki: split: fix off-by-one loop * Range splitting: disable for logs volume queries * Range splitting: combine any number of fields, not just hardcoded 2 * Range splitting: optimize frame-combining function * Range splitting: further optimize * Range splitting: combine frame length * Range splitting: combine stats * Range splitting: combine stats without assuming the same order * Query splitting: catch and raise errors * Range splitting: create feature flag * Range splitting: implement feature flag * Range splitting: add unit test for datasource query * Range splitting: add basic test for runPartitionedQuery * Range splitting: add unit test for resultLimitReached * Range splitting: test frame merging * Chore: fix unit test --------- Co-authored-by: Sven Grossmann <svennergr@gmail.com> Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>
2 years ago
}
// we make the duration a multiple of `step`, lowering it if necessary
const alignedDuration = Math.trunc(idealRangeDuration / step) * step;
const [alignedStartTime, alignedEndTime] = expandTimeRange(startTime, endTime, step);
const result: Array<[number, number]> = [];
// we iterate it from the end, because we want to have the potentially smaller chunk at the end, not at the beginning
for (let chunkEndTime = alignedEndTime; chunkEndTime > alignedStartTime; chunkEndTime -= alignedDuration + step) {
// when we get close to the start of the time range, we need to be sure not
// to cross over the startTime
const chunkStartTime = Math.max(chunkEndTime - alignedDuration, alignedStartTime);
result.push([chunkStartTime, chunkEndTime]);
}
// because we walked backwards, we need to reverse the array
result.reverse();
return result;
}