The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
grafana/public/app/plugins/datasource/loki/datasource.ts

783 lines
26 KiB

// Libraries
import { cloneDeep, isEmpty, map as lodashMap } from 'lodash';
import { lastValueFrom, merge, Observable, of, throwError } from 'rxjs';
import { catchError, map, switchMap } from 'rxjs/operators';
import Prism from 'prismjs';
// Types
import {
AnnotationEvent,
AnnotationQueryRequest,
DataFrame,
DataFrameView,
DataQuery,
DataQueryError,
DataQueryRequest,
DataQueryResponse,
DataSourceApi,
DataSourceInstanceSettings,
DataSourceWithLogsContextSupport,
Loki: Full range logs volume (#39327) * Basic implementation of getLogsVolumeQuery method * Add todos * Add a switcher to automatically load logs volume * De-scope dismissing logs volume panel * De-scope logs volume query cancellation * Remove todo * Aggregate logs volume components in single panel * Show logs volume only when it's available * Aggregate logs volume by level * Simplify aggregation * Handle no logs volume data * Add error handling * Do not show auto-load logs volume switcher when loading logs volume is not available * Remove old logs volume graph * Clean up * Make getting data provider more generic * Provide complete logs volume data (error, isLoading) * Display more specific error message * Add missing props to mocks * Remove setRequest method * Mark getQueryRelatedDataProviders as internal * Add missing dataQueryRequest and add a todo * Remove redundant loading state * Do not mutate existing queries * Apply fix for zooming-in from main * Post-merge fixes * Create collection for data provider results * Use more generic names * Move aggregation logic to Loki logs volume provider * Move LogsVolume to common types * Update tests * Post-merge fixes * Fix mapping related data values * Simplify prop mappings * Add docs * Fix property name * Clean-up * Mark new types as internal * Reduce number of providers to logs volume only * Simplify data structure to DataQueryResponse * Move Logs Volume panel to a separate component * Test logsVolumeProvider.ts * Add observable version of datasource mock * Test getLogsVolumeDataProvider method * Test LogsVolumePanel * Test logs volume reducer * Clean up * Clean up * Fix test * Use sum by to use level field directly * Fix strict type errors * Fix strict type errors * Use "logs" instead of "unknown" if only one level was detected * Add docs about logs volume * Rename histogramRequest to logsVolumeRequest * Use LogsVolumeContentWrapper all content types * Move `autoLoadLogsVolume` local storage handling * Fix strict error * Move getting autoLoadLogsVolume to initial state * Cancel current logs volume subscription * Test cancelling subscriptions * Update docs/sources/datasources/loki.md Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Update packages/grafana-data/src/types/explore.ts Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Inline container styles * Ensure logs volume is aggregated per each subscription separately * Simplify logs volume provider * Type-guard support for logs volume provider * Simplify event handlers to avoid casting * Clean up and docs * Move auto-load switcher to logs volume panel * Fix test * Move DataSourceWithLogsVolumeSupport to avoid cross referencing * Simplify interface * Bring back old histogram and hide the new one behind a feature flag * Add missing props to logs histogram panel * Clean up the provider when it's not supported * Simplify storing autoLoadLogsVolume * Remove docs * Update packages/grafana-data/src/types/logsVolume.ts Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com> * Skip dataframes without fields (instant queries) * Revert styles changes * Revert styles changes * Add release tag Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com>
4 years ago
DataSourceWithLogsVolumeSupport,
dateMath,
DateTime,
FieldCache,
FieldType,
getLogLevelFromKey,
Labels,
LoadingState,
LogLevel,
LogRowModel,
QueryResultMeta,
ScopedVars,
TimeRange,
} from '@grafana/data';
import { BackendSrvRequest, FetchError, getBackendSrv } from '@grafana/runtime';
import { getTemplateSrv, TemplateSrv } from 'app/features/templating/template_srv';
import { addLabelToQuery } from 'app/plugins/datasource/prometheus/add_label_to_query';
import { getTimeSrv, TimeSrv } from 'app/features/dashboard/services/TimeSrv';
import { convertToWebSocketUrl } from 'app/core/utils/explore';
import {
lokiResultsToTableModel,
lokiStreamResultToDataFrame,
lokiStreamsToDataFrames,
processRangeQueryResponse,
} from './result_transformer';
import { addParsedLabelToQuery, queryHasPipeParser } from './query_utils';
import {
LokiOptions,
LokiQuery,
LokiRangeQueryRequest,
LokiResultType,
LokiStreamResponse,
LokiStreamResult,
} from './types';
import { LiveStreams, LokiLiveTarget } from './live_streams';
import LanguageProvider from './language_provider';
import { serializeParams } from '../../../core/utils/fetch';
import { RowContextOptions } from '@grafana/ui/src/components/Logs/LogRowContextProvider';
import syntax from './syntax';
import { DEFAULT_RESOLUTION } from './components/LokiOptionFields';
import { queryLogsVolume } from 'app/core/logs_model';
import config from 'app/core/config';
export type RangeQueryOptions = DataQueryRequest<LokiQuery> | AnnotationQueryRequest<LokiQuery>;
6 years ago
export const DEFAULT_MAX_LINES = 1000;
export const LOKI_ENDPOINT = '/loki/api/v1';
const NS_IN_MS = 1000000;
/**
* Loki's logs volume query may be expensive as it requires counting all logs in the selected range. If such query
* takes too much time it may need be made more specific to limit number of logs processed under the hood.
*/
const LOGS_VOLUME_TIMEOUT = 10000;
const RANGE_QUERY_ENDPOINT = `${LOKI_ENDPOINT}/query_range`;
const INSTANT_QUERY_ENDPOINT = `${LOKI_ENDPOINT}/query`;
const DEFAULT_QUERY_PARAMS: Partial<LokiRangeQueryRequest> = {
direction: 'BACKWARD',
6 years ago
limit: DEFAULT_MAX_LINES,
query: '',
};
Loki: Full range logs volume (#39327) * Basic implementation of getLogsVolumeQuery method * Add todos * Add a switcher to automatically load logs volume * De-scope dismissing logs volume panel * De-scope logs volume query cancellation * Remove todo * Aggregate logs volume components in single panel * Show logs volume only when it's available * Aggregate logs volume by level * Simplify aggregation * Handle no logs volume data * Add error handling * Do not show auto-load logs volume switcher when loading logs volume is not available * Remove old logs volume graph * Clean up * Make getting data provider more generic * Provide complete logs volume data (error, isLoading) * Display more specific error message * Add missing props to mocks * Remove setRequest method * Mark getQueryRelatedDataProviders as internal * Add missing dataQueryRequest and add a todo * Remove redundant loading state * Do not mutate existing queries * Apply fix for zooming-in from main * Post-merge fixes * Create collection for data provider results * Use more generic names * Move aggregation logic to Loki logs volume provider * Move LogsVolume to common types * Update tests * Post-merge fixes * Fix mapping related data values * Simplify prop mappings * Add docs * Fix property name * Clean-up * Mark new types as internal * Reduce number of providers to logs volume only * Simplify data structure to DataQueryResponse * Move Logs Volume panel to a separate component * Test logsVolumeProvider.ts * Add observable version of datasource mock * Test getLogsVolumeDataProvider method * Test LogsVolumePanel * Test logs volume reducer * Clean up * Clean up * Fix test * Use sum by to use level field directly * Fix strict type errors * Fix strict type errors * Use "logs" instead of "unknown" if only one level was detected * Add docs about logs volume * Rename histogramRequest to logsVolumeRequest * Use LogsVolumeContentWrapper all content types * Move `autoLoadLogsVolume` local storage handling * Fix strict error * Move getting autoLoadLogsVolume to initial state * Cancel current logs volume subscription * Test cancelling subscriptions * Update docs/sources/datasources/loki.md Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Update packages/grafana-data/src/types/explore.ts Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Inline container styles * Ensure logs volume is aggregated per each subscription separately * Simplify logs volume provider * Type-guard support for logs volume provider * Simplify event handlers to avoid casting * Clean up and docs * Move auto-load switcher to logs volume panel * Fix test * Move DataSourceWithLogsVolumeSupport to avoid cross referencing * Simplify interface * Bring back old histogram and hide the new one behind a feature flag * Add missing props to logs histogram panel * Clean up the provider when it's not supported * Simplify storing autoLoadLogsVolume * Remove docs * Update packages/grafana-data/src/types/logsVolume.ts Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com> * Skip dataframes without fields (instant queries) * Revert styles changes * Revert styles changes * Add release tag Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com>
4 years ago
export class LokiDatasource
extends DataSourceApi<LokiQuery, LokiOptions>
implements DataSourceWithLogsContextSupport, DataSourceWithLogsVolumeSupport<LokiQuery> {
private streams = new LiveStreams();
languageProvider: LanguageProvider;
6 years ago
maxLines: number;
constructor(
private instanceSettings: DataSourceInstanceSettings<LokiOptions>,
private readonly templateSrv: TemplateSrv = getTemplateSrv(),
private readonly timeSrv: TimeSrv = getTimeSrv()
) {
super(instanceSettings);
this.languageProvider = new LanguageProvider(this);
const settingsData = instanceSettings.jsonData || {};
this.maxLines = parseInt(settingsData.maxLines ?? '0', 10) || DEFAULT_MAX_LINES;
}
_request(apiUrl: string, data?: any, options?: Partial<BackendSrvRequest>): Observable<Record<string, any>> {
const baseUrl = this.instanceSettings.url;
const params = data ? serializeParams(data) : '';
const url = `${baseUrl}${apiUrl}${params.length ? `?${params}` : ''}`;
if (this.instanceSettings.withCredentials || this.instanceSettings.basicAuth) {
options = { ...options, withCredentials: true };
if (this.instanceSettings.basicAuth) {
options.headers = { ...options.headers, Authorization: this.instanceSettings.basicAuth };
}
}
const req = {
...options,
url,
};
return getBackendSrv().fetch<Record<string, any>>(req);
}
Loki: Full range logs volume (#39327) * Basic implementation of getLogsVolumeQuery method * Add todos * Add a switcher to automatically load logs volume * De-scope dismissing logs volume panel * De-scope logs volume query cancellation * Remove todo * Aggregate logs volume components in single panel * Show logs volume only when it's available * Aggregate logs volume by level * Simplify aggregation * Handle no logs volume data * Add error handling * Do not show auto-load logs volume switcher when loading logs volume is not available * Remove old logs volume graph * Clean up * Make getting data provider more generic * Provide complete logs volume data (error, isLoading) * Display more specific error message * Add missing props to mocks * Remove setRequest method * Mark getQueryRelatedDataProviders as internal * Add missing dataQueryRequest and add a todo * Remove redundant loading state * Do not mutate existing queries * Apply fix for zooming-in from main * Post-merge fixes * Create collection for data provider results * Use more generic names * Move aggregation logic to Loki logs volume provider * Move LogsVolume to common types * Update tests * Post-merge fixes * Fix mapping related data values * Simplify prop mappings * Add docs * Fix property name * Clean-up * Mark new types as internal * Reduce number of providers to logs volume only * Simplify data structure to DataQueryResponse * Move Logs Volume panel to a separate component * Test logsVolumeProvider.ts * Add observable version of datasource mock * Test getLogsVolumeDataProvider method * Test LogsVolumePanel * Test logs volume reducer * Clean up * Clean up * Fix test * Use sum by to use level field directly * Fix strict type errors * Fix strict type errors * Use "logs" instead of "unknown" if only one level was detected * Add docs about logs volume * Rename histogramRequest to logsVolumeRequest * Use LogsVolumeContentWrapper all content types * Move `autoLoadLogsVolume` local storage handling * Fix strict error * Move getting autoLoadLogsVolume to initial state * Cancel current logs volume subscription * Test cancelling subscriptions * Update docs/sources/datasources/loki.md Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Update packages/grafana-data/src/types/explore.ts Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Inline container styles * Ensure logs volume is aggregated per each subscription separately * Simplify logs volume provider * Type-guard support for logs volume provider * Simplify event handlers to avoid casting * Clean up and docs * Move auto-load switcher to logs volume panel * Fix test * Move DataSourceWithLogsVolumeSupport to avoid cross referencing * Simplify interface * Bring back old histogram and hide the new one behind a feature flag * Add missing props to logs histogram panel * Clean up the provider when it's not supported * Simplify storing autoLoadLogsVolume * Remove docs * Update packages/grafana-data/src/types/logsVolume.ts Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com> * Skip dataframes without fields (instant queries) * Revert styles changes * Revert styles changes * Add release tag Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com>
4 years ago
getLogsVolumeDataProvider(request: DataQueryRequest<LokiQuery>): Observable<DataQueryResponse> | undefined {
if (!config.featureToggles.fullRangeLogsVolume) {
return undefined;
}
Loki: Full range logs volume (#39327) * Basic implementation of getLogsVolumeQuery method * Add todos * Add a switcher to automatically load logs volume * De-scope dismissing logs volume panel * De-scope logs volume query cancellation * Remove todo * Aggregate logs volume components in single panel * Show logs volume only when it's available * Aggregate logs volume by level * Simplify aggregation * Handle no logs volume data * Add error handling * Do not show auto-load logs volume switcher when loading logs volume is not available * Remove old logs volume graph * Clean up * Make getting data provider more generic * Provide complete logs volume data (error, isLoading) * Display more specific error message * Add missing props to mocks * Remove setRequest method * Mark getQueryRelatedDataProviders as internal * Add missing dataQueryRequest and add a todo * Remove redundant loading state * Do not mutate existing queries * Apply fix for zooming-in from main * Post-merge fixes * Create collection for data provider results * Use more generic names * Move aggregation logic to Loki logs volume provider * Move LogsVolume to common types * Update tests * Post-merge fixes * Fix mapping related data values * Simplify prop mappings * Add docs * Fix property name * Clean-up * Mark new types as internal * Reduce number of providers to logs volume only * Simplify data structure to DataQueryResponse * Move Logs Volume panel to a separate component * Test logsVolumeProvider.ts * Add observable version of datasource mock * Test getLogsVolumeDataProvider method * Test LogsVolumePanel * Test logs volume reducer * Clean up * Clean up * Fix test * Use sum by to use level field directly * Fix strict type errors * Fix strict type errors * Use "logs" instead of "unknown" if only one level was detected * Add docs about logs volume * Rename histogramRequest to logsVolumeRequest * Use LogsVolumeContentWrapper all content types * Move `autoLoadLogsVolume` local storage handling * Fix strict error * Move getting autoLoadLogsVolume to initial state * Cancel current logs volume subscription * Test cancelling subscriptions * Update docs/sources/datasources/loki.md Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Update packages/grafana-data/src/types/explore.ts Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Inline container styles * Ensure logs volume is aggregated per each subscription separately * Simplify logs volume provider * Type-guard support for logs volume provider * Simplify event handlers to avoid casting * Clean up and docs * Move auto-load switcher to logs volume panel * Fix test * Move DataSourceWithLogsVolumeSupport to avoid cross referencing * Simplify interface * Bring back old histogram and hide the new one behind a feature flag * Add missing props to logs histogram panel * Clean up the provider when it's not supported * Simplify storing autoLoadLogsVolume * Remove docs * Update packages/grafana-data/src/types/logsVolume.ts Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com> * Skip dataframes without fields (instant queries) * Revert styles changes * Revert styles changes * Add release tag Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com>
4 years ago
const isLogsVolumeAvailable = request.targets.some((target) => target.expr && !isMetricsQuery(target.expr));
if (!isLogsVolumeAvailable) {
return undefined;
}
const logsVolumeRequest = cloneDeep(request);
logsVolumeRequest.targets = logsVolumeRequest.targets
.filter((target) => target.expr && !isMetricsQuery(target.expr))
.map((target) => {
return {
...target,
instant: false,
volumeQuery: true,
expr: `sum by (level) (count_over_time(${target.expr}[$__interval]))`,
};
});
return queryLogsVolume(this, logsVolumeRequest, {
timeout: LOGS_VOLUME_TIMEOUT,
extractLevel,
range: request.range,
targets: request.targets,
});
Loki: Full range logs volume (#39327) * Basic implementation of getLogsVolumeQuery method * Add todos * Add a switcher to automatically load logs volume * De-scope dismissing logs volume panel * De-scope logs volume query cancellation * Remove todo * Aggregate logs volume components in single panel * Show logs volume only when it's available * Aggregate logs volume by level * Simplify aggregation * Handle no logs volume data * Add error handling * Do not show auto-load logs volume switcher when loading logs volume is not available * Remove old logs volume graph * Clean up * Make getting data provider more generic * Provide complete logs volume data (error, isLoading) * Display more specific error message * Add missing props to mocks * Remove setRequest method * Mark getQueryRelatedDataProviders as internal * Add missing dataQueryRequest and add a todo * Remove redundant loading state * Do not mutate existing queries * Apply fix for zooming-in from main * Post-merge fixes * Create collection for data provider results * Use more generic names * Move aggregation logic to Loki logs volume provider * Move LogsVolume to common types * Update tests * Post-merge fixes * Fix mapping related data values * Simplify prop mappings * Add docs * Fix property name * Clean-up * Mark new types as internal * Reduce number of providers to logs volume only * Simplify data structure to DataQueryResponse * Move Logs Volume panel to a separate component * Test logsVolumeProvider.ts * Add observable version of datasource mock * Test getLogsVolumeDataProvider method * Test LogsVolumePanel * Test logs volume reducer * Clean up * Clean up * Fix test * Use sum by to use level field directly * Fix strict type errors * Fix strict type errors * Use "logs" instead of "unknown" if only one level was detected * Add docs about logs volume * Rename histogramRequest to logsVolumeRequest * Use LogsVolumeContentWrapper all content types * Move `autoLoadLogsVolume` local storage handling * Fix strict error * Move getting autoLoadLogsVolume to initial state * Cancel current logs volume subscription * Test cancelling subscriptions * Update docs/sources/datasources/loki.md Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Update packages/grafana-data/src/types/explore.ts Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Inline container styles * Ensure logs volume is aggregated per each subscription separately * Simplify logs volume provider * Type-guard support for logs volume provider * Simplify event handlers to avoid casting * Clean up and docs * Move auto-load switcher to logs volume panel * Fix test * Move DataSourceWithLogsVolumeSupport to avoid cross referencing * Simplify interface * Bring back old histogram and hide the new one behind a feature flag * Add missing props to logs histogram panel * Clean up the provider when it's not supported * Simplify storing autoLoadLogsVolume * Remove docs * Update packages/grafana-data/src/types/logsVolume.ts Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com> * Skip dataframes without fields (instant queries) * Revert styles changes * Revert styles changes * Add release tag Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com>
4 years ago
}
query(options: DataQueryRequest<LokiQuery>): Observable<DataQueryResponse> {
const subQueries: Array<Observable<DataQueryResponse>> = [];
const scopedVars = {
...options.scopedVars,
...this.getRangeScopedVars(options.range),
};
const filteredTargets = options.targets
.filter((target) => target.expr && !target.hide)
.map((target) => {
const expr = this.addAdHocFilters(target.expr);
return {
...target,
expr: this.templateSrv.replace(expr, scopedVars, this.interpolateQueryExpr),
};
});
for (const target of filteredTargets) {
if (target.instant) {
subQueries.push(this.runInstantQuery(target, options, filteredTargets.length));
} else {
subQueries.push(this.runRangeQuery(target, options, filteredTargets.length));
}
}
// No valid targets, return the empty result to save a round trip.
if (isEmpty(subQueries)) {
return of({
data: [],
state: LoadingState.Done,
});
}
return merge(...subQueries);
}
runInstantQuery = (
target: LokiQuery,
options: DataQueryRequest<LokiQuery>,
responseListLength = 1
): Observable<DataQueryResponse> => {
const timeNs = this.getTime(options.range.to, true);
const queryLimit = isMetricsQuery(target.expr) ? options.maxDataPoints : target.maxLines;
const query = {
query: target.expr,
time: `${timeNs + (1e9 - (timeNs % 1e9))}`,
limit: Math.min(queryLimit || Infinity, this.maxLines),
};
/** Used only for results of metrics instant queries */
const meta: QueryResultMeta = {
preferredVisualisationType: 'table',
};
return this._request(INSTANT_QUERY_ENDPOINT, query).pipe(
map((response) => {
if (response.data.data.resultType === LokiResultType.Stream) {
return {
data: response.data
? lokiStreamsToDataFrames(
response.data as LokiStreamResponse,
target,
query.limit,
this.instanceSettings.jsonData
)
: [],
key: `${target.refId}_instant`,
};
}
QueryProcessing: Observable query interface and RxJS for query & stream processing (#18899) * I needed to learn some rxjs and understand this more, so just playing around * Updated * Removed all the complete calls * Refactoring * StreamHandler -> observable start * progress * simple singal works * Handle update time range * added error handling * wrap old function * minor changes * handle data format in the subscribe function * Use replay subject to return last value to subscribers * Set loading state after no response in 50ms * added missing file * updated comment * Added cancelation of network requests * runRequest: Added unit test scenario framework * Progress on tests * minor refactor of unit tests * updated test * removed some old code * Shared queries work again, and also became so much simplier * unified query and observe methods * implict any fix * Fixed closed subject issue * removed comment * Use last returned data for loading state * WIP: Explore to runRequest makover step1 * Minor progress * Minor progress on explore and runRequest * minor progress * Things are starting to work in explore * Updated prometheus to use new observable query response, greatly simplified code * Revert refId change * Found better solution for key/refId/requestId problem * use observable with loki * tests compile * fix loki query prep * Explore: correct first response handling * Refactorings * Refactoring * Explore: Fixes LoadingState and GraphResults between runs (#18986) * Refactor: Adds state to DataQueryResponse * Fix: Fixes so we do not empty results before new data arrives Fixes: #17409 * Transformations work * observable test data * remove single() from loki promise * Fixed comment * Explore: Fixes failing Loki and Prometheus unit tests (#18995) * Tests: Makes datasource tests work again * Fix: Fixes loki datasource so highligthing works * Chore: Runs Prettier * Fixed query runner tests * Delay loading state indication to 200ms * Fixed test * fixed unit tests * Clear cached calcs * Fixed bug getProcesedDataFrames * Fix the correct test is a better idea * Fix: Fixes so queries in Explore are only run if Graph/Table is shown (#19000) * Fix: Fixes so queries in Explore are only run if Graph/Table is shown Fixes: #18618 * Refactor: Removes unnecessary condition * PanelData: provide legacy data only when needed (#19018) * no legacy * invert logic... now compiles * merge getQueryResponseData and getDataRaw * update comment about query editor * use single getData() function * only send legacy when it is used in explore * pre process rather than post process * pre process rather than post process * Minor refactoring * Add missing tags to test datasource response * MixedDatasource: Adds query observable pattern to MixedDatasource (#19037) * start mixed datasource * Refactor: Refactors into observable parttern * Tests: Fixes tests * Tests: Removes console.log * Refactor: Adds unique requestId
6 years ago
return {
data: [lokiResultsToTableModel(response.data.data.result, responseListLength, target.refId, meta, true)],
key: `${target.refId}_instant`,
QueryProcessing: Observable query interface and RxJS for query & stream processing (#18899) * I needed to learn some rxjs and understand this more, so just playing around * Updated * Removed all the complete calls * Refactoring * StreamHandler -> observable start * progress * simple singal works * Handle update time range * added error handling * wrap old function * minor changes * handle data format in the subscribe function * Use replay subject to return last value to subscribers * Set loading state after no response in 50ms * added missing file * updated comment * Added cancelation of network requests * runRequest: Added unit test scenario framework * Progress on tests * minor refactor of unit tests * updated test * removed some old code * Shared queries work again, and also became so much simplier * unified query and observe methods * implict any fix * Fixed closed subject issue * removed comment * Use last returned data for loading state * WIP: Explore to runRequest makover step1 * Minor progress * Minor progress on explore and runRequest * minor progress * Things are starting to work in explore * Updated prometheus to use new observable query response, greatly simplified code * Revert refId change * Found better solution for key/refId/requestId problem * use observable with loki * tests compile * fix loki query prep * Explore: correct first response handling * Refactorings * Refactoring * Explore: Fixes LoadingState and GraphResults between runs (#18986) * Refactor: Adds state to DataQueryResponse * Fix: Fixes so we do not empty results before new data arrives Fixes: #17409 * Transformations work * observable test data * remove single() from loki promise * Fixed comment * Explore: Fixes failing Loki and Prometheus unit tests (#18995) * Tests: Makes datasource tests work again * Fix: Fixes loki datasource so highligthing works * Chore: Runs Prettier * Fixed query runner tests * Delay loading state indication to 200ms * Fixed test * fixed unit tests * Clear cached calcs * Fixed bug getProcesedDataFrames * Fix the correct test is a better idea * Fix: Fixes so queries in Explore are only run if Graph/Table is shown (#19000) * Fix: Fixes so queries in Explore are only run if Graph/Table is shown Fixes: #18618 * Refactor: Removes unnecessary condition * PanelData: provide legacy data only when needed (#19018) * no legacy * invert logic... now compiles * merge getQueryResponseData and getDataRaw * update comment about query editor * use single getData() function * only send legacy when it is used in explore * pre process rather than post process * pre process rather than post process * Minor refactoring * Add missing tags to test datasource response * MixedDatasource: Adds query observable pattern to MixedDatasource (#19037) * start mixed datasource * Refactor: Refactors into observable parttern * Tests: Fixes tests * Tests: Removes console.log * Refactor: Adds unique requestId
6 years ago
};
}),
catchError((err) => throwError(() => this.processError(err, target)))
QueryProcessing: Observable query interface and RxJS for query & stream processing (#18899) * I needed to learn some rxjs and understand this more, so just playing around * Updated * Removed all the complete calls * Refactoring * StreamHandler -> observable start * progress * simple singal works * Handle update time range * added error handling * wrap old function * minor changes * handle data format in the subscribe function * Use replay subject to return last value to subscribers * Set loading state after no response in 50ms * added missing file * updated comment * Added cancelation of network requests * runRequest: Added unit test scenario framework * Progress on tests * minor refactor of unit tests * updated test * removed some old code * Shared queries work again, and also became so much simplier * unified query and observe methods * implict any fix * Fixed closed subject issue * removed comment * Use last returned data for loading state * WIP: Explore to runRequest makover step1 * Minor progress * Minor progress on explore and runRequest * minor progress * Things are starting to work in explore * Updated prometheus to use new observable query response, greatly simplified code * Revert refId change * Found better solution for key/refId/requestId problem * use observable with loki * tests compile * fix loki query prep * Explore: correct first response handling * Refactorings * Refactoring * Explore: Fixes LoadingState and GraphResults between runs (#18986) * Refactor: Adds state to DataQueryResponse * Fix: Fixes so we do not empty results before new data arrives Fixes: #17409 * Transformations work * observable test data * remove single() from loki promise * Fixed comment * Explore: Fixes failing Loki and Prometheus unit tests (#18995) * Tests: Makes datasource tests work again * Fix: Fixes loki datasource so highligthing works * Chore: Runs Prettier * Fixed query runner tests * Delay loading state indication to 200ms * Fixed test * fixed unit tests * Clear cached calcs * Fixed bug getProcesedDataFrames * Fix the correct test is a better idea * Fix: Fixes so queries in Explore are only run if Graph/Table is shown (#19000) * Fix: Fixes so queries in Explore are only run if Graph/Table is shown Fixes: #18618 * Refactor: Removes unnecessary condition * PanelData: provide legacy data only when needed (#19018) * no legacy * invert logic... now compiles * merge getQueryResponseData and getDataRaw * update comment about query editor * use single getData() function * only send legacy when it is used in explore * pre process rather than post process * pre process rather than post process * Minor refactoring * Add missing tags to test datasource response * MixedDatasource: Adds query observable pattern to MixedDatasource (#19037) * start mixed datasource * Refactor: Refactors into observable parttern * Tests: Fixes tests * Tests: Removes console.log * Refactor: Adds unique requestId
6 years ago
);
};
createRangeQuery(target: LokiQuery, options: RangeQueryOptions, limit: number): LokiRangeQueryRequest {
const query = target.expr;
let range: { start?: number; end?: number; step?: number } = {};
if (options.range) {
const startNs = this.getTime(options.range.from, false);
const endNs = this.getTime(options.range.to, true);
const rangeMs = Math.ceil((endNs - startNs) / 1e6);
const resolution = target.resolution || (DEFAULT_RESOLUTION.value as number);
const adjustedInterval =
this.adjustInterval((options as DataQueryRequest<LokiQuery>).intervalMs || 1000, resolution, rangeMs) / 1000;
// We want to ceil to 3 decimal places
const step = Math.ceil(adjustedInterval * 1000) / 1000;
range = {
start: startNs,
end: endNs,
step,
};
}
return {
...DEFAULT_QUERY_PARAMS,
...range,
query,
limit,
};
}
/**
* Attempts to send a query to /loki/api/v1/query_range
*/
runRangeQuery = (
target: LokiQuery,
options: RangeQueryOptions,
responseListLength = 1
): Observable<DataQueryResponse> => {
// For metric query we use maxDataPoints from the request options which should be something like width of the
// visualisation in pixels. In case of logs request we either use lines limit defined in the query target or
// global limit defined for the data source which ever is lower.
let maxDataPoints = isMetricsQuery(target.expr)
? // We fallback to maxLines here because maxDataPoints is defined as possibly undefined. Not sure that can
// actually happen both Dashboards and Explore should send some value here. If not maxLines does not make that
// much sense but nor any other arbitrary value.
(options as DataQueryRequest<LokiQuery>).maxDataPoints || this.maxLines
: // If user wants maxLines 0 we still fallback to data source limit. I think that makes sense as why would anyone
// want to do a query and not see any results?
target.maxLines || this.maxLines;
if ((options as DataQueryRequest<LokiQuery>).liveStreaming) {
return this.runLiveQuery(target, maxDataPoints);
Explore: Adds Loki explore query editor (#21497) * Explore: updates grafana-data explore query field props with explore mode * Explore: updates query row to pass down explore mode to query fields * Explore: adds LokiExploreQueryEditor * Explore: updates loki query field form to render children * Explore: adds loki explore extra field component * Explore: adds extra field element to loki query field form * Explore: updates loki explore query editor to use extra field element * Explore: moves ExploreMode to grafana-data * Explore: updates query row limit string * Explore: adds maxLines to DataQuery * Explore: adds maxLines to loki datasource runRangeQueryWithFallback * Explore: adds onChangeQueryLimit to LokiExploreQueryEditor * Explore: updates loki explore query editor to render extra field only in logs mode * Explore: fixes query limits for live and legacy queries * Explore: fixes result processor max lines limit in get logs result * Explore: fixes Loki datasource limit test * Explore: removes unnecessary ExploreMode from Loki language provider * Explore: fixes formatting * Explore: updates grafana-data datasource types - replaces strings with explore mode enum * Explore: updates loki explore query field props to take ReactNode * Explore: updates the way we calculate loki query lines limit to fall back to 0 lines on negative or invalid input instead of datasource maxLines * Explore: updates result processor get logs result method to avoid counting invalid/negative line limits * Explore: updates loki result transformer to process only an appropriate slice of a result instead of an entire one * Explore: adds a method for query limit preprocessing/mapping * Explore: updates loki datasource run range query with fallback method to use options.maxDataPoints in dashboards * Explore: removes unnecessary maxlineslimt from getLogsResult in resultProcessor * Explore: moves line limit to metadata * Explore: adds an ability to specify input type of extra field * Explore: updates LokiExploreQueryEditor - adds an input type * Explore: updates LokiExploreQueryEditor to run queries when maxLines is positive * Explore: fixes failing import of ExploreMode * Explore: fixes reducers test imports formatting * Explore: updates Loki extra field with min value set to 0 * Explore: exports LokiExploreExtraFieldProps * Explore: adds render test of LokiExploreQueryEditor * Explore: adds LokiExploreQueryEditor snapshot * Explore: updates LokiExploreQueryEditor onChangeQueryLimit method to prevent it from running when query input is empty - fixes cheatsheet display issue * Explore: updates Loki editor snapshots * Explore: fixes typo in test set name in LokiExploreQueryEditor * Explore: adds a render test of LokiExploreExtraField * Explore: fixes typo in LokiExploreQueryEditor * Explore: updates LokiExploreQueryEditor snapshot due to timezone issues * Explore: updates LokiExploreExtraField to export both functional component and a version using memo * Explore: updates LokiExploreQueryEditor to export both functional component and memoized function * Explore: updates LokiExploreQueryEditor - removes unnecessary react fragment * Explore: updates LokiExploreQueryEditor snapshot * Explore: adds LokiExploreQueryEditor tests for different explore mode cases * Explore: fixes Loki datasource and result transformer * Explore: updates LokiExploreQueryEditor snapshot * Explore: updates LokiExploreQueryEditor tests and test setup * Explore: updates LokiExploreQueryEditor - refactors component * Explore: updates LokiExploreQueryEditor to use default import from LokiExploreExtraField * Explore: updates LokiExploreQueryEditor snapshot * Explore: fixes formatting * Explore: updates LokiExploreQueryEditor max lines change * Explore: updates LokiExploreQueryEditor tests checking ExtraFieldElement * Explore: adds mock loki datasource to LokiExploreQueryEditor * Explore: updates LokiExploreQueryEditor test mock - adds language provider * Explore: updates LokiExploreQueryEditor snapshot * Explore: updates Loki ResultTransformer to filter out rows on limit - logic to be moved into a component with new form styles * Explore: updates LokiExploreQueryEditor tests
5 years ago
}
const query = this.createRangeQuery(target, options, maxDataPoints);
const headers = target.volumeQuery ? { 'X-Query-Tag': 'Source=logvolhist' } : undefined;
return this._request(RANGE_QUERY_ENDPOINT, query, { headers }).pipe(
catchError((err) => throwError(() => this.processError(err, target))),
switchMap((response) =>
processRangeQueryResponse(
response.data,
target,
query,
responseListLength,
maxDataPoints,
this.instanceSettings.jsonData,
(options as DataQueryRequest<LokiQuery>).scopedVars,
(options as DataQueryRequest<LokiQuery>).reverse
)
)
);
};
createLiveTarget(target: LokiQuery, maxDataPoints: number): LokiLiveTarget {
const query = target.expr;
const baseUrl = this.instanceSettings.url;
const params = serializeParams({ query });
return {
query,
url: convertToWebSocketUrl(`${baseUrl}/loki/api/v1/tail?${params}`),
refId: target.refId,
size: maxDataPoints,
};
}
/**
* Runs live queries which in this case means creating a websocket and listening on it for new logs.
* This returns a bit different dataFrame than runQueries as it returns single dataframe even if there are multiple
* Loki streams, sets only common labels on dataframe.labels and has additional dataframe.fields.labels for unique
* labels per row.
*/
runLiveQuery = (target: LokiQuery, maxDataPoints: number): Observable<DataQueryResponse> => {
const liveTarget = this.createLiveTarget(target, maxDataPoints);
return this.streams.getStream(liveTarget).pipe(
map((data) => ({
data: data || [],
key: `loki-${liveTarget.refId}`,
state: LoadingState.Streaming,
})),
catchError((err: any) => {
return throwError(() => `Live tailing was stopped due to following error: ${err.reason}`);
})
);
};
getRangeScopedVars(range: TimeRange = this.timeSrv.timeRange()) {
const msRange = range.to.diff(range.from);
const sRange = Math.round(msRange / 1000);
return {
__range_ms: { text: msRange, value: msRange },
__range_s: { text: sRange, value: sRange },
__range: { text: sRange + 's', value: sRange + 's' },
};
}
interpolateVariablesInQueries(queries: LokiQuery[], scopedVars: ScopedVars): LokiQuery[] {
let expandedQueries = queries;
if (queries && queries.length) {
expandedQueries = queries.map((query) => ({
...query,
datasource: this.getRef(),
expr: this.templateSrv.replace(query.expr, scopedVars, this.interpolateQueryExpr),
}));
}
return expandedQueries;
}
getQueryDisplayText(query: LokiQuery) {
return query.expr;
}
getTimeRangeParams() {
const timeRange = this.timeSrv.timeRange();
return { start: timeRange.from.valueOf() * NS_IN_MS, end: timeRange.to.valueOf() * NS_IN_MS };
}
async importQueries(queries: DataQuery[], originDataSource: DataSourceApi): Promise<LokiQuery[]> {
return this.languageProvider.importQueries(queries, originDataSource);
}
async metadataRequest(url: string, params?: Record<string, string | number>) {
const res = await lastValueFrom(this._request(url, params, { hideFromInspector: true }));
return res.data.data || res.data.values || [];
}
async metricFindQuery(query: string) {
if (!query) {
return Promise.resolve([]);
}
const interpolated = this.templateSrv.replace(query, {}, this.interpolateQueryExpr);
return await this.processMetricFindQuery(interpolated);
}
async processMetricFindQuery(query: string) {
const labelNamesRegex = /^label_names\(\)\s*$/;
const labelValuesRegex = /^label_values\((?:(.+),\s*)?([a-zA-Z_][a-zA-Z0-9_]*)\)\s*$/;
const labelNames = query.match(labelNamesRegex);
if (labelNames) {
return await this.labelNamesQuery();
}
const labelValues = query.match(labelValuesRegex);
if (labelValues) {
// If we have query expr, use /series endpoint
if (labelValues[1]) {
return await this.labelValuesSeriesQuery(labelValues[1], labelValues[2]);
}
return await this.labelValuesQuery(labelValues[2]);
}
return Promise.resolve([]);
}
async labelNamesQuery() {
const url = `${LOKI_ENDPOINT}/label`;
const params = this.getTimeRangeParams();
const result = await this.metadataRequest(url, params);
return result.map((value: string) => ({ text: value }));
}
async labelValuesQuery(label: string) {
const params = this.getTimeRangeParams();
const url = `${LOKI_ENDPOINT}/label/${label}/values`;
const result = await this.metadataRequest(url, params);
return result.map((value: string) => ({ text: value }));
}
async labelValuesSeriesQuery(expr: string, label: string) {
const timeParams = this.getTimeRangeParams();
const params = {
...timeParams,
'match[]': expr,
};
const url = `${LOKI_ENDPOINT}/series`;
const streams = new Set();
const result = await this.metadataRequest(url, params);
result.forEach((stream: { [key: string]: string }) => {
if (stream[label]) {
streams.add({ text: stream[label] });
}
});
return Array.from(streams);
}
// By implementing getTagKeys and getTagValues we add ad-hoc filtters functionality
async getTagKeys() {
return await this.labelNamesQuery();
}
async getTagValues(options: any = {}) {
return await this.labelValuesQuery(options.key);
}
interpolateQueryExpr(value: any, variable: any) {
// if no multi or include all do not regexEscape
if (!variable.multi && !variable.includeAll) {
return lokiRegularEscape(value);
}
if (typeof value === 'string') {
return lokiSpecialRegexEscape(value);
}
const escapedValues = lodashMap(value, lokiSpecialRegexEscape);
return escapedValues.join('|');
}
modifyQuery(query: LokiQuery, action: any): LokiQuery {
let expression = query.expr ?? '';
switch (action.type) {
case 'ADD_FILTER': {
expression = this.addLabelToQuery(expression, action.key, action.value, '=');
break;
}
case 'ADD_FILTER_OUT': {
expression = this.addLabelToQuery(expression, action.key, action.value, '!=');
break;
}
default:
break;
}
return { ...query, expr: expression };
}
getTime(date: string | DateTime, roundUp: boolean) {
if (typeof date === 'string') {
date = dateMath.parse(date, roundUp)!;
}
return Math.ceil(date.valueOf() * 1e6);
}
getLogRowContext = (row: LogRowModel, options?: RowContextOptions): Promise<{ data: DataFrame[] }> => {
const target = this.prepareLogRowContextQueryTarget(
row,
(options && options.limit) || 10,
(options && options.direction) || 'BACKWARD'
);
const reverse = options && options.direction === 'FORWARD';
return lastValueFrom(
this._request(RANGE_QUERY_ENDPOINT, target).pipe(
catchError((err) => {
const error: DataQueryError = {
message: 'Error during context query. Please check JS console logs.',
status: err.status,
statusText: err.statusText,
};
throw error;
}),
switchMap((res) =>
of({
data: res.data
? res.data.data.result.map((stream: LokiStreamResult) => lokiStreamResultToDataFrame(stream, reverse))
: [],
})
)
)
);
};
prepareLogRowContextQueryTarget = (row: LogRowModel, limit: number, direction: 'BACKWARD' | 'FORWARD') => {
const labels = this.languageProvider.getLabelKeys();
const query = Object.keys(row.labels)
.map((label: string) => {
if (labels.includes(label)) {
// escape backslashes in label as users can't escape them by themselves
return `${label}="${row.labels[label].replace(/\\/g, '\\\\')}"`;
}
return '';
})
// Filter empty strings
.filter((label) => !!label)
.join(',');
const contextTimeBuffer = 2 * 60 * 60 * 1000; // 2h buffer
const commonTargetOptions = {
limit,
query: `{${query}}`,
expr: `{${query}}`,
direction,
};
const fieldCache = new FieldCache(row.dataFrame);
const nsField = fieldCache.getFieldByName('tsNs')!;
const nsTimestamp = nsField.values.get(row.rowIndex);
if (direction === 'BACKWARD') {
return {
...commonTargetOptions,
// convert to ns, we loose some precision here but it is not that important at the far points of the context
start: row.timeEpochMs - contextTimeBuffer + '000000',
end: nsTimestamp,
direction,
};
} else {
return {
...commonTargetOptions,
// start param in Loki API is inclusive so we'll have to filter out the row that this request is based from
// and any other that were logged in the same ns but before the row. Right now these rows will be lost
// because the are before but came it he response that should return only rows after.
start: nsTimestamp,
// convert to ns, we loose some precision here but it is not that important at the far points of the context
end: row.timeEpochMs + contextTimeBuffer + '000000',
};
}
};
testDatasource() {
// Consider only last 10 minutes otherwise request takes too long
const startMs = Date.now() - 10 * 60 * 1000;
const start = `${startMs}000000`; // API expects nanoseconds
return lastValueFrom(
this._request(`${LOKI_ENDPOINT}/label`, { start }).pipe(
map((res) => {
const values: any[] = res?.data?.data || res?.data?.values || [];
const testResult =
values.length > 0
? { status: 'success', message: 'Data source connected and labels found.' }
: {
status: 'error',
message:
'Data source connected, but no labels received. Verify that Loki and Promtail is configured properly.',
};
return testResult;
}),
catchError((err: any) => {
let message = 'Loki: ';
if (err.statusText) {
message += err.statusText;
} else {
message += 'Cannot connect to Loki';
}
if (err.status) {
message += `. ${err.status}`;
}
if (err.data && err.data.message) {
message += `. ${err.data.message}`;
} else if (err.data) {
message += `. ${err.data}`;
}
return of({ status: 'error', message: message });
})
)
);
}
async annotationQuery(options: any): Promise<AnnotationEvent[]> {
const {
expr,
maxLines,
instant,
stepInterval,
tagKeys = '',
titleFormat = '',
textFormat = '',
} = options.annotation;
if (!expr) {
return [];
}
const interpolatedExpr = this.templateSrv.replace(expr, {}, this.interpolateQueryExpr);
const query = {
refId: `annotation-${options.annotation.name}`,
expr: interpolatedExpr,
maxLines,
instant,
stepInterval,
};
const { data } = instant
? await lastValueFrom(this.runInstantQuery(query, options as any))
: await lastValueFrom(this.runRangeQuery(query, options as any));
const annotations: AnnotationEvent[] = [];
const splitKeys: string[] = tagKeys.split(',').filter((v: string) => v !== '');
QueryProcessing: Observable query interface and RxJS for query & stream processing (#18899) * I needed to learn some rxjs and understand this more, so just playing around * Updated * Removed all the complete calls * Refactoring * StreamHandler -> observable start * progress * simple singal works * Handle update time range * added error handling * wrap old function * minor changes * handle data format in the subscribe function * Use replay subject to return last value to subscribers * Set loading state after no response in 50ms * added missing file * updated comment * Added cancelation of network requests * runRequest: Added unit test scenario framework * Progress on tests * minor refactor of unit tests * updated test * removed some old code * Shared queries work again, and also became so much simplier * unified query and observe methods * implict any fix * Fixed closed subject issue * removed comment * Use last returned data for loading state * WIP: Explore to runRequest makover step1 * Minor progress * Minor progress on explore and runRequest * minor progress * Things are starting to work in explore * Updated prometheus to use new observable query response, greatly simplified code * Revert refId change * Found better solution for key/refId/requestId problem * use observable with loki * tests compile * fix loki query prep * Explore: correct first response handling * Refactorings * Refactoring * Explore: Fixes LoadingState and GraphResults between runs (#18986) * Refactor: Adds state to DataQueryResponse * Fix: Fixes so we do not empty results before new data arrives Fixes: #17409 * Transformations work * observable test data * remove single() from loki promise * Fixed comment * Explore: Fixes failing Loki and Prometheus unit tests (#18995) * Tests: Makes datasource tests work again * Fix: Fixes loki datasource so highligthing works * Chore: Runs Prettier * Fixed query runner tests * Delay loading state indication to 200ms * Fixed test * fixed unit tests * Clear cached calcs * Fixed bug getProcesedDataFrames * Fix the correct test is a better idea * Fix: Fixes so queries in Explore are only run if Graph/Table is shown (#19000) * Fix: Fixes so queries in Explore are only run if Graph/Table is shown Fixes: #18618 * Refactor: Removes unnecessary condition * PanelData: provide legacy data only when needed (#19018) * no legacy * invert logic... now compiles * merge getQueryResponseData and getDataRaw * update comment about query editor * use single getData() function * only send legacy when it is used in explore * pre process rather than post process * pre process rather than post process * Minor refactoring * Add missing tags to test datasource response * MixedDatasource: Adds query observable pattern to MixedDatasource (#19037) * start mixed datasource * Refactor: Refactors into observable parttern * Tests: Fixes tests * Tests: Removes console.log * Refactor: Adds unique requestId
6 years ago
for (const frame of data) {
const labels: { [key: string]: string } = {};
for (const field of frame.fields) {
if (field.labels) {
for (const [key, value] of Object.entries(field.labels)) {
labels[key] = String(value).trim();
}
}
}
const tags: string[] = [
...new Set(
Object.entries(labels).reduce((acc: string[], [key, val]) => {
if (val === '') {
return acc;
}
if (splitKeys.length && !splitKeys.includes(key)) {
return acc;
}
acc.push.apply(acc, [val]);
return acc;
}, [])
),
];
const view = new DataFrameView<{ ts: string; line: string }>(frame);
QueryProcessing: Observable query interface and RxJS for query & stream processing (#18899) * I needed to learn some rxjs and understand this more, so just playing around * Updated * Removed all the complete calls * Refactoring * StreamHandler -> observable start * progress * simple singal works * Handle update time range * added error handling * wrap old function * minor changes * handle data format in the subscribe function * Use replay subject to return last value to subscribers * Set loading state after no response in 50ms * added missing file * updated comment * Added cancelation of network requests * runRequest: Added unit test scenario framework * Progress on tests * minor refactor of unit tests * updated test * removed some old code * Shared queries work again, and also became so much simplier * unified query and observe methods * implict any fix * Fixed closed subject issue * removed comment * Use last returned data for loading state * WIP: Explore to runRequest makover step1 * Minor progress * Minor progress on explore and runRequest * minor progress * Things are starting to work in explore * Updated prometheus to use new observable query response, greatly simplified code * Revert refId change * Found better solution for key/refId/requestId problem * use observable with loki * tests compile * fix loki query prep * Explore: correct first response handling * Refactorings * Refactoring * Explore: Fixes LoadingState and GraphResults between runs (#18986) * Refactor: Adds state to DataQueryResponse * Fix: Fixes so we do not empty results before new data arrives Fixes: #17409 * Transformations work * observable test data * remove single() from loki promise * Fixed comment * Explore: Fixes failing Loki and Prometheus unit tests (#18995) * Tests: Makes datasource tests work again * Fix: Fixes loki datasource so highligthing works * Chore: Runs Prettier * Fixed query runner tests * Delay loading state indication to 200ms * Fixed test * fixed unit tests * Clear cached calcs * Fixed bug getProcesedDataFrames * Fix the correct test is a better idea * Fix: Fixes so queries in Explore are only run if Graph/Table is shown (#19000) * Fix: Fixes so queries in Explore are only run if Graph/Table is shown Fixes: #18618 * Refactor: Removes unnecessary condition * PanelData: provide legacy data only when needed (#19018) * no legacy * invert logic... now compiles * merge getQueryResponseData and getDataRaw * update comment about query editor * use single getData() function * only send legacy when it is used in explore * pre process rather than post process * pre process rather than post process * Minor refactoring * Add missing tags to test datasource response * MixedDatasource: Adds query observable pattern to MixedDatasource (#19037) * start mixed datasource * Refactor: Refactors into observable parttern * Tests: Fixes tests * Tests: Removes console.log * Refactor: Adds unique requestId
6 years ago
view.forEach((row) => {
annotations.push({
time: new Date(row.ts).valueOf(),
title: renderTemplate(titleFormat, labels),
text: renderTemplate(textFormat, labels) || row.line,
tags,
});
});
}
return annotations;
}
showContextToggle(row?: LogRowModel): boolean {
return (row && row.searchWords && row.searchWords.length > 0) === true;
}
processError(err: FetchError, target: LokiQuery) {
let error = cloneDeep(err);
if (err.data.message.includes('escape') && target.expr.includes('\\')) {
error.data.message = `Error: ${err.data.message}. Make sure that all special characters are escaped with \\. For more information on escaping of special characters visit LogQL documentation at https://grafana.com/docs/loki/latest/logql/.`;
}
return error;
}
adjustInterval(dynamicInterval: number, resolution: number, range: number) {
// Loki will drop queries that might return more than 11000 data points.
// Calibrate interval if it is too small.
let safeInterval = range / 11000;
if (safeInterval > 1) {
safeInterval = Math.ceil(safeInterval);
}
let adjustedInterval = Math.max(resolution * dynamicInterval, safeInterval);
return adjustedInterval;
}
addAdHocFilters(queryExpr: string) {
const adhocFilters = this.templateSrv.getAdhocFilters(this.name);
let expr = queryExpr;
expr = adhocFilters.reduce((acc: string, filter: { key?: any; operator?: any; value?: any }) => {
const { key, operator } = filter;
let { value } = filter;
if (operator === '=~' || operator === '!~') {
value = lokiRegularEscape(value);
}
return this.addLabelToQuery(acc, key, value, operator);
}, expr);
return expr;
}
addLabelToQuery(queryExpr: string, key: string, value: string | number, operator: string) {
if (queryHasPipeParser(queryExpr) && !isMetricsQuery(queryExpr)) {
// If query has parser, we treat all labels as parsed and use | key="value" syntax
return addParsedLabelToQuery(queryExpr, key, value, operator);
} else {
return addLabelToQuery(queryExpr, key, value, operator, true);
}
}
}
export function renderTemplate(aliasPattern: string, aliasData: { [key: string]: string }) {
const aliasRegex = /\{\{\s*(.+?)\s*\}\}/g;
return aliasPattern.replace(aliasRegex, (_match, g1) => {
if (aliasData[g1]) {
return aliasData[g1];
}
return '';
});
}
export function lokiRegularEscape(value: any) {
if (typeof value === 'string') {
return value.replace(/'/g, "\\\\'");
}
return value;
}
export function lokiSpecialRegexEscape(value: any) {
if (typeof value === 'string') {
return lokiRegularEscape(value.replace(/\\/g, '\\\\\\\\').replace(/[$^*{}\[\]+?.()|]/g, '\\\\$&'));
}
return value;
}
/**
* Checks if the query expression uses function and so should return a time series instead of logs.
* Sometimes important to know that before we actually do the query.
*/
Loki: Full range logs volume (#39327) * Basic implementation of getLogsVolumeQuery method * Add todos * Add a switcher to automatically load logs volume * De-scope dismissing logs volume panel * De-scope logs volume query cancellation * Remove todo * Aggregate logs volume components in single panel * Show logs volume only when it's available * Aggregate logs volume by level * Simplify aggregation * Handle no logs volume data * Add error handling * Do not show auto-load logs volume switcher when loading logs volume is not available * Remove old logs volume graph * Clean up * Make getting data provider more generic * Provide complete logs volume data (error, isLoading) * Display more specific error message * Add missing props to mocks * Remove setRequest method * Mark getQueryRelatedDataProviders as internal * Add missing dataQueryRequest and add a todo * Remove redundant loading state * Do not mutate existing queries * Apply fix for zooming-in from main * Post-merge fixes * Create collection for data provider results * Use more generic names * Move aggregation logic to Loki logs volume provider * Move LogsVolume to common types * Update tests * Post-merge fixes * Fix mapping related data values * Simplify prop mappings * Add docs * Fix property name * Clean-up * Mark new types as internal * Reduce number of providers to logs volume only * Simplify data structure to DataQueryResponse * Move Logs Volume panel to a separate component * Test logsVolumeProvider.ts * Add observable version of datasource mock * Test getLogsVolumeDataProvider method * Test LogsVolumePanel * Test logs volume reducer * Clean up * Clean up * Fix test * Use sum by to use level field directly * Fix strict type errors * Fix strict type errors * Use "logs" instead of "unknown" if only one level was detected * Add docs about logs volume * Rename histogramRequest to logsVolumeRequest * Use LogsVolumeContentWrapper all content types * Move `autoLoadLogsVolume` local storage handling * Fix strict error * Move getting autoLoadLogsVolume to initial state * Cancel current logs volume subscription * Test cancelling subscriptions * Update docs/sources/datasources/loki.md Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Update packages/grafana-data/src/types/explore.ts Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Inline container styles * Ensure logs volume is aggregated per each subscription separately * Simplify logs volume provider * Type-guard support for logs volume provider * Simplify event handlers to avoid casting * Clean up and docs * Move auto-load switcher to logs volume panel * Fix test * Move DataSourceWithLogsVolumeSupport to avoid cross referencing * Simplify interface * Bring back old histogram and hide the new one behind a feature flag * Add missing props to logs histogram panel * Clean up the provider when it's not supported * Simplify storing autoLoadLogsVolume * Remove docs * Update packages/grafana-data/src/types/logsVolume.ts Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com> * Skip dataframes without fields (instant queries) * Revert styles changes * Revert styles changes * Add release tag Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com>
4 years ago
export function isMetricsQuery(query: string): boolean {
const tokens = Prism.tokenize(query, syntax);
return tokens.some((t) => {
// Not sure in which cases it can be string maybe if nothing matched which means it should not be a function
return typeof t !== 'string' && t.type === 'function';
});
}
function extractLevel(dataFrame: DataFrame): LogLevel {
let valueField;
try {
valueField = new FieldCache(dataFrame).getFirstFieldOfType(FieldType.number);
} catch {}
return valueField?.labels ? getLogLevelFromLabels(valueField.labels) : LogLevel.unknown;
}
function getLogLevelFromLabels(labels: Labels): LogLevel {
const labelNames = ['level', 'lvl', 'loglevel'];
let levelLabel;
for (let labelName of labelNames) {
if (labelName in labels) {
levelLabel = labelName;
break;
}
}
return levelLabel ? getLogLevelFromKey(labels[levelLabel]) : LogLevel.unknown;
}
export default LokiDatasource;