diff --git a/docs/sources/features/datasources/mssql.md b/docs/sources/features/datasources/mssql.md index dabb896ec0f..524a93a943b 100644 --- a/docs/sources/features/datasources/mssql.md +++ b/docs/sources/features/datasources/mssql.md @@ -81,7 +81,9 @@ Macro example | Description *$__timeFrom()* | Will be replaced by the start of the currently active time selection. For example, *'2017-04-21T05:01:17Z'* *$__timeTo()* | Will be replaced by the end of the currently active time selection. For example, *'2017-04-21T05:06:17Z'* *$__timeGroup(dateColumn,'5m'[, fillvalue])* | Will be replaced by an expression usable in GROUP BY clause. Providing a *fillValue* of *NULL* or *floating value* will automatically fill empty series in timerange with that value.
For example, *CAST(ROUND(DATEDIFF(second, '1970-01-01', time_column)/300.0, 0) as bigint)\*300*. -*$__timeGroup(dateColumn,'5m', 0)* | Same as above but with a fill parameter so all null values will be converted to the fill value (all null values would be set to zero using this example). +*$__timeGroup(dateColumn,'5m', 0)* | Same as above but with a fill parameter so missing points in that series will be added by grafana and 0 will be used as value. +*$__timeGroup(dateColumn,'5m', NULL)* | Same as above but NULL will be used as value for missing points. +*$__timeGroup(dateColumn,'5m', last)* | Same as above but the last seen value in that series will be used as fill value if no value has been seen yet NULL will be used. *$__timeGroupAlias(dateColumn,'5m')* | Will be replaced identical to $__timeGroup but with an added column alias (only available in Grafana 5.3+). *$__unixEpochFilter(dateColumn)* | Will be replaced by a time range filter using the specified column name with times represented as unix timestamp. For example, *dateColumn > 1494410783 AND dateColumn < 1494497183* *$__unixEpochFrom()* | Will be replaced by the start of the currently active time selection as unix timestamp. For example, *1494410783* diff --git a/docs/sources/features/datasources/mysql.md b/docs/sources/features/datasources/mysql.md index a0e67037005..153b3d7bbf5 100644 --- a/docs/sources/features/datasources/mysql.md +++ b/docs/sources/features/datasources/mysql.md @@ -64,7 +64,9 @@ Macro example | Description *$__timeFrom()* | Will be replaced by the start of the currently active time selection. For example, *'2017-04-21T05:01:17Z'* *$__timeTo()* | Will be replaced by the end of the currently active time selection. For example, *'2017-04-21T05:06:17Z'* *$__timeGroup(dateColumn,'5m')* | Will be replaced by an expression usable in GROUP BY clause. For example, *cast(cast(UNIX_TIMESTAMP(dateColumn)/(300) as signed)*300 as signed),* -*$__timeGroup(dateColumn,'5m',0)* | Same as above but with a fill parameter so all null values will be converted to the fill value (all null values would be set to zero using this example). +*$__timeGroup(dateColumn,'5m', 0)* | Same as above but with a fill parameter so missing points in that series will be added by grafana and 0 will be used as value. +*$__timeGroup(dateColumn,'5m', NULL)* | Same as above but NULL will be used as value for missing points. +*$__timeGroup(dateColumn,'5m', last)* | Same as above but the last seen value in that series will be used as fill value if no value has been seen yet NULL will be used. *$__timeGroupAlias(dateColumn,'5m')* | Will be replaced identical to $__timeGroup but with an added column alias (only available in Grafana 5.3+). *$__unixEpochFilter(dateColumn)* | Will be replaced by a time range filter using the specified column name with times represented as unix timestamp. For example, *dateColumn > 1494410783 AND dateColumn < 1494497183* *$__unixEpochFrom()* | Will be replaced by the start of the currently active time selection as unix timestamp. For example, *1494410783* diff --git a/docs/sources/features/datasources/postgres.md b/docs/sources/features/datasources/postgres.md index 35dfcac15c0..b776b7cbe58 100644 --- a/docs/sources/features/datasources/postgres.md +++ b/docs/sources/features/datasources/postgres.md @@ -61,7 +61,9 @@ Macro example | Description *$__timeFrom()* | Will be replaced by the start of the currently active time selection. For example, *'2017-04-21T05:01:17Z'* *$__timeTo()* | Will be replaced by the end of the currently active time selection. For example, *'2017-04-21T05:06:17Z'* *$__timeGroup(dateColumn,'5m')* | Will be replaced by an expression usable in GROUP BY clause. For example, *(extract(epoch from dateColumn)/300)::bigint*300* -*$__timeGroup(dateColumn,'5m', 0)* | Same as above but with a fill parameter so all null values will be converted to the fill value (all null values would be set to zero using this example). +*$__timeGroup(dateColumn,'5m', 0)* | Same as above but with a fill parameter so missing points in that series will be added by grafana and 0 will be used as value. +*$__timeGroup(dateColumn,'5m', NULL)* | Same as above but NULL will be used as value for missing points. +*$__timeGroup(dateColumn,'5m', last)* | Same as above but the last seen value in that series will be used as fill value if no value has been seen yet NULL will be used. *$__timeGroupAlias(dateColumn,'5m')* | Will be replaced identical to $__timeGroup but with an added column alias (only available in Grafana 5.3+). *$__unixEpochFilter(dateColumn)* | Will be replaced by a time range filter using the specified column name with times represented as unix timestamp. For example, *dateColumn >= 1494410783 AND dateColumn <= 1494497183* *$__unixEpochFrom()* | Will be replaced by the start of the currently active time selection as unix timestamp. For example, *1494410783* diff --git a/pkg/tsdb/mssql/macros.go b/pkg/tsdb/mssql/macros.go index f33ab1d40be..57a37d618e0 100644 --- a/pkg/tsdb/mssql/macros.go +++ b/pkg/tsdb/mssql/macros.go @@ -99,9 +99,13 @@ func (m *msSqlMacroEngine) evaluateMacro(name string, args []string) (string, er if len(args) == 3 { m.query.Model.Set("fill", true) m.query.Model.Set("fillInterval", interval.Seconds()) - if args[2] == "NULL" { - m.query.Model.Set("fillNull", true) - } else { + switch args[2] { + case "NULL": + m.query.Model.Set("fillMode", "null") + case "last": + m.query.Model.Set("fillMode", "last") + default: + m.query.Model.Set("fillMode", "value") floatVal, err := strconv.ParseFloat(args[2], 64) if err != nil { return "", fmt.Errorf("error parsing fill value %v", args[2]) diff --git a/pkg/tsdb/mssql/macros_test.go b/pkg/tsdb/mssql/macros_test.go index ea50c418de7..b808666d967 100644 --- a/pkg/tsdb/mssql/macros_test.go +++ b/pkg/tsdb/mssql/macros_test.go @@ -76,12 +76,25 @@ func TestMacroEngine(t *testing.T) { _, err := engine.Interpolate(query, timeRange, "GROUP BY $__timeGroup(time_column,'5m', NULL)") fill := query.Model.Get("fill").MustBool() - fillNull := query.Model.Get("fillNull").MustBool() + fillMode := query.Model.Get("fillMode").MustString() fillInterval := query.Model.Get("fillInterval").MustInt() So(err, ShouldBeNil) So(fill, ShouldBeTrue) - So(fillNull, ShouldBeTrue) + So(fillMode, ShouldEqual, "null") + So(fillInterval, ShouldEqual, 5*time.Minute.Seconds()) + }) + + Convey("interpolate __timeGroup function with fill (value = last)", func() { + _, err := engine.Interpolate(query, timeRange, "GROUP BY $__timeGroup(time_column,'5m', last)") + + fill := query.Model.Get("fill").MustBool() + fillMode := query.Model.Get("fillMode").MustString() + fillInterval := query.Model.Get("fillInterval").MustInt() + + So(err, ShouldBeNil) + So(fill, ShouldBeTrue) + So(fillMode, ShouldEqual, "last") So(fillInterval, ShouldEqual, 5*time.Minute.Seconds()) }) diff --git a/pkg/tsdb/mysql/macros.go b/pkg/tsdb/mysql/macros.go index a56fd1ceb2a..bebf4b396bb 100644 --- a/pkg/tsdb/mysql/macros.go +++ b/pkg/tsdb/mysql/macros.go @@ -94,9 +94,13 @@ func (m *mySqlMacroEngine) evaluateMacro(name string, args []string) (string, er if len(args) == 3 { m.query.Model.Set("fill", true) m.query.Model.Set("fillInterval", interval.Seconds()) - if args[2] == "NULL" { - m.query.Model.Set("fillNull", true) - } else { + switch args[2] { + case "NULL": + m.query.Model.Set("fillMode", "null") + case "last": + m.query.Model.Set("fillMode", "last") + default: + m.query.Model.Set("fillMode", "value") floatVal, err := strconv.ParseFloat(args[2], 64) if err != nil { return "", fmt.Errorf("error parsing fill value %v", args[2]) diff --git a/pkg/tsdb/mysql/mysql_test.go b/pkg/tsdb/mysql/mysql_test.go index 9947c23498b..fe262a3f758 100644 --- a/pkg/tsdb/mysql/mysql_test.go +++ b/pkg/tsdb/mysql/mysql_test.go @@ -295,7 +295,7 @@ func TestMySQL(t *testing.T) { }) - Convey("When doing a metric query using timeGroup with float fill enabled", func() { + Convey("When doing a metric query using timeGroup with value fill enabled", func() { query := &tsdb.TsdbQuery{ Queries: []*tsdb.Query{ { @@ -320,6 +320,35 @@ func TestMySQL(t *testing.T) { points := queryResult.Series[0].Points So(points[3][0].Float64, ShouldEqual, 1.5) }) + + Convey("When doing a metric query using timeGroup with last fill enabled", func() { + query := &tsdb.TsdbQuery{ + Queries: []*tsdb.Query{ + { + Model: simplejson.NewFromAny(map[string]interface{}{ + "rawSql": "SELECT $__timeGroup(time, '5m', last) as time_sec, avg(value) as value FROM metric GROUP BY 1 ORDER BY 1", + "format": "time_series", + }), + RefId: "A", + }, + }, + TimeRange: &tsdb.TimeRange{ + From: fmt.Sprintf("%v", fromStart.Unix()*1000), + To: fmt.Sprintf("%v", fromStart.Add(34*time.Minute).Unix()*1000), + }, + } + + resp, err := endpoint.Query(nil, nil, query) + So(err, ShouldBeNil) + queryResult := resp.Results["A"] + So(queryResult.Error, ShouldBeNil) + + points := queryResult.Series[0].Points + So(points[2][0].Float64, ShouldEqual, 15.0) + So(points[3][0].Float64, ShouldEqual, 15.0) + So(points[6][0].Float64, ShouldEqual, 20.0) + }) + }) Convey("Given a table with metrics having multiple values and measurements", func() { diff --git a/pkg/tsdb/postgres/macros.go b/pkg/tsdb/postgres/macros.go index 9e337caf3ec..3ab21ea0c6e 100644 --- a/pkg/tsdb/postgres/macros.go +++ b/pkg/tsdb/postgres/macros.go @@ -116,9 +116,13 @@ func (m *postgresMacroEngine) evaluateMacro(name string, args []string) (string, if len(args) == 3 { m.query.Model.Set("fill", true) m.query.Model.Set("fillInterval", interval.Seconds()) - if args[2] == "NULL" { - m.query.Model.Set("fillNull", true) - } else { + switch args[2] { + case "NULL": + m.query.Model.Set("fillMode", "null") + case "last": + m.query.Model.Set("fillMode", "last") + default: + m.query.Model.Set("fillMode", "value") floatVal, err := strconv.ParseFloat(args[2], 64) if err != nil { return "", fmt.Errorf("error parsing fill value %v", args[2]) diff --git a/pkg/tsdb/postgres/postgres_test.go b/pkg/tsdb/postgres/postgres_test.go index 3e864dca1e6..ac0964e912c 100644 --- a/pkg/tsdb/postgres/postgres_test.go +++ b/pkg/tsdb/postgres/postgres_test.go @@ -276,7 +276,7 @@ func TestPostgres(t *testing.T) { }) - Convey("When doing a metric query using timeGroup with float fill enabled", func() { + Convey("When doing a metric query using timeGroup with value fill enabled", func() { query := &tsdb.TsdbQuery{ Queries: []*tsdb.Query{ { @@ -303,6 +303,34 @@ func TestPostgres(t *testing.T) { }) }) + Convey("When doing a metric query using timeGroup with last fill enabled", func() { + query := &tsdb.TsdbQuery{ + Queries: []*tsdb.Query{ + { + Model: simplejson.NewFromAny(map[string]interface{}{ + "rawSql": "SELECT $__timeGroup(time, '5m', last), avg(value) as value FROM metric GROUP BY 1 ORDER BY 1", + "format": "time_series", + }), + RefId: "A", + }, + }, + TimeRange: &tsdb.TimeRange{ + From: fmt.Sprintf("%v", fromStart.Unix()*1000), + To: fmt.Sprintf("%v", fromStart.Add(34*time.Minute).Unix()*1000), + }, + } + + resp, err := endpoint.Query(nil, nil, query) + So(err, ShouldBeNil) + queryResult := resp.Results["A"] + So(queryResult.Error, ShouldBeNil) + + points := queryResult.Series[0].Points + So(points[2][0].Float64, ShouldEqual, 15.0) + So(points[3][0].Float64, ShouldEqual, 15.0) + So(points[6][0].Float64, ShouldEqual, 20.0) + }) + Convey("Given a table with metrics having multiple values and measurements", func() { type metric_values struct { Time time.Time diff --git a/pkg/tsdb/sql_engine.go b/pkg/tsdb/sql_engine.go index 3f681a5cdd7..f2f8b17db5f 100644 --- a/pkg/tsdb/sql_engine.go +++ b/pkg/tsdb/sql_engine.go @@ -274,9 +274,15 @@ func (e *sqlQueryEndpoint) transformToTimeSeries(query *Query, rows *core.Rows, fillMissing := query.Model.Get("fill").MustBool(false) var fillInterval float64 fillValue := null.Float{} + fillLast := false + if fillMissing { fillInterval = query.Model.Get("fillInterval").MustFloat64() * 1000 - if !query.Model.Get("fillNull").MustBool(false) { + switch query.Model.Get("fillMode").MustString() { + case "null": + case "last": + fillLast = true + case "value": fillValue.Float64 = query.Model.Get("fillValue").MustFloat64() fillValue.Valid = true } @@ -352,6 +358,14 @@ func (e *sqlQueryEndpoint) transformToTimeSeries(query *Query, rows *core.Rows, intervalStart = series.Points[len(series.Points)-1][1].Float64 + fillInterval } + if fillLast { + if len(series.Points) > 0 { + fillValue = series.Points[len(series.Points)-1][0] + } else { + fillValue.Valid = false + } + } + // align interval start intervalStart = math.Floor(intervalStart/fillInterval) * fillInterval @@ -377,6 +391,14 @@ func (e *sqlQueryEndpoint) transformToTimeSeries(query *Query, rows *core.Rows, intervalStart := series.Points[len(series.Points)-1][1].Float64 intervalEnd := float64(tsdbQuery.TimeRange.MustGetTo().UnixNano() / 1e6) + if fillLast { + if len(series.Points) > 0 { + fillValue = series.Points[len(series.Points)-1][0] + } else { + fillValue.Valid = false + } + } + // align interval start intervalStart = math.Floor(intervalStart/fillInterval) * fillInterval for i := intervalStart + fillInterval; i < intervalEnd; i += fillInterval { diff --git a/public/app/plugins/datasource/mssql/partials/query.editor.html b/public/app/plugins/datasource/mssql/partials/query.editor.html index e1320aabde2..e873d60ebbf 100644 --- a/public/app/plugins/datasource/mssql/partials/query.editor.html +++ b/public/app/plugins/datasource/mssql/partials/query.editor.html @@ -53,7 +53,9 @@ Macros: - $__timeEpoch(column) -> DATEDIFF(second, '1970-01-01', column) AS time - $__timeFilter(column) -> column BETWEEN '2017-04-21T05:01:17Z' AND '2017-04-21T05:01:17Z' - $__unixEpochFilter(column) -> column >= 1492750877 AND column <= 1492750877 -- $__timeGroup(column, '5m'[, fillvalue]) -> CAST(ROUND(DATEDIFF(second, '1970-01-01', column)/300.0, 0) as bigint)*300. Providing a fillValue of NULL or floating value will automatically fill empty series in timerange with that value. +- $__timeGroup(column, '5m'[, fillvalue]) -> CAST(ROUND(DATEDIFF(second, '1970-01-01', column)/300.0, 0) as bigint)*300. + by setting fillvalue grafana will fill in missing values according to the interval + fillvalue can be either a literal value, NULL or last; last will fill in the last seen value or NULL if none has been seen yet - $__timeGroupAlias(column, '5m'[, fillvalue]) -> CAST(ROUND(DATEDIFF(second, '1970-01-01', column)/300.0, 0) as bigint)*300 AS [time] Example of group by and order by with $__timeGroup: diff --git a/public/app/plugins/datasource/mysql/partials/query.editor.html b/public/app/plugins/datasource/mysql/partials/query.editor.html index db12a3fe8ce..664481ec8dc 100644 --- a/public/app/plugins/datasource/mysql/partials/query.editor.html +++ b/public/app/plugins/datasource/mysql/partials/query.editor.html @@ -53,7 +53,9 @@ Macros: - $__timeEpoch(column) -> UNIX_TIMESTAMP(column) as time_sec - $__timeFilter(column) -> column BETWEEN '2017-04-21T05:01:17Z' AND '2017-04-21T05:01:17Z' - $__unixEpochFilter(column) -> time_unix_epoch > 1492750877 AND time_unix_epoch < 1492750877 -- $__timeGroup(column,'5m') -> cast(cast(UNIX_TIMESTAMP(column)/(300) as signed)*300 as signed) +- $__timeGroup(column,'5m'[, fillvalue]) -> cast(cast(UNIX_TIMESTAMP(column)/(300) as signed)*300 as signed) + by setting fillvalue grafana will fill in missing values according to the interval + fillvalue can be either a literal value, NULL or last; last will fill in the last seen value or NULL if none has been seen yet - $__timeGroupAlias(column,'5m') -> cast(cast(UNIX_TIMESTAMP(column)/(300) as signed)*300 as signed) AS "time" Example of group by and order by with $__timeGroup: diff --git a/public/app/plugins/datasource/postgres/partials/query.editor.html b/public/app/plugins/datasource/postgres/partials/query.editor.html index 1b7278f6809..c455c0ebaf9 100644 --- a/public/app/plugins/datasource/postgres/partials/query.editor.html +++ b/public/app/plugins/datasource/postgres/partials/query.editor.html @@ -53,7 +53,9 @@ Macros: - $__timeEpoch -> extract(epoch from column) as "time" - $__timeFilter(column) -> column BETWEEN '2017-04-21T05:01:17Z' AND '2017-04-21T05:01:17Z' - $__unixEpochFilter(column) -> column >= 1492750877 AND column <= 1492750877 -- $__timeGroup(column,'5m') -> (extract(epoch from column)/300)::bigint*300 +- $__timeGroup(column,'5m'[, fillvalue]) -> (extract(epoch from column)/300)::bigint*300 + by setting fillvalue grafana will fill in missing values according to the interval + fillvalue can be either a literal value, NULL or last; last will fill in the last seen value or NULL if none has been seen yet - $__timeGroupAlias(column,'5m') -> (extract(epoch from column)/300)::bigint*300 AS "time" Example of group by and order by with $__timeGroup: