275 Commits (6e2b7d25898f321bfd6f94de05d6ce934c785da9)

Author	SHA1	Message	Date
Dylan Guedes	f9f1eddfc5	Loki: Improve query timeouts behavior (#7230) ... **What this PR does / why we need it**: Improves the different query-related timeouts behavior by fixing a bug and having better defaults. The changes are: - Fix the timeout middleware by creating a new request that will use the new WithDeadline context - `engine:timeout` was being ignored. This PR improves such scenario by using **the query timeout as a fallback** instead of the default `5min` for all cases	3 years ago
Susana Ferreira	10e9695b21	Fix rangemapper with VectorExpr support (#7288) ... **What this PR does / why we need it**: The new expression type VectorExpr introduced recently in [PR](https://github.com/grafana/loki/pull/7007), is not taken into account in the query mapping in the AST mapper rangemapper. The AST mapper shardmapper was updated in [PR](https://github.com/grafana/loki/pull/7045). **Special notes for your reviewer**: **Checklist** - [ ] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [x] Tests updated - [x] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Eng Zer Jun	e2842c69c5	Replace deprecated `io/ioutil` with `io` and `os` packages (#7212) ... The `io/ioutil` package has been deprecated in Go 1.16 (See https://pkg.go.dev/io/ioutil). This PR replaces the existing `io/ioutil` functions with their new definitions in `io` and `os` packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
李国忠	557fdf2206	[bugfix] logql: vectorExpr run fail when query_range: parallelise_shardable_queries: true (#7045) ... …hardable_queries: true <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: A grafana error occurred when I deployed the vector(0) function to our online loki cluster. It was finally found because online loki has a configuration `unexpected expr type (*syntax.VectorExpr) for ASTMapper type (logql.ShardMapper) ` ```yaml query_range: parallelise_shardable_queries: true ``` **Which issue(s) this PR fixes**: Fixes # **Special notes for your reviewer**: error snapssnapshot  with this pr，success snapshot  <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Owen Diehl	bac52165d3	highly sharded subqueries are ellided in expr.String (#7203) ... This is mostly a quality of life improvement. Highly sharded subqueries can be burdensome to view and they're logged on every request. This PR will only display the first four subqueries in a `Concat{Log,Sample}Expr`, which is easier on the eyes when viewing logs and is more efficient.	3 years ago
李国忠	3296c6dbe8	[new feature] logql engine support exec `vector(0)` grama (#7023) ... **What this PR does / why we need it**: logql engine support exec `vector(0)` grama. new PR of :https://github.com/grafana/loki/pull/7007 **Which issue(s) this PR fixes**: Fixes #6946 **Special notes for your reviewer**: test passed. test1: vectorExpr only  test2: binExpr and vectorExpr  **Checklist** - [ ] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
李国忠	6f4671a572	[new feature] syntax: support "vector" grama (#7007) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: Implement vector() function. The "vector function" feature is divided into multiple PRs to avoid excessively large PRs resulting in too long review time. **Which issue(s) this PR fixes**: Fixes #6946 **Special notes for your reviewer**: The test for syntax has passed  <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [ ] Documentation added - [ ] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`	3 years ago
Vladyslav Diachenko	615941234a	Logql Analyzer (#6171) ... * created logql-debugger page and API endpoint that returns information for each stage of the query * added docker-compose setup for local testing * refactored analyzer and added unit tests * removed docs page to release it in a separate PR when back-end API is published * changes structure of results object in response to return origin log line needed for case when only stream selector is sent in query * merged the latest main and resolved conflicts * added endpoint for readiness probe and configured CI * fixed drone.yml discrepancy * fixed method signature * fixed path to Dockerfile * added clean step	3 years ago
Ed Welch	6d54505485	Loki: add `__line__` and `__timestamp__` to label_format (#6983) ... <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: We have the `__line__` and `__timestamp__` functions currently available in the `line_format` stage but not in the `label_format` stage. This PR fixes that. Specifically `__timestamp__` can be valuable in a label because you can do things like ``` label_format hour_of_day=`{{__timestamp__ | date "15"}}` ``` To be able to conditionaly select log lines based on the hour of the day they were created, this can be useful for conditional alerting as an example. **Which issue(s) this PR fixes**: Fixes #6402 **Special notes for your reviewer**: <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [x] Documentation added - [ ] Tests updated - [x] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` Signed-off-by: Edward Welch <edward.welch@grafana.com>	3 years ago
Dylan Guedes	15f8f42295	Adapt queryTimeout to be a per-tenant configuration (#6835) ... What this PR does / why we need it: * Add a new per-tenant query timeout * Adds new middleware to query calls. This new middleware will timeout requests based on the new per-tenant query timeout * Add span logs to query calls missing it * Deprecate `engine.timeout` configuration The motivation to change this configuration to be per-tenant instead of global is that it makes sense to alleviate the timeout for particular tenants or make it more strict for others. Especially useful for scenarios where the Loki client doesn't handle context cancellations correctly; in such scenarios, since Loki would still process expensive queries that were canceled, this allows one to have small timeouts for most tenants, which will help mitigate unnecessary work without having a timeout that is too short for important tenants.	3 years ago
Susana Ferreira	3f3f4eda1d	Fix topk and bottomk operations with int <= 0 (#6937) ... * Fix topk and bottomk operations with int <= 0 * Fix VectorAggregationExpr String method * Update rangemapper_test expression * Add tests * Update CHANGELOG.md	3 years ago
Owen Diehl	72d8627a55	log query which causes panic in logql.clone (#6934) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	3 years ago
Christian Simon	ee2fe6514d	Propagate error detail properly in all cases (#6890) ... The error handling was missing appending the error detail properly in all cases. Fixes #6265	3 years ago
Kaviraj Kanagaraj	7b89f20378	fix(logql): Make `LabelSampleExtractor` work as expected when label doesn't exist in logline (#6766) ... * fix(logql): Make `StringLabelfilter` work as expected Consider Log lines ``` msg=hello subqueries=5 msg=hello subqueries=0 msg=hello ``` If you run following logql ``` {foo="bar"}|logfmt|subqueries!=0 ``` it returns ``` msg=hello subqueries=5 ``` But if you run following logql ``` {foo="bar"}|logfmt|subqueries!="0" # NOTE: "0" as string. ``` it returns ``` msg=hello subqueries=5 msg=hello ``` NOTE: it also returns log lines that doesn't contain `subqueries` label in the first place. This cause subtle error if used in metric queries using that `label` in unwrap. e.g: following logql on above log lines returns ``` sum_over_time({foo="bar"}|logfmt|subqueries!="0"|unwrap subqueries[1m]) ``` Returns `pipeline error: 'SampleExtractionErr'`. Because, the lines without `subqueries` labels are not skipped during `ProcessLine` and during metric extraction, our extractor sets this `SampleExtractionerror`. Fixes: https://github.com/grafana/loki/issues/6713 * Add it on changelog Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add more tests and fix the edge case with equality to empty strings Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * typo on the comment Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * handle this edge case during metric extraction. Not in label filter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix linter Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Remove commented code Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Update changelog Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Danny Kopping	fa42c12f4c	Adding "start_delta" entry to metrics.go line (#6550) ... * Adding "start_delta" entry to metrics.go line This will help us determine if a 3h query (derived from "length") is for some random 3h period in the past, or within the last 3h Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing test Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter; for once it was actually useful Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding "end_delta" as well Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Removing rounding Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing test Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Ed Welch	1a75bb8e22	Loki: Return an __error_details__ label for any line which incurs a __error__ while being processed (#6543) ... * adds an __error_details__ label which includes any details available about a logql error. Signed-off-by: Edward Welch <edward.welch@grafana.com> * fix lint, changed a test a little Signed-off-by: Edward Welch <edward.welch@grafana.com> * reset error details in Reset() function Signed-off-by: Edward Welch <edward.welch@grafana.com>	4 years ago
Owen Diehl	d2c099ac1d	query_frontend_shard_factor uses exponential buckets (#6689) ... Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Ed Welch	33761194d9	Loki: Add the ability to use the log line timestamp in label and line formatter (#6517) ... * Change Stage interface Process method to include the timestamp of the line Signed-off-by: Edward Welch <edward.welch@grafana.com> * allow using log line timestamp in formatter Signed-off-by: Edward Welch <edward.welch@grafana.com> * update formatting in test Signed-off-by: Edward Welch <edward.welch@grafana.com> * update docs Signed-off-by: Edward Welch <edward.welch@grafana.com> * Update docs/sources/logql/template_functions.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>	4 years ago
Susana Ferreira	79b2d01970	Add label_replace and literal expressions to instant query split by range (#6515) ... * Add label_replace logic to instant query split by range * Add label_replace and literal expressions support * Support binary operations with literal expressions * Remove Literal expression as splittable * Fix lint * Address comments	4 years ago
Christian Haudum	798677ae7f	Fix panic in instant query splitting when using unwrapped rate (#6348) ... * Fix panic in instant query splitting when using unwrapped rate The range aggregation `rate()` supports both log ranges and unwrapped ranges, e.g. `rate({app="foo"} [$__interval])` and `rate({app="foo"} | unwrap bar [$__interval])` Since `rate()` was split into multiple `count_over_time()` over total duration, but `count_over_time()` does not support `unwrap`, unwrapped rate queries caused panics. This fix changes the splitting of `rate({app="foo"} | unwrap bar [$__interval]` into multiple `sum_over_time()` over total duration. Fixes #6344 Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add tests Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Integrate review feedback Co-authored-by: Susana Ferreira <susana.ferreira@grafana.com> Co-authored-by: Susana Ferreira <susana.ferreira@grafana.com>	4 years ago
Cyril Tovena	8ee0d62d9e	Fixes bad initialization for uniqueString (#6432)	4 years ago
Christian Haudum	a56a88bd35	Add unwrapped aggregation `rate_counter()` (#6412) ... `rate_counter()` takes an unwrapped range as input, but unlike `rate()` it treats the values extracted from the log lines as "counter metric" like in Prometheus' `rate()` function. This is a replacement for the reverted change of #5013 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Owen Diehl	d6f50ca5e1	[TSDB] Use index sampling to determine shard factor (#6396) ... * use index sampling to determine shard factor * logging/tracing + better parallelism planning * queryrange downstreamer now checks max_query_parallelism * lint * handles interval, offset in tsdb planning and adds logging * fixes ns->ms confusion in index stats proto * handle zero shard value without panics * shardmapper will downstream a single unsharded query * uses concat expr with no shards to avoid query parsing errors * better logging * fixes wrong Size() method call and rounds to nearest KB when calculating chunk size * humanize bytes in log line * only adds defaultLookback to index sampling when interval is zero * removes comment * more logging * adjust through correctly * adds query length for index queries	4 years ago
Christian Haudum	b315ed0ddf	Sum values in unwrapped rate aggregation instead of treating them as counter (#6361) ... * Revert unwrapped rate aggregation to previous implementation This PR reverts the implementation done in #5013 to the original implementation that sums the extracted values from the log lines instead of treating them like a Prometheus counter metric. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Move changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove unused/dead code Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Clean changelog Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Jesus Vazquez	db9dbe8e1d	Bump prometheus dependency (#6403) ... * Bump prometheus dependency We had to add a replace of prometheus in go mod because thanos is importing an older version and then replacing it. * Undo textparser.New variable mismatch * modtimevfs path update * Fix labels withlabels and withoutlabels usages WithoutLabels and WithLabels methods have dissappeared. Now labels.NewBuilder has to be used. * Fix RuleManager interface for Update() Update now accepts a new param called ruleGroupPostProcessFunc which is an optional function to be executed after rule group processing. In this case I've set it to nil in the usages i've found. * NewQueryRange now accepts QueryOpts The NewQueryRange function now accepts QueryOptions so I had to update all the calls. I've set the opts to nil since we're not using them. * gofmt	4 years ago
Danny Kopping	a649e0d7c4	Naïve fix for optimize bug with `line_format` and `json` expression parser (#6375) ... * Naïve fix Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * CHANGELOG Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Update CHANGELOG.md Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Mathew Heard	771e218b58	JSON label extraction field validation expression overly strict (#6372) ... * Extend json label extraction test * Add json label extraction to aggregation * fix lexeer issue, it's value to have numbers inside the field (just not at the start) Also simplified logic * integrate feedback from reviews	4 years ago
Owen Diehl	9218e4654b	TSDB/query-planning-groundwork (#6367) ... * begins building queryrange compatible types for index sampling * adds MatcherGroups() method to SampleExpr * MatcherGroups test + ShardResolver interface * index stats codec wiring * millisecond precision on index stats codec * lint * cleanMatchers in tsdb.IndexClient for consistency + adds catchall matcher if nil	4 years ago
Kaviraj Kanagaraj	ad65cc29bc	Support `series|labels` query_type in `logql_query_duration` (#6341) ... So that you can do quantile calculation on metadata queries. Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Christopher Norris	874aa6495e	Fix max series test loop that returns early (#6345) ... Signed-off-by: Christopher Norris <christopher.norris@grafana.com>	4 years ago
Callum Styan	97ee1b93f3	[logql] don't try to execute a modified function that is not allowed by the AST (#6346) ... * Panic with an actual error if clone (which calls range aggregation expr's validate) errors, instead of swallowing the error and then attempting to execute a nil function, which panics anyways. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Call the validation function on a modified expression before attempting to execute it. If the validation fails we should return the original expression. Signed-off-by: Callum Styan <callumstyan@gmail.com>	4 years ago
Danny Kopping	36e0979cf5	Add cache usage statistics (#6317) ... * Adding cache statistics Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding metrics to metrics.go Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Creating new stats context for use in metric queries middleware Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Clean up unnecessary log fields Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Adding stats tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * CHANGELOG entry Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Appeasing the linter Documenting function Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Moving CHANGELOG entry to appropriate section Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Implementing a stats collector cache wrapper to simplify stats collection If we keep the stats collection in pkg/storage/chunk/cache/instrumented.go, then any implementation that wraps it will cause the stats collected to be incomplete. For example: NewBackground(cacheName, cfg.Background, Instrument(cacheName, cache, reg), reg)) - the background cache requests are not collected Signed-off-by: Danny Kopping <danny.kopping@grafana.com> * Fixing tests Signed-off-by: Danny Kopping <danny.kopping@grafana.com>	4 years ago
Cyril Tovena	7e53381eee	Avoid line copy during LogQL line_format (#6104)	4 years ago
Bayan Taani	50ef4f2ebd	Add duration filter unit test (#6271)	4 years ago
Cyril Tovena	67e7285bf5	Improve labels computation during LogQL pipeline (#6110) ... * Improve labels computation during LogQL pipeline * Fixes a bug that was mutating the base labels	4 years ago
Susana Ferreira	0dfd2cbbcc	Fix label extraction pushdown on instant queries (#6215) ... * Add downstream label extraction tests * Fix label extraction downstream pushdown	4 years ago
Owen Diehl	f14f9dd8c4	prevents sharding avg and avg_over_time when their children arent shardable. (#6204)	4 years ago
Aditya C S	c6af640798	feat: support simple selection of labels in json expr parser (#6151)	4 years ago
Justin Burnham	97c7d7ca3c	LogQL: add the `default` sprig template function to logql label/line formatter (#6163) ... * add the `default` sprig template function to logql label/line formatter With the `default` sprig template function, we will be able to make it easier to output a default value if the source string is otherwise empty. Useful for json fields that can be missing, like HTTP headers in a log line that aren't required as in the following: ``` {job="access_log"} | json | line_format `{{.http_request_headers_x_forwarded_for | default "-"}}` ``` Previously this could only be done (that I found) with the much more verbose ``` {{regexReplaceAllLiteral "^$" .http_request_headers_x_forwarded_for "-"}} ``` * update changelog entry * add pr number * enhance docs * Update docs/sources/logql/template_functions.md Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> * Update docs/sources/logql/template_functions.md Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com> Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com> Co-authored-by: Vladyslav Diachenko <82767850+vlad-diachenko@users.noreply.github.com>	4 years ago
Christian Haudum	ee4ab1b394	Fix certain binary expression queries by optimizing through push down (#6132) ... * Add test cases that fail Instant queries of the following type fail and return an `unimplemented` error: ``` sum(count_over_time({foo="bar"} | logfmt | duration > 2s [3s])) / sum(count_over_time({foo="bar"} [3s])) ``` Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix certain binary expressions for instant queries If either the left hand side or the right hand side of a binary expression is a noop, we need to return the original expression so the whole expression is a noop as well, and thus not executed using the downstream engine. Otherwise, a binary expression that has a noop on either side, results in an `unimplemented` error when executed using the downstream engine. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Optimize instant vector aggregations with log extraction stage This change optimizes queries that use a vector aggregation without grouping around a range aggregation with a label extraction stage such as `json` or `logfmt`. Since the vector aggregation can be pushed down to the downstream query, the downstream query does not create a massive amount of streams, even though it contains a generic label extraction stage. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Add test cases that fail Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Add test cases that fail Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Kaviraj Kanagaraj	1389857846	`metrics.go` support for metadata queries(labels and series) (#5971) ... * Add different functions to record labels and series queries * Add tests for logging series and labels queries * Add statistics to the Series and Labels proto response types * Plug in the statistics recorder on the query handler * Remove duplicate imports and cleanup comments * Remove the need to add usage statistics for metadata queries. * Add status based on returned error from the api * Fixing lint error by fixin return values order * Use explicit logger * Remove unused recorders * Add `total_entries` representing length of total result returned * Inject statscollector middleware for metadata queries * Fix query frontend stats middleware Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tests with change in extra `totalEntriesReturned` field Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * `make check-generated-files` Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix typo Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix tripperware middleware for label values endpoint Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix bug with label extraction Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Add changelog entry Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Fix `total_entries` value for range query Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>	4 years ago
Owen Diehl	93de7a7061	renames skip -> ok in ProcessString (#6064) ... * renames skip -> ok in ProcessString Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * streampipeline matches convention Signed-off-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Susana Ferreira	6b6521f791	Add metrics to range mapper (#6030) ... * Refactor ShardingMetrics to MapperMetrics * Add mapper metrics to range mapper	4 years ago
Cyril Tovena	986fe5cf05	Remove empty match within concat of regex (#6026) ... * Remove empty match within concat of regex * add back tests	4 years ago
Christian Haudum	38e5e410fe	Do not parse string of empty matchers (#5980) ... This fixes a bug in the index gateway when querying values for a label. Since the gRPC handler for LabelValuesForMetricName in the index gateway allows empty matchers in the LabelValuesForMetricNameRequest, we need to check if the matchers string is an empty matcher (`{}`) before we parse the string. This bug was introduced with the index gateway api refactoring in #5892 Fixes: #5965 Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	4 years ago
Susana Ferreira	ea59e4bdc4	Add changelog entry for split by range of instant queries (#5966)	4 years ago
Travis Patterson	5939d5e9d8	Query filtering in the ingester and storage (#5629) ... * Add timestamp to pipeline and extractor interfaces * implement filter extractor and pipeline * implement storage filtering * implement ingester filtering * test cleanup * Refactor pipeline setup to util * linter * enable filtering based on mode * review feedback * review comments * review feedback * no need to include label filters in the test pipeline * review feedpack	4 years ago
Christian Haudum	99c5b35895	Add more function comments to the rangemapper implementation (#5861) ... * Add more function comments to the rangemapper implementation Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Apply suggestions from code review Co-authored-by: Susana Ferreira <susana.ferreira@grafana.com> Co-authored-by: Susana Ferreira <susana.ferreira@grafana.com>	4 years ago
Susana Ferreira	0bce8d95a2	Split by range of instant queries (#5662) ... * Split by range on Instant queries POC v3 Co-authored-by: Christian Haudum <christian.haudum@gmail.com> * Handle uneven split by duration * Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Register SplitByRangeMiddleware in roundtripper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove rewrite if range aggr has label extraction stage In case a range aggregation has a generic label extraction stage, such as `| json` or `| logfmt` and no group by, we cannot split it, because otherwise the downstream queries would result in too many series. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Implement range splitting for rate() and bytes_rate() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Calculate offset of downstream queries correctly if the outer query range contains an offset as well. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linting * Add optimization by moving the outer label grouping downstream * Add label grouping downstream optimization to rate and bytes_rate expressions * Add changelog entry Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Simplify types in rangemapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check in Map function if query is splittable by range Since this is the main function of the mapper, we can ensure here that only supported vector/range aggregations are handled. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Some code cleanups and variable renaming Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Extract duplicate code in range aggr mapping into function Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add topk to supported splittable vector aggregations Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Check if query is splittable by range before calling Map() Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add more function comments Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Rename RangeVectorMapper to RangeMapper Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix incorrect import due to rebase Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Add equivalence test cases with `logfmt` pipeline stage Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Remove TestRangeMappingEquivalenceMockMapper test This test is essentially the same as the test Test_SplitRangeVectorMapping, just using a different representation of the result. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * fixup! fixup! Remove limitation of pushing down vector aggr only if grouping is present Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Fix linter errors Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Better naming of variable Signed-off-by: Christian Haudum <christian.haudum@gmail.com> * Split SplitRangeVectorMapping test into two to have the test for noop queries separated Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	4 years ago
Ed Welch	f313a9ff4f	Build: update package for fuzzer after refactoring (#5776) ... * Update the package name to reflect the moved package after refactoring Signed-off-by: Ed Welch <edward.welch@grafana.com> * also include makefile Signed-off-by: Ed Welch <edward.welch@grafana.com>	4 years ago