prometheus

Commit Graph

Author	SHA1	Message	Date
Neeraj Gartia	8b0d33e5b2	promql: support variable scalar parameter in aggregations in range queries (#16404 ) This fixes the regression introduced in https://github.com/prometheus/prometheus/issues/15971 while preserving the performance improvements. Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	7 months ago
Charles Korn	2c0f02a702	promql: don't emit a value from `histogram_fraction` or `histogram_quantile` if classic and native histograms are present at the same timestamp Signed-off-by: Charles Korn <charles.korn@grafana.com>	7 months ago
Arve Knudsen	e7e3ab2824	Fix linting issues found by golangci-lint v2.0.2 (#16368 ) * Fix linting issues found by golangci-lint v2.0.2 --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	8 months ago
Michael Hoffmann	d6d9f97bac	promql: histogram_fraction for bucket histograms (#16095 ) * promql: histogram_fraction for bucket histograms This PR extends the histogram_fraction function to also work with classic bucket histograms. This is beneficial because it allows expressions like sum(increase(my_bucket{le="0.5"}[10m]))/sum(increase(my_total[10m])) to be written without knowing the actual values of the "le" label, easing the transition to native histograms later on. It also feels natural since histogram_quantile also can deal with classic histograms. Signed-off-by: Michael Hoffmann <mhoffmann@cloudflare.com> * promql: histogram_fraction for bucket histograms * Add documentation and reduce code duplication * Fix a bug in linear interpolation between bucket boundaries * Add more PromQL tests Signed-off-by: Michael Hoffmann <mhoffmann@cloudflare.com> * Update docs/querying/functions.md Co-authored-by: Björn Rabenstein <github@rabenste.in> Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> --------- Signed-off-by: Michael Hoffmann <mhoffmann@cloudflare.com> Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> Co-authored-by: Björn Rabenstein <github@rabenste.in>	8 months ago
Julien Pivotto	2f6ad79edf	Move calculation to the engine Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>	9 months ago
Neeraj Gartia	9084c5247f	fix subquery bug Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	9 months ago
Charles Korn	8356990709	promql: fix double quoting in invalid label name error from `count_values` (#16054 ) promql: fix double quoting in invalid label name error from `count_values` --------- Signed-off-by: Charles Korn <charles.korn@grafana.com> Signed-off-by: Charles Korn <charleskorn@users.noreply.github.com> Co-authored-by: Björn Rabenstein <github@rabenste.in>	10 months ago
Matthieu MOREL	c7d4b53ec1	chore: enable unused-parameter from revive Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	10 months ago
beorn7	14bb63c467	promql: Fix counter reset detection in subqueries with histograms Signed-off-by: beorn7 <beorn@grafana.com>	10 months ago
Matthieu MOREL	b472ce7010	chore: enable early-return from revive Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	10 months ago
Charles Korn	69ce0c24db	Fix issue Signed-off-by: Charles Korn <charles.korn@grafana.com>	10 months ago
Neeraj Gartia	8be416a67c	[FIX] PromQL: Updates annotation for bin op between incompatible histograms (#15895 ) PromQL: Updates annotation for bin op between incompatible histograms --------- Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	11 months ago
Linas Medziunas	940016e002	promql: use histogram stats decoder for histogram_avg Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>	11 months ago
machine424	9823a93c42	fix(main.go): avoid closing the query engine until it is guaranteed to no longer be in use. partially reverts https://github.com/prometheus/prometheus/pull/14064 fixes https://github.com/prometheus/prometheus/issues/15232 supersedes https://github.com/prometheus/prometheus/pull/15533 reusing Engine.Close() outside of tests will require more consideration. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	12 months ago
Joel Beckmeyer	41dabfb464	fix topk/bottomk with numbers greater than int maxsize on 32-bit Signed-off-by: Joel Beckmeyer <joel@beckmeyer.us>	1 year ago
TJ Hoplock	4d54c304f8	ref: make query logger more efficient by building list of attrs Addresses PR feedback to make query logger more efficient. Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	1 year ago
TJ Hoplock	e0104a6b7e	ref: JSONFileLogger slog handler, add scrape.FailureLogger interface Improves upon #15434, better resolves #15433. This commit introduces a few changes to ensure safer handling of the JSONFileLogger: - the JSONFileLogger struct now implements the slog.Handler interface, so it can directly be used to create slog Loggers. This pattern more closely aligns with upstream slog usage (which is generally based around handlers), as well as making it clear that devs are creating a whole new logger when interacting with it (vs silently modifying internal configs like it did previously). - updates the `promql.QueryLogger` interface to be a union of the methods of both the `io.Closer` interface and the `slog.Handler` interface. This allows for plugging in/using slog-compatible loggers other than the JSONFileLogger, if desired (ie, for downstream projects). - introduces new `scrape.FailureLogger` interface; just like `promql.QueryLogger`, it is a union of `io.Closer` and `slog.Handler` interfaces. Similar logic applies to reasoning. - updates tests where needed; have the `FakeQueryLogger` from promql's engine_test implement the `slog.Handler`, improve JSONFileLogger test suite, etc. Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	1 year ago
Neeraj Gartia	38bb6ece25	[BUGFIX] PromQL: Fix behaviour of some aggregations with histograms (#15432 ) promql: fix some aggregations for histograms This PR fixes the behaviour of `topk`,`bottomk`, `limitk` and `limit_ratio` with histograms. The fixed behaviour are as follows: - For `topk` and `bottomk` histograms are ignored and add info annotations added. - For `limitk` and `limit_ratio` histograms are included in the results(if applicable). Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	1 year ago
TJ Hoplock	3e24e84172	fix!: stop unbounded memory usage from query log Resolves: #15433 When I converted prometheus to use slog in #14906, I update both the `QueryLogger` interface, as well as how the log calls to the `QueryLogger` were built up in `promql.Engine.exec()`. The backing logger for the `QueryLogger` in the engine is a `util/logging.JSONFileLogger`, and it's implementation of the `With()` method updates the logger the logger in place with the new keyvals added onto the underlying slog.Logger, which means they get inherited onto everything after. All subsequent calls to `With()`, even in later queries, would continue to then append on more and more keyvals for the various params and fields built up in the logger. In turn, this causes unbounded growth of the logger, leading to increased memory usage, and in at least one report was the likely cause of an OOM kill. More information can be found in the issue and the linked slack thread. This commit does a few things: - It was referenced in feedback in #14906 that it would've been better to not change the `QueryLogger` interface if possible, this PR proposes changes that bring it closer to alignment with the pre-3.0 `QueryLogger` interface contract - reverts `promql.Engine.exec()`'s usage of the query logger to the pattern of building up an array of args to pass at once to the end log call. Avoiding the repetitious calls to `.With()` are what resolve the issue with the logger growth/memory usage. - updates the scrape failure logger to use the update `QueryLogger` methods in the contract. - updates tests accordingly - cleans up unused methods Builds and passes tests successfully. Tested locally and confirmed I could no longer reproduce the issue/it resolved the issue. Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	1 year ago
beorn7	4b573e0521	promql: Fix subqueries to be really left-open Previously, we managed to get rid of the sample on the left bound later, so the problem didn't show up in the framework tests. But the subqueries were still evaluation with the sample on the left bound, taking space and showing up if returning the subquery result directly (without further processing through PromQL like in all the framework tests). Signed-off-by: beorn7 <beorn@grafana.com>	1 year ago
Neeraj Gartia	048222867a	fix count_values for histograms Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	1 year ago
Charles Korn	62e6e55c07	promql: fix issues with comparison binary operations with `bool` modifier and native histograms (#15413 ) * Fix issue where comparison operations with `bool` modifier and native histograms return histograms rather than 0 or 1 * Don't emit anything for comparisons between floats and histograms when `bool` modifier is used * Don't emit anything for comparisons between floats and histograms when `bool` modifier is used between a vector and a scalar --------- Signed-off-by: Charles Korn <charles.korn@grafana.com>	1 year ago
Charles Korn	45db23617a	promql: fix incorrect "native histogram ignored in aggregation" annotations (#15414 ) Signed-off-by: Charles Korn <charles.korn@grafana.com>	1 year ago
huochexizhan	4f48e76086	chore: fix some function names in comment Signed-off-by: huochexizhan <huochexizhan@outlook.com>	1 year ago
Neeraj Gartia	789c9b1a5e	[BUGFIX] PromQL: Corrects the behaviour of some operator and aggregators with Native Histograms (#15245 ) PromQL: Correct the behaviour of some operator and aggregators with Native Histograms --------- Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	1 year ago
Matthieu MOREL	af1a19fc78	enable errorf rule from perfsprint linter Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	1 year ago
Joshua Hesketh	5a4e4f6936	Fix stddev/stdvar when aggregating histograms, NaNs, and infinities (#14941 ) promql: Fix stddev/stdvar when aggregating histograms, NaNs, and Infs Native histograms are ignored when calculating stddev or stdvar. However, for the first series of each group, a `groupedAggregation` is always created. If the first series that was encountered is a histogram then it acts as the equivalent of a 0 point. This change creates the first `groupedAggregation` with the `seen` field set to `false` if the point is a histogram, thus ignoring it like the rest of the aggregation function does. A new `groupedAggregation` will then be created once an actual float value is encountered. This commit also sets the `floatValue` field of the `groupedAggregation` to `NaN`, if the first float value of a group is `NaN` or `±Inf`, so that the outcome is consistently `NaN` once those values are in the mix. (The added tests fail without this change). Signed-off-by: Joshua Hesketh <josh@nitrotech.org> Signed-off-by: beorn7 <beorn@grafana.com> --------- Signed-off-by: Joshua Hesketh <josh@nitrotech.org> Signed-off-by: beorn7 <beorn@grafana.com> Co-authored-by: beorn7 <beorn@grafana.com>	1 year ago
Arve Knudsen	de16f5e387	[FEATURE] PromQL: Add experimental info function MVP (#14495 ) The `info` function is an experiment to improve UX around including labels from info metrics. `info` has to be enabled via the feature flag `--enable-feature=promql-experimental-functions`. This MVP of info simplifies the implementation by assuming: * Only support for the target_info metric * That target_info's identifying labels are job and instance Also: * Encode info samples' original timestamp as sample value * Deduce info series select hints from top-most VectorSelector --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Ying WANG <ying.wang@grafana.com> Co-authored-by: Augustin Husson <augustin.husson@amadeus.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Arve Knudsen	e05e97cdd7	evaluator.rangeEval: Split out gatherVector method Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	1 year ago
Arve Knudsen	f7b396a1dc	promql.Engine: Refactor vector selector evaluation into a method (#14900 ) New method is named `evalVectorSelector`. --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	1 year ago
Neeraj Gartia	d4b1f9eb33	Corrects the behaviour of binary opperators between histogram and float (#14726 ) promql: corrects binary operators functioning for mixed sample with histogram and float For invalid pairings of sample types, an annotation is added now. Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com> --------- Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	1 year ago
TJ Hoplock	6ebfbd2d54	chore!: adopt log/slog, remove go-kit/log For: #14355 This commit updates Prometheus to adopt stdlib's log/slog package in favor of go-kit/log. As part of converting to use slog, several other related changes are required to get prometheus working, including: - removed unused logging util func `RateLimit()` - forward ported the util/logging/Deduper logging by implementing a small custom slog.Handler that does the deduping before chaining log calls to the underlying real slog.Logger - move some of the json file logging functionality to use prom/common package functionality - refactored some of the new json file logging for scraping - changes to promql.QueryLogger interface to swap out logging methods for relevant slog sugar wrappers - updated lots of tests that used/replicated custom logging functionality, attempting to keep the logical goal of the tests consistent after the transition - added a healthy amount of `if logger == nil { $makeLogger }` type conditional checks amongst various functions where none were provided -- old code that used the go-kit/log.Logger interface had several places where there were nil references when trying to use functions like `With()` to add keyvals on the new *slog.Logger type Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	1 year ago
Arve Knudsen	c2bbabb4a7	promql.Engine: Refactor vector selector evaluation into a method (#14900 ) * PromQL.Engine: Refactor Matrix expansion into a method Add utility method promql.evaluator.expandSeriesToMatrix, for expanding a slice of storage.Series into a promql.Matrix. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Rename to generateMatrix Rename evaluator.expandSeriesToMatrix into generateMatrix, while also dropping the start, end, interval arguments since they are evaluator fields. Write more extensive method documentation. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Rename to evalVectorSelector Rename to evalVectorSelector after discussing with @michahoffmann. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	1 year ago
Björn Rabenstein	1639450172	Merge pull request #14821 from charleskorn/nh-negative-multiplication-division promql: correctly handle unary negation of native histograms and add tests for multiplication and division of native histograms by negative scalars Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Joshua Hesketh	b6107cc888	Make rate possible non-counter annotation consistent (#14910 ) * Make rate possible non-counter annotation consistent Previously a PossibleNonCounterInfo annotation would be left in cases where a range-vector selects 1 float data point, even if no more points are selected in order to calculate a rate. This change ensures an output float exists before emitting such an annotation. This fixes an inconsistency where a series with mixed data (ie, a float and a native histogram) would emit an annotation without any points. For example, ``` load 1m series{label="a"} 1 {{schema:1 sum:10 count:5 buckets:[1 2 3]}} eval instant at 1m rate(series[1m1s]) ``` Would have a PossibleNonCounterInfo annotation. Wheras ``` load 1m series{label="a"} {{schema:1 sum:10 count:5 buckets:[1 2 3]}} {{schema:1 sum:15 count:10 buckets:[1 2 3]}} eval instant at 1m rate(series[1m1s]) ``` Would not. --------- Signed-off-by: Joshua Hesketh <josh@nitrotech.org>	1 year ago
Charles Korn	9852855084	Implement unary negation for native histograms Signed-off-by: Charles Korn <charles.korn@grafana.com>	1 year ago
Arve Knudsen	db5e48dc33	promql.Engine.Close: No-op if nil (#14861 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	1 year ago
Bryan Boreham	abb0502685	[ENHANCEMENT] PromQL: Add detail to tracing spans For aggregates, operators, calls, show what operation is performed. Also add an event when series are expanded, typically time spent accessing TSDB. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Bryan Boreham	8742077498	[BUGFIX] PromQL: pass Context so spans parent correctly Assigning to `evaluator.ctx` in `eval()` broke the parent-child relationship. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Jorge Creixell	e9e3d64b7c	PromQL engine: Delay deletion of __name__ label to the end of the query evaluation (#14477 ) PromQL engine: Delay deletion of __name__ label to the end of the query evaluation - This change allows optionally preserving the `__name__` label via the `label_replace` and `label_join` functions, and helps prevent the dreaded "vector cannot contain metrics with the same labelset" error. - The implementation extends the `Series` and `Sample` structs with a boolean flag indicating whether the `__name__` label should be deleted at the end of the query evaluation. - The `label_replace` and `label_join` functions can still access the value of the `__name__` label, even if it has been previously marked for deletion. If `__name__` is used as target label, it won't be dropped at the end of the query evaluation. - Fixes https://github.com/prometheus/prometheus/issues/11397 - See https://github.com/jcreixell/prometheus/pull/2 for previous discussion, including the decision to create this PR and benchmark it before considering other alternatives (like refactoring `labels.Labels`). - See https://github.com/jcreixell/prometheus/pull/1 for an alternative implementation using a special label instead of boolean flags. - Note: a feature flag `promql-delayed-name-removal` has been added as it changes the behavior of some "weird" queries (see https://github.com/prometheus/prometheus/issues/11397#issuecomment-1451998792) Example (this always fails, as `__name__` is being dropped by `count_over_time`): ``` count_over_time({__name__!=""}[1m]) => Error executing query: vector cannot contain metrics with the same labelset ``` Before: ``` label_replace(count_over_time({__name__!=""}[1m]), "__name__", "count_$1", "__name__", "(.+)") => Error executing query: vector cannot contain metrics with the same labelset ``` After: ``` label_replace(count_over_time({__name__!=""}[1m]), "__name__", "count_$1", "__name__", "(.+)") => count_go_gc_cycles_automatic_gc_cycles_total{instance="localhost:9090", job="prometheus"} 1 count_go_gc_cycles_forced_gc_cycles_total{instance="localhost:9090", job="prometheus"} 1 ... ``` Signed-off-by: Jorge Creixell <jcreixell@gmail.com> --------- Signed-off-by: Jorge Creixell <jcreixell@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in>	1 year ago
beorn7	0f760f63dd	lint: Revamp our linting rules, mostly around doc comments Several things done here: - Set `max-issues-per-linter` to 0 so that we actually see all linter warnings and not just 50 per linter. (As we also set `max-same-issues` to 0, I assume this was the intention from the beginning.) - Stop using the golangci-lint default excludes (by setting `exclude-use-default: false`. Those are too generous and don't match our style conventions. (I have re-added some of the excludes explicitly in this commit. See below.) - Re-add the `errcheck` exclusion we have used so far via the defaults. - Exclude the signature requirement `govet` has for `Seek` methods because we use non-standard `Seek` methods a lot. (But we keep other requirements, while the default excludes completely disabled the check for common method segnatures.) - Exclude warnings about missing doc comments on exported symbols. (We used to be pretty adamant about doc comments, but stopped that at some point in the past. By now, we have about 500 missing doc comments. We may consider reintroducing this check, but that's outside of the scope of this commit. The default excludes of golangci-lint essentially ignore doc comments completely.) - By stop using the default excludes, we now get warnings back on malformed doc comments. That's the most impactful change in this commit. It does not enforce doc comments (again), but _if_ there is a doc comment, it has to have the recommended form. (Most of the changes in this commit are fixing this form.) - Improve wording/spelling of some comments in .golangci.yml, and remove an outdated comment. - Leave `package-comments` inactive, but add a TODO asking if we should change that. - Add a new sub-linter `comment-spacings` (and fix corresponding comments), which avoids missing spaces after the leading `//`. Signed-off-by: beorn7 <beorn@grafana.com>	1 year ago
cuiweiyuan	1800af54f0	chore: fix some function names Signed-off-by: cuiweiyuan <cuiweiyuan@aliyun.com>	1 year ago
Arve Knudsen	0503d4f372	PromQL: Fix comment regarding non-nil histogram pointer Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	1 year ago
Charles Korn	82bb35fabb	Address PR feedback: fix typo and rename variable Signed-off-by: Charles Korn <charles.korn@grafana.com>	1 year ago
George Krajcsovits	1d7fe4be5c	Update promql/engine.go Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>	1 year ago
György Krajcsovits	909785b97f	Fix histogram pool poisoning bu chunkenc.Iterator chunkenc.Iterator.AtFloatHistogram may do a shallow copy if it receives nil as input pointer. This can in turn share the span slice with multiple histograms in the matrixSelectorHPool, leading to unexpected errors. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
Charles Korn	f91009aa2e	promql: clarify error message when panic occurs during query evaluation Signed-off-by: Charles Korn <charles.korn@grafana.com>	1 year ago
George Krajcsovits	17b0b788da	Update promql/engine.go Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>	1 year ago
Charles Korn	0f4bc87b4f	Make linter happy Signed-off-by: Charles Korn <charles.korn@grafana.com>	1 year ago
Charles Korn	f07b3ae67b	Fix issue where `avg` over mixed exponential and custom buckets, or incompatible custom buckets, produces incorrect results or panics Signed-off-by: Charles Korn <charles.korn@grafana.com>	1 year ago

1 2 3 4 5 ...

482 Commits (8b0d33e5b20475318a1a72857ed8a9947b13e1aa)