prometheus

Commit Graph

Author	SHA1	Message	Date
George Krajcsovits	5b7ff92d95	fix(promql): histogram_quantile and histogram_fraction NaN observed in native histogram (#16724 ) * fix(promql): histogram_quantile NaN observed in native histogram Fixes: #16578 See the issue for detailed explanation. When a histogram had only NaN observations and no normal observations, we returned 0 from the quantile, which is completely wrong. If there were normal observations but we went over them, we returned the upper bound of the existing buckets, however that contradicts expectations on histogram_fraction. Now we return NaN if the quantile is calculated to be over all normal observations, falling into NaNs (in a virtual +Inf bucket). We also return info level annotations if we see any NaN observations. The annotation calls out if we returned NaN or even if we took the virtual +Inf bucket into account. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * fix(promql): histogram_fraction NaN observed in native histogram Fixes: #16580 According to the specification we should not take NaN observations into account when calculating the fraction. This commit fixes that and adds an info level annotation to let the user know about this. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	7 months ago
Michael Hoffmann	a5fa9431d8	promql: add ts_of_(max,min,last)_over_time functions This commit adds the ts_of_(max,min,last)_over_time functions behind the experimental feature flag. Signed-off-by: Michael Hoffmann <mhoffmann@cloudflare.com>	7 months ago
Julius Volz	13564c03ef	Standardize doc page title handling See https://groups.google.com/g/prometheus-developers/c/cwL3cM66Em8 Signed-off-by: Julius Volz <julius.volz@gmail.com>	8 months ago
Aman	26bddcf068	promql: histogram_stddev and histogram_stdvar should use arithmetic mean for custom buckets Signed-off-by: amanycodes <amanycodes@gmail.com>	9 months ago
Michael Hoffmann	d6d9f97bac	promql: histogram_fraction for bucket histograms (#16095 ) * promql: histogram_fraction for bucket histograms This PR extends the histogram_fraction function to also work with classic bucket histograms. This is beneficial because it allows expressions like sum(increase(my_bucket{le="0.5"}[10m]))/sum(increase(my_total[10m])) to be written without knowing the actual values of the "le" label, easing the transition to native histograms later on. It also feels natural since histogram_quantile also can deal with classic histograms. Signed-off-by: Michael Hoffmann <mhoffmann@cloudflare.com> * promql: histogram_fraction for bucket histograms * Add documentation and reduce code duplication * Fix a bug in linear interpolation between bucket boundaries * Add more PromQL tests Signed-off-by: Michael Hoffmann <mhoffmann@cloudflare.com> * Update docs/querying/functions.md Co-authored-by: Björn Rabenstein <github@rabenste.in> Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> --------- Signed-off-by: Michael Hoffmann <mhoffmann@cloudflare.com> Signed-off-by: Michael Hoffmann <mhoffm@posteo.de> Co-authored-by: Björn Rabenstein <github@rabenste.in>	9 months ago
beorn7	993d4e71c9	docs: Update PromQL documentation to match the native histogram spec This is also meant to document the actual implementation, but see #13934 for the current state. This also improves and streamlines some parts of the documentation that are not strictly native histogram related, but are colocated with them. In particular, the section about aggregation operators got restructured quite a bit, including the removal of a quite verbose example for `limit_ratio` (which was just too long an this location and also a bit questionabl in its usefulness). Signed-off-by: beorn7 <beorn@grafana.com>	11 months ago
asymmetric	96f31e370b	doc: clarify `rate` values are averaged (#14045 ) * doc: clarify `rate` values are averaged Signed-off-by: asymmetric <101816+asymmetric@users.noreply.github.com> Co-authored-by: Björn Rabenstein <github@rabenste.in>	11 months ago
Kateryna Pavlova	d6e43b89f4	docs: add regex section in querying basics Signed-off-by: Kateryna Pavlova <kathrinett@gmail.com>	1 year ago
Arve Knudsen	de16f5e387	[FEATURE] PromQL: Add experimental info function MVP (#14495 ) The `info` function is an experiment to improve UX around including labels from info metrics. `info` has to be enabled via the feature flag `--enable-feature=promql-experimental-functions`. This MVP of info simplifies the implementation by assuming: * Only support for the target_info metric * That target_info's identifying labels are job and instance Also: * Encode info samples' original timestamp as sample value * Deduce info series select hints from top-most VectorSelector --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Ying WANG <ying.wang@grafana.com> Co-authored-by: Augustin Husson <augustin.husson@amadeus.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Jan Fajerski	96e5a94d29	promql: rename holt_winters to double_exponential_smoothing Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	1 year ago
beorn7	6fcd225aee	promql(native histograms): Introduce exponential interpolation The linear interpolation (assuming that observations are uniformly distributed within a bucket) is a solid and simple assumption in lack of any other information. However, the exponential bucketing used by standard schemas of native histograms has been chosen to cover the whole range of observations in a way that bucket populations are spread out over buckets in a reasonably way for typical distributions encountered in real-world scenarios. This is the origin of the idea implemented here: If we divide a given bucket into two (or more) smaller exponential buckets, we "most naturally" expect that the samples in the original buckets will split among those smaller buckets in a more or less uniform fashion. With this assumption, we end up with an "exponential interpolation", which therefore appears to be a better match for histograms with exponential bucketing. This commit leaves the linear interpolation in place for NHCB, but changes the interpolation for exponential native histograms to exponential. This affects `histogram_quantile` and `histogram_fraction` (because the latter is more or less the inverse of the former). The zero bucket has to be treated specially because the assumption above would lead to an "interpolation to zero" (the bucket density approaches infinity around zero, and with the postulated uniform usage of buckets, we would end up with an estimate of zero for all quantiles ending up in the zero bucket). We simply fall back to linear interpolation within the zero bucket. At the same time, this commit makes the call to stick with the assumption that the zero bucket only contains positive observations for native histograms without negative buckets (and vice versa). (This is an assumption relevant for interpolation. It is a mostly academic point, as the zero bucket is supposed to be very small anyway. However, in cases where it _is_ relevantly broad, the assumption helps a lot in practice.) This commit also updates and completes the documentation to match both details about interpolation. As a more high level note: The approach here attempts to strike a balance between a more simplistic approach without any assumption, and a more involved approach with more sophisticated assumptions. I will shortly describe both for reference: The "zero assumption" approach would be to not interpolate at all, but _always_ return the harmonic mean of the bucket boundaries of the bucket the quantile ends up in. This has the advantage of minimizing the maximum possible relative error of the quantile estimation. (Depending on the exact definition of the relative error of an estimation, there is also an argument to return the arithmetic mean of the bucket boundaries.) While limiting the maximum possible relative error is a good property, this approach would throw away the information if a quantile is closer to the upper or lower end of the population within a bucket. This can be valuable trending information in a dashboard. With any kind of interpolation, the maximum possible error of a quantile estimation increases to the full width of a bucket (i.e. it more than doubles for the harmonic mean approach, and precisely doubles for the arithmetic mean approach). However, in return the _expectation value_ of the error decreases. The increase of the theoretical maximum only has practical relevance for pathologic distributions. For example, if there are thousand observations within a bucket, they could _all_ be at the upper bound of the bucket. If the quantile calculation picks the 1st observation in the bucket as the relevant one, an interpolation will yield a value close to the lower bucket boundary, while the true quantile value is close to the upper boundary. The "fancy interpolation" approach would be one that analyses the _actual_ distribution of samples in the histogram. A lot of statistics could be applied based on the information we have available in the histogram. This would include the population of neighboring (or even all) buckets in the histogram. In general, the resolution of a native histogram should be quite high, and therefore, those "fancy" approaches would increase the computational cost quite a bit with very little practical benefits (i.e. just tiny corrections of the estimated quantile value). The results are also much harder to reason with. Signed-off-by: beorn7 <beorn@grafana.com>	1 year ago
Jan Fajerski	15cea39136	promql: put holt_winters behind experimental feature flag Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	1 year ago
suntala	ce4eac859a	Link to specific feature flag entry Signed-off-by: suntala <arati.rana@grafana.com>	1 year ago
suntala	532904a1d6	Document changes to sort by label Co-authored-by: Aleks Fazlieva <britishrum@users.noreply.github.com> Signed-off-by: suntala <arati.rana@grafana.com>	1 year ago
suntala	77d111e501	Fix links to feature flags page Signed-off-by: suntala <arati.rana@grafana.com>	1 year ago
咸鱼暄	bab098a4c1	change all lists to bullets Signed-off-by: 咸鱼暄 <30610597+smd1121@users.noreply.github.com>	2 years ago
咸鱼暄	ad03ede602	fix markdown list Signed-off-by: 咸鱼暄 <30610597+smd1121@users.noreply.github.com>	2 years ago
Rick Rackow	9290d1308d	fix(docs/querying): explain `ceil` behaviour more explicitly with examples (#11987 ) * fix(docs/querying): explain `ceil` behaviour more explicitly with examples Signed-off-by: Rick Rackow <rick.rackow@gmail.com> * fix(docs/querying): explain `floor` behaviour more explicitly with examples Signed-off-by: Rick Rackow <rick.rackow@paymenttools.com> --------- Signed-off-by: Rick Rackow <rick.rackow@gmail.com> Signed-off-by: Rick Rackow <rick.rackow@paymenttools.com> Co-authored-by: Rick Rackow <rick.rackow@paymenttools.com>	2 years ago
Charles Korn	76b1237215	Document sorting behaviour Signed-off-by: Charles Korn <charles.korn@grafana.com>	2 years ago
Faustas Butkus	6feffeb92e	promql: add histogram_avg function (#13467 ) Add histogram_avg function --------- Signed-off-by: Faustas Butkus <faustas.butkus@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: Björn Rabenstein <github@rabenste.in>	2 years ago
Björn Rabenstein	89523cf9b3	doc: Mark `mad_over_time` as experimental (#13440 ) We forgot to do that in https://github.com/prometheus/prometheus/pull/13059 Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Ivan Babrou	a6b35ff304	promql: use natural sort in sort_by_label and sort_by_label_desc (#13411 ) These functions are intended for humans, as robots can already sort the results however they please. Humans like things sorted "naturally": * https://blog.codinghorror.com/sorting-for-humans-natural-sort-order/ A similar thing has been done to Grafana, which is also used by humans: * https://github.com/grafana/grafana/pull/78024 * https://github.com/grafana/grafana/pull/78494 Signed-off-by: Ivan Babrou <github@ivan.computer>	2 years ago
Jeanette Tan	9bf4cc993e	Add mad_over_time function Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2 years ago
beorn7	0eb0ca42c5	Update “conventional histogram” → “classic histogram” Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Julien Pivotto	c1ec6ae851	sort_by_label: Switch to feature flag Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	2 years ago
Alexander Trost	5051a993ab	promql: add sort_by_label and sort_by_label_desc functions This adds functions to sort a vector by its label value. Based on https://github.com/prometheus/prometheus/pull/1533 Signed-off-by: Alexander Trost <galexrt@googlemail.com>	2 years ago
zenador	ccfe14d7e7	PromQL: ignore small errors for bucketQuantile (#13153 ) promql: Improve histogram_quantile calculation for classic buckets Tiny differences between classic buckets are most likely caused by floating point precision issues. With this commit, relative changes below a certain threshold are ignored. This makes the result of histogram_quantile more meaningful, and also avoids triggering the _input to histogram_quantile needed to be fixed for monotonicity_ annotations in unactionable cases. This commit also adds explanation of the new adjustment and of the monotonicity annotation to the documentation of `histogram_quantile`. --------- Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2 years ago
Julien Pivotto	ac0919d48c	Update docs/querying/functions.md Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	2 years ago
lasea75	f15f0ac16a	Update functions.md Signed-off-by: lasea75 <lasea75@gmail.com>	2 years ago
zenador	54aaa2bd7e	Add `histogram_stdvar` and `histogram_stddev` functions (#12614 ) * Add new function: histogram_stdvar and histogram_stddev Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2 years ago
Salar Nosrati-Ershad	fd96996b75	docs: fix: correct reference to native histograms feature flag Signed-off-by: Salar Nosrati-Ershad <snosratiershad@gmail.com>	2 years ago
Guillaume Berche	f5fb37dbab	Update functions.md Add missing linefeed as requested by https://github.com/prometheus/prometheus/pull/11404#discussion_r1266625301 Signed-off-by: Guillaume Berche <guillaume.berche@orange.com>	3 years ago
Guillaume Berche	60b380da70	Refine functions.md as suggested during review See https://github.com/prometheus/prometheus/pull/11404#issuecomment-1631165746 Signed-off-by: Guillaume Berche <guillaume.berche@orange.com>	3 years ago
Ziqi Zhao	42d9169ba1	enhance histogram_quantile to get min/max value Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	3 years ago
Carrie Edwards	f93ac97867	Update querying function docs Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>	3 years ago
marcoderama	a308ea773d	Update functions.md Fix small typo Signed-off-by: marcoderama <marcoderamagit@gmail.com>	3 years ago
zenador	191bf9055b	Handle more arithmetic operators for native histograms (#12262 ) Handle more arithmetic operators and aggregators for native histograms This includes operators for multiplication (formerly known as scaling), division, and subtraction. Plus aggregations for average and the avg_over_time function. Stdvar and stddev will (for now) ignore histograms properly (rather than counting them but adding a 0 for them). Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	3 years ago
beorn7	c0879d64cf	promql: Separate `Point` into `FPoint` and `HPoint` In other words: Instead of having a “polymorphous” `Point` that can either contain a float value or a histogram value, use an `FPoint` for floats and an `HPoint` for histograms. This seemingly small change has a _lot_ of repercussions throughout the codebase. The idea here is to avoid the increase in size of `Point` arrays that happened after native histograms had been added. The higher-level data structures (`Sample`, `Series`, etc.) are still “polymorphous”. The same idea could be applied to them, but at each step the trade-offs needed to be evaluated. The idea with this change is to do the minimum necessary to get back to pre-histogram performance for functions that do not touch histograms. Here are comparisons for the `changes` function. The test data doesn't include histograms yet. Ideally, there would be no change in the benchmark result at all. First runtime v2.39 compared to directly prior to this commit: ``` name old time/op new time/op delta RangeQuery/expr=changes(a_one[1d]),steps=1-16 391µs ± 2% 542µs ± 1% +38.58% (p=0.000 n=9+8) RangeQuery/expr=changes(a_one[1d]),steps=10-16 452µs ± 2% 617µs ± 2% +36.48% (p=0.000 n=10+10) RangeQuery/expr=changes(a_one[1d]),steps=100-16 1.12ms ± 1% 1.36ms ± 2% +21.58% (p=0.000 n=8+10) RangeQuery/expr=changes(a_one[1d]),steps=1000-16 7.83ms ± 1% 8.94ms ± 1% +14.21% (p=0.000 n=10+10) RangeQuery/expr=changes(a_ten[1d]),steps=1-16 2.98ms ± 0% 3.30ms ± 1% +10.67% (p=0.000 n=9+10) RangeQuery/expr=changes(a_ten[1d]),steps=10-16 3.66ms ± 1% 4.10ms ± 1% +11.82% (p=0.000 n=10+10) RangeQuery/expr=changes(a_ten[1d]),steps=100-16 10.5ms ± 0% 11.8ms ± 1% +12.50% (p=0.000 n=8+10) RangeQuery/expr=changes(a_ten[1d]),steps=1000-16 77.6ms ± 1% 87.4ms ± 1% +12.63% (p=0.000 n=9+9) RangeQuery/expr=changes(a_hundred[1d]),steps=1-16 30.4ms ± 2% 32.8ms ± 1% +8.01% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=10-16 37.1ms ± 2% 40.6ms ± 2% +9.64% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=100-16 105ms ± 1% 117ms ± 1% +11.69% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1000-16 783ms ± 3% 876ms ± 1% +11.83% (p=0.000 n=9+10) ``` And then runtime v2.39 compared to after this commit: ``` name old time/op new time/op delta RangeQuery/expr=changes(a_one[1d]),steps=1-16 391µs ± 2% 547µs ± 1% +39.84% (p=0.000 n=9+8) RangeQuery/expr=changes(a_one[1d]),steps=10-16 452µs ± 2% 616µs ± 2% +36.15% (p=0.000 n=10+10) RangeQuery/expr=changes(a_one[1d]),steps=100-16 1.12ms ± 1% 1.26ms ± 1% +12.20% (p=0.000 n=8+10) RangeQuery/expr=changes(a_one[1d]),steps=1000-16 7.83ms ± 1% 7.95ms ± 1% +1.59% (p=0.000 n=10+8) RangeQuery/expr=changes(a_ten[1d]),steps=1-16 2.98ms ± 0% 3.38ms ± 2% +13.49% (p=0.000 n=9+10) RangeQuery/expr=changes(a_ten[1d]),steps=10-16 3.66ms ± 1% 4.02ms ± 1% +9.80% (p=0.000 n=10+9) RangeQuery/expr=changes(a_ten[1d]),steps=100-16 10.5ms ± 0% 10.8ms ± 1% +3.08% (p=0.000 n=8+10) RangeQuery/expr=changes(a_ten[1d]),steps=1000-16 77.6ms ± 1% 78.1ms ± 1% +0.58% (p=0.035 n=9+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1-16 30.4ms ± 2% 33.5ms ± 4% +10.18% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=10-16 37.1ms ± 2% 40.0ms ± 1% +7.98% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=100-16 105ms ± 1% 107ms ± 1% +1.92% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1000-16 783ms ± 3% 775ms ± 1% -1.02% (p=0.019 n=9+9) ``` In summary, the runtime doesn't really improve with this change for queries with just a few steps. For queries with many steps, this commit essentially reinstates the old performance. This is good because the many-step queries are the one that matter most (longest absolute runtime). In terms of allocations, though, this commit doesn't make a dent at all (numbers not shown). The reason is that most of the allocations happen in the sampleRingIterator (in the storage package), which has to be addressed in a separate commit. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Chance Feick	52270d6216	Fix relative link to use .md file extension Signed-off-by: Chance Feick <cfeick@gitlab.com>	3 years ago
Björn Rabenstein	41035469d3	Document the native histogram feature flag and PromQL (#11446 ) Signed-off-by: beorn7 <beorn@grafana.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Guillaume Berche	e6b84ac1e0	functions.md doc refinement Removed named capture group example in label_replace Signed-off-by: Guillaume Berche <guillaume.berche@orange.com>	3 years ago
Guillaume Berche	ea40c15fe9	Refine documentation for label_replace Illustrate use of named capturing group syntax available from https://github.com/google/re2/wiki/Syntax and their usage in the replacement field Signed-off-by: Guillaume Berche <guillaume.berche@orange.com>	3 years ago
beorn7	9eafed0f79	promql: Add `histogram_count` and `histogram_sum` This follow a simple function-based approach to access the count and sum fields of a native Histogram. It might be more elegant to implement “accessors” via the dot operator, as considered in the brainstorming doc [1]. However, that would require the introduction of a whole new concept in PromQL. For the PoC, we should be fine with the function-based approch. Even the obvious inefficiencies (rate'ing a whole histogram twice when we only want to rate each the count and the sum once) could be optimized behind the scenes. Note that the function-based approach elegantly solves the problem of detecting counter resets in the sum of observations in the case of negative observations. (Since the whole native Histogram is rate'd, the counter reset is detected for the Histogram as a whole.) We will decide later if an “accessor” approach is really needed. It would change the example expression for average duration in functions.md from histogram_sum(rate(http_request_duration_seconds[10m])) / histogram_count(rate(http_request_duration_seconds[10m])) to rate(http_request_duration_seconds.sum[10m]) / rate(http_request_duration_seconds.count[10m]) [1]: https://docs.google.com/document/d/1ch6ru8GKg03N02jRjYriurt-CZqUVY09evPg6yKTA1s/edit Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	a3a8f58bb3	promql: Add histogram_fraction function Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Julien Pivotto	42f574d5ac	Remove "This function was added in Prometheus 2.0" (#10719 ) Since Prometheus documentation is versioned, do not write down that a specific function was added in Prom 2.0, for consistency. Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	4 years ago
Ivo Gosemann	e22b54e253	Adds day_of_year function to PromQL Signed-off-by: Ivo Gosemann <ivo.gosemann@sap.com>	4 years ago
Matthew	b6630876d3	remove impl details from docs Signed-off-by: mtfoley <mtfoley.mae@gmail.com>	4 years ago
jyz0309	02e032884a	add doc Signed-off-by: jyz0309 <45495947@qq.com>	4 years ago
Matthew	57b86cfe9e	docs: update function docs on deriv Signed-off-by: mtfoley <mtfoley.mae@gmail.com>	4 years ago
Levi Harrison	6faca22eec	Add inverse hyperbolic functions Signed-off-by: Levi Harrison <git@leviharrison.dev>	4 years ago

1 2

78 Commits (4eafbcae93d4848bcf064a4dcbf2a3ca7157ed43)