prometheus

Commit Graph

Author	SHA1	Message	Date
Bryan Boreham	74b73d1e2c	Labels: Add DropMetricName function, used in PromQL (#13446 ) This function is called very frequently when executing PromQL functions, and we can do it much more efficiently inside Labels. In the common case that `__name__` comes first in the labels, we simply re-point to start at the next label, which is nearly free. `DropMetricName` is now so cheap I removed the cache - benchmarks show everything still goes faster. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Filip Petkovski	583f3e587c	Optimize histogram iterators (#13340 ) Optimize histogram iterators Histogram iterators allocate new objects in the AtHistogram and AtFloatHistogram methods, which makes calculating rates over long ranges expensive. In #13215 we allowed an existing object to be reused when converting an integer histogram to a float histogram. This commit follows the same idea and allows injecting an existing object in the AtHistogram and AtFloatHistogram methods. When the injected value is nil, iterators allocate new histograms, otherwise they populate and return the injected object. The commit also adds a CopyTo method to Histogram and FloatHistogram which is used in the BufferedIterator to overwrite items in the ring instead of making new copies. Note that a specialized HPoint pool is needed for all of this to work (`matrixSelectorHPool`). --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>	2 years ago
Ben Ye	d778591fd3	add more context cancellation check at evaluation time Signed-off-by: Ben Ye <benye@amazon.com>	2 years ago
zenador	72a8f1084b	Restore more efficient version of NewPossibleNonCounterInfo annotation (#13022 ) Restore more efficient version of NewPossibleNonCounterInfo annotation Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> --------- Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2 years ago
Filip Petkovski	0e1ae1d1ca	Add comment Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	2 years ago
Filip Petkovski	5df3820c7a	Copy last histogram point Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	2 years ago
Filip Petkovski	1f69dcfa6b	Fix reusing float histograms In https://github.com/prometheus/prometheus/pull/13276 we started reusing float histogram objects to reduce allocations in PromQL. That PR introduces a bug where histogram pointers gets copied to the beginning of the histograms slice, but are still kept in the end of the slice. When a new histogram is read into the last element, it can overwrite a previous element because the pointer is the same. This commit fixes the issue by moving outdated points to the end of the slice so that we don't end up with duplicate pointers in the same buffer. In other words, the slice gets rotated so that old objects can get reused. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	2 years ago
Filip Petkovski	bb8363dbb3	Add comment on SampleRingIterator Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	2 years ago
Filip Petkovski	e2a9f8ac0f	Reuse float histogram objects This commit reduces the memory needed to query native histogram objects by reusing existing HPoint instances. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	2 years ago
Linas Medziunas	7319ad6a0b	promql: simplify Native Histogram arithmetics Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>	2 years ago
beorn7	0eb0ca42c5	Update “conventional histogram” → “classic histogram” Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Julien Pivotto	c1ec6ae851	sort_by_label: Switch to feature flag Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	2 years ago
Alexander Trost	5051a993ab	promql: add sort_by_label and sort_by_label_desc functions This adds functions to sort a vector by its label value. Based on https://github.com/prometheus/prometheus/pull/1533 Signed-off-by: Alexander Trost <galexrt@googlemail.com>	2 years ago
Dimitar Dimitrov	9e3df532d8	Export `promql.Engine.FindMinMaxTime` This function is useful to analyze promQL queries. We want to use this in Mimir to record the time range which the query touches. I also chose to remove the `Engine` receiver because it was unnecessary, and it makes it easier to use, but happy to refactor that if you disagree. The function is untested on its own. If you prefer to have unit tests now that its exported, I can look into adding some. Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>	2 years ago
Oleksandr Redko	8e5f0387a2	ci(lint): enable nolintlint and remove redundant comments (#12926 ) Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>	2 years ago
Bryan Boreham	49c5e7afe1	PromQL: reduce garbage in range-query evaluation The temporary variable was allocated on the heap, and it is unnecessary. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
zenador	80e977aae6	Remove `NewPossibleNonCounterInfo` and minimise creating empty annotations (#13012 ) * Remove NewPossibleNonCounterInfo until it can be made more efficient, and avoid creating empty annotations as much as possible Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2 years ago
Marc Tuduri	af7c31ee10	PR feedback Signed-off-by: Marc Tuduri <marctc@protonmail.com>	2 years ago
Marc Tuduri	8fededf6ad	promql(histograms): Change sample total calculation for histograms Signed-off-by: Marc Tuduri <marctc@protonmail.com>	2 years ago
Alan Protasio	a15e884e7a	Prevent Prometheus from overallocating memory on subquery with large amount of steps. (#12734 ) * change initial points slice size Signed-off-by: Alan Protasio <alanprot@gmail.com> * refactor on the steps calculation and moving the getXPoint/putXPoint method to the evaluator Signed-off-by: Alan Protasio <alanprot@gmail.com> * prevent potential panic Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update promql/engine.go Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update promql/engine.go Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update promql/engine.go Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update promql/engine.go Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> * Allocating slice with maximum size of 5k Signed-off-by: Alan Protasio <alanprot@gmail.com> * adding comments Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: Alan Protasio <alanprot@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	2 years ago
zenador	69edd8709b	Add warnings (and annotations) to PromQL query results (#12152 ) Return annotations (warnings and infos) from PromQL queries This generalizes the warnings we have already used before (but only for problems with remote read) as "annotations". Annotations can be warnings or infos (the latter could be false positives). We do not treat them different in the API for now and return them all as "warnings". It would be easy to distinguish them and return infos separately, should that appear useful in the future. The new annotations are then used to create a lot of warnings or infos during PromQL evaluations. Partially these are things we have wanted for a long time (e.g. inform the user that they have applied `rate` to a metric that doesn't look like a counter), but the new native histograms have created even more needs for those annotations (e.g. if a query tries to aggregate float numbers with histograms). The annotations added here are not yet complete. A prominent example would be a warning about a range too short for a rate calculation. But such a warnings is more tricky to create with good fidelity and we will tackle it later. Another TODO is to take annotations into account when evaluating recording rules. --------- Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2 years ago
Arve Knudsen	6daee89e5f	Add context argument to Querier.Select (#12660 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Bryan Boreham	e4dd3469ac	lint Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	5ce990cabc	promql: simplify rangeEval a bit more We can't have both a float and a histogram at the same timestep. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	3879488476	promql: simplify inner loop of rangeEval Took out the loops with break after one iteration, and extract some common code to a function. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	0670e4771a	promql engine: check unique labels using existing map `ContainsSameLabelset` constructs a map with the same hash key as the one used to compile the output of `rangeEval`, so we can use that one and save work. Need to hold the timestamp so we can be sure we saw the same series in the same evaluation. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Charles Korn	6087c555ed	Address PR feedback: clarify comment Signed-off-by: Charles Korn <charles.korn@grafana.com>	2 years ago
Charles Korn	fb3935e8f9	Address PR feedback: rename method Signed-off-by: Charles Korn <charles.korn@grafana.com>	2 years ago
Julius Volz	531567d46e	Drop metric name for "atan2" binary operator The operator changes the meaning of the metric, so the metric name should be dropped. Technically this would be a breaking change, but it's also very obviously a bug and not likely that anyone depends on it. Signed-off-by: Julius Volz <julius.volz@gmail.com>	3 years ago
Charles Korn	fde6ebb17d	Create per-series iterators only once per selector, rather than recreating it for each time step. Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 years ago
Charles Korn	993618adea	Don't create a new iterator for every time step. Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 years ago
Charles Korn	b114c0888d	Simplify loop Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 years ago
Charles Korn	a142998052	Expand series set just once Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 years ago
Charles Korn	eeface2e17	Inline method Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 years ago
Charles Korn	a2a2cc757e	Extract timestamp special case to its own method. Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 years ago
Giedrius Statkevičius	3f230fc9f8	promql: convert QueryOpts to interface Convert QueryOpts to an interface so that downstream projects like https://github.com/thanos-community/promql-engine could extend the query options with engine specific options that are not in the original engine. Will be used to enable query analysis per-query. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	3 years ago
Bryan Boreham	67d2ef004d	Placate lint I think the version using scoping was better, but I'm out of energy to fight the linter. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	bb0d8320dd	promql: include parsing in active-query tracking So that the max-concurrency limit is applied. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	71fc4f1516	promql: refactor: create query object before parsing Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	1f3821379c	promql: refactor: extract fn to wait on concurrency limit Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
zenador	191bf9055b	Handle more arithmetic operators for native histograms (#12262 ) Handle more arithmetic operators and aggregators for native histograms This includes operators for multiplication (formerly known as scaling), division, and subtraction. Plus aggregations for average and the avg_over_time function. Stdvar and stddev will (for now) ignore histograms properly (rather than counting them but adding a 0 for them). Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	3 years ago
Justin Lei	7bbf24b707	Make MemoizedSeriesIterator not implement chunkenc.Iterator Signed-off-by: Justin Lei <justin.lei@grafana.com>	3 years ago
Justin Lei	6985dcbe73	Optimize and test MemoizedSeriesIterator Signed-off-by: Justin Lei <justin.lei@grafana.com>	3 years ago
Matthieu MOREL	7e9acc2e46	golangci-lint: remove skip-cache and restore singleCaseSwitch rule Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	3 years ago
beorn7	5b53aa1108	style: Replace `else if` cascades with `switch` Wiser coders than myself have come to the conclusion that a `switch` statement is almost always superior to a statement that includes any `else if`. The exceptions that I have found in our codebase are just these two: * The `if else` is followed by an additional statement before the next condition (separated by a `;`). * The whole thing is within a `for` loop and `break` statements are used. In this case, using `switch` would require tagging the `for` loop, which probably tips the balance. Why are `switch` statements more readable? For one, fewer curly braces. But more importantly, the conditions all have the same alignment, so the whole thing follows the natural flow of going down a list of conditions. With `else if`, in contrast, all conditions but the first are "hidden" behind `} else if `, harder to spot and (for no good reason) presented differently from the first condition. I'm sure the aforemention wise coders can list even more reasons. In any case, I like it so much that I have found myself recommending it in code reviews. I would like to make it a habit in our code base, without making it a hard requirement that we would test on the CI. But for that, there has to be a role model, so this commit eliminates all `if else` occurrences, unless it is autogenerated code or fits one of the exceptions above. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	c3c7d44d84	lint: Adjust to the lint warnings raised by current versions of golint-ci We haven't updated golint-ci in our CI yet, but this commit prepares for that. There are a lot of new warnings, and it is mostly because the "revive" linter got updated. I agree with most of the new warnings, mostly around not naming unused function parameters (although it is justified in some cases for documentation purposes – while things like mocks are a good example where not naming the parameter is clearer). I'm pretty upset about the "empty block" warning to include `for` loops. It's such a common pattern to do something in the head of the `for` loop and then have an empty block. There is still an open issue about this: https://github.com/mgechev/revive/issues/810 I have disabled "revive" altogether in files where empty blocks are used excessively, and I have made the effort to add individual `// nolint:revive` where empty blocks are used just once or twice. It's borderline noisy, though, but let's go with it for now. I should mention that none of the "empty block" warnings for `for` loop bodies were legitimate. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Ben Ye	fd3630b9a3	add ctx to QueryEngine interface Signed-off-by: Ben Ye <benye@amazon.com>	3 years ago
Matthieu MOREL	fb3eb21230	enable gocritic, unconvert and unused linters Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	3 years ago
beorn7	551de0346f	promql: Do not return nil slices to the pool Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	c0879d64cf	promql: Separate `Point` into `FPoint` and `HPoint` In other words: Instead of having a “polymorphous” `Point` that can either contain a float value or a histogram value, use an `FPoint` for floats and an `HPoint` for histograms. This seemingly small change has a _lot_ of repercussions throughout the codebase. The idea here is to avoid the increase in size of `Point` arrays that happened after native histograms had been added. The higher-level data structures (`Sample`, `Series`, etc.) are still “polymorphous”. The same idea could be applied to them, but at each step the trade-offs needed to be evaluated. The idea with this change is to do the minimum necessary to get back to pre-histogram performance for functions that do not touch histograms. Here are comparisons for the `changes` function. The test data doesn't include histograms yet. Ideally, there would be no change in the benchmark result at all. First runtime v2.39 compared to directly prior to this commit: ``` name old time/op new time/op delta RangeQuery/expr=changes(a_one[1d]),steps=1-16 391µs ± 2% 542µs ± 1% +38.58% (p=0.000 n=9+8) RangeQuery/expr=changes(a_one[1d]),steps=10-16 452µs ± 2% 617µs ± 2% +36.48% (p=0.000 n=10+10) RangeQuery/expr=changes(a_one[1d]),steps=100-16 1.12ms ± 1% 1.36ms ± 2% +21.58% (p=0.000 n=8+10) RangeQuery/expr=changes(a_one[1d]),steps=1000-16 7.83ms ± 1% 8.94ms ± 1% +14.21% (p=0.000 n=10+10) RangeQuery/expr=changes(a_ten[1d]),steps=1-16 2.98ms ± 0% 3.30ms ± 1% +10.67% (p=0.000 n=9+10) RangeQuery/expr=changes(a_ten[1d]),steps=10-16 3.66ms ± 1% 4.10ms ± 1% +11.82% (p=0.000 n=10+10) RangeQuery/expr=changes(a_ten[1d]),steps=100-16 10.5ms ± 0% 11.8ms ± 1% +12.50% (p=0.000 n=8+10) RangeQuery/expr=changes(a_ten[1d]),steps=1000-16 77.6ms ± 1% 87.4ms ± 1% +12.63% (p=0.000 n=9+9) RangeQuery/expr=changes(a_hundred[1d]),steps=1-16 30.4ms ± 2% 32.8ms ± 1% +8.01% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=10-16 37.1ms ± 2% 40.6ms ± 2% +9.64% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=100-16 105ms ± 1% 117ms ± 1% +11.69% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1000-16 783ms ± 3% 876ms ± 1% +11.83% (p=0.000 n=9+10) ``` And then runtime v2.39 compared to after this commit: ``` name old time/op new time/op delta RangeQuery/expr=changes(a_one[1d]),steps=1-16 391µs ± 2% 547µs ± 1% +39.84% (p=0.000 n=9+8) RangeQuery/expr=changes(a_one[1d]),steps=10-16 452µs ± 2% 616µs ± 2% +36.15% (p=0.000 n=10+10) RangeQuery/expr=changes(a_one[1d]),steps=100-16 1.12ms ± 1% 1.26ms ± 1% +12.20% (p=0.000 n=8+10) RangeQuery/expr=changes(a_one[1d]),steps=1000-16 7.83ms ± 1% 7.95ms ± 1% +1.59% (p=0.000 n=10+8) RangeQuery/expr=changes(a_ten[1d]),steps=1-16 2.98ms ± 0% 3.38ms ± 2% +13.49% (p=0.000 n=9+10) RangeQuery/expr=changes(a_ten[1d]),steps=10-16 3.66ms ± 1% 4.02ms ± 1% +9.80% (p=0.000 n=10+9) RangeQuery/expr=changes(a_ten[1d]),steps=100-16 10.5ms ± 0% 10.8ms ± 1% +3.08% (p=0.000 n=8+10) RangeQuery/expr=changes(a_ten[1d]),steps=1000-16 77.6ms ± 1% 78.1ms ± 1% +0.58% (p=0.035 n=9+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1-16 30.4ms ± 2% 33.5ms ± 4% +10.18% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=10-16 37.1ms ± 2% 40.0ms ± 1% +7.98% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=100-16 105ms ± 1% 107ms ± 1% +1.92% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1000-16 783ms ± 3% 775ms ± 1% -1.02% (p=0.019 n=9+9) ``` In summary, the runtime doesn't really improve with this change for queries with just a few steps. For queries with many steps, this commit essentially reinstates the old performance. This is good because the many-step queries are the one that matter most (longest absolute runtime). In terms of allocations, though, this commit doesn't make a dent at all (numbers not shown). The reason is that most of the allocations happen in the sampleRingIterator (in the storage package), which has to be addressed in a separate commit. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago

1 2 3 4 5 ...

338 Commits (0e8a2e44a4c54f70d1de052d52bb1304148ddfa5)