482 Commits (8b0d33e5b20475318a1a72857ed8a9947b13e1aa)

Author	SHA1	Message	Date
Matthieu MOREL	fb3eb21230	enable gocritic, unconvert and unused linters ... Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	3 years ago
beorn7	551de0346f	promql: Do not return nil slices to the pool ... Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	c0879d64cf	promql: Separate `Point` into `FPoint` and `HPoint` ... In other words: Instead of having a “polymorphous” `Point` that can either contain a float value or a histogram value, use an `FPoint` for floats and an `HPoint` for histograms. This seemingly small change has a _lot_ of repercussions throughout the codebase. The idea here is to avoid the increase in size of `Point` arrays that happened after native histograms had been added. The higher-level data structures (`Sample`, `Series`, etc.) are still “polymorphous”. The same idea could be applied to them, but at each step the trade-offs needed to be evaluated. The idea with this change is to do the minimum necessary to get back to pre-histogram performance for functions that do not touch histograms. Here are comparisons for the `changes` function. The test data doesn't include histograms yet. Ideally, there would be no change in the benchmark result at all. First runtime v2.39 compared to directly prior to this commit: ``` name old time/op new time/op delta RangeQuery/expr=changes(a_one[1d]),steps=1-16 391µs ± 2% 542µs ± 1% +38.58% (p=0.000 n=9+8) RangeQuery/expr=changes(a_one[1d]),steps=10-16 452µs ± 2% 617µs ± 2% +36.48% (p=0.000 n=10+10) RangeQuery/expr=changes(a_one[1d]),steps=100-16 1.12ms ± 1% 1.36ms ± 2% +21.58% (p=0.000 n=8+10) RangeQuery/expr=changes(a_one[1d]),steps=1000-16 7.83ms ± 1% 8.94ms ± 1% +14.21% (p=0.000 n=10+10) RangeQuery/expr=changes(a_ten[1d]),steps=1-16 2.98ms ± 0% 3.30ms ± 1% +10.67% (p=0.000 n=9+10) RangeQuery/expr=changes(a_ten[1d]),steps=10-16 3.66ms ± 1% 4.10ms ± 1% +11.82% (p=0.000 n=10+10) RangeQuery/expr=changes(a_ten[1d]),steps=100-16 10.5ms ± 0% 11.8ms ± 1% +12.50% (p=0.000 n=8+10) RangeQuery/expr=changes(a_ten[1d]),steps=1000-16 77.6ms ± 1% 87.4ms ± 1% +12.63% (p=0.000 n=9+9) RangeQuery/expr=changes(a_hundred[1d]),steps=1-16 30.4ms ± 2% 32.8ms ± 1% +8.01% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=10-16 37.1ms ± 2% 40.6ms ± 2% +9.64% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=100-16 105ms ± 1% 117ms ± 1% +11.69% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1000-16 783ms ± 3% 876ms ± 1% +11.83% (p=0.000 n=9+10) ``` And then runtime v2.39 compared to after this commit: ``` name old time/op new time/op delta RangeQuery/expr=changes(a_one[1d]),steps=1-16 391µs ± 2% 547µs ± 1% +39.84% (p=0.000 n=9+8) RangeQuery/expr=changes(a_one[1d]),steps=10-16 452µs ± 2% 616µs ± 2% +36.15% (p=0.000 n=10+10) RangeQuery/expr=changes(a_one[1d]),steps=100-16 1.12ms ± 1% 1.26ms ± 1% +12.20% (p=0.000 n=8+10) RangeQuery/expr=changes(a_one[1d]),steps=1000-16 7.83ms ± 1% 7.95ms ± 1% +1.59% (p=0.000 n=10+8) RangeQuery/expr=changes(a_ten[1d]),steps=1-16 2.98ms ± 0% 3.38ms ± 2% +13.49% (p=0.000 n=9+10) RangeQuery/expr=changes(a_ten[1d]),steps=10-16 3.66ms ± 1% 4.02ms ± 1% +9.80% (p=0.000 n=10+9) RangeQuery/expr=changes(a_ten[1d]),steps=100-16 10.5ms ± 0% 10.8ms ± 1% +3.08% (p=0.000 n=8+10) RangeQuery/expr=changes(a_ten[1d]),steps=1000-16 77.6ms ± 1% 78.1ms ± 1% +0.58% (p=0.035 n=9+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1-16 30.4ms ± 2% 33.5ms ± 4% +10.18% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=10-16 37.1ms ± 2% 40.0ms ± 1% +7.98% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=100-16 105ms ± 1% 107ms ± 1% +1.92% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1000-16 783ms ± 3% 775ms ± 1% -1.02% (p=0.019 n=9+9) ``` In summary, the runtime doesn't really improve with this change for queries with just a few steps. For queries with many steps, this commit essentially reinstates the old performance. This is good because the many-step queries are the one that matter most (longest absolute runtime). In terms of allocations, though, this commit doesn't make a dent at all (numbers not shown). The reason is that most of the allocations happen in the sampleRingIterator (in the storage package), which has to be addressed in a separate commit. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Łukasz Mierzwa	b6573353c1	Add query_samples_total metric ... query_samples_total is a counter that tracks the total number of samples loaded by all queries. The goal with this metric is to be able to see the amount of 'work' done by Prometheus to service queries. At the moment we have metrics with the number of queries, plus more detailed metrics showing how much time each step of a query takes. While those metrics do help they don't show us the whole picture. Queries that do load more samples are (in general) more expensive than queries that do load fewer samples. This means that looking only at the number of queries doesn't tell us how much 'work' Prometheus received. Adding a counter that tracks the total number of samples loaded allows us to see if there was a spike in the cost of queries, not just the number of them. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	3 years ago
Oleg Zaytsev	6e2905a4d4	Use zeropool.Pool to workaround SA6002 (#12189) ... * Use zeropool.Pool to workaround SA6002 I built a tiny library called https://github.com/colega/zeropool to workaround the SA6002 staticheck issue. While searching for the references of that SA6002 staticheck issues on Github first results was Prometheus itself, with quite a lot of ignores of it. This changes the usages of `sync.Pool` to `zeropool.Pool[T]` where a pointer is not available. Also added a benchmark for HeadAppender Append/Commit when series already exist, which is one of the most usual cases IMO, as I didn't find any. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Improve BenchmarkHeadAppender with more cases Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * A little copying is better than a little dependency https://www.youtube.com/watch?v=PAAkCSZUG1c&t=9m28s Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Fix imports order Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Add license header Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Copyright should be on one of the first 3 lines Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Use require.Equal for testing I don't depend on testify in my lib, but here we have it available. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Avoid flaky test Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Also use zeropool for pointsPool in engine.go Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> --------- Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	3 years ago
Bryan Boreham	f2fd85df82	promql: use faster heap method for topk/bottomk ... Call `Fix()` instead of `Pop()` followed by `Push()`. This is slightly faster. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	b987afa7ef	labels: simplify call to get Labels from Builder ... It took a `Labels` where the memory could be re-used, but in practice this hardly ever benefitted. Especially after converting `relabel.Process` to `relabel.ProcessBuilder`. Comparing the parameter to `nil` was a bug; `EmptyLabels` is not `nil` so the slice was reallocated multiple times by `append`. Lastly `Builder.Labels()` now estimates that the final size will depend on labels added and deleted. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	1b0a29701b	promql: optimise aggregation with no labels ... For a query like 'sum (foo)', we can quickly skip to the empty labels that its result needs. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	aafef011b7	Promql: reuse LabelBuilder in aggregations ... We have a LabelBuilder in EvalNodeHelper; use it instead of creating a new one at every step. Need to take some care that different uses of enh.lb do not overlap. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	2c382f5e24	promql: extract function to initialize LabelBuilder ... Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	56fefcd812	Update package promql for new labels.Labels type ... We use `labels.Builder` to parse metrics, to avoid depending on the internal implementation. This is not efficient, but the feature is only used in tests. It wasn't efficient previously either - calling `Sort()` after adding each label. `createLabelsForAbsentFunction` also uses a Builder now, and gets an extra `map` to replace the previous `Has()` usage. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> Fix up promql to compile with changes to Labels	3 years ago
Bryan Boreham	3c7de69059	storage: allow re-use of iterators ... Patterned after `Chunk.Iterator()`: pass the old iterator in so it can be re-used to avoid allocating a new object. (This commit does not do any re-use; it is just changing all the method signatures so re-use is possible in later commits.) Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Alan Protasio	8460807475	fix blank lines ... Signed-off-by: Alan Protasio <approtas@amazon.com>	3 years ago
Alan Protasio	f8f4ac14a8	Finishing evalSpanTimer always before return ... Signed-off-by: Alan Protasio <approtas@amazon.com>	3 years ago
Giedrius Statkevičius	a1d6ba59ac	promql: pass down subquery interval (#11163) ... If we are populating series for a subquery then set the interval parameter accordingly so that downstream users could use that information. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	3 years ago
Bryan Boreham	3330d85ba8	Replace sort.Strings and sort.Ints with faster slices.Sort (#11318) ... Use new experimental package `golang.org/x/exp/slices`. slices.Sort works on values that are directly comparable, like ints, so avoids the overhad of an interface call to `.Less()`. Left tests unchanged, because they don't need the speed and it may be a cross-check that slices.Sort gives the same answer. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Ganesh Vernekar	71489d0e3d	Fix count() for histograms and add test case ... Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Bryan Boreham	8b863c42dd	Optimise relabeling by re-using memory (#11147) ... * model/relabel: Add benchmark Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * model/relabel: re-use Builder across relabels Saves memory allocations. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * labels.Builder: allow re-use of result slice This reduces memory allocations where the caller has a suitable slice available. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * model/relabel: re-use source values slice To reduce memory allocations. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Unwind one change causing test failures Restore original behaviour in PopulateLabels, where we must not overwrite the input set. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * relabel: simplify values optimisation Use a stack-based array for up to 16 source labels, which will be the vast majority of cases. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * lint Signed-off-by: Bryan Boreham <bjboreham@gmail.com> Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Vilius Pranckaitis	4660656312	Allow setting custom lookback delta for instant queries (#9946) ... * Allow setting custom lookback delta for instant queries Signed-off-by: Vilius Pranckaitis <vpranckaitis@gmail.com>	3 years ago
Łukasz Mierzwa	54a3c3ba3f	Print query that caused a panic (#10995) ... We print the stacktrace of a panic when query causes one, but there's no information about the query itself, which makes it harder to debug and reproduce the issue. This adds the 'expr' string to the logged panic. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	3 years ago
Matthieu MOREL	0906f2eafa	refactor (promql): move from github.com/pkg/errors to 'errors' and 'fmt' (#10817) ... Signed-off-by: Matthieu MOREL <mmorel-35@users.noreply.github.com> Co-authored-by: Matthieu MOREL <mmorel-35@users.noreply.github.com>	4 years ago
Bryan Boreham	2e2c014d52	Labels: optimise creation of signature with/without labels (#10667) ... * Labels: create signature with/without labels Instead of creating a new Labels slice then converting to signature, go directly to the signature and save time. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Labels: refactor Builder tests Have one test with a range of cases, and have them check the final output rather than checking the internal structure of the Builder. Also add a couple of cases where the value is "", which should be interpreted as 'delete'. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Labels: add 'Keep' function to Builder This lets us replace `Labels.WithLabels` with the more general `Builder`. In `engine.resultMetric()` we can call `Keep()` instead of checking and calling `Del()`. Avoid calling `Sort()` in `Builder.Labels()` if we didn't add anything, so that `Keep()` has the same performance as `WithLabels()`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	4 years ago
Alan Protasio	ce6a643ee8	Changing TotalQueryableSamples from int to int64 (#10549) ... * Changing TotalQueryableSamples from int to int64 Signed-off-by: Alan Protasio <approtas@amazon.com>	4 years ago
beorn7	106e20cde5	Histogram: Fix and simplify histogram_quantile ... For conventional histograms, we need to gather all the individual bucket timeseries at a data point to do the quantile calculation. The code so far mirrored this behavior for the new native histograms. However, since a single data point contains all the buckets alreade, that's actually not needed. This PR simplifies the code while still detecting a mix of conventional and native histograms. The weird signature calculation for the conventional histograms is getting even weirder because of that. If this PR turns out to do the right thing, I will implement a proper fix for the signature calculation upstream. Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Andrew Bloomgarden	a64b9fe323	Report PeakSamples in query statistics ... This exactly corresponds to the statistic compared against MaxSamples during the course of query execution, so users can see how close their queries are to a limit. Co-authored-by: Harkishen Singh <harkishensingh@hotmail.com> Co-authored-by: Andrew Bloomgarden <blmgrdn@amazon.com> Signed-off-by: Andrew Bloomgarden <blmgrdn@amazon.com>	4 years ago
Alan Protasio	606ef33d91	Track and report Samples Queried per query ... We always track total samples queried and add those to the standard set of stats queries can report. We also allow optionally tracking per-step samples queried. This must be enabled both at the engine and query level to be tracked and rendered. The engine flag is exposed via a Prometheus feature flag, while the query flag is set when stats=all. Co-authored-by: Alan Protasio <approtas@amazon.com> Co-authored-by: Andrew Bloomgarden <blmgrdn@amazon.com> Co-authored-by: Harkishen Singh <harkishensingh@hotmail.com> Signed-off-by: Andrew Bloomgarden <blmgrdn@amazon.com>	4 years ago
beorn7	9fbcf14e5c	histogram: Handle changes of the ZeroThreshold and the Schema ... Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Julien Pivotto	9a2e93228e	Switch to grafana/regexp everywhere (#10268) ... Let's have a consistent library for regexp. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Peter Štibraný	6d76f09c58	Extract interface from ActivityQueryTracker and allows passing custom implementation (#10071) ... * Extract interface from ActivityQueryTracker and allows passing custom implementation. Signed-off-by: Peter Štibraný <pstibrany@gmail.com>	4 years ago
Matej Gera	2c61d29b2a	Tracing: Migrate to OpenTelemetry library (#9724) ... Signed-off-by: Matej Gera <matejgera@gmail.com>	4 years ago
beorn7	b39f2739e5	PromQL: Always enable negative offset and @ modifier ... This follows the line of argument that the invariant of not looking ahead of the query time was merely emerging behavior and not a documented stable feature. Any query that looks ahead of the query time was simply invalid before the introduction of the negative offset and the @ modifier. Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	53ca375345	promql: Add a guard against a nil histogram in sum aggregation ... This can happen if the aggregation starts with a float and later encounters a histogram. In that case, the newly encountered histogram would have been added to a nil histogram. This should be tested, of course, but that's best done within the PromQL testing framework, which we still need to enable for histograms (for which we have a TODO in the code and now also a card in the GH project). Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Ganesh Vernekar	f580248759	Support + operator for sparse histograms (#9949) ... Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	4 years ago
Ganesh Vernekar	187a767292	Implement sum() for sparse histograms (#9948) ... Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	4 years ago
Ganesh Vernekar	4a43349aca	`histogram_quantile` for sparse histograms (#9935) ... * MergeFloatBucketIterator for []FloatBucketIterator Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * histogram_quantile for histograms Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix histogram_quantile Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Unit test and enhancements Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Iterators to iterate buckets in reverse and all buckets together including zero bucket Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Consider all buckets for histogram_quantile and fix the implementation Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Remove unneeded code Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix lint Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	4 years ago
Björn Rabenstein	4ce01e9770	storage: Rename ...Values methods to At... (#9889) ... This mirrors #9888 for the richer iterators we have with histograms in the game. Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Björn Rabenstein	d677aa4b29	storage: Consolidate iterator method names (Values -> At) (#9888) ... `BufferedSeriesIterator` and `MemoizedSeriesIterator` use a method called `Values` for exactly the purpose for which all other iterators of the same kind use a method called `At`. That alone is confusing, but on top of that, the `Values` method only returns a single sample, not multiple values. I assume the naming has historical reasons. This commit makes it more consistent. It is now easier to read, and now `BufferedSeriesIterator` and `MemoizedSeriesIterator` implement `chunkenc.Iterator` like many other iterators, too. Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Björn Rabenstein	7e42acd3b1	tsdb: Rework iterators (#9877) ... - Pick At... method via return value of Next/Seek. - Do not clobber returned buckets. - Add partial FloatHistogram suppert. Note that the promql package is now _only_ dealing with FloatHistograms, following the idea that PromQL only knows float values. As a byproduct, I have removed the histogramSeries metric. In my understanding, series can have both float and histogram samples, so that metric doesn't make sense anymore. As another byproduct, I have converged the sampleBuf and the histogramSampleBuf in memSeries into one. The sample type stored in the sampleBuf has been extended to also contain histograms even before this commit. Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	8e4e8726bb	promql: Fix another ChunkEncoding call ... Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	9de3ab60df	promql: improve histogram support in engine.go ... Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	73858d7f82	storage: histogram support in memoized_iterator ... Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	4c28d9fac7	Move to histogram.Histogram pointers ... This is to avoid copying the many fields of a histogram.Histogram all the time. This also fixes a bunch of formerly broken tests. Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Thomas Jackson	f0003bc0ba	Don't drop ParenExpr when creating StepInvariantExpr (#9591) ... * Add test case to showcase the problem in #9590 Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com> * Don't unwrap ParenExpr in newStepInvariantExpr Fixes #9590 Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>	4 years ago
beorn7	c954cd9d1d	Move packages out of deprecated pkg directory ... This creates a new `model` directory and moves all data-model related packages over there: exemplar labels relabel rulefmt textparse timestamp value All the others are more or less utilities and have been moved to `util`: gate logging modetimevfs pool runtime Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	8f92c90897	Add TODOs and some minor tweaks ... Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Ganesh Vernekar	c8b267efd6	Get histograms from TSDB to the rate() function implementation ... Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	4 years ago
Mateusz Gozdek	1a6c2283a3	Format Go source files using 'gofumpt -w -s -extra' ... Part of #9557 Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>	4 years ago
Bryan Boreham	a278ea4b58	promql: copy data when short-circuiting (#9552) ... * promql: copy data when short-circuiting Because the range query loop re-uses the output buffer each time round, we must copy results into the buffer rather than using input as output. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	4 years ago
Julien Pivotto	a18224d02d	make aggregations deterministic (#9459) ... * Add deterministic test for aggregations Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> * Make aggregations deterministic Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu> * Increase testing Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
ziollek	55f9147b44	Add atan2 to scalar operators - issue #9485 (#9515) ... * Add atan2 to scalar operators Signed-off-by: Tomasz Ziolkowski <tomasz.ziolkowski@allegro.pl>	4 years ago