80 Commits (9fe8bcf4be00a7ebd73932df0cf39cc837cbb993)

Author	SHA1	Message	Date
Brian Brazil	dd6781add2	Optimise PromQL (#3966) ... * Move range logic to 'eval' Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make aggregegate range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * PromQL is statically typed, so don't eval to find the type. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Extend rangewrapper to multiple exprs Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Start making function evaluation ranged Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make instant queries a special case of range queries Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Eliminate evalString Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Evaluate range vector functions one series at a time Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make unary operators range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make binops range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Pass time to range-aware functions. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make simple _over_time functions range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reduce allocs when working with matrix selectors Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Add basic benchmark for range evaluation Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reuse objects for function arguments Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Do dropmetricname and allocating output vector only once. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Add range-aware support for range vector functions with params Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Optimise holt_winters, cut cpu and allocs by ~25% Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make rate&friends range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make more functions range aware. Document calling convention. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make date functions range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make simple math functions range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Convert more functions to be range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make more functions range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Specialcase timestamp() with vector selector arg for range awareness Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove transition code for functions Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove the rest of the engine transition code Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove more obselete code Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove the last uses of the eval* functions Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove engine finalizers to prevent corruption The finalizers set by matrixSelector were being called just before the value they were retruning to the pool was then being provided to the caller. Thus a concurrent query could corrupt the data that the user has just been returned. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Add new benchmark suite for range functinos Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Migrate existing benchmarks to new system Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Expand promql benchmarks Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Simply test by removing unused range code Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * When testing instant queries, check range queries too. To protect against subsequent steps in a range query being affected by the previous steps, add a test that evaluates an instant query that we know works again as a range query with the tiimestamp we care about not being the first step. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reuse ring for matrix iters. Put query results back in pool. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reuse buffer when iterating over matrix selectors Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Unary minus should remove metric name Cut down benchmarks for faster runs. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reduce repetition in benchmark test cases Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Work series by series when doing normal vectorSelectors Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Optimise benchmark setup, cuts time by 60% Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Have rangeWrapper use an evalNodeHelper to cache across steps Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Use evalNodeHelper with functions Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Cache dropMetricName within a node evaluation. This saves both the calculations and allocs done by dropMetricName across steps. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reuse input vectors in rangewrapper Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reuse the point slices in the matrixes input/output by rangeWrapper Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make benchmark setup faster using AddFast Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Simplify benchmark code. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Add caching in VectorBinop Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Use xor to have one-level resultMetric hash key Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Add more benchmarks Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Call Query.Close in apiv1 This allows point slices allocated for the response data to be reused by later queries, saving allocations. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Optimise histogram_quantile It's now 5-10% faster with 97% less garbage generated for 1k steps Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make the input collection in rangeVector linear rather than quadratic Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Optimise label_replace, for 1k steps 15x fewer allocs and 3x faster Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Optimise label_join, 1.8x faster and 11x less memory for 1k steps Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Expand benchmarks, cleanup comments, simplify numSteps logic. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Address Fabian's comments Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Comments from Alin. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Address jrv's comments Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove dead code Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Address Simon's comments. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Rename populateIterators, pre-init some sizes Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Handle case where function has non-matrix args first Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Split rangeWrapper out to rangeEval function, improve comments Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Cleanup and make things more consistent Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make EvalNodeHelper public Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Fabian's comments. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	8 years ago
David King	6286c10df0	Fix OOM when a large K is used in topk queries (#4087) ... This attempts to close #3973. Handles cases where the length of the input vector to an aggregate topk / bottomk function is less than the K paramater. The change updates Prometheus to allocate a result vector the same length as the input vector in these cases. Previously Prometheus would out-of-memory panic for large K values. This change makes that unlikely unless the size of the input vector is equally large. Signed-off-by: David King <dave@davbo.org>	8 years ago
Krasi Georgiev	a53d4ed197	drop metric name for bool modifier (#3821) ... fixes #3820	8 years ago
Krasi Georgiev	4801573b64	time() return milliseconds (#3811)	8 years ago
Brian Brazil	b418063d1a	Add tests for negative selectors. (#3616) ... https://github.com/prometheus/prometheus/issues/3575	8 years ago
Brian Brazil	99905f82a6	Remove keep_common modifier. ... See #3060	8 years ago
Brian Brazil	b2ac3d2d86	Remove count_scalar and drop_common_labels. ... For #3060	8 years ago
Brian Brazil	67274f0794	Remove 4 interval staleness heuristic. (#3244) ... This means that if there is no stale marker, only the usual staleness delta (5m) applies. It has occured to me that there is an oddity in the heurestic. It works fine as long as you have 2 points within the last 5m, but breaks down when the time window advances to the point where you have just 1 point. Consider you had points at t=0 and t=10. With the heurestic it goes stale at t=51, up until t=300. However from t=301 until t=310 we only see the t=10 point and the series comes back to life. That is not desirable. I don't see a way to keep this form of heurestic working given this issue, so thus I'm removing it.	8 years ago
Harsh Agarwal	16867c89a7	implement label_join issue 1147 (#2806) ... Replace OptionalArgs int with Variadic int.	9 years ago
Brian Brazil	220e78b9c3	Consider a series stale after 4.1 intervals with no data. ... To cover the cases where stale markers may not be available, we need to infer the interval and mark series stale based on that. As we're lacking stale markers this is less accurate, however it should be good enough for these cases. We need 4 intervals as if say we had data at t=0 and t=10, coming via federation. The next data point should be at t=20 however it could take up to t=30 for it actually to be ingested, t=40 for it to be scraped via federation and t=50 for it to be ingested. We then add 10% on to that for slack, as we do elsewhere.	9 years ago
Brian Brazil	80b40e6d91	Add initial staleness handing to promql. ... For instant vectors, if "stale" is the newest sample ignore the timeseries. For range vectors, filter out "stale" samples. Make it possible to inject "stale" samples in promql tests.	9 years ago
Brian Brazil	517b81f927	Add timestamp() function. ... Make the timestamp of instant vectors be the timestamp of the sample rather than the evaluation. We were not using this anywhere, so this is safe. Add a function to return the timestamp of samples in an instant vector. Fixes #1557	9 years ago
Jack Neely	896f951e68	Force buckets in a histogram to be monotonic for quantile estimation (#2610) ... * Force buckets in a histogram to be monotonic for quantile estimation The assumption that bucket counts increase monotonically with increasing upperBound may be violated during: * Recording rule evaluation of histogram_quantile, especially when rate() has been applied to the underlying bucket timeseries. * Evaluation of histogram_quantile computed over federated bucket timeseries, especially when rate() has been applied This is because scraped data is not made available to RR evalution or federation atomically, so some buckets are computed with data from the N most recent scrapes, but the other buckets are missing the most recent observations. Monotonicity is usually guaranteed because if a bucket with upper bound u1 has count c1, then any bucket with a higher upper bound u > u1 must have counted all c1 observations and perhaps more, so that c >= c1. Randomly interspersed partial sampling breaks that guarantee, and rate() exacerbates it. Specifically, suppose bucket le=1000 has a count of 10 from 4 samples but the bucket with le=2000 has a count of 7, from 3 samples. The monotonicity is broken. It is exacerbated by rate() because under normal operation, cumulative counting of buckets will cause the bucket counts to diverge such that small differences from missing samples are not a problem. rate() removes this divergence.) bucketQuantile depends on that monotonicity to do a binary search for the bucket with the qth percentile count, so breaking the monotonicity guarantee causes bucketQuantile() to return undefined (nonsense) results. As a somewhat hacky solution until the Prometheus project is ready to accept the changes required to make scrapes atomic, we calculate the "envelope" of the histogram buckets, essentially removing any decreases in the count between successive buckets. * Fix up comment docs for ensureMonotonic * ensureMonotonic: Use switch statement Use switch statement rather than if/else for better readability. Process the most frequent cases first.	9 years ago
beorn7	4e3abc6cbf	Simply use `math.Mod(float64, float64)` after all ... This circumvents all the problems with int overflow, plus it is what was originally intended.	9 years ago
beorn7	5cf5bb427a	Check for int64 overflow when converting from float64	9 years ago
beorn7	07f1bdfe94	Fix MOD binop for scalars and vectors ... Previously, a floating point number that would round down to 0 would cause a "division by zero" panic.	9 years ago
Fabian Reinartz	ccbce0c51f	promql: handle NaN in changes() correctly	9 years ago
Matt Bostock	a0201036fa	PromQL: Add tests for time/date funcs with arg ... Add tests for the date and time functions where an argument is specified. Suggested by @grobie: https://github.com/prometheus/prometheus/pull/1984#issuecomment-246508286 `1136239445` is the reference time used by Go: https://golang.org/src/time/format.go	9 years ago
Matt Bostock	9628eb5998	PromQL: Add minute() function ... Returns the minutes from the current time in UTC. Related to the `hour()` function. Fixes #1983.	9 years ago
Brian Brazil	4680daf237	Default date functions to current time.	9 years ago
Brian Brazil	ea1318f38b	Short names of some date related functions	10 years ago
Brian Brazil	d2ca2b496a	Add days_in_month function.	10 years ago
Brian Brazil	fd7822829c	Add date related functions. ... Add day_of_month, day_of_week, hour_of_day, month_of_year and year. This only work for UTC, and ignore leap seconds the same as Go.	10 years ago
Fabian Stäber	08b6556ee6	Assume counters start at zero after reset.	10 years ago
Frederic Branczyk	dbf83666bb	add idelta function ... similar to the irate function the idelta function calculates the delta function with the last two values	10 years ago
Frederic Branczyk	0ce5e7fe6d	move legacy test for delta function	10 years ago
Brian Brazil	0303ccc6a7	Add quantile aggregator.	10 years ago
Brian Brazil	b0342ba9ec	Add quantile_over_time function	10 years ago
Brian Brazil	1edd6875f5	Add stddev_over_time and stdvar_over_time.	10 years ago
Brian Brazil	16690736ab	Add count_values() aggregator. ... This is useful for counting how many instances of a job are running a particular version/build. Fixes #622	10 years ago
Brian Brazil	fa9cc15573	Add topk/bottomk tests for multiple buckets.	10 years ago
Brian Brazil	3b0c182eee	Move topk/bottomk unittests over to aggregators.	10 years ago
Brian Brazil	3b89616d82	Allow on, ignoring, by and without wit empty laberls. ... This offers new semantics in allowing on() for matching two single-element vectors with no known common labels. Previosuly this was often done using on(dummy). This also allows making it explict that you meant to do an aggregation without labels via by(). Fixes #1597.	10 years ago
royels	2fdc5717a3	promql: add power binary operation	10 years ago
Brian Brazil	ef55fd6176	Add unittest for using a metric for thresholds with group_left.	10 years ago
Brian Brazil	d991f0cf47	For many-to-one matches, always copy label from one side. ... This is a breaking change for everyone using the machine roles labeling approach.	10 years ago
Brian Brazil	768d09fd2a	Change on+group_* to take copy from the one side. ... If the label doesn't exist on the one side, it's not copied. All labels on the many inside are included, this is a breaking change but likely low impact.	10 years ago
Brian Brazil	d1edfb25b3	Add support for OneToMany with IGNORING. ... The labels listed in the group_ modifier will be copied from the one side to the many side. It will be valid to specify no labels. This is intended to replace the existing ON/GROUP_* support.,	10 years ago
Brian Brazil	1d08c4fef0	Add 'ignoring' as modifier for binops. ... Where 'on' uses the given labels to match, 'ignoring' uses all other labels to match. group_left/right is not supported yet.	10 years ago
Brian Brazil	f5084ab1c5	Add tests for group_left/group_right	10 years ago
Tobias Schmidt	8cc86f25c0	Implement relative complement set operator "unless" ... The `unless` set operator can be used to return all vector elements from the LHS which do not match the elements on the RHS. A use case is to return all metrics for nodes which do not have a specific role: node_load1 unless on(instance) chef_role{role="app"}	10 years ago
Tobias Schmidt	4c3dc25e35	Fix whitespace in promql test data	10 years ago
eliothedeman	1543ef92b2	Adds holt-winters query function	10 years ago
Brian Brazil	8788701ce7	Add test for incorrect behaviour	10 years ago
Brian Brazil	39d556f0d5	Move all the operator tests into one file	10 years ago
beorn7	79a2ae2d2e	Add missing test file	10 years ago
beorn7	c740789ce3	Improve predict_linear ... Fixes https://github.com/prometheus/prometheus/issues/1401 This remove the last (and in fact bogus) use of BoundaryValues. Thus, a whole lot of unused (and arguably sub-optimal / ugly) code can be removed here, too.	10 years ago
Brian Brazil	9d0112d7cf	Add without aggregator modifier. ... This has the advantage that the user doesn't need to list all labels they want to keep (as with "by") but without having to worry about inconsistent labels as when there's only one time series (as with "keeping_common"). Almost all aggregation should use this rather than the existing two options as it's much less error prone and easier to maintain due to not having to always add in "job" plus whatever other common job-level labels you have like "region".	10 years ago
Brian Brazil	b7ef0b45e8	Break aggregation tests out. Add missing tests.	10 years ago
Brian Brazil	c77c3a8c56	promql: Limit extrapolation of delta/rate/increase ... The new implementation detects the start and end of a series by looking at the average sample interval within the range. If the first (last) sample in the range is more than 1.1*interval distant from the beginning (end) of the range, it is considered the first (last) sample of the series as a whole, and extrapolation is limited to half the interval (rather than all the way to the beginning (end) of the range). In addition, if the extrapolated starting point of a counter (where it is zero) is within the range, it is used as the starting point of the series. Fixes #581	10 years ago