prometheus

Commit Graph

Author	SHA1	Message	Date
Callum Styan	9bca041285	WIP: keep track of samples per query, set a max # of samples (#4513 ) * keep track of samples per query, set a max # of samples that can be in memory at once Signed-off-by: Callum Styan <callumstyan@gmail.com>	7 years ago
Tom Wilkie	4c52400708	Limit concurrent remote reads. (#4656 ) Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Harsh Agarwal	18a9a390b5	Add duplicate-labelset check for range/instant vectors (#4589 ) Signed-off-by: Harsh Agarwal <cs15btech11019@iith.ac.in>	7 years ago
Ganesh Vernekar	576ee4d309	Label name check for 'count_values' (#4585 ) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	7 years ago
Dan Cech	9f4cb06a37	use Welford/Knuth method to compute standard deviation and variance (#4533 ) * use Welford/Knuth method to compute standard deviation and variance, avoids float precision issues * use better method for calculating avg and avg_over_time Signed-off-by: Dan Cech <dcech@grafana.com>	7 years ago
Goutham Veeramachaneni	71855a22a4	Add tracing spans to promql (#4436 ) * Add spans to promql Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Simplify timer and span tracking. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago
Thomas Jackson	56daa1f28a	Only add LookbackDelta to vector selectors (#4399 ) Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com> Related to #4226	8 years ago
Alin Sinpalean	372e7652b7	Reuse (copy) overlapping matrix samples between range evaluation steps (#4315 ) * Reuse (copy) overlapping matrix samples between range evaluation steps. Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>	8 years ago
Tony Lee	bcdaf8e2d2	add unused pointslices to the pool (#4363 ) Signed-off-by: Tony Lee <tl@hudson-trading.com>	8 years ago
Alin Sinpalean	e3b775b78b	Simplify BufferedSeriesIterator usage (#4294 ) * Allow for BufferedSeriesIterator instances to be created without an underlying iterator, to simplify their usage. Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>	8 years ago
Julius Volz	219e477272	Fix some (valid) lint errors (#4287 ) Signed-off-by: Julius Volz <julius.volz@gmail.com>	8 years ago
Thomas Jackson	92c6f0c92e	Add offset to selectParams (#4226 ) * Add Start/End to SelectParams * Make remote read use the new selectParams for start/end This commit will continue sending the start/end time of the remote read query as the overarching promql time and the specific range of data that the query is intersted in receiving a response to is now part of the ReadHints (upstream discussion in #4226). * Remove unused vendored code The genproto.sh script was updated, but the code wasn't regenerated. This simply removes the vendored deps that are no longer part of the codegen output. Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>	8 years ago
Alin Sinpalean	96fb0b2155	Optimize PromQL aggregations (#4248 ) * Compute hash of label subsets without creating a LabelSet first. Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>	8 years ago
Tom Wilkie	3228814456	Don't forget to register query_duration_seconds{slice="queue_time"} (#4381 ) Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	8 years ago
Thomas Jackson	86239ee5a5	Check for timeout in each iteration of matrixSelector (#4300 ) Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com> Fixes #4288	8 years ago
Thomas Jackson	fc7f45ba9e	Timeout if populating iterators takes too long (#4291 ) Right now promql won't time out a request if populating the iterators takes a long time. Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com> Fixes #4289	8 years ago
Thomas Jackson	63b8e4fb88	Bubble up errors to promql from populating iterators (#4136 ) This changes the Walk/Inspect API inside the promql package to bubble up errors. This is done by having the inspector return an error (instead of a bool) and then bubbling that up in the Walk. This way if any error is encountered in the Walk() the walk will stop and return the error. This avoids issues where errors from the Querier where being ignored (causing incorrect promql evaluation). Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com> Fixes #4136	8 years ago
Thomas Jackson	a6dace8829	Check for timeout in each iteration of matrixSelector (#4300 ) Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com> Fixes #4288	8 years ago
Thomas Jackson	630f42fcf1	Timeout if populating iterators takes too long (#4291 ) Right now promql won't time out a request if populating the iterators takes a long time. Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com> Fixes #4289	8 years ago
Thomas Jackson	404abe0f1c	Bubble up errors to promql from populating iterators (#4136 ) This changes the Walk/Inspect API inside the promql package to bubble up errors. This is done by having the inspector return an error (instead of a bool) and then bubbling that up in the Walk. This way if any error is encountered in the Walk() the walk will stop and return the error. This avoids issues where errors from the Querier where being ignored (causing incorrect promql evaluation). Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com> Fixes #4136	8 years ago
Mario Trangoni	0e2aa35771	promql: fix unconvert issues (#4040 ) See, $ gometalinter --vendor --disable-all --enable=unconvert --deadline 6m ./... promql/engine.go:1396:26⚠️ unnecessary conversion (unconvert) promql/engine.go:1396:40⚠️ unnecessary conversion (unconvert) promql/engine.go:1398:26⚠️ unnecessary conversion (unconvert) promql/engine.go:1398:40⚠️ unnecessary conversion (unconvert) promql/engine.go:1427:26⚠️ unnecessary conversion (unconvert) promql/engine.go:1427:40⚠️ unnecessary conversion (unconvert) promql/engine.go:1429:26⚠️ unnecessary conversion (unconvert) promql/engine.go:1429:40⚠️ unnecessary conversion (unconvert) promql/engine.go:1505:50⚠️ unnecessary conversion (unconvert) promql/engine.go:1573:46⚠️ unnecessary conversion (unconvert) promql/engine.go:1578:46⚠️ unnecessary conversion (unconvert) promql/engine.go:1591:80⚠️ unnecessary conversion (unconvert) promql/engine.go:1602:94⚠️ unnecessary conversion (unconvert) promql/engine.go:1630:18⚠️ unnecessary conversion (unconvert) promql/engine.go:1631:24⚠️ unnecessary conversion (unconvert) promql/engine.go:1634:18⚠️ unnecessary conversion (unconvert) promql/engine.go:1635:34⚠️ unnecessary conversion (unconvert) promql/functions.go:302:42⚠️ unnecessary conversion (unconvert) promql/functions.go:315:42⚠️ unnecessary conversion (unconvert) promql/functions.go:334:26⚠️ unnecessary conversion (unconvert) promql/functions.go:395:31⚠️ unnecessary conversion (unconvert) promql/functions.go:406:31⚠️ unnecessary conversion (unconvert) promql/functions.go:454:27⚠️ unnecessary conversion (unconvert) promql/functions.go:701:46⚠️ unnecessary conversion (unconvert) promql/functions.go:701:78⚠️ unnecessary conversion (unconvert) promql/functions.go:730:43⚠️ unnecessary conversion (unconvert) promql/functions.go:1220:23⚠️ unnecessary conversion (unconvert) promql/functions.go:1249:23⚠️ unnecessary conversion (unconvert) promql/quantile.go:107:54⚠️ unnecessary conversion (unconvert) promql/quantile.go:182:16⚠️ unnecessary conversion (unconvert) promql/quantile.go:182:64⚠️ unnecessary conversion (unconvert) Signed-off-by: Mario Trangoni <mjtrangoni@gmail.com>	8 years ago
Brian Brazil	dd6781add2	Optimise PromQL (#3966 ) * Move range logic to 'eval' Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make aggregegate range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * PromQL is statically typed, so don't eval to find the type. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Extend rangewrapper to multiple exprs Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Start making function evaluation ranged Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make instant queries a special case of range queries Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Eliminate evalString Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Evaluate range vector functions one series at a time Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make unary operators range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make binops range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Pass time to range-aware functions. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make simple _over_time functions range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reduce allocs when working with matrix selectors Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Add basic benchmark for range evaluation Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reuse objects for function arguments Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Do dropmetricname and allocating output vector only once. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Add range-aware support for range vector functions with params Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Optimise holt_winters, cut cpu and allocs by ~25% Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make rate&friends range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make more functions range aware. Document calling convention. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make date functions range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make simple math functions range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Convert more functions to be range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make more functions range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Specialcase timestamp() with vector selector arg for range awareness Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove transition code for functions Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove the rest of the engine transition code Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove more obselete code Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove the last uses of the eval* functions Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove engine finalizers to prevent corruption The finalizers set by matrixSelector were being called just before the value they were retruning to the pool was then being provided to the caller. Thus a concurrent query could corrupt the data that the user has just been returned. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Add new benchmark suite for range functinos Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Migrate existing benchmarks to new system Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Expand promql benchmarks Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Simply test by removing unused range code Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * When testing instant queries, check range queries too. To protect against subsequent steps in a range query being affected by the previous steps, add a test that evaluates an instant query that we know works again as a range query with the tiimestamp we care about not being the first step. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reuse ring for matrix iters. Put query results back in pool. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reuse buffer when iterating over matrix selectors Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Unary minus should remove metric name Cut down benchmarks for faster runs. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reduce repetition in benchmark test cases Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Work series by series when doing normal vectorSelectors Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Optimise benchmark setup, cuts time by 60% Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Have rangeWrapper use an evalNodeHelper to cache across steps Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Use evalNodeHelper with functions Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Cache dropMetricName within a node evaluation. This saves both the calculations and allocs done by dropMetricName across steps. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reuse input vectors in rangewrapper Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reuse the point slices in the matrixes input/output by rangeWrapper Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make benchmark setup faster using AddFast Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Simplify benchmark code. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Add caching in VectorBinop Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Use xor to have one-level resultMetric hash key Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Add more benchmarks Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Call Query.Close in apiv1 This allows point slices allocated for the response data to be reused by later queries, saving allocations. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Optimise histogram_quantile It's now 5-10% faster with 97% less garbage generated for 1k steps Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make the input collection in rangeVector linear rather than quadratic Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Optimise label_replace, for 1k steps 15x fewer allocs and 3x faster Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Optimise label_join, 1.8x faster and 11x less memory for 1k steps Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Expand benchmarks, cleanup comments, simplify numSteps logic. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Address Fabian's comments Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Comments from Alin. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Address jrv's comments Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove dead code Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Address Simon's comments. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Rename populateIterators, pre-init some sizes Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Handle case where function has non-matrix args first Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Split rangeWrapper out to rangeEval function, improve comments Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Cleanup and make things more consistent Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make EvalNodeHelper public Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Fabian's comments. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	8 years ago
David King	6286c10df0	Fix OOM when a large K is used in topk queries (#4087 ) This attempts to close #3973. Handles cases where the length of the input vector to an aggregate topk / bottomk function is less than the K paramater. The change updates Prometheus to allocate a result vector the same length as the input vector in these cases. Previously Prometheus would out-of-memory panic for large K values. This change makes that unlikely unless the size of the input vector is equally large. Signed-off-by: David King <dave@davbo.org>	8 years ago
Tony Lee	7cd56f56df	add queue_time slice to query_duration_seconds (#4050 )	8 years ago
Anton Tereshchenkov	18bbec050c	promql: propagate storage errors	8 years ago
Nikunj Aggarwal	998dfcbac6	Expose itemtype outside the package (#3933 )	8 years ago
Fabian Reinartz	7ccd4b39b8	*: implement query params This adds a parameter to the storage selection interface which allows query engine(s) to pass information about the operations surrounding a data selection. This can for example be used by remote storage backends to infer the correct downsampling aggregates that need to be provided.	8 years ago
Krasi Georgiev	a53d4ed197	drop metric name for bool modifier (#3821 ) fixes #3820	8 years ago
Fabian Reinartz	f8fccc73d8	promql: remove global metrics	8 years ago
Fabian Reinartz	83cd270ea4	*: adapt to storage interface changes	8 years ago
David Kaltschmidt	87c46ea6c3	Renamed TotalEvalTime to EvalTotalTime * TotalFoo suggested a comprehensive timing, but TotalEvalTime was part of the Exec timings, together with Queue timings * The other option was to rename ExecTotalTime to TotalExecTime, but there was already ExecQueueTime, suggesting Exec to be some sort of group	8 years ago
David Kaltschmidt	c93e54d240	Adds execution timer stats to the range query API consumers should be able to get insight into the query run times. The UI currently measures total roundtrip times. This PR allows for more fine grained metrics to be exposed. * adds new timer for total execution time (queue + eval) * expose new timer, queue timer, and eval timer in stats field of the range query response: ```json { "status": "success", "data": { "resultType": "matrix", "result": [], "stats": { "execQueueTimeNs": 4683, "execTotalTimeNs": 2086587, "totalEvalTimeNs": 2077851 } } } ``` * stats field is optional, only set when query parameter `stats` is not empty Try it via ```sh curl 'http://localhost:9090/api/v1/query_range?query=up&start=1486480279&end=1486483879&step=14000&stats=true' ``` Review feedback * moved query stats json generation to query_stats.go * use seconds for all query timers * expose all timers available * Changed ExecTotalTime string representation from Exec queue total time to Exec total time	8 years ago
Brian Brazil	99905f82a6	Remove keep_common modifier. See #3060	8 years ago
Brian Brazil	67274f0794	Remove 4 interval staleness heuristic. (#3244 ) This means that if there is no stale marker, only the usual staleness delta (5m) applies. It has occured to me that there is an oddity in the heurestic. It works fine as long as you have 2 points within the last 5m, but breaks down when the time window advances to the point where you have just 1 point. Consider you had points at t=0 and t=10. With the heurestic it goes stale at t=51, up until t=300. However from t=301 until t=310 we only see the t=10 point and the series comes back to life. That is not desirable. I don't see a way to keep this form of heurestic working given this issue, so thus I'm removing it.	8 years ago
Julius Volz	f7e8348a88	Re-add contexts to storage.Storage.Querier() (#3230 ) * Re-add contexts to storage.Storage.Querier() These are needed when replacing the storage by a multi-tenant implementation where the tenant is stored in the context. The 1.x query interfaces already had contexts, but they got lost in 2.x. * Convert promql.Engine to use native contexts	8 years ago
Fabian Reinartz	d21f149745	*: migrate to go-kit/log	8 years ago
Alexey Palazhchenko	695ec0b981	Fix few typos. (#2962 )	9 years ago
Goutham Veeramachaneni	4194d2ac79	Call At() only if Next() is true Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	9 years ago
Goutham Veeramachaneni	d407bd150c	Consolidate the duration params in CLI * All CLI params moved to model.Duration Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	9 years ago
Goutham Veeramachaneni	507790a357	Rework logging to use explicitly passed logger Mostly cleaned up the global logger use. Still some uses in discovery package. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	9 years ago
Brian Brazil	220e78b9c3	Consider a series stale after 4.1 intervals with no data. To cover the cases where stale markers may not be available, we need to infer the interval and mark series stale based on that. As we're lacking stale markers this is less accurate, however it should be good enough for these cases. We need 4 intervals as if say we had data at t=0 and t=10, coming via federation. The next data point should be at t=20 however it could take up to t=30 for it actually to be ingested, t=40 for it to be scraped via federation and t=50 for it to be ingested. We then add 10% on to that for slack, as we do elsewhere.	9 years ago
Brian Brazil	c02c25d5ba	Allow peeking back further in buffer.	9 years ago
Brian Brazil	a5cf25743c	Move stalness check into a function	9 years ago
Brian Brazil	80b40e6d91	Add initial staleness handing to promql. For instant vectors, if "stale" is the newest sample ignore the timeseries. For range vectors, filter out "stale" samples. Make it possible to inject "stale" samples in promql tests.	9 years ago
Brian Brazil	fcc88f0e1e	query/query_range should return eval timestamp Query and query_range should return the timestamp at which an evaluation is performed, not the timestamp of the data. This is as that's what query range asked for, and we need to keep query consistent with that. Query for a matrix remains unchanged, returning the literal matrix.	9 years ago
Brian Brazil	517b81f927	Add timestamp() function. Make the timestamp of instant vectors be the timestamp of the sample rather than the evaluation. We were not using this anywhere, so this is safe. Add a function to return the timestamp of samples in an instant vector. Fixes #1557	9 years ago
Tom Wilkie	4d9b917d11	Instrument Prometheus with OpenTracing (#2554 ) * Use request.Context() instead of a global map of contexts. * Add some basic opentracing instrumentation on the query path. * Remove tracehandler endpoint.	9 years ago
Tom Wilkie	f0e8a5f37c	Add promql.ErrStorage, which is interpreted by the API as a 500.	9 years ago
Fabian Reinartz	0ecd205794	promql: Use buffer pool for matrix allocations	9 years ago
Fabian Reinartz	b09b90a940	Correctly close querier on error, revendor tsdb	9 years ago

1 2 3

135 Commits (9bca041285e226410809b64ce2d178bc4ccabbcf)