187 Commits (ee5bba07c003af88d233083cdd65ac6d064b5b82)

Author	SHA1	Message	Date
beorn7	0e202dacb4	Streamline series iterator creation ... This will fix issue #1035 and will also help to make issue #1264 less bad. The fundamental problem in the current code: In the preload phase, we quite accurately determine which chunks will be used for the query being executed. However, in the subsequent step of creating series iterators, the created iterators are referencing _all_ in-memory chunks in their series, even the un-pinned ones. In iterator creation, we copy a pointer to each in-memory chunk of a series into the iterator. While this creates a certain amount of allocation churn, the worst thing about it is that copying the chunk pointer out of the chunkDesc requires a mutex acquisition. (Remember that the iterator will also reference un-pinned chunks, so we need to acquire the mutex to protect against concurrent eviction.) The worst case happens if a series doesn't even contain any relevant samples for the query time range. We notice that during preloading but then we will still create a series iterator for it. But even for series that do contain relevant samples, the overhead is quite bad for instant queries that retrieve a single sample from each series, but still go through all the effort of series iterator creation. All of that is particularly bad if a series has many in-memory chunks. This commit addresses the problem from two sides: First, it merges preloading and iterator creation into one step, i.e. the preload call returns an iterator for exactly the preloaded chunks. Second, the required mutex acquisition in chunkDesc has been greatly reduced. That was enabled by a side effect of the first step, which is that the iterator is only referencing pinned chunks, so there is no risk of concurrent eviction anymore, and chunks can be accessed without mutex acquisition. To simplify the code changes for the above, the long-planned change of ValueAtTime to ValueAtOrBefore time was performed at the same time. (It should have been done first, but it kind of accidentally happened while I was in the middle of writing the series iterator changes. Sorry for that.) So far, we actively filtered the up to two values that were returned by ValueAtTime, i.e. we invested work to retrieve up to two values, and then we invested more work to throw one of them away. The SeriesIterator.BoundaryValues method can be removed once #1401 is fixed. But I really didn't want to load even more changes into this PR. Benchmarks: The BenchmarkFuzz.* benchmarks run 83% faster (i.e. about six times faster) and allocate 95% fewer bytes. The reason for that is that the benchmark reads one sample after another from the time series and creates a new series iterator for each sample read. To find out how much these improvements matter in practice, I have mirrored a beefy Prometheus server at SoundCloud that suffers from both issues #1035 and #1264. To reach steady state that would be comparable, the server needs to run for 15d. So far, it has run for 1d. The test server currently has only half as many memory time series and 60% of the memory chunks the main server has. The 90th percentile rule evaluation cycle time is ~11s on the main server and only ~3s on the test server. However, these numbers might get much closer over time. In addition to performance improvements, this commit removes about 150 LOC.	10 years ago
Brian Brazil	c77c3a8c56	promql: Limit extrapolation of delta/rate/increase ... The new implementation detects the start and end of a series by looking at the average sample interval within the range. If the first (last) sample in the range is more than 1.1*interval distant from the beginning (end) of the range, it is considered the first (last) sample of the series as a whole, and extrapolation is limited to half the interval (rather than all the way to the beginning (end) of the range). In addition, if the extrapolated starting point of a counter (where it is zero) is within the range, it is used as the starting point of the series. Fixes #581	10 years ago
Brian Brazil	bac1f28cad	Similar to topk/bottomk, have sort/sort_desc put NaN at end. ... This makes topk and bottomk consistent with the sorting functions, as per #1271.	10 years ago
Brian Brazil	88ca82304c	Make topk/bottomk prefer returning real numbers over NaN.	10 years ago
James Sanford	5b53262b7a	promql: Add clamp_max/clamp_min functions.	10 years ago
Brian Brazil	5740a8fade	promql: Remove deprecated 2nd argument to delta() ... This change is breaking, use increase() instead. I'm not cleaning up the function in this PR, as my solution to #581 will rewrite and simplify increase/rate/delta.	10 years ago
Brian Brazil	f08abdb48b	promql: Add irate() function ... irate is a rate function that only looks at the most recent two data points, and calucaltes a per-second value from that. This produces much more granular graphs for fast moving data, and works sanely across many scrape intervals. It doesn't do so well for slowly moving data.	10 years ago
Fabian Reinartz	e005f939fd	Fix scalar construction in function	10 years ago
Fabian Reinartz	eca41f5319	Run gofmt	10 years ago
Brian Brazil	9b382647b5	Remove optional vector() 2nd argument	10 years ago
Brian Brazil	69f5fa0c1e	promql: Add vector function. ... Currently the only way to convert a scalar to a vector is to use absent(), which isn't very clean. This adds a vector() function that's the inverse of scalar() and lets your optionally set labels. Example usage would be vector(time() % 86400) < 3600 to filter to only the first hour of the day.	10 years ago
Julius Volz	61c42c8da0	Change relabel_replace() to do full-string matches. ... THIS IS A BREAKING CHANGE. Fixes part of https://github.com/prometheus/prometheus/issues/996	10 years ago
Fabian Reinartz	d6b8da8d43	Switch promql types to common/model	10 years ago
Fabian Reinartz	1535ef1457	Replace metric.SamplePair with model.SamplePair	10 years ago
Fabian Reinartz	306e8468a0	Switch from client_golang/model to common/model	10 years ago
Julius Volz	27ed874358	Implement label_replace() ... Implements part of https://github.com/prometheus/prometheus/issues/959.	10 years ago
Brian Brazil	d6a80c2b76	promql: Add support for predict_linear(my_timeseries[1h], 7200) ... This will give a prediction for the value of my_timeseries in 2 hours, based on the last hour of data.	11 years ago
Brian Brazil	f34de493d5	Add increase() function, to replace delta(..., 1). ... This calculates how much a counter increases over a given period of time, which is the area under the curve of it's rate. increase(x[5m]) is equivilent to rate(x[5m]) * 300.	11 years ago
Julius Volz	d44a89c6e8	Implement changes() function. ... changes() takes a range vector and returns the number of times a value has changed in the given time window for each time series as an instant vector.	11 years ago
Julius Volz	6f33ed9e59	Add resets() function to count counter resets. ... resets() returns for every range vector element how many counter resets there have been in the specified range.	11 years ago
Fabian Reinartz	5602328c7c	Refactor query evaluation. ... This copies the evaluation logic from the current rules/ package. The new engine handles the execution process from query string to final result. It provides query timeout and cancellation and general flexibility for future changes. functions.go: Add evaluation implementation. Slight changes to in/out data but not to the processing logic. quantile.go: No changes. analyzer.go: No changes. engine.go: Actually new part. Mainly consists of evaluation methods which were not changed. setup_test.go: Copy of rules/helpers_test.go to setup test storage. promql_test.go: Copy of rules/rules_test.go.	11 years ago
Fabian Reinartz	32b7595c47	Create promql package with lexer/parser. ... This commit creates a (so far unused) package. It contains the a custom lexer/parser for the query language. ast.go: New AST that interacts well with the parser. lex.go: Custom lexer (new). lex_test.go: Lexer tests (new). parse.go: Custom parser (new). parse_test.go: Parser tests (new). functions.go: Changed function type, dummies for parser testing (barely changed/dummies). printer.go: Adapted from rules/ and adjusted to new AST (mostly unchanged, few additions).	11 years ago