100 Commits (c8665eaee3b4ecb3890a9d1c4085d0d7767ad394)

Author	SHA1	Message	Date
Tom Wilkie	de0a772b8e	Port tsdb to use pkg/labels. (#6326) ... * Port tsdb to use pkg/labels. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Get tests passing. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Remove useless cast. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Appease linters. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Sharad Gaur	a85e7aac0e	Adding TSDB Stats Page in React UI (#6281) ... Signed-off-by: Sharad Gaur <sgaur@splunk.com>	6 years ago
Dustin Hooten	ca60bf298c	React UI: Implement /targets page (#6276) ... * Add LastScrapeDuration to targets endpoint Signed-off-by: Dustin Hooten <dhooten@splunk.com> * Add Scrape job name to targets endpoint Signed-off-by: Dustin Hooten <dhooten@splunk.com> * Implement the /targets page in react Signed-off-by: Dustin Hooten <dhooten@splunk.com> * Add state query param to targets endpoint Signed-off-by: Dustin Hooten <dhooten@splunk.com> * Use state filter in api call Signed-off-by: Dustin Hooten <dhooten@splunk.com> * api feedback Signed-off-by: Dustin Hooten <dhooten@splunk.com> * pr feedback frontend Signed-off-by: Dustin Hooten <dhooten@splunk.com> * Implement and use localstorage hook Signed-off-by: Dustin Hooten <dhooten@splunk.com> * PR feedback Signed-off-by: Dustin Hooten <dhooten@splunk.com>	6 years ago
Yao Zengzeng	8744afdd1e	cleanup redundant code of `TestEndpoints` (#6022) ... Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>	6 years ago
Yao Zengzeng	21c9789083	multiple queries test for StreamRead (#5969) ... Signed-off-by: YaoZengzeng <yaozengzeng@zju.edu.cn>	6 years ago
Bartek Płotka	48b2c9c8ea	remote-read: streamed chunked server side; Extended protobuf; Added chunked, checksumed reader (#5703) ... Part of: https://github.com/prometheus/prometheus/issues/4517 and https://github.com/improbable-eng/thanos/issues/488 Changes: * Extended protobuf for chunked remote read and negotation. * Added checksumed, chunked Writer/Reader. * Added Server side implementation for chunked streamed remote-read. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>	6 years ago
Ganesh Vernekar	5ecef3542d	Cleanup after merging tsdb into prometheus ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Chris Marchbanks	0685eb5395	Refactor testutil.NewStorage into a new package ... This avoids a circular dependency between the testutil and storage packages. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	6 years ago
@aifsair	0f00737308	Fix log config setup (#5807) ... The return value of Set() was not checked. Hence, the typo ("al" instead of "af") wasn't catched. Signed-off-by: François (fser) <fser@code-libre.org>	6 years ago
Thomas Jackson	fef150f1b5	Add tests to ensure we can marshal and unmarshal our min/max times (#5734) ... * Add tests to ensure we can marshal and unmarshal our min/max times Related to https://github.com/prometheus/client_golang/issues/614 Instead of implementing all the time parsing, we can special-case handle these 2 times. This means if times in this format show up that time.Parse can't handle they will still error, but we can marshal/parse our own min/max time Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>	6 years ago
Simon Pasquier	45506841e6	*: enable all default linters (#5504) ... Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Bjoern Rabenstein	38d518c0fe	Rework #5009 after comments ... Signed-off-by: Bjoern Rabenstein <bjoern@rabenste.in>	6 years ago
Bob Shannon	8c8bb82d04	Add support for POSTing to /series endpoint (#5422) ... * Add support for POSTing to /series endpoint * Document query API POST support Signed-off-by: Bob Shannon <bob.m.shannon@gmail.com>	6 years ago
Tariq Ibrahim	8fdfa8abea	refine error handling in prometheus (#5388) ... i) Uses the more idiomatic Wrap and Wrapf methods for creating nested errors. ii) Fixes some incorrect usages of fmt.Errorf where the error messages don't have any formatting directives. iii) Does away with the use of fmt package for errors in favour of pkg/errors Signed-off-by: tariqibrahim <tariq181290@gmail.com>	6 years ago
Tom Wilkie	c7b3535997	Use pkg/relabelling in remote write. ... - Unmarshall external_labels config as labels.Labels, add tests. - Convert some more uses of model.LabelSet to labels.Labels. - Remove old relabel pkg (fixes #3647). - Validate external label names. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Callum Styan	6f69e31398	Tail the TSDB WAL for remote_write ... This change switches the remote_write API to use the TSDB WAL. This should reduce memory usage and prevent sample loss when the remote end point is down. We use the new LiveReader from TSDB to tail WAL segments. Logic for finding the tracking segment is included in this PR. The WAL is tailed once for each remote_write endpoint specified. Reading from the segment is based on a ticker rather than relying on fsnotify write events, which were found to be complicated and unreliable in early prototypes. Enqueuing a sample for sending via remote_write can now block, to provide back pressure. Queues are still required to acheive parallelism and batching. We have updated the queue config based on new defaults for queue capacity and pending samples values - much smaller values are now possible. The remote_write resharding code has been updated to prevent deadlocks, and extra tests have been added for these cases. As part of this change, we attempt to guarantee that samples are not lost; however this initial version doesn't guarantee this across Prometheus restarts or non-retryable errors from the remote end (eg 400s). This changes also includes the following optimisations: - only marshal the proto request once, not once per retry - maintain a single copy of the labels for given series to reduce GC pressure Other minor tweaks: - only reshard if we've also successfully sent recently - add pending samples, latest sent timestamp, WAL events processed metrics Co-authored-by: Chris Marchbanks <csmarchbanks.com> (initial prototype) Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com> (sharding changes) Signed-off-by: Callum Styan <callumstyan@gmail.com>	6 years ago
Hrishikesh Barman	a1f34bec2e	Added CORS Origin flag (#5011) ... Signed-off-by: Hrishikesh Barman <hrishikeshbman@gmail.com>	7 years ago
Matt Layher	302148fd69	*: apply gofmt -s ... Signed-off-by: Matt Layher <mdlayher@gmail.com>	7 years ago
Callum Styan	5358f76c5c	update remote write path proto so that Labels/Timeseries can't be nil (#4957) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	7 years ago
Tom Wilkie	6e08029b56	Move err to be the last return value from storage.Select. (#5054) ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Julius Volz	11a52be1d8	Better rounding for incoming query timestamps (#4941) ... Fixes https://github.com/prometheus/prometheus/issues/4939 Signed-off-by: Julius Volz <julius.volz@gmail.com>	7 years ago
mknapphrt	f0e9196dca	Return warnings on a remote read fail (#4832) ... Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>	7 years ago
Alex Yu	5dcce32ef8	update promlog to latest version (#4876) ... * update promlog to latest version Signed-off-by: Alex Yu <yu.alex96@gmail.com> * Update api tests, fix main setup Signed-off-by: Alex Yu <yu.alex96@gmail.com> * tidy go.sum Signed-off-by: Alex Yu <yu.alex96@gmail.com> * revendor prometheus/common Signed-off-by: Alex Yu <yu.alex96@gmail.com> * only initialize config; use kingpin for remote_storage_adapter Signed-off-by: Alex Yu <yu.alex96@gmail.com> * actually parse the flags Signed-off-by: Alex Yu <yu.alex96@gmail.com> * clean up imports Signed-off-by: Alex Yu <yu.alex96@gmail.com>	7 years ago
Ganesh Vernekar	ca93fd544b	/api/v1/labels endpoint for getting all label names (#4835) ... * vendor: update tsdb Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * /api/v1/labels endpoint Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * regex matchers for API Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Add docs Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Matchers behaving as OR Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Removed the matchers Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * vendor: update tsdb using go mod Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * vendor update: tsdb Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Added LabelNames() to storage.Querier Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Test for api.labelNames Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Nits Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	7 years ago
Simon Pasquier	6fa8de132b	web/v1/api: add tests for admin actions (#4767) ... Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago
Simon Pasquier	a308a186e4	web/api/v1: fix targets endpoint ... Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago
Simon Pasquier	c4a6acfb1e	*: move to go 1.11 (#4626) ... * *: move to go 1.11 Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Reduce number of places where we specify the Go version Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago
Callum Styan	9bca041285	WIP: keep track of samples per query, set a max # of samples (#4513) ... * keep track of samples per query, set a max # of samples that can be in memory at once Signed-off-by: Callum Styan <callumstyan@gmail.com>	7 years ago
Krasi Georgiev	47a673c3a0	process scrape loops reloading in parallel (#4526) ... The scrape manage receiver's channel now just saves the target sets and another backgorund runner updates the scrape loops every 5 seconds. This is so that the scrape manager doesn't block the receiving channel when it does the long background reloading of the scrape loops. Active and dropped targets are now saved in each scrape pool instead of the scrape manager. This is mainly to avoid races when getting the targets via the web api. When reloading the scrape loops now happens in parallel to speed up the final disared state and this also speeds up the prometheus's shutting down. Also updated some funcs signatures in the web package for consistency. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
Tom Wilkie	4c52400708	Limit concurrent remote reads. (#4656) ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	d3a1ff1abf	Reduce memory usage of remote read by reducing pointer usage. (#4655) ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	457e4bb58e	Limit the number of samples remote read can return. (#4532) ... * Limit the number of samples remote read can return. - Return 413 entity too large. - Limit can be set be a flag. Allow 0 to mean no limit. - Include limit in error message. - Set default limit to 50M (* 16 bytes = 800MB). Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Max Inden	ecf676cf97	web/api: Expose rule health and last error (#4501) ... Expose rule health and last evaluation error on `/api/v1/rules`. Signed-off-by: Max Leonard Inden <IndenML@gmail.com>	7 years ago
Ganesh Vernekar	f1db699dff	Persist alert 'for' state across restarts (#4061) ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	7 years ago
Max Leonard Inden	71fafad099	api/v1: Coninue work exposing rules and alerts ... Signed-off-by: Max Leonard Inden <IndenML@gmail.com>	7 years ago
mg03	31f8ca0dfb	api v1 alerts/rules json endpoint ... Signed-off-by: mg03 <mgeng03@gmail.com>	7 years ago
Thomas Jackson	92c6f0c92e	Add offset to selectParams (#4226) ... * Add Start/End to SelectParams * Make remote read use the new selectParams for start/end This commit will continue sending the start/end time of the remote read query as the overarching promql time and the specific range of data that the query is intersted in receiving a response to is now part of the ReadHints (upstream discussion in #4226). * Remove unused vendored code The genproto.sh script was updated, but the code wasn't regenerated. This simply removes the vendored deps that are no longer part of the codegen output. Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>	7 years ago
Tom Wilkie	ccb2ee607b	Log errors encountered when marshalling and writing responses. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	4e4f0d4e41	spelling. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	e518f51a99	Extend API tests to cover remote read API. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	fcc3f43acd	spelling. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Tom Wilkie	ae29512444	Extend API tests to cover remote read API. ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Brian Brazil	dd6781add2	Optimise PromQL (#3966) ... * Move range logic to 'eval' Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make aggregegate range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * PromQL is statically typed, so don't eval to find the type. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Extend rangewrapper to multiple exprs Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Start making function evaluation ranged Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make instant queries a special case of range queries Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Eliminate evalString Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Evaluate range vector functions one series at a time Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make unary operators range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make binops range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Pass time to range-aware functions. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make simple _over_time functions range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reduce allocs when working with matrix selectors Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Add basic benchmark for range evaluation Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reuse objects for function arguments Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Do dropmetricname and allocating output vector only once. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Add range-aware support for range vector functions with params Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Optimise holt_winters, cut cpu and allocs by ~25% Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make rate&friends range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make more functions range aware. Document calling convention. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make date functions range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make simple math functions range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Convert more functions to be range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make more functions range aware Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Specialcase timestamp() with vector selector arg for range awareness Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove transition code for functions Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove the rest of the engine transition code Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove more obselete code Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove the last uses of the eval* functions Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove engine finalizers to prevent corruption The finalizers set by matrixSelector were being called just before the value they were retruning to the pool was then being provided to the caller. Thus a concurrent query could corrupt the data that the user has just been returned. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Add new benchmark suite for range functinos Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Migrate existing benchmarks to new system Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Expand promql benchmarks Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Simply test by removing unused range code Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * When testing instant queries, check range queries too. To protect against subsequent steps in a range query being affected by the previous steps, add a test that evaluates an instant query that we know works again as a range query with the tiimestamp we care about not being the first step. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reuse ring for matrix iters. Put query results back in pool. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reuse buffer when iterating over matrix selectors Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Unary minus should remove metric name Cut down benchmarks for faster runs. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reduce repetition in benchmark test cases Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Work series by series when doing normal vectorSelectors Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Optimise benchmark setup, cuts time by 60% Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Have rangeWrapper use an evalNodeHelper to cache across steps Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Use evalNodeHelper with functions Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Cache dropMetricName within a node evaluation. This saves both the calculations and allocs done by dropMetricName across steps. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reuse input vectors in rangewrapper Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Reuse the point slices in the matrixes input/output by rangeWrapper Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make benchmark setup faster using AddFast Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Simplify benchmark code. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Add caching in VectorBinop Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Use xor to have one-level resultMetric hash key Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Add more benchmarks Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Call Query.Close in apiv1 This allows point slices allocated for the response data to be reused by later queries, saving allocations. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Optimise histogram_quantile It's now 5-10% faster with 97% less garbage generated for 1k steps Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make the input collection in rangeVector linear rather than quadratic Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Optimise label_replace, for 1k steps 15x fewer allocs and 3x faster Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Optimise label_join, 1.8x faster and 11x less memory for 1k steps Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Expand benchmarks, cleanup comments, simplify numSteps logic. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Address Fabian's comments Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Comments from Alin. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Address jrv's comments Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Remove dead code Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Address Simon's comments. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Rename populateIterators, pre-init some sizes Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Handle case where function has non-matrix args first Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Split rangeWrapper out to rangeEval function, improve comments Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Cleanup and make things more consistent Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Make EvalNodeHelper public Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> * Fabian's comments. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	7 years ago
Henri DF	2952387ed1	Pass query hints down into remote read query proto (#4122) ... Signed-off-by: Henri DF <henridf@gmail.com>	7 years ago
Krasi Georgiev	ddd46de6f4	Races/3994 (#4005) ... Fix race by properly locking access to scrape pools. Use separate mutex for information needed by UI so that UI isn't blocked when targets are being updated.	7 years ago
Brian Brazil	cc39021b2b	Provide custom marshalling for Point ... Point has a non-standard marshalling, and is also where the vast majority of CPU time is spent so it is worth optimising.	7 years ago
Brian Brazil	8ede14b24c	Add unittests for Point json output	7 years ago
Brian Brazil	ecd0a9c6ba	web: Add benchmark for respond()	7 years ago
Conor Broderick	99006d3baf	Added dropped targets API to targets endpoint (#3870)	7 years ago
Conor Broderick	1fd20fc954	Add dropped alertmanagers to alertmanagers API (#3865)	7 years ago