123 Commits (2209fa98b42aee3aed864d7ea3534775ca8cd6e9)

Author	SHA1	Message	Date
Ganesh Vernekar	1c99adb9fd	Callbacks for lifecycle of series in TSDB (#7159) ... * Callbacks for lifecycle of series in TSDB Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Add more comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
gotjosh	bfcd9282a8	API: Allow AlertmanagerRetriever and RulesRetriever to receive a Context (#7256) ... * API: Allow AlertmanagerRetriever and RulesRetriever to receive a Context Signed-off-by: gotjosh <josue@grafana.com> * Use single line functions where possible Signed-off-by: gotjosh <josue@grafana.com>	5 years ago
Ganesh Vernekar	d4b9fe801f	M-map full chunks of Head from disk (#6679) ... When appending to the head and a chunk is full it is flushed to the disk and m-mapped (memory mapped) to free up memory Prom startup now happens in these stages - Iterate the m-maped chunks from disk and keep a map of series reference to its slice of mmapped chunks. - Iterate the WAL as usual. Whenever we create a new series, look for it's mmapped chunks in the map created before and add it to that series. If a head chunk is corrupted the currpted one and all chunks after that are deleted and the data after the corruption is recovered from the existing WAL which means that a corruption in m-mapped files results in NO data loss. [Mmaped chunks format](https://github.com/prometheus/prometheus/blob/master/tsdb/docs/format/head_chunks.md) - main difference is that the chunk for mmaping now also includes series reference because there is no index for mapping series to chunks. [The block chunks](https://github.com/prometheus/prometheus/blob/master/tsdb/docs/format/chunks.md) are accessed from the index which includes the offsets for the chunks in the chunks file - example - chunks of series ID have offsets 200, 500 etc in the chunk files. In case of mmaped chunks, the offsets are stored in memory and accessed from that. During WAL replay, these offsets are restored by iterating all m-mapped chunks as stated above by matching the series id present in the chunk header and offset of that chunk in that file. **Prombench results** _WAL Replay_ 1h Wal reply time 30% less wal reply time - 4m31 vs 3m36 2h Wal reply time 20% less wal reply time - 8m16 vs 7m _Memory During WAL Replay_ High Churn: 10-15% less RAM - 32gb vs 28gb 20% less RAM after compaction 34gb vs 27gb No Churn: 20-30% less RAM - 23gb vs 18gb 40% less RAM after compaction 32.5gb vs 20gb Screenshots are in [this comment](https://github.com/prometheus/prometheus/pull/6679#issuecomment-621678932) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Ben Ye	1e4e37144d	Fixed wrongly handled not ready TSDB on web and API. (#7182) ... * fix federate endpoint panic Signed-off-by: yeya24 <yb532204897@gmail.com> * Fixed all cases of not ready TSDB being wrongly handled. * Fixed issue for federation. * Ensured this will never happen again thanks to interfaces * Fixes same issue for stats. * Added tests for readiness. * Fixed bug in stats. It was: status.MaxTime = db.Head().MaxTime() status.MinTime = db.Head().MaxTime() Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Brian's comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Brian's comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
gotjosh	24af5049bb	API: Allow `TargetRetriever` to receive a Context (#7125) ... Fixes #7103 Signed-off-by: gotjosh <josue@grafana.com>	5 years ago
Bartlomiej Plotka	c4eefd1b3a	storage: Removed SelectSorted method; Simplified interface; Added requirement for remote read to sort response. ... This is technically BREAKING CHANGE, but it was like this from the beginning: I just notice that we rely in Prometheus on remote read being sorted. This is because we use selected data from remote reads in MergeSeriesSet which rely on sorting. I found during work on https://github.com/prometheus/prometheus/pull/5882 that we do so many repetitions because of this, for not good reason. I think I found a good balance between convenience and readability with just one method. Smaller the interface = better. Also I don't know what TestSelectSorted was testing, but now it's testing sorting. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Bartlomiej Plotka	fe802f29c9	storage: Removed SelectSorted method; Simplified interface; Added requirement for remote read to sort response. ... This is technically BREAKING CHANGE, but it was like this from the beginning: I just notice that we rely in Prometheus on remote read being sorted. This is because we use selected data from remote reads in MergeSeriesSet which rely on sorting. I found during work on https://github.com/prometheus/prometheus/pull/5882 that we do so many repetitions because of this, for not good reason. I think I found a good balance between convenience and readability with just one method. Smaller the interface = better. Also I don't know what TestSelectSorted was testing, but now it's testing sorting. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Boyko	84b00564f4	API_V1: Extract time param parsing logic (#6860) ... * extract API time param parsing logic in a func Signed-off-by: blalov <boiskila@gmail.com>	5 years ago
Bartlomiej Plotka	fb79f515fc	Fixed second bug. ... Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Harkishen Singh	489a9aa7b9	Adds normalization of localhost urls in targets page react (#6794) ... * support for globalurls in targets page react Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com> * fixed tests Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com> * removed fmts Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com> * implemented suggestions Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com> * formatted Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com> * implemented suggestions. fixed tests. Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com> * formated go code Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com> * implemented suggestions Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>	5 years ago
Tobias Guggenmos	6c00f2ffcb	Comment fixes ... Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>	5 years ago
Julien Pivotto	135cc30063	rules: Make deleted rule series as stale after a reload (#6745) ... * rules: Make deleted rule series as stale after a reload Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Thor	17d8c49919	made stripe size configurable (#6644) ... Signed-off-by: Thor <thansen@digitalocean.com>	5 years ago
gotjosh	8b49c9285d	scrape: Add metrics to track bytes and entries in the metadata cache (#6675) ... Signed-off-by: gotjosh <josue@grafana.com>	5 years ago
Julien Pivotto	9adad8ad30	Remove MaxConcurrent from the PromQL engine opts (#6712) ... Since we use ActiveQueryTracker to check for concurrency in d992c36b3a it does not make sense to keep the MaxConcurrent value as an option of the PromQL engine. This pull request removes it from the PromQL engine options, sets the max concurrent metric to -1 if there is no active query tracker, and use the value of the active query tracker otherwise. It removes dead code and also will inform people who import the promql package that we made that change, as it breaks the EngineOpts struct. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Julien Pivotto	9d9bc524e5	Add query log (#6520) ... * Add query log, make stats logged in JSON like in the API Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	6 years ago
gotjosh	e1f2da74a3	api: provide metadata for a single metric ... This adds support for a new query param on the new `api/v1/metadata` endpoint that provides metadata for a specified metric via the V1 API. It collapses metadata that is equal across all targets, and aggregates under the same metric name the ones that differ. Signed-off-by: gotjosh <josue@grafana.com>	6 years ago
gotjosh	4a13c844b7	Add periods to comments on tests. ... Signed-off-by: gotjosh <josue@grafana.com>	6 years ago
gotjosh	0a0a228db3	api: provide per metric metadata (#6420) ... * api: provide per metric metadata This adds a new endpoint that provides per metric metadata via the V1 API. It collapses metadata that is equal across all targets, and aggregates under the same metric name the ones that differ. * Allow tests to be asserted on response length Some tests e.g. limit on API responses, don't require an assertion on equality. This allows us to assert against response length instead of equality. Signed-off-by: gotjosh <josue@grafana.com>	6 years ago
Boyko	1c66aea992	React UI: Implement alerts page (#6402) ... * url filter rules param Signed-off-by: blalov <boiskila@gmail.com> Signed-off-by: Boyko Lalov <boiskila@gmail.com> Signed-off-by: blalov <boiskila@gmail.com> * address review changes Signed-off-by: blalov <boiskila@gmail.com> Signed-off-by: Boyko Lalov <boiskila@gmail.com> Signed-off-by: blalov <boiskila@gmail.com> * ui initial commit Signed-off-by: blalov <boiskila@gmail.com> Signed-off-by: Boyko Lalov <boiskila@gmail.com> Signed-off-by: blalov <boiskila@gmail.com> * improve ui Signed-off-by: blalov <boiskila@gmail.com> Signed-off-by: Boyko Lalov <boiskila@gmail.com> Signed-off-by: blalov <boiskila@gmail.com> * fix typo in component name Signed-off-by: Boyko Lalov <boiskila@gmail.com> Signed-off-by: blalov <boiskila@gmail.com> * create query link + ui enhancements Signed-off-by: Boyko Lalov <boiskila@gmail.com> Signed-off-by: blalov <boiskila@gmail.com> * add count to state labels Signed-off-by: blalov <boiskila@gmail.com> * put alerts table render in the right place Signed-off-by: blalov <boiskila@gmail.com> * refactoring Signed-off-by: blalov <boiskila@gmail.com> * fix rules endpoint test Signed-off-by: blalov <boiskila@gmail.com> * lint fixes Signed-off-by: blalov <boiskila@gmail.com> * test query params Signed-off-by: blalov <boiskila@gmail.com> * refactoring Signed-off-by: blalov <boiskila@gmail.com> * review changes Signed-off-by: blalov <boiskila@gmail.com> * adding down arrow as click indicator in Alert Signed-off-by: blalov <boiskila@gmail.com> * add period at the end of the comment Signed-off-by: blalov <boiskila@gmail.com> * review changes Signed-off-by: blalov <boiskila@gmail.com> * remove left-over css Signed-off-by: blalov <boiskila@gmail.com> * adding expand/collapse arrows on Alert Signed-off-by: blalov <boiskila@gmail.com> * create proper expression for alert name Signed-off-by: blalov <boiskila@gmail.com>	6 years ago
gotjosh	7bb73a9abd	fix: flaky test for api/v1/targets/metadata. (#6436) ... * Allows sorting of responses from the API in tests Fixes flaky test for api/v1/targets/metadata. Allows sorting of responses from the API. For our tests to be deterministic, we need to ensure the response from the API follows an order. This structure allows us to define one. Fixes #6431 Signed-off-by: gotjosh <josue@grafana.com>	6 years ago
gotjosh	428089f83f	api: tests for /target/metadata API endpoint ... This commit introduces several test cases for the current /targets/metadata API endpoint. To achieve so, we use a mock of the metadataStore and inject it to the targets under test. Currently, three success cases are covered: with a metric name, with a target matcher, and with both. As for the failure scenario, the one where we couldn't match against a particular metric is covered. Signed-off-by: gotjosh <josue@grafana.com>	6 years ago
gotjosh	4bf9c6bb82	Allow targets to be injected as arguments when creating a testTargetRetriever ... Previously, the struct `testTargetRetriever` had hardcoded active and dropped targets. This made it difficult to change the target information depending on the test case. This change introduces a way to define them as arguments and pass it to a constructor for building. It lays a foundation for dynamically defining targets with various set of arguments to test different scenarios. Signed-off-by: gotjosh <josue@grafana.com>	6 years ago
Tom Wilkie	de0a772b8e	Port tsdb to use pkg/labels. (#6326) ... * Port tsdb to use pkg/labels. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Get tests passing. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Remove useless cast. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Appease linters. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Sharad Gaur	a85e7aac0e	Adding TSDB Stats Page in React UI (#6281) ... Signed-off-by: Sharad Gaur <sgaur@splunk.com>	6 years ago
Dustin Hooten	ca60bf298c	React UI: Implement /targets page (#6276) ... * Add LastScrapeDuration to targets endpoint Signed-off-by: Dustin Hooten <dhooten@splunk.com> * Add Scrape job name to targets endpoint Signed-off-by: Dustin Hooten <dhooten@splunk.com> * Implement the /targets page in react Signed-off-by: Dustin Hooten <dhooten@splunk.com> * Add state query param to targets endpoint Signed-off-by: Dustin Hooten <dhooten@splunk.com> * Use state filter in api call Signed-off-by: Dustin Hooten <dhooten@splunk.com> * api feedback Signed-off-by: Dustin Hooten <dhooten@splunk.com> * pr feedback frontend Signed-off-by: Dustin Hooten <dhooten@splunk.com> * Implement and use localstorage hook Signed-off-by: Dustin Hooten <dhooten@splunk.com> * PR feedback Signed-off-by: Dustin Hooten <dhooten@splunk.com>	6 years ago
Yao Zengzeng	8744afdd1e	cleanup redundant code of `TestEndpoints` (#6022) ... Signed-off-by: YaoZengzeng <yaozengzeng@huawei.com>	6 years ago
Yao Zengzeng	21c9789083	multiple queries test for StreamRead (#5969) ... Signed-off-by: YaoZengzeng <yaozengzeng@zju.edu.cn>	6 years ago
Bartek Płotka	48b2c9c8ea	remote-read: streamed chunked server side; Extended protobuf; Added chunked, checksumed reader (#5703) ... Part of: https://github.com/prometheus/prometheus/issues/4517 and https://github.com/improbable-eng/thanos/issues/488 Changes: * Extended protobuf for chunked remote read and negotation. * Added checksumed, chunked Writer/Reader. * Added Server side implementation for chunked streamed remote-read. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>	6 years ago
Ganesh Vernekar	5ecef3542d	Cleanup after merging tsdb into prometheus ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Chris Marchbanks	0685eb5395	Refactor testutil.NewStorage into a new package ... This avoids a circular dependency between the testutil and storage packages. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	6 years ago
@aifsair	0f00737308	Fix log config setup (#5807) ... The return value of Set() was not checked. Hence, the typo ("al" instead of "af") wasn't catched. Signed-off-by: François (fser) <fser@code-libre.org>	6 years ago
Thomas Jackson	fef150f1b5	Add tests to ensure we can marshal and unmarshal our min/max times (#5734) ... * Add tests to ensure we can marshal and unmarshal our min/max times Related to https://github.com/prometheus/client_golang/issues/614 Instead of implementing all the time parsing, we can special-case handle these 2 times. This means if times in this format show up that time.Parse can't handle they will still error, but we can marshal/parse our own min/max time Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>	6 years ago
Simon Pasquier	45506841e6	*: enable all default linters (#5504) ... Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Bjoern Rabenstein	38d518c0fe	Rework #5009 after comments ... Signed-off-by: Bjoern Rabenstein <bjoern@rabenste.in>	6 years ago
Bob Shannon	8c8bb82d04	Add support for POSTing to /series endpoint (#5422) ... * Add support for POSTing to /series endpoint * Document query API POST support Signed-off-by: Bob Shannon <bob.m.shannon@gmail.com>	6 years ago
Tariq Ibrahim	8fdfa8abea	refine error handling in prometheus (#5388) ... i) Uses the more idiomatic Wrap and Wrapf methods for creating nested errors. ii) Fixes some incorrect usages of fmt.Errorf where the error messages don't have any formatting directives. iii) Does away with the use of fmt package for errors in favour of pkg/errors Signed-off-by: tariqibrahim <tariq181290@gmail.com>	6 years ago
Tom Wilkie	c7b3535997	Use pkg/relabelling in remote write. ... - Unmarshall external_labels config as labels.Labels, add tests. - Convert some more uses of model.LabelSet to labels.Labels. - Remove old relabel pkg (fixes #3647). - Validate external label names. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Callum Styan	6f69e31398	Tail the TSDB WAL for remote_write ... This change switches the remote_write API to use the TSDB WAL. This should reduce memory usage and prevent sample loss when the remote end point is down. We use the new LiveReader from TSDB to tail WAL segments. Logic for finding the tracking segment is included in this PR. The WAL is tailed once for each remote_write endpoint specified. Reading from the segment is based on a ticker rather than relying on fsnotify write events, which were found to be complicated and unreliable in early prototypes. Enqueuing a sample for sending via remote_write can now block, to provide back pressure. Queues are still required to acheive parallelism and batching. We have updated the queue config based on new defaults for queue capacity and pending samples values - much smaller values are now possible. The remote_write resharding code has been updated to prevent deadlocks, and extra tests have been added for these cases. As part of this change, we attempt to guarantee that samples are not lost; however this initial version doesn't guarantee this across Prometheus restarts or non-retryable errors from the remote end (eg 400s). This changes also includes the following optimisations: - only marshal the proto request once, not once per retry - maintain a single copy of the labels for given series to reduce GC pressure Other minor tweaks: - only reshard if we've also successfully sent recently - add pending samples, latest sent timestamp, WAL events processed metrics Co-authored-by: Chris Marchbanks <csmarchbanks.com> (initial prototype) Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com> (sharding changes) Signed-off-by: Callum Styan <callumstyan@gmail.com>	6 years ago
Hrishikesh Barman	a1f34bec2e	Added CORS Origin flag (#5011) ... Signed-off-by: Hrishikesh Barman <hrishikeshbman@gmail.com>	7 years ago
Matt Layher	302148fd69	*: apply gofmt -s ... Signed-off-by: Matt Layher <mdlayher@gmail.com>	7 years ago
Callum Styan	5358f76c5c	update remote write path proto so that Labels/Timeseries can't be nil (#4957) ... Signed-off-by: Callum Styan <callumstyan@gmail.com>	7 years ago
Tom Wilkie	6e08029b56	Move err to be the last return value from storage.Select. (#5054) ... Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	7 years ago
Julius Volz	11a52be1d8	Better rounding for incoming query timestamps (#4941) ... Fixes https://github.com/prometheus/prometheus/issues/4939 Signed-off-by: Julius Volz <julius.volz@gmail.com>	7 years ago
mknapphrt	f0e9196dca	Return warnings on a remote read fail (#4832) ... Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>	7 years ago
Alex Yu	5dcce32ef8	update promlog to latest version (#4876) ... * update promlog to latest version Signed-off-by: Alex Yu <yu.alex96@gmail.com> * Update api tests, fix main setup Signed-off-by: Alex Yu <yu.alex96@gmail.com> * tidy go.sum Signed-off-by: Alex Yu <yu.alex96@gmail.com> * revendor prometheus/common Signed-off-by: Alex Yu <yu.alex96@gmail.com> * only initialize config; use kingpin for remote_storage_adapter Signed-off-by: Alex Yu <yu.alex96@gmail.com> * actually parse the flags Signed-off-by: Alex Yu <yu.alex96@gmail.com> * clean up imports Signed-off-by: Alex Yu <yu.alex96@gmail.com>	7 years ago
Ganesh Vernekar	ca93fd544b	/api/v1/labels endpoint for getting all label names (#4835) ... * vendor: update tsdb Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * /api/v1/labels endpoint Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * regex matchers for API Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Add docs Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Matchers behaving as OR Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Removed the matchers Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * vendor: update tsdb using go mod Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * vendor update: tsdb Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Added LabelNames() to storage.Querier Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Test for api.labelNames Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Nits Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	7 years ago
Simon Pasquier	6fa8de132b	web/v1/api: add tests for admin actions (#4767) ... Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago
Simon Pasquier	c7a48a4a93	web/api/v1: fix targets endpoint ... Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago
Simon Pasquier	a308a186e4	web/api/v1: fix targets endpoint ... Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago