prometheus

Commit Graph

Author	SHA1	Message	Date
Bryan Boreham	d0dee51aac	scrape tests: check NaN values directly Normally, a NaN value is never equal to any other value. Compare sample values via `Float64bits` so that NaN values which are exactly the same will compare equal. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Bryan Boreham	39af788dbd	Tests: use replacement DeepEquals using go-cmp Use DeepEqual replacement using go-cmp, which is more flexible. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Paweł Szulik	b0c538787d	Refactor scrape tests to use testify. Signed-off-by: Paweł Szulik <paul.szulik@gmail.com>	1 year ago
Ziqi Zhao	df2a0ecf3b	Native Histograms: support `native_histogram_min_bucket_factor` in scrape_config (#13222 ) Native Histograms: support native_histogram_min_bucket_factor in scrape_config --------- Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Co-authored-by: Björn Rabenstein <github@rabenste.in>	1 year ago
machine424	2f60177203	scrape_test.go: Increase scrape interval in TestScrapeLoopCache to reduce potential flakiness Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2 years ago
Bryan Boreham	8065bef172	Move metric type definitions to common/model They are used in multiple repos, so common is a better place for them. Several packages now don't depend on `model/textparse`, e.g. `storage/remote`. Also remove `metadata` struct from `api.go`, since it was identical to a struct in the `metadata` package. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Arthur Silva Sens	5082655392	Append Created Timestamps (#12733 ) * Append created timestamps. Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> * Log when created timestamps are ignored Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> * Proposed changes to Append CT PR. Changes: * Changed textparse Parser interface for consistency and robustness. * Changed CT interface to be more explicit and handle validation. * Simplified test, change scrapeManager to allow testability. * Added TODOs. Signed-off-by: bwplotka <bwplotka@gmail.com> * Updates. Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> * Refactor head_appender test Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> * Fix linter issues Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> * Use model.Sample in head appender test Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> --------- Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com> Signed-off-by: bwplotka <bwplotka@gmail.com> Co-authored-by: bwplotka <bwplotka@gmail.com>	2 years ago
Matthieu MOREL	9c4782f1cc	golangci-lint: enable testifylint linter (#13254 ) Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2 years ago
Bryan Boreham	3e287e0170	Scraping tests: refactor scrapeLoop creation Pull boilerplate code into a function. Where appropriate we set some config on the returned object. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Paulin Todev	0102425af1	Use only one scrapeMetrics object per test. (#13051 ) The scrape loop and scrape cache should use the same instance. This brings the tests' behavior more in line with production. Signed-off-by: Paulin Todev <paulin.todev@gmail.com>	2 years ago
Bryan Boreham	9051100aba	Scraping: share buffer pool across all scrapes Previously we had one per scrapePool, and one of those per configured scraping job. Each pool holds a few unused buffers, so sharing one across all scrapePools reduces total heap memory. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Ziqi Zhao	8fe9250f7d	optimize the logic of break the loop of reducing resolution Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	2 years ago
Łukasz Mierzwa	870627fbed	Add enable_compression scrape config option Currently Prometheus will always request gzip compression from the target when sending scrape requests. HTTP compression does reduce the amount of bytes sent over the wire and so is often desirable. The downside of compression is that it requires extra resources - cpu & memory. This also affects the resource usage on the target since it has to compress the response before sending it to Prometheus. This change adds a new option to the scrape job configuration block: enable_compression. The default is true so it remains the same as current Prometheus behaviour. Setting this option to false allows users to disable compression between Prometheus and the scraped target, which will require more bandwidth but it lowers the resource usage of both Prometheus and the target. Fixes #12319. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	2 years ago
zenador	32ee1b15de	Fix error on ingesting out-of-order exemplars (#13021 ) Fix and improve ingesting exemplars for native histograms. See code comment for a detailed explanation of the algorithm. Note that this changes the current behavior for all kind of samples slightly: We now allow exemplars with the same timestamp as during the last scrape if the value or the labels have changed. Also note that we now do not ingest exemplars without timestamps for native histograms anymore. Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: Björn Rabenstein <github@rabenste.in> --------- Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Signed-off-by: zenador <zenador@users.noreply.github.com> Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: Björn Rabenstein <github@rabenste.in>	2 years ago
Julien Pivotto	0fe34f6d78	Follow-up to #13060 : Add test to ensure staleness tracking This commit introduces an additional test in `scrape_test.go` to verify staleness tracking when `trackTimestampStaleness` is enabled. The new `TestScrapeLoopAppendStalenessIfTrackTimestampStaleness` function asserts that the scrape loop correctly appends staleness markers when necessary, reflecting the expected behavior with the feature flag turned on. The previous tests were only testing end of scrape staleness. Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	2 years ago
Matthieu MOREL	7eaefcf379	ci(lint): enable errorlint on scrape (#12923 ) Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com> Signed-off-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com> Co-authored-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>	2 years ago
George Krajcsovits	e399395b01	Native histograms vs labels (#13005 ) * Document le and quantile label transition due to native histograms Fixes: #12984 For full explanation see the related issue. The le and quantile labels are formatted as float with trailing .0 for whole number values when native histograms is enabled, e.g. 10.0. This changes the resulting series in Prometheus if previously we scraped the whole number itself, e.g. 10 over the text format. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>	2 years ago
Julien Pivotto	fed6f81377	scrape: Added trackTimestampsStaleness configuration option Add the ability to track staleness when an explicit timestamp is set. Useful for cAdvisor. Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	2 years ago
Julien Pivotto	84aadfc45b	scrape: Added trackTimestampsStaleness configuration option Add the ability to track staleness when an explicit timestamp is set. Useful for cAdvisor. Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	2 years ago
Paulin Todev	5752050b42	Scrape metrics can now be registered with a non-default registry. * A registerer is passed to the scrape Manager, and all scrape metrics register with it. * For now the registry which we pass to the scrape Manager is still the global one. Signed-off-by: Paulin Todev <paulin.todev@gmail.com>	2 years ago
Bartlomiej Plotka	624b973ebf	Added ability to specify scrape protocols to accept during HTTP content type negotiation. (#12738 ) * Added ability to specify scrape protocols to accept during HTTP content type negotiation. This is done via new option in GlobalConfig and ScrapeConfig: "scrape_protocol" Signed-off-by: bwplotka <bwplotka@gmail.com> * Fixed readability and log message. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com>	2 years ago
Bryan Boreham	f6d9c84fde	scraping: delay creating buffer, to save memory (#12953 ) We don't need the buffer to read the response until the scrape http call returns; creating it earlier makes the buffer pool larger. I split `scrape()` into `scrape()` which returns with the http response, and `readResponse()` which decompresses and copies the data into the supplied buffer. This design was chosen to minimize impact on the logic. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Arve Knudsen	6daee89e5f	Add context argument to Querier.Select (#12660 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
György Krajcsovits	983c0c5e9d	Add missing buckets My previous proposal for a fix was wrong and also missed these. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
György Krajcsovits	2ae8c2bd3d	Set expected values in test The parsing doesn't seem to be perfect as I don't get all classic buckets possibly another bug found? Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
György Krajcsovits	2a781ec5ac	Replicate infinite loop in native-classic histogram scrape Enable scraping a native histogram with exemplars that leads to infinite loop. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
Bryan Boreham	627c99424b	scrape: extend TestDroppedTargetsList to check counts Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	1e3fef6ab0	scraping: limit detail on dropped targets, to save memory (#12647 ) It's possible (quite common on Kubernetes) to have a service discovery return thousands of targets then drop most of them in relabel rules. The main place this data is used is to display in the web UI, where you don't want thousands of lines of display. The new limit is `keep_dropped_targets`, which defaults to 0 for backwards-compatibility. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
beorn7	536a487af4	scrape: Refactor names of float samples Continue to remove confusion that histogram samples are also samples and histogram values are also values etc. by renaming float values and float samples using the same schema as for histograms. Concretely: - result → resultFloats (corresponding to resultHistograms) - pendingResult → pendingFloats (corresponding to pendingHistograms) - rolledbackResult → rolledbackFloats (corresponding to rolledbackHistograms) - sample → floatSample (corresponding to histogramSample) This also order the fields in `collectResultAppender` more consistently. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	0e3f35324b	scrape: Enable ingestion of multiple exemplars per sample This has become a requirement for native histograms, as a single histogram sample commonly has many buckets, so that providing many exemplars makes sense. Since OM text doesn't support native histograms yet, the test had to be expanded to also support protobuf test cases. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	9e500345f3	textparse/scrape: Add option to scrape both classic and native histograms So far, if a target exposes a histogram with both classic and native buckets, a native-histogram enabled Prometheus would ignore the classic buckets. With the new scrape config option `scrape_classic_histograms` set, both buckets will be ingested, creating all the series of a classic histogram in parallel to the native histogram series. For example, a histogram `foo` would create a native histogram series `foo` and classic series called `foo_sum`, `foo_count`, and `foo_bucket`. This feature can be used in a migration strategy from classic to native histograms, where it is desired to have a transition period during which both native and classic histograms are present. Note that two bugs in classic histogram parsing were found and fixed as a byproduct of testing the new feature: 1. Series created from classic _gauge_ histograms didn't get the _sum/_count/_bucket prefix set. 2. Values of classic _float_ histograms weren't parsed properly. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
György Krajcsovits	19a4f314f5	Refactor testutil/protobuf.go into scrape package Renamed to clientprotobuf.go and added comments to indicate the intended usage. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
Russ Cox	28f5502828	scrape: fix two loop variable scoping bugs in test Consider code like: for i := 0; i < numTargets; i++ { stopFuncs = append(stopFuncs, func() { time.Sleep(i20time.Millisecond) }) } Because the loop variable i is shared by all closures, all the stopFuncs sleep for numTargets20 ms. If the i were made per-iteration, as we are considering for a future Go release, the stopFuncs would have sleep durations ranging from 0 to (numTargets-1)20 ms. Two tests had code like this and were checking that the aggregate sleep was at least numTargets20 ms ("at least as long as the last target slept"). This is only true today because i == numTarget during all the sleeps. To keep the code working even if the semantics of this loop change, this PR computes d := time.Duration((i+1)20) * time.Millisecond outside the closure (but inside the loop body), and then each closure has its own d. Now the sleeps range from 20 ms to numTargets*20 ms, keeping the test passing (and probably behaving closer to the intent of the test author). The failure being fixed can be reproduced by using the current Go development branch with GOEXPERIMENT=loopvar go test Signed-off-by: Russ Cox <rsc@golang.org>	2 years ago
Jeanette Tan	dfabc69303	Add tests according to code review Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2 years ago
Jeanette Tan	2ad39baa72	Treat bucket limit like sample limit and make it fail the whole scrape and return an error Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2 years ago
György Krajcsovits	071426f72f	Add unit test for bucket limit appender Refactors textparser test to use a common test utility to create protobuf representation from MetricFamily Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
Jeanette Tan	4d21ac23e6	Implement bucket limit for native histograms Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2 years ago
beorn7	5b53aa1108	style: Replace `else if` cascades with `switch` Wiser coders than myself have come to the conclusion that a `switch` statement is almost always superior to a statement that includes any `else if`. The exceptions that I have found in our codebase are just these two: * The `if else` is followed by an additional statement before the next condition (separated by a `;`). * The whole thing is within a `for` loop and `break` statements are used. In this case, using `switch` would require tagging the `for` loop, which probably tips the balance. Why are `switch` statements more readable? For one, fewer curly braces. But more importantly, the conditions all have the same alignment, so the whole thing follows the natural flow of going down a list of conditions. With `else if`, in contrast, all conditions but the first are "hidden" behind `} else if `, harder to spot and (for no good reason) presented differently from the first condition. I'm sure the aforemention wise coders can list even more reasons. In any case, I like it so much that I have found myself recommending it in code reviews. I would like to make it a habit in our code base, without making it a hard requirement that we would test on the CI. But for that, there has to be a role model, so this commit eliminates all `if else` occurrences, unless it is autogenerated code or fits one of the exceptions above. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	c3c7d44d84	lint: Adjust to the lint warnings raised by current versions of golint-ci We haven't updated golint-ci in our CI yet, but this commit prepares for that. There are a lot of new warnings, and it is mostly because the "revive" linter got updated. I agree with most of the new warnings, mostly around not naming unused function parameters (although it is justified in some cases for documentation purposes – while things like mocks are a good example where not naming the parameter is clearer). I'm pretty upset about the "empty block" warning to include `for` loops. It's such a common pattern to do something in the head of the `for` loop and then have an empty block. There is still an open issue about this: https://github.com/mgechev/revive/issues/810 I have disabled "revive" altogether in files where empty blocks are used excessively, and I have made the effort to add individual `// nolint:revive` where empty blocks are used just once or twice. It's borderline noisy, though, but let's go with it for now. I should mention that none of the "empty block" warnings for `for` loop bodies were legitimate. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Matthieu MOREL	fb3eb21230	enable gocritic, unconvert and unused linters Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2 years ago
Julien Pivotto	0c56e5d014	Update our own dependencies, support proxy from env Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	2 years ago
Bryan Boreham	bec5abc4dc	scrape: remove unsafe code The `yolostring` routine was intended to avoid an allocation when converting from a `[]byte` to a `string` for map lookup. However, since 2014 Go has recognized this pattern and does not make a copy of the data when looking up a map. So the unsafe code is not necessary. In line with this, constants like `scrapeHealthMetricName` also become `[]byte`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	9bc6d7a7db	Update package scrape tests for new labels.Labels type Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	3c7de69059	storage: allow re-use of iterators Patterned after `Chunk.Iterator()`: pass the old iterator in so it can be re-used to avoid allocating a new object. (This commit does not do any re-use; it is just changing all the method signatures so re-use is possible in later commits.) Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Xiaochao Dong (@damnever)	9979024a30	Report error if the series contains invalid metric names or labels during scrape Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>	3 years ago
Ganesh Vernekar	3cbf87b83d	Enable protobuf negotiation only when histograms are enabled Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Bryan Boreham	4927e13537	scrape tests: undo EmptyLabels change Needs other code changes otherwise tests fail Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	14780c3b4e	scrape: in tests use labels.FromStrings And a few cases of `EmptyLabels()`. Replacing code which assumes the internal structure of `Labels`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bogdan Drutu	3cde9287a6	scrape: remove unused member from cacheEntry (#11281 ) Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>	3 years ago
Bogdan Drutu	c8cfe5c25d	scrape: remove unused argument in newScrapeLoop (#11283 ) Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com> Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>	3 years ago

1 2 3 4

156 Commits (ff6c83269c61cbec2442daf8bcc92ff5ef35fcef)