prometheus

Commit Graph

Author	SHA1	Message	Date
Owen Williams	aa1d46a9da	scrape: set validation and escaping defaults in default config vars (#16751 ) Fixes https://github.com/prometheus/prometheus/issues/16750 Signed-off-by: Owen Williams <owen.williams@grafana.com>	2 days ago
Carrie Edwards	7289d41940	Add tests for relabeling of type and unit labels (#16743 ) Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>	4 days ago
Łukasz Mierzwa	c528293376	Enhancement: Reload all scrape pools concurrently (#16595 ) * Reload all scrape pools concurrently At the moment all scrape pools that need to be reloaded are reloaded one by one. While reloads are ongoing mtxScrape is locked. For each pool that's being reloaded we need to wait until all targets are updated. This whole process can take a while and the more scrape pools to reload the longer. At the same time all pools are independent and there's no real reason to do them one-by-one. Reload each pool in a seperate goroutine so we finish config reload as ASAP as possible and unlock the mtxScrape. Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Address PR review feedback Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> --------- Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>	3 weeks ago
Bartlomiej Plotka	8e6b008608	feature: type-and-unit-labels (PROM-39 implementation) (#16228 ) * feature: type-and-unit-labels (extended MetricIdentity) Experimental implementation of https://github.com/prometheus/proposals/pull/39 Previous (unmerged) experiments: * https://github.com/prometheus/prometheus/compare/main...dashpole:prometheus:type_and_unit_labels * https://github.com/prometheus/prometheus/pull/16025 Signed-off-by: bwplotka <bwplotka@gmail.com> feature: type-and-unit-labels (extended MetricIdentity) Experimental implementation of https://github.com/prometheus/proposals/pull/39 Previous (unmerged) experiments: * https://github.com/prometheus/prometheus/compare/main...dashpole:prometheus:type_and_unit_labels * https://github.com/prometheus/prometheus/pull/16025 Signed-off-by: bwplotka <bwplotka@gmail.com> * Fix compilation errors Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Lint Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Revert change made to protobuf 'Accept' header Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Fix compilation errors for 'dedupelabels' tag Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> * Rectored into schema.Metadata Signed-off-by: bwplotka <bwplotka@gmail.com> * texparse: Added tests for PromParse Signed-off-by: bwplotka <bwplotka@gmail.com> * add OM tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * add proto tests Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> * add schema label tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> * fix tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * add promql tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * lint Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>	1 month ago
chardch	a1c157aaef	Add global config option for always_scrape_classic_histograms Addresses https://github.com/prometheus/prometheus/issues/16371 This will help with migrating to native histograms with `convert_classic_histograms_to_nhcb` since users may still need to keep the classic histograms during a migration Signed-off-by: chardch <otwordsne@gmail.com>	2 months ago
Arve Knudsen	e7e3ab2824	Fix linting issues found by golangci-lint v2.0.2 (#16368 ) * Fix linting issues found by golangci-lint v2.0.2 --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 months ago
chardch	357b5ed640	Fix small issues and clarify documentation Signed-off-by: chardch <otwordsne@gmail.com>	3 months ago
chardch	2f59d38054	Add global config option for convert_classic_histograms_to_nhcb Addresses the global part of https://github.com/prometheus/prometheus/issues/13532 Signed-off-by: chardch <otwordsne@gmail.com>	3 months ago
Owen Williams	6566c5a2b3	scrape: Add config option for escaping scheme request. (#16066 ) The new metric_name_escaping_scheme config option works in parallel with metric_name_validation_scheme and controls which escaping scheme is requested when scraping. When not specified, the scheme will request underscores if the validation scheme is set to legacy, and will request allow-utf-8 when the validation scheme is set to utf8. This setting allows users to allow utf8 names if they like, but explicitly request an escaping scheme rather than UTF-8. Fixes https://github.com/prometheus/prometheus/issues/16034 Built on https://github.com/prometheus/prometheus/pull/16080 Signed-off-by: Owen Williams <owen.williams@grafana.com>	3 months ago
György Krajcsovits	f71ec8d5d5	fix(scrape): skip native histogram series if ingestion is disabled No need to validate, track, add sample, add exemplar if series isn't going to be ingested. Basically this is the same as if this series was dropped by relabelling. Fixes #16217 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	3 months ago
Matthieu MOREL	5fa1146e21	chore: enable gci linter (#16245 ) Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	3 months ago
Bryan Boreham	e86b9812f4	[BUGFIX] Scraping: bump cache iteration after error Backport of #16174 to LTS release-2.53 branch. We use the cache iteration number to detect when the same metric has occurred twice in a scrape. We need to bump this number at the end of every scrape, not just successful scrapes. The `iter` number is also used: * After a successful scrape, to delete older metrics and metadata. * To detect when metadata changed in this scrape. None of those additional cases is broken by incrementing the number on error. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 months ago
Bryan Boreham	63d1e56525	[TESTS] Scrape: make caching work in benchmark Returning 0 from Append means 'unknown', so the series is never cached. Return arbitrary numbers instead. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 months ago
György Krajcsovits	fec2f2603a	fix(scrape): skip native histogram series if ingestion is disabled No need to validate, track, add sample, add exemplar if series isn't going to be ingested. Basically this is the same as if this series was dropped by relabelling. Fixes #16217 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	3 months ago
Owen Williams	94b43c5d4c	utf8: Remove support for legacy global validation setting Global and Data Source configurations can specify legacy mode, but Prometheus now requires that the overall validation mode be set to UTF-8 Signed-off-by: Owen Williams <owen.williams@grafana.com>	4 months ago
MarkDaveny	53be282396	refactor: use slices.Equal to simplify code Signed-off-by: MarkDaveny <peicuiping@aliyun.com>	4 months ago
Bryan Boreham	e32d89af7f	[BUGFIX] Scraping: bump cache iteration after error (#16174 ) We use the cache iteration number to detect when the same metric has occurred twice in a scrape. We need to bump this number at the end of every scrape, not just successful scrapes. The `iter` number is also used: * After a successful scrape, to delete older metrics and metadata. * To detect when metadata changed in this scrape. None of those additional cases is broken by incrementing the number on error. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	4 months ago
Bartlomiej Plotka	dc85d677d8	textparse: Optimize CreatedTimestamp; It returns int64 value now. (#16072 ) ..instead of *int64. This is as an optimization and ease of use. We already accepted in many places (proto histograms, PRW) that CT (or any timestamp really) 0 means not set. Signed-off-by: bwplotka <bwplotka@gmail.com>	4 months ago
boforetech	6ec89f14c7	refactor: use a more straightforward return value (#16070 ) Signed-off-by: beforetech <mail@before.tech>	4 months ago
Matt Hughes	528b3fc60f	scraper: fix UTF-8 scraping header always sent with PrometheusText1.0.0 The `Accept` header should not include `escape=allow-utf-8` unless explicitly requested. Conveniently, there was already a test covering this header's value, it just required updating so it also asserts that this value in the header is not set in the cases we don't expect it to be set. I also converted those tests into table tests to help make failures clearer. Issue: https://github.com/prometheus/prometheus/issues/15857 Signed-off-by: Matt Hughes <mhughes@uw.co.uk>	4 months ago
Matt Hughes	5868e36d91	scraper: fix UTF-8 scraping header always sent with PrometheusText1.0.0 The `Accept` header should not include `escape=allow-utf-8` unless explicitly requested. Conveniently, there was already a test covering this header's value, it just required updating so it also asserts that this value in the header is not set in the cases we don't expect it to be set. I also converted those tests into table tests to help make failures clearer. Issue: https://github.com/prometheus/prometheus/issues/15857 Signed-off-by: Matt Hughes <mhughes@uw.co.uk>	4 months ago
Matthieu MOREL	c7d4b53ec1	chore: enable unused-parameter from revive Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	4 months ago
Laimis Juzeliūnas	a5ffa83be8	scrape: reorder the switch case of errors (#15991 ) Change case order for switch scrapeLoop This commit changes the ordering in error identification switch cases for better production performance and adds reasonings behind error switch case order as comments. --------- Signed-off-by: Laimis Juzeliūnas <asnelaimis@gmail.com>	4 months ago
Bartlomiej Plotka	733a5e9eb4	textparse: Optimized protobuf parser with custom streaming unmarshal. (#15731 ) * textparse: Optimized protobuf parser with custom streaming decoder. Signed-off-by: bwplotka <bwplotka@gmail.com> Update model/textparse/protobufparse.go Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Addressing comments. Signed-off-by: bwplotka <bwplotka@gmail.com> decoder: reuse histograms and summaries. Signed-off-by: bwplotka <bwplotka@gmail.com> optimize help returning (5% of mem utilization). Signed-off-by: bwplotka <bwplotka@gmail.com> Apply suggestions from code review Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Update prompb/io/prometheus/client/decoder.go Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Fix build. Signed-off-by: bwplotka <bwplotka@gmail.com> * Update model/textparse/protobufparse.go Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	4 months ago
Bartlomiej Plotka	00b69efabb	model/textparse: Change parser interface Metric(...) string to Labels(...) (#16012 ) * model/textparse: Change parser interface Metric(...) string to Labels(...) Simplified the interface given no one is using the return argument. Renamed for clarity too. Found and discussed https://github.com/prometheus/prometheus/pull/15731#discussion_r1950916842 Signed-off-by: bwplotka <bwplotka@gmail.com> * Fixed comments; optimized not needed copy for om and text. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com>	4 months ago
Bartlomiej Plotka	8cd9069cf1	textparse: Refactor benchmark testdata for all types. (#15998 ) Also: * split benchmark functions to make sure no one compares across parsers. * testdata file have meaningful names reflecting the type representation * promtestdata.txt now has all types, taken directly from long running Prometheus (https://demo.do.prometheus.io/) Needed for https://github.com/prometheus/prometheus/pull/15731 Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
Bartlomiej Plotka	7427753922	scrape: Add realistic data case for scrape loop append bench. (#15966 ) * scrape: Add realistic data case for scrape loop append bench. Signed-off-by: bwplotka <bwplotka@gmail.com> * Update scrape/scrape_test.go Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>	5 months ago
Matthieu MOREL	dd5ab743ea	chore(deps): use version.PrometheusUserAgent Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	5 months ago
Bartlomiej Plotka	6f1ad64e27	Update scrape/scrape.go Co-authored-by: Julius Volz <julius.volz@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 months ago
bwplotka	3119567d5b	scrape: Add metadata for automatic metrics. Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
bwplotka	af928a1ba4	Addressed Krajo's comment. Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
bwplotka	9385f31147	scrape: Fix metadata in WAL not working for histograms and summaries. The was a bug (due to confusion?) on the local metadata cache that is cached by metric family not the series metric name. The fix is to NOT use that local cache at all (it's still needed for current metadata API implementation, added TODO on how we can get rid of it). I went ahead and also rename Metric field in metadata structs to MetricFamily to make clear it's not always __name__. Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
Arve Knudsen	f030894c2c	Fix issues raised by staticcheck (#15722 ) Fix issues raised by staticcheck We are not enabling staticcheck explicitly, though, because it has too many false positives. --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	6 months ago
bwplotka	281306765e	scrape: Unified scrape loop benchmark. Signed-off-by: bwplotka <bwplotka@gmail.com>	6 months ago
Bryan Boreham	bc9210e393	[TESTS] Scrape: make caching work in benchmark Returning 0 from Append means 'unknown', so the series is never cached. Return arbitrary numbers instead. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago
Bryan Boreham	b4ef38cfc8	Scraping: Add benchmark for protobuf format Extract helper function textToProto(). Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago
Bryan Boreham	8f4557b0b1	Scraping benchmark: more realistic test Don't repeat type and help text. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago
Bryan Boreham	7b03796d0f	Scraping: stop storing discovered labels (#15261 ) Instead of storing discovered labels on every target, recompute them if required. The `Target` struct now needs to hold some more data required to recompute them, such as ScrapeConfig. This moves the load from every Prometheus all of the time, to just when someone views Service Discovery in the UI. The way `PopulateLabels` is used changes; you are no longer expected to call it with a part-populated `labels.Builder`. The signature of `Target.Labels` changes to take a `labels.Builder` instead of a `ScratchBuilder`, for consistency with `DiscoveredLabels`. This will save a lot of work when many targets are filtered out in relabeling. Combine with `keep_dropped_targets` to avoid ever computing most labels for dropped targets. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago
Joel Beckmeyer	39f5a07236	fix TestOOOHeadChunkReader_Chunk on 32-bit Signed-off-by: Joel Beckmeyer <joel@beckmeyer.us>	6 months ago
David Ashpole	953a873342	update links to openmetrics to reference the v1.0.0 release Signed-off-by: David Ashpole <dashpole@google.com>	7 months ago
bwplotka	f44bba31b3	config: Remove validation GetScrapeConfigs; require using config.Load. We should never modify (or even shallow copy) Config after config.Load; added comments and modified GetScrapeConfigs to do so. For GetScrapeConfigs the validation (even repeated) was likely doing writes (because global fields was 0). We GetScrapeConfigs concurrently in tests and ApplyConfig causing test races. In prod there were races but likelyt only to replace 0 with 0, so not too severe. I removed validation since I don't see anyone using our config.Config without Load. I had to refactor one test that was doing it, all others use yaml config. Fixes #15538 Previous attempt: https://github.com/prometheus/prometheus/pull/15634 Signed-off-by: bwplotka <bwplotka@gmail.com>	7 months ago
Giedrius Statkevičius	5d76510bd6	scrape: update test name Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	7 months ago
Giedrius Statkevičius	5479fb3cfd	scrape: update test after review Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	7 months ago
machine424	7719ed18d7	test: add TestTargetsFromGroupWithLabelKeepDrop to reinforce a relabelling behaviour. fixes https://github.com/prometheus/prometheus/issues/12355 Signed-off-by: machine424 <ayoubmrini424@gmail.com>	7 months ago
Giedrius Statkevičius	90e832c861	scrape: fix nil panic after scrape loop reload Don't forget to set `metrics` field as otherwise scraping will lead to a nil panic in case the body size limit is reached. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	7 months ago
Ben Ye	e10bbf0a84	hot reload always_scrape_classic_histograms and convert_classic_histograms_to_nhcb configs properly Signed-off-by: Ben Ye <benye@amazon.com>	7 months ago
TJ Hoplock	e0104a6b7e	ref: JSONFileLogger slog handler, add scrape.FailureLogger interface Improves upon #15434, better resolves #15433. This commit introduces a few changes to ensure safer handling of the JSONFileLogger: - the JSONFileLogger struct now implements the slog.Handler interface, so it can directly be used to create slog Loggers. This pattern more closely aligns with upstream slog usage (which is generally based around handlers), as well as making it clear that devs are creating a whole new logger when interacting with it (vs silently modifying internal configs like it did previously). - updates the `promql.QueryLogger` interface to be a union of the methods of both the `io.Closer` interface and the `slog.Handler` interface. This allows for plugging in/using slog-compatible loggers other than the JSONFileLogger, if desired (ie, for downstream projects). - introduces new `scrape.FailureLogger` interface; just like `promql.QueryLogger`, it is a union of `io.Closer` and `slog.Handler` interfaces. Similar logic applies to reasoning. - updates tests where needed; have the `FakeQueryLogger` from promql's engine_test implement the `slog.Handler`, improve JSONFileLogger test suite, etc. Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	7 months ago
TJ Hoplock	3e24e84172	fix!: stop unbounded memory usage from query log Resolves: #15433 When I converted prometheus to use slog in #14906, I update both the `QueryLogger` interface, as well as how the log calls to the `QueryLogger` were built up in `promql.Engine.exec()`. The backing logger for the `QueryLogger` in the engine is a `util/logging.JSONFileLogger`, and it's implementation of the `With()` method updates the logger the logger in place with the new keyvals added onto the underlying slog.Logger, which means they get inherited onto everything after. All subsequent calls to `With()`, even in later queries, would continue to then append on more and more keyvals for the various params and fields built up in the logger. In turn, this causes unbounded growth of the logger, leading to increased memory usage, and in at least one report was the likely cause of an OOM kill. More information can be found in the issue and the linked slack thread. This commit does a few things: - It was referenced in feedback in #14906 that it would've been better to not change the `QueryLogger` interface if possible, this PR proposes changes that bring it closer to alignment with the pre-3.0 `QueryLogger` interface contract - reverts `promql.Engine.exec()`'s usage of the query logger to the pattern of building up an array of args to pass at once to the end log call. Avoiding the repetitious calls to `.With()` are what resolve the issue with the logger growth/memory usage. - updates the scrape failure logger to use the update `QueryLogger` methods in the contract. - updates tests accordingly - cleans up unused methods Builds and passes tests successfully. Tested locally and confirmed I could no longer reproduce the issue/it resolved the issue. Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	7 months ago
Arve Knudsen	89bbb885e5	Upgrade to golangci-lint v1.62.0 (#15424 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	7 months ago
Bryan Boreham	6380229c71	[REFACTOR] Scraping: rename variables for clarity Instead of shadowing the outer variables, use different names. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	8 months ago

1 2 3 4 5 ...

471 Commits (main)