prometheus

Commit Graph

Author	SHA1	Message	Date
Owen Williams	aa1d46a9da	scrape: set validation and escaping defaults in default config vars (#16751 ) Fixes https://github.com/prometheus/prometheus/issues/16750 Signed-off-by: Owen Williams <owen.williams@grafana.com>	2 days ago
Bartlomiej Plotka	8e6b008608	feature: type-and-unit-labels (PROM-39 implementation) (#16228 ) * feature: type-and-unit-labels (extended MetricIdentity) Experimental implementation of https://github.com/prometheus/proposals/pull/39 Previous (unmerged) experiments: * https://github.com/prometheus/prometheus/compare/main...dashpole:prometheus:type_and_unit_labels * https://github.com/prometheus/prometheus/pull/16025 Signed-off-by: bwplotka <bwplotka@gmail.com> feature: type-and-unit-labels (extended MetricIdentity) Experimental implementation of https://github.com/prometheus/proposals/pull/39 Previous (unmerged) experiments: * https://github.com/prometheus/prometheus/compare/main...dashpole:prometheus:type_and_unit_labels * https://github.com/prometheus/prometheus/pull/16025 Signed-off-by: bwplotka <bwplotka@gmail.com> * Fix compilation errors Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Lint Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Revert change made to protobuf 'Accept' header Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Fix compilation errors for 'dedupelabels' tag Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> * Rectored into schema.Metadata Signed-off-by: bwplotka <bwplotka@gmail.com> * texparse: Added tests for PromParse Signed-off-by: bwplotka <bwplotka@gmail.com> * add OM tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * add proto tests Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> * add schema label tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> * fix tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * add promql tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * lint Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>	1 month ago
chardch	a1c157aaef	Add global config option for always_scrape_classic_histograms Addresses https://github.com/prometheus/prometheus/issues/16371 This will help with migrating to native histograms with `convert_classic_histograms_to_nhcb` since users may still need to keep the classic histograms during a migration Signed-off-by: chardch <otwordsne@gmail.com>	2 months ago
chardch	357b5ed640	Fix small issues and clarify documentation Signed-off-by: chardch <otwordsne@gmail.com>	3 months ago
chardch	2f59d38054	Add global config option for convert_classic_histograms_to_nhcb Addresses the global part of https://github.com/prometheus/prometheus/issues/13532 Signed-off-by: chardch <otwordsne@gmail.com>	3 months ago
Owen Williams	6566c5a2b3	scrape: Add config option for escaping scheme request. (#16066 ) The new metric_name_escaping_scheme config option works in parallel with metric_name_validation_scheme and controls which escaping scheme is requested when scraping. When not specified, the scheme will request underscores if the validation scheme is set to legacy, and will request allow-utf-8 when the validation scheme is set to utf8. This setting allows users to allow utf8 names if they like, but explicitly request an escaping scheme rather than UTF-8. Fixes https://github.com/prometheus/prometheus/issues/16034 Built on https://github.com/prometheus/prometheus/pull/16080 Signed-off-by: Owen Williams <owen.williams@grafana.com>	3 months ago
György Krajcsovits	f71ec8d5d5	fix(scrape): skip native histogram series if ingestion is disabled No need to validate, track, add sample, add exemplar if series isn't going to be ingested. Basically this is the same as if this series was dropped by relabelling. Fixes #16217 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	3 months ago
Bryan Boreham	e86b9812f4	[BUGFIX] Scraping: bump cache iteration after error Backport of #16174 to LTS release-2.53 branch. We use the cache iteration number to detect when the same metric has occurred twice in a scrape. We need to bump this number at the end of every scrape, not just successful scrapes. The `iter` number is also used: * After a successful scrape, to delete older metrics and metadata. * To detect when metadata changed in this scrape. None of those additional cases is broken by incrementing the number on error. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 months ago
György Krajcsovits	fec2f2603a	fix(scrape): skip native histogram series if ingestion is disabled No need to validate, track, add sample, add exemplar if series isn't going to be ingested. Basically this is the same as if this series was dropped by relabelling. Fixes #16217 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	3 months ago
Bryan Boreham	e32d89af7f	[BUGFIX] Scraping: bump cache iteration after error (#16174 ) We use the cache iteration number to detect when the same metric has occurred twice in a scrape. We need to bump this number at the end of every scrape, not just successful scrapes. The `iter` number is also used: * After a successful scrape, to delete older metrics and metadata. * To detect when metadata changed in this scrape. None of those additional cases is broken by incrementing the number on error. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	4 months ago
Bartlomiej Plotka	dc85d677d8	textparse: Optimize CreatedTimestamp; It returns int64 value now. (#16072 ) ..instead of *int64. This is as an optimization and ease of use. We already accepted in many places (proto histograms, PRW) that CT (or any timestamp really) 0 means not set. Signed-off-by: bwplotka <bwplotka@gmail.com>	4 months ago
Matthieu MOREL	c7d4b53ec1	chore: enable unused-parameter from revive Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	4 months ago
Laimis Juzeliūnas	a5ffa83be8	scrape: reorder the switch case of errors (#15991 ) Change case order for switch scrapeLoop This commit changes the ordering in error identification switch cases for better production performance and adds reasonings behind error switch case order as comments. --------- Signed-off-by: Laimis Juzeliūnas <asnelaimis@gmail.com>	4 months ago
Bartlomiej Plotka	00b69efabb	model/textparse: Change parser interface Metric(...) string to Labels(...) (#16012 ) * model/textparse: Change parser interface Metric(...) string to Labels(...) Simplified the interface given no one is using the return argument. Renamed for clarity too. Found and discussed https://github.com/prometheus/prometheus/pull/15731#discussion_r1950916842 Signed-off-by: bwplotka <bwplotka@gmail.com> * Fixed comments; optimized not needed copy for om and text. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
Matthieu MOREL	dd5ab743ea	chore(deps): use version.PrometheusUserAgent Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	5 months ago
Bartlomiej Plotka	6f1ad64e27	Update scrape/scrape.go Co-authored-by: Julius Volz <julius.volz@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 months ago
bwplotka	3119567d5b	scrape: Add metadata for automatic metrics. Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
bwplotka	af928a1ba4	Addressed Krajo's comment. Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
bwplotka	9385f31147	scrape: Fix metadata in WAL not working for histograms and summaries. The was a bug (due to confusion?) on the local metadata cache that is cached by metric family not the series metric name. The fix is to NOT use that local cache at all (it's still needed for current metadata API implementation, added TODO on how we can get rid of it). I went ahead and also rename Metric field in metadata structs to MetricFamily to make clear it's not always __name__. Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
Bryan Boreham	7b03796d0f	Scraping: stop storing discovered labels (#15261 ) Instead of storing discovered labels on every target, recompute them if required. The `Target` struct now needs to hold some more data required to recompute them, such as ScrapeConfig. This moves the load from every Prometheus all of the time, to just when someone views Service Discovery in the UI. The way `PopulateLabels` is used changes; you are no longer expected to call it with a part-populated `labels.Builder`. The signature of `Target.Labels` changes to take a `labels.Builder` instead of a `ScratchBuilder`, for consistency with `DiscoveredLabels`. This will save a lot of work when many targets are filtered out in relabeling. Combine with `keep_dropped_targets` to avoid ever computing most labels for dropped targets. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago
Giedrius Statkevičius	90e832c861	scrape: fix nil panic after scrape loop reload Don't forget to set `metrics` field as otherwise scraping will lead to a nil panic in case the body size limit is reached. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	7 months ago
Ben Ye	e10bbf0a84	hot reload always_scrape_classic_histograms and convert_classic_histograms_to_nhcb configs properly Signed-off-by: Ben Ye <benye@amazon.com>	7 months ago
TJ Hoplock	e0104a6b7e	ref: JSONFileLogger slog handler, add scrape.FailureLogger interface Improves upon #15434, better resolves #15433. This commit introduces a few changes to ensure safer handling of the JSONFileLogger: - the JSONFileLogger struct now implements the slog.Handler interface, so it can directly be used to create slog Loggers. This pattern more closely aligns with upstream slog usage (which is generally based around handlers), as well as making it clear that devs are creating a whole new logger when interacting with it (vs silently modifying internal configs like it did previously). - updates the `promql.QueryLogger` interface to be a union of the methods of both the `io.Closer` interface and the `slog.Handler` interface. This allows for plugging in/using slog-compatible loggers other than the JSONFileLogger, if desired (ie, for downstream projects). - introduces new `scrape.FailureLogger` interface; just like `promql.QueryLogger`, it is a union of `io.Closer` and `slog.Handler` interfaces. Similar logic applies to reasoning. - updates tests where needed; have the `FakeQueryLogger` from promql's engine_test implement the `slog.Handler`, improve JSONFileLogger test suite, etc. Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	7 months ago
TJ Hoplock	3e24e84172	fix!: stop unbounded memory usage from query log Resolves: #15433 When I converted prometheus to use slog in #14906, I update both the `QueryLogger` interface, as well as how the log calls to the `QueryLogger` were built up in `promql.Engine.exec()`. The backing logger for the `QueryLogger` in the engine is a `util/logging.JSONFileLogger`, and it's implementation of the `With()` method updates the logger the logger in place with the new keyvals added onto the underlying slog.Logger, which means they get inherited onto everything after. All subsequent calls to `With()`, even in later queries, would continue to then append on more and more keyvals for the various params and fields built up in the logger. In turn, this causes unbounded growth of the logger, leading to increased memory usage, and in at least one report was the likely cause of an OOM kill. More information can be found in the issue and the linked slack thread. This commit does a few things: - It was referenced in feedback in #14906 that it would've been better to not change the `QueryLogger` interface if possible, this PR proposes changes that bring it closer to alignment with the pre-3.0 `QueryLogger` interface contract - reverts `promql.Engine.exec()`'s usage of the query logger to the pattern of building up an array of args to pass at once to the end log call. Avoiding the repetitious calls to `.With()` are what resolve the issue with the logger growth/memory usage. - updates the scrape failure logger to use the update `QueryLogger` methods in the contract. - updates tests accordingly - cleans up unused methods Builds and passes tests successfully. Tested locally and confirmed I could no longer reproduce the issue/it resolved the issue. Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	7 months ago
Bryan Boreham	6380229c71	[REFACTOR] Scraping: rename variables for clarity Instead of shadowing the outer variables, use different names. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	8 months ago
alexgreenbank	6b09f094e1	scrape: stop erroring on empty scrapes Signed-off-by: alexgreenbank <alex.greenbank@grafana.com>	8 months ago
Simon Pasquier	37f3f3f2db	Fix scrape failure logs Before this change, logs would show like: ``` {...,"target":"http://localhost:8080/metrics","!BADKEY":"Get ..."} ``` After this change ``` {...,"msg":"Get ...","job_name":...,"target":...} ``` Signed-off-by: Simon Pasquier <spasquie@redhat.com>	8 months ago
Vanshika	cccbe72514	TSDB: Fix some edge cases when OOO is enabled (#14710 ) Fix some edge cases when OOO is enabled Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> Signed-off-by: Vanshika <102902652+Vanshikav123@users.noreply.github.com> Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com> Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com>	8 months ago
George Krajcsovits	877fd2a60e	Update scrape/scrape.go Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>	8 months ago
György Krajcsovits	4283ae73dc	Rename convert_classic_histograms to convert_classic_histograms_to_nhcb On reviewer request. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	8 months ago
Alex Greenbank	421a3c22ea	scrape: provide a fallback format (#15136 ) scrape: Remove implicit fallback to the Prometheus text format Remove implicit fallback to the Prometheus text format in case of invalid/missing Content-Type and fail the scrape instead. Add ability to specify a `fallback_scrape_protocol` in the scrape config. --------- Signed-off-by: alexgreenbank <alex.greenbank@grafana.com> Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> Co-authored-by: Björn Rabenstein <beorn@grafana.com>	8 months ago
Bartlomiej Plotka	efc43d0714	s/scrape_classic_histograms/always_scrape_classic_histograms (3.0 breaking change) (#15178 ) This is for readability, especially when we can converting to nhcb option. See discussion https://cloud-native.slack.com/archives/C077Z4V13AM/p1729155873397889 Signed-off-by: bwplotka <bwplotka@gmail.com>	8 months ago
huochexizhan	ff16fa1827	fix: fix slice init length Signed-off-by: huochexizhan <huochexizhan@outlook.com>	9 months ago
György Krajcsovits	8dfa733596	Fix labels handling with dedupelabels tag Use scratch builder. Use hash compare instead of compare by label. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	9 months ago
TJ Hoplock	6ebfbd2d54	chore!: adopt log/slog, remove go-kit/log For: #14355 This commit updates Prometheus to adopt stdlib's log/slog package in favor of go-kit/log. As part of converting to use slog, several other related changes are required to get prometheus working, including: - removed unused logging util func `RateLimit()` - forward ported the util/logging/Deduper logging by implementing a small custom slog.Handler that does the deduping before chaining log calls to the underlying real slog.Logger - move some of the json file logging functionality to use prom/common package functionality - refactored some of the new json file logging for scraping - changes to promql.QueryLogger interface to swap out logging methods for relevant slog sugar wrappers - updated lots of tests that used/replicated custom logging functionality, attempting to keep the logical goal of the tests consistent after the transition - added a healthy amount of `if logger == nil { $makeLogger }` type conditional checks amongst various functions where none were provided -- old code that used the go-kit/log.Logger interface had several places where there were nil references when trying to use functions like `With()` to add keyvals on the new *slog.Logger type Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	9 months ago
Manik Rana	f1c57a95ed	change: No longer ingest OM _created as timeseries if feature-flag 'enable-ct-zero-ingestion' is enabled; fixed OM text CT conversion bug (#14738 ) * chore: revert TypeRequiresCT to private Signed-off-by: Manik Rana <manikrana54@gmail.com> * feat: init NewOpenMetricsParser with skipCT true Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: allow opt-in to OM CT ingestion Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: use textparse interface to set om options Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: set skipOMSeries in test Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: gofumpt Signed-off-by: Manik Rana <manikrana54@gmail.com> * wip: add tests for OM CR parse Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: merge ct tests Signed-off-by: Manik Rana <manikrana54@gmail.com> * tests: add cases for OM text Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: check correct test cases Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: use both scrape protocols in config Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: fix inputs and output tests for OM Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: cleanup Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: rename skipOMSeries to skipOMCTSeries Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> * fix: finish refac Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: move setup code outside test Signed-off-by: Manik Rana <manikrana54@gmail.com> * tests: verify _created lines create new metric in certain cases Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: post merge fixes Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * manager: Fixed CT OMText conversion bug; Refactored tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: gofumpt Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: imports Signed-off-by: Manik Rana <manikrana54@gmail.com> --------- Signed-off-by: Manik Rana <manikrana54@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> Signed-off-by: bwplotka <bwplotka@gmail.com> Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com> Co-authored-by: bwplotka <bwplotka@gmail.com>	9 months ago
Arthur Silva Sens	6bd9b1a7cc	Histogram CT Zero ingestion Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>	9 months ago
Alex Johnson	be0f10054e	Remove no-default-scrape-port featureFlag Signed-off-by: Alex Johnson <alex.kattathra.johnson@gmail.com>	9 months ago
Jeanette Tan	97ba2fc39d	use caps for NHCB Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	9 months ago
Jeanette Tan	172d4f2405	insert nhcb parser as intermediate layer Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	9 months ago
Jeanette Tan	f596f17024	allow option to convert classic histograms to nhcb entirely (don't append classic histogram series) Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	9 months ago
Jeanette Tan	02d5abf60e	don't use cache for nhcb maps Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	9 months ago
Jeanette Tan	4503145c8b	convert classic histograms to int nhcb where possible instead Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	9 months ago
Jeanette Tan	62e7f0438d	implement basic conversion of classic to nhcb in scrape Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	9 months ago
Nicolás Pazos	569b6abfa3	fix(utf8): propagate validationScheme config to scraping options Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com>	10 months ago
Owen Williams	88bb05c3e8	utf8: enable utf-8 support by default This change causes Prometheus to allow all UTF-8 characters in metric and label names. This means that names that were previously invalid and would have been previously rejected will be allowed through. Signed-off-by: Owen Williams <owen.williams@grafana.com>	10 months ago
Julien	ce0f09b125	Scrape: Add scrape_failure_log_file to log Scrape Failures Signed-off-by: Julien <roidelapluie@o11y.eu>	10 months ago
Bartlomiej Plotka	f1d3c3f464	fix: Properly put OM text p.CreatedTimestamp behind feature flag (#14809 ) (#14815 ) Mitigates https://github.com/prometheus/prometheus/issues/14808 Signed-off-by: Manik Rana <manikrana54@gmail.com> Co-authored-by: Manik Rana <Manikrana54@gmail.com>	10 months ago
Owen Williams	9da75328ea	fix(utf8): ensure correct validation when legacy mode turned on (#14736 ) fix(utf8): ensure correct validation when legacy mode turned on This depends on the included update of the prometheus/common dependency. --------- Signed-off-by: Owen Williams <owen.williams@grafana.com>	10 months ago
Bryan Boreham	d697719db6	[BUGFIX] Scraping: allow multiple samples on same series (#14685 ) So long as they specify timestamps. We don't check that the timestamps are different. Extend test, and use client_golang/prometheus/testutil to simplify metric check. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	10 months ago

1 2 3 4 5 ...

278 Commits (aa1d46a9da0f9710ca1528bef7623f1b4fb93f53)