prometheus

Commit Graph

Author	SHA1	Message	Date
chardch	2f59d38054	Add global config option for convert_classic_histograms_to_nhcb Addresses the global part of https://github.com/prometheus/prometheus/issues/13532 Signed-off-by: chardch <otwordsne@gmail.com>	9 months ago
Owen Williams	6566c5a2b3	scrape: Add config option for escaping scheme request. (#16066 ) The new metric_name_escaping_scheme config option works in parallel with metric_name_validation_scheme and controls which escaping scheme is requested when scraping. When not specified, the scheme will request underscores if the validation scheme is set to legacy, and will request allow-utf-8 when the validation scheme is set to utf8. This setting allows users to allow utf8 names if they like, but explicitly request an escaping scheme rather than UTF-8. Fixes https://github.com/prometheus/prometheus/issues/16034 Built on https://github.com/prometheus/prometheus/pull/16080 Signed-off-by: Owen Williams <owen.williams@grafana.com>	9 months ago
György Krajcsovits	f71ec8d5d5	fix(scrape): skip native histogram series if ingestion is disabled No need to validate, track, add sample, add exemplar if series isn't going to be ingested. Basically this is the same as if this series was dropped by relabelling. Fixes #16217 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	9 months ago
Bryan Boreham	e86b9812f4	[BUGFIX] Scraping: bump cache iteration after error Backport of #16174 to LTS release-2.53 branch. We use the cache iteration number to detect when the same metric has occurred twice in a scrape. We need to bump this number at the end of every scrape, not just successful scrapes. The `iter` number is also used: * After a successful scrape, to delete older metrics and metadata. * To detect when metadata changed in this scrape. None of those additional cases is broken by incrementing the number on error. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	9 months ago
György Krajcsovits	fec2f2603a	fix(scrape): skip native histogram series if ingestion is disabled No need to validate, track, add sample, add exemplar if series isn't going to be ingested. Basically this is the same as if this series was dropped by relabelling. Fixes #16217 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	9 months ago
Owen Williams	94b43c5d4c	utf8: Remove support for legacy global validation setting Global and Data Source configurations can specify legacy mode, but Prometheus now requires that the overall validation mode be set to UTF-8 Signed-off-by: Owen Williams <owen.williams@grafana.com>	9 months ago
Bryan Boreham	e32d89af7f	[BUGFIX] Scraping: bump cache iteration after error (#16174 ) We use the cache iteration number to detect when the same metric has occurred twice in a scrape. We need to bump this number at the end of every scrape, not just successful scrapes. The `iter` number is also used: * After a successful scrape, to delete older metrics and metadata. * To detect when metadata changed in this scrape. None of those additional cases is broken by incrementing the number on error. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	9 months ago
Matt Hughes	528b3fc60f	scraper: fix UTF-8 scraping header always sent with PrometheusText1.0.0 The `Accept` header should not include `escape=allow-utf-8` unless explicitly requested. Conveniently, there was already a test covering this header's value, it just required updating so it also asserts that this value in the header is not set in the cases we don't expect it to be set. I also converted those tests into table tests to help make failures clearer. Issue: https://github.com/prometheus/prometheus/issues/15857 Signed-off-by: Matt Hughes <mhughes@uw.co.uk>	10 months ago
Matt Hughes	5868e36d91	scraper: fix UTF-8 scraping header always sent with PrometheusText1.0.0 The `Accept` header should not include `escape=allow-utf-8` unless explicitly requested. Conveniently, there was already a test covering this header's value, it just required updating so it also asserts that this value in the header is not set in the cases we don't expect it to be set. I also converted those tests into table tests to help make failures clearer. Issue: https://github.com/prometheus/prometheus/issues/15857 Signed-off-by: Matt Hughes <mhughes@uw.co.uk>	10 months ago
Matthieu MOREL	c7d4b53ec1	chore: enable unused-parameter from revive Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	10 months ago
Bartlomiej Plotka	733a5e9eb4	textparse: Optimized protobuf parser with custom streaming unmarshal. (#15731 ) * textparse: Optimized protobuf parser with custom streaming decoder. Signed-off-by: bwplotka <bwplotka@gmail.com> Update model/textparse/protobufparse.go Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Addressing comments. Signed-off-by: bwplotka <bwplotka@gmail.com> decoder: reuse histograms and summaries. Signed-off-by: bwplotka <bwplotka@gmail.com> optimize help returning (5% of mem utilization). Signed-off-by: bwplotka <bwplotka@gmail.com> Apply suggestions from code review Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Update prompb/io/prometheus/client/decoder.go Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Fix build. Signed-off-by: bwplotka <bwplotka@gmail.com> * Update model/textparse/protobufparse.go Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	10 months ago
Bartlomiej Plotka	00b69efabb	model/textparse: Change parser interface Metric(...) string to Labels(...) (#16012 ) * model/textparse: Change parser interface Metric(...) string to Labels(...) Simplified the interface given no one is using the return argument. Renamed for clarity too. Found and discussed https://github.com/prometheus/prometheus/pull/15731#discussion_r1950916842 Signed-off-by: bwplotka <bwplotka@gmail.com> * Fixed comments; optimized not needed copy for om and text. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com>	10 months ago
Bartlomiej Plotka	8cd9069cf1	textparse: Refactor benchmark testdata for all types. (#15998 ) Also: * split benchmark functions to make sure no one compares across parsers. * testdata file have meaningful names reflecting the type representation * promtestdata.txt now has all types, taken directly from long running Prometheus (https://demo.do.prometheus.io/) Needed for https://github.com/prometheus/prometheus/pull/15731 Signed-off-by: bwplotka <bwplotka@gmail.com>	10 months ago
Bartlomiej Plotka	7427753922	scrape: Add realistic data case for scrape loop append bench. (#15966 ) * scrape: Add realistic data case for scrape loop append bench. Signed-off-by: bwplotka <bwplotka@gmail.com> * Update scrape/scrape_test.go Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>	11 months ago
bwplotka	3119567d5b	scrape: Add metadata for automatic metrics. Signed-off-by: bwplotka <bwplotka@gmail.com>	11 months ago
bwplotka	9385f31147	scrape: Fix metadata in WAL not working for histograms and summaries. The was a bug (due to confusion?) on the local metadata cache that is cached by metric family not the series metric name. The fix is to NOT use that local cache at all (it's still needed for current metadata API implementation, added TODO on how we can get rid of it). I went ahead and also rename Metric field in metadata structs to MetricFamily to make clear it's not always __name__. Signed-off-by: bwplotka <bwplotka@gmail.com>	11 months ago
bwplotka	281306765e	scrape: Unified scrape loop benchmark. Signed-off-by: bwplotka <bwplotka@gmail.com>	12 months ago
Bryan Boreham	b4ef38cfc8	Scraping: Add benchmark for protobuf format Extract helper function textToProto(). Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	12 months ago
Bryan Boreham	8f4557b0b1	Scraping benchmark: more realistic test Don't repeat type and help text. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	12 months ago
Bryan Boreham	7b03796d0f	Scraping: stop storing discovered labels (#15261 ) Instead of storing discovered labels on every target, recompute them if required. The `Target` struct now needs to hold some more data required to recompute them, such as ScrapeConfig. This moves the load from every Prometheus all of the time, to just when someone views Service Discovery in the UI. The way `PopulateLabels` is used changes; you are no longer expected to call it with a part-populated `labels.Builder`. The signature of `Target.Labels` changes to take a `labels.Builder` instead of a `ScratchBuilder`, for consistency with `DiscoveredLabels`. This will save a lot of work when many targets are filtered out in relabeling. Combine with `keep_dropped_targets` to avoid ever computing most labels for dropped targets. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Giedrius Statkevičius	5d76510bd6	scrape: update test name Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	1 year ago
Giedrius Statkevičius	5479fb3cfd	scrape: update test after review Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	1 year ago
Giedrius Statkevičius	90e832c861	scrape: fix nil panic after scrape loop reload Don't forget to set `metrics` field as otherwise scraping will lead to a nil panic in case the body size limit is reached. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	1 year ago
TJ Hoplock	e0104a6b7e	ref: JSONFileLogger slog handler, add scrape.FailureLogger interface Improves upon #15434, better resolves #15433. This commit introduces a few changes to ensure safer handling of the JSONFileLogger: - the JSONFileLogger struct now implements the slog.Handler interface, so it can directly be used to create slog Loggers. This pattern more closely aligns with upstream slog usage (which is generally based around handlers), as well as making it clear that devs are creating a whole new logger when interacting with it (vs silently modifying internal configs like it did previously). - updates the `promql.QueryLogger` interface to be a union of the methods of both the `io.Closer` interface and the `slog.Handler` interface. This allows for plugging in/using slog-compatible loggers other than the JSONFileLogger, if desired (ie, for downstream projects). - introduces new `scrape.FailureLogger` interface; just like `promql.QueryLogger`, it is a union of `io.Closer` and `slog.Handler` interfaces. Similar logic applies to reasoning. - updates tests where needed; have the `FakeQueryLogger` from promql's engine_test implement the `slog.Handler`, improve JSONFileLogger test suite, etc. Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	1 year ago
Arve Knudsen	89bbb885e5	Upgrade to golangci-lint v1.62.0 (#15424 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	1 year ago
Bryan Boreham	3f6a133f26	[Test] Scraping: check reload metrics Need to extend `newTestScrapeMetrics`` to get at the Registry. `gatherLabels` function could go upstream to `client_golang`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
alexgreenbank	6b09f094e1	scrape: stop erroring on empty scrapes Signed-off-by: alexgreenbank <alex.greenbank@grafana.com>	1 year ago
Matthieu MOREL	af1a19fc78	enable errorf rule from perfsprint linter Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	1 year ago
Vanshika	cccbe72514	TSDB: Fix some edge cases when OOO is enabled (#14710 ) Fix some edge cases when OOO is enabled Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> Signed-off-by: Vanshika <102902652+Vanshikav123@users.noreply.github.com> Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com> Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com>	1 year ago
George Krajcsovits	aa81210c8b	NHCB scrape: refactor state handling and speed up scrape test (#15193 ) * NHCB: scrape use state field and not booleans From comment https://github.com/prometheus/prometheus/pull/14978#discussion_r1800898724 Also make compareLabels read only and move storeLabels to the first processed classic histogram series. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * Speed up TestConvertClassicHistogramsToNHCB 3x Reduce the startup time and timeouts Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * lint fix Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
György Krajcsovits	4283ae73dc	Rename convert_classic_histograms to convert_classic_histograms_to_nhcb On reviewer request. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
György Krajcsovits	a23aed5634	More followup to #15164 Scrape test for NHCB modified. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
György Krajcsovits	70742a64aa	Follow up #15178 Renaming Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
Alex Greenbank	421a3c22ea	scrape: provide a fallback format (#15136 ) scrape: Remove implicit fallback to the Prometheus text format Remove implicit fallback to the Prometheus text format in case of invalid/missing Content-Type and fail the scrape instead. Add ability to specify a `fallback_scrape_protocol` in the scrape config. --------- Signed-off-by: alexgreenbank <alex.greenbank@grafana.com> Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> Co-authored-by: Björn Rabenstein <beorn@grafana.com>	1 year ago
Bartlomiej Plotka	efc43d0714	s/scrape_classic_histograms/always_scrape_classic_histograms (3.0 breaking change) (#15178 ) This is for readability, especially when we can converting to nhcb option. See discussion https://cloud-native.slack.com/archives/C077Z4V13AM/p1729155873397889 Signed-off-by: bwplotka <bwplotka@gmail.com>	1 year ago
Yi	2cabd1b707	config: remove expand-external-labels flag in release 3.0 (#14657 ) remove expand-external-labels feature flag and enabled env arg expansion for external labels by default. Signed-off-by: jyz0309 <45495947@qq.com>	1 year ago
TJ Hoplock	6ebfbd2d54	chore!: adopt log/slog, remove go-kit/log For: #14355 This commit updates Prometheus to adopt stdlib's log/slog package in favor of go-kit/log. As part of converting to use slog, several other related changes are required to get prometheus working, including: - removed unused logging util func `RateLimit()` - forward ported the util/logging/Deduper logging by implementing a small custom slog.Handler that does the deduping before chaining log calls to the underlying real slog.Logger - move some of the json file logging functionality to use prom/common package functionality - refactored some of the new json file logging for scraping - changes to promql.QueryLogger interface to swap out logging methods for relevant slog sugar wrappers - updated lots of tests that used/replicated custom logging functionality, attempting to keep the logical goal of the tests consistent after the transition - added a healthy amount of `if logger == nil { $makeLogger }` type conditional checks amongst various functions where none were provided -- old code that used the go-kit/log.Logger interface had several places where there were nil references when trying to use functions like `With()` to add keyvals on the new *slog.Logger type Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	1 year ago
Matthieu MOREL	ab64966e9d	fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()" (#15094 ) * fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()" --------- Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com> Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	1 year ago
Manik Rana	f1c57a95ed	change: No longer ingest OM _created as timeseries if feature-flag 'enable-ct-zero-ingestion' is enabled; fixed OM text CT conversion bug (#14738 ) * chore: revert TypeRequiresCT to private Signed-off-by: Manik Rana <manikrana54@gmail.com> * feat: init NewOpenMetricsParser with skipCT true Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: allow opt-in to OM CT ingestion Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: use textparse interface to set om options Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: set skipOMSeries in test Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: gofumpt Signed-off-by: Manik Rana <manikrana54@gmail.com> * wip: add tests for OM CR parse Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: merge ct tests Signed-off-by: Manik Rana <manikrana54@gmail.com> * tests: add cases for OM text Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: check correct test cases Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: use both scrape protocols in config Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: fix inputs and output tests for OM Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: cleanup Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: rename skipOMSeries to skipOMCTSeries Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> * fix: finish refac Signed-off-by: Manik Rana <manikrana54@gmail.com> * refac: move setup code outside test Signed-off-by: Manik Rana <manikrana54@gmail.com> * tests: verify _created lines create new metric in certain cases Signed-off-by: Manik Rana <manikrana54@gmail.com> * fix: post merge fixes Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * manager: Fixed CT OMText conversion bug; Refactored tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * chore: lint Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: gofumpt Signed-off-by: Manik Rana <manikrana54@gmail.com> * chore: imports Signed-off-by: Manik Rana <manikrana54@gmail.com> --------- Signed-off-by: Manik Rana <manikrana54@gmail.com> Signed-off-by: Manik Rana <Manikrana54@gmail.com> Signed-off-by: bwplotka <bwplotka@gmail.com> Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com> Co-authored-by: bwplotka <bwplotka@gmail.com>	1 year ago
George Krajcsovits	79a6238e19	scrape/scrape_test.go: reduce the time it takes to reload the manager (#14447 ) * scrape/scrape_test.go: reduce the time it takes to reload the manager TestNativeHistogramMaxSchemaSet took over 3x5s to complete because there's a minimum reload interval. I've made the testcases run in parallel and reduced the reload interval to 10ms. Now the test runs in around 0.1-0.2 seconds. Ran test 10000 times to check if it's flaky. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
Arthur Silva Sens	6bd9b1a7cc	Histogram CT Zero ingestion Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>	1 year ago
Jeanette Tan	90c266845b	fix lint Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	1 year ago
Jeanette Tan	050b5fc257	refine test cases according to spec Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	1 year ago
Jeanette Tan	de9de320a4	start to cover all test cases for scrape Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	1 year ago
Jeanette Tan	f35c6649e4	don't blindly convert series with the classic histogram name suffixes if they are not actually histograms based on metadata Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	1 year ago
Jeanette Tan	8b3ae15ad5	expand tests for classic and exponential native histograms Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	1 year ago
Jeanette Tan	e3899187da	expand tests for protobuf and fix problems Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	1 year ago
Jeanette Tan	cd498964e6	expand tests and support conversion to nhcb in the middle of scrape Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	1 year ago
Jeanette Tan	f596f17024	allow option to convert classic histograms to nhcb entirely (don't append classic histogram series) Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	1 year ago
Jeanette Tan	0a321fe4d8	improve new scrape test Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	1 year ago

1 2 3 4 5

243 Commits (d5731fd5f87004a92f8f6e3800f01a79ddc9f66d)