prometheus

Commit Graph

Author	SHA1	Message	Date
Björn Rabenstein	b8d19543b8	Add histogram validation in remote-read and during reducing resolution (#17561 ) ReduceResolution is currently called before validation during ingestion. This will cause a panic if there are not enough buckets in the histogram. If there are too many buckets, the spurious buckets are ignored, and therefore the error in the input histogram is masked. Furthermore, invalid negative offsets might cause problems, too. Therefore, we need to do some minimal validation in reduceResolution. Fortunately, it is easy and shouldn't slow things down. Sadly, it requires to return errors, which triggers a bunch of code changes. Even here is a bright side, we can get rud of a few panics. (Remember: Don't panic!) In different news, we haven't done a full validation of histograms read via remote-read. This is not so much a security concern (as you can throw off Prometheus easily by feeding it bogus data via remote-read) but more that remote-read sources might be makeshift and could accidentally create invalid histograms. We really don't want to panic in that case. So this commit does not only add a check of the spans and buckets as needed for resolution reduction but also a full validation during remote-read. Signed-off-by: beorn7 <beorn@grafana.com>	1 month ago
Grégoire	1174b0ce4f	model/textparse: Remove unit validation in protobuf parsing (#16834 ) Signed-off-by: Gregoire Verdier <gregoire.verdier@gmail.com>	1 month ago
Bartlomiej Plotka	f50ff0a40a	feat: rename CreatedTimestamp to StartTimestamp (#17523 ) Partially fixes https://github.com/prometheus/prometheus/issues/17416 by renaming all CT* names to ST* in the whole codebase except RW2 (this is done in separate [PR](https://github.com/prometheus/prometheus/pull/17411)) and PrometheusProto exposition proto. ``` CreatedTimestamp -> StartTimestamp CreatedTimeStamp -> StartTimestamp created_timestamp -> start_timestamp CT -> ST ct -> st ``` Signed-off-by: bwplotka <bwplotka@gmail.com>	1 month ago
Ben Kochie	204249fcb5	Update golangci-lint (#17478 ) * Update golangci-lint to v2.6.0 * Fixup various linting issues. * Fixup deprecations. * Add exception for `labels.MetricName` deprecation. Signed-off-by: SuperQ <superq@gmail.com>	2 months ago
Ben Kochie	48956f60d7	Update modernize (#17471 ) Apply additional Go modernize tool improvements. Signed-off-by: SuperQ <superq@gmail.com>	2 months ago
beorn7	ad7d1aed99	Phase out native histogram feature flag The detailed plan for this is laid out in https://github.com/prometheus/prometheus/issues/16572 . This commit adds a global and local scrape config option `scrape_native_histograms`, which has to be set to true to ingest native histograms. To ease the transition, the feature flag is changed to simply set the default of `scrape_native_histograms` to true. Further implications: - The default scrape protocols now depend on the `scrape_native_histograms` setting. - Everywhere else, histograms are now "on by default". Documentation beyond the one for the feature flag and the scrape config are deliberately left out. See https://github.com/prometheus/prometheus/pull/17232 for that. Signed-off-by: beorn7 <beorn@grafana.com>	2 months ago
Bartlomiej Plotka	a4da440dad	fix: Fix slicelabels corruption when used with proto decoding (#17150 ) * fix: Fix slicelabels corruption when used with proto decoding Alternative to https://github.com/prometheus/prometheus/pull/16957/ Signed-off-by: bwplotka <bwplotka@gmail.com> * addressed comments Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com>	3 months ago
Naman-B-Parlecha	083d0fa835	refactor(convert): updated tests and moved formatOpenMetricsFloat Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com>	3 months ago
NamanParlecha	594f9d63a5	refactor(textparse): Introduce Variadic options in textParse.New (#17155 ) * refactor(textparse): introduce ParserOptions struct for cleaner parser initialization Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(fuzz): update fuzzParseMetricWithContentType to use ParserOptions Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(parser): simplify ParserOptions usage in tests and implementations Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(parse): using variadic options Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(parser): add fallbackType & SymbolTable to variadic options Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(parser): private fields Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(scrape): compose parser options Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(parser): add comments Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(parser): update to use ParserOptions struct for configuration Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(scrape): remove unused parserOptions field from scrapeLoop Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(parser): update ParserOptions field names and add comments for clarity Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> --------- Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com>	4 months ago
George Krajcsovits	acd9aa0afb	fix(textparse/protobuf): metric family name corrupted by NHCB parser (#17156 ) * fix(textparse): implement NHCB parsing in ProtoBuf parser directly The NHCB conversion does some validation, but we can only return error from Parser.Next() not Parser.Histogram(). So the conversion needs to happen in Next(). There are 2 cases: 1. "always_scrape_classic_histograms" is enabled, in which case we convert after returning the classic series. This is to be consistent with the PromParser text parser, which collects NHCB while spitting out classic series; then returns the NHCB. 2. "always_scrape_classic_histograms" is disabled. In which case we never return the classic series. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * refactor(textparse): skip classic series instead of adding NHCB around Do not return the first classic series from the EntryType state, switch to EntrySeries. This means we need to start the histogram field state from -3 , not -2. In EntrySeries, skip classic series if needed. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * reuse nhcb converter Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * test(textparse/nhcb): test corrupting metric family name NHCB parse doesn't always copy the metric name from the underlying parser. When called via HELP, UNIT, the string is directly referenced which means that the read-ahead of NHCB can corrupt it. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	4 months ago
George Krajcsovits	31e4d84edd	refactor(textparse): allow for parsers with direct NHCB support (#17153 ) Hide adding NHCB parser on top another parser in New() function so we can easily add direct NHCB capable parsers. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	4 months ago
beorn7	747c5ee2b1	Apply analyzer "modernize" to the whole codebase See https://pkg.go.dev/golang.org/x/tools/gopls/internal/analysis/modernize for details. This ran into a few issues (arguably bugs in the modernize tool), which I will fix in the next commit, so that we have transparency what was done automatically. Beyond those hiccups, I believe all the changes applied are legitimate. Even where there might be no tangible direct gain, I would argue it's still better to use the "modern" way to avoid micro discussions in tiny style PRs later. Signed-off-by: beorn7 <beorn@grafana.com>	4 months ago
Arve Knudsen	0a40df33fb	Make metric/label name validation scheme explicit (#16928 ) * Parameterize metric/label name validation scheme Parameterized metric/label name validation scheme --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Julius Hinze <julius.hinze@grafana.com>	4 months ago
Matthieu MOREL	cef219c31c	chore: enable unused-receiver rule from revive Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	5 months ago
Bryan Boreham	d9e5748a27	[REFACTOR] Scraping: Remove unnecessary yolostring calls (#16927 ) Go will not allocate when reading from a map with a key cast from []byte to string. Also remove some yoloString calls in package `textparse` - call a more suitable library function. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	5 months ago
György Krajcsovits	85d63ba861	refactor(nhcb): get rid of extra lastHistogramExponential state and make it a regular state of inhibiting NHCB conversion. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	7 months ago
György Krajcsovits	3844a21738	fix(nhcb): do not loose NHCB after exponential with different labels The problem happens when we parse a standalone native histogram, which sets the p.lastHistogramExponential state flag. We never unset it. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	7 months ago
Ayoub Mrini	89a72bf1b5	[BACKPORT-3.4] fix: add reproducer for a dangling-reference issue in parsers and fix (#16633 )	7 months ago
machine424	50a6efd5ec	fix(model/textparse): Labels(): copy the input to avoid dangling references Signed-off-by: machine424 <ayoubmrini424@gmail.com>	7 months ago
Bartlomiej Plotka	8e6b008608	feature: type-and-unit-labels (PROM-39 implementation) (#16228 ) * feature: type-and-unit-labels (extended MetricIdentity) Experimental implementation of https://github.com/prometheus/proposals/pull/39 Previous (unmerged) experiments: * https://github.com/prometheus/prometheus/compare/main...dashpole:prometheus:type_and_unit_labels * https://github.com/prometheus/prometheus/pull/16025 Signed-off-by: bwplotka <bwplotka@gmail.com> feature: type-and-unit-labels (extended MetricIdentity) Experimental implementation of https://github.com/prometheus/proposals/pull/39 Previous (unmerged) experiments: * https://github.com/prometheus/prometheus/compare/main...dashpole:prometheus:type_and_unit_labels * https://github.com/prometheus/prometheus/pull/16025 Signed-off-by: bwplotka <bwplotka@gmail.com> * Fix compilation errors Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Lint Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Revert change made to protobuf 'Accept' header Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Fix compilation errors for 'dedupelabels' tag Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> * Rectored into schema.Metadata Signed-off-by: bwplotka <bwplotka@gmail.com> * texparse: Added tests for PromParse Signed-off-by: bwplotka <bwplotka@gmail.com> * add OM tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * add proto tests Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> * add schema label tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> * fix tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * add promql tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * lint Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>	7 months ago
Rafal Rosa	c7ea28c36b	parser: remove expired link from comment (#16189 ) * Replace OpenMetrics link with the current specification --------- Signed-off-by: pomyslowynick <dark0mlet@gmail.com> Signed-off-by: Rafal Rosa <dark0mlet@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	9 months ago
Matthieu MOREL	5fa1146e21	chore: enable gci linter (#16245 ) Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	9 months ago
Owen Williams	94b43c5d4c	utf8: Remove support for legacy global validation setting Global and Data Source configurations can specify legacy mode, but Prometheus now requires that the overall validation mode be set to UTF-8 Signed-off-by: Owen Williams <owen.williams@grafana.com>	10 months ago
Bartlomiej Plotka	dc85d677d8	textparse: Optimize CreatedTimestamp; It returns int64 value now. (#16072 ) ..instead of *int64. This is as an optimization and ease of use. We already accepted in many places (proto histograms, PRW) that CT (or any timestamp really) 0 means not set. Signed-off-by: bwplotka <bwplotka@gmail.com>	10 months ago
Matthieu MOREL	c7d4b53ec1	chore: enable unused-parameter from revive Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	10 months ago
Bartlomiej Plotka	733a5e9eb4	textparse: Optimized protobuf parser with custom streaming unmarshal. (#15731 ) * textparse: Optimized protobuf parser with custom streaming decoder. Signed-off-by: bwplotka <bwplotka@gmail.com> Update model/textparse/protobufparse.go Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Addressing comments. Signed-off-by: bwplotka <bwplotka@gmail.com> decoder: reuse histograms and summaries. Signed-off-by: bwplotka <bwplotka@gmail.com> optimize help returning (5% of mem utilization). Signed-off-by: bwplotka <bwplotka@gmail.com> Apply suggestions from code review Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Update prompb/io/prometheus/client/decoder.go Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Fix build. Signed-off-by: bwplotka <bwplotka@gmail.com> * Update model/textparse/protobufparse.go Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	11 months ago
Bartlomiej Plotka	00b69efabb	model/textparse: Change parser interface Metric(...) string to Labels(...) (#16012 ) * model/textparse: Change parser interface Metric(...) string to Labels(...) Simplified the interface given no one is using the return argument. Renamed for clarity too. Found and discussed https://github.com/prometheus/prometheus/pull/15731#discussion_r1950916842 Signed-off-by: bwplotka <bwplotka@gmail.com> * Fixed comments; optimized not needed copy for om and text. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com>	11 months ago
Bartlomiej Plotka	8cd9069cf1	textparse: Refactor benchmark testdata for all types. (#15998 ) Also: * split benchmark functions to make sure no one compares across parsers. * testdata file have meaningful names reflecting the type representation * promtestdata.txt now has all types, taken directly from long running Prometheus (https://demo.do.prometheus.io/) Needed for https://github.com/prometheus/prometheus/pull/15731 Signed-off-by: bwplotka <bwplotka@gmail.com>	11 months ago
Bartlomiej Plotka	7427753922	scrape: Add realistic data case for scrape loop append bench. (#15966 ) * scrape: Add realistic data case for scrape loop append bench. Signed-off-by: bwplotka <bwplotka@gmail.com> * Update scrape/scrape_test.go Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>	11 months ago
George Krajcsovits	cfcb00a716	perf(nhcbparse): unroll recursion (#15776 ) https://github.com/prometheus/prometheus/pull/15467#issuecomment-2563585979 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	12 months ago
David Ashpole	953a873342	update links to openmetrics to reference the v1.0.0 release Signed-off-by: David Ashpole <dashpole@google.com>	1 year ago
György Krajcsovits	a48d05912d	nhcb: optimize, do not recalculate suffixes multiple times Reduce string manipulation by just cutting off the histogram suffixes from the series name label once. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
huochexizhan	4f48e76086	chore: fix some function names in comment Signed-off-by: huochexizhan <huochexizhan@outlook.com>	1 year ago
Tristan Colgate-McFarlane	48abdcd83f	bugfix: allow quoted exemplar keys in openmetrics text format Signed-off-by: Tristan Colgate-McFarlane <tristan@cerbos.dev>	1 year ago
Matthieu MOREL	af1a19fc78	enable errorf rule from perfsprint linter Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	1 year ago
György Krajcsovits	eafe72a0d0	perf(nhcb): optimize away most allocations in convertnhcb In general aim for the happy case when the exposer lists the buckets in ascending order. Use Compact(2) to compact the result of nhcb convert. This is more in line with how client_golang optimizes spans vs buckets. `aef8aedb4b/prometheus/histogram.go (L1485)` Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
George Krajcsovits	eb3b349024	fix(nhcb): created timestamp fails when keeping classic histograms (#15218 ) The wrong source was used to return the created timestamp, leading to index out of bound panic. One line fix. Refactor the requirement test to be generic and be able to test OpenMetrics and Prom parsers as well. There are some differencies in what the parsers support, the Prom parser doesn't have created timestamp. The protobuf parser uses different formatting to identify the metric for the scrape loop. Each parser represents the sample timestamp differently. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
George Krajcsovits	469573b13b	fix(nhcb): do not return nhcb from parse if exponential is present (#15209 ) From: https://github.com/prometheus/prometheus/pull/14978#discussion_r1800755481 Also encode the requirement table set in #13532 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
George Krajcsovits	2182b83271	feat(nhcb): implement created timestamp handling (#15198 ) Call through to the underlaying parser if we are not in a histogram and the entry is a series or exponential native histogram. Otherwise store and retrieve CT for NHCB. * fix(omparser): losing exemplars when CT is parsed Fixes: #15137 Ignore exemplars while peeking ahead during CT parsing. Simplify state reset with defer(). Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
George Krajcsovits	aa81210c8b	NHCB scrape: refactor state handling and speed up scrape test (#15193 ) * NHCB: scrape use state field and not booleans From comment https://github.com/prometheus/prometheus/pull/14978#discussion_r1800898724 Also make compareLabels read only and move storeLabels to the first processed classic histogram series. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * Speed up TestConvertClassicHistogramsToNHCB 3x Reduce the startup time and timeouts Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * lint fix Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
György Krajcsovits	25ef4d3483	benchmark, rename parser omtext_with_nhcb Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
György Krajcsovits	bee1eb7720	goimports run Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
György Krajcsovits	555bd6292a	Better docstring on test Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
György Krajcsovits	a6947e1e6d	Remove omcounterdata.txt as redundant Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
György Krajcsovits	eaee6bacc7	Fix failing benchmarks Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
György Krajcsovits	5ee0980cd1	Add unit test to show that current wrapper is sub-optimal https://github.com/prometheus/prometheus/pull/14978#discussion_r1800755481 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
György Krajcsovits	482bb453c6	Followup to #15164 Update test cases Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
machine424	8bcb4d865d	feat: normalize "le" and "quantile" labels values upon ingestion Signed-off-by: machine424 <ayoubmrini424@gmail.com> Co-authored-by: beorn7 <beorn@grafana.com>	1 year ago
Alex Greenbank	421a3c22ea	scrape: provide a fallback format (#15136 ) scrape: Remove implicit fallback to the Prometheus text format Remove implicit fallback to the Prometheus text format in case of invalid/missing Content-Type and fail the scrape instead. Add ability to specify a `fallback_scrape_protocol` in the scrape config. --------- Signed-off-by: alexgreenbank <alex.greenbank@grafana.com> Signed-off-by: Alex Greenbank <alex.greenbank@grafana.com> Co-authored-by: Björn Rabenstein <beorn@grafana.com>	1 year ago
machine424	18b81ad79d	feat: ProtobufParse.formatOpenMetricsFloat: improve float formatting by using strconv.AppendFloat instead of fmt.Sprint Signed-off-by: machine424 <ayoubmrini424@gmail.com>	1 year ago

1 2 3 4

178 Commits (main)