prometheus

Commit Graph

Author	SHA1	Message	Date
beorn7	bf0847073d	histogram: Modify getBound to deal properly with infinity The bucket receiving math.MaxFloat64 observations now has math.MaxFloat64 as upper bound, while the bucket after it (the last possible bucket) has +Inf. This also adds a test for getBound and moves the getBound code to generic.go (where it should have been in the first place). Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Björn Rabenstein	dccfb9db4e	histogram: Remove code replication via generics (#11361 ) * histogram: Simplify iterators We don't really need currLower and currUpper and can calculate it when needed (as already done for the floatBucketIterator). The calculation is cheap, while keeping those extra variables around costs RAM (potentially a lot with many iterators). * histogram: Convert Bucket/FloatBucket to one generic type * histogram: Move some bucket iterator code into generic base iterator * histogram: Remove cumulative iterator for FloatHistogram We added it in the past for completeness (Histogram has one), but it has never been used. Plus, even the cumulative iterator for Histogram is only there for test reasons. We can always add it back, and then maybe even using generics. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Bryan Boreham	8120af22e2	benchmarks: SetBytes takes bytes per operation Where the code was multiplying bytes by number of operations, this resulted in absurdly high throughput numbers. Also, in `BenchmarkParse()`, don't run the `expfmt` case twice. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
beorn7	a7c519930e	histograms: Add Compact method to the normal integer Histogram And use the new method to call to compact Histograms during parsing. This happens for both `Histogram` and `FloatHistogram`. In this way, if targets decide to optimize the exposition size by merging spans with empty buckets in between, we still get a normalized results. It will also normalize away any valid but weird representations like empty spans, spans with offset zero, and empty buckets at the start or end of a span. The implementation seemed easy at first as it just turns the `compactBuckets` helper into a generic function (which now got its own file). However, the integer Histograms have delta buckets instead of absolute buckets, which had to be treated specially in the generic `compactBuckets` function. To make sure it works, I have added plenty of explicit tests for `Histogram` in addition to the `FloatHistogram` tests. I have also updated the doc comment for the `Compact` method. Based on the insights now expressed in the doc comment, compacting with a maxEmptyBuckets > 0 is rarely useful. Therefore, this commit also sets the value to 0 in the two cases we were using 3 so far. We might still want to reconsider, so I don't want to remove the maxEmptyBuckets parameter right now. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Ganesh Vernekar	2474c6fb2c	Error on amending histograms on append (#11308 ) * Error on amending histograms on append Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Rename Matches to Equals Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Bryan Boreham	5421c778ba	labels: in tests use labels.FromStrings Replacing code which assumes the internal structure of `Labels`. Add a convenience function `EmptyLabels()` which is more efficient than calling `New()`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Julien Pivotto	96d5a32659	Update go to 1.19, set min version to 1.18 (#11279 ) * Update go to 1.19, set min version to 1.18 Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu> * Update golangci-lint Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu> Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	3 years ago
Cosrider	bef6556ca5	delete redundant alias (#11180 ) Signed-off-by: Cosrider <cosrider7@gmail.com> Signed-off-by: Cosrider <cosrider7@gmail.com>	3 years ago
Marc Tudurí	f7df3b86ba	histograms: parse float histograms from proto definition (#11149 ) * histograms: parse float histograms from proto definition Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Improve comment Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Ignore float buckets Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Refactor Histogram() function Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Fix test_float_histogram Signed-off-by: Marc Tuduri <marctc@protonmail.com> * Update model/textparse/protobufparse.go Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com> Signed-off-by: Marc Tudurí <marctc@protonmail.com> * Update protobufparse.go Signed-off-by: Marc Tudurí <marctc@protonmail.com> * Update scrape.go Signed-off-by: Marc Tudurí <marctc@protonmail.com> * Update scrape/scrape.go Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com> Signed-off-by: Marc Tudurí <marctc@protonmail.com> Signed-off-by: Marc Tuduri <marctc@protonmail.com> Signed-off-by: Marc Tudurí <marctc@protonmail.com> Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>	3 years ago
Bryan Boreham	8b863c42dd	Optimise relabeling by re-using memory (#11147 ) * model/relabel: Add benchmark Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * model/relabel: re-use Builder across relabels Saves memory allocations. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * labels.Builder: allow re-use of result slice This reduces memory allocations where the caller has a suitable slice available. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * model/relabel: re-use source values slice To reduce memory allocations. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Unwind one change causing test failures Restore original behaviour in PopulateLabels, where we must not overwrite the input set. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * relabel: simplify values optimisation Use a stack-based array for up to 16 source labels, which will be the vast majority of cases. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * lint Signed-off-by: Bryan Boreham <bjboreham@gmail.com> Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	a7f19b5775	labels: add a test for JSON and YAML marshalling Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	10699c37a3	labels: test BytesWithoutLabels does not remove __name__ by default Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	d46ef0aa8e	labels: tweak BenchmarkLabels_Get() So the benchmark works without requiring `Labels` internals to be a slice. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	24ebff9c4a	labels: don't test that Hash() works on unordered labels Labels are required to be ordered. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Bryan Boreham	fb2d883f1e	labels: test Compare without knowing the exact result Only the sign of the result is important Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
beorn7	89fb7117c3	textparse: Remove TODO that is actually done Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	c40b105efd	histograms: Move to new exposition protobuf format This is an incompatible protobuf change. Instrumented targets must include https://github.com/prometheus/client_golang/pull/1092 to make this work. Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Paschalis Tsilias	d1122e0743	Introduce TSDB changes for appending metadata to the WAL (#10972 ) * Append metadata to the WAL Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Remove extra whitespace; Reword some docstrings and comments Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Use RLock() for hasNewMetadata check Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Use single byte for metric type in RefMetadata Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Update proposed WAL format for single-byte type metadata Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Implementa MetadataAppender interface for the Agent Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Address first round of review comments Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Amend description of metadata in wal.md Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Correct key used to retrieve metadata from cache When we're setting metadata entries in the scrapeCace, we're using the p.Help(), p.Unit(), p.Type() helpers, which retrieve the series name and use it as the cache key. When checking for cache entries though, we used p.Series() as the key, which included the metric name _with_ its labels. That meant that we were never actually hitting the cache. We're fixing this by utiling the __name__ internal label for correctly getting the cache entries after they've been set by setHelp(), setType() or setUnit(). Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Put feature behind a feature flag Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Fix AppendMetadata docstring Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Reorder WAL format document Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Change error message of AppendMetadata; Fix access of s.meta in AppendMetadata Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Reuse temporary buffer in Metadata encoder Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Only keep latest metadata for each refID during checkpointing Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Fix test that's referencing decoding metadata Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Avoid creating metadata block if no new metadata are present Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Add tests for corrupt metadata block and relevant record type Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Fix CR comments Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Extract logic about changing metadata in an anonymous function Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Implement new proposed WAL format and amend relevant tests Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Use 'const' for metadata field names Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Apply metadata to head memSeries in Commit, not in AppendMetadata Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Add docstring and rename extracted helper in scrape.go Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Add tests for tsdb-related cases Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Fix linter issues vol1 Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Fix linter issues vol2 Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Fix Windows test by closing WAL reader files Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Use switch instead of two if statements in metadata decoding Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Fix review comments around TestMetadata* tests Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Add code for replaying WAL; test correctness of in-memory data after a replay Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Remove scrape-loop related code from PR Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Address first round of comments Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Simplify tests by sorting slices before comparison Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Fix test to use separate transactions Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Empty out buffer and record slices after encoding latest metadata Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Fix linting issue Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Update calculation for DroppedMetadata metric Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Rename MetadataAppender interface and AppendMetadata method to MetadataUpdater/UpdateMetadata Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Reuse buffer when encoding latest metadata for each series Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Fix review comments; Check all returned error values using two helpers Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Simplify use of helpers Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Satisfy linter Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>	4 years ago
Matthieu MOREL	c2b4de3611	refactor (package model): move from github.com/pkg/errors to 'errors' and 'fmt' packages (#10747 ) Signed-off-by: Matthieu MOREL <mmorel-35@users.noreply.github.com> Co-Authored-By: Julien Pivotto <roidelapluie@gmail.com> Co-authored-by: Julien Pivotto <roidelapluie@gmail.com>	4 years ago
Łukasz Mierzwa	648b12d8c5	Implement Unwrap() on errors returned from rulefmt I'd like to unwrap errors returned from rulefmt but both Error and WrappedError types are missing Unwrap() method. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	4 years ago
Paschalis Tsilias	4f3791024a	Drop extra string held in relabel.Regexp struct (#10846 ) * Drop extra string held in relabel.Regexp struct Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Use slice operations instead of TrimPrefix/TrimSuffix; Override String() method Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>	4 years ago
Bryan Boreham	2e2c014d52	Labels: optimise creation of signature with/without labels (#10667 ) * Labels: create signature with/without labels Instead of creating a new Labels slice then converting to signature, go directly to the signature and save time. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Labels: refactor Builder tests Have one test with a range of cases, and have them check the final output rather than checking the internal structure of the Builder. Also add a couple of cases where the value is "", which should be interpreted as 'delete'. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Labels: add 'Keep' function to Builder This lets us replace `Labels.WithLabels` with the more general `Builder`. In `engine.resultMetric()` we can call `Keep()` instead of checking and calling `Del()`. Avoid calling `Sort()` in `Builder.Labels()` if we didn't add anything, so that `Keep()` has the same performance as `WithLabels()`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	4 years ago
Łukasz Mierzwa	b11062bfcc	Don't count added labels when initializing slice (#10756 ) This was added in #10749 but based on feedback it might be over-allocating so worth removing. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	4 years ago
Łukasz Mierzwa	08262454a3	Preallocate Labels in labels.Builder (#10749 ) This tries to avoid re-allocations of labels slice since we know possible max size Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	4 years ago
Łukasz Mierzwa	89de30a0b7	Avoid reallocating map in MatchLabels (#10715 ) We know the max size of our map so we can create it with that information and avoid extra allocations Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	4 years ago
beorn7	2d233cf95e	Histogram: Fix allFloatBucketIterator If the zero threshold overlaps with the highest negative bucket and/or the lowest positive bucket, its upper or lower boundary, respectively, has to be adjusted. In valid histograms, only ever the highest negative bucket and/or the lowest positive bucket may overlap with the zero bucket. This is assumed in this code to simplify the checks. Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Julien Pivotto	71dbb4d091	Add lowercase and uppercase relabel action (#10641 ) Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	4 years ago
Matthieu MOREL	e2ede285a2	refactor: move from io/ioutil to io and os packages (#10528 ) * refactor: move from io/ioutil to io and os packages * use fs.DirEntry instead of os.FileInfo after os.ReadDir Signed-off-by: MOREL Matthieu <matthieu.morel@cnp.fr>	4 years ago
beorn7	56db51c826	Histgram: Fix Compact for spans of only empty buckets Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	15583af9bb	Histogram: Fix crash when compacting only empty buckets Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	08efde4de5	Adjust to newest gofumpt Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Peter Štibraný	cd18da3605	Fix OpenMetrics parser to sort uppercase labels correctly. (#10510 ) Signed-off-by: Peter Štibraný <pstibrany@gmail.com>	4 years ago
beorn7	f9c411604d	Fix spelling errors Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	9fbcf14e5c	histogram: Handle changes of the ZeroThreshold and the Schema Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Nick Pillitteri	53ac9d6d66	Compare equality of label.Label structs directly (#10427 ) * labels.Equal benchmark for equal, not equal, and differing lengths Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Compare equality of label.Label structs directly Compare the structs using `==` instead of the name and value of each label. This is functionally equivalent and about ~10% faster in my testing. Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Use longer more realistic names and values in benchmark Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>	4 years ago
Levi Harrison	a9e2ffb717	Specify type in metadata parser errors (#10269 )	4 years ago
Łukasz Mierzwa	da23c4649a	Enable misspell check in golangci-lint (#10393 ) Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	4 years ago
Julien Pivotto	9a2e93228e	Switch to grafana/regexp everywhere (#10268 ) Let's have a consistent library for regexp. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Bryan Boreham	579331446a	Allow downstream projects to use faster regexp (#10251 ) * Add benchmark for FastRegexMatcher Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Use modified regexp package with optimisations See https://github.com/grafana/regexp/tree/speedup#readme Includes the following changes proposed upstream: * [regexp: allow patterns with no alternates to be one-pass](https://go-review.googlesource.com/c/go/+/353711) * [regexp: speed up onepass prefix check](https://go-review.googlesource.com/c/go/+/354909) * [regexp: handle prefix string with fold-case](https://go-review.googlesource.com/c/go/+/358756) * [regexp: avoid copying each instruction executed](https://go-review.googlesource.com/c/go/+/355789) * [regexp: allow prefix string anchored at beginning](https://go-review.googlesource.com/c/go/+/377294) Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Use regexp code identical to upstream Go Change `grafana/regexp` import to use `main` branch. This means Prometheus is not using the proposed optimisations, but downstream users of Prometheus code are able to `replace` the library with the `speedup` branch which does. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	4 years ago
Julien Pivotto	f695df843f	Improve content-type error handling - Call err everywhere - Change log message to underscore-separated field Followup on #10186 Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Matheus Pimenta	8d8ce641a4	error for invalid media type should not be completely swallowed (#10186 ) * error for invalid media type should not be completely swallowed Signed-off-by: Matheus Pimenta <matheuscscp@gmail.com>	4 years ago
beorn7	3b4d6c3fdb	model: Implement FloatHistogram.Compact Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	e7592fe353	sparsehistogram: Address two TODOs Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	27f865ec49	histograms: avoid initialization tracking in reverseFloatBucketIterator Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	a6acdfe346	histograms: Doc comment and naming improvements Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Ganesh Vernekar	4a43349aca	`histogram_quantile` for sparse histograms (#9935 ) * MergeFloatBucketIterator for []FloatBucketIterator Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * histogram_quantile for histograms Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix histogram_quantile Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Unit test and enhancements Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Iterators to iterate buckets in reverse and all buckets together including zero bucket Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Consider all buckets for histogram_quantile and fix the implementation Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Remove unneeded code Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix lint Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	4 years ago
Björn Rabenstein	0e1b9dd308	Promql: Initial rate implementation for sparse histograms (#9926 ) Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Björn Rabenstein	7e42acd3b1	tsdb: Rework iterators (#9877 ) - Pick At... method via return value of Next/Seek. - Do not clobber returned buckets. - Add partial FloatHistogram suppert. Note that the promql package is now _only_ dealing with FloatHistograms, following the idea that PromQL only knows float values. As a byproduct, I have removed the histogramSeries metric. In my understanding, series can have both float and histogram samples, so that metric doesn't make sense anymore. As another byproduct, I have converged the sampleBuf and the histogramSampleBuf in memSeries into one. The sample type stored in the sampleBuf has been extended to also contain histograms even before this commit. Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	6a820a646c	histogram: Add FloatHistogram Including a few adjustments for normal Histogram, too, e.g. use pointer receiver to avoid the large copy on method calls. Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	73858d7f82	storage: histogram support in memoized_iterator Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago

1 2 3

136 Commits (8ee4dfd40cf4cc65733a27129db235e347bd9eaf)