prometheus

Commit Graph

Author	SHA1	Message	Date
Matthieu MOREL	af1a19fc78	enable errorf rule from perfsprint linter Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	1 year ago
Jeanette Tan	9adc1699c3	fix according to code review Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2 years ago
zenador	4acbb7dea6	Add custom buckets to native histogram chunks encoding (#13706 ) * add custom bounds to chunks encoding * change custom buckets schema number * rename custom bounds to custom values Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2 years ago
machine424	f477e0539a	Move from golang.org/x/exp/slices into slices now that we only support Go >= 1.21 Prevent adding back golang.org/x/exp/slices. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2 years ago
György Krajcsovits	5d0a0a7542	Add custom buckets to native histogram model (#13592 ) * add custom buckets to native histogram model * simple copy for custom bounds * return errors for unsupported add/sub operations * add test cases for string and update appendhistogram in scrape to account for new schema * check fields which are supposed to be unused but may affect results in equals * allow appending custom buckets histograms regardless of max schema Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2 years ago
Filip Petkovski	583f3e587c	Optimize histogram iterators (#13340 ) Optimize histogram iterators Histogram iterators allocate new objects in the AtHistogram and AtFloatHistogram methods, which makes calculating rates over long ranges expensive. In #13215 we allowed an existing object to be reused when converting an integer histogram to a float histogram. This commit follows the same idea and allows injecting an existing object in the AtHistogram and AtFloatHistogram methods. When the injected value is nil, iterators allocate new histograms, otherwise they populate and return the injected object. The commit also adds a CopyTo method to Histogram and FloatHistogram which is used in the BufferedIterator to overwrite items in the ring instead of making new copies. Note that a specialized HPoint pool is needed for all of this to work (`matrixSelectorHPool`). --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>	2 years ago
Filip Petkovski	9008271df5	Use cap to determine slice capacity Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	2 years ago
Filip Petkovski	10a82f87fd	Enable reusing memory when converting between histogram types The 'ToFloat' method on integer histograms currently allocates new memory each time it is called. This commit adds an optional *FloatHistogram parameter that can be used to reuse span and bucket slices. It is up to the caller to make sure the input float histogram is not used anymore after the call. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	2 years ago
Linas Medziunas	49613823f8	Reuse slices in [Float]Histogram.ReduceResolution Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>	2 years ago
Linas Medziunas	ea1862aab4	Explicit schema check in [Float]Histogram.ReduceResolution Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>	2 years ago
Ziqi Zhao	e250f09b5d	change origin schema in `ReduceResolution` method of histogram and float histogram (#13116 ) * change origin schema in ReduceResolution method of histogram and float histogram Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> --------- Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	2 years ago
Ziqi Zhao	ab2a7bb74f	add generic shrink function (#13001 ) Add `ReduceResolution` method to `Histogram` and `FloatHistogram` This takes the original `mergeToSchema` function and turns it into a more generic `reduceResolution` function, which is the building block for the new methods. The methods will help with addressing #12864. --------- Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	2 years ago
Matthieu MOREL	05fba53e57	web : use Go standard package Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2 years ago
Linas Medziunas	ebed7d0612	Change Validate to be a method on histogram structs Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>	2 years ago
Linas Medziunas	ec823d9daf	Update comments, bitwise comparison of float buckets Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>	2 years ago
Linas Medziunas	c5c5c569fa	Histogram performance: optimize floatBucketIterator Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>	2 years ago
Linas Medziunas	3c047a3518	Expand docs comments Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>	2 years ago
Linas Medziunas	cbd01fc296	Fix NaN sum check in [Float]Histogram.Equals method Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>	2 years ago
Ganesh Vernekar	a87e7e9e33	tsdb: Add counter reset hint to histograms and support in WAL Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Marc Tudurí	9474610baf	Support FloatHistogram in TSDB (#11522 ) Extends Appender.AppendHistogram function to accept the FloatHistogram. TSDB supports appending, querying, WAL replay, for this new type of histogram. Signed-off-by: Marc Tudurí <marctc@protonmail.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
beorn7	bf0847073d	histogram: Modify getBound to deal properly with infinity The bucket receiving math.MaxFloat64 observations now has math.MaxFloat64 as upper bound, while the bucket after it (the last possible bucket) has +Inf. This also adds a test for getBound and moves the getBound code to generic.go (where it should have been in the first place). Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Björn Rabenstein	dccfb9db4e	histogram: Remove code replication via generics (#11361 ) * histogram: Simplify iterators We don't really need currLower and currUpper and can calculate it when needed (as already done for the floatBucketIterator). The calculation is cheap, while keeping those extra variables around costs RAM (potentially a lot with many iterators). * histogram: Convert Bucket/FloatBucket to one generic type * histogram: Move some bucket iterator code into generic base iterator * histogram: Remove cumulative iterator for FloatHistogram We added it in the past for completeness (Histogram has one), but it has never been used. Plus, even the cumulative iterator for Histogram is only there for test reasons. We can always add it back, and then maybe even using generics. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	a7c519930e	histograms: Add Compact method to the normal integer Histogram And use the new method to call to compact Histograms during parsing. This happens for both `Histogram` and `FloatHistogram`. In this way, if targets decide to optimize the exposition size by merging spans with empty buckets in between, we still get a normalized results. It will also normalize away any valid but weird representations like empty spans, spans with offset zero, and empty buckets at the start or end of a span. The implementation seemed easy at first as it just turns the `compactBuckets` helper into a generic function (which now got its own file). However, the integer Histograms have delta buckets instead of absolute buckets, which had to be treated specially in the generic `compactBuckets` function. To make sure it works, I have added plenty of explicit tests for `Histogram` in addition to the `FloatHistogram` tests. I have also updated the doc comment for the `Compact` method. Based on the insights now expressed in the doc comment, compacting with a maxEmptyBuckets > 0 is rarely useful. Therefore, this commit also sets the value to 0 in the two cases we were using 3 so far. We might still want to reconsider, so I don't want to remove the maxEmptyBuckets parameter right now. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Ganesh Vernekar	2474c6fb2c	Error on amending histograms on append (#11308 ) * Error on amending histograms on append Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Rename Matches to Equals Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
beorn7	08efde4de5	Adjust to newest gofumpt Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	f9c411604d	Fix spelling errors Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Björn Rabenstein	7e42acd3b1	tsdb: Rework iterators (#9877 ) - Pick At... method via return value of Next/Seek. - Do not clobber returned buckets. - Add partial FloatHistogram suppert. Note that the promql package is now _only_ dealing with FloatHistograms, following the idea that PromQL only knows float values. As a byproduct, I have removed the histogramSeries metric. In my understanding, series can have both float and histogram samples, so that metric doesn't make sense anymore. As another byproduct, I have converged the sampleBuf and the histogramSampleBuf in memSeries into one. The sample type stored in the sampleBuf has been extended to also contain histograms even before this commit. Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	6a820a646c	histogram: Add FloatHistogram Including a few adjustments for normal Histogram, too, e.g. use pointer receiver to avoid the large copy on method calls. Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	73858d7f82	storage: histogram support in memoized_iterator Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	f1065e44a4	model: String method for histogram.Histogram This includes a regular bucket iterator and a string method for histogram.Bucket. Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
beorn7	7a8bb8222c	Style cleanup of all the changes in sparsehistogram so far A lot of this code was hacked together, literally during a hackathon. This commit intends not to change the code substantially, but just make the code obey the usual style practices. A (possibly incomplete) list of areas: * Generally address linter warnings. * The `pgk` directory is deprecated as per dev-summit. No new packages should be added to it. I moved the new `pkg/histogram` package to `model` anticipating what's proposed in #9478. * Make the naming of the Sparse Histogram more consistent. Including abbreviations, there were just too many names for it: SparseHistogram, Histogram, Histo, hist, his, shs, h. The idea is to call it "Histogram" in general. Only add "Sparse" if it is needed to avoid confusion with conventional Histograms (which is rare because the TSDB really has no notion of conventional Histograms). Use abbreviations only in local scope, and then really abbreviate (not just removing three out of seven letters like in "Histo"). This is in the spirit of https://github.com/golang/go/wiki/CodeReviewComments#variable-names * Several other minor name changes. * A lot of formatting of doc comments. For one, following https://github.com/golang/go/wiki/CodeReviewComments#comment-sentences , but also layout question, anticipating how things will look like when rendered by `godoc` (even where `godoc` doesn't render them right now because they are for unexported types or not a doc comment at all but just a normal code comment - consistency is queen!). * Re-enabled `TestQueryLog` and `TestEndopints` (they pass now, leaving them disabled was presumably an oversight). * Bucket iterator for histogram.Histogram is now created with a method. * HistogramChunk.iterator now allows iterator recycling. (I think @dieterbe only commented it out because he was confused by the question in the comment.) * HistogramAppender.Append panics now because we decided to treat staleness marker differently. Signed-off-by: beorn7 <beorn@grafana.com>	4 years ago
Ganesh Vernekar	f0688c21d6	Log sparse histograms into WAL and replay from it (#9191 ) Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	4 years ago
Dieter Plaetinck	bef872bf3c	clarify Signed-off-by: Dieter Plaetinck <dieter@grafana.com>	5 years ago
Ganesh Vernekar	4c01ff5194	Bunch of fixes for sparse histograms (#9043 ) * Do not panic on histoAppender.Append Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * M-map all chunks on shutdown Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Support negative schema for querying Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	5 years ago
Ganesh Vernekar	78d68d5972	Make query_range serve histograms (#9036 ) * Modify query_range to serve only sparse histograms Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Finish CumulativeExpandSparseHistogram for positive schema Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix lint Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix bug and comment out tests for query_range Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix lint 2 Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	5 years ago
Dieter Plaetinck	4d27816ea5	Sparsehistogram: improve dod encoding, testing, encode chunk metadata (#9015 ) * factor out different varbit schemes and include Beorn's "optimum" for buckets Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * use more compact dod encoding scheme for SHS chunk columns Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * remove FB VB and xor dod encoding because we won't use it Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * HistoChunk metadata encoding Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * add SparseHistogram.Copy() Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * histogram test: test appending a few histograms Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * add license headers Signed-off-by: Dieter Plaetinck <dieter@grafana.com>	5 years ago
Dieter Plaetinck	58917d1b76	sparsehistogram: integer types and timestamp separation (#9014 ) * integer types and timestamp separation 1) unify types to int64. as suggested by beorn. we want to support counters going down (resets) even if we plan to create new chunks for now, in that case 2) histogram type doesn't know its own timestamp. include it separately in appending and iteration Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * correction: count and zeroCount to remain unsigned to make api more resilient and that's what we use in protobuf anyway Signed-off-by: Dieter Plaetinck <dieter@grafana.com> * temp hack. Ganesh will fix Signed-off-by: Dieter Plaetinck <dieter@grafana.com>	5 years ago
Ganesh Vernekar	64bea6999e	HistogramAppender interface for sparse histograms (#9007 ) Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	5 years ago
Ben Kochie	269e7c8091	Fix golint issues. Signed-off-by: Ben Kochie <superq@gmail.com>	6 years ago
Krasi Georgiev	12fe204ea6	move runtime debug funcs in own package (#4494 ) To make local debuging with `go run` easyer moved all files into a dedicate package `runtime`. This allows running prometheus just by using `go run main.go` instead of passing mani files like `go run main.go limits_default.go ...` Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
Simon Pasquier	ba22b10113	prometheus: log virtual memory limits Signed-off-by: Simon Pasquier <spasquie@redhat.com>	8 years ago
pasquier-s	2440696961	Log file descriptor limits at startup (#3567 ) Fixes #3564	8 years ago
Shashank Varanasi	61235fd851	Print system information (uname) at Prometheus startup (#2709 ) * Print uname on prom startup * Make uname file linux-only * Add missing license headers Add missing license headers * Print OS when uname is not available * Print only OS name when uname not available * Remove extra space, fix cmd/prometheus/main.go license header * Add fix for int8 and uint8 systems * Better formatting for build tags in cmd/prometheus/uname files * Remove newline	9 years ago
Fabian Reinartz	7d441abd7b	vendor: update prometheus org dependencies	10 years ago
Fabian Reinartz	468d2eb60e	Vendoring via govendor	10 years ago
Fabian Reinartz	1535ef1457	Replace metric.SamplePair with model.SamplePair	11 years ago
Fabian Reinartz	306e8468a0	Switch from client_golang/model to common/model	11 years ago
Julius Volz	d4374a9265	More efficient JSON query result format. This depends on https://github.com/prometheus/client_golang/pull/51. For vectors, the result format looks like this: ```json { "version": 1, "type" : "vector", "value" : [ { "timestamp" : 1421765411.045, "value" : "65.475000", "metric" : { "quantile" : "0.5", "instance" : "http://localhost:9090/metrics", "job" : "prometheus", "__name__" : "http_request_duration_microseconds", "handler" : "/static/", "method" : "get", "code" : "304" } }, { "timestamp" : 1421765411.045, "value" : "5826.339000", "metric" : { "quantile" : "0.9", "instance" : "http://localhost:9090/metrics", "job" : "prometheus", "__name__" : "http_request_duration_microseconds", "handler" : "prometheus", "method" : "get", "code" : "200" } }, /* ... / ] } ``` For matrices, it looks like this: ```json { "version": 1, "type" : "matrix", "value" : [ { "metric" : { "quantile" : "0.99", "instance" : "http://localhost:9090/metrics", "job" : "prometheus", "__name__" : "http_request_duration_microseconds", "handler" : "/static/", "method" : "get", "code" : "200" }, "values" : [ [ 1421765547.659, "29162.953000" ], [ 1421765548.659, "29162.953000" ], [ 1421765549.659, "29162.953000" ], / ... */ ] } ] } ```	11 years ago
Bjoern Rabenstein	5859b74f1b	Clean up license issues. - Move CONTRIBUTORS.md to the more common AUTHORS. - Added the required NOTICE file. - Changed "Prometheus Team" to "The Prometheus Authors". - Reverted the erroneous changes to the Apache License.	11 years ago
Julius Volz	c9618d11e8	Introduce copy-on-write for metrics in AST. This depends on changes in: https://github.com/prometheus/client_golang/tree/cow-metrics. Change-Id: I80b94833a60ddf954c7cd92fd2cfbebd8dd46142	11 years ago

34 Commits (bwplotka-patch-3)