prometheus

Commit Graph

Author	SHA1	Message	Date
George Krajcsovits	5b7ff92d95	fix(promql): histogram_quantile and histogram_fraction NaN observed in native histogram (#16724 ) * fix(promql): histogram_quantile NaN observed in native histogram Fixes: #16578 See the issue for detailed explanation. When a histogram had only NaN observations and no normal observations, we returned 0 from the quantile, which is completely wrong. If there were normal observations but we went over them, we returned the upper bound of the existing buckets, however that contradicts expectations on histogram_fraction. Now we return NaN if the quantile is calculated to be over all normal observations, falling into NaNs (in a virtual +Inf bucket). We also return info level annotations if we see any NaN observations. The annotation calls out if we returned NaN or even if we took the virtual +Inf bucket into account. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * fix(promql): histogram_fraction NaN observed in native histogram Fixes: #16580 According to the specification we should not take NaN observations into account when calculating the fraction. This commit fixes that and adds an info level annotation to let the user know about this. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 days ago
Arthur Silva Sens	8d0a747237	PROM-39: Provide PromQL info annotations when rate()/increase() over series without `__type__`="counter" label (#16632 ) * Provide PromQL info annotations when rate()/increase() over series without counter label Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> * Address comments Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> --------- Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>	3 weeks ago
Innokentii Konstantinov	cd5bd11bd8	Export QuerySamples StartTimestamp and Interval to enable custom statistics implementations to populate these fields as needed. Signed-off-by: Innokentii Konstantinov <innokenty.konstantinov@grafana.com>	3 weeks ago
Julius Volz	13564c03ef	Standardize doc page title handling See https://groups.google.com/g/prometheus-developers/c/cwL3cM66Em8 Signed-off-by: Julius Volz <julius.volz@gmail.com>	4 weeks ago
jub0bs	4bc8df0f54	util/httputil: Always add Vary header in SetCORS Closes #15406 Signed-off-by: jub0bs <jcretel-infosec+github@protonmail.com>	1 month ago
Arve Knudsen	e7e3ab2824	Fix linting issues found by golangci-lint v2.0.2 (#16368 ) * Fix linting issues found by golangci-lint v2.0.2 --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 months ago
Arthur Silva Sens	95f49dd84b	Bump prometheus/common to v0.63.0 (#16210 ) * Bump prometheus/common to v0.63.0 Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> * nolint usage of deprecated model.NameValidationScheme Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> --------- Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>	4 months ago
Bartlomiej Plotka	7a7bc65237	Add util/compression package to consolidate snappy/zstd use in Prometheus. (#16156 ) # Conflicts: # tsdb/db_test.go Apply suggestions from code review tmp Addressed comments. Update util/compression/buffers.go Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>	4 months ago
Matthieu MOREL	c7d4b53ec1	chore: enable unused-parameter from revive Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	4 months ago
jub0bs	329ec6831a	util/httputil: reduce heap allocations in newCompressedResponseWriter (#16001 ) * util/httputil: Benchmark newCompressedResponseWriter This benchmark illustrates that newCompressedResponseWriter incurs a prohibitive amount of heap allocations when handling a request containing a malicious Accept-Encoding header.¬ Signed-off-by: jub0bs <jcretel-infosec+github@protonmail.com> * util/httputil: Improve newCompressedResponseWriter This change dramatically reduces the heap allocations (in bytes) incurred when handling a request containing a malicious Accept-Encoding header. Below are some benchmark results; for conciseness, I've omitted the name of the benchmark function (BenchmarkNewCompressionHandler_MaliciousAcceptEncoding): ``` goos: darwin goarch: amd64 pkg: github.com/prometheus/prometheus/util/httputil cpu: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz │ old │ new │ │ sec/op │ sec/op vs base │ 18.60m ± 2% 13.54m ± 3% -27.17% (p=0.000 n=10) │ old │ new │ │ B/op │ B/op vs base │ 16785442.50 ± 0% 32.00 ± 0% -100.00% (p=0.000 n=10) │ old │ new │ │ allocs/op │ allocs/op vs base │ 2.000 ± 0% 1.000 ± 0% -50.00% (p=0.000 n=10) ``` Signed-off-by: jub0bs <jcretel-infosec+github@protonmail.com> --------- Signed-off-by: jub0bs <jcretel-infosec+github@protonmail.com>	5 months ago
Neeraj Gartia	8be416a67c	[FIX] PromQL: Updates annotation for bin op between incompatible histograms (#15895 ) PromQL: Updates annotation for bin op between incompatible histograms --------- Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	5 months ago
crystalstall	616914abe2	Signed-off-by: crystalstall <crystalruby@qq.com> refactor: using slices.Contains to simplify the code Signed-off-by: crystalstall <crystalruby@qq.com>	6 months ago
Neeraj Gartia	0e99ca3e8c	[BUGFIX] PromQL: Fix `deriv`, `predict_linear` and `double_exponential_smoothing` with histograms (#15686 ) PromQL: Fix deriv, predict_linear and double_exponential_smoothing with histograms Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com> --------- Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	6 months ago
Joel Beckmeyer	bad1c75514	fix alignment of atomic uint64 on 32-bit Signed-off-by: Joel Beckmeyer <joel@beckmeyer.us>	6 months ago
TJ Hoplock	437362a7a7	fix(deduper): use ptr to sync.RWMutex, fix panic during concurrent use Resolves: #15559 As accurately noted in the issue description, the map is shared among child loggers that get created when `WithAttr()`/`WithGroup()` are called on the underlying handler, which happens via `log.With()` and `log.WithGroup()` respectively. The RW mutex was a value in the previous implementation that used go-kit/log, and I should've updated it to use a pointer when I converted the deduper. Also adds a test. Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	7 months ago
TJ Hoplock	e0104a6b7e	ref: JSONFileLogger slog handler, add scrape.FailureLogger interface Improves upon #15434, better resolves #15433. This commit introduces a few changes to ensure safer handling of the JSONFileLogger: - the JSONFileLogger struct now implements the slog.Handler interface, so it can directly be used to create slog Loggers. This pattern more closely aligns with upstream slog usage (which is generally based around handlers), as well as making it clear that devs are creating a whole new logger when interacting with it (vs silently modifying internal configs like it did previously). - updates the `promql.QueryLogger` interface to be a union of the methods of both the `io.Closer` interface and the `slog.Handler` interface. This allows for plugging in/using slog-compatible loggers other than the JSONFileLogger, if desired (ie, for downstream projects). - introduces new `scrape.FailureLogger` interface; just like `promql.QueryLogger`, it is a union of `io.Closer` and `slog.Handler` interfaces. Similar logic applies to reasoning. - updates tests where needed; have the `FakeQueryLogger` from promql's engine_test implement the `slog.Handler`, improve JSONFileLogger test suite, etc. Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	7 months ago
György Krajcsovits	eb9ce70c3e	Set hasCount after setting count to be consistent Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	7 months ago
György Krajcsovits	a48d05912d	nhcb: optimize, do not recalculate suffixes multiple times Reduce string manipulation by just cutting off the histogram suffixes from the series name label once. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	7 months ago
György Krajcsovits	41f051e9a4	Small improvement in handling cases without count Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	7 months ago
TJ Hoplock	3e24e84172	fix!: stop unbounded memory usage from query log Resolves: #15433 When I converted prometheus to use slog in #14906, I update both the `QueryLogger` interface, as well as how the log calls to the `QueryLogger` were built up in `promql.Engine.exec()`. The backing logger for the `QueryLogger` in the engine is a `util/logging.JSONFileLogger`, and it's implementation of the `With()` method updates the logger the logger in place with the new keyvals added onto the underlying slog.Logger, which means they get inherited onto everything after. All subsequent calls to `With()`, even in later queries, would continue to then append on more and more keyvals for the various params and fields built up in the logger. In turn, this causes unbounded growth of the logger, leading to increased memory usage, and in at least one report was the likely cause of an OOM kill. More information can be found in the issue and the linked slack thread. This commit does a few things: - It was referenced in feedback in #14906 that it would've been better to not change the `QueryLogger` interface if possible, this PR proposes changes that bring it closer to alignment with the pre-3.0 `QueryLogger` interface contract - reverts `promql.Engine.exec()`'s usage of the query logger to the pattern of building up an array of args to pass at once to the end log call. Avoiding the repetitious calls to `.With()` are what resolve the issue with the logger growth/memory usage. - updates the scrape failure logger to use the update `QueryLogger` methods in the contract. - updates tests accordingly - cleans up unused methods Builds and passes tests successfully. Tested locally and confirmed I could no longer reproduce the issue/it resolved the issue. Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	7 months ago
TJ Hoplock	16a4354b48	chore(deduper): add compile check that slog.Handler int is satisfied Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	7 months ago
Neeraj Gartia	789c9b1a5e	[BUGFIX] PromQL: Corrects the behaviour of some operator and aggregators with Native Histograms (#15245 ) PromQL: Correct the behaviour of some operator and aggregators with Native Histograms --------- Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	8 months ago
Julien	5c05c89b8b	Update prometheus/common Signed-off-by: Julien <roidelapluie@o11y.eu>	8 months ago
Julien	28f22b4d80	Update prometheus/common Signed-off-by: Julien <roidelapluie@o11y.eu>	8 months ago
György Krajcsovits	eafe72a0d0	perf(nhcb): optimize away most allocations in convertnhcb In general aim for the happy case when the exposer lists the buckets in ascending order. Use Compact(2) to compact the result of nhcb convert. This is more in line with how client_golang optimizes spans vs buckets. `aef8aedb4b/prometheus/histogram.go (L1485)` Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	8 months ago
TJ Hoplock	4f9e4dc016	ref: remove unused deduper log wrapper methods I used these wrapper methods during initial development of the custom handler that the deduper now implements. Since the deduper implements slog.Handler and can be used directly as a logger, these wrapper methods are no longer needed. Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	8 months ago
TJ Hoplock	b602393473	fix: avoid data race in log deduper This change should have been included in the initial prometheus slog conversion, but I must've lost track of it in all the rebases involved in that PR. This changes the dedupe logger so that the only method that needs to use the lock is the `Handle()` method that actually interacts with the deduplication map. Ex: ``` ================== WARNING: DATA RACE Write at 0x00c000518bc0 by goroutine 29481: github.com/prometheus/prometheus/util/logging.(Deduper).WithAttrs() /home/tjhop/go/src/github.com/prometheus/prometheus/util/logging/dedupe.go:89 +0xef log/slog.(Logger).With() /home/tjhop/.asdf/installs/golang/1.23.1/go/src/log/slog/logger.go:132 +0x106 github.com/prometheus/prometheus/storage/remote.NewQueueManager() /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:483 +0x7a9 github.com/prometheus/prometheus/storage/remote.(WriteStorage).ApplyConfig() /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/write.go:201 +0x102c github.com/prometheus/prometheus/storage/remote.(Storage).ApplyConfig() /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage.go:92 +0xfd github.com/prometheus/prometheus/storage/remote.TestWriteStorageApplyConfigsDuringCommit.func1() /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage_test.go:172 +0x3e4 github.com/prometheus/prometheus/storage/remote.TestWriteStorageApplyConfigsDuringCommit.gowrap1() /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage_test.go:174 +0x41 Previous read at 0x00c000518bc0 by goroutine 31261: github.com/prometheus/prometheus/util/logging.(Deduper).Handle() /home/tjhop/go/src/github.com/prometheus/prometheus/util/logging/dedupe.go:82 +0x2b1 log/slog.(Logger).log() /home/tjhop/.asdf/installs/golang/1.23.1/go/src/log/slog/logger.go:257 +0x228 log/slog.(Logger).Error() /home/tjhop/.asdf/installs/golang/1.23.1/go/src/log/slog/logger.go:230 +0x3d4 github.com/prometheus/prometheus/tsdb/wlog.(Watcher).loop() /home/tjhop/go/src/github.com/prometheus/prometheus/tsdb/wlog/watcher.go:254 +0x2db github.com/prometheus/prometheus/tsdb/wlog.(Watcher).Start.gowrap1() /home/tjhop/go/src/github.com/prometheus/prometheus/tsdb/wlog/watcher.go:227 +0x33 Goroutine 29481 (running) created at: github.com/prometheus/prometheus/storage/remote.TestWriteStorageApplyConfigsDuringCommit() /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage_test.go:164 +0xe4 testing.tRunner() /home/tjhop/.asdf/installs/golang/1.23.1/go/src/testing/testing.go:1690 +0x226 testing.(T).Run.gowrap1() /home/tjhop/.asdf/installs/golang/1.23.1/go/src/testing/testing.go:1743 +0x44 Goroutine 31261 (running) created at: github.com/prometheus/prometheus/tsdb/wlog.(Watcher).Start() /home/tjhop/go/src/github.com/prometheus/prometheus/tsdb/wlog/watcher.go:227 +0x177 github.com/prometheus/prometheus/storage/remote.(QueueManager).Start() /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:934 +0x304 github.com/prometheus/prometheus/storage/remote.(WriteStorage).ApplyConfig() /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/write.go:232 +0x151b github.com/prometheus/prometheus/storage/remote.(Storage).ApplyConfig() /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage.go:92 +0xfd github.com/prometheus/prometheus/storage/remote.TestWriteStorageApplyConfigsDuringCommit.func1() /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage_test.go:172 +0x3e4 github.com/prometheus/prometheus/storage/remote.TestWriteStorageApplyConfigsDuringCommit.gowrap1() /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage_test.go:174 +0x41 ================== --- FAIL: TestWriteStorageApplyConfigsDuringCommit (2.26s) testing.go:1399: race detected during execution of test FAIL FAIL github.com/prometheus/prometheus/storage/remote 68.321s ``` Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	8 months ago
Vanshika	cccbe72514	TSDB: Fix some edge cases when OOO is enabled (#14710 ) Fix some edge cases when OOO is enabled Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> Signed-off-by: Vanshika <102902652+Vanshikav123@users.noreply.github.com> Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com> Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com>	8 months ago
Neeraj Gartia	d4b1f9eb33	Corrects the behaviour of binary opperators between histogram and float (#14726 ) promql: corrects binary operators functioning for mixed sample with histogram and float For invalid pairings of sample types, an annotation is added now. Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com> --------- Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	8 months ago
György Krajcsovits	78de9bd10f	convertnhcb: use CutSuffix instead of regex replace for histogram name This is much quicker. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	9 months ago
George Krajcsovits	522149a2ae	model: move classic NHCB conversion into its own file (#15156 ) * model: move classic to NHCB conversion into its own file In preparation for #14978. Author: Jeanette Tan <jeanette.tan@grafana.com> 2024-07-03 11:56:48 Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: Jeanette Tan <jeanette.tan@grafana.com> Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * Better naming from review comment Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * Add doc strings. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> --------- Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com> Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: Jeanette Tan <jeanette.tan@grafana.com>	9 months ago
TJ Hoplock	6ebfbd2d54	chore!: adopt log/slog, remove go-kit/log For: #14355 This commit updates Prometheus to adopt stdlib's log/slog package in favor of go-kit/log. As part of converting to use slog, several other related changes are required to get prometheus working, including: - removed unused logging util func `RateLimit()` - forward ported the util/logging/Deduper logging by implementing a small custom slog.Handler that does the deduping before chaining log calls to the underlying real slog.Logger - move some of the json file logging functionality to use prom/common package functionality - refactored some of the new json file logging for scraping - changes to promql.QueryLogger interface to swap out logging methods for relevant slog sugar wrappers - updated lots of tests that used/replicated custom logging functionality, attempting to keep the logical goal of the tests consistent after the transition - added a healthy amount of `if logger == nil { $makeLogger }` type conditional checks amongst various functions where none were provided -- old code that used the go-kit/log.Logger interface had several places where there were nil references when trying to use functions like `With()` to add keyvals on the new *slog.Logger type Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	9 months ago
Julien	21e0f83b68	Move notifications in utils Signed-off-by: Julien <roidelapluie@o11y.eu>	9 months ago
Jorge Alberto Díaz Orozco	06945b3933	Add a mutex and used ports list to the random port generator to avoid port collisions Signed-off-by: Jorge Alberto Díaz Orozco <diazorozcoj@gmail.com>	9 months ago
Jeanette Tan	f35c6649e4	don't blindly convert series with the classic histogram name suffixes if they are not actually histograms based on metadata Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	9 months ago
Jeanette Tan	cd498964e6	expand tests and support conversion to nhcb in the middle of scrape Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	9 months ago
Jeanette Tan	f596f17024	allow option to convert classic histograms to nhcb entirely (don't append classic histogram series) Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	9 months ago
Jeanette Tan	4503145c8b	convert classic histograms to int nhcb where possible instead Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	9 months ago
Jeanette Tan	62e7f0438d	implement basic conversion of classic to nhcb in scrape Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	9 months ago
Nathan Baulch	50cd453c8f	chore: Fix typos (#14868 ) * Fix typos --------- Signed-off-by: Nathan Baulch <nathan.baulch@gmail.com>	10 months ago
Bryan Boreham	d550c4a0b9	[COMMENT] Improve comment on almost.Equal; add tests At first it wasn't obvious to me why the first check would give a different result to the second. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	10 months ago
Ziqi Zhao	87eab0aaad	fix golang lint Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	10 months ago
Ziqi Zhao	c7c4a5c347	add helper function to compare native histograms in testing Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	10 months ago
beorn7	0f760f63dd	lint: Revamp our linting rules, mostly around doc comments Several things done here: - Set `max-issues-per-linter` to 0 so that we actually see all linter warnings and not just 50 per linter. (As we also set `max-same-issues` to 0, I assume this was the intention from the beginning.) - Stop using the golangci-lint default excludes (by setting `exclude-use-default: false`. Those are too generous and don't match our style conventions. (I have re-added some of the excludes explicitly in this commit. See below.) - Re-add the `errcheck` exclusion we have used so far via the defaults. - Exclude the signature requirement `govet` has for `Seek` methods because we use non-standard `Seek` methods a lot. (But we keep other requirements, while the default excludes completely disabled the check for common method segnatures.) - Exclude warnings about missing doc comments on exported symbols. (We used to be pretty adamant about doc comments, but stopped that at some point in the past. By now, we have about 500 missing doc comments. We may consider reintroducing this check, but that's outside of the scope of this commit. The default excludes of golangci-lint essentially ignore doc comments completely.) - By stop using the default excludes, we now get warnings back on malformed doc comments. That's the most impactful change in this commit. It does not enforce doc comments (again), but _if_ there is a doc comment, it has to have the recommended form. (Most of the changes in this commit are fixing this form.) - Improve wording/spelling of some comments in .golangci.yml, and remove an outdated comment. - Leave `package-comments` inactive, but add a TODO asking if we should change that. - Add a new sub-linter `comment-spacings` (and fix corresponding comments), which avoids missing spaces after the leading `//`. Signed-off-by: beorn7 <beorn@grafana.com>	10 months ago
Ziqi Zhao	8f828d45c1	convert TestNativeHistogram_Sum_Count_Add_AvgOperator into testing framework Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	10 months ago
Julien Pivotto	9b5e7623f4	Add support for multiple listening addresses Fixes #9105 Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu> Signed-off-by: Julien <roidelapluie@o11y.eu>	10 months ago
Arve Knudsen	3a78e76282	Upgrade golangci-lint to v1.60.1 Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	10 months ago
Kushal shukla	fe12924638	promtool: JUnit-Format XML Test Results (#14506 ) * Junit compatible output Signed-off-by: Kushal Shukla <kushalshukla110@gmail.com>	11 months ago
B1F030	3648386345	fix fmt Signed-off-by: B1F030 <b1fzhang@gmail.com>	12 months ago
B1F030	15ac95ddd0	change documentcli Signed-off-by: B1F030 <b1fzhang@gmail.com>	12 months ago

1 2 3 4 5 ...

290 Commits (main)