prometheus

Commit Graph

Author	SHA1	Message	Date
Matt	cf9d093f8f	prw2: Move Remote Write 2.0 CT to be per Sample; Rename to ST (start timestamp) (#17411 ) (#17600 ) Relates to https://github.com/prometheus/prometheus/issues/16944#issuecomment-3164760343 Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: matt-gp <small_minority@hotmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	4 months ago
Bartlomiej Plotka	7429a75932	scrape: switch scrape manager to AppenderV2 (#17978 ) Signed-off-by: bwplotka <bwplotka@gmail.com>	4 months ago
Abu	0495130e06	tests: improve error handling in main_test.go - Add t.Helper() to getCurrentGaugeValuesFor helper function for better error attribution in test failures - Add require.NoError checks for os.WriteFile calls in TestRuntimeGOGCConfig and TestHeadCompactionWhileScraping to catch file write failures - Strengthen error handling in TestDocumentation to assert command success rather than silently continuing on failure - Improve log message clarity in TestAgentSuccessfulStartup to accurately describe early exit scenario These changes improve test reliability and follow Go testing best practices. Signed-off-by: Abu <abdullahfakrudeen2020@gmail.com>	4 months ago
Julien Pivotto	3bc688e5cb	Features API: Add OpenAPI 3.1 and 3.2 Now that #17825 is merged. Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>	4 months ago
Bartlomiej Plotka	2597a12080	st: Add a hidden 'st-storage' feature flag for PROM-60 (#17907 ) Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	4 months ago
Arve Knudsen	a5f86c3fb6	cmd/prometheus: fix flaky TestQueryLog race condition (#17933 ) Add waitForQueryLog helper that polls for query log entries to appear before asserting, rather than reading the file immediately after making a query. This fixes a race condition where the query log wasn't flushed to disk before the test read the file. The helper uses a 5 second timeout with 100ms polling intervals, which is generous enough to handle slow CI environments while keeping the test responsive. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 months ago
Ganesh Vernekar	3e4a094dbb	Add stale_series_compaction_threshold config file option Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>	4 months ago
Bartlomiej Plotka	bec70227f1	feat(scrape)[PART5b]: Add AppenderV2 support to scrape.NewManager constructor (#17872 ) * feat(scrape)[PART5b]: Add AppenderV2 support to scrape.NewManager optionally to V1 Signed-off-by: bwplotka <bwplotka@gmail.com> * Update scrape/manager.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * fixes after rebase Signed-off-by: bwplotka <bwplotka@gmail.com> * Apply suggestions from code review Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	4 months ago
Bartlomiej Plotka	0d116b0994	tests(teststorage): Close Storage in the helper (#17902 ) Signed-off-by: bwplotka <bwplotka@gmail.com>	4 months ago
wangyazhou	629bd3c5d5	Fix logger syntax for timingsLogger (#17870 ) Fix logger syntax for timingsLogger Signed-off-by: wangyazhou <564631192@qq.com>	5 months ago
Julius Volz	d3b6e61487	Put binop fill modifiers behind a feature flag Signed-off-by: Julius Volz <julius.volz@gmail.com>	5 months ago
Julius Volz	d6aa6a3fc7	Add new fill modifiers to features test data Signed-off-by: Julius Volz <julius.volz@gmail.com>	5 months ago
bwplotka	49c3aea56d	feat(storage)[PART4b]: add AppenderV2 to the rest of storage.Storage impl Signed-off-by: bwplotka <bwplotka@gmail.com>	5 months ago
Ben Kochie	d9ed026658	Refractor promtool errors (#17842 ) Replace use of `tsdb/errors` with standard library `errors`. Signed-off-by: SuperQ <superq@gmail.com>	5 months ago
Arthur Silva Sens	1e317d0098	Add configuration option to control `extra-scrape-metrics` (#17606 )	5 months ago
Ben Kochie	e14795bbf4	Remove copyright date from headers (#17785 ) Remove copyright dates from various files as part of [PROM-50]. [PROM-50]: https://github.com/prometheus/proposals/blob/main/proposals/0050-remove-copyright-dates.md Signed-off-by: SuperQ <superq@gmail.com>	5 months ago
Aditya Tiwari	146080186d	promtool: Add --lint flag to check metrics command to allow disabling linting (#17669 ) * promtool: allow cardinality with metrics linting and add --lint to check metrics Signed-off-by: ADITYA TIWARI <adityatiwari342005@gmail.com> * fix/ci: Simplify test case variable declaration Remove unnecessary variable declaration in test cases. Signed-off-by: ADITYA TIWARI <142050150+ADITYATIWARI342005@users.noreply.github.com> * promtool: avoid Tee for --lint=none Signed-off-by: ADITYA TIWARI <adityatiwari342005@gmail.com> * promtool: validate at least one feature enabled in check metrics addresses feedback to ensure the command does something useful now fails with clear error when both --lint=none and no --extended flag. Signed-off-by: ADITYA TIWARI <adityatiwari342005@gmail.com> --------- Signed-off-by: ADITYA TIWARI <adityatiwari342005@gmail.com> Signed-off-by: ADITYA TIWARI <142050150+ADITYATIWARI342005@users.noreply.github.com>	6 months ago
NamanParlecha	c94101d023	TSDB: Option to configure TSDB Block Reload Interval (#16728 ) Add --storage.tsdb.block-reload-interval flag to configure TSDB block reload interval. --------- Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> Signed-off-by: NamanParlecha <namanparlecha@gmail.com> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>	6 months ago
bwplotka	ad367b504b	refactor(tsdb/agent)[PART3]: add AppenderV2 support to agent Signed-off-by: bwplotka <bwplotka@gmail.com>	6 months ago
Julien Pivotto	a5671a002f	API: Add a /api/v1/features endpoint Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>	6 months ago
dongjiang	3239723098	Update golangci-lint and add modernize check (#17640 ) * add modernize check Signed-off-by: dongjiang1989 <dongjiang1989@126.com> * fix golangci lint Signed-off-by: dongjiang1989 <dongjiang1989@126.com> --------- Signed-off-by: dongjiang1989 <dongjiang1989@126.com>	6 months ago
Julien Pivotto	4620c8ac71	Simplify StartTime assignment in unit test setup. Remove redundant IsZero check since promqltest.LazyLoader already handles zero StartTime by defaulting to Unix epoch. Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>	6 months ago
Julien Pivotto	1a853e23db	Add start_timestamp field for unit tests. This commit adds support for configuring a custom start timestamp for Prometheus unit tests, allowing tests to use realistic timestamps instead of starting at Unix epoch 0. Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>	6 months ago
Łukasz Mierzwa	8a1086a128	feat: Add flag that blocks lvl 1 compactions until upload is confirmed in an external JSON file (#17435 ) * Delay compactions until Thanos uploads all blocks Using Thanos sidecar with Prometheus requires us to disable TSDB compactions on Prometheus side by setting --storage.tsdb.min-block-duration and --storage.tsdb.max-block-duration to the same value. See https://thanos.io/tip/components/sidecar.md. The main problem this avoids is that Prometheus might compact given block before Thanos uploads it, creating a gap in Thanos metrics. Thanos does not upload compacted blocks because that would upload the same sample multiple times. You can tell Thanos to upload compacted blocks but that is aimed at one time migrations. This patch creates a bridge between Thanos and Prometheus by allowing Prometheus to read the shipper file Thanos creates, where it tracks which blocks were already uploaded, and using that data delays compaction of blocks until they are marked as uploaded by Thanos. Thanks to this both services can coordinate with each other (in a way) and we can stop disabling compaction on Prometheus side when Thanos uploads are enabled. The reason to have this is that disabling compactions have very dramatic performance cost. Since most time series exist for longer than a single block duration (2h by default) large chunks of block index will reference the same series, so 10 * 2h blocks will each have an index that is usually fairly big and is almost the same for all 10 blocks. Compaction de-duplicates the index so merging 10 blocks together would leave us with a single index that is around the same size as each of these 10 2h blocks would have (plus some extra for series that only exists in some blocks, but not all). Every range query that iterates over all 10 blocks would then have to read each index and so we're doing 10x more work then if we had a single compacted block. Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Rename structs and functions to make this more generic Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Address review comments Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> * Cache UploadMeta for 1 minute Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> --------- Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>	6 months ago
Ayoub Mrini	e7999528fa	fix(test): make TestRemoteWrite_ReshardingWithoutDeadlock more reliable and re-enable it (#17490 ) Improve test stability by waiting for the relevant metrics to appear on /metrics before the first check on the desired shard count. Increase the scrape interval to avoid timeouts, as 100 ms may be insufficient for Prometheus to scrape itself in some environments (e.g., CI). Have Prometheus scrape itself multiple times to increase the volume of data sent and help fill the queue more quickly. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	6 months ago
Jan Fajerski	26f8c92de8	test: skip TestRemoteWrite_ReshardingWithoutDeadlock temporarily as flaky (#17534 ) (#17543 ) (cherry picked from commit `35c3232a2e`) Signed-off-by: machine424 <ayoubmrini424@gmail.com> Signed-off-by: Jan Fajerski <jfajersk@redhat.com> Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>	7 months ago
beorn7	be4efd740c	cmd: Make feature flag `native-histograms` a no-op. Signed-off-by: beorn7 <beorn@grafana.com>	7 months ago
Ayoub Mrini	35c3232a2e	test: skip TestRemoteWrite_ReshardingWithoutDeadlock temporarily as flaky (#17534 ) Signed-off-by: machine424 <ayoubmrini424@gmail.com>	7 months ago
Bartlomiej Plotka	f50ff0a40a	feat: rename CreatedTimestamp to StartTimestamp (#17523 ) Partially fixes https://github.com/prometheus/prometheus/issues/17416 by renaming all CT* names to ST* in the whole codebase except RW2 (this is done in separate [PR](https://github.com/prometheus/prometheus/pull/17411)) and PrometheusProto exposition proto. ``` CreatedTimestamp -> StartTimestamp CreatedTimeStamp -> StartTimestamp created_timestamp -> start_timestamp CT -> ST ct -> st ``` Signed-off-by: bwplotka <bwplotka@gmail.com>	7 months ago
Bryan Boreham	e17742902b	lint Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	7 months ago
Bryan Boreham	11cf858166	Small refactor: use clearer name formatSeriesSetLabelsToJSON Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	7 months ago
Bryan Boreham	89b92f7880	[TEST] Promtool: test dump series together with dump samples Add data files for series tests. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	7 months ago
Bryan Boreham	beea578b20	Make dump-series a --format flag on the dump command Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	7 months ago
smallfish	5560397a70	promtool: add dump-series Signed-off-by: smallfish <smallfish.xy@gmail.com>	7 months ago
Ben Kochie	204249fcb5	Update golangci-lint (#17478 ) * Update golangci-lint to v2.6.0 * Fixup various linting issues. * Fixup deprecations. * Add exception for `labels.MetricName` deprecation. Signed-off-by: SuperQ <superq@gmail.com>	7 months ago
Minh Nguyen	30992dd032	[RW2] Fix: Only update metadata to WAL when metadata-wal-records feature is enabled (#17470 ) * add feature check when UpdateMetadata Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * add appendMetadata boolean to write_hander Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	7 months ago
Ben Kochie	48956f60d7	Update modernize (#17471 ) Apply additional Go modernize tool improvements. Signed-off-by: SuperQ <superq@gmail.com>	7 months ago
Minh Nguyen	9f93c2d2e1	promtool: Add Remote Write 2.0 support to push metrics command (#17417 ) * add feature flag for remote write v2 Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * change from number to protobuf_message Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix test Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix name Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * run make cli-documentation Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix help Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * run make cli-documentation Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	7 months ago
Ayoub Mrini	6806b68f93	[release-3.7] fix: Remote-write: revert changes in the queue resharding logic (#17412 ) * Revert "chore: deprecate prometheus_remote_storage_{samples,exemplars,histograms}_in_total and prometheus_remote_storage_highest_timestamp_in_seconds" This reverts commit `ba14bc49db`. Signed-off-by: machine424 <ayoubmrini424@gmail.com> * Revert "storage/remote: compute highestTimestamp and dataIn at QueueManager level" This reverts commit `184c7eb918`. Signed-off-by: machine424 <ayoubmrini424@gmail.com> * fix(remote-write): bring back the per queue metrics Signed-off-by: machine424 <ayoubmrini424@gmail.com> * test(remote): add TestRemoteWrite_ReshardingWithoutDeadlock to reproduce the sharding scale up deadlock Signed-off-by: machine424 <ayoubmrini424@gmail.com> --------- Signed-off-by: machine424 <ayoubmrini424@gmail.com>	7 months ago
Arve Knudsen	92ccadc96d	promtool: generate rulesLintConfig when `--lint=none` is set (#17414 ) * Correctly generate rulesLintConfig when --lint=none is set (#17399) If I run promtool check config --lint=none I get: ``` Checking rules.yml FAILED: rules.yml: unset nameValidationScheme ``` This is because passing --lint=none stops newConfigLintConfig from generating rulesLintConfig which is needed for validation. It means that defaults are used then, one of which is unset value for metric name validation, causing this error. Fix this by handling --lint=none case correctly and still generating rulesLintConfig. --------- Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com> Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	7 months ago
Łukasz Mierzwa	0e187f5542	Correctly generate rulesLintConfig when --lint=none is set (#17399 ) If I run promtool check config --lint=none I get: ``` Checking rules.yml FAILED: rules.yml: unset nameValidationScheme ``` This is becuase passing --lint=none stops newConfigLintConfig from generating rulesLintConfig which is needed for validation. It means that defaults are used then, one of which is unset value for metric name validation, causing this error. Fix this by handling --lint=none case correctly and still generating rulesLintConfig. Fixes #17398. Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>	7 months ago
Minh Nguyen	ad4b59c504	tsdb: Deprecate retention flags; add tsdb.retention runtime configuration (#17026 ) * Move storage from CL to config file Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * Fix .md Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * run make cli-documentation Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * run make cli-documentation Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * nit_fixed Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * add test and update configuration.md Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix lint Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	7 months ago
Minh Nguyen	6bb367970e	feat(promtool): add RW2 support to promtool push metrics using client_golang library (#17280 ) * Add WriteProto method and tests for promtool metrics This commit adds: 1. WriteProto method to storage/remote/client.go that handles marshaling and compression of protobuf messages 2. Updated parseAndPushMetrics in cmd/promtool/metrics.go to use the new WriteProto method 3. Comprehensive tests for PushMetrics functionality The WriteProto method provides a cleaner API for sending protobuf messages without manually handling marshaling and compression. Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * use Write method from exp/api/remote Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix lint Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix test Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * nit fixed Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix lint Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	7 months ago
Minh Nguyen	f070e35358	[RW]: Adopt client_golang/exp/api/remote types for receiving RW1 and RW2 (#17197 ) Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> # Conflicts: # storage/remote/write_handler.go * add comment Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix failling test Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * nit_fixing Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> * fix comment Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com> --------- Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	7 months ago
beorn7	ad7d1aed99	Phase out native histogram feature flag The detailed plan for this is laid out in https://github.com/prometheus/prometheus/issues/16572 . This commit adds a global and local scrape config option `scrape_native_histograms`, which has to be set to true to ingest native histograms. To ease the transition, the feature flag is changed to simply set the default of `scrape_native_histograms` to true. Further implications: - The default scrape protocols now depend on the `scrape_native_histograms` setting. - Everywhere else, histograms are now "on by default". Documentation beyond the one for the feature flag and the scrape config are deliberately left out. See https://github.com/prometheus/prometheus/pull/17232 for that. Signed-off-by: beorn7 <beorn@grafana.com>	8 months ago
Julien	4199c2f45a	Add anchored and smoothed to vector selectors. (#16457 ) * Add anchored and smoothed to vector selectors. This adds "anchored" and "smoothed" keywords that can be used following a matrix selector. "Anchored" selects the last point before the range (or the first one after the range) and adds it at the boundary of the matrix selector. "Smoothed" applies linear interpolation at the edges using the points around the edges. In the absence of a point before or after the edge, the first or the last point is added to the edge, without interpolation. Exemple usage * `increase(caddy_http_requests_total[5m] anchored)` (equivalent of caddy_http_requests_total - caddy_http_requests_total offset 5m but takes counter reset into consideration) * `rate(caddy_http_requests_total[step()] smoothed)` Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> * Update docs/feature_flags.md Co-authored-by: Charles Korn <charleskorn@users.noreply.github.com> Signed-off-by: Julien <291750+roidelapluie@users.noreply.github.com> * Smoothed/Anchored rate: Add more tests Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> * Anchored/Smoothed modifier: error out with histograms Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> --------- Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com> Signed-off-by: Julien <291750+roidelapluie@users.noreply.github.com> Co-authored-by: Charles Korn <charleskorn@users.noreply.github.com>	8 months ago
Arve Knudsen	913cc8f72b	Replace gopkg.in/yaml.v2 with go.yaml.in/yaml/v2 (#17151 ) * Replace gopkg.in/yaml.v2 with go.yaml.in/yaml/v2 * Upgrade to client_golang@v1.23.2 --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	9 months ago
Owen Williams	6ee965c255	common: Update to prom/common v0.66.0, fix TextParser creation (#17139 ) TextParser as of prom/common v0.66.0 requires an explicit validation scheme. Signed-off-by: Owen Williams <owen.williams@grafana.com>	9 months ago
Duciwuci	76a0cce3ac	adjust windows error message Signed-off-by: Duciwuci <duciwuci@gmail.com>	9 months ago
machine424	184c7eb918	storage/remote: compute highestTimestamp and dataIn at QueueManager level Because of relabelling, an endpoint can only select a subset of series that go through WriteStorage Having a highestTimestamp at WriteStorage level yields wrong values if the corresponding sample won't even make it to a remote queue. Currently PrometheusRemoteWriteBehind is based on that, and would fire if an endpoint is only interested in a subset of series that take time to appear. A "prometheus_remote_storage_queue_highest_timestamp_seconds" that only takes into account samples in the queue is introduced, and used in PrometheusRemoteWriteBehind and dashboards in documentation/prometheus-mixin Same applies to samplesIn/dataIn, QueueManager should know more about when to update those; when data is enqueued. That makes dataDropped unnecessary, thus help simplify the logic in QueueManager.calculateDesiredShards() Signed-off-by: machine424 <ayoubmrini424@gmail.com>	9 months ago

1 2 3 4 5 ...

987 Commits (1dcdb07d30fc37cd0ef11c7cf0552da71f9823be)