prometheus

Commit Graph

Author	SHA1	Message	Date
Will Bollock	b70a871988	fix(discovery): delete expired refresh metrics on reload (#17614 ) Building off config-specific Prometheus refresh metrics from an earlier PR (https://github.com/prometheus/prometheus/pull/17138), this deletes refresh metrics like `prometheus_sd_refresh_duration_seconds` and `prometheus_sd_refresh_failures_total` when the underlying scrape job configuration is removed on reload. This reduces un-needed cardinality from scrape job specific metrics while still preserving metrics that indicate overall health of a service discovery engine. For example, `prometheus_sd_refresh_failures_total{config="linode-servers",mechanism="linode"} 1` will no longer be exported by Prometheus when the `linode-servers` scrape job for the Linode service provider is removed. The generic, service discovery specific `prometheus_sd_linode_failures_total` metric will persist however. * fix: add targetsMtx lock for targets access * test: validate refresh/discover metrics are gone * ref: combine sdMetrics and refreshMetrics Good idea from @bboreham to combine sdMetrics and refreshMetrics! They're always passed around together and don't have much of a reason not to be combined. mechanismMetrics makes it clear what kind of metrics this is used for (service discovery mechanisms). --------- Signed-off-by: Will Bollock <wbollock@linode.com>	1 month ago
machine424	dcfb8ce59c	chore: remove util/testutil/synctest now that we use Go>=1.25 Signed-off-by: machine424 <ayoubmrini424@gmail.com>	1 month ago
Arve Knudsen	020a0b30a0	notifier: fix flaky TestStop_DrainingEnabled and TestStop_DrainingDisabled race conditions (#17938 ) Fix flaky TestStop_DrainingEnabled and TestStop_DrainingDisabled tests. The tests used real HTTP servers and real time, making them susceptible to race conditions and timing-dependent failures. The solution is to convert both tests to use synctest for deterministic fake time. --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 months ago
Arve Knudsen	ade3f08eca	notifier: fix flaky TestHangingNotifier race condition (#17934 ) * notifier: fix flaky TestHangingNotifier race condition Make deterministic through `synctest.Test()`. --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 months ago
Siavash Safi	2437977bff	fix(notify): apply config sendloop cleanup fix (#17915 ) These bugs were discovered accidentally with code analysis: - https://app.devin.ai/review/prometheus/prometheus/pull/16355 Upon further inspection and performing more analysis, 3 potential bugs were found: 1. sendloops could continue running if corresponding AM changed position in the config 2. multiple configs with the same hash would share sendloops resulting in sets without sendloops 3. sendloops could continue running if the config hash was changed - `TestApplyConfigSendLoopsNotStoppedOnKeyChange`: Verifies sendLoops work when keys swap (no fix needed) - `TestApplyConfigDuplicateHashSharesSendLoops`: Verifies sendLoops are independent with duplicate hashes (bug fixed) - `TestApplyConfigHashChangeLeaksSendLoops`: Verifies sendLoops are cleaned up when hash changes (bug fixed) Signed-off-by: Siavash Safi <siavash@cloudflare.com>	3 months ago
Siavash Safi	d9ccd70ac1	fix(notify): flaky tests (#17899 ) Add a helper function to set up AlertmanagerSets. Fix all flaky tests. Signed-off-by: Siavash Safi <siavash@cloudflare.com>	3 months ago
Siavash Safi	a89c665f47	feat(notifier): independent alertmanager sendloops (#16355 ) * notifier: unit test for dropping throughput on stuck AM Ref: https://github.com/prometheus/prometheus/issues/7676 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Signed-off-by: Siavash Safi <siavash@cloudflare.com> * chore(notifier): remove year from copyrights Signed-off-by: Siavash Safi <siavash@cloudflare.com> * feat(notifier): independent alertmanager sendloops Independent Alertmanager sendloops avoid issues with queue overflowing when one or more Alertmanager instances are unavailable which could result in lost alert notifications. The sendloops are managed per AlertmanagerSet which are dynamically added/removed with service discovery or configuration reload. The following metrics now include an extra dimention for alertmanager label: - prometheus_notifications_dropped_total - prometheus_notifications_queue_capacity - prometheus_notifications_queue_length This change also includes the test from #14099 Closes #7676 Signed-off-by: machine424 <ayoubmrini424@gmail.com> Signed-off-by: Siavash Safi <siavash@cloudflare.com> --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Signed-off-by: Siavash Safi <siavash@cloudflare.com> Signed-off-by: machine424 <ayoubmrini424@gmail.com> Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	3 months ago
Ben Kochie	e14795bbf4	Remove copyright date from headers (#17785 ) Remove copyright dates from various files as part of [PROM-50]. [PROM-50]: https://github.com/prometheus/proposals/blob/main/proposals/0050-remove-copyright-dates.md Signed-off-by: SuperQ <superq@gmail.com>	4 months ago
Kevin Hellemun	33082be0e2	feat: add histogram metric for notification_latency_seconds (#16637 ) This metric can be used to create alerting based on how many notifications finish or do not finish within a certain amount of time. Change-Id: afbf3d8ceb3994c7d6220389353cff92 Signed-Off-By: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com> Co-authored-by: Björn Rabenstein <github@rabenste.in> --------- Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com> Co-authored-by: Björn Rabenstein <github@rabenste.in>	6 months ago
Ben Kochie	48956f60d7	Update modernize (#17471 ) Apply additional Go modernize tool improvements. Signed-off-by: SuperQ <superq@gmail.com>	6 months ago
Arve Knudsen	913cc8f72b	Replace gopkg.in/yaml.v2 with go.yaml.in/yaml/v2 (#17151 ) * Replace gopkg.in/yaml.v2 with go.yaml.in/yaml/v2 * Upgrade to client_golang@v1.23.2 --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	8 months ago
machine424	8f79470ca9	fix(notifier): create a new alert when relabeling alters labels Signed-off-by: machine424 <ayoubmrini424@gmail.com>	8 months ago
machine424	bd725fd6b8	test(notifier): add a test showing an alert mutation bug between alertmanager_config (alertmanagersets) The alert_relabel_configs should only apply to the corresponding alertmanagerset Signed-off-by: machine424 <ayoubmrini424@gmail.com>	8 months ago
Arve Knudsen	0a40df33fb	Make metric/label name validation scheme explicit (#16928 ) * Parameterize metric/label name validation scheme Parameterized metric/label name validation scheme --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Julius Hinze <julius.hinze@grafana.com>	9 months ago
Matthieu MOREL	cef219c31c	chore: enable unused-receiver rule from revive Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	9 months ago
Yandi Lee	8eb445b8a4	Discovery.Manager: close sync ch after sender() is stopped (#14465 ) * close sync ch after sender() is stopped * break if chan is closed Signed-off-by: liyandi <littlepangdi@163.com> Co-authored-by: liyandi <liyandi@xiaomi.com>	10 months ago
Siavash Safi	ef48e4cb9f	chore: refactor notifier package Split the notifier package into smaller source files. Signed-off-by: Siavash Safi <siavash@cloudflare.com>	1 year ago
Dimitar Dimitrov	bd5b2ea95c	ruler notifier: make batch size configurable (#16254 ) * ruler notifier: make batch size configurable In Mimir we experimented with setting a higher value for the batch size. A 4x increase in batch size decreased the time to process a single notification by about 2x. This reduces the processing time of the notifications queue and increases the throughput of the queue. Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> * Update cmd/prometheus/main.go Co-authored-by: gotjosh <josue.abreu@gmail.com> Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> * Update docs Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> * Use a string constant Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> * Add godoc comment on exported constant Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> --------- Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Co-authored-by: gotjosh <josue.abreu@gmail.com>	1 year ago
Matthieu MOREL	5fa1146e21	chore: enable gci linter (#16245 ) Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	1 year ago
Justin Cichra	78599d0ec6	notifier: Consider alert relabeling when deciding if alerts are dropped When intentionally dropping all alerts in a batch, `SendAll` returns false, increasing the dropped_total metric. This makes it difficult to tell if there is a connection issue between Prometheus and AlertManager or a result of intentionally dropping alerts. Have `SendAll` return `true` when no batches were sent by keeping track of the number of AlertManager request attempts. If no attempts were made, then the send is successful. Fixes: #15978 Signed-off-by: Justin Cichra <jcichra@cloudflare.com>	1 year ago
Matthieu MOREL	c7d4b53ec1	chore: enable unused-parameter from revive Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	1 year ago
Matthieu MOREL	dd5ab743ea	chore(deps): use version.PrometheusUserAgent Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	1 year ago
beorn7	e01c5cefac	notifier: fix increment of metric prometheus_notifications_errors_total Previously, prometheus_notifications_errors_total was incremented by one whenever a batch of alerts was affected by an error during sending to a specific alertmanager. However, the corresponding metric prometheus_notifications_sent_total, counting all alerts that were sent (including those where the sent ended in error), is incremented by the batch size, i.e. the number of alerts. Therefore, the ratio used in the mixin for the PrometheusErrorSendingAlertsToSomeAlertmanagers alert is inconsistent. This commit changes the increment of prometheus_notifications_errors_total to the number of alerts that were sent in the attempt that ended in an error. It also adjusts the metrics help string accordingly and makes the wording in the alert in the mixin more precise. Signed-off-by: beorn7 <beorn@grafana.com>	1 year ago
SuperQ	bc1bc5c118	Update sigv4 library Migrate use of prometheus/common/sigv4 to prometheus/sigv4. Related to: https://github.com/prometheus/common/issues/709 Signed-off-by: SuperQ <superq@gmail.com>	2 years ago
Alan Protasio	c78d5b94af	Disallowing configure AM with the v1 api (#13883 ) * Stop supporting Alertmanager v1 * Disallowing configure AM with the v1 api Signed-off-by: alanprot <alanprot@gmail.com> * Update config/config_test.go Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update config/config.go Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> * Addressing coments Signed-off-by: alanprot <alanprot@gmail.com> * Update notifier/notifier.go Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update config/config_test.go Co-authored-by: Jan Fajerski <jan--f@users.noreply.github.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: alanprot <alanprot@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com> Co-authored-by: Jan Fajerski <jan--f@users.noreply.github.com>	2 years ago
TJ Hoplock	6ebfbd2d54	chore!: adopt log/slog, remove go-kit/log For: #14355 This commit updates Prometheus to adopt stdlib's log/slog package in favor of go-kit/log. As part of converting to use slog, several other related changes are required to get prometheus working, including: - removed unused logging util func `RateLimit()` - forward ported the util/logging/Deduper logging by implementing a small custom slog.Handler that does the deduping before chaining log calls to the underlying real slog.Logger - move some of the json file logging functionality to use prom/common package functionality - refactored some of the new json file logging for scraping - changes to promql.QueryLogger interface to swap out logging methods for relevant slog sugar wrappers - updated lots of tests that used/replicated custom logging functionality, attempting to keep the logical goal of the tests consistent after the transition - added a healthy amount of `if logger == nil { $makeLogger }` type conditional checks amongst various functions where none were provided -- old code that used the go-kit/log.Logger interface had several places where there were nil references when trying to use functions like `With()` to add keyvals on the new *slog.Logger type Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	2 years ago
machine424	83ee57343a	fix(notifier): stop dropping known alertmanagers on each ApplyConfig and waiting on SD to update them. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2 years ago
machine424	3dc623d30b	test(notifier): add reproducer Signed-off-by: machine424 <ayoubmrini424@gmail.com> Co-authored-by: tommy0 <tommy0@mail.ru>	2 years ago
Bryan Boreham	5710ddf24f	[ENHANCEMENT] Alerts: remove metrics for removed Alertmanagers (#13909 ) * [ENHANCEMENT] Alerts: remove metrics for removed Alertmanagers So they don't continue to report stale values. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Nathan Baulch	50cd453c8f	chore: Fix typos (#14868 ) * Fix typos --------- Signed-off-by: Nathan Baulch <nathan.baulch@gmail.com>	2 years ago
Arve Knudsen	250aa5031d	Remove empty line Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Arve Knudsen	3a78e76282	Upgrade golangci-lint to v1.60.1 Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
Lukasz Mierzwa	7694c89497	Increase TestHangingNotifier timeout This test keeps timing out on our arm64 CI server, it does use a very slow timeout and that 5ms doesn't seem to be enough. But it 10x. Signed-off-by: Lukasz Mierzwa <lukasz@cloudflare.com>	2 years ago
Charles Korn	2dd07fbb1b	notifier: optionally drain queued notifications before shutting down (#14290 ) * Add draining of queued notifications to `notifier.Manager` Signed-off-by: Charles Korn <charles.korn@grafana.com> * Update docs Signed-off-by: Charles Korn <charles.korn@grafana.com> * Address PR feedback Signed-off-by: Charles Korn <charles.korn@grafana.com> * Add more logging Signed-off-by: Charles Korn <charles.korn@grafana.com> * Address offline feedback: remove timeout Signed-off-by: Charles Korn <charles.korn@grafana.com> * Ensure stopping takes priority over further processing, make tests more robust Signed-off-by: Charles Korn <charles.korn@grafana.com> * Make channel unbuffered Signed-off-by: Charles Korn <charles.korn@grafana.com> * Update docs Signed-off-by: Charles Korn <charles.korn@grafana.com> * Fix race in test Signed-off-by: Charles Korn <charles.korn@grafana.com> * Remove unnecessary context Signed-off-by: Charles Korn <charles.korn@grafana.com> * Make Stop safe to call multiple times Signed-off-by: Charles Korn <charles.korn@grafana.com> --------- Signed-off-by: Charles Korn <charles.korn@grafana.com>	2 years ago
Arve Knudsen	d902116b41	Fix various linting errors Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 years ago
machine424	70beda092a	fix(notifier): take alertmanagerSet.mtx before checking alertmanagerSet.ams in sendAll Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2 years ago
machine424	690de487e2	chore(notifier): Split 'Run()' into two goroutines: one to receive target updates and trigger reloads and the other one to send notifications. This is done to prevent the latter operation from blocking/starving the former, as previously, the `tsets` channel was consumed by the same goroutine that consumes and feeds the buffered `n.more` channel, the `tsets` channel was less likely to be ready as it's unbuffered and only fed every `SDManager.updatert` seconds. See https://github.com/prometheus/prometheus/issues/13676 and https://github.com/prometheus/prometheus/issues/8768 The synchronization with the sendLoop goroutine is managed through the n.mtx mutex. This uses a similar approach than scrape manager's `efbd6e41c5/scrape/manager.go (L115-L117)` The old TestHangingNotifier was replaced by the new one to more closely reflect reality. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	2 years ago
machine424	94d28cd6cf	chore(notifier): add a reproducer for https://github.com/prometheus/prometheus/issues/13676 to show "targets groups update" starvation when the notifications queue is full and an Alertmanager is down. The existing `TestHangingNotifier` that was added in https://github.com/prometheus/prometheus/pull/10948 doesn't really reflect the reality as the SD changes are manually fed into `syncCh` in a continuous way, whereas in reality, updates are only resent every `updatert`. The test added here sets up an SD manager and links it to the notifier. The SD changes will be triggered by that manager as it's done in reality. Signed-off-by: machine424 <ayoubmrini424@gmail.com> Co-authored-by: Ethan Hunter <ehunter@hudson-trading.com>	2 years ago
Oleksandr Redko	f10c3454e9	Enable perfsprint linter and fix up code Signed-off-by: Oleksandr Redko <oleksandr.red+github@gmail.com>	2 years ago
Matthieu MOREL	d496687c8e	golangci-lint: enable usestdlibvars linter Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2 years ago
Simon Pasquier	8bd6ae1b20	Notifier: fix deadlock when zero alerts When all alerts were dropped after alert relabeling, the `sendAll()` function didn't release the lock properly which created a deadlock with the Alertmanager target discovery. In addition, the commit detects early when there are no Alertmanager endpoint to notify to avoid unnecessary work. Signed-off-by: Simon Pasquier <spasquie@redhat.com>	2 years ago
Bryan Boreham	8c4e4b72a8	Notifier: pass parameters to goroutine explicitly Avoids possible false sharing between loops. Plausibly there is no problem in the current code, but it's easy enough to write it more safely. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	57c799132b	Notifier: don't reuse payload after relabeling Also clarify why these variables are being cleared. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
darshanime	61ce18950f	Fix pipeline golangci-lint error Signed-off-by: darshanime <deathbullet@gmail.com>	2 years ago
Matthieu MOREL	9c4782f1cc	golangci-lint: enable testifylint linter (#13254 ) Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2 years ago
TJ Hoplock	26b78da281	ci: use go1.21.0 fmt to make ci happy https://github.com/prometheus/prometheus/actions/runs/6044443719/job/16403043771?pr=12774 Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	3 years ago
TJ Hoplock	51d1d2cd96	feat: add AWS sigv4 support to alertmanager endpoints Addresses: #12536 This commit adds support for configuring sigv4 to an `alertmanager_config`. Based heavily on the sigv4 work in the remote write client. Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	3 years ago
m.nabokikh	9d8463339d	Fixes according to the code review Signed-off-by: m.nabokikh <maksim.nabokikh@flant.com>	3 years ago
m.nabokikh	39d008f94f	Route different alerts to different alertmanagers Signed-off-by: m.nabokikh <maksim.nabokikh@flant.com>	3 years ago
Bryan Boreham	3711339a7d	Alerts: more efficient relabel on Send Re-use `labels.Builder` and use `relabel.ProcessBuilder` to skip a conversion step. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago

1 2 3 4

164 Commits (main)