prometheus

Commit Graph

Author	SHA1	Message	Date
Lukasz Mierzwa	59761f631b	Move m.targetsMtx.Lock down into the loop Make sure the order of locks is always the same in all functions. In ApplyConfig() we have m.targetsMtx.Lock() after provider is locked, so replicate the same in allGroups(). Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>	7 months ago
Lukasz Mierzwa	7d55ee8cc8	Try fixing potential deadlocks in discovery Manager.ApplyConfig() uses multiple locks: - Provider.mu - Manager.targetsMtx Manager.cleaner() uses the same locks but in the opposite order: - First it locks Manager.targetsMtx - The it locks Provider.mu I've seen a few strange cases of Prometheus hanging up on shutdown and never compliting that shutdown. From a few traces I was given it appears that while Prometheus is still running only discovery.Manager and notifier.Manager are running running. From that trace it also seems like they are stuck on a lock from two functions: - cleaner waits on a RLock() - ApplyConfig waits on a Lock() I cannot reproduce it but I suspect this is a race between locks. Imagine this scenario: - Manager.ApplyConfig() is called - Manager.ApplyConfig locks Provider.mu.Lock() - at the same time cleaner() is called on the same Provider instance and it calls Manager.targetsMtx.Lock() - Manager.ApplyConfig() now calls Manager.targetsMtx.Lock() but that lock is already held by cleaner() function so ApplyConfig() hangs there - at the same time cleaner() now wants to lock Provider.mu.Rlock() but that lock is already held by Manager.ApplyConfig() - we end up with both functions locking each other out without any way to break that lock Re-order lock calls to try to avoid this scenario. I tried writing a test case for it but couldn't hit this issue. Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>	7 months ago
Matthieu MOREL	b472ce7010	chore: enable early-return from revive Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	10 months ago
TJ Hoplock	6ebfbd2d54	chore!: adopt log/slog, remove go-kit/log For: #14355 This commit updates Prometheus to adopt stdlib's log/slog package in favor of go-kit/log. As part of converting to use slog, several other related changes are required to get prometheus working, including: - removed unused logging util func `RateLimit()` - forward ported the util/logging/Deduper logging by implementing a small custom slog.Handler that does the deduping before chaining log calls to the underlying real slog.Logger - move some of the json file logging functionality to use prom/common package functionality - refactored some of the new json file logging for scraping - changes to promql.QueryLogger interface to swap out logging methods for relevant slog sugar wrappers - updated lots of tests that used/replicated custom logging functionality, attempting to keep the logical goal of the tests consistent after the transition - added a healthy amount of `if logger == nil { $makeLogger }` type conditional checks amongst various functions where none were provided -- old code that used the go-kit/log.Logger interface had several places where there were nil references when trying to use functions like `With()` to add keyvals on the new *slog.Logger type Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	1 year ago
machine424	d23d196db5	fix(discovery): prevent the manager from storing stale targetGroups Signed-off-by: machine424 <ayoubmrini424@gmail.com>	1 year ago
machine424	c586c15ae6	fix(discovery): make discovery manager notify consumers of dropped targets for still defined jobs scrape/manager_test.go: add a test to check that the manager gets notified for targets that got dropped by discovery to reproduce: https://github.com/prometheus/prometheus/issues/12858#issuecomment-1732318102 Signed-off-by: machine424 <ayoubmrini424@gmail.com>	1 year ago
beorn7	0f760f63dd	lint: Revamp our linting rules, mostly around doc comments Several things done here: - Set `max-issues-per-linter` to 0 so that we actually see all linter warnings and not just 50 per linter. (As we also set `max-same-issues` to 0, I assume this was the intention from the beginning.) - Stop using the golangci-lint default excludes (by setting `exclude-use-default: false`. Those are too generous and don't match our style conventions. (I have re-added some of the excludes explicitly in this commit. See below.) - Re-add the `errcheck` exclusion we have used so far via the defaults. - Exclude the signature requirement `govet` has for `Seek` methods because we use non-standard `Seek` methods a lot. (But we keep other requirements, while the default excludes completely disabled the check for common method segnatures.) - Exclude warnings about missing doc comments on exported symbols. (We used to be pretty adamant about doc comments, but stopped that at some point in the past. By now, we have about 500 missing doc comments. We may consider reintroducing this check, but that's outside of the scope of this commit. The default excludes of golangci-lint essentially ignore doc comments completely.) - By stop using the default excludes, we now get warnings back on malformed doc comments. That's the most impactful change in this commit. It does not enforce doc comments (again), but _if_ there is a doc comment, it has to have the recommended form. (Most of the changes in this commit are fixing this form.) - Improve wording/spelling of some comments in .golangci.yml, and remove an outdated comment. - Leave `package-comments` inactive, but add a TODO asking if we should change that. - Add a new sub-linter `comment-spacings` (and fix corresponding comments), which avoids missing spaces after the leading `//`. Signed-off-by: beorn7 <beorn@grafana.com>	1 year ago
machine424	94d28cd6cf	chore(notifier): add a reproducer for https://github.com/prometheus/prometheus/issues/13676 to show "targets groups update" starvation when the notifications queue is full and an Alertmanager is down. The existing `TestHangingNotifier` that was added in https://github.com/prometheus/prometheus/pull/10948 doesn't really reflect the reality as the SD changes are manually fed into `syncCh` in a continuous way, whereas in reality, updates are only resent every `updatert`. The test added here sets up an SD manager and links it to the notifier. The SD changes will be triggered by that manager as it's done in reality. Signed-off-by: machine424 <ayoubmrini424@gmail.com> Co-authored-by: Ethan Hunter <ehunter@hudson-trading.com>	2 years ago
David Ashpole	bbfc72b4e2	support unregistering discovery manager metrics (#13896 ) Signed-off-by: David Ashpole <dashpole@google.com>	2 years ago
Paulin Todev	78411d5e8b	SD Managers taking over responsibility for registration of debug metrics (#13375 ) SD Managers take over responsibility for SD metrics registration --------- Signed-off-by: Paulin Todev <paulin.todev@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in> Co-authored-by: Björn Rabenstein <github@rabenste.in>	2 years ago
Paulin Todev	6a5306a53c	Use const labels for Discovery Manager metrics. Signed-off-by: Paulin Todev <paulin.todev@gmail.com>	2 years ago
Paulin Todev	6de80d7fb0	Allow non-default registry to be used for metrics of SD components Signed-off-by: Paulin Todev <paulin.todev@gmail.com>	2 years ago
Oleksandr Redko	fa90ca46e5	ci(lint): enable godot; append dot at the end of comments Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>	2 years ago
Julien Pivotto	009017a3fb	Revert "Remove deleted target from discovery manager" Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	2 years ago
haleyao	c5a37ddad5	Remove deleted target from discovery manager Signed-off-by: haleyao <haleyao@tencent.com>	2 years ago
Bryan Boreham	2f58be840d	service discovery: add config name to log messages This makes it easier to connect a log message with the config it relates to. Each SD config has a name, either the scrape job name or something like "config-0" for Alertmanager config. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Sebastian Poxhofer	3f9a9d1e62	chore(discoveryManager): expose Discoverer refresh function (#10531 ) Signed-off-by: secustor <sebastian@poxhofer.at>	4 years ago
Jinwook Jeong	c7c7847b6f	Fix discovery managers to be properly cancelled Signed-off-by: Jinwook Jeong <vustthat@gmail.com>	4 years ago
Robert Fratto	44a5e705be	discovery: Expose custom HTTP client options to discoverers (#10462 ) * discovery: expose HTTP client options to discoverers Signed-off-by: Robert Fratto <robertfratto@gmail.com> * discovery/http: use HTTP client options for created client Signed-off-by: Robert Fratto <robertfratto@gmail.com> * scrape: use a list of HTTP client options instead of just dial context Signed-off-by: Robert Fratto <robertfratto@gmail.com> * discovery: rephrase comment Signed-off-by: Robert Fratto <robertfratto@gmail.com>	4 years ago
Julien Pivotto	9621c2c0cc	Fix race with targets update during ApplyConfig (#9656 ) I ended up extending the lock so refTargets remains valid for the duration of the update. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Vladimir Kononov	1043d2b594	Discovery: abstain from restarting providers if possible (#9321 ) (#9349 ) * Abstain from restarting discovery providers if possible (#9321) Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com>	4 years ago
Julien Pivotto	432005826d	Add a feature flag to enable the new discovery manager (#9537 ) * Add a feature flag to enable the new manager This PR creates a copy of the legacy manager and uses it by default. It is a companion PR to #9349. With this PR, users can enable the new discovery manager and provide us with any feedback / side effects that the new behaviour might have. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Levi Harrison	b5f6f8fb36	Switched to go-kit/log Signed-off-by: Levi Harrison <git@leviharrison.dev>	5 years ago
Julien Pivotto	e1774b6f83	Fix the computation of prometheus_sd_discovered_targets prometheus_sd_discovered_targets is wrongly calculated when there are multiple SD configurations in place. One discovery manager can have multiple groups coming from multiple service discoveries. When multiple service discovery configs are used, we do not compute the metric correctly, and instead just set the metric to one of the service discoveries. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Andy Bursavich	4e6a94a27d	Invert service discovery dependencies (#7701 ) This also fixes a bug in query_log_file, which now is relative to the config file like all other paths. Signed-off-by: Andy Bursavich <abursavich@gmail.com>	5 years ago
Julien Pivotto	59de58d380	Docker Swarm service discovery (#7420 ) * Docker Swarm service discovery Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	6 years ago
Julien Pivotto	c61141ce51	Add DigitalOcean service discovery (#7407 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	6 years ago
Marek Slabicki	8224ddec23	Capitalizing first letter of all log lines (#7043 ) Signed-off-by: Marek Slabicki <thaniri@gmail.com>	6 years ago
Julien Pivotto	c67f81937c	discovery: updateGroup should not create targets[poolKey] in the loop (#6903 ) We can assume that not all target groups are nil in normal scernarios, so we can create targets[poolKey] outside the loop. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	6 years ago
johncming	17683d074c	discovery: fix bug that use rlock for read. (#5928 ) Signed-off-by: johncming <johncming@yahoo.com>	6 years ago
Nevill	55661ab004	Set failedConfigs only once right after registerProviders finished Signed-off-by: Nevill <nevill.dutt@gmail.com>	6 years ago
Nevill	048f81218d	Change prometheus_sd_configs_failed_total to Gauge Signed-off-by: Nevill <nevill.dutt@gmail.com>	6 years ago
Harkishen Singh	d98d4a9bf0	remove resetting of manager properties and init manager props under locking (#5979 ) Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>	6 years ago
Matt Layher	302148fd69	*: apply gofmt -s Signed-off-by: Matt Layher <mdlayher@gmail.com>	7 years ago
Ilya Gladyshev	922c17e119	added name label to all discovery metrics (#5002 ) Signed-off-by: Ilya Gladyshev <ilya.v.gladyshev@gmail.com>	7 years ago
Simon Pasquier	8b91d39c43	discovery: send empty group on empty SD config (#4819 ) * discovery: send empty group on blank SD config Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Update comments Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Add another comment Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago
Simon Pasquier	a30348f1a4	discovery: add config label to discovered targets metric (#4753 ) * discovery: add labels to discovered targets metric Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago
Simon Pasquier	657199af22	Address Krasi comments Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago
Simon Pasquier	5df757fdd4	zookeeper: fix panic Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago
Simon Pasquier	365931ea83	discovery: add metrics + send updates from one goroutine only The added metrics are: * prometheus_sd_discovered_targets * prometheus_sd_received_updates_total * prometheus_sd_updates_delayed_total * prometheus_sd_updates_total Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago
Augustin Husson	97950a3fae	remove group if the target is empty at adapter level Signed-off-by: Augustin Husson <husson.augustin@gmail.com>	7 years ago
Simon Pasquier	48989d8996	discovery: add more tests Co-authored-by: Camille Janicki <camille.janicki@gmail.com> Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago
Augustin Husson	9970e4e7e4	unregister source when the target is empty Signed-off-by: Augustin Husson <husson.augustin@gmail.com>	7 years ago
Krasi Georgiev	ba7eb733e8	tidy up the discovery logs,updating loops and selects (#4556 ) * tidy up the discovery logs,updating loops and selects few objects renamings removed a very noise debug log on the k8s discovery. It would be usefull to show some summary rather than every update as this is impossible to follow. added most comments as debug logs so each block becomes self explanatory. when the discovery receiving channel is full will retry again on the next cycle. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com> * add noop logger for the SD manager tests. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com> * spelling nits Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
Simon Pasquier	674c76adb8	discovery: coalesce identical SD configurations (#3912 ) * discovery: coalesce identical SD configurations Instead of creating as many SD providers as declared in the configuration, the discovery manager merges identical configurations into the same provider and keeps track of the subscribers. When the manager receives target updates from a SD provider, it will broadcast the updates to all interested subscribers. Signed-off-by: Simon Pasquier <spasquie@redhat.com>	7 years ago
Krasi Georgiev	53691ae261	Simplify SD update throttling (#4523 ) * simplfied SD updates throtling Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com> * add default to catch cases when we don't have new updates. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
Paul Gier	5c70213f9f	config: set target group source index during unmarshalling (#4245 ) * config: set target group source index during unmarshalling Fixes issue #4214 where the scrape pool is unnecessarily reloaded for a config reload where the config hasn't changed. Previously, the discovery manager changed the static config after loading which caused the in-memory config to differ from a freshly reloaded config. Signed-off-by: Paul Gier <pgier@redhat.com> * [issue #4214] Test that static targets are not modified by discovery manager Signed-off-by: Paul Gier <pgier@redhat.com>	8 years ago
Paul Gier	d24d2acd11	config: set target group source index during unmarshalling (#4245 ) * config: set target group source index during unmarshalling Fixes issue #4214 where the scrape pool is unnecessarily reloaded for a config reload where the config hasn't changed. Previously, the discovery manager changed the static config after loading which caused the in-memory config to differ from a freshly reloaded config. Signed-off-by: Paul Gier <pgier@redhat.com> * [issue #4214] Test that static targets are not modified by discovery manager Signed-off-by: Paul Gier <pgier@redhat.com>	8 years ago
Callum Styan	03578d5df8	add example usage of SD adapter for converting unsupported SD type to filesd (#3720 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	8 years ago
ferhat elmas	ffa673f7d8	General simplifications (#3887 ) Another try as in #1516	8 years ago

1 2

74 Commits (eb8d34c2add51d6f35a2327de10ce16139a7a86c)