116 Commits (9d182986f19c20aa700fc0577d79bb738adb14c2)

Author	SHA1	Message	Date
Alexander Akhmetov	9d182986f1	Alerting: make StatePersister more configurable to support custom rule-level state persisters (#94590)	9 months ago
Alexander Akhmetov	0a4e6ff86b	Alerting: Add SaveAlertInstancesForRule instance store method (#94505) ... Alerting: Add SaveAlertInstancesForRule method to the InstanceStore interface	9 months ago
Alexander Weaver	393faa8732	Alerting: Move rule evaluation status logic out of prometheus API and into scheduler (#89141) ... * Add health fields to rules and an aggregator method to the scheduler * Move health, last error, and last eval time in together to minimize state processing * Wire up a readonly scheduler to prom api * Extract to exported function * Use health in api_prometheus and fix up tests * Rename health struct to status * Fix tests one more time * Several new tests * Handle inactive rules * Push state mapping into state manager * rename to StatusReader * Rectify cyclo complexity rebase * Convert existing package local status implementation to models one * fix tests * undo RuleDefs rename	10 months ago
Matthew Jacobson	a397bca02e	Alerting: Fix panic with nil annotations & Nodata=alerting/ok/keep (#91506)	12 months ago
Nihal	9ad9b4989b	Alerting: Include a list of ref_Id and aggregated datasource UIDs to alerts when state reason is NoData (#88819) ... * include a list of ref_Id and datasource UID to alerts when state reason is NoData. --------- Signed-off-by: Syed Nihal <syed.nihal@nokia.com>	12 months ago
Matthew Jacobson	ba800692c6	Alerting: Persist AlertInstance ResolvedAt & LastSentAt (#89135) ... * Alerting: Persist AlertInstance ResolvedAt & LastSentAt * Fix test * Modify existing tests * Fix merge conflicts from nullable LastSentAt & ResolvedAt	1 year ago
Alexander Weaver	3b6a8775bb	Alerting: Fix stale values associated with states that have gone to NoData, unify values calculation (#89807) ... * Unify values * Fix with latest changes on main * Fix up NaN test * Keep refIDs with -1 as value * Test that refIDs are preserved on Normal to Error transition * Alerting to err test too * Add a blurb to docs about this behavior	1 year ago
Matthew Jacobson	47c9259d75	Alerting: Ensure we update State.LastSentAt before persisting (#89427)	1 year ago
Matthew Jacobson	3228b64fe6	Alerting: Resend resolved notifications for ResolvedRetention duration (#88938) ... * Simple replace of State.Resolved with State.ResolvedAt * Retain ResolvedAt time between Normal->Normal transition * Introduce ResolvedRetention to keep sending recently resolved alerts * Make ResolvedRetention configurable with resolved_alert_retention * Tick-based LastSentAt for testing of ResendDelay and ResolvedRetention * Do not reset ResolvedAt during Normal->Pending transition Initially this was done to be inline with Prom ruler. However, Prom ruler doesn't keep track of Inactive->Pending/Alerting using the same alert instance, so it's more understandable that they choose not to retain ResolvedAt. In our case, since we use the same cached instance to represent the transition, it makes more sense to retain it. This should help alleviate some odd situations where temporarily entering Pending will stop future resolved notifications that would have happened because of ResolvedRetention. * Pointers for ResolvedAt & LastSentAt To avoid awkward time.Time{}.Unix() defaults on persist	1 year ago
Alexander Akhmetov	667fea6623	Alerting: use hash of labels instead of labels string as the alert state cache key (#88956) ... * Alerting: use hash instead of labels as the cache key * Use data.Labels.Fingerprint to calculate the cache key	1 year ago
Alexander Weaver	a6a9ab4008	Alerting: Do not store series values from past evaluations in state manager for no reason (#87525) ... Do not store previous execution results on states	1 year ago
Jean-Philippe Quéméner	7cfd470c91	fix(alerting): only expose metrics if executing alerts (#85512)	1 year ago
William Wernert	10dc6c6d75	Alerting: Add "Keep Last State" backend functionality (#83940) ... * Implement keep last state for state transitions * Respect For duration when keeping state * Only keep transition from recording an annotation * Add keep last state option for nodata/error in UI	1 year ago
Alexander Weaver	99fa064576	Alerting: Emit warning when creating or updating unusually large groups (#82279) ... * Add config for limit of rules per rule group * Warn when editing big groups through normal API * Warn on prov api writes for groups * Wire up comp root, tests * Also add warning to state manager warm * Drop unnecessary conversion	1 year ago
Jean-Philippe Quéméner	aa25776f81	Alerting: Add a feature flag to periodically save states (#80987)	2 years ago
Jean-Philippe Quéméner	82638d059f	feat(alerting): add state persister interface (#80384)	2 years ago
Matthew Jacobson	1d4419fbe4	Alerting: Fix NoData & Error alerts not resolving when rule is reset (#80184) ... * Alerting: Fix NoData & Error alerts not resolving when rule is reset On rule reset, when creating the PostableAlerts StateToPostableAlert did not attach the correct NoData/Error alertname and rulename labels to expire/resolve the active alerts when the previous cached state was NoData/Error.	2 years ago
Yuri Tseretyan	f6a46744a6	Alerting: Support hysteresis command expression (#75189) ... Backend: * Update the Grafana Alerting engine to provide feedback to HysteresisCommand. The feedback information is stored in state.Manager as a fingerprint of each state. The fingerprint is persisted to the database. Only fingerprints that belong to Pending and Alerting states are considered as "loaded" and provided back to the command. - add ResultFingerprint to state.State. It's different from other fingerprints we store in the state because it is calculated from the result labels. - add rule_fingerprint column to alert_instance - update alerting evaluator to accept AlertingResultsReader via context, and update scheduler to provide it. - add AlertingResultsFromRuleState that implements the new interface in eval package - update getExprRequest to patch the hysteresis command. * Only one "Recovery Threshold" query is allowed to be used in the alert rule and it must be the Condition. Frontend: * Add hysteresis option to Threshold in UI. It's called "Recovery Threshold" * Add test for getUnloadEvaluatorTypeFromCondition * Hide hysteresis in panel expressions * Refactor isInvalid and add test for it * Remove unnecesary React.memo * Add tests for updateEvaluatorConditions --------- Co-authored-by: Sonia Aguilar <soniaaguilarpeiron@gmail.com>	2 years ago
Marcus Efraimsson	e4c1a7a141	Tracing: Standardize on otel tracing (#75528)	2 years ago
gotjosh	e877174501	Alerting: Expose metrics for Alertmanager Alerts - `grafana_alerting_alertmanager_alerts` (#75802) ... * Alerting: Expose metrics for Alertmanager Alerts In Grafana, the alert evaluation and alert delivery are combined. We're always used a metric named `grafana_alerting_alerts` to get a sense of what are the alerts that are currently firing (these come from the evaluation side) and opted to not map the alertmanager alerts metric directly. I think it's important that we make a disction between alerts that happen at evaluation vs alerts that are received for delivery by the internal Alertmanager as we have options to skip the delivery of these alerts to the internal alertmanager altogether.	2 years ago
gotjosh	59694fb2be	Alerting: Don't use a separate collection system for metrics (#75296) ... * Alerting: Don't use a separate collection system for metrics The state package had a metric collection system that ran every 15s updating the values of the metrics - there is a common pattern for this in the Prometheus ecosystem called "collectors". I have removed the behaviour of using a time-based interval to "set" the metrics in favour of a set of functions as the "value" that get called at scrape time.	2 years ago
Steve Simpson	894f420014	Alerting: Pass loggers into SchedulerCfg and ManagerCfg. (#75158)	2 years ago
Yuri Tseretyan	938e26b59f	Alerting: Add new metrics and tracings to state manager and scheduler (#71398) ... * add metrics and tracing to state manager * propagate tracer to state manager * add scheduler metrics * fix backtesting * add test for state metrics * remove StateUpdateCount * update docs * metrics can be null * add tracer to new tests	2 years ago
Yuri Tseretyan	0717ec11d6	Alerting: Update state manager to change all current states in the case when Error\NoData is executed as Ok\Nomal (#68142)	2 years ago
Yuri Tseretyan	78fc3bcdf4	Alerting: Fix state manager to not keep datasource_uid and ref_id labels in state after Error (#72216)	2 years ago
George Robinson	594c851d4b	Alerting: Add duration to saving alert states done (#70844)	2 years ago
George Robinson	7edbe72483	Alerting: Support concurrent queries for saving alert instances (#70525) ... This commit adds support for concurrent queries when saving alert instances to the database. This is an experimental feature in response to some customers experiencing delays between rule evaluation and sending alerts to Alertmanager, resulting in flapping. It is disabled by default.	2 years ago
George Robinson	8a13ee3cd4	Alerting: Add debug logs when saving instances is finished (#70447)	2 years ago
Santiago	b0881daf23	Alerting: Use URLs in image annotations (#66804) ... * use tokens or urls in image annotations * improve tests, fix some comments * fix empty tokens * code review changes, check for url before checking for token (support old token formats)	2 years ago
Matthew Jacobson	63187fae0c	Alerting: Remove and revert flag alertingBigTransactions (#65976) ... * Alerting: Remove and revert flag alertingBigTransactions This is a partial revert of #56575 and a removal of the `alertingBigTransactions` flag. Real-word use has seen no clear performance incentive to maintain this flag. Lowered db connection count came at the cost of significant increase in CPU usage and query latency. * Fix lint backend * Removed last bits of alertingBigTransactions --------- Co-authored-by: Armand Grillet <2117580+armandgrillet@users.noreply.github.com>	2 years ago
Yuri Tseretyan	622c23716a	Alerting: Use logger with context in the state cache (#65663)	2 years ago
Serge Zaitsev	0beb768427	Chore: Remove result fields from ngalert (#65410) ... * remove result fields from ngalert * remove duplicate imports	2 years ago
Alexander Weaver	a31672fa40	Alerting: Create new state history "fanout" backend that dispatches to multiple other backends at once (#64774) ... * Rename RecordStatesAsync to Record * Rename QueryStates to Query * Implement fanout writes * Implement primary queries * Simplify error joining * Add test for query path * Add tests for writes and error propagation * Allow fanout backend to be configured * Touch up log messages and config validation * Consistent documentation for all backend structs * Parse and normalize backend names more consistently against an enum * Touch-ups to documentation * Improve clarity around multi-record blocking * Keep primary and secondaries more distinct * Rename fanout backend to multiple backend * Simplify config keys for multi backend mode	2 years ago
Yuri Tseretyan	05bf241952	Alerting: Update state manager to return StateTransitions when Delete or Reset (#62264) ... * update Delete and Reset methods to return state transitions this will be used by notifier code to decide whether alert needs to be sent or not. * update scheduler to provide reason to delete states and use transitions * update FromAlertsStateToStoppedAlert to accept StateTransition and filter by old state * fixup * fix tests	3 years ago
Alex Moreno	531b439cf1	Alerting: Add alert pausing feature (#60734) ... * Add field in alert_rule model, add state to alert_instance model, and state to eval * Remove paused state from eval package * Skip paused alert rules in scheduler * Add migration to add is_paused field to alert_rule table * Convert to postable alerts only if not normal, pernding, or paused * Handle paused eval results in state manager * Add Paused state to eval package * Add paused alerts logic in scheduler * Skip alert on scheduler * Remove paused status from eval package * Apply suggestions from code review Co-authored-by: George Robinson <george.robinson@grafana.com> * Remove state * Rethink schedule and manager for paused alerts * Change return to continue * Remove unused var * Rethink alert pausing * Paused alerts storing annotations * Only add one state transition * Revert boolean method renaming refactor * Revert take image refactor * Make registry errors public * Revert method extraction for getting a folder title * Revert variable renaming refactor * Undo unnecessary changes * Revert changes in test * Remove IsPause check in PatchPartiLAlertRule function * Use SetNormal to set state * Fix text by returning to old behaviour on alert rule deletion * Add test in schedule_unit_test.go to test ticks with paused alerts * Add coment to clarify usage of context.Background() * Add comment to clarify resetStateByRuleUID method usage * Move rule get to a more limited scope * Update pkg/services/ngalert/schedule/schedule.go Co-authored-by: George Robinson <george.robinson@grafana.com> * rum gofmt on pkg/services/ngalert/schedule/schedule.go * Remove defer cancel for context * Update pkg/services/ngalert/models/instance_test.go Co-authored-by: Santiago <santiagohernandez.1997@gmail.com> * Update pkg/services/ngalert/models/testing.go Co-authored-by: Santiago <santiagohernandez.1997@gmail.com> * Update pkg/services/ngalert/schedule/schedule_unit_test.go Co-authored-by: Santiago <santiagohernandez.1997@gmail.com> * Update pkg/services/ngalert/schedule/schedule_unit_test.go Co-authored-by: Santiago <santiagohernandez.1997@gmail.com> * Update pkg/services/ngalert/models/instance_test.go Co-authored-by: Santiago <santiagohernandez.1997@gmail.com> * skip scheduler rule state clean up on paused alert rule * Update pkg/services/ngalert/schedule/schedule.go Co-authored-by: Santiago <santiagohernandez.1997@gmail.com> * Fix mock in test * Add (hopefully) final suggestions * Use error channel from recordAnnotationsSync to cancel context * Run make gen-cue * Place pause alert check in channel update after version check * Reduce branching un update channel select * Add if for error and move code inside if in state manager ResetStateByRuleUID * Add reason to logs * Update pkg/services/ngalert/schedule/schedule.go Co-authored-by: George Robinson <george.robinson@grafana.com> * Do not delete alert rule routine, just exit on eval if is paused * Reduce branching and create-close a channel to avoid deadlocks * Separate state deletion and state reset (includes history saving) * Add current pause state in rule route in scheduler * Split clearState and bring errCh closer to RecordStatesAsync call * Change rule to ruleMeta in RecordStatesAsync * copy state to be able to modify it * Add timeout to context creation * Shorten the timeout * Use resetState is rule is paused and deleteState if rule is not paused * Remove Empty state reason * Save every rule change in historian * Add tests for DeleteStateByRuleUID and ResetStateByRuleUID * Remove useless line * Remove outdated comment Co-authored-by: George Robinson <george.robinson@grafana.com> Co-authored-by: Santiago <santiagohernandez.1997@gmail.com> Co-authored-by: Armand Grillet <2117580+armandgrillet@users.noreply.github.com>	3 years ago
Alexander Weaver	046a9bb7c1	Alerting: Copy rule definitions into state history (#62032) ... * Copy rules instead of accepting pointer * Deep-copy the rule, for even more guarantees * Create struct just for needed fields * Move RuleMeta to historian/model package, iron out package dependencies * Move tests for dash ID parsing to model package along with code	3 years ago
Denis Limarev	e6dee8a723	Perfomance: Preallocate slices (#61580)	3 years ago
Yuri Tseretyan	9d57b1c72e	Alerting: Do not persist noop transition from Normal state. (#61201) ... * add feature flag `alertingNoNormalState` * update instance database to support exclusion of state in list operation * do not save normal state and delete transitions to normal * update get methods to filter out normal state	3 years ago
Yuri Tseretyan	86b5fbbf60	Alerting: Introduce state manager config structure (#61249)	3 years ago
Yuri Tseretyan	4d989860fb	Alerting: Fix conversion of alert state from db state during manager warmup (#60933)	3 years ago
George Robinson	6359dab040	Alerting: Change resultError in preparation for supporting ForError duration (#59894)	3 years ago
Yuri Tseretyan	abb49d96b5	Alerting: update state manager to return StateTransition instead of State (#58867) ... * improve test for stale states * update state manager return StateTransition * update scheduler to accept state transitions	3 years ago
Yuri Tseretyan	a85adeed96	Alerting: Update state history service to filter states transitions (#58863) ... * rename the method to better reflect its behavior * make historian filter transition on itself * call historian with all changes	3 years ago
Yuri Tseretyan	28d39d35fd	Alerting: Update state manager to save state transitions in one batch (#58358) ... * change stale results handler to not update database but return transitions * save all transitions in one call	3 years ago
George Robinson	c5ae1bcfe0	Alerting: Fix logging pointer address of DashboardUID and PanelID variables (#58539)	3 years ago
Alexander Weaver	2bfdda5b68	Alerting: Break dependency between state and image packages (#58381) ... * Refactor state and manager to not depend directly on image interface * Move generic errors to models package * Move NotAvailableImageService to state as its only references are in state tests * Move NoopImageService to state package * Move mock to state package * Fix linter error * Fix comment styling * Fix a couple added references introduced by rebase * Empty commit to kick build	3 years ago
George Robinson	1290951b65	Alerting: Small improvements to staleResultsHandler (#58007)	3 years ago
Yuri Tseretyan	3621cf5a12	Alerting: Update handling of stale state (#58276) ... * delete all stale states in one lock * do not use touched states to detect stale rely only on LastEvaluationTime maintained correctly * fix tests to use correct eval time * delete unused method	3 years ago
Yuri Tseretyan	623de12e35	Alerting: Create AlertInstanceKey in one place (#58278) ... * use method GetAlertInstanceKey * do not add key if error	3 years ago
Yuri Tseretyan	f9c88e72ae	Alerting: Update saveAlertStates in state manager to not return results (#58279)	3 years ago