333 Commits (bccc980b902987fc9a2707deca766412b47bbe2a)

Author	SHA1	Message	Date
Vadim Stepanov	bccc980b90	Alerting: Notifiication history (#107644) ... * Add unified_alerting.notification_history to ini files * Parse notification history settings * Move Loki client to a separate package * Loki client: add params for metrics and traces * add NotificationHistorian * rm writeDuration * remove RangeQuery stuff * wip * wip * wip * wip * pass notification historian in tests * unify loki settings * unify loki settings * add test * update grafana/alerting * make update-workspace * add feature toggle * fix configureNotificationHistorian * Revert "add feature toggle" This reverts commit de7af8f7 * add feature toggle * more tests * RuleUID * fix metrics test * met.Info.Set(0)	4 days ago
Serge Zaitsev	f66a693438	Chore: Rename integration tests to follow the common convention (#105987) ... * automatically rename integration tests to follow the common convention * name tests differently * alter column type to bigint * update another column to bigint * add another alter * fix subquery for mysql	3 weeks ago
Alexander Akhmetov	abbae41f60	Alerting: Sanitize Prometheus state history labels before writing (#107181)	4 weeks ago
Alexander Akhmetov	478f9bf597	Alerting: Emit metrics from prometheus state history backend (#107121) ... Alerting: Emit metrics from prometheus historian backend	4 weeks ago
Alexander Akhmetov	ac832c157e	Alerting: Maintain endsAt for missing alert instances that are not stale yet (#107011)	1 month ago
Alexander Akhmetov	ad683f83ff	Alerting: Add state history backend to write ALERTS metric (#104361) ... **What is this feature?** This PR implements a new Prometheus historian backend that allows Grafana alerting to write alert state history as Prometheus-compatible `ALERTS` metrics to remote Prometheus-compatible data sources. The metric includes a few additional labels: * `grafana_alertstate`: Grafana's full alert state, more granular than Prometheus. * `grafana_rule_uid`: Grafana's alert rule UID. Grafana states are included in the `grafana_alertstate` label also mapped to Prometheus-compatible `alertstate` values: | Grafana alert state | `alertstate` | `grafana_alertstate` | |---------------------|-----------------------|-----------------------| | `Alerting` | `firing` | `alerting` | | `Recovering` | `firing` | `recovering` | | `Pending` | `pending` | `pending` | | `Error` | `firing` | `error` | | `NoData` | `firing` | `nodata` | | `Normal` | _(no metric emitted)_ | _(no metric emitted)_ |	1 month ago
Stephanie Hingtgen	a8886ad5ec	Annotations: Use dashboard uids instead of dashboard ids (#106676)	1 month ago
Fayzal Ghantiwala	85df859589	Alerting: Correctly persist FiredAt in SyncRuleStatePersister (#106658) ... Correctly persist FiredAt	1 month ago
Alexander Akhmetov	1a75787e74	Alerting: Send notifications immediately on Error|NoData -> Normal transitions (#106421)	1 month ago
Alexander Akhmetov	3bb4c92028	Alerting: Fix resolved notifications for same-label Error to Normal transitions (#106210) ... What is this feature? Ensures that resolved notifications are sent when alert states transition from Error to Normal after the configured number of evaluation intervals: Missing series evaluations to resolve. Why do we need this feature? Before this change, when an alert was transitioning from Error to Normal, in case when the labels on the new Normal alert instance are the same, Grafana would not send resolved notifications for the Error alert state. The alert would be resolved after a few evaluation intervals automatically in the alertmanager, following the endsAt. With this change the resolved notification is sent after the configured number of evaluation intervals: Missing series evaluations to resolve.	1 month ago
Moustafa Baiou	0ce086bd2e	Alerting: Ensure errors cleared when Alerting after error (#105246) ... When a rule configured with `ExecErrState` state of `Alerting`, has an instance which is Alerting then has a data source error, then successfully evaluates and continues to be Alerting, the cached instance keeps the error cached until it is no longer firing. This is unexpected and leads to misleading results.	2 months ago
Alexander Akhmetov	4cde79e802	Alerting: Clean up join errors code (#106243)	2 months ago
Alexander Akhmetov	eae77aa695	Alerting: Resend alerts for states that are missing in the eval results (#105965) ... What is this feature? This PR fixes the MissingSeriesEvalsToResolve behavior when it's set to more than 4 evaluation intervals. Why do we need this feature? The MissingSeriesEvalsToResolve setting was not working correctly due to alerts being auto-resolved by Alertmanager after 4 evaluation intervals (via the endsAt field). Before we had deleteStaleStatesFromCache method that was returning only stale states that had to be resolved. Non-stale states for which the current evaluation does not have a series never had endsAt updated and were never resend to the Alertmanager, so they were automatically resolved after 4 evaluations regardless of the setting. The new processMissingSeriesStates returns state for each missing series on every evaluation, and resolves the stale ones. This guarantees that alerts without series still alert for the configured number of evaluations.	2 months ago
Alexander Akhmetov	faeddf334a	Alerting: Fix $value type when single data source is queried (#106080)	2 months ago
Fayzal Ghantiwala	589046bcdc	Alerting: Persist alert instance FiredAt field (#105927) ... * Persist alert instance fired at * Update protos and tests	2 months ago
Alexander Akhmetov	0743689d42	Alerting: Add recovering state to the grafana_alerting_alerts metric (#104380)	3 months ago
Fayzal Ghantiwala	3a054d5e00	Alerting: Add FiredAt field to State (#104046) ... * Add FiredAt field to the State * Update featuretoggle files * Fix lint errors * Fix test compilation * Remove random print line + formatting * Address PR comments	3 months ago
Alexander Akhmetov	acfd998fa6	Alerting: Send resolved notifications immediately for state deleted states (#103996) ... What is this feature? Send resolved notifications not only when an alert state becomes stale (series is missing) and transitions from Alerting to Normal, but also from Error, NoData and Recovering. Why do we need this feature? Previously, when an alert state became stale or was deleted, it would transition to Normal but wouldn't trigger resolved notifications to the Alertmanager. This meant we relied on the Alertmanager to send resolved notifications when the alert expires. However, if the Alertmanager state is lost, these resolved notifications would never be sent, leaving users with firing alerts in their notification channels. This PR ensures that any transition from a firing state (Alerting, Error, NoData, Recovering) to Normal triggers a resolved notification.	3 months ago
Mariell Hoversholm	757be6365a	CI: Bump golangci-lint to 2.0.2 (#103572)	3 months ago
Alexander Akhmetov	b49c532999	Alerting: Fix state transition from Recovering back to Alerting (#103286) ... What is this feature? This PR fixes a state transition issue where alerts transitioning from the Recovering state back to the Alerting state incorrectly entered the Pending state first if the rule had a For duration configured. Why do we need this feature? When an alert goes from Alerting to Recovering (when using the Keep firing for) and then back to Alerting, the existing logic would incorrectly put the alert into Pending state while it should be alerting and still sending notifications to the Alertmanager.	4 months ago
maicon	d8c5c2d3b8	K8s: Folders: Modify GetChildren to return only Folder References (#103072) ... * Return FolderReference instead of Folder on GetChildren Signed-off-by: Maicon Costa <maiconscosta@gmail.com> --------- Signed-off-by: Maicon Costa <maiconscosta@gmail.com>	4 months ago
Alexander Akhmetov	c54da8f955	Alerting: Make $value return the query value in case when a single datasource is used (#102301) ... What is this feature? This PR changes the behavior of the $value and .Value variables in alerting templating to be more compatible with Prometheus templating. When a single datasource is used in the alerting rule, these variables will now return the numeric value from the query instead of the evaluation string. Why do we need this feature? It makes Grafana templating more compatible with Prometheus templates. In Prometheus, $value returns the numeric value of the query, but in Grafana it's the evaluation string: [ var='A' labels={instance=instance1} value=81.234 ]. This is because in Grafana multiple datasources can be used in the alert rule, and it's not always possible to get a single value. This change makes Grafana's behavior consistent with Prometheus when a single datasource is used, and in case when multiple datasources are used in the query, it keeps the old behaviour. Both $value and .Value are not recommended to use (documentation), and it's better to use .Values instead.	4 months ago
Alexander Akhmetov	695ac91290	Alerting: Add backend support for keep_firing_for (#100750) ... What is this feature? This PR introduces a new alert rule configuration option, keep_firing_for (Prometheus documentation). keep_firing_for prevents alerts from resolving immediately after the alert condition returns to normal. Instead, they transition into a "Recovering" state and are not considered resolved by the Alertmanager. Once the recovery period ends (or after the next evaluation if it is bigger than keep_firing_for), the alert transitions to "Normal" if it doesn't start alerting again: Before +----------+ +----------+ | Alerting |---->| Normal | +----------+ +----------+ ----- After +----------+ +------------+ +----------+ | Alerting |----->| Recovering |---->| Normal | +----------+ +------------+ +----------+ Why do we need this feature? This feature prevents flapping alerts by adding a recovery period. This helps avoid false resolutions caused by brief alert	4 months ago
Yuri Tseretyan	e30034a42a	Alerting: Remove feature flag `alertingNoDataErrorExecution` (#102156) ... * remove feature flag * remove feature flag in state manager * make sure no data with empty results is handled Signed-off-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com> --------- Signed-off-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>	4 months ago
Alexander Akhmetov	7dd6f52630	Alerting: Add MissingSeriesEvalsToResolve option to the AlertRule (#101184)	4 months ago
Yuri Tseretyan	67b44ad22a	Alerting: Fix state reason (#101530) ... --------- Signed-off-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>	5 months ago
Matthew Jacobson	b78a63b0ad	Alerting: Use new image TokenProvider and send image url in annotation (#99989) ... * Send new annotation containing image url * Use new image TokenProvider with TokenStore New abstraction GetImage no longer needs to support parsing both token and url from annotations, as remote AM will use the new URLProvider. Instead, we use the new generic TokenProvider and give it a TokenStore backed by the grafana database. That means we revert back to always using token simplifying code and security considerations. * Upgrade grafana/alerting to merged commit SHA	5 months ago
Yuri Tseretyan	9dd75aee32	Alerting: Refactor State Transition (part 2 of n) (#99985) ... * split create to create and patch and move to state patch will be refactored further * move setNextState to state transition * move tests * split tests for patch function	5 months ago
Yuri Tseretyan	807f94b2c7	Alerting: Remove feature toggle alertingNoNormalState (#99905)	6 months ago
Alexander Akhmetov	d6c1e3bb45	Alerting: Use org store to read organization IDs (#99938)	6 months ago
Alexander Akhmetov	f45265b5f7	Alerting: Read from both proto and simple DB instance stores on startup (#99855)	6 months ago
Alexander Akhmetov	a0bf9202f5	Alerting: Clear the state cache when the alert routine stops (#99681)	6 months ago
Alexander Akhmetov	cb43f4b696	Alerting: Add compressed protobuf-based alert state storage (#99193)	6 months ago
Alexander Akhmetov	651430e34a	Alerting: Add sync state persister to save entire state of the rule (#96628)	6 months ago
Yuri Tseretyan	d025523a8b	Alerting: Log reason for taking image. (#99036)	6 months ago
Matthew Jacobson	fc90a446c6	Alerting: Ensure long-lived repeat alerts keep images after 24h expiry (#98993) ... Ensures we retake images after expiration on long-lived repeat alerts. Otherwise, logs would show "Image not found in database" and notifications would cease to contain an image after 24h of continuous firing.	6 months ago
Yuri Tseretyan	4f62c8a160	Alerting: Update state manager to take image only once per rule evaluation (#98289) ... * add test * update state manager to take image only once per rule evaluation process execution * update test	6 months ago
Alexander Akhmetov	1f8f9a45d7	Alerting: Add state_periodic_save_batch_size config option (#98019) ... * Alerting: Add state_periodic_save_batch_size config option --------- Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>	7 months ago
Yuri Tseretyan	420db99d16	Alerting: Update state manager to have immutable state in cache (#95985) ... * create a new state and set at the end * propagate labels datasource_uid and ref_id from current state if it's error * copy the state when apply to all	8 months ago
Alexander Akhmetov	580d073b96	Alerting: Add context to the logger in state manager Warm (#96228)	8 months ago
Alexander Akhmetov	305123df91	Alerting: Keep state manager cache during cache warm-up (#95727) ... * Alerting: Keep state manager cache during cache warm-up Instead of overwriting the state manager cache during warm-up, we update the data in the cache if it is not there yet. If the cache already contains a state entry with the same key, we do not overwrite it.	9 months ago
Will Browne	25abd57029	Plugins: Update to latest go plugin SDK (0.256.0) (#95065) ... * update to latest go plugin SDK * make update-workspace * update alerting tests	9 months ago
Alexander Akhmetov	d0481bb568	Alerting: Refactor state manager Warm method to accept instance store as an argument (#95098)	9 months ago
Alexander Akhmetov	9d182986f1	Alerting: make StatePersister more configurable to support custom rule-level state persisters (#94590)	9 months ago
Alexander Akhmetov	0a4e6ff86b	Alerting: Add SaveAlertInstancesForRule instance store method (#94505) ... Alerting: Add SaveAlertInstancesForRule method to the InstanceStore interface	9 months ago
Alexander Weaver	393faa8732	Alerting: Move rule evaluation status logic out of prometheus API and into scheduler (#89141) ... * Add health fields to rules and an aggregator method to the scheduler * Move health, last error, and last eval time in together to minimize state processing * Wire up a readonly scheduler to prom api * Extract to exported function * Use health in api_prometheus and fix up tests * Rename health struct to status * Fix tests one more time * Several new tests * Handle inactive rules * Push state mapping into state manager * rename to StatusReader * Rectify cyclo complexity rebase * Convert existing package local status implementation to models one * fix tests * undo RuleDefs rename	10 months ago
Steve Simpson	acb051b314	Alerting: Fix logging for failed annotations writing. (#93856)	10 months ago
Alexander Akhmetov	d32e1e009b	Alerting: Update prometheus/client_golang to v1.20 (#92070) ... Update prometheus/client_golang to v1.20	11 months ago
Alexander Akhmetov	c7fdf8ce70	Alerting: Add error to annotations on data source errors (#91594)	11 months ago
Matthew Jacobson	a397bca02e	Alerting: Fix panic with nil annotations & Nodata=alerting/ok/keep (#91506)	12 months ago