grafana

Commit Graph

Author	SHA1	Message	Date
Yuriy Tseretyan	6e1e4a4215	Alerting: Update DbStore to use disabled orgs from the config (#52156 ) * update DbStore to use UnifiedAlerting settings * remove disabled orgs from scheduler and use config in db store instead * remove test	3 years ago
Yuriy Tseretyan	a7509ba18b	Alerting: rule evaluation loop's update channel to provide version (#52170 ) * handler for update message in rule evaluation routine ignores the message if its version greater or equal. * replace messages to update the channel if it is not empty	3 years ago
Yuriy Tseretyan	e5e8747ee9	Alerting: Update state manager to accept reserved labels (#52189 ) * add tests for cache getOrCreate * update ProcessEvalResults to accept extra lables * extract to getRuleExtraLabels * move populating of constant rule labels to extra labels	3 years ago
Yuriy Tseretyan	429ed4b4ee	remove unused orgStore from scheduler (#52157 )	3 years ago
Yuriy Tseretyan	0d4c503d3d	update Evaluator interface to accept context (#52151 )	3 years ago
Yuriy Tseretyan	554ebd647b	Alerting: Refactor Evaluator (#51673 ) * AlertRule to return condition * update ConditionEval to not return an error because it's always nil * make getExprRequest private * refactor executeCondition to just converter and move execution to the ConditionEval as this makes code more readable. * log error if results have errors * change signature of evaluate function to not return an error	3 years ago
Yuriy Tseretyan	a6b1090879	Alerting: refactor scheduler and separate notification logic (#48144 ) * Introduce AlertsRouter in the sender package, and move all fields and methods related to notifications out of the scheduler to this router. * Introduce a new interface AlertsSender in the schedule package and replace calls of anonymous function `notify` inside the ruleRoutine to calling methods of that interface. * Rename interface Scheduler in api package to ExternalAlertmanagerProvider, and replace scheduler with AlertRouter as struct that implements the interface.	3 years ago
Matthew Jacobson	28dd413c1d	Alerting: Add config disabled_labels to disable reserved labels (#51832 ) * Alerting: Add config disabled_labels to disable reserved labels [unified_alerting.reserved_labels] disabled_labels * Replace IsGrafanaFolderDisabled with more generic IsReservedLabelDisabled * Simplify SchedulerCfg by including UnifiedAlertingSettings	3 years ago
Yuriy Tseretyan	94e709fdcb	Alerting: Simplify eval.Evaluator interface (#51463 ) * remove ExpressionService from argument list of Evaluator's methods	3 years ago
Yuriy Tseretyan	157c12211d	Alerting: State manager to use tick time to determine stale states (#50991 ) * use correct stale timestamp * calculate stale using tick time instead of time.now * remove unused dependency on sql store	3 years ago
Matthew Jacobson	5dee2ed24c	Alerting: Add first Grafana reserved label grafana_folder (#50262 ) * Alerting: Add first Grafana reserved label g_label g_label holds the title of the folder container the alert. The intention of this label is to use it as part of the new default notification policy groupBy. * Add nil check on updateRule labels map * Disable gocyclo lint on schedule.ruleRoutine will remove later in a separate refactoring PR to reduce complexity. * Address doc suggestions * Update g_folder for rules in folder when folder title changes * Remove global bus in FolderService * Modify tests to fit new common g_folder label * Add changelog entry * Fix merge conflicts * Switch GrafanaReservedLabelPrefix from `g_` to `grafana_`	3 years ago
gotjosh	c59938b235	Alerting: Schedule Alert rules metric tracking (#50415 ) * Alerting: Schedule Alert rules metric tracking Change the record of metrics from one place to two as an attempt to have a semi-accurate record.	3 years ago
Yuriy Tseretyan	a89d4a5be7	Alerting: Scheduler to drop ticks if a rule's evaluation is too slow (#48885 ) * drop ticks if evaluation of a rule is too slow. * add metric schedule_rule_evaluations_missed_total	3 years ago
gotjosh	0cde283505	Alerting: Logs should not be capitalized and the errors key should be "err" (#50333 ) * Alerting: decapitalize log lines and use "err" as the key for errors Found using (logger\|log).(Warn\|Debug\|Info\|Error)\([A-Z] and (logger\|log).(Warn\|Debug\|Info\|Error)\(.+"error"	3 years ago
George Robinson	c83f84348c	Alerting: Fix database unavailable removes rules from scheduler (#49874 )	3 years ago
Yuriy Tseretyan	c8d891785d	Alerting: Ticker to support stopping (#48142 ) * add stop for ticker * stop ticker when scheduler stops * stop ticker when legacy engine stops	3 years ago
Joe Blubaugh	1cc034d960	Alerting: Add a "Reason" to Alert Instances to show underlying cause of state. (#49259 ) This change adds a field to state.State and models.AlertInstance that indicate the "Reason" that an instance has its current state. This helps us account for cases where the state is "Normal" but the underlying evaluation returned "NoData" or "Error", for example. Fixes #42606 Signed-off-by: Joe Blubaugh <joe.blubaugh@grafana.com>	3 years ago
Yuriy Tseretyan	99156b40bd	Alerting: Move alertRuleRegistry to its own file (#48890 ) * move alertRuleRegistry to its own file * move tests to separate file	3 years ago
Yuriy Tseretyan	dc33e09b24	simplify getting a slice of keys (#48889 )	3 years ago
Yuriy Tseretyan	75ba4e98c6	Alerting: Remove unused features from ticker + metric + tests (#47828 ) * remove not used code: - remove offset in ticket because it is not used - remove unused ticker and scheduler methods * use duration for interval * add metrics grafana_alerting_ticker_last_consumed_tick_timestamp_seconds, grafana_alerting_ticker_next_tick_timestamp_seconds, grafana_alerting_ticker_interval_seconds	3 years ago
Yuriy Tseretyan	e94d0c1b96	Alerting: update rule test endpoints to respect data source permissions (#47169 ) * make eval.Evaluator an interface * inject Evaluator to TestingApiSrv * move conditionEval to RouteTestGrafanaRuleConfig because it is the only place where it is used * update rule test api to check data source permissions	3 years ago
Yuriy Tseretyan	e20d157a9b	Alerting: rules delete API to check data source authorization (#46906 ) * merge RuleSrv rule delete methods * remove unused store methods * implement delete by uid for fake store * add scheduler mock * implement tests for RouteDeleteAlertRules	3 years ago
George Robinson	f87bfdf2ff	Update comment for scheduler_behind_seconds metric (#45918 )	3 years ago
George Robinson	6cccbb5a09	Fix incorrect metric values for scheduler_behind_seconds (#45830 )	3 years ago
George Robinson	2ca79ca0c7	Rename evalCtx to avoid confusion with context.Context (#45144 )	3 years ago
George Robinson	67a3e1d6fd	Add context.Context to InstanceStore (#45049 )	3 years ago
George Robinson	a9399ab3cd	Alerting: Add context.Context to RuleStore (#45004 ) Alerting: Add context.Context to RuleStore	3 years ago
George Robinson	9df43abbb5	Fix evaluation of alert rules for datasources with custom headers (#44862 ) * Fix evaluation of alert rules for datasources with custom headers * Fix unit tests * Fix integration tests * Evaluator fields should be package private	3 years ago
Santiago	04d93751b8	Alerting: send alerts to external, internal, or both alertmanagers (#40341 ) * (WIP) send alerts to external, internal, or both alertmanagers * Modify admin configuration endpoint, update swagger docs * Integration test for admin config updated * Code review changes * Fix alertmanagers choice not changing bug, add unit test * Add AlertmanagersChoice as enum in swagger, code review changes * Fix API and tests errors * Change enum from int to string, use 'SendAlertsTo' instead of 'AlertmanagerChoice' where necessary * Fix tests to reflect last changes * Keep senders running when alerts are handled just internally * Check if any external AM has been discovered before sending alerts, update tests * remove duplicate data from logs * update comment * represent alertmanagers choice as an int instead of a string * default alertmanagers choice to all alertmanagers, test cases * update definitions and generate spec	3 years ago
George Robinson	5e2280ceee	Add metrics to ngalert scheduler (#44602 ) This pull request adds metrics to the ngalert scheduler so we can see how long it takes to evaluate a tick.	3 years ago
Yuriy Tseretyan	ed5c664e4a	Alerting: Stop firing of alert when it is updated (#39975 ) * Update API to call the scheduler to remove\update an alert rule. When a rule is updated by a user, the scheduler will remove the currently firing alert instances and clean up the state cache. * Update evaluation loop in the scheduler to support one more channel that is used to communicate updates to it. * Improved rule deletion from the internal registry. * Move alert rule version from the internal registry (structure alertRuleInfo) closer rule evaluation loop (to evaluation task structure), which will make the registry values immutable. * Extract notification code to a separate function to reuse in update flow.	3 years ago
Yuriy Tseretyan	1a762083d7	Alerting: make alert rule routine evaluation control be thread-safe (#41220 ) * change registry.delete to return deleted struct * use pointer to alertRuleInfo instead copying. * do not access evaluation channel when routine is stopped * remove stopCh and use context cancellation * do not return ctx.Err when channel is cancelled because it cancels all other routines * make alertRuleInfo fields and functions package private	3 years ago
George Robinson	1e5b0e64ac	Alerting: Add comments to ScheduleService interface (#42228 )	4 years ago
Marcus Efraimsson	baab021fec	Chore: Refactor usage of legacy data contracts (#41218 ) Refactor usage of legacy data contracts. Moves legacy data contracts to pkg/tsdb/legacydata package. Refactor pkg/expr to be a proper service/dependency that can be provided to wire to remove some unneeded dependencies to SSE in ngalert and other places. Refactor pkg/expr to not use the legacydata,RequestHandler and use backend.QueryDataHandler instead.	4 years ago
ying-jeanne	54de1078c8	remove the global log error/warn etc functions (#41404 ) * remove the global log error/warn etc functions and use request context logger whenever possible	4 years ago
Yuriy Tseretyan	8a88caaa71	Alerting: Rule evaluation loop refactoring (#40238 ) * move evaluation function out of loop * extract updateRule function * isolate alertRule change. update returns new rule and the evaluation accepts the rule as argument * extract retry loop into a function * add function wide log context. * refactor metrics + add tests + replace timeNow with schedule.clock	4 years ago
gotjosh	f33f098b73	Alerting: Disable flaky assertion in TestSendingToExternalAlertmanager_WithMultipleOrgs (#40197 ) * Alerting: Disable flaky assertion in TestSendingToExternalAlertmanager_WithMultipleOrgs * Update pkg/services/ngalert/schedule/schedule_unit_test.go Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com> * Update pkg/services/ngalert/schedule/schedule_unit_test.go Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com> Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>	4 years ago
idafurjes	2759b16ef5	Chore: Add context for dashboards (#39844 ) * Add context for dashboards * Remove GetDashboardCtx * Remove ctx.TODO	4 years ago
Yuriy Tseretyan	526961f298	Alerting: rule evaluation loop to not access multiorg Alertmanager if no alerts to add (#39872 )	4 years ago
Yuriy Tseretyan	2b4e51f478	Alerting: Parse App URL only once (#39855 )	4 years ago
Sofia Papagiannaki	012d4f0905	Alerting: Remove `ngalert` feature toggle and introduce two new settings for enabling Grafana 8 alerts and disabling them for specific organisations (#38746 ) * Remove `ngalert` feature toggle * Update frontend Remove all references of ngalert feature toggle * Update docs * Disable unified alerting for specific orgs * Add backend tests * Apply suggestions from code review Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Disabled unified alerting by default * Ensure backward compatibility with old ngalert feature toggle * Apply suggestions from code review Co-authored-by: gotjosh <josue@grafana.com>	4 years ago
Sofia Papagiannaki	f6f3a54742	Alerting: tune rule evaluation via configuration (#35623 ) * Alerting: Configure max evaluation retries * Alerting: Enforce minimum rule evaluation interval * Alerting: Disable rule evaluation from configuration * Update docs * Alerting: Configure rule evaluation timeout * Move options on unified_alerting config section * Apply suggestions from code review Co-authored-by: gotjosh <josue@grafana.com>	4 years ago
gotjosh	a2f4344bf2	Alerting: Refactor & fix unified alerting metrics structure (#39151 ) * Alerting: Refactor & fix unified alerting metrics structure Fixes and refactors the metrics structure we have for the ngalert service. Now, each component has its own metric struct that includes the JUST the metrics it uses. Additionally, I have fixed the configuration metrics and added new metrics to determine if we have discovered and started all the necessary configurations of an instance. This allows us to alert on `grafana_alerting_discovered_configurations - grafana_alerting_active_configurations != 0` to know whether an alertmanager instance did not start successfully.	4 years ago
David Parrott	7fbeefc090	Alerting: create wrapper for Alertmanager to enable org level isolation (#37320 ) Introduces org-level isolation for the Alertmanager and its components. Silences, Alerts and Contact points are not separated by org and are not shared between them. Co-authored with @davidmparrott and @papagian	4 years ago
gotjosh	f3f3fcc727	Alerting: Introduces `/api/v1/ngalert/alertmanagers` to expose discovered and dropped Alertmanager(s) (#37632 ) * Alerting: Expose discovered and dropped Alertmanagers Exposes the API for discovered and dropped Alertmanagers. * make admin config poll interval configurable * update after rebase * wordsmith * More wordsmithing * change name of the config * settings package too	4 years ago
gotjosh	f83cd401e5	Alerting: Send alerts to external Alertmanager(s) (#37298 ) * Alerting: Send alerts to external Alertmanager(s) Within this PR we're adding support for registering or unregistering sending to a set of external alertmanagers. A few of the things that are going are: - Introduce a new table to hold "admin" (either org or global) configuration we can change at runtime. - A new periodic check that polls for this configuration and adjusts the "senders" accordingly. - Introduces a new concept of "senders" that are responsible for shipping the alerts to the external Alertmanager(s). In a nutshell, this is the Prometheus notifier (the one in charge of sending the alert) mapped to a multi-tenant map. There are a few code movements here and there but those are minor, I tried to keep things intact as much as possible so that we could have an easier diff.	4 years ago
gotjosh	442a6677fc	Alerting: Refactor `Run` of the scheduler (#37157 ) * Alerting: Refactor `Run` of the scheduler A bit of a refactor to make the diff easier to read for supporting external Alertmanagers. We'll introduce another routine that checks the database for configuration and spawns other routines accordingly. * Block the wait. * Fix test	4 years ago
gotjosh	a86ad1190c	Alerting: Refactor state manager as a dependency (#36513 ) * Alerting: Refactor state manager as a dependency Within the scheduler, the state manager was being passed around a certain number of functions. I've introduced it as a dependency to keep the "service" interfaces as clean and homogeneous as possible. This is relevant, because I'm going to introduce live reload of these components as part of my next PR and it is better if dependencies are self-contained. * remove unused functions * Fix a few more tests * Make sure the `stateManager` is declared before the schedule	4 years ago
Ganesh Vernekar	dcd4bf1615	Alerting: Fill the empty GeneratorURL (#35740 ) Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	4 years ago
David Parrott	7a83d1f9ff	Alerting resend delay for sending to notifiers (#34312 ) * adds resend delay to avoid saturating notifier * correct method signatures * pr feedback	4 years ago

1 2

72 Commits (6e1e4a42151ccaf250b925ccc25f396784e77943)