153 Commits (01d274852c389f7c33b22254acb5dec35913ff92)

Author	SHA1	Message	Date
Santiago	01d274852c	Alerting: Add GetFullState method to FileStore (#78701) ... * Alerting: Add GetFullState method to FileStore * make tests compile, create stateStore in NewAlertmanager * return errors instead of logging, accept an arbitrary number of strings * make NewAlertmanager() accept a stateStore	2 years ago
Tania	39754ba2d6	Nested Folders: Wrap create/update operations with transactions (#78000) ... * Nested Folders: Add transaction to create and update methods * Update tests * Make IncreaseVersionForAllRulesInNamespace synchronous * Resolve merge conflicts	2 years ago
Ryan McKinley	f69fd3726b	FeatureToggles: Add context and and an explicit global check (#78081)	2 years ago
Santiago	488a60aee6	Alerting: Rename remote.ExternalAlertmanager to remote.Alertmanager (#76956)	2 years ago
gotjosh	866acbd5ac	Alerting: Move `ExternalAlertmanager` to its own package (#76854) ... * Alerting: Move `ExternalAlertmanager` to its own package We'll avoid import cycles when using components from other packages. In addition to that, I've created an `Options` approach for the multiorg alertmanger to allow us to override how per tenant alertmanagers are created. * switch things around * address review comments * fix references and warnings	2 years ago
Matthew Jacobson	c2efcdde09	Alerting: Fix flaky SQLITE_BUSY when migrating with provisioned dashboards (#76658) ... * Alerting: Move migration from background service run to ngalert init sqlite database write contention between the migration's single transaction and dashboard provisioning's frequent commits was causing the migration to fail with SQLITE_BUSY/SQLITE_BUSY_SNAPSHOT on all retries. This is not a new issue for sqlite+grafana, but the discrepancy between the length of the transactions was causing it to be very consistent. In addition, since a failed migration has implications on the assumed correctness of the alertmanager and alert rule definition state, we cause a server shutdown on error. This can make e2e tests as well as some high-load provisioned sqlite installations flaky on startup. The correct fix for this is better transaction management across various services and is out of scope for this change as we're primarily interested in mitigating the current bout of server failures in e2e tests when using sqlite.	2 years ago
Matthew Jacobson	82f3127e23	Alerting: Move legacy alert migration from sqlstore migration to service (#72702)	2 years ago
Alexander Weaver	f6649d7a97	Revert "Alerting: Remove vendored models in migration service" (#76387) ... Revert "Alerting: Remove vendored models in migration service (#74503)" This reverts commit 6a8649d544.	2 years ago
Matthew Jacobson	6a8649d544	Alerting: Remove vendored models in migration service (#74503) ... This PR replaces the vendored models in the migration with their equivalent ngalert models. It also replaces the raw SQL selects and inserts with service calls. It also fills in some gaps in the testing suite around: - Migration of alert rules: verifying that the actual data model (queries, conditions) are correct 9a7cfa9 - Secure settings migration: verifying that secure fields remain encrypted for all available notifiers and certain fields migrate from plain text to encrypted secure settings correctly e7d3993 Replacing the checks for custom dashboard ACLs will be replaced in a separate targeted PR as it will be complex enough alone.	2 years ago
gotjosh	59694fb2be	Alerting: Don't use a separate collection system for metrics (#75296) ... * Alerting: Don't use a separate collection system for metrics The state package had a metric collection system that ran every 15s updating the values of the metrics - there is a common pattern for this in the Prometheus ecosystem called "collectors". I have removed the behaviour of using a time-based interval to "set" the metrics in favour of a set of functions as the "value" that get called at scrape time.	2 years ago
Steve Simpson	894f420014	Alerting: Pass loggers into SchedulerCfg and ManagerCfg. (#75158)	2 years ago
Will Browne	e855efb13d	Plugins: Move store and plugin dto to pluginsintegration (#74655) ... move store and plugin dto	2 years ago
Yuri Tseretyan	938e26b59f	Alerting: Add new metrics and tracings to state manager and scheduler (#71398) ... * add metrics and tracing to state manager * propagate tracer to state manager * add scheduler metrics * fix backtesting * add test for state metrics * remove StateUpdateCount * update docs * metrics can be null * add tracer to new tests	2 years ago
Yuri Tseretyan	0717ec11d6	Alerting: Update state manager to change all current states in the case when Error\NoData is executed as Ok\Nomal (#68142)	2 years ago
Yuri Tseretyan	6b4a9d73d7	Alerting: Export contact points to check access control action instead legacy role (#71990) ... * introduce a new action "alert.provisioning.secrets:read" and role "fixed:alerting.provisioning.secrets:reader" * update alerting API authorization layer to let the user read provisioning with the new action * let new action use decrypt flag * add action and role to docs	2 years ago
Alexander Weaver	18b910e654	Alerting: Refactor annotation historian to isolate dashboard service dependency (#71689) ... * Refactor annotation historian to isolate dashboard service dependency * Export PanelKey * Don't export parsePanelKey * Remove commented out code	2 years ago
Steve Simpson	21ac224c45	Alerting: Make ImageService public in NGAlert. (#70737)	2 years ago
George Robinson	7edbe72483	Alerting: Support concurrent queries for saving alert instances (#70525) ... This commit adds support for concurrent queries when saving alert instances to the database. This is an experimental feature in response to some customers experiencing delays between rule evaluation and sending alerts to Alertmanager, resulting in flapping. It is disabled by default.	2 years ago
Arati R	6cb1a5e368	Nested folders: Add alert rule counts and deletion to folder registry (#67259) ... * Let alert rule service implement registry service * Add count method to RuleStore interface * Add implementation for deletion of alert rules * Rename uid to folderUID in registry methods * Check forceDeleteRule value for registry deletion * Register alerting store with folder service * Move folder test functions to separate package * Add testing for alert rule counting, deletion * Remove redundant count method * Fix deleteChildrenInFolder signature * Update pkg/services/ngalert/store/alert_rule.go Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com> * Add tests for nested folder deletion * Refactor TestIntegrationNestedFolderService * Add rules store as parameter for alertng provider --------- Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>	2 years ago
Steve Simpson	9effb9a708	Alerting: Allow hooking into request handler functions. (#67000) ... * Alerting: Allow hooking into request handler functions. Adds a facility to AlertNG for hooking into API handlers, allowing the replacement of request handlers for specific paths. One of goals of this approach was to allow hooking as late as possible in the request, e.g. after all middleware has been applied, to simplfiy usage. * Update pkg/services/ngalert/api/hooks.go Co-authored-by: gotjosh <josue.abreu@gmail.com> * Update pkg/services/ngalert/api/hooks.go Co-authored-by: gotjosh <josue.abreu@gmail.com> * Update pkg/services/ngalert/ngalert.go Co-authored-by: gotjosh <josue.abreu@gmail.com> * Fixes to review comments * Fix passing logger in --------- Co-authored-by: gotjosh <josue.abreu@gmail.com>	2 years ago
Alexander Weaver	da4832724e	Alerting: Delete stub for SQL alert state history backend (#65667) ... Delete stub for SQL backend	2 years ago
Alexander Weaver	b2abb63286	Alerting: Introduce proper feature toggles for common state history backend combinations (#65497) ... * define 3 feature toggles for rollout phases * Pass feature toggles along * Implement first feature toggle * Try a different strategy with fall-throughs to specific configurations * Apply toggle overrides once outside of backend composition * Emit log messages when we coerce backends * Run code generator for feature toggle files * Improve wording in flag descs * Re-run generator * Use code-generated constants instead of plain strings * Use converted enum values rather than strings for pre-parsing	2 years ago
Alexander Weaver	a31672fa40	Alerting: Create new state history "fanout" backend that dispatches to multiple other backends at once (#64774) ... * Rename RecordStatesAsync to Record * Rename QueryStates to Query * Implement fanout writes * Implement primary queries * Simplify error joining * Add test for query path * Add tests for writes and error propagation * Allow fanout backend to be configured * Touch up log messages and config validation * Consistent documentation for all backend structs * Parse and normalize backend names more consistently against an enum * Touch-ups to documentation * Improve clarity around multi-record blocking * Keep primary and secondaries more distinct * Rename fanout backend to multiple backend * Simplify config keys for multi backend mode	2 years ago
gotjosh	02a8f62021	Alerting: Fix stats that display alert count when using unified alerting (#64852) ... * Alerting: Fix stats when using unified alerting	2 years ago
Yuri Tseretyan	85a954cd81	Alerting: Update scheduler to get updates only from database (#64635) ... * stop using the scheduler's Update and Delete methods all communication must be via the database * update scheduler's registry to calculate diff before re-setting the cache * update fetcher to return the diff generated by registry * update processTick to update rule eval routine if the rule was updated and it is not going to be evaluated at this tick. * remove references to the scheduler from api package * remove unused methods in the scheduler	2 years ago
Alexander Weaver	faef3a8258	Alerting: Log error but don't fail initialization if state history connection test fails (#64699) ... Don't return init error if ping fails, add tests	2 years ago
Alexander Weaver	19d01dff91	Alerting: Expose Prometheus metrics for persisting state history (#63157) ... * Create historian metrics and dependency inject * Record counter for total number of state transitions logged * Track write failures * Track current number of active write goroutines * Record histogram of how long it takes to write history data * Don't copy the registerer * Adjust naming of write failures metric * Introduce WritesTotal to complement WritesFailedTotal * Measure TransitionsFailedTotal to complement TransitionsTotal * Rename all to state_history * Remove redundant Total suffix * Increment totals all the time, not just on success * Drop ActiveWriteGoroutines * Drop PersistDuration in favor of WriteDuration * Drop unused gauge * Make writes and writesFailed per org * Add metric indicating backend and a spot for future metadata * Drop _batch_ from names and update help * Add metric for bytes written * Better pairing of total + failure metric updates * Few tweaks to wording and naming * Record info metric during composition * Create fakeRequester and simple happy path test using it * Blocking test for the full historian and test for happy path metrics * Add tests for failure case metrics * Smoke test for full annotation persistence * Create test for metrics on annotation persistence, both happy and failing paths * Address linter complaints * More linter complaints * Remove unnecessary whitespace * Consistency improvements to help texts * Update tests to match new descs	2 years ago
Alexander Weaver	e77621649d	Alerting: Instrument outgoing state history requests using weaveworks/common (#63600) ... * Loki backend and client depend on a requester * Instrument all requests to loki using weaveworks TimedClient * Construct collector in metrics package	2 years ago
Alexander Weaver	6ad1cfef38	Alerting: Add endpoint for querying state history (#62166) ... * Define endpoint and generate * Wire up and register endpoint * Cleanup, define authorization * Forgot the leading slash * Wire up query and SignedInUser * Wire up timerange query params * Add todo for label queries * Drop comment * Update path to rules subtree	2 years ago
Alexander Weaver	e7ace4ed62	Alerting: Allow separate read and write path URLs for Loki state history (#62268) ... Extract config parsing and add tests	3 years ago
Matthew Jacobson	c006df375a	Alerting: Create endpoints for exporting in provisioning file format (#58623) ... This adds provisioning endpoints for downloading alert rules and alert rule groups in a format that is compatible with file provisioning. Each endpoint supports both json and yaml response types via Accept header as well as a query parameter download=true/false that will set Content-Disposition to recommend initiating a download or inline display. This also makes some package changes to keep structs with potential to drift closer together. Eventually, other alerting file structs should also move into this new file package, but the rest require some refactoring that is out of scope for this PR.	3 years ago
Alex Moreno	531b439cf1	Alerting: Add alert pausing feature (#60734) ... * Add field in alert_rule model, add state to alert_instance model, and state to eval * Remove paused state from eval package * Skip paused alert rules in scheduler * Add migration to add is_paused field to alert_rule table * Convert to postable alerts only if not normal, pernding, or paused * Handle paused eval results in state manager * Add Paused state to eval package * Add paused alerts logic in scheduler * Skip alert on scheduler * Remove paused status from eval package * Apply suggestions from code review Co-authored-by: George Robinson <george.robinson@grafana.com> * Remove state * Rethink schedule and manager for paused alerts * Change return to continue * Remove unused var * Rethink alert pausing * Paused alerts storing annotations * Only add one state transition * Revert boolean method renaming refactor * Revert take image refactor * Make registry errors public * Revert method extraction for getting a folder title * Revert variable renaming refactor * Undo unnecessary changes * Revert changes in test * Remove IsPause check in PatchPartiLAlertRule function * Use SetNormal to set state * Fix text by returning to old behaviour on alert rule deletion * Add test in schedule_unit_test.go to test ticks with paused alerts * Add coment to clarify usage of context.Background() * Add comment to clarify resetStateByRuleUID method usage * Move rule get to a more limited scope * Update pkg/services/ngalert/schedule/schedule.go Co-authored-by: George Robinson <george.robinson@grafana.com> * rum gofmt on pkg/services/ngalert/schedule/schedule.go * Remove defer cancel for context * Update pkg/services/ngalert/models/instance_test.go Co-authored-by: Santiago <santiagohernandez.1997@gmail.com> * Update pkg/services/ngalert/models/testing.go Co-authored-by: Santiago <santiagohernandez.1997@gmail.com> * Update pkg/services/ngalert/schedule/schedule_unit_test.go Co-authored-by: Santiago <santiagohernandez.1997@gmail.com> * Update pkg/services/ngalert/schedule/schedule_unit_test.go Co-authored-by: Santiago <santiagohernandez.1997@gmail.com> * Update pkg/services/ngalert/models/instance_test.go Co-authored-by: Santiago <santiagohernandez.1997@gmail.com> * skip scheduler rule state clean up on paused alert rule * Update pkg/services/ngalert/schedule/schedule.go Co-authored-by: Santiago <santiagohernandez.1997@gmail.com> * Fix mock in test * Add (hopefully) final suggestions * Use error channel from recordAnnotationsSync to cancel context * Run make gen-cue * Place pause alert check in channel update after version check * Reduce branching un update channel select * Add if for error and move code inside if in state manager ResetStateByRuleUID * Add reason to logs * Update pkg/services/ngalert/schedule/schedule.go Co-authored-by: George Robinson <george.robinson@grafana.com> * Do not delete alert rule routine, just exit on eval if is paused * Reduce branching and create-close a channel to avoid deadlocks * Separate state deletion and state reset (includes history saving) * Add current pause state in rule route in scheduler * Split clearState and bring errCh closer to RecordStatesAsync call * Change rule to ruleMeta in RecordStatesAsync * copy state to be able to modify it * Add timeout to context creation * Shorten the timeout * Use resetState is rule is paused and deleteState if rule is not paused * Remove Empty state reason * Save every rule change in historian * Add tests for DeleteStateByRuleUID and ResetStateByRuleUID * Remove useless line * Remove outdated comment Co-authored-by: George Robinson <george.robinson@grafana.com> Co-authored-by: Santiago <santiagohernandez.1997@gmail.com> Co-authored-by: Armand Grillet <2117580+armandgrillet@users.noreply.github.com>	3 years ago
George Robinson	a7eab8e46e	Alerting: Support context.Context in Loki interface (#61979) ... This commit adds support for canceleable contexts in the Loki interface.	3 years ago
Alexander Weaver	c10713ea76	Alerting: Create query interface for state history along with annotation-based implementation (#61646)	3 years ago
Jean-Philippe Quéméner	44b11d3228	Alerting: support basic auth for the state history loki client (#61696)	3 years ago
Alexander Weaver	1ac89ea040	Alerting: Add client configuration for remote Loki historian backend and test connection (#61114) ... * Create loki client type and ping method * Expose TestConnection on client * Configure and ping Loki URL * Close response body reader if present * Add 30 second timeout * Remove duplicate close	3 years ago
Yuri Tseretyan	9d57b1c72e	Alerting: Do not persist noop transition from Normal state. (#61201) ... * add feature flag `alertingNoNormalState` * update instance database to support exclusion of state in list operation * do not save normal state and delete transitions to normal * update get methods to filter out normal state	3 years ago
Yuri Tseretyan	86b5fbbf60	Alerting: Introduce state manager config structure (#61249)	3 years ago
Yuri Tseretyan	48f1db63ff	Alerting: Add support for tracing to alerting scheduler (#61057)	3 years ago
Alexander Weaver	eb960d9725	Alerting: Add un-documented toggle for changing state history backend, add shells for remote loki and sql (#61072) ... * Add toggle for state history backend and shells * Extract some shared logic and add tests	3 years ago
Alexander Weaver	8c3a5f6da0	Alerting: Allow state history to be disabled through configuration (#61006) ... * Add configuration option for if state history should be enabled * Inject no-op when history is disabled	3 years ago
Yuri Tseretyan	ad09feed83	Alerting: rule backtesting API (#57318) ... * Implement backtesting engine that can process regular rule specification (with queries to datasource) as well as special kind of rules that have data frame instead of query. * declare a new API endpoint and model * add feature toggle `alertingBacktesting`	3 years ago
Yuri Tseretyan	c5ee4e4ae1	Alerting: Improve rule validation to check if rule uses backend datasources (#58986) ... * validate if rule uses backend datasources * add backend datasource to test * fix tests * another forgotten import * remove unused var	3 years ago
Alexander Weaver	9977c7ea43	Alerting: Simplify scheduler configuration and remove dependency on Grafana-wide settings (#59735) ... * Make scheduler not depend directly on grafana-wide settings * Re-add missing interval	3 years ago
Sofia Papagiannaki	9855e74b92	Chore: Refactor quota service (#58643) ... Chore: Refactor quota service (#57586) * Chore: refactore quota service * Apply suggestions from code review	3 years ago
Sofia Papagiannaki	96cdf77995	Revert "Chore: Refactor quota service (#57586)" (#58394) ... This reverts commit 326ea86a57.	3 years ago
Sofia Papagiannaki	326ea86a57	Chore: Refactor quota service (#57586) ... * Chore: refactore quota service * Apply suggestions from code review	3 years ago
Yuri Tseretyan	978f1119d7	Alerting: Run state manager as regular sub-service (#58246)	3 years ago
Yuri Tseretyan	dce8879145	Alerting: Update state manager to accept rule store as Warm method argument (#58244)	3 years ago
Yuriy Tseretyan	e3a4bde622	Alerting: Condition evaluator with cached pipeline (#57479) ... * create rule evaluator * load header from the context * init one factory * update scheduler	3 years ago