grafana

Commit Graph

Author	SHA1	Message	Date
Joe Blubaugh	22c937340e	Revert "Alerting: Write and Delete multiple alert instances. (#54072 )" (#54885 ) This reverts commit `5e4fd94413`.	3 years ago
Joe Blubaugh	5e4fd94413	Alerting: Write and Delete multiple alert instances. (#54072 ) Prior to this change, all alert instance writes and deletes happened individually, in their own database transaction. This change batches up writes or deletes for a given rule's evaluation loop into a single transaction before applying it. Before: ``` goos: darwin goarch: arm64 pkg: github.com/grafana/grafana/pkg/services/ngalert/store BenchmarkAlertInstanceOperations-8 398 2991381 ns/op 1133537 B/op 27703 allocs/op --- BENCH: BenchmarkAlertInstanceOperations-8 util.go:127: alert definition: {orgID: 1, UID: FovKXiRVzm} with title: "an alert definition FTvFXmRVkz" interval: 60 created util.go:127: alert definition: {orgID: 1, UID: foDFXmRVkm} with title: "an alert definition fovFXmRVkz" interval: 60 created util.go:127: alert definition: {orgID: 1, UID: VQvFuigVkm} with title: "an alert definition VwDKXmR4kz" interval: 60 created PASS ok github.com/grafana/grafana/pkg/services/ngalert/store 1.619s ``` After: ``` goos: darwin goarch: arm64 pkg: github.com/grafana/grafana/pkg/services/ngalert/store BenchmarkAlertInstanceOperations-8 1440 816484 ns/op 352297 B/op 6529 allocs/op --- BENCH: BenchmarkAlertInstanceOperations-8 util.go:127: alert definition: {orgID: 1, UID: 302r_igVzm} with title: "an alert definition q0h9lmR4zz" interval: 60 created util.go:127: alert definition: {orgID: 1, UID: 71hrlmR4km} with title: "an alert definition nJ29_mR4zz" interval: 60 created util.go:127: alert definition: {orgID: 1, UID: Cahr_mR4zm} with title: "an alert definition ja2rlmg4zz" interval: 60 created PASS ok github.com/grafana/grafana/pkg/services/ngalert/store 1.383s ``` So we cut time by about 75% and memory allocations by about 60% when storing and deleting 100 instances. This change also updates some of our tests so that they run successfully against postgreSQL - we were using random Int64s, but postgres integers, which our tables use, max out at 2^31-1	3 years ago
Yuriy Tseretyan	9f90a7b54d	Alerting: State manager to use InstanceStore (#53852 ) * move saving the state to state manager when scheduler stops * move saving state to ProcessEvalResults * add GetRuleKey to State * add LogContext to AlertRuleKey	3 years ago
Yuriy Tseretyan	e5e8747ee9	Alerting: Update state manager to accept reserved labels (#52189 ) * add tests for cache getOrCreate * update ProcessEvalResults to accept extra lables * extract to getRuleExtraLabels * move populating of constant rule labels to extra labels	3 years ago
George Robinson	34d45977ca	Alerting: Fix bug where state did not change between Alerting and Error (#52204 ) This commit fixes a bug where the state did not change from Alerting to Error if the evaluation result returned an error, or from Error to Alerting if evaluations stopped returning errors.	3 years ago
Yuriy Tseretyan	4b42cd3c1d	Alerting: State manager to use clock (#51219 ) * manager to use clock, to be able to mock real time	3 years ago
Yuriy Tseretyan	157c12211d	Alerting: State manager to use tick time to determine stale states (#50991 ) * use correct stale timestamp * calculate stale using tick time instead of time.now * remove unused dependency on sql store	3 years ago
Joe Blubaugh	1cc034d960	Alerting: Add a "Reason" to Alert Instances to show underlying cause of state. (#49259 ) This change adds a field to state.State and models.AlertInstance that indicate the "Reason" that an instance has its current state. This helps us account for cases where the state is "Normal" but the underlying evaluation returned "NoData" or "Error", for example. Fixes #42606 Signed-off-by: Joe Blubaugh <joe.blubaugh@grafana.com>	3 years ago
Joe Blubaugh	1d724810de	Alerting: State Manager takes screenshots. (#49338 ) The State Manager will now take screenshots when an alert instance switches to an Alerting or Resolved state. Signed-off-by: Joe Blubaugh joe.blubaugh@grafana.com	3 years ago
Joe Blubaugh	687e79538b	Alerting: Add a general screenshot service and alerting-specific image service. (#49293 ) This commit adds a pkg/services/screenshot package for taking and uploading screenshots of Grafana dashboards. It supports taking screenshots of both dashboards and individual panels within a dashboard, using the rendering service. The screenshot package has the following services, most of which can be composed: BrowserScreenshotService (Takes screenshots with headless Chrome) CachableScreenshotService (Caches screenshots taken with another service such as BrowserScreenshotService) NoopScreenshotService (A no-op screenshot service for tests) SingleFlightScreenshotService (Prevents duplicate screenshots when taking screenshots of the same dashboard or panel in parallel) ScreenshotUnavailableService (A screenshot service that returns ErrScreenshotsUnavailable) UploadingScreenshotService (A screenshot service that uploads taken screenshots) The screenshot package does not support wire dependency injection yet. ngalert constructs its own version of the service. See https://github.com/grafana/grafana/issues/49296 This PR also adds an ImageScreenshotService to ngAlert. This is used to take screenshots with a screenshotservice and then store their location reference for use by alert instances and notifiers.	3 years ago
Kristin Laemmert	1df340ff28	backend/services: Move GetDashboard from sqlstore to dashboard service (#48971 ) * rename folder to match package name * backend/sqlstore: move GetDashboard into DashboardService This is a stepping-stone commit which copies the GetDashboard function - which lets us remove the sqlstore from the interfaces in dashboards - without changing any other callers. * checkpoint: moving GetDashboard calls into dashboard service * finish refactoring api tests for dashboardService.GetDashboard	3 years ago
Yuriy Tseretyan	884c885289	Alerting: Support OK option for Error state (#47670 ) * support OK state for Error	3 years ago
gotjosh	cb6124c921	Alerting: Accurately set value for prom-compatible APIs (#47216 ) * Alerting: Accurately set value for prom-compatible APIs Sets the value fields for the prometheus compatible API based on a combination of condition `refID` and the values extracted from the different frames. * Fix an extra test * Ensure a consitent ordering * Address review comments * address review comments	3 years ago
George Robinson	79769132c0	Alerting: Alert rule should wait For duration when execution error state is Alerting (#47052 ) Alerting: Alert rule should wait For duration when execution error state is Alerting	3 years ago
George Robinson	789cfc31e3	Alerting: Fix use of > instead of >= when checking the For duration (#46011 )	3 years ago
George Robinson	8d57318941	Alerting: Use expanded labels in dashboard annotations (#45726 )	3 years ago
Yuriy Tseretyan	02f8e99ca1	Alerting: move fake stores to store package (#45428 ) * make fake storage public * move fake storages to store package	3 years ago
George Robinson	67a3e1d6fd	Add context.Context to InstanceStore (#45049 )	3 years ago
George Robinson	a9399ab3cd	Alerting: Add context.Context to RuleStore (#45004 ) Alerting: Add context.Context to RuleStore	3 years ago
idafurjes	7a23700e1a	Remove unused GetDashboard method (#44890 ) * Remove unused GetDashboard method * Uncomment test * Fix dashboard service integration test * Remove comment	3 years ago
Yuriy Tseretyan	984c95de63	Do not store EvaluationString in Evaluation. (#44606 ) * do not store evaluation string in Evaluation. * reduce number of buckets to store for a single state	3 years ago
George Robinson	5e2280ceee	Add metrics to ngalert scheduler (#44602 ) This pull request adds metrics to the ngalert scheduler so we can see how long it takes to evaluate a tick.	3 years ago
George Robinson	c932dc959c	Alerting: Add Ref ID to DatasourceNoData and DatasourceError alerts (#42630 )	4 years ago
gotjosh	357e9ed1ea	Alerting: Fix Annotation Creation when the alerting state changes (#42479 ) * Fix Annotation creation - Remove validation of panelID, now annotations are created irrespective on whether they're attached to a panel or not. - Alwasy attach the annotation to an AlertID * Fix annotation creation * fix tests	4 years ago
gotjosh	dd5a2e5128	Alerting: Clear alerting rule evaluation errors after intermittent failures (#42386 ) * Alerting: Clear alerting rule evaluation errors after intermittent failures When an alert transitioned in a way that `alerting -> error -> (alerting\|nodata)`, the error provided by the `error` state would never be cleared thus the API and UI would show the health as an error.	4 years ago
George Robinson	1b26d4d88e	Alerting: Create DatasourceError alert if evaluation returns error (#41869 ) * Alerting: Create DatasourceError alert if evaluation returns error * Alerting: Add docs for DatasourceError alert * Alerting: Fix DatasourceError alert does not have dashboard_uid label * Alerting: Add break when datasource_uid found * Alerting: Update TestProcessEvalResults	4 years ago
Yuriy Tseretyan	1b5b747885	Alerting: Additional Tests for State Manager (#41291 ) * rename fakeInstanceStore to FakeInstanceStore * update test for state manager to initialize instance store with FakeInstanceStore	4 years ago
idafurjes	2759b16ef5	Chore: Add context for dashboards (#39844 ) * Add context for dashboards * Remove GetDashboardCtx * Remove ctx.TODO	4 years ago
Santiago	562cd9e44e	Alerting template functions (#39261 ) * Alerting: (wip) add template funcs * Alerting: (wip) numeric template functions * Alerting: (wip) template functions * Test for the "args" function * Alerting: (wip) Documentation for template functions * Alerting: template functions - refactor * code review changes * disable linter error * Use Prometheus implementation of TemplateExpander * Update docs/sources/alerting/unified-alerting/alerting-rules/create-grafana-managed-rule.md Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * change templateCaptureValue to support using template functions * Update pkg/services/ngalert/state/template.go Co-authored-by: gotjosh <josue.abreu@gmail.com> * Test and documentation added for reReplaceAll template function * complete missing functions, documentation and tests * Use the alert instance's evaluation time for expanding the template * strvalue graphlink and tablelink functions * delete duplicate test * make strvalue return an empty string Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> Co-authored-by: gotjosh <josue.abreu@gmail.com>	4 years ago
Sofia Papagiannaki	012d4f0905	Alerting: Remove `ngalert` feature toggle and introduce two new settings for enabling Grafana 8 alerts and disabling them for specific organisations (#38746 ) * Remove `ngalert` feature toggle * Update frontend Remove all references of ngalert feature toggle * Update docs * Disable unified alerting for specific orgs * Add backend tests * Apply suggestions from code review Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com> * Disabled unified alerting by default * Ensure backward compatibility with old ngalert feature toggle * Apply suggestions from code review Co-authored-by: gotjosh <josue@grafana.com>	4 years ago
gotjosh	a2f4344bf2	Alerting: Refactor & fix unified alerting metrics structure (#39151 ) * Alerting: Refactor & fix unified alerting metrics structure Fixes and refactors the metrics structure we have for the ngalert service. Now, each component has its own metric struct that includes the JUST the metrics it uses. Additionally, I have fixed the configuration metrics and added new metrics to determine if we have discovered and started all the necessary configurations of an instance. This allows us to alert on `grafana_alerting_discovered_configurations - grafana_alerting_active_configurations != 0` to know whether an alertmanager instance did not start successfully.	4 years ago
gotjosh	dd502f22eb	Alerting: Fix alert flapping in the internal alertmanager (#38648 ) * Alerting: Fix alert flapping in the alertmanager fixes a bug that caused Alerts that are evaluated at low intervals (sub 1 minute), to flap in the Alertmanager. Mostly due to a combination of `EndsAt` and resend delay. The Alertmanager uses `EndsAt` as a heuristic to know whenever it should resolve a firing alert, in the case that it hasn't heard back from the alert generation system. Because grafana sent the alert with an `EndsAt` which is equal to the `For` of the alert itself, and we had a hard-coded 1 minute re-send delay (only applicable to firing alerts) this meant that a firing alert would resolve in the Alertmanager before we re-notify that it still firing. This commit, increases the `EndsAt` by 3x the the resend delay or alert interval (depending on which one is higher). The resendDelay has been decreased to 30 seconds.	4 years ago
Arve Knudsen	78596a6756	Migrate to Wire for dependency injection (#32289 ) Fixes #30144 Co-authored-by: dsotirakis <sotirakis.dim@gmail.com> Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com> Co-authored-by: Ida Furjesova <ida.furjesova@grafana.com> Co-authored-by: Jack Westbrook <jack.westbrook@gmail.com> Co-authored-by: Will Browne <wbrowne@users.noreply.github.com> Co-authored-by: Leon Sorokin <leeoniya@gmail.com> Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com> Co-authored-by: spinillos <selenepinillos@gmail.com> Co-authored-by: Karl Persson <kalle.persson@grafana.com> Co-authored-by: Leonard Gram <leo@xlson.com>	4 years ago
David Parrott	b5f464412d	Alerting: automatically remove stale alerting states (#36767 ) * initial attempt at automatic removal of stale states * test case, need espected states * finish unit test * PR feedback * still multiply by time.second * pr feedback	4 years ago
George Robinson	456dac1303	Expand the value of math and reduce expressions in annotations and labels (#36611 ) * Expand the value of math and reduce expressions in annotations and labels This commit makes it possible to use the values of reduce and math expressions in annotations and labels via their RefIDs. It uses the Stringer interface to ensure that "{{ $values.A }}" still prints the value in decimal format while also making the labels for each RefID available with "{{ $values.A.Labels }}" and the float64 value with "{{ $values.A.Value }}"	4 years ago
gotjosh	a86ad1190c	Alerting: Refactor state manager as a dependency (#36513 ) * Alerting: Refactor state manager as a dependency Within the scheduler, the state manager was being passed around a certain number of functions. I've introduced it as a dependency to keep the "service" interfaces as clean and homogeneous as possible. This is relevant, because I'm going to introduce live reload of these components as part of my next PR and it is better if dependencies are self-contained. * remove unused functions * Fix a few more tests * Make sure the `stateManager` is declared before the schedule	4 years ago
Ganesh Vernekar	8417088969	Alerting: Expand `{{$labels.xyz}}` template in labels and annotations (#35159 ) * Alerting: Expand `{{$labels.xyz}}` template in labels and annotations Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix annotation not updating for same alert Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	4 years ago
David Parrott	20d356947c	set state correctly and test (#34680 )	4 years ago
David Parrott	bbb7bbf891	Alerting: Remove back end logic for supporting KeepLastState (#34242 ) * Removed back end logic for supporting KeepLastState * Map keep_state correctly in migrations	4 years ago
Kyle Brandt	babb17afd6	Alerting/Chore: Move tests from tests package (#34059 ) Instead put in package folder but with package name suffixed with _test This enables code coverage within the pkg while still allow the tests to operate from external to package perspective (only exported things).	4 years ago
David Parrott	b1a8c67689	Alerting return evaluation errors to /rules (#33663 ) * Set and return errors produced by evaluation results * test fixup	4 years ago
David Parrott	39099bf3c0	Alerting nested state cache (#33666 ) * nest cache by orgID, ruleUID, stateID * update accessors to use new cache structure * test and linter fixup * fix panic Co-authored-by: Kyle Brandt <kyle@grafana.com> * add comment to identify what's going on with nested maps in cache Co-authored-by: Kyle Brandt <kyle@grafana.com>	4 years ago
Owen Diehl	5e48b54549	Alerting/metrics (#33547 ) * moves alerting metrics to their own pkg * adds grafana_alerting_alerts (by state) metric * alerts_received_{total,invalid} * embed alertmanager alerting struct in ng metrics & remove duplicated notification metrics (already embed alertmanager notifier metrics) * use silence metrics from alertmanager lib * fix - manager has metrics * updates ngalert tests * comment lint Signed-off-by: Owen Diehl <ow.diehl@gmail.com> * cleaner prom registry code * removes ngalert global metrics * new registry use in all tests * ngalert metrics impl service, hack testinfra code to prevent duplicate metric registrations * nilmetrics unexported	4 years ago
Kyle Brandt	914443c816	Alerting: Fix state cache id duplication (#33480 )	4 years ago
David Parrott	788bc2a793	Alerting: refactor state tracker (#33292 ) * set processing time * merge labels and set on response * use state cache for adding alerts to rules * minor cleanup * add support for NoData and Error results * rename test * bring in changes from other PRs tha have been merged * pr feedback * add integration test * close state tracker cleanup on context.Done * fixup test * rename state tracker * set EvaluationDuration on Result * default labels set as constants * separate cache and state from manager * use RWMutex in cache	4 years ago
David Parrott	ca79206498	Alerting: Handle NoData and Error evaluation results (#33194 ) * set processing time * merge labels and set on response * use state cache for adding alerts to rules * minor cleanup * add support for NoData and Error results * rename test * bring in changes from other PRs tha have been merged * pr feedback * add integration test * close state tracker cleanup on context.Done * fixup test * not those annotations	4 years ago
gotjosh	de0802cf3b	Alerting: Fixes the integration test currently failing at master (#33233 ) * Alerting: Fixes the integration test currently failing at master * Skip the state tracker test for now	4 years ago
David Parrott	4be1d84f23	Alerting: Enhancements to /rules (#33085 ) * set processing time * merge labels and set on response * use state cache for adding alerts to rules * minor cleanup * pr feedback * Do not initialize mutex unnecessarily Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com> * linter Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>	4 years ago
David Parrott	555da77527	Dparrott/labels on alert rule (#33057 ) * move state tracker tests to /tests * set default labels on alerts * handle empty labels in result.Instance * create annotation on transition to alerting state	4 years ago
David Parrott	567a6a09bd	Alerting: Return RuleResponse for api/prometheus/grafana/api/v1/rules (#32919 ) * Return RuleResponse for api/prometheus/grafana/api/v1/rules * change TODO to note Co-authored-by: gotjosh <josue@grafana.com> * pr feedback * test fixup Co-authored-by: gotjosh <josue@grafana.com>	4 years ago

40 Commits (22c937340e7ae239eb5b3e3f0d1764bc54da266b)