* Basic eval flow
* Wiring-up
* fix
* Extend todo
* Start with tests
* Include some relevant tests, skip ones that seem to have timing-based race conditions
* Some tests, touch up linter and todo
* Solve TODO
* Add tracing
* Tests to make sure an eval went through
* Wire up feature toggles
* Update pkg/services/ngalert/schedule/recording_rule.go
Co-authored-by: Steve Simpson <steve.simpson@grafana.com>
* Update pkg/services/ngalert/schedule/recording_rule_test.go
Co-authored-by: Steve Simpson <steve.simpson@grafana.com>
* Update pkg/services/ngalert/schedule/recording_rule_test.go
Co-authored-by: Steve Simpson <steve.simpson@grafana.com>
* Update pkg/services/ngalert/schedule/recording_rule_test.go
Co-authored-by: Steve Simpson <steve.simpson@grafana.com>
---------
Co-authored-by: Steve Simpson <steve.simpson@grafana.com>
* Add shim rule implementation for recording rules
* Give ruleFactory access to the original rule definition
* Schedule shim implementation if the rule is a recording rule
* Fix or suppress linter
* Fix nolint
* Create recording rule fields in model
* Add migration
* Write to database, support in version table
* extend fingerprint
* Force fields to be empty on validate
* Another storage spot, tests for fingerprint
* Explicitly set defaults in provisioning API
* Tests for main API validation
* Add diff tests even though fields are unpopulated for now
* Use struct tag approach instead of FromDB/ToDB hooks as it better handles nulls when deserializing
* test for deser
* Backout RecordTo for now since it's not decided in the doc
* back out of migration too
* Drop datasourceref for now
* address linter complaints
* Try a single outer struct with all fields embedded
* export Evaluation
* Export Evaluation
* Export RuleVersionAndPauseStatus
* export Eval, create interface
* Export update and add to interface
* Export Stop and Run and add to interface
* Registry and scheduler use rule by interface and not concrete type
* Update factory to use interface, update tests to work over public API rather than writing to channels directly
* Rename map in registry
* Rename getOrCreateInfo to not reference a specific implementation
* Genericize alertRuleInfoRegistry into ruleRegistry
* Rename alertRuleInfo to alertRule
* Comments on interface
* Update pkg/services/ngalert/schedule/schedule.go
Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>
---------
Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>
* create rule factory for more complicated dep injection into rules
* Rules get direct access to metrics, logs, traces utilities, use factory in tests
* Use clock internal to rule
* Use sender, statemanager, evalfactory directly
* evalApplied and stopApplied
* use schedulableAlertRules behind interface
* loaded metrics reader
* 3 relevant config options
* Drop unused scheduler parameter
* Rename ruleRoutine to run
* Update READMED
* Handle long parameter lists
* remove dead branch
* Add notification settings to storage\domain and API models. Settings are a slice to workaround XORM mapping
* Support validation of notification settings when rules are updated
* Implement route generator for Alertmanager configuration. That fetches all notification settings.
* Update multi-tenant Alertmanager to run the generator before applying the configuration.
* Add notification settings labels to state calculation
* update the Multi-tenant Alertmanager to provide validation for notification settings
* update GET API so only admins can see auto-gen
* stop using the scheduler's Update and Delete methods all communication must be via the database
* update scheduler's registry to calculate diff before re-setting the cache
* update fetcher to return the diff generated by registry
* update processTick to update rule eval routine if the rule was updated and it is not going to be evaluated at this tick.
* remove references to the scheduler from api package
* remove unused methods in the scheduler
* Add field in alert_rule model, add state to alert_instance model, and state to eval
* Remove paused state from eval package
* Skip paused alert rules in scheduler
* Add migration to add is_paused field to alert_rule table
* Convert to postable alerts only if not normal, pernding, or paused
* Handle paused eval results in state manager
* Add Paused state to eval package
* Add paused alerts logic in scheduler
* Skip alert on scheduler
* Remove paused status from eval package
* Apply suggestions from code review
Co-authored-by: George Robinson <george.robinson@grafana.com>
* Remove state
* Rethink schedule and manager for paused alerts
* Change return to continue
* Remove unused var
* Rethink alert pausing
* Paused alerts storing annotations
* Only add one state transition
* Revert boolean method renaming refactor
* Revert take image refactor
* Make registry errors public
* Revert method extraction for getting a folder title
* Revert variable renaming refactor
* Undo unnecessary changes
* Revert changes in test
* Remove IsPause check in PatchPartiLAlertRule function
* Use SetNormal to set state
* Fix text by returning to old behaviour on alert rule deletion
* Add test in schedule_unit_test.go to test ticks with paused alerts
* Add coment to clarify usage of context.Background()
* Add comment to clarify resetStateByRuleUID method usage
* Move rule get to a more limited scope
* Update pkg/services/ngalert/schedule/schedule.go
Co-authored-by: George Robinson <george.robinson@grafana.com>
* rum gofmt on pkg/services/ngalert/schedule/schedule.go
* Remove defer cancel for context
* Update pkg/services/ngalert/models/instance_test.go
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
* Update pkg/services/ngalert/models/testing.go
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
* Update pkg/services/ngalert/schedule/schedule_unit_test.go
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
* Update pkg/services/ngalert/schedule/schedule_unit_test.go
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
* Update pkg/services/ngalert/models/instance_test.go
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
* skip scheduler rule state clean up on paused alert rule
* Update pkg/services/ngalert/schedule/schedule.go
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
* Fix mock in test
* Add (hopefully) final suggestions
* Use error channel from recordAnnotationsSync to cancel context
* Run make gen-cue
* Place pause alert check in channel update after version check
* Reduce branching un update channel select
* Add if for error and move code inside if in state manager ResetStateByRuleUID
* Add reason to logs
* Update pkg/services/ngalert/schedule/schedule.go
Co-authored-by: George Robinson <george.robinson@grafana.com>
* Do not delete alert rule routine, just exit on eval if is paused
* Reduce branching and create-close a channel to avoid deadlocks
* Separate state deletion and state reset (includes history saving)
* Add current pause state in rule route in scheduler
* Split clearState and bring errCh closer to RecordStatesAsync call
* Change rule to ruleMeta in RecordStatesAsync
* copy state to be able to modify it
* Add timeout to context creation
* Shorten the timeout
* Use resetState is rule is paused and deleteState if rule is not paused
* Remove Empty state reason
* Save every rule change in historian
* Add tests for DeleteStateByRuleUID and ResetStateByRuleUID
* Remove useless line
* Remove outdated comment
Co-authored-by: George Robinson <george.robinson@grafana.com>
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
Co-authored-by: Armand Grillet <2117580+armandgrillet@users.noreply.github.com>
* Update GetAlertRulesForScheduling to query for folders (if needed)
* Update scheduler's alertRulesRegistry to cache folder titles along with rules
* Update rule eval loop to take folder title from the
* Extract interface RuleStore
* Pre-fetch the rule keys with the version to detect changes, and query the full table only if there are changes.
* update GetAlertRulesForSchedulingQuery to have result AlertRule
* update fetcher utils and registry to support AlertRule
* alertRuleInfo to use alert rule instead of version
* update updateCh hanlder of ruleRoutine to just clean up the state. The updated rule will be provided at the next evaluation
* update evalCh handler of ruleRoutine to use rule from the message and clear state as well as update extra labels
* remove unused function in ruleRoutine
* remove unused model SchedulableAlertRule
* store rule version in ruleRoutine instead of rule
* do not call the sender if nothing to send
* handler for update message in rule evaluation routine ignores the message if its version greater or equal.
* replace messages to update the channel if it is not empty
