We have received a lot of feedback regarding the ValueString in alert notifications. Perhaps one of the most frequent complaints about ValueString is that it is difficult to read because it contains a lot of information, and the information is shown as a JSON-like string. Users have often asked how it can be templated and the answer is that it can't.
Until now users have been able to add custom annotations to their alert rules which contains values via the $values variable added in previous versions of Grafana. However, these custom annotations must be added for each of the user's alert rule, instead of once in a template that all of their alerts can be notified via.
This commit adds then the much requested feature to support values in notification templates. Users can then create a single template that prints the annotations, labels and values of their alerts in a format of their choice!
The State Manager will now take screenshots when an alert instance
switches to an Alerting or Resolved state.
Signed-off-by: Joe Blubaugh joe.blubaugh@grafana.com
Before this change, notifications generated by the Grafana Alertmanager
pointed to '/alerting/:ruleID/edit'. This change instead points them to
the view path '/alerting/grafana/:ruleID/view'. The view page has a
better UX, including timeseries display. It's also where many alert
state improvements will land in the next few versions of Grafana.
Fixes#45301
Signed-off-by: Joe Blubaugh <joe.blubaugh@grafana.com>
* Update API to call the scheduler to remove\update an alert rule. When a rule is updated by a user, the scheduler will remove the currently firing alert instances and clean up the state cache.
* Update evaluation loop in the scheduler to support one more channel that is used to communicate updates to it.
* Improved rule deletion from the internal registry.
* Move alert rule version from the internal registry (structure alertRuleInfo) closer rule evaluation loop (to evaluation task structure), which will make the registry values immutable.
* Extract notification code to a separate function to reuse in update flow.
* do not suppress NoData state
* extract conversion of state to postable alert + tests
* create a special alert instance if nodata
* use NoData when converting from Keep Last State instead of Alerting
* add silence during migration if NoData is mapped to KeepLastState.
* Alerting: Allow __value__ label in notifications
was being removed by removePrivateItems
discoverd in #36020, but issue is not about that specifically
* __value__ label to __value_string__ annotation
and .ValueString extended property for notifications
When, and currently only when using a classic condition, evaluation information is added (which is like the EvalMatches from dashboard alerting).
This is returned via the API and can be included in notifications by reading the `__value__` label attached `.Alerts` in the template. It is a string.
* set processing time
* merge labels and set on response
* use state cache for adding alerts to rules
* minor cleanup
* add support for NoData and Error results
* rename test
* bring in changes from other PRs tha have been merged
* pr feedback
* add integration test
* close state tracker cleanup on context.Done
* fixup test
* rename state tracker
* set EvaluationDuration on Result
* default labels set as constants
* separate cache and state from manager
* use RWMutex in cache
* init
* autogens AM route
* POST dashboards/db spec
* POST alert-notifications spec
* fix description
* re inits vendor, updates grafana to master
* go mod updates
* alerting routes
* renames to receivers
* prometheus endpoints
* align config endpoint with cortex, include templates
* Change grafana receiver type
* Update receivers.go
* rename struct to stop swagger thrashing
* add rules API
* index html
* standalone swagger ui html page
* Update README.md
* Expose GrafanaManagedAlert properties
* Some fixes
- /api/v1/rules/{Namespace} should return a map
- update ExtendedUpsertAlertDefinitionCommand properties
* am alerts routes
* rename prom swagger section for clarity, remove example endpoints
* Add missing json and yaml tags
* folder perms
* make folders POST again
* fix grafana receiver type
* rename fodler->namespace for perms
* make ruler json again
* PR fixes
* silences
* fix Ok -> Ack
* Add id to POST /api/v1/silences (#9)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* Add POST /api/v1/alerts (#10)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
* fix silences
* Add testing endpoints
* removes grpc replace directives
* [wip] starts validation
* pkg cleanup
* go mod tidy
* ignores vendor dir
* Change response type for Cortex/Loki alerts
* receiver unmarshaling tests
* ability to split routes between AM & Grafana
* api marshaling & validation
* begins work on routing lib
* [hack] ignores embedded field in generation
* path specific datasource for alerting
* align endpoint names with cloud
* single route per Alerting config
* removes unused routing pkg
* regens spec
* adds datasource param to ruler/prom route paths
* Modifications for supporting migration
* Apply suggestions from code review
* hack for cleaning circular refs in swagger definition
* generates files
* minor fixes for prom endpoints
* decorate prom apis with required: true where applicable
* Revert "generates files"
This reverts commit ef7e975584.
* removes server autogen
* Update imported structs from ngalert
* Fix listing rules response
* Update github.com/prometheus/common dependency
* Update get silence response
* Update get silences response
* adds ruler validation & backend switching
* Fix GET /alertmanager/{DatasourceId}/config/api/v1/alerts response
* Distinct gettable and postable grafana receivers
* Remove permissions routes
* Latest JSON specs
* Fix testing routes
* inline yaml annotation on apirulenode
* yaml test & yamlv3 + comments
* Fix yaml annotations for embedded type
* Rename DatasourceId path parameter
* Implement Backend.String()
* backend zero value is a real backend
* exports DiscoveryBase
* Fix GO initialisms
* Silences: Use PostableSilence as the base struct for creating silences
* Use type alias instead of struct embedding
* More fixes to alertmanager silencing routes
* post and spec JSONs
* Split rule config to postable/gettable
* Fix empty POST /silences payload
Recreating the generated JSON specs fixes the issue
without further modifications
* better yaml unmarshaling for nested yaml docs in cortex-am configs
* regens spec
* re-adds config.receivers
* omitempty to align with prometheus API behavior
* Prefix routes with /api
* Update Alertmanager models
* Make adjustments to follow the Alertmanager API
* ruler: add for and annotations to grafana alert (#45)
* Modify testing API routes
* Fix grafana rule for field type
* Move PostableUserConfig validation to this library
* Fix PostableUserConfig YAML encoding/decoding
* Use common fields for grafana and lotex rules
* Add namespace id in GettableGrafanaRule
* Apply suggestions from code review
* fixup
* more changes
* Apply suggestions from code review
* aligns structure pre merge
* fix new imports & tests
* updates tooling readme
* goimports
* lint
* more linting!!
* revive lint
Co-authored-by: Sofia Papagiannaki <papagian@gmail.com>
Co-authored-by: Domas <domasx2@gmail.com>
Co-authored-by: Sofia Papagiannaki <papagian@users.noreply.github.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Co-authored-by: gotjosh <josue@grafana.com>
Co-authored-by: David Parrott <stomp.box.yo@gmail.com>
Co-authored-by: Kyle Brandt <kyle@grafana.com>
* move state tracker tests to /tests
* set default labels on alerts
* handle empty labels in result.Instance
* create annotation on transition to alerting state
* Initial commit for state tracking
* basic state transition logic and tests
* constructor. test and interface fixup
* use new sig for sch.definitionRoutine()
* test fixup
* make the linter happy
* more minor linting cleanup
* Alerting: Send alerts from state tracker to notifier
* Add evaluation time and test
Add evaluation time and test
* Add cleanup routine and logging
* Pull in compact.go and reconcile differences
* Save alert transitions and save all state on shutdown
* pr feedback
* WIP
* WIP
* Persist alerts on evaluation and shutdown. Warm cache on startup
* Filter non-firing alerts before sending to notifier
Co-authored-by: Josue Abreu <josue@grafana.com>
* Initial commit for state tracking
* basic state transition logic and tests
* constructor. test and interface fixup
* use new sig for sch.definitionRoutine()
* test fixup
* make the linter happy
* more minor linting cleanup
* Alerting: Send alerts from state tracker to notifier
* Add evaluation time and test
Add evaluation time and test
* Add cleanup routine and logging
* Pull in compact.go and reconcile differences
* pr feedback
* pr feedback
Pull in compact.go and reconcile differences
Co-authored-by: Josue Abreu <josue@grafana.com>
