* Alerting: Persist notification log and silences to the database
This removes the dependency of having persistent disk to run grafana alerting. Instead of regularly flushing the notification log and silences to disk we now flush the binary content of those files to the database encoded as a base64 string.
* Change templateCaptureValue to support using template functions
This commit changes templateCaptureValue to use float64 for the value
instead of *float64. This change means that annotations and labels can
use the float64 value with functions such as printf and avoid having to
check for nil. It also means that absent values are now printed as 0.
* Use math.NaN() instead of 0 for absent value
* Alerting: Fix alert flapping in the alertmanager
fixes a bug that caused Alerts that are evaluated at low intervals (sub 1 minute), to flap in the Alertmanager.
Mostly due to a combination of `EndsAt` and resend delay.
The Alertmanager uses `EndsAt` as a heuristic to know whenever it should resolve a firing alert, in the case that it hasn't heard
back from the alert generation system.
Because grafana sent the alert with an `EndsAt` which is equal to the `For` of the alert itself,
and we had a hard-coded 1 minute re-send delay (only applicable to firing alerts) this meant that a firing alert would resolve in the Alertmanager before we re-notify that it still firing.
This commit, increases the `EndsAt` by 3x the the resend delay or alert interval (depending on which one is higher). The resendDelay has been decreased to 30 seconds.
Fixes#30144
Co-authored-by: dsotirakis <sotirakis.dim@gmail.com>
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
Co-authored-by: Ida Furjesova <ida.furjesova@grafana.com>
Co-authored-by: Jack Westbrook <jack.westbrook@gmail.com>
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
Co-authored-by: Leon Sorokin <leeoniya@gmail.com>
Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com>
Co-authored-by: spinillos <selenepinillos@gmail.com>
Co-authored-by: Karl Persson <kalle.persson@grafana.com>
Co-authored-by: Leonard Gram <leo@xlson.com>
Introduces org-level isolation for the Alertmanager and its components.
Silences, Alerts and Contact points are not separated by org and are not shared between them.
Co-authored with @davidmparrott and @papagian
This commit adds contact point testing to ngalerts via a new API
endpoint. This endpoint accepts JSON containing a list of
receiver configurations which are validated and then tested
with a notification for a test alert. The endpoint returns JSON
for each receiver with a status and error message. It accepts
a configurable timeout via the Request-Timeout header (in seconds)
up to a maximum of 30 seconds.
* Alerting: Expose discovered and dropped Alertmanagers
Exposes the API for discovered and dropped Alertmanagers.
* make admin config poll interval configurable
* update after rebase
* wordsmith
* More wordsmithing
* change name of the config
* settings package too
* Alerting: modify table and accessors to limit org access appropriately
* Update migration to create multiple Alertmanager configs
* Apply suggestions from code review
Co-authored-by: gotjosh <josue@grafana.com>
* replace mg.ClearMigrationEntry()
mg.ClearMigrationEntry() would create a new session.
This commit introduces a new migration for clearing an entry from migration log for replacing mg.ClearMigrationEntry() so that all dashboard alert migration operations will run inside the same transaction.
It adds also `SkipMigrationLog()` in Migrator interface for skipping adding an entry in the migration_log.
Co-authored-by: gotjosh <josue@grafana.com>
* Alerting: Send alerts to external Alertmanager(s)
Within this PR we're adding support for registering or unregistering
sending to a set of external alertmanagers. A few of the things that are
going are:
- Introduce a new table to hold "admin" (either org or global)
configuration we can change at runtime.
- A new periodic check that polls for this configuration and adjusts the
"senders" accordingly.
- Introduces a new concept of "senders" that are responsible for
shipping the alerts to the external Alertmanager(s). In a nutshell,
this is the Prometheus notifier (the one in charge of sending the alert)
mapped to a multi-tenant map.
There are a few code movements here and there but those are minor, I
tried to keep things intact as much as possible so that we could have an
easier diff.
* Alerting: Refactor `Run` of the scheduler
A bit of a refactor to make the diff easier to read for supporting
external Alertmanagers.
We'll introduce another routine that checks the database for
configuration and spawns other routines accordingly.
* Block the wait.
* Fix test
* initial attempt at automatic removal of stale states
* test case, need espected states
* finish unit test
* PR feedback
* still multiply by time.second
* pr feedback
* Alerting: deactivate an Alertmanager configuration
Implement DELETE /api/alertmanager/grafana/config/api/v1/alerts
by storing the default configuration which stops existing cnfiguration
from being in use.
* Apply suggestions from code review
* Expand the value of math and reduce expressions in annotations and labels
This commit makes it possible to use the values of reduce and math
expressions in annotations and labels via their RefIDs. It uses the
Stringer interface to ensure that "{{ $values.A }}" still prints the
value in decimal format while also making the labels for each RefID
available with "{{ $values.A.Labels }}" and the float64 value with
"{{ $values.A.Value }}"
* Alerting: Refactor state manager as a dependency
Within the scheduler, the state manager was being passed around a
certain number of functions. I've introduced it as a dependency to keep
the "service" interfaces as clean and homogeneous as possible.
This is relevant, because I'm going to introduce live reload of these
components as part of my next PR and it is better if dependencies are
self-contained.
* remove unused functions
* Fix a few more tests
* Make sure the `stateManager` is declared before the schedule
* Alerting: Allow __value__ label in notifications
was being removed by removePrivateItems
discoverd in #36020, but issue is not about that specifically
* __value__ label to __value_string__ annotation
and .ValueString extended property for notifications
* Fix deleting labels and annotations
* Add test
* Keep no data and error start if not provided
* Allow setting interval and for to zero during rule updates
* Alerting: Implement /status for the notification system
Implements the necessary plumbing to have a /status endpoint on the
notification system.
* Add API examples
* Update API specs
* Update prometheus/common dependency
Co-authored-by: Sofia Papagiannaki <sofia@grafana.com>
* Fix dashboard alert and nootifier migration for MySQL
* Fix POSTing Alertmanager configuration if no current configuration exists
in case the default configuration has not be stored yet
or has failed to get stored
* Change CreatedAt field type
* Alerting: Expand `{{$labels.xyz}}` template in labels and annotations
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix annotation not updating for same alert
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Alerting: Do not hard fail on templating errors in channels
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix review
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
When using mulit-dimensional Grafana managed alerts (e.g. SSE math) extract refIds values and labels so they can be shown in the notification and dashboards.
Threema Gateway supports two types of IDs: Basic IDs (where the
encryption is managed by the API server) and End-to-End IDs (where the
keys are managed by the user).
This plugin currently does not support End-to-End IDs (since it's much
more complex to implement, because the encryption needs to happen
locally). Add a few clarifications to the UI.
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
* starting to add eval logic.
* wip
* first version of test rule.
* reverted file.
* add info colum to result to show error or (with CC evalmatches)
* fix labels in evalmatch
* fix be test
* refactored using observables.
* moved widht/height div to outside panel rendere.
* adding docs api level.
* adding container styles to error div.
* increasing size of preview.
Co-authored-by: kyle <kyle@grafana.com>
* [Alerting]: forbid viewers for updating rules if viewers can edit
check for CanSave instead of CanEdit
* Clear ngalert tables when deleting the folder
* Apply suggestions from code review
* Log failure to check save permission
Co-authored-by: gotjosh <josue@grafana.com>
* Alerting: Opsgenie notification channel
This translate the opsgenie notification channel from the old alerting
system to the new alerting system with a few changes:
- The tag system has been replaced in favour of annotation.
- TBD
- TBD
Signed-off-by: Josue Abreu <josue@grafana.com>
* Fix template URL
* Bugfig: dont send resolved when autoClose is false
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix integration tests
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix URLs in all other channels
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Add discord notifier channel and test
* Correct payload
* remove print statement
* PR feedback and update due to changes in main
* Add discord notifier channel and test
* Correct payload
* remove print statement
* PR feedback and update due to changes in main
* update constructor and tests
* group imports sensibly
* Fix lint
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
* [Alerting]: Add sensugo notification channel
* Apply suggestions from code review
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
* Do not include labels with concatenated rule UID and names
* Modifications after syncing with main
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
When, and currently only when using a classic condition, evaluation information is added (which is like the EvalMatches from dashboard alerting).
This is returned via the API and can be included in notifications by reading the `__value__` label attached `.Alerts` in the template. It is a string.
* Alerting: Allow the notifier to log
The notifier upstream code uses go-kit as its logging library. The
grafana specific logger is not compatible with this API. In this PR, I
have created a wrapper that implements io.Writer to make them
compatible.
Instead put in package folder but with package name suffixed with _test
This enables code coverage within the pkg while still allow the tests to operate from external to package perspective (only exported things).
* nest cache by orgID, ruleUID, stateID
* update accessors to use new cache structure
* test and linter fixup
* fix panic
Co-authored-by: Kyle Brandt <kyle@grafana.com>
* add comment to identify what's going on with nested maps in cache
Co-authored-by: Kyle Brandt <kyle@grafana.com>
* Quota: Extend service to set limit on alerts
* Add test for applying quota to alert rules
* Apply suggestions from code review
Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>
* Get used alert quota only if naglert is enabled
* Set alert limit to zero if nglalert is not enabled
Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>
* Not complete, put migration behind env flag for now:
UALERT_MIG=iDidBackup
* Important to backup, and not expect the same DB to keep working until the env trigger is removed.
* Alerting: Migrate dashboard alert permissions
* Do not use imported models
* Change folder titles
Co-authored-by: Sofia Papagiannaki <papagian@users.noreply.github.com>
* Fix fialure when adding state annotations
* Fix get org rules API
Do not fail response if user has no access to view a namespace.
Do not include the namespace in the response instead.
* lint
* update swagger json files match datasourceUid change
underlying change made in https://github.com/grafana/grafana/pull/33282
* Document DatasourceUID field in AlertQuery model
* Run spec generation from inside a docker container
* Generate latest spec
Co-authored-by: Sofia Papagiannaki <sofia@grafana.com>
* set processing time
* merge labels and set on response
* use state cache for adding alerts to rules
* minor cleanup
* add support for NoData and Error results
* rename test
* bring in changes from other PRs tha have been merged
* pr feedback
* add integration test
* close state tracker cleanup on context.Done
* fixup test
* rename state tracker
* set EvaluationDuration on Result
* default labels set as constants
* separate cache and state from manager
* use RWMutex in cache
* set processing time
* merge labels and set on response
* use state cache for adding alerts to rules
* minor cleanup
* add support for NoData and Error results
* rename test
* bring in changes from other PRs tha have been merged
* pr feedback
* add integration test
* close state tracker cleanup on context.Done
* fixup test
* not those annotations
* AlertingNG: Fix TODOs in email notification channel
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Test fixup
* Remove the receiver field it is not needed for the email notification
Co-authored-by: Josue Abreu <josue@grafana.com>
* [Alerting]: Add alerting endpoint for Query Evaluation
* Fix passing down now parameter
* Add validations and test
* Fix eval queries and expressions test
* Add eval tests
