Commit Graph

1373 Commits

Author SHA1 Message Date
gotjosh
f3f3fcc727 Alerting: Introduces /api/v1/ngalert/alertmanagers to expose discovered and dropped Alertmanager(s) (#37632)
* Alerting: Expose discovered and dropped Alertmanagers

Exposes the API for discovered and dropped Alertmanagers.

* make admin config poll interval configurable

* update after rebase

* wordsmith

* More wordsmithing

* change name of the config

* settings package too
2021-08-13 13:14:36 +01:00
Kyle Brandt
aef67994a1 Annotations: Fix alerting annotation coloring (#37412)
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2021-08-12 09:37:54 -07:00
Sofia Papagiannaki
04d5dcb7c8 Alerting: modify DB table, accessors and migration to restrict org access (#37414)
* Alerting: modify table and accessors to limit org access appropriately

* Update migration to create multiple Alertmanager configs

* Apply suggestions from code review

Co-authored-by: gotjosh <josue@grafana.com>

* replace mg.ClearMigrationEntry()

mg.ClearMigrationEntry() would create a new session.
This commit introduces a new migration for clearing an entry from migration log for replacing  mg.ClearMigrationEntry() so that all dashboard alert migration operations will run inside the same transaction.
It adds also `SkipMigrationLog()` in Migrator interface for skipping adding an entry in the migration_log.

Co-authored-by: gotjosh <josue@grafana.com>
2021-08-12 16:04:09 +03:00
SLAMA
5986d99f51 Alerting frontend : fix line notifier (#37744)
- Fixes #37425
- change `line` type string to uppercase
2021-08-10 18:59:53 +01:00
gotjosh
f83cd401e5 Alerting: Send alerts to external Alertmanager(s) (#37298)
* Alerting: Send alerts to external Alertmanager(s)

Within this PR we're adding support for registering or unregistering
sending to a set of external alertmanagers. A few of the things that are
going are:

- Introduce a new table to hold "admin" (either org or global)
  configuration we can change at runtime.
- A new periodic check that polls for this configuration and adjusts the
  "senders" accordingly.
- Introduces a new concept of "senders" that are responsible for
  shipping the alerts to the external Alertmanager(s). In a nutshell,
this is the Prometheus notifier (the one in charge of sending the alert)
mapped to a multi-tenant map.

There are a few code movements here and there but those are minor, I
tried to keep things intact as much as possible so that we could have an
easier diff.
2021-08-06 13:06:56 +01:00
Kyle Brandt
aa904a5a04 NGAlert: Send resolve signal to alertmanager on alerting -> Normal (#37363) 2021-07-29 20:29:17 +02:00
gotjosh
442a6677fc Alerting: Refactor Run of the scheduler (#37157)
* Alerting: Refactor `Run` of the scheduler

A bit of a refactor to make the diff easier to read for supporting
external Alertmanagers.

We'll introduce another routine that checks the database for
configuration and spawns other routines accordingly.

* Block the wait.

* Fix test
2021-07-27 11:52:59 +01:00
Ganesh Vernekar
e8ac802e4f Alerting: Remove unused fields in Pagerduty struct (#37198)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-07-27 10:41:48 +05:30
David Parrott
b5f464412d Alerting: automatically remove stale alerting states (#36767)
* initial attempt at automatic removal of stale states

* test case, need espected states

* finish unit test

* PR feedback

* still multiply by time.second

* pr feedback
2021-07-26 18:12:04 +02:00
Ganesh Vernekar
a65975cca0 Alerting: Remove the fixed wait for notification delivery (#37203)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-07-26 15:15:09 +02:00
George Robinson
2f4c893cf3 Expand the value string in annotations and labels of alerts (#37051)
This commit makes it possible to use the value string in
annotations and labels for alerts with "{{ $value }}"
2021-07-22 15:20:44 +01:00
Sofia Papagiannaki
7815ed511f Alerting: Refactor API endpoints for fetching alert rules (#37055)
* Refactor ruler API endpoint for listing rules

* Refactor prometheus API endpoint for listing rules

* Update HTTP API docs
2021-07-22 09:53:14 +03:00
David Parrott
fa0bed7118 do not over write alerting rule duration (#36930) 2021-07-20 11:49:35 +05:30
Djairho Geuens
4cadbba686 Email: Allow configuration of content types for email notifications (#34530)
* Alerting: Allow configuration of content types for email notifications

* Fix lint error

* Improves email templates

* Improve configuration documentation

Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>

* Improve code comments

Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>

* Improve configuration documentation

Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>

* Improve email template

* Remove unnecessary predeclaration

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* Adds handling for unrecognized content type

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* Move utility function outside of util package

* Fixes syntax

* Remove unused package

* Fix lint error

* improve email templates

* Fix test

* Alerting: Allow configuration of content types for email notifications

* Fix lint error

* Improves email templates

* Improve configuration documentation

Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>

* Improve code comments

Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>

* Improve configuration documentation

Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>

* Improve email template

* Remove unnecessary predeclaration

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* Adds handling for unrecognized content type

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

* Move utility function outside of util package

* Fixes syntax

* Remove unused package

* Fix lint error

* improve email templates

* Fix test

* Fix comment style

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>

* Fix template formatting

* Add test and improve error handling

* Fix test

* Fix formatting

* Fix formatting

* Improve documentation and regenerates txt template

* Update docs/sources/administration/configuration.md

Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>

Co-authored-by: Djairho Geuens <djairho.geuens@ae.be>
Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
2021-07-19 13:31:51 +03:00
Sofia Papagiannaki
f308ba91e3 Alerting: Improve receiver initialisation errors (#36814)
* Alerting: Improve receiver initialisation errors
2021-07-19 11:58:35 +03:00
Sofia Papagiannaki
afe6e793ff Alerting: deactivate an Alertmanager configuration (#36794)
* Alerting: deactivate an Alertmanager configuration

Implement DELETE /api/alertmanager/grafana/config/api/v1/alerts
by storing the default configuration which stops existing cnfiguration
from being in use.

* Apply suggestions from code review
2021-07-16 20:07:31 +03:00
George Robinson
456dac1303 Expand the value of math and reduce expressions in annotations and labels (#36611)
* Expand the value of math and reduce expressions in annotations and labels

This commit makes it possible to use the values of reduce and math
expressions in annotations and labels via their RefIDs. It uses the
Stringer interface to ensure that "{{ $values.A }}" still prints the
value in decimal format while also making the labels for each RefID
available with "{{ $values.A.Labels }}" and the float64 value with
"{{ $values.A.Value }}"
2021-07-15 13:10:56 +01:00
David Parrott
19f18bcecc Alerting: annotation on state change (#36535)
* WIP

* Add annotation on alert state change

* move annotation creation to manager

* praise the linter!

* add debug msg when creating annotation
2021-07-13 09:50:10 -07:00
David Parrott
310d3ebe3d change template expansion missing value handling (#36679) 2021-07-13 06:57:18 -07:00
Ganesh Vernekar
8efe1856e2 Alerting: A better and cleaner way to know if Alertmanager is initialised (#36659)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-07-12 18:53:01 +05:30
Ganesh Vernekar
e19c690426 Alerting: Fix potential panic in Alertmanager when starting up (#36562)
* Alerting: Fix potential panic in Alertmanager when starting up

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix reviews

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-07-12 11:15:16 +05:30
Andrej Ocenas
ea2ba06b93 CloudWatch/Logs: Fix log alerts in new unified alerting (#36558)
* Pass FromAlert header from new alerting

* Add better error messages
2021-07-09 13:43:22 +02:00
Ganesh Vernekar
94d2520a84 Alerting: Allow space in label and annotation names (#36549)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-07-08 18:26:09 +05:30
gotjosh
a86ad1190c Alerting: Refactor state manager as a dependency (#36513)
* Alerting: Refactor state manager as a dependency

Within the scheduler, the state manager was being passed around a
certain number of functions. I've introduced it as a dependency to keep
the "service" interfaces as clean and homogeneous as possible.

This is relevant, because I'm going to introduce live reload of these
components as part of my next PR and it is better if dependencies are
self-contained.

* remove unused functions

* Fix a few more tests

* Make sure the `stateManager` is declared before the schedule
2021-07-07 17:18:31 +01:00
Sofia Papagiannaki
fc90d47863 Alerting API: Restrict access to Alertmanager configuration (#36507)
* Alerting API: Restrict access to Alertmanager configuration to viewers
2021-07-07 16:29:18 +03:00
Sofia Papagiannaki
8a3edf280e Alerting: Fix prometheus API to check folder permissions (#36301) 2021-07-05 10:49:14 +03:00
Ganesh Vernekar
8fe58fc2e3 Alerting: Add additional newlines to Microsoft Teams notification message where necessary (#36126)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-06-24 20:52:00 +05:30
Kyle Brandt
19f764739b Alerting: Change __value__ label to __value_string__ annotation and add ValueString variable in notifications (#36032)
* Alerting: Allow __value__ label in notifications
was being removed by removePrivateItems
discoverd in #36020, but issue is not about that specifically

* __value__ label to __value_string__ annotation
and .ValueString extended property for notifications
2021-06-24 12:45:49 +05:30
Ganesh Vernekar
9a5c1f06df Alerting: Template all possible variables in notification channels (#35749)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-06-22 15:12:54 +05:30
Ganesh Vernekar
33d6e11175 Alerting: Decouple default template from channel tests (#35239)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-06-21 07:59:09 +05:30
David Parrott
4732f832f7 Alerting: recalculate EndsAt (#35830)
* setEndsAt

* one more test case

* add should clause to tests
2021-06-17 10:01:46 -07:00
Ganesh Vernekar
dcd4bf1615 Alerting: Fill the empty GeneratorURL (#35740)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-06-16 15:34:12 +05:30
Sofia Papagiannaki
e5a5b8e3fe Alerting: Fix updating alert rule properties with missing/zero values (#35512)
* Fix deleting labels and annotations

* Add test

* Keep no data and error start if not provided

* Allow setting interval and for to zero during rule updates
2021-06-15 20:55:25 +03:00
Sofia Papagiannaki
abe35c8c01 Alerting: Add error recovery during rule evaluations (#35450)
* Alerting: Eval recovery after query failure

* Apply suggestions from code review
2021-06-15 19:30:21 +03:00
gotjosh
f7ed35336d Alerting: Implement /status for the notification system (#33227)
* Alerting: Implement /status for the notification system

Implements the necessary plumbing to have a /status endpoint on the
notification system.

* Add API examples

* Update API specs

* Update prometheus/common dependency

Co-authored-by: Sofia Papagiannaki <sofia@grafana.com>
2021-06-15 19:14:02 +03:00
Sofia Papagiannaki
fba90b8f9b Alerting: Recact html responses (#35277) 2021-06-04 20:57:24 +03:00
Sofia Papagiannaki
8cda1f5153 Alerting: Allow rules with same title across folders (#35270)
* Alerting: Allow rules with same title across folders

* Add test
2021-06-04 20:45:26 +03:00
Sofia Papagiannaki
15c55b0115 Alerting: Fix notification channel migration and handle case when Alertmanager default configuration is absent (#35086)
* Fix dashboard alert and nootifier migration for MySQL

* Fix POSTing Alertmanager configuration if no current configuration exists

in case the default configuration has not be stored yet
or has failed to get stored

* Change CreatedAt field type
2021-06-04 15:52:41 +03:00
Ganesh Vernekar
8417088969 Alerting: Expand {{$labels.xyz}} template in labels and annotations (#35159)
* Alerting: Expand `{{$labels.xyz}}` template in labels and annotations

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix annotation not updating for same alert

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-06-03 19:24:36 +02:00
Ganesh Vernekar
a30e60a0b8 Alerting: Do not hard fail on templating errors in channels (#35165)
* Alerting: Do not hard fail on templating errors in channels

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix review

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-06-03 19:39:32 +05:30
Ganesh Vernekar
a23674ef99 Alerting: Migrate tags as labels and not annotations (#34990)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-05-31 19:47:17 +05:30
Sofia Papagiannaki
355be158b7 [Alerting]: fix/cleanup API examples (#34588) 2021-05-31 11:18:29 +03:00
Chip Wolf ‮
badec6c6ad Alerting: Add support for configuring avatar URL for the Discord notifier (#33355)
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
2021-05-28 23:00:21 +02:00
Owen Diehl
cc38613ba4 alerting: fixes per-receiver metric cardinality (#34915) 2021-05-28 12:31:23 -04:00
Owen Diehl
9aca032d10 Alerting/consistent api errors (#34858)
* consolidates alertmanager api errors

* util & testing consistent errors

* consistent errors for rest of ngalert apis

* updates expected errors in testware

* bump ci

* linting

* unrelated: dashboard.go lint
2021-05-28 11:55:03 -04:00
Kyle Brandt
b47e7d12e6 Alerting: Extract values from MD expr alerts (#34757)
When using mulit-dimensional Grafana managed alerts (e.g. SSE math) extract refIds values and labels so they can be shown in the notification and dashboards.
2021-05-28 11:04:20 -04:00
Danilo Bargen
83a83de10a Clarify that Threema Gateway Alerts support only Basic IDs (#34828)
Threema Gateway supports two types of IDs: Basic IDs (where the
encryption is managed by the API server) and End-to-End IDs (where the
keys are managed by the user).

This plugin currently does not support End-to-End IDs (since it's much
more complex to implement, because the encryption needs to happen
locally). Add a few clarifications to the UI.

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2021-05-28 08:54:55 +02:00
Domas
347273cdea Alerting: check upstream response content type in lotex proxy (#34760) 2021-05-27 14:12:29 +03:00
David Parrott
20d356947c set state correctly and test (#34680) 2021-05-26 11:37:42 -07:00
Ganesh Vernekar
d69c21acb6 NGAlert: Update the default template to include more URLs (#34715)
* NGAlert: Update the default template to include more URLs

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix tests

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-05-26 16:49:39 +02:00