This change marks tests in the `sender` package that use an external
process as integration tests instead of unit tests. This speeds up the
package's unit tests by about 20 seconds.
This change also reduces the number of alert instances in the `store`
package's bulk write integration test from 20_000 to 10_000. This is
still enough to exercise the bulk-write code but speeds up the package
tests from about 250s to 130s.
Put together, integration tests go to about 160s while also speeding up
unit tests by 20s.
Before this change, the alerting provisioning system incorrectly used
the QuotaTarget to check if alerting's request quota had been reached.
The quota service requires the QuotaTargetSrv, which is what's
registered with the service at startup time. This is leading to errors
in the provisioning system.
The existing code uses `instanceof Error` to check for a `message` field on the thrown object. The objects that are thrown are never instances of the error interface. This change introduces a new type that extends Error so that the check works properly and displays a meaningful error message in the UI.
grafana.com/grafana/prometheus-alertmanager has been updated to a
version that fixes some bugs upstream. This change just updates that
dependency and a few shared ones.
Adds t.Run calls that supply the name of a receiver and its integrations
when the API response is being verified. This test has a rare flake and
it's difficult to track down without more context for the test failure.
Prior to this change, all alert instance writes and deletes happened
individually, in their own database transaction. This change batches up
writes or deletes for a given rule's evaluation loop into a single
transaction before applying it.
These new transactions are off by default, guarded by the feature toggle "alertingBigTransactions"
Before:
```
goos: darwin
goarch: arm64
pkg: github.com/grafana/grafana/pkg/services/ngalert/store
BenchmarkAlertInstanceOperations-8 398 2991381 ns/op 1133537 B/op 27703 allocs/op
--- BENCH: BenchmarkAlertInstanceOperations-8
util.go:127: alert definition: {orgID: 1, UID: FovKXiRVzm} with title: "an alert definition FTvFXmRVkz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: foDFXmRVkm} with title: "an alert definition fovFXmRVkz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: VQvFuigVkm} with title: "an alert definition VwDKXmR4kz" interval: 60 created
PASS
ok github.com/grafana/grafana/pkg/services/ngalert/store 1.619s
```
After:
```
goos: darwin
goarch: arm64
pkg: github.com/grafana/grafana/pkg/services/ngalert/store
BenchmarkAlertInstanceOperations-8 1440 816484 ns/op 352297 B/op 6529 allocs/op
--- BENCH: BenchmarkAlertInstanceOperations-8
util.go:127: alert definition: {orgID: 1, UID: 302r_igVzm} with title: "an alert definition q0h9lmR4zz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: 71hrlmR4km} with title: "an alert definition nJ29_mR4zz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: Cahr_mR4zm} with title: "an alert definition ja2rlmg4zz" interval: 60 created
PASS
ok github.com/grafana/grafana/pkg/services/ngalert/store 1.383s
```
So we cut time by about 75% and memory allocations by about 60% when
storing and deleting 100 instances.
Prior to this change, all alert instance writes and deletes happened
individually, in their own database transaction. This change batches up
writes or deletes for a given rule's evaluation loop into a single
transaction before applying it.
Before:
```
goos: darwin
goarch: arm64
pkg: github.com/grafana/grafana/pkg/services/ngalert/store
BenchmarkAlertInstanceOperations-8 398 2991381 ns/op 1133537 B/op 27703 allocs/op
--- BENCH: BenchmarkAlertInstanceOperations-8
util.go:127: alert definition: {orgID: 1, UID: FovKXiRVzm} with title: "an alert definition FTvFXmRVkz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: foDFXmRVkm} with title: "an alert definition fovFXmRVkz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: VQvFuigVkm} with title: "an alert definition VwDKXmR4kz" interval: 60 created
PASS
ok github.com/grafana/grafana/pkg/services/ngalert/store 1.619s
```
After:
```
goos: darwin
goarch: arm64
pkg: github.com/grafana/grafana/pkg/services/ngalert/store
BenchmarkAlertInstanceOperations-8 1440 816484 ns/op 352297 B/op 6529 allocs/op
--- BENCH: BenchmarkAlertInstanceOperations-8
util.go:127: alert definition: {orgID: 1, UID: 302r_igVzm} with title: "an alert definition q0h9lmR4zz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: 71hrlmR4km} with title: "an alert definition nJ29_mR4zz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: Cahr_mR4zm} with title: "an alert definition ja2rlmg4zz" interval: 60 created
PASS
ok github.com/grafana/grafana/pkg/services/ngalert/store 1.383s
```
So we cut time by about 75% and memory allocations by about 60% when
storing and deleting 100 instances.
This change also updates some of our tests so that they run successfully against postgreSQL - we were using random Int64s, but postgres integers, which our tables use, max out at 2^31-1
As stated in the documentation, Grafana will support the versions of our backend database options that are supported by their projects at the time that a Grafana release is issued.
This changes the API codegen template (controller-api.mustache) to simplify some names. When this package was created, most APIs "forked" to either a Grafana backend implementation or a "Lotex" remote implementation. As we have added APIs it's no longer the case. Provisioning, configuration, and testing APIs do not fork, and we are likely to add additional APIs that don't fork.
This change replaces {{classname}}ForkingService with {{classname}} for interface names, and names the concrete implementation {{classname}}Handler. It changes the implied implementation of a route handler from fork{{nickname}} to handle{{nickname}}. So PrometheusApiForkingService becomes PrometheusApi, ForkedPrometheusApi becomes PrometheusApiHandler and forkRouteGetGrafanaAlertStatuses becomes handleRouteGetGrafanaAlertStatuses
It also renames some files - APIs that do no forking go from forked_{{name}}.go to {{name}}.go and APIs that still fork go from forked_{{name}}.go to forking_{{name}}.go to capture the idea that those files a "doing forking" rather than "are a fork of something."
Signed-off-by: Joe Blubaugh <joe.blubaugh@grafana.com>
/api/admin/pause-all-alerts only takes effect for legacy alerts. This
change returns a 403 if it's called when legacy alerting is disabled.
Fixes#51729
Because some contact points only support a single image, and the order
of alerts sent to a contact point is not guaranteed, which alert rule's
image is attached to a notification can change from alert to alert. This
documents that behavior.
After migrating to unified alerting, users must explicitly allow rolling
back to legacy alerting by setting force_migration = true in config.
This updates the panic message to clarify why that's required and what
the consequences of rolling back will be.
Fixes#50469
The ng_alert_notification email template did not include templating for
linked or embedded images. This change updates that.
Additionally, this change supports embedding an image for each alert in
an email batch.
Fixes#50315
This change adds documentation about how the configuration of Grafana
Alerting affects the performance of the Grafana backend, and how to
control the load that Grafana Alerting generates.
Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>
The image file upload code as it is now simply doesn't work - it's
missing several important steps in the file upload process. There is
more information in the fixed issue as to the steps required.
After this change, screenshots will still be attached to slack messages
when external image storage is used with Grafana (an S3 bucket, for
example).
Fixes#50056
Tests that file bytes are correctly read from storage when an image is
available without a URL. Tests that not found and success cases are both
handled.
Adds three functions:
`withStoredImages` iterates over a list of models.Alerts, extracting a stored image's data from storage, if available, and executing a user-provided function.
`withStoredImage` does this for an image attached to a specific alert.
`openImage` finds and opens an image file on disk.
Moves `store.Image` to `models.Image`
Simplifies `channels.ImageStore` interface and updates notifiers that use it to use the simpler methods.
Updates all pkg/alert/notifier/channels to use withStoredImage routines.
If an image token is present in an alert instance, the email notifier will attempt to find a public URL for the image token. If found, it will add that to the email as the `ImageLink` field. If only local file data is available, the notifier will attach the file to the outgoing email using the `EmbeddedImage` field.
Adds an array of image_urls to the OpsGenie details field in a message, if image urls are available.
```json
{
"message": "Alert with Images!",
"details": {
"image_urls": ["http://www.example.com"]
}
}
```
If there are screenshot images with URLs, they will be attached to the
Microsoft teams notification in the first sections, as a slice of image
objects.
This change adds a field to state.State and models.AlertInstance
that indicate the "Reason" that an instance has its current state. This
helps us account for cases where the state is "Normal" but the
underlying evaluation returned "NoData" or "Error", for example.
Fixes#42606
Signed-off-by: Joe Blubaugh <joe.blubaugh@grafana.com>
This change extracts screenshot data from alert messages via a private annotation `__alertScreenshotToken__` and attaches a URL to a Slack message or uploads the data to an image upload endpoint if needed.
This change also implements a few foundational functions for use in other notifiers.
The State Manager will now take screenshots when an alert instance
switches to an Alerting or Resolved state.
Signed-off-by: Joe Blubaugh joe.blubaugh@grafana.com
This commit adds a pkg/services/screenshot package for taking and uploading screenshots of Grafana dashboards. It supports taking screenshots of both dashboards and individual panels within a dashboard, using the rendering service.
The screenshot package has the following services, most of which can be composed:
BrowserScreenshotService (Takes screenshots with headless Chrome)
CachableScreenshotService (Caches screenshots taken with another service such as BrowserScreenshotService)
NoopScreenshotService (A no-op screenshot service for tests)
SingleFlightScreenshotService (Prevents duplicate screenshots when taking screenshots of the same dashboard or panel in parallel)
ScreenshotUnavailableService (A screenshot service that returns ErrScreenshotsUnavailable)
UploadingScreenshotService (A screenshot service that uploads taken screenshots)
The screenshot package does not support wire dependency injection yet. ngalert constructs its own version of the service. See https://github.com/grafana/grafana/issues/49296
This PR also adds an ImageScreenshotService to ngAlert. This is used to take screenshots with a screenshotservice and then store their location reference for use by alert instances and notifiers.
Unlike dashboard queries, alerting queries were not correctly applying
custom headers to datasource queries. This change mimics the dashboard
query code to apply custom headers.
Fixes#44460
Signed-off-by: Joe Blubaugh <joe.blubaugh@grafana.com>
Invalid PostableSilences could be passed to the Alerting API - if they
are passed all the way down into the alertmanager data layer, they can
cause a panic. This change adds validation to avoid a panic in the
alertmanager.
Before this change, notifications generated by the Grafana Alertmanager
pointed to '/alerting/:ruleID/edit'. This change instead points them to
the view path '/alerting/grafana/:ruleID/view'. The view page has a
better UX, including timeseries display. It's also where many alert
state improvements will land in the next few versions of Grafana.
Fixes#45301
Signed-off-by: Joe Blubaugh <joe.blubaugh@grafana.com>
PR #47674 attempted to sort a read-only managed async array. This change
moves the sort logic to the fetch code so sort happens once on fetch, to
a mutable array, rather than trying on each render for an immutable
array.
Signed-off-by: Joe Blubaugh <joe.blubaugh@grafana.com>
This change sorts the State History list returned by the backend by the
id in addition to the timeEnd and time fields by which it is
already sorted. This results in a stable view of the State History
table.
Fixes#45873
Signed-off-by: Joe Blubaugh <joe.blubaugh@grafana.com>
This change makes it explicit in the UI that a notification policy with no matching labels matches all alerts that it processes. There are visual changes in both the Notification Policy editor, and in the Notification Policy table where matching columns are shown.
It's valid to have a notification policy with no label matchers attached. Such a policy matches all alerts sent to it. It's a common stumbling block for users. Where are all my alerts going?
Co-authored-by: gillesdemey <gilles.de.mey@gmail.com>
This test of silence cleanup was flaky because of its use of real wall
time. In CI environments with slow execution, delays could cause the
test to fail. This change mitigates the problem by increasing the end time of
silences in the test.
After Prometheus merges this PR: https://github.com/prometheus/alertmanager/pull/2867
we can make the test fully deterministic by using a fake clock.
Fixes#47470
Signed-off-by: Joe Blubaugh <joe.blubaugh@grafana.com>
FilterInput escapes all input strings for special characters that might
be used in a RegExp by calling escapeStringForRegex, and using
unEscapeStringFromRegex for display. Both of these functions used
string.prototype.replace() for escaping. replace() only replaces the
first occurence of a search string unless called with a global RegExp.
The output of these functions was not necessarily safe to compile into a
RegExp literal.
This change creates RegExps for escapeStringForRegex and
unEscapeStringFromRegex to match all occurrences of the special
characters instead of just their first occurrence. This makes a variety
of strings safe for RegExp compilation.
Joe Blubaugh
