* Fix flaky tests in several notifiers
- Non-mocked time in sensu go tests
- Close server in Slack tests
- Use a mutex for writing responses in the fake slack server
* Remove mutex at the fake slack server
Fixes a panic that would ocurr as we proxy 4xx responses. When this happens and the content type of the response is JSON we try to check if the response has a "message" key. Then, we assume that the key will contain a value of string but we don't take into account that this value can potentially be `null`.
This adds a type assertion check to to this assumption so that we can keep the original JSON body as the response if we're unable to extract an `message`.
* Fix Annotation creation
- Remove validation of panelID, now annotations are created irrespective on whether they're attached to a panel or not.
- Alwasy attach the annotation to an AlertID
* Fix annotation creation
* fix tests
* Alerting: Clear alerting rule evaluation errors after intermittent failures
When an alert transitioned in a way that `alerting -> error -> (alerting|nodata)`, the error provided by the `error` state would never be cleared thus the API and UI would show the health as an error.
* Alerting: Validate contact point configuration during the migration
This minimises the chances of generating broken configuration as part of the migration. Originally, we wanted to generate it and not produce a hard stop in Grafana but this strategy has the chance to avoid delivering notifications for our users.
We now think it's better to hard stop the migration and let the user take care of resolving the configuration manually.
* Alerting: Remove invalid Slack URL as we migrate notification channels
Grafana will accept any type of utf8 valid string as the Slack URL and will simply fail as we try to deliver the notification of the channel. The Alertmanager will fail to apply a configuration if the URL of the Slack Receiver is invalid.
This change takes that into account by removing the URL for the receiver as we migrate notification channels that do not pass the url validation. As we assume the notification was not being delivered to being with.
* Add a log line when we modify the channel
Co-authored-by: Yuriy Tseretyan <yuriy.tseretyan@grafana.com>
* Alerting: Fixes a bug when trying to sync broken alertmanager config
Broken alertmanager configuration has the potential to be introduced as part of a migration e.g. due to incompatible data between what grafana accepts and what the Alertmanager expects. When this happens, we expect an eventually consistent behaviour where we'll keep trying to apply the configuration until it works.
As part of change in https://github.com/grafana/grafana/pull/39237 we introduced a regression that modified this behaviour and instead tried to create a new Alertmanager for that organization everytime, which eventually ended up in a panic due to a duplicate metrics being registered.
This PR fixes that and introduces a test to catch further regressions.
* Remove disable orgs
Remove validation for labels to be accepted in the Alertmanager, This helps with datasources that produce non-compatible labels.
Adds an "object_matchers" to alert manager routers so we can support labels names with extended characters beyond prometheus/openmetrics. It only does this for the internal Grafana managed Alert Manager.
This requires a change to alert manager, so for now we use grafana/alertmanager which is a slight fork, with the intention of going back to upstream.
The frontend handles the migration of "matchers" -> "object_matchers" when the route is edited and saved. Once this is done, downgrades will not work old versions will not recognize the "object_matchers".
Co-authored-by: Kyle Brandt <kyle@grafana.com>
Co-authored-by: Nathan Rodman <nathanrodman@gmail.com>
* Alerting: Refactor & fix unified alerting metrics structure
Fixes and refactors the metrics structure we have for the ngalert service. Now, each component has its own metric struct that includes the JUST the metrics it uses. Additionally, I have fixed the configuration metrics and added new metrics to determine if we have discovered and started all the necessary configurations of an instance.
This allows us to alert on `grafana_alerting_discovered_configurations - grafana_alerting_active_configurations != 0` to know whether an alertmanager instance did not start successfully.
* Alerting: Persist notification log and silences to the database
This removes the dependency of having persistent disk to run grafana alerting. Instead of regularly flushing the notification log and silences to disk we now flush the binary content of those files to the database encoded as a base64 string.
* Alerting: Fix alert flapping in the alertmanager
fixes a bug that caused Alerts that are evaluated at low intervals (sub 1 minute), to flap in the Alertmanager.
Mostly due to a combination of `EndsAt` and resend delay.
The Alertmanager uses `EndsAt` as a heuristic to know whenever it should resolve a firing alert, in the case that it hasn't heard
back from the alert generation system.
Because grafana sent the alert with an `EndsAt` which is equal to the `For` of the alert itself,
and we had a hard-coded 1 minute re-send delay (only applicable to firing alerts) this meant that a firing alert would resolve in the Alertmanager before we re-notify that it still firing.
This commit, increases the `EndsAt` by 3x the the resend delay or alert interval (depending on which one is higher). The resendDelay has been decreased to 30 seconds.
* Alerting: Expose discovered and dropped Alertmanagers
Exposes the API for discovered and dropped Alertmanagers.
* make admin config poll interval configurable
* update after rebase
* wordsmith
* More wordsmithing
* change name of the config
* settings package too
* Alerting: Send alerts to external Alertmanager(s)
Within this PR we're adding support for registering or unregistering
sending to a set of external alertmanagers. A few of the things that are
going are:
- Introduce a new table to hold "admin" (either org or global)
configuration we can change at runtime.
- A new periodic check that polls for this configuration and adjusts the
"senders" accordingly.
- Introduces a new concept of "senders" that are responsible for
shipping the alerts to the external Alertmanager(s). In a nutshell,
this is the Prometheus notifier (the one in charge of sending the alert)
mapped to a multi-tenant map.
There are a few code movements here and there but those are minor, I
tried to keep things intact as much as possible so that we could have an
easier diff.
* Alerting: Refactor `Run` of the scheduler
A bit of a refactor to make the diff easier to read for supporting
external Alertmanagers.
We'll introduce another routine that checks the database for
configuration and spawns other routines accordingly.
* Block the wait.
* Fix test
* Alerting: Refactor state manager as a dependency
Within the scheduler, the state manager was being passed around a
certain number of functions. I've introduced it as a dependency to keep
the "service" interfaces as clean and homogeneous as possible.
This is relevant, because I'm going to introduce live reload of these
components as part of my next PR and it is better if dependencies are
self-contained.
* remove unused functions
* Fix a few more tests
* Make sure the `stateManager` is declared before the schedule
* Alerting: Implement /status for the notification system
Implements the necessary plumbing to have a /status endpoint on the
notification system.
* Add API examples
* Update API specs
* Update prometheus/common dependency
Co-authored-by: Sofia Papagiannaki <sofia@grafana.com>
* Alerting: Opsgenie notification channel
This translate the opsgenie notification channel from the old alerting
system to the new alerting system with a few changes:
- The tag system has been replaced in favour of annotation.
- TBD
- TBD
Signed-off-by: Josue Abreu <josue@grafana.com>
* Fix template URL
* Bugfig: dont send resolved when autoClose is false
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix integration tests
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Fix URLs in all other channels
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
* Alerting: Allow the notifier to log
The notifier upstream code uses go-kit as its logging library. The
grafana specific logger is not compatible with this API. In this PR, I
have created a wrapper that implements io.Writer to make them
compatible.
A set of fixes for the GET alert and groups endpoints.
- First, is the fact that the default values where not being for the query params. I've introduced a new method in the Grafana context that allow us to do this.
- Second, is the fact that alerts were never being transitioned to active. To my surprise this is actually done by the inhibitor in the pipeline - if an alert is not muted, or inhibited then it's active.
- Third, I have added an integration test to cover for regressions.
Signed-off-by: Josue Abreu <josue@grafana.com>
* Alerting: Use a default configuration and periodically poll for new ones
Use a default configuration to make sure we always start the grafana
instance. Then, regularly poll for new ones.
I've also made sure that failures to apply configuration do not stop the
Grafana server but instead keep polling until it is a success.
* Alerting: Cleanup and move legacy to a legacy file
A quick cleanup of the ngalert/api directory, optimising for an easy
removal of what is will be considered legacy at some point. A quick
summary of what's done is:
- Add a prefix `generated` prefix to files that are auto-generated by
our swagger definitions.
- Create a legacy file to place all the legacy API routes implementation
and helpers. Deleting files that where no longer needed after this
move.
- Rename the `lotex` file to `lotex_ruler`
- Adding a couple of comments here and there.
With this, I hope to organise our code in this directory a bit better
given there's a lot going on.
* Alerting: Introduce the silencing interface
The operations introduced are:
- Listing silences
- Retrieving an specific silence
- Deleting a silence
- Creating a silence
Signed-off-by: Josue Abreu <josue@grafana.com>
* Add a comment to listing silences
* Update to upstream alertmanager
* Remove copied code from the Alertmanager
* Alerting: Add database table for persisting alerting configuration
* Fix the linter
* Address review comments
* Don't split templates and configuration
It is already bundled together as part of a of the API so might as well
marshall it directly.
The arching goal of this commit is to enable single user
synchronisation with LDAP. Also, it included minor fixes of style,
error messages and minor bug fixing.
The changes are:
- bug: The `multildap` package has its own errors when the user is
not found. We fixed the conditional branch on this error by asserting
on the `multildap` errors as opposed to the `ldap` one
- bug: The previous interface usage of `RevokeAllUserTokens` did not
work as expected. This replaces the manual injection of the service by
leveraging the service injected as part of the `server` struct.
- chore: Better error messages around not finding the user in LDAP.
- fix: Enable the single sync button and disable it when we receive an
error from LDAP. Please note, that you can enable it by dispatching
the error. This allows you to try again without having to reload the
page.
- fix: Move the sync info to the top, then move the sync button above
that information and clearfix to have more harmony with the UI.
* Dependency: Bump our SAML Library to the latest version
Fixes a non-obligatory check for RSA encrypted assertions. Previously they required a certificate embedded in the assertion, this is not mandatory according to the SAML standard.
* Fix: Handle empty nested keys on YAML provisioning datasources
As we provision a datasource via a YAML file, we attempt to transform the
file into sensible Go types that the provisioning code can use.
While this happens, there is a chance some of the keys nested within
the YAML array are empty.
This fix allows the YAML parser to handle empty keys by null checking
the return of `reflect.TypeOf` which according to the documentation:
> TypeOf returns the reflection Type that represents the dynamic type of i. If i is a nil interface value, TypeOf returns nil.
Can return nil.
* Add tests
* Docs: Add information regarding the LDAP Debug View
* Add instructions to compress image when adding them to the documentation
* Add a troubleshooting section to the docs documentation
* API: Add `createdAt` and `updatedAt` to api/users/lookup
In the past, we have added both `updatedAt` (#19004) and `createdAt` (#19475) to /api/users/:id
Turns out, api/users/lookup uses the same DTO for both. This fixes the serialization of both `createdAt` and `updatedAt`for this endpoint.
Also, adds a test to ensure no further regressions.
* Updated API documentation
* Multi-LDAP: Do not fail-fast on invalid credentials
When configuring LDAP authentication, it is very common to have multiple
servers configured. When using user bind (authenticating with LDAP using
the same credentials as the user authenticating to Grafana) we don't
expect all the users to be on all LDAP servers.
Because of this use-case, we should not fail-fast when authenticating on
multiple LDAP server configurations. Instead, we should continue to try
the credentials with the next LDAP server configured.
Fixes#19066
* LDAP: Show all LDAP groups
* Use the returned LDAP groups as the reference when debugging LDAP
We need to use the LDAP groups returned as the main reference for
assuming what we were able to match and what wasn't. Before, we were
using the configured groups in LDAP TOML configuration file.
* s/User name/Username
* Add a title to for the LDAP mapping results
* LDAP: UI Updates to debug view
* LDAP: Make it explicit when we weren't able to match teams
* LDAP: Allow an user to be synchronised against LDAP
This PR introduces the /ldap/sync/:id endpoint. It allows a user to be synchronized against LDAP on demand.
A few things to note are:
LDAP needs to be enabled for the sync to work
It only works against users that originally authenticated against LDAP
If the user is the Grafana admin and it needs to be disabled - it will not sync the information
Includes a tiny refactor that favours the JSONEq assertion helper instead of manually parsing JSON strings.
* API: Add `updatedAt` to api/users/:id
This adds the timestamp of when a particular user was last updated to
the `api/users/:id` endpoint.
This helps our administrators understand when was the user information last
updated. Particularly when it comes from external systems e.g. LDAP
Adds the definition of `GetTeamsForLDAPGroupCommand` which handles the lookup of team information based on LDAP groupDNs.
This is an Enterprise only feature. To diferentiate,a response will contain the `team` key as `null` on OSS while on Enterprise the key will contain an empty array `[]` when no teams are found.
* LDAP: Add API endpoint to query the LDAP server(s) status|
This endpoint returns the current status(es) of the configured LDAP server(s).
The status of each server is verified by dialling and if no error is returned we assume the server is operational.
This is the last piece I'll produce as an API before moving into #18759 and see the view come to life.
* Move the ReloadLDAPCfg function to the debug file
Appears to be a better suite place for this.
* LDAP: Return the server information when we find a specific user
We allow you to specify multiple LDAP servers as part of LDAP authentication integration. As part of searching for specific users, we need to understand from which server they come from. Returning the server configuration as part of the search will help us do two things:
- Understand in which server we found the user
- Have access the groups specified as part of the server configuration
* LDAP: Adds the /api/admin/ldap/:username endpoint
This endpoint returns a user found within the configured LDAP server(s). Moreso, it provides the mapping information for the user to help administrators understand how the users would be created within Grafana based on the current configuration.
No changes are executed or saved to the database, this is all an in-memory representation of how the final result would look like.
* Fix: Avoid glob of single-value array variables
Based on our current implementation of templates, when multi-select
variables are part of a dashboard query the default/fallback formatting option is `glob`.
Some data sources do not support glob (e.g. metrics.{a}.* instead of
metrics.a.*) for single variable queries. This behaviour breaks dashboards.
This commit introduces an alternative formatting option where globing is avoided if it's there is only one value as part of the query variable.
This means, queries previously formatted as `query=metrics.{a}.*.*`, are
now formatted as `query=metrics.a.*.*`. However, queries formatted as
`query=metrics.{a,b}.*.*` continue to be as is.
* Imperative mood
The English language suggests that when giving out instructions or considerations is best to use imperative mood as the writing style. Done as part of this commit:
- some worth smithing to match this style
- changed the format to match the fact that this is commonly read as part of a text box when opening up your PRs
- and extended a few descriptions to make it more friendly for newcomers
* Auth Proxy: Include additional headers as part of the cache key
Auth proxy has support to send additional user attributes as part of the
authentication flow. These attributes (e.g. Groups) need to be monitored
as part of the process in case of change.
This commit changes the way we compute the cache key to include all of the
attributes sent as part of the authentication request. That way, if we
change any user attributes we'll upsert the user information.
* Update defaults.ini and sample.ini with the SAML assertion mapping
fields
* Document Grafana's ability to map ACS attributes while a Grafana user is created
* chore: mirror the Dockerhub readme
Seems like the dockerhub readme is out of sync with this one. Let's update this one to make them identical.
* chore: add changes part of 6.3.0-beta1 to the docker image README
* chore: add the date to the 6.3.0-beta1 changes in the CHANGELOG
* docs: Link to SAML docs and document configuration options
- Document configuration options `defaults.ini` and `sample.ini`
- Add the SAML documentation
- Link to the SAML documentation from "what's new in 6.3"
* Add SAML configuration options
* Add crewjam/saml as a depdency
Needed as part of the enterprise SAML integration.
* Vendor github.com/stretchr/testify/require
The package require implements the same assertions as the `assert` package but stops test execution when a test fails.
* Feature: Introduce a grafana-cli wrapper
When our users install the *nix packed version of grafana, tendency is to use the services and scripts installed as part of the package for grafana-server. These leverage the default configuration options by specifying the several default paths.
This introduces a similar approach for the grafana-cli binary. We exposed it through a wrapper to ensure a proper configuration is in place. To enable that, we add the .real suffix to the original binary (grafana-cli.real) and then use a bash script named grafana-cli as the wrapper.
* Make the config and homepath flags global
* Introduce `configOverrides` as a global flag
This flag allows us to pass configuration overrides as a string.
The string follows the convention of configuration arguments separated by a space e.g. "cfg:default.paths.data=/dev/nullX cfg:default.paths.logs=/dev/nullX"
Also, it is backwards compatible with similar the previous configuration method through tailing arguments. Tailing arguments take presedence over the configuration options string.
* Only log configuration information in debug mode
* Move the grafana-cli binary to $GRAFANA_HOME/bin
As part of the package install process, we copy all the release files and
directories into the grafana home directory. This includes the /bin folder
from where we copied the binaries into their respective destinations.
After that, the /bin folder gets deleted as we don't want to keep
duplicates of the binaries around.
As part of this commit, we moved the re-creation of /bin within
grafana-home and the copy of the original binary (again) after the
folder gets deleted.
* Don't display changelog category title when no items
The output of the changelog is meant to be copy/pasted with ease. When a
changelog category does not contain items is better to not display title
at all thus avoiding having the manually modify the output as we include
it in the steps of the process.
* Introduce a CLI task to close milestones whilst doing a Grafana release
As part of a Grafana release, we need to eventually close the GitHub
milestone to indicate is done and remove all the cherry-pick labels from
issues/prs within the milestone to avoid our cherry-pick CLI command to
pick them up on the next release.
* Abstract the GitHub client into a module
* Introduce `GitHubClient` to all CLI tasks
`grafana-cli` uses the third-party library to define the flags and not
the standard library. Using `flag.Parse` conflicts with the defined
flags from our third-party library.
In the case where `flag.Parse` is used, the CLI assumes that definitions
provided are not needed and will not define them; producing errors of
the kind `flag provided but not defined --example-flag`.
Using the context to read any arguments (including flags) is the
recommended approach by the third-party library.
grafana-cli should allow configuration overrides to be received from the command line. e.g.
```
grafana-cli admin reset-password cfg:default.paths.logs=custom/log/directory/
```
Seems like we missed the inclusion of `flag.Parse` as we run the command, to be able to consume them.
Additionally, it'll be useful for the user to know whenever these are being overriden or not - hence the addition of logging the configuration to be used as we run the command.
gotjosh
