Commit Graph
100 Commits
Author SHA1 Message Date
Alexander WeaverandJean-Philippe Quéméner 6c5e94095d Alerting: Scheduler and registry handle rules by an interface (#84044)
* export Evaluation

* Export Evaluation

* Export RuleVersionAndPauseStatus

* export Eval, create interface

* Export update and add to interface

* Export Stop and Run and add to interface

* Registry and scheduler use rule by interface and not concrete type

* Update factory to use interface, update tests to work over public API rather than writing to channels directly

* Rename map in registry

* Rename getOrCreateInfo to not reference a specific implementation

* Genericize alertRuleInfoRegistry into ruleRegistry

* Rename alertRuleInfo to alertRule

* Comments on interface

* Update pkg/services/ngalert/schedule/schedule.go

Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>

---------

Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>
2024-03-11 22:57:38 +02:00
Alexander Weaver 201f5d3ac9 Alerting: Extract large closures in ruleRoutine (#84035)
* extract notify

* extract resetState

* move evaluate metrics inside evaluate

* split out evaluate
2024-03-06 16:39:23 -06:00
Alexander WeaverandRyan McKinley 7a171fd14a Regenerate openapidocs at 1.21.8 to match ci (#84037)
* Regenerate openapidocs at 1.21.8 to match ci

* Adjust trigger to work on the actual outputted files

* Also put go.mod and go.sum in the triggers

* manually fix

* Make an arbitrary change rather than touching the trigger to force a run

* Drop all triggers - run all the time

* Print diff - taken from @papagian's PR

* Manual fixes to swagger doc

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2024-03-06 16:08:45 -06:00
Alexander Weaver d5fda06147 Alerting: Decouple rule routine from scheduler (#84018)
* create rule factory for more complicated dep injection into rules

* Rules get direct access to metrics, logs, traces utilities, use factory in tests

* Use clock internal to rule

* Use sender, statemanager, evalfactory directly

* evalApplied and stopApplied

* use schedulableAlertRules behind interface

* loaded metrics reader

* 3 relevant config options

* Drop unused scheduler parameter

* Rename ruleRoutine to run

* Update READMED

* Handle long parameter lists

* remove dead branch
2024-03-06 13:44:53 -06:00
Alexander Weaver 1bb38e8f95 Alerting: Move ruleRoutine to be a method on ruleInfo (#83866)
* Move ruleRoutine to ruleInfo file

* Move tests as well

* swap ruleInfo and scheduler parameters on ruleRoutine

* Fix linter complaint, receiver name
2024-03-04 17:15:55 -06:00
Alexander Weaver f2a9d0a89d Alerting: Refactor ruleRoutine to take an entire ruleInfo instance (#83858)
* Make stop a real method

* ruleRoutine takes a ruleInfo reference directly rather than pieces of it

* Fix whitespace
2024-03-04 15:15:01 -06:00
Alexander Weaver fa51724bc6 Alerting: Move alertRuleInfo and tests to new files (#83854)
Move ruleinfo and tests to new files
2024-03-04 11:24:49 -06:00
Alexander Weaver a862a4264d Alerting: Export rule validation logic and make it portable (#83555)
* ValidateInterval doesn't need the entire config

* Validation no longer depends on entire folder now that we've dropped foldertitle from api

* Don't depend on entire config struct

* Export validate group
2024-02-28 14:40:13 -06:00
Alexander Weaver d4ae10ecc6 Alerting: Small refactor, move unrelated functions out of fetcher (#82459)
Move unrelated functions out of fetcher
2024-02-14 20:01:32 +02:00
Alexander Weaver ccb4533a86 Alerting: Remove unused AlertRuleVersionWithPauseStatus (#82383)
Remove unused AlertRuleVersionWithPauseStatus
2024-02-13 10:56:24 -06:00
Alexander Weaver 99fa064576 Alerting: Emit warning when creating or updating unusually large groups (#82279)
* Add config for limit of rules per rule group

* Warn when editing big groups through normal API

* Warn on prov api writes for groups

* Wire up comp root, tests

* Also add warning to state manager warm

* Drop unnecessary conversion
2024-02-13 08:29:03 -06:00
Alexander Weaver 5bbe9c6e61 Alerting: Enable group-level rule evaluation jittering by default, remove feature toggle (#82212)
* remove jitter feature flag

* Add an out so users can manually disable jitter

* Pass in cfg

* Add TODO to remove knob in future
2024-02-09 15:53:58 -06:00
Alexander Weaver 843c477899 Alerting: Add exported API to scheduler to access currently loaded rules (#82031)
* Add exported API to fetch rule definitions from scheduler

* Add comment
2024-02-07 09:31:22 -06:00
Alexander Weaver ae2e1aeee4 Alerting: Upgrade feature toggle stages for jitterAlertRules and jitterAlertRulesWithinGroups (#81314)
Upgrade jitter toggle stages
2024-01-25 16:46:22 -06:00
Alexander Weaver 18b9c8fd5f Alerting: Nilcheck JitterStrategyFrom so it can be used in contexts without feature toggles (#80841)
Nilcheck so tests can have a nil feature toggles
2024-01-18 15:43:41 -06:00
Alexander Weaver 00a260effa Alerting: Add setting to distribute rule group evaluations over time (#80766)
* Simple, per-base-interval jitter

* Add log just for test purposes

* Add strategy approach, allow choosing between group or rule

* Add flag to jitter rules

* Add second toggle for jittering within a group

* Wire up toggles to strategy

* Slightly improve comment ordering

* Add tests for offset generation

* Rename JitterStrategyFrom

* Improve debug log message

* Use grafana SDK labels rather than prometheus labels
2024-01-18 12:48:11 -06:00
Alexander Weaver 3c796ecc8f Alerting: Add metric counting rule groups per org (#80669)
* Refactor, fix bad map hint

* Count groups per org
2024-01-16 16:35:56 -06:00
Alexander Weaver 68d4e8a930 Annotations: Remove extraneous, debug log messages (#80670)
* drop log messages

* Revert timer

* fix returns, no need to capture vars for log lines anymore
2024-01-16 14:48:11 -06:00
Alexander WeaverandSofia Papagiannaki 81c45bfe44 Annotations: Split cleanup into separate queries and deletes to avoid deadlocks on MySQL (#80329)
* Split subquery when cleaning annotations

* update comment

* Raise batch size, now that we pay attention to it

* Iterate in batches

* Separate cancellable batch implementation to allow for multi-statement callbacks, add overload for single-statement use

* Use split-out utility in outer batching loop so it respects context cancellation

* guard against empty queries

* Use SQL parameters

* Use same approach for tags

* drop unused function

* Work around parameter limit on sqlite for large batches

* Bulk insert test data in DB

* Refactor test to customise test data creation

* Add test for catching SQLITE_MAX_VARIABLE_NUMBER limit

* Turn annotation cleanup test to integration tests

* lint

---------

Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
2024-01-12 14:05:04 -06:00
Alexander Weaver 542741f748 Alerting: Log scheduler maxAttempts, guard against invalid retry counts, log retry errors (#80234)
* Log maxAttempts, add guard, log retry errors

* fix whitespace

* Initialize evaluator in TestProcessTicks
2024-01-09 13:19:37 -06:00
Alexander Weaver 4623b49947 Drop weaveworks/common dependency (#80090)
Drop weaveworks/common
2024-01-05 10:24:04 -06:00
Alexander Weaver a8fb01a502 Swap weaveworks/common utilities for equivalents in grafana/dskit (#80051)
* Replace histogram collector and grpc injectors

* Extract request timing utility

* Also vendor test file

* Suppress erroneous linter warn
2024-01-05 10:08:38 -06:00
Alexander Weaver a03eca29eb Upgrade grafana/dskit (#80049)
Upgrade dskit
2024-01-04 16:16:51 -06:00
Alexander Weaver 90d4704cd7 Alerting: Fix URL timestamp conversion in historian API in annotation mode (#80026)
Fix timestamp conversion when calling annotation store
2024-01-04 12:40:21 -06:00
Alexander WeaverandWilliam Wernert 65ecde6eed Alerting: Don't record annotations for mapped NoData transitions, when NoData is mapped to OK (#77164)
* Exclude mapped nodata transitions when nodata mapped to OK

* Fix processEvalResults test

* Don't check NoDataState when filtering transition

* Add comment to explain purpose of separate function

---------

Co-authored-by: William Wernert <william.wernert@grafana.com>
2023-12-18 16:59:32 -05:00
Alexander Weaver cf8e8852c3 Alerting: Drop NamespaceID from responses on unstable ngalert API endpoints in favor of NamespaceUID (#79359)
* Drop from API response

* Drop from swagger docs

* Drop from integration tests

* regenerate public swagger docs

* Drop from frontend

* Drop asserts for namespaceID field
2023-12-15 11:06:53 -06:00
Alexander Weaver aa63e91a43 Alerting: Use mux router to match hooks, add support for path variables and methods (#79345)
* Use a router inside hooks rather than plain string matching

* Add test for mismatched method
2023-12-12 14:43:11 -06:00
Alexander Weaver b867505bd4 alerting: Add tests for hooks (#79284)
Add tests for hooks
2023-12-11 13:20:48 -06:00
Alexander Weaver ab0ef5276f Alerting: Decouple quota configuration logic from API interfaces and add tests (#78930)
* Separate usage reporter from API

* Extract quota registration

* Decouple from API store interface

* Move to ngalert package and add tests

* linter
2023-12-01 10:47:19 -06:00
Alexander Weaver 8a56a94781 Usagestats: Add stat group for alert rule groups (#78825)
* Add rule group support to usagestats service

* Quote column name
2023-11-29 14:37:36 -06:00
Alexander Weaver 27b7d1de6f Stats: Fix unregistered unified alerting metric (#78777)
Fix unregistered metric
2023-11-29 08:28:13 -06:00
Alexander Weaver 36dba7584d grpcserver: Extract existing span context from grpc request metadata prior to span creation (#77833)
Extract existing span context from grpc metadata
2023-11-09 11:25:19 -06:00
Alexander Weaver 6ee52ac80c Alerting: Allow more time before Alertmanager expire-resolves alerts (#77094)
* Sync endsAt factor with prometheus

* Fix state tests
2023-10-25 10:03:46 -05:00
Alexander Weaver 39599fa7f7 Alerting: Alert rule constraint violations return as 400s in provisioning API (#76396)
Constraint violations become 400s
2023-10-23 10:28:40 -05:00
Alexander Weaver acee3efcf9 Alerting: Use common StateReason values for NoData/Error mapped states (#76781)
Fix hardcoded state reasons
2023-10-18 17:26:41 -05:00
Alexander Weaver 0592fe7edb Alerting: Add support for msteams contact point in external Alertmanagers (#76392)
* Add support for msteams contact point in external AMs

* Update docs
2023-10-13 10:43:42 -05:00
Alexander Weaver f6649d7a97 Revert "Alerting: Remove vendored models in migration service" (#76387)
Revert "Alerting: Remove vendored models in migration service (#74503)"

This reverts commit 6a8649d544.
2023-10-11 14:21:21 -05:00
Alexander Weaver 5c9aeaef41 Alerting: Do not exit if Redis ping fails when using redis-based Alertmanager clustering (#74144)
Do not fail redis peer construction if ping fails
2023-09-05 10:43:13 -05:00
Alexander Weaver dfba94e052 Alerting: Limit redis pool size to 5 and make configurable (#74057)
* Limit redis pool size to 5 and expose it in config ini

* Coerce negative pool sizes to the default
2023-08-29 14:59:12 -05:00
Alexander Weaver 2245a3d0d1 Tracer: Export ContextWithSpan (#73715)
Export ContextWithSpan
2023-08-25 13:48:58 -05:00
Alexander Weaver 8c8b3ecb5b Alerting: Add dashboardUID and panelID query parameters for loki state history (#72119)
* read query parameters

* Generate loki query from params
2023-07-24 23:46:46 -05:00
Alexander Weaver ff48a145cc Alerting: Add exported getters for PanelKey fields (#72064)
Add getters
2023-07-20 15:47:20 -05:00
Alexander Weaver d6db9a5b3c Alerting: Add exported constructor for panelKey (#71872)
Exported constructor for panelKey
2023-07-18 13:37:43 -05:00
Alexander Weaver 18b910e654 Alerting: Refactor annotation historian to isolate dashboard service dependency (#71689)
* Refactor annotation historian to isolate dashboard service dependency

* Export PanelKey

* Don't export parsePanelKey

* Remove commented out code
2023-07-18 08:18:55 -05:00
Alexander Weaver 128990dbb7 Docs: Consolidation and add call-outs for semantics in Alert Provisioning documentation (#68848)
* Provisioning landing page

* Move editing bit to view page

* Specify API vs File, and provide terraform example

* Improvements around notif policies

* Docs around contact points

* Use h2 instead of h1

* Address feedback

* Move text about policy granularity
2023-07-17 11:15:51 -05:00
Alexander WeaverandKonrad Lalik f94fb765b5 Alerting: Add limit query parameter to Loki-based ASH api, drop default limit from 5000 to 1000, extend visible time range for new ASH UI (#70769)
* Add limit query parameter

* Drop copy paste comment

* Extend history query limit to 30 days and 250 entries

* Fix history log entries ordering

* Update no history message, add empty history test

---------

Co-authored-by: Konrad Lalik <konrad.lalik@grafana.com>
2023-06-28 13:32:28 -05:00
Alexander Weaver ce6f73bd32 Alerting: Add two missing tests which cover missing URLs for Loki state history (#70460)
Add two missing tests which cover individual missing URLs
2023-06-21 12:58:37 -05:00
Alexander Weaver 0ed5d3bdf2 Revert "Alerting: Refactor the ImageStore/Provider to provide image URL/bytes" (#69265)
Revert "Alerting: Refactor the ImageStore/Provider to provide image URL/bytes (#67693)"

This reverts commit 72a187b0be.
2023-05-30 11:33:33 -05:00
Alexander Weaver 0f88b117dc Alerting: Skip flaky test TestRouteGetRuleStatuses (#69258)
Skip TestRouteGetRuleStatuses
2023-05-30 09:48:02 -05:00
Alexander Weaver 117636e8ca Alerting: Fix panic when reparenting receivers to groups following an attempted rename via Provisioning (#67167) 2023-04-24 21:23:23 -04:00
Alexander Weaver 3634079b8f Alerting: Attach hash of instance labels to state history log lines (#65968)
* Add instanceID which is hash of labels

* Rename field to fingerprint

* Move to prometheus style signature

* Appease linter
2023-04-19 14:22:19 -05:00
Alexander Weaver a384194e15 Alerting: Use default page size of 5000 when querying Loki for state history (#66315)
Always specify limit of 5000
2023-04-18 14:31:29 -05:00
Alexander Weaver cf7157f683 Alerting: Capture refID of rule's condition expression in Loki state history entries (#66419)
* Capture condition from rule

* Add test
2023-04-18 14:21:28 -05:00
Alexander Weaver 1791c6043f Alerting: Document state history config options in default and sample configuration files (#65748)
* Document configuration options

* Improve documentation
2023-04-11 14:38:06 -05:00
Alexander Weaver fb520edd72 Alerting: Use a completely isolated context for state history writes (#64989)
* Add fresh context with timeout and same log properties, re-derive logger

* Unify timeout constants

* Move ctx after shortcut that got added through rebasing

* Unify timeouts

* Port opentracing's SpanFromContext and ContextFromSpan to the grafana tracing package

* Support both opentracing and otel variants

* Better document why we're creating a new ctx

* Add new func to FakeSpan which was added after rebase

* Support grafana-specific traceID key in both tracer implementations
2023-04-04 16:41:46 -05:00
Alexander Weaver da4832724e Alerting: Delete stub for SQL alert state history backend (#65667)
Delete stub for SQL backend
2023-03-31 11:15:56 -05:00
Alexander Weaver b2abb63286 Alerting: Introduce proper feature toggles for common state history backend combinations (#65497)
* define 3 feature toggles for rollout phases

* Pass feature toggles along

* Implement first feature toggle

* Try a different strategy with fall-throughs to specific configurations

* Apply toggle overrides once outside of backend composition

* Emit log messages when we coerce backends

* Run code generator for feature toggle files

* Improve wording in flag descs

* Re-run generator

* Use code-generated constants instead of plain strings

* Use converted enum values rather than strings for pre-parsing
2023-03-30 13:53:21 -05:00
Alexander Weaver 5e87ea745d Alerting: Fix and re-enable filters instance labels in log line test (#65618)
Fix and reenable test
2023-03-30 09:02:18 -05:00
Alexander Weaver a416100abc Alerting: No longer index state history log streams by instance labels (#65474)
* Remove private labels

* No longer index by instance labels

* Labels are now invariant, only build them once

* Remove bucketing since everything is in a single stream

* Refactor statesToStreams to only return a single unified log stream

* Don't query on labels that no longer exist

* Move selector logic to loki layer, genericize client to work in terms of straight logQL

* Add support for line-level label filters in query

* Combine existing selector tests for better parallelism

* Tests for logQL construction

* Underscore instead of dot for unwrapping labels in logql
2023-03-29 11:52:11 -05:00
Alexander Weaver de1637afe5 Alerting: Add alert instance labels to Loki log lines in addition to stream labels (#65403)
Add instance labels to log line
2023-03-28 08:57:51 -05:00
Alexander Weaver dd04757fc9 Alerting: Add "backend" label to state history writes metrics (#65395)
* Add backend label to state history writes metrics

* Update test expectations
2023-03-28 08:49:51 -05:00
Alexander Weaver 07368dec74 Alerting: Fix attachment of external labels to Loki state history log streams (#65140)
Fix attachment of external labels, add tests
2023-03-21 18:00:59 -05:00
Alexander Weaver bf54f2672e Alerting: Switch to snappy-compressed-protobuf for outgoing push requests to Loki (#65077)
* Encode with snappy, always

* JSON encoder type

* Headers

* Copy labels formatter from promtail

* Implement snappy-proto encoding

* Create encoder interface, test both encoders, choose snappy-proto by default

* Make encoder configurable at the LokiCfg level

* Export both encoders

* Touch up comment and tests

* Drop unnecessary conversions after move to plain strings to appease linter
2023-03-21 13:38:42 -05:00
Alexander Weaver cc7e5ce62e Alerting: Fix ambiguous handling of equals in labels when bucketing Loki state history streams (#65013)
* Use JSON instead of data.Labels string format as label repr

* Drop debug log line
2023-03-21 12:33:27 -05:00
Alexander Weaver e39d7f44c9 Alerting: Elide requests to Loki if nothing should be recorded (#65011)
Exit early if no log streams or annotations
2023-03-21 09:30:56 -05:00
Alexander Weaver 40c5713cbd Vendor errors.Join from Go standard library to avoid version incompatibilities (#64985)
Vendor errors.Join from std lib
2023-03-17 14:07:58 -05:00
Alexander Weaver a31672fa40 Alerting: Create new state history "fanout" backend that dispatches to multiple other backends at once (#64774)
* Rename RecordStatesAsync to Record

* Rename QueryStates to Query

* Implement fanout writes

* Implement primary queries

* Simplify error joining

* Add test for query path

* Add tests for writes and error propagation

* Allow fanout backend to be configured

* Touch up log messages and config validation

* Consistent documentation for all backend structs

* Parse and normalize backend names more consistently against an enum

* Touch-ups to documentation

* Improve clarity around multi-record blocking

* Keep primary and secondaries more distinct

* Rename fanout backend to multiple backend

* Simplify config keys for multi backend mode
2023-03-17 12:41:18 -05:00
Alexander Weaver 9bcf8819d3 Alerting: Handful of small adjustments to log levels and parameters (#64572)
Calculate duration earlier in scheduler
2023-03-17 12:15:49 +00:00
Alexander Weaver faef3a8258 Alerting: Log error but don't fail initialization if state history connection test fails (#64699)
Don't return init error if ping fails, add tests
2023-03-13 15:54:46 -05:00
Alexander Weaver 4a1c18abf6 Alerting: Fix intermittency when seeding database in rule store tests (#64322)
Force unique IDs when seeding database
2023-03-07 09:40:55 -05:00
Alexander Weaver 19d01dff91 Alerting: Expose Prometheus metrics for persisting state history (#63157)
* Create historian metrics and dependency inject

* Record counter for total number of state transitions logged

* Track write failures

* Track current number of active write goroutines

* Record histogram of how long it takes to write history data

* Don't copy the registerer

* Adjust naming of write failures metric

* Introduce WritesTotal to complement WritesFailedTotal

* Measure TransitionsFailedTotal to complement TransitionsTotal

* Rename all to state_history

* Remove redundant Total suffix

* Increment totals all the time, not just on success

* Drop ActiveWriteGoroutines

* Drop PersistDuration in favor of WriteDuration

* Drop unused gauge

* Make writes and writesFailed per org

* Add metric indicating backend and a spot for future metadata

* Drop _batch_ from names and update help

* Add metric for bytes written

* Better pairing of total + failure metric updates

* Few tweaks to wording and naming

* Record info metric during composition

* Create fakeRequester and simple happy path test using it

* Blocking test for the full historian and test for happy path metrics

* Add tests for failure case metrics

* Smoke test for full annotation persistence

* Create test for metrics on annotation persistence, both happy and failing paths

* Address linter complaints

* More linter complaints

* Remove unnecessary whitespace

* Consistency improvements to help texts

* Update tests to match new descs
2023-03-06 10:40:37 -06:00
Alexander Weaver 8ea71d37c2 SQLStore: Enable clientFoundRows for MySQL connections (#64070)
Enable clientFoundRows for MySQL connections
2023-03-02 16:31:01 -06:00
Alexander Weaver e77621649d Alerting: Instrument outgoing state history requests using weaveworks/common (#63600)
* Loki backend and client depend on a requester

* Instrument all requests to loki using weaveworks TimedClient

* Construct collector in metrics package
2023-02-23 17:52:02 -06:00
Alexander Weaver 958fb2c50a Alerting: Unify structs in Loki client and make them more consistent with Prometheus (#63055)
* Use existing row struct instead of [2]string, add deserialization helper

* Replace Stream struct with stream struct which is exactly the same

* Drop unused status field

* Don't export queryRes and queryData

* Tests for custom marshalling

* Rename row fields to T and V for consistency with prometheus samples

* Rename row to sample
2023-02-11 05:17:44 -06:00
Alexander Weaver f80bf11782 Alerting: Make time range query parameters not required when querying Loki (#62985)
* Make from and to not required

* Move default range calculation up to loki.go
2023-02-07 14:26:43 -06:00
Alexander Weaver 0efb84617e Alerting: Create benchmarking test for state.ProcessEvalResults (#62041)
* Create benchmark for ProcessEvalResults

* Simplify the test
2023-02-03 15:38:08 -06:00
Alexander Weaver 9eeea8f5ea Alerting: Add label query parameters to state history endpoint (#62831)
* Allow equality-only matching of arbitrary labels via query params

* Pre-initialize map
2023-02-02 16:52:08 -06:00
Alexander Weaver 647f73ddc5 Alerting: Add static label to all state history entries (#62817)
* Add static label to all state history entries

* Separate label and value visually
2023-02-02 13:25:26 -06:00
Alexander Weaver 6ad1cfef38 Alerting: Add endpoint for querying state history (#62166)
* Define endpoint and generate

* Wire up and register endpoint

* Cleanup, define authorization

* Forgot the leading slash

* Wire up query and SignedInUser

* Wire up timerange query params

* Add todo for label queries

* Drop comment

* Update path to rules subtree
2023-02-02 11:34:00 -06:00
Alexander Weaver 9fa28c11c5 Alerting: Usability adjustments to Loki representation of state history values (#62643)
* Extract label merge, add test file

* Extract error/NoData to first class fields, remove a layer from values

* Include dashUID and panelID as line-level fields

* Drop unnecessary object receiver

* Add tests for stream building

* Drop NoData field from log lines
2023-02-02 10:54:10 -06:00
Alexander Weaver fcecf4d3cb Alerting: Refactor away a layer of indirection around the goroutine in Loki state history (#62644)
Inline recordStreamsAsync in loki backend
2023-02-01 11:57:29 -06:00
Alexander Weaver 03f3fbec0d Alerting: Fix handling of special floating-point cases when writing observed values to annotations (#61074)
* Fix json serialization of state values

* Simplify two of the tests

* Fix linter complaint

* Don't return error if we fail to look up dashboard, just log it and move on

* Address linter complaint
2023-01-31 15:27:51 -06:00
Alexander Weaver e7ace4ed62 Alerting: Allow separate read and write path URLs for Loki state history (#62268)
Extract config parsing and add tests
2023-01-30 16:30:05 -06:00
Alexander Weaver b4682fe3cb Alerting: Configurable externalLabels for Loki state history (#62404)
* Add config option for external labels

* Remove redundant nilcheck
2023-01-30 14:24:45 -06:00
Alexander Weaver eb1293ebb1 Alerting: Re-generate swagger definitions (#62154)
Regenerate swagger, add body binding so parameters work
2023-01-25 15:01:42 -06:00
Alexander Weaver 046a9bb7c1 Alerting: Copy rule definitions into state history (#62032)
* Copy rules instead of accepting pointer

* Deep-copy the rule, for even more guarantees

* Create struct just for needed fields

* Move RuleMeta to historian/model package, iron out package dependencies

* Move tests for dash ID parsing to model package along with code
2023-01-25 11:29:57 -06:00
Alexander Weaver 7ccc845187 Alerting: Push state history entries to Loki (#61724)
* Implement push endpoint

* Drop duplicated struct

* Genericize auth/tenant headers and improve logging in error case

* Flesh out the data model

* Drop dead code

* Drop log line entirely

* Drop unused arg

* Rename a few type manipulation functions

* Extract label keys as constants

* Improve logs when loki responds with error

* Inline lokiRepresentation function
2023-01-23 16:31:03 -06:00
Alexander Weaver c10713ea76 Alerting: Create query interface for state history along with annotation-based implementation (#61646) 2023-01-19 10:45:31 +01:00
Alexander Weaver 1ac89ea040 Alerting: Add client configuration for remote Loki historian backend and test connection (#61114)
* Create loki client type and ping method

* Expose TestConnection on client

* Configure and ping Loki URL

* Close response body reader if present

* Add 30 second timeout

* Remove duplicate close
2023-01-17 13:58:52 -06:00
Alexander Weaver 4f1bdc0607 Alerting: Skip flaky test in TestIntegrationUpdateAlertRules (#61627)
* Skip flaky test

* Add comment
2023-01-17 10:39:16 -06:00
Alexander Weaver b289b8ac6e Alerting: Set error annotation on EvaluationError regardless of underlying error type (#61506)
Set error annotation regardless of underlying error type
2023-01-13 13:58:02 -06:00
Alexander Weaver 499fdf1ae4 Docs: Unified Alerting is now compatible with AWS Aurora (#61001)
Remove line that aurora is incompatible
2023-01-11 08:43:18 +00:00
Alexander Weaver eb960d9725 Alerting: Add un-documented toggle for changing state history backend, add shells for remote loki and sql (#61072)
* Add toggle for state history backend and shells

* Extract some shared logic and add tests
2023-01-06 12:06:01 -06:00
Alexander Weaver 8c3a5f6da0 Alerting: Allow state history to be disabled through configuration (#61006)
* Add configuration option for if state history should be enabled

* Inject no-op when history is disabled
2023-01-05 12:21:07 -06:00
Alexander Weaver 0e7640475f Alerting: Store alertmanager configuration history in a separate table in the database (#60492)
* Update config store to split between active and history tables

* Migrations to fix up indexes

* Implement migration from old format to new

* Move add migrations call

* Delete duplicated rows

* Explicitly map fields

* Quote the column name because it's a reserved word

* Lift migrations to top

* Use XORM for nearly everything, avoid any non trivial raw SQL

* Touch up indexes and zero out IDs on move

* Drop TODO that's already completed

* Fix assignment of IDs
2023-01-04 10:43:26 -06:00
Alexander Weaver 1381fb6dfc Annotations: Fix EpochEnd being zero for Alert-generated annotations (#60931)
* Revert linter suggestion

* Re-add nolint

* Work in terms of pointer rather than copy

* Add tests covering validation

* Add comment
2023-01-04 10:16:54 -06:00
Alexander Weaver b88b8bc291 Alerting: Fix missing dashboard/panelID links in annotations (#60926)
Assign thru ref
2023-01-03 14:12:27 -06:00
Alexander WeaverandYuri Tseretyan ca3f8ba6f4 Alerting: Refactor alertmanager notifier to use encoding/json to parse settings instead of simplejson (#55507)
* replace basic auth header with method call

Co-authored-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
2022-12-19 15:12:49 -05:00
Alexander Weaver 91bd1cdb41 Revert "Alerting: Store alertmanager configuration history in a separate table in the database" (#60470)
Revert "Alerting: Store alertmanager configuration history in a separate table in the database (#60197)"

This reverts commit ec80f38c34.
2022-12-16 10:07:44 -05:00
Alexander Weaver ec80f38c34 Alerting: Store alertmanager configuration history in a separate table in the database (#60197)
* Update config store to split between active and history tables

* Migrations to fix up indexes

* Implement migration from old format to new

* Move add migrations call

* Delete duplicated rows

* Explicitly map fields

* Quote the column name because it's a reserved word

* Lift migrations to top
2022-12-15 17:35:00 -06:00