Commit Graph

65 Commits

Author SHA1 Message Date
Alexander Weaver
a8fb01a502
Swap weaveworks/common utilities for equivalents in grafana/dskit (#80051)
* Replace histogram collector and grpc injectors

* Extract request timing utility

* Also vendor test file

* Suppress erroneous linter warn
2024-01-05 10:08:38 -06:00
Alexander Weaver
90d4704cd7
Alerting: Fix URL timestamp conversion in historian API in annotation mode (#80026)
Fix timestamp conversion when calling annotation store
2024-01-04 12:40:21 -06:00
Alexander Weaver
65ecde6eed
Alerting: Don't record annotations for mapped NoData transitions, when NoData is mapped to OK (#77164)
* Exclude mapped nodata transitions when nodata mapped to OK

* Fix processEvalResults test

* Don't check NoDataState when filtering transition

* Add comment to explain purpose of separate function

---------

Co-authored-by: William Wernert <william.wernert@grafana.com>
2023-12-18 16:59:32 -05:00
William Wernert
9171bf92bb
Alerting: Add rule ID and title to alert state history Loki entry (#79481)
* Add rule ID and title to Loki entry

* Combine related tests
2023-12-14 13:06:23 -05:00
William Wernert
f7bf818527
Alerting: Make alert state history Loki http client public (#78291)
* Make state history Loki client public

* Make historian metrics subsystem configurable
2023-11-27 09:20:50 -05:00
Ryan McKinley
5d5f8dfc52
Chore: Upgrade Go to 1.21.3 (#77304) 2023-11-01 09:17:38 -07:00
Marcus Efraimsson
e4c1a7a141
Tracing: Standardize on otel tracing (#75528) 2023-10-03 14:54:20 +02:00
Alexander Weaver
8c8b3ecb5b
Alerting: Add dashboardUID and panelID query parameters for loki state history (#72119)
* read query parameters

* Generate loki query from params
2023-07-24 23:46:46 -05:00
Alexander Weaver
ff48a145cc
Alerting: Add exported getters for PanelKey fields (#72064)
Add getters
2023-07-20 15:47:20 -05:00
Alexander Weaver
d6db9a5b3c
Alerting: Add exported constructor for panelKey (#71872)
Exported constructor for panelKey
2023-07-18 13:37:43 -05:00
Alexander Weaver
18b910e654
Alerting: Refactor annotation historian to isolate dashboard service dependency (#71689)
* Refactor annotation historian to isolate dashboard service dependency

* Export PanelKey

* Don't export parsePanelKey

* Remove commented out code
2023-07-18 08:18:55 -05:00
Alexander Weaver
f94fb765b5
Alerting: Add limit query parameter to Loki-based ASH api, drop default limit from 5000 to 1000, extend visible time range for new ASH UI (#70769)
* Add limit query parameter

* Drop copy paste comment

* Extend history query limit to 30 days and 250 entries

* Fix history log entries ordering

* Update no history message, add empty history test

---------

Co-authored-by: Konrad Lalik <konrad.lalik@grafana.com>
2023-06-28 13:32:28 -05:00
William Wernert
4aa477f48f
Alerting: Move rule UID from Loki stream labels into log lines (#70637)
Move rule uid into log line to reduce cardinality
2023-06-26 09:57:45 -04:00
Alexander Weaver
ce6f73bd32
Alerting: Add two missing tests which cover missing URLs for Loki state history (#70460)
Add two missing tests which cover individual missing URLs
2023-06-21 12:58:37 -05:00
Alexander Weaver
3634079b8f
Alerting: Attach hash of instance labels to state history log lines (#65968)
* Add instanceID which is hash of labels

* Rename field to fingerprint

* Move to prometheus style signature

* Appease linter
2023-04-19 14:22:19 -05:00
Alexander Weaver
a384194e15
Alerting: Use default page size of 5000 when querying Loki for state history (#66315)
Always specify limit of 5000
2023-04-18 14:31:29 -05:00
Alexander Weaver
cf7157f683
Alerting: Capture refID of rule's condition expression in Loki state history entries (#66419)
* Capture condition from rule

* Add test
2023-04-18 14:21:28 -05:00
Alexander Weaver
fb520edd72
Alerting: Use a completely isolated context for state history writes (#64989)
* Add fresh context with timeout and same log properties, re-derive logger

* Unify timeout constants

* Move ctx after shortcut that got added through rebasing

* Unify timeouts

* Port opentracing's SpanFromContext and ContextFromSpan to the grafana tracing package

* Support both opentracing and otel variants

* Better document why we're creating a new ctx

* Add new func to FakeSpan which was added after rebase

* Support grafana-specific traceID key in both tracer implementations
2023-04-04 16:41:46 -05:00
Alexander Weaver
da4832724e
Alerting: Delete stub for SQL alert state history backend (#65667)
Delete stub for SQL backend
2023-03-31 11:15:56 -05:00
Alexander Weaver
5e87ea745d
Alerting: Fix and re-enable filters instance labels in log line test (#65618)
Fix and reenable test
2023-03-30 09:02:18 -05:00
Dimitris Sotirakis
e758b017d0
Alerting: Disable filters instance labels in log line test (#65610)
* Disable filters instance labels in log line test

* Add drone reference
2023-03-30 16:04:29 +03:00
Alexander Weaver
a416100abc
Alerting: No longer index state history log streams by instance labels (#65474)
* Remove private labels

* No longer index by instance labels

* Labels are now invariant, only build them once

* Remove bucketing since everything is in a single stream

* Refactor statesToStreams to only return a single unified log stream

* Don't query on labels that no longer exist

* Move selector logic to loki layer, genericize client to work in terms of straight logQL

* Add support for line-level label filters in query

* Combine existing selector tests for better parallelism

* Tests for logQL construction

* Underscore instead of dot for unwrapping labels in logql
2023-03-29 11:52:11 -05:00
Alexander Weaver
de1637afe5
Alerting: Add alert instance labels to Loki log lines in addition to stream labels (#65403)
Add instance labels to log line
2023-03-28 08:57:51 -05:00
Alexander Weaver
dd04757fc9
Alerting: Add "backend" label to state history writes metrics (#65395)
* Add backend label to state history writes metrics

* Update test expectations
2023-03-28 08:49:51 -05:00
Serge Zaitsev
0beb768427
Chore: Remove result fields from ngalert (#65410)
* remove result fields from ngalert

* remove duplicate imports
2023-03-28 10:34:35 +02:00
Alexander Weaver
07368dec74
Alerting: Fix attachment of external labels to Loki state history log streams (#65140)
Fix attachment of external labels, add tests
2023-03-21 18:00:59 -05:00
Alexander Weaver
bf54f2672e
Alerting: Switch to snappy-compressed-protobuf for outgoing push requests to Loki (#65077)
* Encode with snappy, always

* JSON encoder type

* Headers

* Copy labels formatter from promtail

* Implement snappy-proto encoding

* Create encoder interface, test both encoders, choose snappy-proto by default

* Make encoder configurable at the LokiCfg level

* Export both encoders

* Touch up comment and tests

* Drop unnecessary conversions after move to plain strings to appease linter
2023-03-21 13:38:42 -05:00
Alexander Weaver
cc7e5ce62e
Alerting: Fix ambiguous handling of equals in labels when bucketing Loki state history streams (#65013)
* Use JSON instead of data.Labels string format as label repr

* Drop debug log line
2023-03-21 12:33:27 -05:00
Alexander Weaver
e39d7f44c9
Alerting: Elide requests to Loki if nothing should be recorded (#65011)
Exit early if no log streams or annotations
2023-03-21 09:30:56 -05:00
Alexander Weaver
40c5713cbd
Vendor errors.Join from Go standard library to avoid version incompatibilities (#64985)
Vendor errors.Join from std lib
2023-03-17 14:07:58 -05:00
Alexander Weaver
a31672fa40
Alerting: Create new state history "fanout" backend that dispatches to multiple other backends at once (#64774)
* Rename RecordStatesAsync to Record

* Rename QueryStates to Query

* Implement fanout writes

* Implement primary queries

* Simplify error joining

* Add test for query path

* Add tests for writes and error propagation

* Allow fanout backend to be configured

* Touch up log messages and config validation

* Consistent documentation for all backend structs

* Parse and normalize backend names more consistently against an enum

* Touch-ups to documentation

* Improve clarity around multi-record blocking

* Keep primary and secondaries more distinct

* Rename fanout backend to multiple backend

* Simplify config keys for multi backend mode
2023-03-17 12:41:18 -05:00
Alexander Weaver
19d01dff91
Alerting: Expose Prometheus metrics for persisting state history (#63157)
* Create historian metrics and dependency inject

* Record counter for total number of state transitions logged

* Track write failures

* Track current number of active write goroutines

* Record histogram of how long it takes to write history data

* Don't copy the registerer

* Adjust naming of write failures metric

* Introduce WritesTotal to complement WritesFailedTotal

* Measure TransitionsFailedTotal to complement TransitionsTotal

* Rename all to state_history

* Remove redundant Total suffix

* Increment totals all the time, not just on success

* Drop ActiveWriteGoroutines

* Drop PersistDuration in favor of WriteDuration

* Drop unused gauge

* Make writes and writesFailed per org

* Add metric indicating backend and a spot for future metadata

* Drop _batch_ from names and update help

* Add metric for bytes written

* Better pairing of total + failure metric updates

* Few tweaks to wording and naming

* Record info metric during composition

* Create fakeRequester and simple happy path test using it

* Blocking test for the full historian and test for happy path metrics

* Add tests for failure case metrics

* Smoke test for full annotation persistence

* Create test for metrics on annotation persistence, both happy and failing paths

* Address linter complaints

* More linter complaints

* Remove unnecessary whitespace

* Consistency improvements to help texts

* Update tests to match new descs
2023-03-06 10:40:37 -06:00
Alexander Weaver
e77621649d
Alerting: Instrument outgoing state history requests using weaveworks/common (#63600)
* Loki backend and client depend on a requester

* Instrument all requests to loki using weaveworks TimedClient

* Construct collector in metrics package
2023-02-23 17:52:02 -06:00
Alexander Weaver
958fb2c50a
Alerting: Unify structs in Loki client and make them more consistent with Prometheus (#63055)
* Use existing row struct instead of [2]string, add deserialization helper

* Replace Stream struct with stream struct which is exactly the same

* Drop unused status field

* Don't export queryRes and queryData

* Tests for custom marshalling

* Rename row fields to T and V for consistency with prometheus samples

* Rename row to sample
2023-02-11 05:17:44 -06:00
Alexander Weaver
f80bf11782
Alerting: Make time range query parameters not required when querying Loki (#62985)
* Make from and to not required

* Move default range calculation up to loki.go
2023-02-07 14:26:43 -06:00
idafurjes
982939111b
Rename Id to ID for annotation models (#62886)
* Rename Id to ID for annotation models

* Add xorm tags

* Rename Id to ID for API key models

* Add xorm tags
2023-02-03 17:23:09 +01:00
Jean-Philippe Quéméner
1694a35e0c
Alerting: implement loki query for alert state history (#61992)
* Alerting: implement loki query for alert state history

* extract selector building

* add unit tests for selector creation

* backup

* give selectors their own type

* build dataframe

* add some tests

* small changes after manual testing

* use struct client

* golint

* more golint

* Make RuleUID optional for Loki implementation

* Drop initial assumption that we only have one series

* Pare down to three columns, fix timestamp overflows, improve failure cases in loki responses

* Embed structred log lines in the dataframe as objects rather than json strings

* Include state history label filter

* Remove dead code

---------

Co-authored-by: Alex Weaver <weaver.alex.d@gmail.com>
2023-02-02 16:31:51 -06:00
Alexander Weaver
647f73ddc5
Alerting: Add static label to all state history entries (#62817)
* Add static label to all state history entries

* Separate label and value visually
2023-02-02 13:25:26 -06:00
Alexander Weaver
6ad1cfef38
Alerting: Add endpoint for querying state history (#62166)
* Define endpoint and generate

* Wire up and register endpoint

* Cleanup, define authorization

* Forgot the leading slash

* Wire up query and SignedInUser

* Wire up timerange query params

* Add todo for label queries

* Drop comment

* Update path to rules subtree
2023-02-02 11:34:00 -06:00
Alexander Weaver
9fa28c11c5
Alerting: Usability adjustments to Loki representation of state history values (#62643)
* Extract label merge, add test file

* Extract error/NoData to first class fields, remove a layer from values

* Include dashUID and panelID as line-level fields

* Drop unnecessary object receiver

* Add tests for stream building

* Drop NoData field from log lines
2023-02-02 10:54:10 -06:00
Alexander Weaver
fcecf4d3cb
Alerting: Refactor away a layer of indirection around the goroutine in Loki state history (#62644)
Inline recordStreamsAsync in loki backend
2023-02-01 11:57:29 -06:00
Alexander Weaver
03f3fbec0d
Alerting: Fix handling of special floating-point cases when writing observed values to annotations (#61074)
* Fix json serialization of state values

* Simplify two of the tests

* Fix linter complaint

* Don't return error if we fail to look up dashboard, just log it and move on

* Address linter complaint
2023-01-31 15:27:51 -06:00
Alexander Weaver
e7ace4ed62
Alerting: Allow separate read and write path URLs for Loki state history (#62268)
Extract config parsing and add tests
2023-01-30 16:30:05 -06:00
Alexander Weaver
b4682fe3cb
Alerting: Configurable externalLabels for Loki state history (#62404)
* Add config option for external labels

* Remove redundant nilcheck
2023-01-30 14:24:45 -06:00
Serge Zaitsev
d6d4097567
Chore: Fix goimports grouping in alerting (#62424)
* fix goimports

* fix goimports order
2023-01-30 09:55:35 +01:00
Yuri Tseretyan
0c4671e31f
Alerting: Update historian to ignore transitions from Normal Paused and Updated (#62267) 2023-01-27 16:26:22 -05:00
George Robinson
a7eab8e46e
Alerting: Support context.Context in Loki interface (#61979)
This commit adds support for canceleable contexts in the Loki
interface.
2023-01-26 09:31:20 +00:00
Alexander Weaver
046a9bb7c1
Alerting: Copy rule definitions into state history (#62032)
* Copy rules instead of accepting pointer

* Deep-copy the rule, for even more guarantees

* Create struct just for needed fields

* Move RuleMeta to historian/model package, iron out package dependencies

* Move tests for dash ID parsing to model package along with code
2023-01-25 11:29:57 -06:00
idafurjes
b54b80f473
Chore: Remove Result from dashboard models (#61997)
* Chore: Remove Result from dashboard models

* Fix lint tests

* Fix dashboard service tests

* Fix API tests

* Remove commented out code

* Chore: Merge main - cleanup
2023-01-25 10:36:26 +01:00
George Robinson
239d94205a
Alerting: Return chan <-error for #61811 (#61858) 2023-01-24 15:41:38 +00:00