Alerting: Write and Delete multiple alert instances. (#55350)

Prior to this change, all alert instance writes and deletes happened
individually, in their own database transaction. This change batches up
writes or deletes for a given rule's evaluation loop into a single
transaction before applying it.

These new transactions are off by default, guarded by the feature toggle "alertingBigTransactions"

Before:

```
goos: darwin
goarch: arm64
pkg: github.com/grafana/grafana/pkg/services/ngalert/store
BenchmarkAlertInstanceOperations-8           398           2991381 ns/op         1133537 B/op      27703 allocs/op
--- BENCH: BenchmarkAlertInstanceOperations-8
    util.go:127: alert definition: {orgID: 1, UID: FovKXiRVzm} with title: "an alert definition FTvFXmRVkz" interval: 60 created
    util.go:127: alert definition: {orgID: 1, UID: foDFXmRVkm} with title: "an alert definition fovFXmRVkz" interval: 60 created
    util.go:127: alert definition: {orgID: 1, UID: VQvFuigVkm} with title: "an alert definition VwDKXmR4kz" interval: 60 created
PASS
ok      github.com/grafana/grafana/pkg/services/ngalert/store   1.619s
```

After:

```
goos: darwin
goarch: arm64
pkg: github.com/grafana/grafana/pkg/services/ngalert/store
BenchmarkAlertInstanceOperations-8          1440            816484 ns/op          352297 B/op       6529 allocs/op
--- BENCH: BenchmarkAlertInstanceOperations-8
    util.go:127: alert definition: {orgID: 1, UID: 302r_igVzm} with title: "an alert definition q0h9lmR4zz" interval: 60 created
    util.go:127: alert definition: {orgID: 1, UID: 71hrlmR4km} with title: "an alert definition nJ29_mR4zz" interval: 60 created
    util.go:127: alert definition: {orgID: 1, UID: Cahr_mR4zm} with title: "an alert definition ja2rlmg4zz" interval: 60 created
PASS
ok      github.com/grafana/grafana/pkg/services/ngalert/store   1.383s
```

So we cut time by about 75% and memory allocations by about 60% when
storing and deleting 100 instances.
This commit is contained in:
Joe Blubaugh
2022-10-06 14:22:58 +08:00
committed by GitHub
parent b4e23e5d32
commit b476ae62fb
21 changed files with 606 additions and 193 deletions

View File

@@ -17,7 +17,7 @@ import (
)
func TestCalculateChanges(t *testing.T) {
orgId := rand.Int63()
orgId := int64(rand.Int31())
t.Run("detects alerts that need to be added", func(t *testing.T) {
fakeStore := fakes.NewRuleStore(t)
@@ -102,8 +102,8 @@ func TestCalculateChanges(t *testing.T) {
r := models.CopyRule(rule)
// Ignore difference in the following fields as submitted models do not have them set
r.ID = rand.Int63()
r.Version = rand.Int63()
r.ID = int64(rand.Int31())
r.Version = int64(rand.Int31())
r.Updated = r.Updated.Add(1 * time.Minute)
submitted = append(submitted, r)