* SQLStore: Fix Postgres dialect treating "false" migrator default as true
Previously, when creating a migration you could choose a default value for a new
boolean column that looked correct but would be interpreted incorrectly by the
Postgres dialect. For example, values such as "false" or "FALSE" would be treated
as true by the Postgres dialect.
This refactors how migration dialects determine the Default column value for boolean
type columns. Each dialect now uses the same base code to parse the Default literal
and panics if an unknown value is encountered.
So, now AddColumnMigration and AddTableMigration will ensure that across dialects:
- The exact same Default literals will be allowed.
- The literals are converted to equivalent defaults in their DDL.
- An error will be thrown if an invalid literal is provided.
Prior to this change, all alert instance writes and deletes happened
individually, in their own database transaction. This change batches up
writes or deletes for a given rule's evaluation loop into a single
transaction before applying it.
These new transactions are off by default, guarded by the feature toggle "alertingBigTransactions"
Before:
```
goos: darwin
goarch: arm64
pkg: github.com/grafana/grafana/pkg/services/ngalert/store
BenchmarkAlertInstanceOperations-8 398 2991381 ns/op 1133537 B/op 27703 allocs/op
--- BENCH: BenchmarkAlertInstanceOperations-8
util.go:127: alert definition: {orgID: 1, UID: FovKXiRVzm} with title: "an alert definition FTvFXmRVkz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: foDFXmRVkm} with title: "an alert definition fovFXmRVkz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: VQvFuigVkm} with title: "an alert definition VwDKXmR4kz" interval: 60 created
PASS
ok github.com/grafana/grafana/pkg/services/ngalert/store 1.619s
```
After:
```
goos: darwin
goarch: arm64
pkg: github.com/grafana/grafana/pkg/services/ngalert/store
BenchmarkAlertInstanceOperations-8 1440 816484 ns/op 352297 B/op 6529 allocs/op
--- BENCH: BenchmarkAlertInstanceOperations-8
util.go:127: alert definition: {orgID: 1, UID: 302r_igVzm} with title: "an alert definition q0h9lmR4zz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: 71hrlmR4km} with title: "an alert definition nJ29_mR4zz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: Cahr_mR4zm} with title: "an alert definition ja2rlmg4zz" interval: 60 created
PASS
ok github.com/grafana/grafana/pkg/services/ngalert/store 1.383s
```
So we cut time by about 75% and memory allocations by about 60% when
storing and deleting 100 instances.
Prior to this change, all alert instance writes and deletes happened
individually, in their own database transaction. This change batches up
writes or deletes for a given rule's evaluation loop into a single
transaction before applying it.
Before:
```
goos: darwin
goarch: arm64
pkg: github.com/grafana/grafana/pkg/services/ngalert/store
BenchmarkAlertInstanceOperations-8 398 2991381 ns/op 1133537 B/op 27703 allocs/op
--- BENCH: BenchmarkAlertInstanceOperations-8
util.go:127: alert definition: {orgID: 1, UID: FovKXiRVzm} with title: "an alert definition FTvFXmRVkz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: foDFXmRVkm} with title: "an alert definition fovFXmRVkz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: VQvFuigVkm} with title: "an alert definition VwDKXmR4kz" interval: 60 created
PASS
ok github.com/grafana/grafana/pkg/services/ngalert/store 1.619s
```
After:
```
goos: darwin
goarch: arm64
pkg: github.com/grafana/grafana/pkg/services/ngalert/store
BenchmarkAlertInstanceOperations-8 1440 816484 ns/op 352297 B/op 6529 allocs/op
--- BENCH: BenchmarkAlertInstanceOperations-8
util.go:127: alert definition: {orgID: 1, UID: 302r_igVzm} with title: "an alert definition q0h9lmR4zz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: 71hrlmR4km} with title: "an alert definition nJ29_mR4zz" interval: 60 created
util.go:127: alert definition: {orgID: 1, UID: Cahr_mR4zm} with title: "an alert definition ja2rlmg4zz" interval: 60 created
PASS
ok github.com/grafana/grafana/pkg/services/ngalert/store 1.383s
```
So we cut time by about 75% and memory allocations by about 60% when
storing and deleting 100 instances.
This change also updates some of our tests so that they run successfully against postgreSQL - we were using random Int64s, but postgres integers, which our tables use, max out at 2^31-1
Fixes#30144
Co-authored-by: dsotirakis <sotirakis.dim@gmail.com>
Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
Co-authored-by: Ida Furjesova <ida.furjesova@grafana.com>
Co-authored-by: Jack Westbrook <jack.westbrook@gmail.com>
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
Co-authored-by: Leon Sorokin <leeoniya@gmail.com>
Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com>
Co-authored-by: spinillos <selenepinillos@gmail.com>
Co-authored-by: Karl Persson <kalle.persson@grafana.com>
Co-authored-by: Leonard Gram <leo@xlson.com>
* sqlstore: Run tests as integration tests
* Truncate database instead of re-creating it on each test
* Fix test description
See https://github.com/grafana/grafana/pull/12129
* Fix lint issues
* Fix postgres dialect after review suggestion
* Rename and document functions after review suggestion
* Add periods
* Fix auto-increment value for mysql dialect
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
* Implementation of optimistic lock pattern
Try to insert the remote cache key and handle integrity error
* Remove transaction
Integrity error inside a transaction results in deadlock
* Remove check for existing remote cache key
Is no longer needed since integrity constrain violations are handled
* Add check for integrity constrain violation
Do not update the row if the insert statement fails
for other than an integrity constrain violation
* Handle failing inserts because of deadlocks
If the insert statement fails because of a deadlock
try to update the row
* Add utility function for returning SQL error code
Useful for debugging
* Add logging for failing expired cache key deletion
Do not shallow it completely
* Revert "Add utility function for returning SQL error code"
This reverts commit 8e0b82c79633e7d8bc350823cbbab2ac7a58c0a5.
* Better log for failing deletion of expired cache key
* Add some comments
* Remove check for existing cache key
Attempt to insert the key without checking if it's already there
and handle the error situations
* Do not propagate deadlocks created during update
Most probably somebody else is trying to insert/update
the key at the same time so it is safe enough to ignore it
This is basically implementation of the https://github.com/grafana/grafana/issues/8900#issuecomment-435437167
points, except for the type conversion bit.
I tried to implement idea mentioned in cockroachdb ticket (see below).
And it is possible, but it complicates things as lot - not only we have to
have 4 SQL statements instead of one, but we would have to copy the column
structure as well - PK, FG, indexes and stuff, plus there will
be additional downtime with this approach.
So idea for this pull is to prepare our SQL as much as possible, so when
cockroachdb will add support for full type conversions, we could easilly add
support for it as well.
* Add `CASCADE` to `DROP INDEX` statement
* Make string conversions explicit
Thanks @Luit
Ref #8900
Ref cockroach/cockroach#9851
This commit fixes the following golint warnings:
pkg/bus/bus.go:64:9: if block ends with a return statement, so drop this else and outdent its block
pkg/bus/bus.go:84:9: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:137:10: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:177:9: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:183:10: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:199:9: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:208:9: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/components/dynmap/dynmap.go:236:9: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:242:10: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:257:9: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:263:10: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:278:9: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:284:10: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:299:9: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:331:9: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:350:9: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:356:10: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:366:12: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:390:9: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:396:10: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:405:12: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:427:9: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:433:10: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:442:12: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:459:9: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:465:10: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:474:12: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:491:9: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:497:10: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:506:12: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:523:9: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:529:10: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:538:12: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:555:9: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:561:10: if block ends with a return statement, so drop this else and outdent its block
pkg/components/dynmap/dynmap.go:570:12: if block ends with a return statement, so drop this else and outdent its block
pkg/login/ldap.go:55:11: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/login/ldap_test.go:372:10: if block ends with a return statement, so drop this else and outdent its block
pkg/middleware/middleware_test.go:213:12: if block ends with a return statement, so drop this else and outdent its block
pkg/plugins/dashboard_importer.go:153:11: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/plugins/dashboards_updater.go:39:9: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/plugins/dashboards_updater.go:121:10: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/plugins/plugins.go:210:9: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/plugins/plugins.go:235:9: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/services/alerting/eval_context.go:111:9: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/services/alerting/notifier.go:92:9: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/services/alerting/notifier.go:98:9: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/services/alerting/notifier.go:122:10: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/services/alerting/rule.go:108:10: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/services/alerting/rule.go:118:10: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/services/alerting/rule.go:121:11: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/services/alerting/notifiers/telegram.go:94:10: if block ends with a return statement, so drop this else and outdent its block
pkg/services/sqlstore/annotation.go:34:11: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/services/sqlstore/annotation.go:99:11: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/services/sqlstore/dashboard_test.go:107:13: if block ends with a return statement, so drop this else and outdent its block
pkg/services/sqlstore/plugin_setting.go:78:10: if block ends with a return statement, so drop this else and outdent its block
pkg/services/sqlstore/preferences.go:91:10: if block ends with a return statement, so drop this else and outdent its block
pkg/services/sqlstore/user.go:50:10: if block ends with a return statement, so drop this else and outdent its block
pkg/services/sqlstore/migrator/migrator.go:106:11: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/services/sqlstore/migrator/postgres_dialect.go:48:10: if block ends with a return statement, so drop this else and outdent its block
pkg/tsdb/time_range.go:59:9: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/tsdb/time_range.go:67:9: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
pkg/tsdb/cloudwatch/metric_find_query.go:225:9: if block ends with a return statement, so drop this else and outdent its block
pkg/util/filepath.go:68:11: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)
This commit fixes the following golint warnings:
pkg/api/avatar/avatar.go:229:12: should omit type *http.Client from declaration of var client; it will be inferred from the right-hand side
pkg/login/brute_force_login_protection.go:13:26: should omit type time.Duration from declaration of var loginAttemptsWindow; it will be inferred from the right-hand side
pkg/metrics/graphitebridge/graphite.go:58:26: should omit type []string from declaration of var metricCategoryPrefix; it will be inferred from the right-hand side
pkg/metrics/graphitebridge/graphite.go:69:22: should omit type []string from declaration of var trimMetricPrefix; it will be inferred from the right-hand side
pkg/models/alert.go:37:36: should omit type error from declaration of var ErrCannotChangeStateOnPausedAlert; it will be inferred from the right-hand side
pkg/models/alert.go:38:36: should omit type error from declaration of var ErrRequiresNewState; it will be inferred from the right-hand side
pkg/models/datasource.go:61:28: should omit type map[string]bool from declaration of var knownDatasourcePlugins; it will be inferred from the right-hand side
pkg/plugins/update_checker.go:16:13: should omit type http.Client from declaration of var httpClient; it will be inferred from the right-hand side
pkg/services/alerting/engine.go:103:24: should omit type time.Duration from declaration of var unfinishedWorkTimeout; it will be inferred from the right-hand side
pkg/services/alerting/engine.go:105:19: should omit type time.Duration from declaration of var alertTimeout; it will be inferred from the right-hand side
pkg/services/alerting/engine.go:106:19: should omit type int from declaration of var alertMaxAttempts; it will be inferred from the right-hand side
pkg/services/alerting/notifier.go:143:23: should omit type map[string]*NotifierPlugin from declaration of var notifierFactories; it will be inferred from the right-hand side
pkg/services/alerting/rule.go:136:24: should omit type map[string]ConditionFactory from declaration of var conditionFactories; it will be inferred from the right-hand side
pkg/services/alerting/conditions/evaluator.go:12:15: should omit type []string from declaration of var defaultTypes; it will be inferred from the right-hand side
pkg/services/alerting/conditions/evaluator.go:13:15: should omit type []string from declaration of var rangedTypes; it will be inferred from the right-hand side
pkg/services/alerting/notifiers/opsgenie.go:44:19: should omit type string from declaration of var opsgenieAlertURL; it will be inferred from the right-hand side
pkg/services/alerting/notifiers/pagerduty.go:43:23: should omit type string from declaration of var pagerdutyEventApiUrl; it will be inferred from the right-hand side
pkg/services/alerting/notifiers/telegram.go:21:17: should omit type string from declaration of var telegramApiUrl; it will be inferred from the right-hand side
pkg/services/provisioning/dashboards/config_reader_test.go:11:24: should omit type string from declaration of var simpleDashboardConfig; it will be inferred from the right-hand side
pkg/services/provisioning/dashboards/config_reader_test.go:12:24: should omit type string from declaration of var oldVersion; it will be inferred from the right-hand side
pkg/services/provisioning/dashboards/config_reader_test.go:13:24: should omit type string from declaration of var brokenConfigs; it will be inferred from the right-hand side
pkg/services/provisioning/dashboards/file_reader.go:22:30: should omit type time.Duration from declaration of var checkDiskForChangesInterval; it will be inferred from the right-hand side
pkg/services/provisioning/dashboards/file_reader.go:24:23: should omit type error from declaration of var ErrFolderNameMissing; it will be inferred from the right-hand side
pkg/services/provisioning/datasources/config_reader_test.go:15:34: should omit type string from declaration of var twoDatasourcesConfig; it will be inferred from the right-hand side
pkg/services/provisioning/datasources/config_reader_test.go:16:34: should omit type string from declaration of var twoDatasourcesConfigPurgeOthers; it will be inferred from the right-hand side
pkg/services/provisioning/datasources/config_reader_test.go:17:34: should omit type string from declaration of var doubleDatasourcesConfig; it will be inferred from the right-hand side
pkg/services/provisioning/datasources/config_reader_test.go:18:34: should omit type string from declaration of var allProperties; it will be inferred from the right-hand side
pkg/services/provisioning/datasources/config_reader_test.go:19:34: should omit type string from declaration of var versionZero; it will be inferred from the right-hand side
pkg/services/provisioning/datasources/config_reader_test.go:20:34: should omit type string from declaration of var brokenYaml; it will be inferred from the right-hand side
pkg/services/sqlstore/stats.go:16:25: should omit type time.Duration from declaration of var activeUserTimeLimit; it will be inferred from the right-hand side
pkg/services/sqlstore/migrator/mysql_dialect.go:69:14: should omit type bool from declaration of var hasLen1; it will be inferred from the right-hand side
pkg/services/sqlstore/migrator/mysql_dialect.go:70:14: should omit type bool from declaration of var hasLen2; it will be inferred from the right-hand side
pkg/services/sqlstore/migrator/postgres_dialect.go:95:14: should omit type bool from declaration of var hasLen1; it will be inferred from the right-hand side
pkg/services/sqlstore/migrator/postgres_dialect.go:96:14: should omit type bool from declaration of var hasLen2; it will be inferred from the right-hand side
pkg/setting/setting.go:42:15: should omit type string from declaration of var Env; it will be inferred from the right-hand side
pkg/setting/setting.go:161:18: should omit type bool from declaration of var LdapAllowSignup; it will be inferred from the right-hand side
pkg/setting/setting.go:473:30: should omit type bool from declaration of var skipStaticRootValidation; it will be inferred from the right-hand side
pkg/tsdb/interval.go:14:21: should omit type time.Duration from declaration of var defaultMinInterval; it will be inferred from the right-hand side
pkg/tsdb/interval.go:15:21: should omit type time.Duration from declaration of var year; it will be inferred from the right-hand side
pkg/tsdb/interval.go:16:21: should omit type time.Duration from declaration of var day; it will be inferred from the right-hand side
pkg/tsdb/cloudwatch/credentials.go:26:24: should omit type map[string]cache from declaration of var awsCredentialCache; it will be inferred from the right-hand side
pkg/tsdb/influxdb/query.go:15:27: should omit type *regexp.Regexp from declaration of var regexpOperatorPattern; it will be inferred from the right-hand side
pkg/tsdb/influxdb/query.go:16:27: should omit type *regexp.Regexp from declaration of var regexpMeasurementPattern; it will be inferred from the right-hand side
pkg/tsdb/mssql/mssql_test.go:25:14: should omit type string from declaration of var serverIP; it will be inferred from the right-hand side
At least in my Postgresql 9.6.5, the old syntax of capitalized queries doesn't work (Linux).<br>
Running with UTF-8 as standard encoding and the ` notations didn't work either, so removed those.
* use utf8mb4 character set for connections to mysql
* use utf8mb4 character set for tables, shorten varchar fields used in unique indexes
* migration type to update table character set
* update table character sets
* set charset for temp_user.status
* gofmt
