IntenseWebs/grafana
3
0
Fork 0
mirror of https://github.com/grafana/grafana.git synced 2025-02-20 11:48:34 -06:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #13400 from connection-reset/concurrent-sql

Browse Source 
Run queries for sql data sources for panels with multiple queries concurrently
Also make max open connections, max idle connections and max connection 
lifetime configurable in the data source settings
This commit is contained in:
Marcus Efraimsson 2018-10-10 19:12:38 +02:00 committed by GitHub
commit bc3b7357bd
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
8 changed files with 152 additions and 23 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
docs/sources/administration/provisioning.md
View File

@ -168,6 +168,9 @@ Since not all datasources have the same configuration settings we only have the
| sslmode | string | PostgreSQL | SSLmode. 'disable', 'require', 'verify-ca' or 'verify-full' |
| postgresVersion | number | PostgreSQL | Postgres version as a number (903/904/905/906/1000) meaning v9.3, v9.4, ..., v10 |
| timescaledb | boolean | PostgreSQL | Enable usage of TimescaleDB extension |
| maxOpenConns | number | MySQL, PostgreSQL & MSSQL | Maximum number of open connections to the database (Grafana v5.4+) |
| maxIdleConns | number | MySQL, PostgreSQL & MSSQL | Maximum number of connections in the idle connection pool (Grafana v5.4+) |
| connMaxLifetime | number | MySQL, PostgreSQL & MSSQL | Maximum amount of time in seconds a connection may be reused (Grafana v5.4+) |
#### Secure Json Data

7
docs/sources/features/datasources/mssql.md
View File

@ -32,6 +32,9 @@ Name | Description
*Database* | Name of your MSSQL database.
*User* | Database user's login/username
*Password* | Database user's password
*Max open* | The maximum number of open connections to the database, default `unlimited` (Grafana v5.4+).
*Max idle* | The maximum number of connections in the idle connection pool, default `2` (Grafana v5.4+).
*Max lifetime* | The maximum amount of time in seconds a connection may be reused, default `14400`/4 hours (Grafana v5.4+).
### Min time interval
@ -585,6 +588,10 @@ datasources:
url: localhost:1433
database: grafana
user: grafana
jsonData:
maxOpenConns: 0 # Grafana v5.4+
maxIdleConns: 2 # Grafana v5.4+
connMaxLifetime: 14400 # Grafana v5.4+
secureJsonData:
password: "Password!"

7
docs/sources/features/datasources/mysql.md
View File

@ -35,6 +35,9 @@ Name | Description
*Database* | Name of your MySQL database.
*User* | Database user's login/username
*Password* | Database user's password
*Max open* | The maximum number of open connections to the database, default `unlimited` (Grafana v5.4+).
*Max idle* | The maximum number of connections in the idle connection pool, default `2` (Grafana v5.4+).
*Max lifetime* | The maximum amount of time in seconds a connection may be reused, default `14400`/4 hours. This should always be lower than configured [wait_timeout](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_wait_timeout) in MySQL (Grafana v5.4+).
### Min time interval
@ -316,4 +319,8 @@ datasources:
database: grafana
user: grafana
password: password
jsonData:
maxOpenConns: 0 # Grafana v5.4+
maxIdleConns: 2 # Grafana v5.4+
connMaxLifetime: 14400 # Grafana v5.4+
```

6
docs/sources/features/datasources/postgres.md
View File

@ -31,6 +31,9 @@ Name | Description
*User* | Database user's login/username
*Password* | Database user's password
*SSL Mode* | This option determines whether or with what priority a secure SSL TCP/IP connection will be negotiated with the server.
*Max open* | The maximum number of open connections to the database, default `unlimited` (Grafana v5.4+).
*Max idle* | The maximum number of connections in the idle connection pool, default `2` (Grafana v5.4+).
*Max lifetime* | The maximum amount of time in seconds a connection may be reused, default `14400`/4 hours (Grafana v5.4+).
*Version* | This option determines which functions are available in the query builder (only available in Grafana 5.3+).
*TimescaleDB* | TimescaleDB is a time-series database built as a PostgreSQL extension. If enabled, Grafana will use `time_bucket` in the `$__timeGroup` macro and display TimescaleDB specific aggregate functions in the query builder (only available in Grafana 5.3+).
@ -374,6 +377,9 @@ datasources:
password: "Password!"
jsonData:
sslmode: "disable" # disable/require/verify-ca/verify-full
maxOpenConns: 0 # Grafana v5.4+
maxIdleConns: 2 # Grafana v5.4+
connMaxLifetime: 14400 # Grafana v5.4+
postgresVersion: 903 # 903=9.3, 904=9.4, 905=9.5, 906=9.6, 1000=10
timescaledb: false
```

58
pkg/tsdb/sql_engine.go
View File

@ -98,8 +98,12 @@ var NewSqlQueryEndpoint = func(config *SqlQueryEndpointConfiguration, rowTransfo
return nil, err
}
engine.SetMaxOpenConns(10)
engine.SetMaxIdleConns(10)
maxOpenConns := config.Datasource.JsonData.Get("maxOpenConns").MustInt(0)
engine.SetMaxOpenConns(maxOpenConns)
maxIdleConns := config.Datasource.JsonData.Get("maxIdleConns").MustInt(2)
engine.SetMaxIdleConns(maxIdleConns)
connMaxLifetime := config.Datasource.JsonData.Get("connMaxLifetime").MustInt(14400)
engine.SetConnMaxLifetime(time.Duration(connMaxLifetime) * time.Second)
engineCache.versions[config.Datasource.Id] = config.Datasource.Version
engineCache.cache[config.Datasource.Id] = engine
@ -116,9 +120,7 @@ func (e *sqlQueryEndpoint) Query(ctx context.Context, dsInfo *models.DataSource,
Results: make(map[string]*QueryResult),
}
session := e.engine.NewSession()
defer session.Close()
db := session.DB()
var wg sync.WaitGroup
for _, query := range tsdbQuery.Queries {
rawSQL := query.Model.Get("rawSql").MustString()
@ -145,31 +147,41 @@ func (e *sqlQueryEndpoint) Query(ctx context.Context, dsInfo *models.DataSource,
queryResult.Meta.Set("sql", rawSQL)
rows, err := db.Query(rawSQL)
if err != nil {
queryResult.Error = err
continue
}
wg.Add(1)
defer rows.Close()
go func(rawSQL string, query *Query, queryResult *QueryResult) {
defer wg.Done()
session := e.engine.NewSession()
defer session.Close()
db := session.DB()
format := query.Model.Get("format").MustString("time_series")
switch format {
case "time_series":
err := e.transformToTimeSeries(query, rows, queryResult, tsdbQuery)
rows, err := db.Query(rawSQL)
if err != nil {
queryResult.Error = err
continue
return
}
case "table":
err := e.transformToTable(query, rows, queryResult, tsdbQuery)
if err != nil {
queryResult.Error = err
continue
defer rows.Close()
format := query.Model.Get("format").MustString("time_series")
switch format {
case "time_series":
err := e.transformToTimeSeries(query, rows, queryResult, tsdbQuery)
if err != nil {
queryResult.Error = err
return
}
case "table":
err := e.transformToTable(query, rows, queryResult, tsdbQuery)
if err != nil {
queryResult.Error = err
return
}
}
}
}(rawSQL, query, queryResult)
}
wg.Wait()
return result, nil
}

31
public/app/plugins/datasource/mssql/partials/config.html
View File

@ -29,6 +29,37 @@
</div>
</div>
<b>Connection limits</b>
<div class="gf-form-group">
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max open</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxOpenConns" placeholder="unlimited"></input>
<info-popover mode="right-absolute">
The maximum number of open connections to the database. If <i>Max idle connections</i> is greater than 0 and the
<i>Max open connections</i> is less than <i>Max idle connections</i>, then <i>Max idle connections</i> will be
reduced to match the <i>Max open connections</i> limit. If set to 0, there is no limit on the number of open
connections.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max idle</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxIdleConns" placeholder="2"></input>
<info-popover mode="right-absolute">
The maximum number of connections in the idle connection pool. If <i>Max open connections</i> is greater than 0 but
less than the <i>Max idle connections</i>, then the <i>Max idle connections</i> will be reduced to match the
<i>Max open connections</i> limit. If set to 0, no idle connections are retained.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max lifetime</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.connMaxLifetime" placeholder="14400"></input>
<info-popover mode="right-absolute">
The maximum amount of time in seconds a connection may be reused. If set to 0, connections are reused forever.
</info-popover>
</div>
</div>
<h3 class="page-heading">MSSQL details</h3>
<div class="gf-form-group">

32
public/app/plugins/datasource/mysql/partials/config.html
View File

@ -24,6 +24,38 @@
</div>
</div>
<b>Connection limits</b>
<div class="gf-form-group">
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max open</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxOpenConns" placeholder="unlimited"></input>
<info-popover mode="right-absolute">
The maximum number of open connections to the database. If <i>Max idle connections</i> is greater than 0 and the
<i>Max open connections</i> is less than <i>Max idle connections</i>, then <i>Max idle connections</i> will be
reduced to match the <i>Max open connections</i> limit. If set to 0, there is no limit on the number of open
connections.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max idle</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxIdleConns" placeholder="2"></input>
<info-popover mode="right-absolute">
The maximum number of connections in the idle connection pool. If <i>Max open connections</i> is greater than 0 but
less than the <i>Max idle connections</i>, then the <i>Max idle connections</i> will be reduced to match the
<i>Max open connections</i> limit. If set to 0, no idle connections are retained.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max lifetime</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.connMaxLifetime" placeholder="14400"></input>
<info-popover mode="right-absolute">
The maximum amount of time in seconds a connection may be reused. If set to 0, connections are reused forever.<br/><br/>
This should always be lower than configured <a href="https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_wait_timeout" target="_blank">wait_timeout</a> in MySQL.
</info-popover>
</div>
</div>
<h3 class="page-heading">MySQL details</h3>
<div class="gf-form-group">

31
public/app/plugins/datasource/postgres/partials/config.html
View File

@ -38,6 +38,37 @@
</div>
</div>
<b>Connection limits</b>
<div class="gf-form-group">
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max open</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxOpenConns" placeholder="unlimited"></input>
<info-popover mode="right-absolute">
The maximum number of open connections to the database. If <i>Max idle connections</i> is greater than 0 and the
<i>Max open connections</i> is less than <i>Max idle connections</i>, then <i>Max idle connections</i> will be
reduced to match the <i>Max open connections</i> limit. If set to 0, there is no limit on the number of open
connections.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max idle</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxIdleConns" placeholder="2"></input>
<info-popover mode="right-absolute">
The maximum number of connections in the idle connection pool. If <i>Max open connections</i> is greater than 0 but
less than the <i>Max idle connections</i>, then the <i>Max idle connections</i> will be reduced to match the
<i>Max open connections</i> limit. If set to 0, no idle connections are retained.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max lifetime</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.connMaxLifetime" placeholder="14400"></input>
<info-popover mode="right-absolute">
The maximum amount of time in seconds a connection may be reused. If set to 0, connections are reused forever.
</info-popover>
</div>
</div>
<h3 class="page-heading">PostgreSQL details</h3>
<div class="gf-form-group">