IntenseWebs/grafana
3
0
Fork 0
mirror of https://github.com/grafana/grafana.git synced 2025-01-08 23:23:45 -06:00
Code Issues Packages Projects Releases Wiki Activity

live: experimental HA with Redis (#36787)

Browse Source 
Grafana 8.1.0 will include experimental high availability setup support by
using Redis PUB/SUB and data structures for inter-node state sharing.
This commit documents this but also explicitly marks feature as experimental
since integration of Live with Redis may slightly evolve with time as soon as
we get more understanding about Grafana Live usage scenarios.
This commit is contained in:
Alexander Emelin 2021-07-27 10:40:00 +03:00 committed by GitHub
parent e8ac802e4f
commit 75bd0bbacd
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 68 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
docs/sources/administration/configuration.md
View File

@ -1529,6 +1529,30 @@ For example:
allowed_origins = "https://*.example.com"
```
### ha_engine
> **Note**: Available in Grafana v8.1 and later versions.
**Experimental**
The high availability (HA) engine name for Grafana Live. By default, it's not set. The only possible value is "redis".
For more information, refer to [Configure Grafana Live HA setup]({{< relref "../live/live-ha-setup.md" >}}).
### ha_engine_address
> **Note**: Available in Grafana v8.1 and later versions.
**Experimental**
Address string of selected the high availability (HA) Live engine. For Redis, it's a `host:port` string. Example:
```ini
[live]
ha_engine = redis
ha_engine_address = 127.0.0.1:6379
```
<hr>
## [plugin.grafana-image-renderer]

46
docs/sources/live/live-ha-setup.md
View File

@ -7,10 +7,52 @@ weight = 130
# Configure Grafana Live HA setup
Live features in Grafana v8.0 are designed to work with a single Grafana server instance only. We will add the option for HA configuration in future Grafana releases to eliminate the current limitations.
By default, Grafana Live uses in-memory data structures and in-memory PUB/SUB hub for handling subscriptions.
Currently, if you have several Grafana server instances behind a load balancer, you may come across the following limitations:
In a high availability Grafana setup involving several Grafana server instances behind a load balancer, you can find the following limitations:
- Built-in features like dashboard change notifications will only be broadcasted to users connected to the same Grafana server process instance.
- Streaming from Telegraf will deliver data only to clients connected to the same instance which received Telegraf data, active stream cache is not shared between different Grafana instances.
- A separate unidirectional stream between Grafana and backend data source may be opened on different Grafana servers for the same channel.
To bypass these limitations, Grafana v8.1 has an experimental Live HA engine that requires Redis to work.
## Configure Redis Live engine
When the Redis engine is configured, Grafana Live keeps its state in Redis and uses Redis PUB/SUB functionality to deliver messages to all subscribers throughout all Grafana server nodes.
Here is an example configuration:
```
[live]
ha_engine = redis
ha_engine_address = 127.0.0.1:6379
```
After running:
- All built-in real-time notifications like dashboard changes are delivered to all Grafana server instances and broadcasted to all subscribers.
- Streaming from Telegraf delivers messages to all subscribers.
- A separate unidirectional stream between Grafana and backend data source opens on different Grafana servers. Publishing data to a channel delivers messages to instance subscribers, as a result, publications from different instances on different machines do not produce duplicate data on panels.
At the moment we only support single Redis node.
> **Note:** It's possible to use Redis Sentinel and Haproxy to achieve a highly available Redis setup. Redis nodes should be managed by [Redis Sentinel](https://redis.io/topics/sentinel) to achieve automatic failover. Haproxy configuration example:
> ```
> listen redis
> server redis-01 127.0.0.1:6380 check port 6380 check inter 2s weight 1 inter 2s downinter 5s rise 10 fall 2 on-marked-down shutdown-sessions on-marked-up shutdown-backup-sessions
> server redis-02 127.0.0.1:6381 check port 6381 check inter 2s weight 1 inter 2s downinter 5s rise 10 fall 2 backup
> bind *:6379
> mode tcp
> option tcpka
> option tcplog
> option tcp-check
> tcp-check send PING\r\n
> tcp-check expect string +PONG
> tcp-check send info\ replication\r\n
> tcp-check expect string role:master
> tcp-check send QUIT\r\n
> tcp-check expect string +OK
> balance roundrobin
> ```
> Next, point Grafana Live to Haproxy address:port.