Live: docs (#34642)

docs/sources/administration/configuration.md

@@ -1487,6 +1487,20 @@ Custom install/learn more URL for enterprise plugins. Defaults to https://grafan
 
 <hr>
 
+## [live]
+
+### max_connections
+
+> **Note**: Available in Grafana v8.0 and later versions.
+
+The `max_connections` option specifies the maximum number of connections to the Grafana Live WebSocket endpoint per Grafana server instance. Default is `100`.
+
+Refer to [Grafana Live configuration documentation]({{< relref "../live/configure-grafana-live.md" >}}) if you specify a number higher than default since this can require some operating system and infrastructure tuning.
+
+0 disables Grafana Live, -1 means unlimited connections.
+
+<hr>
+
 ## [plugin.grafana-image-renderer]
 
 For more information, refer to [Image rendering]({{< relref "image_rendering.md" >}}).

docs/sources/live/_index.md

@@ -0,0 +1,15 @@
++++
+title = "Grafana Live"
+aliases = []
+weight = 115
++++
+
+# Grafana Live overview
+
+Grafana Live is a real-time messaging engine introduced in Grafana v8.0.
+
+With Grafana Live, you can push event data to a frontend as soon as an event occurs.
+
+This could be notifications about dashboard changes, new frames for rendered data, and so on. Live features can help eliminate a page reload or polling in many places, it can stream Internet of things (IOT) sensors or any other real-time data to panels.
+
+> **Note:** By `real-time`, we indicate a soft real-time. Due to network latencies, garbage collection cycles, and so on, the delay of a delivered message can be up to several hundred milliseconds or higher.

docs/sources/live/configure-grafana-live.md

@@ -0,0 +1,101 @@
++++
+title = "Configure Grafana Live"
+description = "Grafana Live configuration guide"
+keywords = ["Grafana", "live", "guide", "websocket"]
+weight = 120
++++
+
+# Configure Grafana Live
+
+Grafana Live is enabled by default. In Grafana v8.0, it has a strict default for a maximum number of connections per Grafana server instance. 
+
+## Max number of connections
+
+Grafana Live uses persistent connections (WebSocket at the moment) to deliver real-time updates to clients.
+
+WebSocket is a persistent connection that starts with an HTTP Upgrade request (using the same HTTP port as the rest of Grafana) and then switches to a TCP mode where WebSocket frames can travel in both directions between a client and a server. Each logged-in user opens a WebSocket connection – one per browser tab.
+
+The number of maximum WebSocket connections users can establish with Grafana is limited to 100 by default. See [max_connections]({{< relref "../administration/configuration.md#max_connections" >}}) option.
+
+In case you want to increase this limit, ensure that your server and infrastructure allow handling more connections. The following sections discuss several common problems which could happen when managing persistent connections, in particular WebSocket connections.
+
+### Resource usage
+
+Each persistent connection costs some memory on a server. Typically, this should be about 50 KB per connection at this moment. Thus a server with 1 GB RAM is expected to handle about 20k connections max. Each active connection consumes additional CPU resources since the client and server send PING/PONG frames to each other to maintain a connection.
+
+Using the streaming functionality results in additional CPU usage. The exact CPU resource utilization can be hard to estimate as it heavily depends on the Grafana Live usage pattern.
+
+### Open file limit
+
+Each WebSocket connection costs a file descriptor on a server machine where Grafana runs. Most operating systems have a quite low default limit for the maximum number of descriptors that process can open.
+
+To look at the current limit on Unix run:
+
+```
+ulimit -n
+```
+
+On a Linux system, you can also check out the current limits for a running process with:
+
+```
+cat /proc/<PROCESS_PID>/limits
+```
+
+The open files limit shows approximately how many user connections your server can currently handle.
+		
+
+To increase this limit, refer to [these instructions](https://docs.riak.com/riak/kv/2.2.3/using/performance/open-files-limit.1.html)for popular operating systems.
+
+### Ephemeral port exhaustion
+
+Ephemeral port exhaustion problem can happen between your load balancer (or reverse proxy) software and Grafana server. For example, when you load balance requests/connections between different Grafana instances. If you connect directly to a single Grafana server instance, then you should not come across this issue.
+
+The problem arises because each TCP connection uniquely identified in the OS by the 4-part-tuple:
+
+```
+source ip | source port | destination ip | destination port
+```
+
+By default, on load balancer/server boundary you are limited to 65535 possible variants. But actually, due to some OS limits (for example on Unix available ports defined in `ip_local_port_range` sysctl parameter) and sockets in TIME_WAIT state, the number is even less.
+
+In order to eliminate a problem you can:
+
+- Increase the ephemeral port range by tuning `ip_local_port_range` kernel option.
+- Deploy more Grafana server instances to load balance across.
+- Deploy more load balancer instances.
+- Use virtual network interfaces.
+
+### WebSocket and proxies
+
+Not all proxies can transparently proxy WebSocket connections by default. For example, if you are using Nginx before Grafana you need to configure WebSocket proxy like this:
+
+```
+http {
+    map $http_upgrade $connection_upgrade {
+        default upgrade;
+        '' close;
+    }
+ 
+    upstream grafana {
+        server 127.0.0.1:3000;
+    }
+ 
+    server {
+        listen 8000;
+
+        location / {
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "Upgrade";
+            proxy_set_header Host $http_host;
+            proxy_pass http://grafana;
+        }
+    }
+}
+```
+
+See the [Nginx blog on their website](https://www.nginx.com/blog/websocket-nginx/) for more information. Also, refer to your load balancer/reverse proxy documentation to find out more information on dealing with WebSocket connections.
+
+Some corporate proxies can remove headers required to properly establish a WebSocket connection. In this case, you should tune intermediate proxies to not remove required headers. However, the better option is to use Grafana with TLS. Now WebSocket connection will inherit TLS and thus must be handled transparently by proxies.
+
+Proxies like Nginx and Envoy have default limits on maximum number of connections which can be established. Make sure you have a reasonable limit for max number of incoming and outgoing connections in your proxy configuration.

docs/sources/live/live-channel.md

@@ -0,0 +1,36 @@
++++
+title = "Live Channel"
+description = "Grafana Live channel guide"
+keywords = ["Grafana", "live", "guide", "channel"]
+weight = 110
++++
+
+# Grafana Live Channel guide
+
+Grafana Live is a PUB/SUB server, clients subscribe to channels to receive real-time updates published to those channels.
+
+## Channel structure
+
+Channel is a string identifier. In Grafana channel consists of 3 parts delimited by `/`:
+
+- Scope
+- Namespace
+- Path
+
+For example, the channel `grafana/dashboard/xyz` has the scope `grafana`, namespace `dashboard`, and path `xyz`.
+
+Scope, namespace and path can only have ASCII alphanumeric symbols (A-Z, a-z, 0-9), `_` (underscore) and `-` (dash) at the moment. The path part can additionally have `/`, `.` and `=` symbols. The meaning of scope, namespace and path is context-specific.
+
+The maximum length of a channel is 160 symbols.
+
+Scope determines the purpose of a channel in Grafana. For example, for data source plugin channels Grafana uses `ds` scope. For built-in features like dashboard edit notifications Grafana uses `grafana` scope.
+
+Namespace has a different meaning depending on scope. For example, for `grafana` scope this could be a name of built-in real-time feature like `dashboard` (i.e. dashboards events).
+
+The path, which is the final part of a channel, usually contains the identifier of some concrete resource such as the ID of a dashboard that a user is currently looking at. But a path can be anything.
+
+Channels are lightweight and ephemeral - they are created automatically on user subscription and removed as soon as last user left a channel.
+
+## Data format
+
+All data travelling over Live channels must be JSON-encoded.

docs/sources/live/live-feature-overview.md

@@ -0,0 +1,32 @@
++++
+title = "Live feature overview"
+description = "Grafana Live feature overview"
+keywords = ["Grafana", "live", "guide"]
+weight = 100
++++
+
+# Grafana Live feature overview
+
+This topic explains the current Grafana Live capabilities.
+
+## Dashboard change notifications 
+
+As soon as there is a change to the dashboard layout, it is automatically reflected on other devices connected to Grafana Live.
+
+## Data streaming from plugins
+
+With Grafana Live data source, plugins can stream data updates in the form of Grafana data frames to a frontend.
+
+For data source plugin channels Grafana uses `ds` scope. Namespace in the case of data source channels is a data source unique ID (UID) which is issued by Grafana at the moment of data source creation. The path is a custom string that plugin authors free to choose themselves (just make sure it consists of allowed symbols).
+
+For example, a data source channel looks like this: `ds/<DATASOURCE_UID>/<CUSTOM_PATH>`.
+
+Refer to the tutorial about [building a streaming data source backend plugin](https://grafana.com/tutorials/build-a-streaming-data-source-plugin/) for more details.
+
+The basic streaming example included in Grafana core streams frames with some generated data to a panel. To look at it create a new panel and point it to the `-- Grafana --` data source. Next, choose `Live Measurements` and select the `plugin/testdata/random-20Hz-stream` channel. 
+
+## Data streaming from Telegraf
+
+A new API endpoint `/api/live/push/:streamId` allows accepting metrics data in Influx format from Telegraf. These metrics are transformed into Grafana data frames and published to channels.
+
+Refer to the tutorial about [streaming metrics from Telegraf to Grafana](https://grafana.com/tutorials/stream-metrics-from-telegraf-to-grafana/) for more information.

docs/sources/live/live-ha-setup.md

@@ -0,0 +1,16 @@
++++
+title = "Live HA setup"
+description = "Grafana Live HA setup guide"
+keywords = ["Grafana", "live", "guide", "ha"]
+weight = 130
++++
+
+# Configure Grafana Live HA setup
+
+Live features in Grafana v8.0 are designed to work with a single Grafana server instance only. We will add the option for HA configuration in future Grafana releases to eliminate the current limitations.
+
+Currently, if you have several Grafana server instances behind a load balancer, you may come across the following limitations:
+
+- Built-in features like dashboard change notifications will only be broadcasted to users connected to the same Grafana server process instance.
+- Streaming from Telegraf will deliver data only to clients connected to the same instance which received Telegraf data, active stream cache is not shared between different Grafana instances.
+- A separate unidirectional stream between Grafana and backend data source may be opened on different Grafana servers for the same channel.

docs/sources/whatsnew/whats-new-in-v8-0.md

@@ -36,10 +36,12 @@ Library panels allow users to build panels that can be used in multiple dashboar
 
 Data sources can now send real-time updates to dashboards over a websocket connection. This can be used with the [MQTT data source](https://github.com/grafana/mqtt-datasource).
 
-In addition to data source integration, events can be sent to dashboards by posting metrics to the new live endpoint: `/api/live/push endpoint`.
+In addition to data source integration, events can be sent to dashboards by posting metrics to the new live endpoint: `/api/live/push` endpoint.
 
 These metrics will be broadcast to all dashboards connected to that stream endpoint.
 
+For more information about real-time streaming, refer to [Grafana Live documentation]({{< relref "../live/_index.md" >}}).
+
 ### Bar chart visualization (beta)
 
 The Bar chart panel is a new visualization that allows categorical data display.