Update the Alerting HA docs (#60292)

* Update the Alerting HA docs

* Update docs/sources/alerting/set-up/configure-high-availability/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/set-up/configure-high-availability/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/set-up/configure-high-availability/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/set-up/configure-high-availability/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update docs/sources/alerting/set-up/configure-high-availability/_index.md

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

* Update HA introduction paragraph

* Apply prettier

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

docs/sources/alerting/set-up/configure-high-availability/_index.md

@@ -15,9 +15,9 @@ weight: 300
 
 # Enable alerting high availability
 
-You can enable alerting high availability support by updating the Grafana configuration file. On Kubernetes, you can enable alerting high availability by updating the Kubernetes container definition.
+You can enable alerting high availability support by updating the Grafana configuration file. If you run Grafana in a Kubernetes cluster, additional steps are required. Both options are described below.
 
-## Update Grafana configuration file
+## Enable alerting high availability in Grafana
 
 ### Before you begin
 
@@ -31,9 +31,9 @@ Since gossiping of notifications and silences uses both TCP and UDP port `9094`,
 3. Set `[ha_listen_address]` to the instance IP address using a format of `host:port` (or the [Pod's](https://kubernetes.io/docs/concepts/workloads/pods/) IP in the case of using Kubernetes).
    By default, it is set to listen to all interfaces (`0.0.0.0`).
 
-## Update Kubernetes container definition
+## Enable alerting high availability using Kubernetes
 
-If you are using Kubernetes, you can expose the pod IP [through an environment variable](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/) via the container definition such as:
+If you are using Kubernetes, you can expose the pod IP [through an environment variable](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/) via the container definition.
 
 ```bash
 env:
@@ -42,3 +42,57 @@ env:
     fieldRef:
       fieldPath: status.podIP
 ```
+
+1. Add the port 9094 to the Grafana deployment:
+
+```yaml
+ports:
+  - containerPort: 3000
+    name: http-grafana
+    protocol: TCP
+  - containerPort: 9094
+    name: grafana-alert
+    protocol: TCP
+```
+
+2. Add the environment variables to the Grafana deployment:
+
+```yaml
+env:
+  - name: POD_IP
+    valueFrom:
+      fieldRef:
+        fieldPath: status.podIP
+```
+
+3. Create a headless service that returns the pod IP instead of the service IP, which is what the `ha_peers` need:
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: grafana-alerting
+  namespace: grafana
+  labels:
+    app.kubernetes.io/name: grafana-alerting
+    app.kubernetes.io/part-of: grafana
+spec:
+  type: ClusterIP
+  clusterIP: 'None'
+  ports:
+    - port: 9094
+  selector:
+    app: grafana
+```
+
+4. Make sure your grafana deployment has the label matching the selector, e.g. `app:grafana`.
+
+5. Add in the grafana.ini:
+
+```bash
+[unified_alerting]
+enabled = true
+ha_listen_address = "${POD_IP}:9094"
+ha_peers = "grafana-alerting.grafana:9094"
+ha_advertise_address = "${POD_IP}:9094"
+```

docs/sources/setup-grafana/set-up-for-high-availability.md

@@ -28,7 +28,7 @@ Grafana will now persist all long term data in the database. How to configure th
 
 ## Alerting high availability
 
-Grafana Alerting provides a new [high availability mode](https://grafana.com/docs/grafana/latest/alerting/fundamentals/high-availability). It also preserves the semantics of legacy dashboard alerting by executing all alerts on every server and by sending notifications only once per alert. Load distribution between servers is not supported at this time.
+Grafana Alerting provides a [high availability mode](https://grafana.com/docs/grafana/latest/alerting/fundamentals/high-availability). It preserves the semantics of legacy dashboard alerting by executing all alerts on every server and by sending notifications only once per alert. Load distribution between servers is not supported at this time.
 
 For instructions on setting up alerting high availability, refer to [Enable alerting high availability](https://grafana.com/docs/grafana/next/alerting/set-up/configure-high-availability/).