grafana/devenv/docker/ha_test
Gilles De Mey 8765c48389
Alerting: Remove legacy alerting (#83671)
Removes legacy alerting, so long and thanks for all the fish! 🐟

---------

Co-authored-by: Matthew Jacobson <matthew.jacobson@grafana.com>
Co-authored-by: Sonia Aguilar <soniaAguilarPeiron@users.noreply.github.com>
Co-authored-by: Armand Grillet <armandgrillet@users.noreply.github.com>
Co-authored-by: William Wernert <rwwiv@users.noreply.github.com>
Co-authored-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
2024-03-14 15:36:35 +01:00
..
fluentd devenv: send nginx logs to loki in ha test 2019-02-13 11:18:29 +01:00
grafana/provisioning Alerting: Remove legacy alerting (#83671) 2024-03-14 15:36:35 +01:00
prometheus devenv: use grafana/fluent-plugin-loki 2019-02-13 11:18:29 +01:00
.gitignore devenv: grafana high availability (ha) test setup 2018-09-27 09:24:40 +02:00
alerts.sh Alerting: Remove legacy alerting (#83671) 2024-03-14 15:36:35 +01:00
docker-compose.yaml Grafana: Upgrades mysql images from 5.7 to 8 (#67604) 2023-05-04 11:42:48 -06:00
README.md Docs: Refactor inconsistent unordered lists (#27826) 2020-10-02 11:02:11 -07:00

Grafana High Availability (HA) test setup

A set of docker compose services which together creates a Grafana HA test setup with capability of easily scaling up/down number of Grafana instances.

Included services

  • Grafana
  • Mysql - Grafana configuration database and session storage
  • Prometheus - Monitoring of Grafana and used as data source of provisioned alert rules
  • Nginx - Reverse proxy for Grafana and Prometheus. Enables browsing Grafana/Prometheus UI using a hostname

Prerequisites

Build grafana docker container

Build a Grafana docker container from current branch and commit and tag it as grafana/grafana:dev.

$ cd <grafana repo>
$ make build-docker-full

Virtual host names

Alternative 1 - Use dnsmasq

$ sudo apt-get install dnsmasq
$ echo 'address=/loc/127.0.0.1' | sudo tee /etc/dnsmasq.d/dnsmasq-loc.conf > /dev/null
$ sudo /etc/init.d/dnsmasq restart
$ ping whatever.loc
PING whatever.loc (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.076 ms
--- whatever.loc ping statistics ---
1 packet transmitted, 1 received, 0% packet loss, time 1998ms

Alternative 2 - Manually update /etc/hosts

Update your /etc/hosts to be able to access Grafana and/or Prometheus UI using a hostname.

$ cat /etc/hosts
127.0.0.1       grafana.loc
127.0.0.1       prometheus.loc

Start services

$ docker-compose up -d

Browse

Check for any errors

$ docker-compose logs | grep error

Scale Grafana instances up/down

Scale number of Grafana instances to <instances>

$ docker-compose up --scale grafana=<instances> -d
# for example 3 instances
$ docker-compose up --scale grafana=3 -d

Test alerting

Create notification channels

Creates default notification channels, if not already exists

$ ./alerts.sh setup

Slack notifications

Disable

$ ./alerts.sh slack -d

Enable and configure url

$ ./alerts.sh slack -u https://hooks.slack.com/services/...

Enable, configure url and enable reminders

$ ./alerts.sh slack -u https://hooks.slack.com/services/... -r -e 10m

Provision alert dashboards with alert rules

Provision 1 dashboard/alert rule (default)

$ ./alerts.sh provision

Provision 10 dashboards/alert rules

$ ./alerts.sh provision -a 10

Provision 10 dashboards/alert rules and change condition to gt > 100

$ ./alerts.sh provision -a 10 -c 100

Pause/unpause all alert rules

Pause

$ ./alerts.sh pause

Unpause

$ ./alerts.sh unpause