End2end: Make it possible to run e2e tests with BASE_URL (#24041)

.circleci/config.yml

@@ -663,6 +663,9 @@ jobs:
           name: Start grafana-server
           command: ./e2e/start-server
           background: true
+      - run:
+          name: "Wait for Grafana to start"
+          command: './e2e/wait-for-grafana'
       - run:
           name: Run end-to-end tests
           command: ./e2e/run-suite

contribute/developer-guide.md

@@ -97,29 +97,35 @@ go test -v ./pkg/...
 
 ### Run end-to-end tests
 
-The end to end tests in Grafana use [Cypress](https://www.cypress.io/) to run automated scripts in a headless Chromium browser. Read more about our [e2e framework](/contribute/style-guides/e2e.md). 
+The end to end tests in Grafana use [Cypress](https://www.cypress.io/) to run automated scripts in a headless Chromium browser. Read more about our [e2e framework](/contribute/style-guides/e2e.md).
 
 To run the tests:
 
 ```
-yarn e2e-tests
+yarn e2e
 ```
 
-By default, the end-to-end tests assume Grafana is available on `localhost:3000`. To use a specific URL, set the `BASE_URL` environment variable:
+By default, the end-to-end tests starts a Grafana instance listening on `localhost:3001`. To use a specific URL, set the `BASE_URL` environment variable:
 
 ```
-BASE_URL=http://localhost:3333 yarn e2e-tests
+BASE_URL=http://localhost:3333 yarn e2e
 ```
 
-To follow the tests in the browser while they're running, use the `yarn e2e-tests:debug` instead.
+To follow the tests in the browser while they're running, use the `yarn e2e:debug`.
 
 ```
-yarn e2e-tests:debug
+yarn e2e:debug
+```
+
+If you want to pick a test first, use the `yarn e2e:dev`, to pick a test and follow the test in the browser while it runs.
+
+```
+yarn e2e:dev
 ```
 
 ## Configure Grafana for development
 
-The default configuration, `grafana.ini`, is located in the `conf` directory. 
+The default configuration, `grafana.ini`, is located in the `conf` directory.
 
 To override the default configuration, create a `custom.ini` file in the `conf` directory. You only need to add the options you wish to override.
 
@@ -198,14 +204,14 @@ The number of files needed may be different on your environment. To determine th
 find ./conf ./pkg ./public/views | wc -l
 ```
 
-Another alternative is to limit the files being watched. The directories that are watched for changes are listed in the `.bra.toml` file in the root directory. 
+Another alternative is to limit the files being watched. The directories that are watched for changes are listed in the `.bra.toml` file in the root directory.
 
 To retain your `ulimit` configuration, i.e. so it will be remembered for future sessions, you need to commit it to your command line shell initialization file. Which file this will be depends on the shell you are using, here are some examples:
 
 * zsh -> ~/.zshrc
 * bash -> ~/.bashrc
 
-Commit your ulimit configuration to your shell initialization file as follows ($LIMIT being your chosen limit and $INIT_FILE being the initialization file for your shell): 
+Commit your ulimit configuration to your shell initialization file as follows ($LIMIT being your chosen limit and $INIT_FILE being the initialization file for your shell):
 
 ```
 echo ulimit -S -n $LIMIT >> $INIT_FILE

contribute/style-guides/e2e.md

@@ -5,15 +5,22 @@ Grafana Labs uses a minimal home grown solution built on top of Cypress for our
 ## Commands
 
 - `yarn e2e` Creates an isolated grafana-server home under `<repo-root>/e2e/tmp` with provisioned data sources and dashboards. This
-  copies locally build binary and frontned assets from your repo root so you need to have a built backend and frontend
+  copies locally build binary and frontend assets from your repo root so you need to have a built backend and frontend
   for this to run locally. The server starts on port 3001 so it does not conflict with your normal dev server.
 - `yarn e2e:debug` Same as above but runs the tests in chrome and does not shutdown after completion.
 - `yarn e2e:dev` Same as above but does not run any tests on startup. It lets you pick a test first.
 
+If you already have a Grafana instance running, you can provide a specific URL by setting the `BASE_URL` environment variable:
+
+```
+BASE_URL=http://172.0.10.2:3333 yarn e2e
+```
+
 The above commands use some utils scripts under `<repo-root>/e2e` that can also be used for more control.
 
 - `./e2e/start-server` This creates a fresh new grafana server working dir, setup's config and starts the server. It
   will also kill any previously started server that is still running using pid file at `<repo-root>/e2e/tmp/pid`.
+- `./e2e/wait-for-grafana` waits for `$HOST` and `$PORT` to be available. Per default localhost and 3001.
 - `./e2e/run-suite <debug|dev|noarg>` Starts cypress in different modes.
 
 ## Test Suites

e2e/run-suite

@@ -2,15 +2,12 @@
 
 . e2e/variables
 
-echo -e "Waiting for grafana-server to finish starting"
-
-timeout 60 bash -c 'until nc -z $0 $1; do sleep 1; done' localhost $PORT
-
 echo -e "Starting Cypress scenarios"
 
 CMD="start"
 PARAMS=""
 SLOWMO=0
+URL=${BASE_URL:-"http://$DEFAULT_HOST:$DEFAULT_PORT"}
 
 if [ "$1" == "debug" ]; then
     echo -e "Debug mode"
@@ -25,6 +22,6 @@ fi
 
 cd packages/grafana-e2e
 
-yarn $CMD --env BASE_URL=http://localhost:$PORT,CIRCLE_SHA1=$CIRCLE_SHA1,SLOWMO=$SLOWMO \
+yarn $CMD --env BASE_URL=$URL,CIRCLE_SHA1=$CIRCLE_SHA1,SLOWMO=$SLOWMO \
   --config integrationFolder=../../e2e/suite1/specs,screenshotsFolder=../../e2e/suite1/screenshots,videosFolder=../../e2e/suite1/videos,fileServerFolder=./cypress,viewportWidth=1920,viewportHeight=1080,trashAssetsBeforeRuns=false \
   $PARAMS

e2e/start-and-run-suite

@@ -2,7 +2,12 @@
 
 . e2e/variables
 
-# Start it in the background
-./e2e/start-server 2>&1 > e2e/server.log &
+if [ "$BASE_URL" != "" ]; then
+    echo -e "BASE_URL set, skipping starting server"
+else
+  # Start it in the background
+  ./e2e/start-server 2>&1 > e2e/server.log &
+  ./e2e/wait-for-grafana
+fi
 
 ./e2e/run-suite "$@"

e2e/start-server

@@ -3,6 +3,8 @@ set -eo pipefail
 
 . e2e/variables
 
+PORT=${PORT:-$DEFAULT_PORT}
+
 ./e2e/kill-server
 
 mkdir $RUNDIR

e2e/variables

@@ -4,4 +4,5 @@ RUNDIR=e2e/tmp
 PIDFILE=$RUNDIR/pid
 PACKAGE_FILE=dist/grafana-*linux-amd64.tar.gz
 PROV_DIR=$RUNDIR/conf/provisioning
-PORT=3001
+DEFAULT_HOST=localhost
+DEFAULT_PORT=3001

e2e/wait-for-grafana

@@ -0,0 +1,11 @@
+#!/bin/bash
+set -eo pipefail
+
+. e2e/variables
+
+HOST=${HOST:-$DEFAULT_HOST}
+PORT=${PORT:-$DEFAULT_PORT}
+
+echo -e "Waiting for grafana-server to finish starting, host=$HOST, port=$PORT"
+
+timeout 60 bash -c 'until nc -z $0 $1; do sleep 1; done' $HOST $PORT

package.json

@@ -10,7 +10,7 @@
     "build": "grunt build",
     "dev": "webpack --progress --colors --config scripts/webpack/webpack.dev.js",
     "e2e": "./e2e/start-and-run-suite",
-    "e3e:debug": "./e2e/start-and-run-suite debug",
+    "e2e:debug": "./e2e/start-and-run-suite debug",
     "e2e:dev": "./e2e/start-and-run-suite dev",
     "jest": "jest --notify --watch",
     "jest-ci": "mkdir -p reports/junit && export JEST_JUNIT_OUTPUT_DIR=reports/junit && jest --ci --reporters=default --reporters=jest-junit --maxWorkers 2",