E2E: Add new run modes for e2e tests (#22795)

* WIP: started e2e tests for bar gauge

* Updated

* Updated

* Updated cypress from 3.7 -> 4.1

* reverted cypress upgrade

* Updated test

* Update e2e docs

* Updated docs

.circleci/config.yml

@@ -524,11 +524,8 @@ jobs:
           command: ./e2e/run-suite
           no_output_timeout: 5m
       - store_artifacts:
-          path: e2e/suite1/screenshots/expected
-          destination: expected-screenshots
-      - store_artifacts:
-          path: e2e/suite1/screenshots/received
-          destination: received-screenshots
+          path: e2e/suite1/screenshots
+          destination: screenshots
       - store_artifacts:
           path: e2e/suite1/videos
           destination: output-videos

.gitignore

@@ -105,5 +105,6 @@ compilation-stats.json
 /packages/grafana-e2e/cypress/videos
 /packages/grafana-e2e/cypress/logs
 /e2e/server.log
-/e2e/suite1/screenshots/received
+/e2e/suite1/screenshots
+!/e2e/suite1/screenshots/expeced/*
 /e2e/suite1/videos/*

contribute/style-guides/e2e.md

@@ -1,19 +1,43 @@
 # End to end test framework
+
 Grafana Labs uses a minimal home grown solution built on top of Cypress for our end to end (e2e) tests.
 
+## Commands
+
+- `yarn e2e` Creates an isolated grafana-server home under `<repo-root>/e2e/tmp` with provisioned data sources and dashboards. This
+  copies locally build binary and frontned assets from your repo root so you need to have a built backend and frontend
+  for this to run locally. The server starts on port 3001 so it does not conflict with your normal dev server.
+- `yarn e2e:debug` Same as above but runs the tests in chrome and does not shutdown after completion.
+- `yarn e2e:dev` Same as above but does not run any tests on startup. It lets you pick a test first.
+
+The above commands use some utils scripts under `<repo-root>/e2e` that can also be used for more control.
+
+- `./e2e/start-server` This creates a fresh new grafana server working dir, setup's config and starts the server. It
+  will also kill any previously started server that is still running using pid file at `<repo-root>/e2e/tmp/pid`.
+- `./e2e/run-suite <debug|dev|noarg>` Starts cypress in different modes.
+
+## Test Suites
+
+All the integration tests are located at `e2e/suite<x>/specs`. The page objects and reusable flows are in the
+`packages/grafana-e2e` package.
+
 ## Basic concepts
+
 Here is a good introduction to e2e best practices: https://martinfowler.com/bliki/PageObject.html.
+
 - `Selector`: A unique identifier that is used from the e2e framework to retrieve an element from the Browser
 - `Page`: An abstraction for an object that contains one or more `Selectors`
 - `Flow`: An abstraction that contains a sequence of actions on one or more `Pages` that can be reused and shared between tests
 
 ## Basic example
+
 Let's start with a simple example with a single selector. For simplicity, all examples are in JSX.
 
 In our example app, we have an input that we want to type some text into during our e2e test.
+
 ```jsx harmony
 <div>
-    <input type="text" className="gf-form-input login-form-input"/>
+  <input type="text" className="gf-form-input login-form-input" />
 </div>
 ```
 
@@ -21,25 +45,28 @@ We could define a selector using `JQuery` [type selectors](https://api.jquery.co
 
 At Grafana, we use `aria-label` as our preferred way of defining selectors instead of `data-*` attributes. This also aids in accessibility.
 Let's add a descriptive `aria-label` to our simple example.
+
 ```jsx harmony
 <div>
-    <input type="text" className="gf-form-input login-form-input" aria-label="Username input field"/>
+  <input type="text" className="gf-form-input login-form-input" aria-label="Username input field" />
 </div>
 ```
 
 Now that we added the `aria-label` we suddenly get more information about this particular field. It's an input field that represents a username, but there it's still not really signaling that it's part of an e2e test.
 
 The next step is to create a `Page` representation in our e2e test framework to glue the test with the real implementation using the `pageFactory` function. For that function we can supply a `url` and `selectors` like in the example below:
+
 ```typescript
 export const Login = pageFactory({
-  url: '/login', // used when called from Login.visit()
+  url: "/login", // used when called from Login.visit()
   selectors: {
-    username: 'Username input field', // used when called from Login.username().type('Hello World')
+    username: "Username input field", // used when called from Login.username().type('Hello World')
   },
 });
 ```
 
 The next step is to add the `Login` page to the exported const `Pages` in `packages/grafana-e2e/src/pages/index.ts` so that it appears when we type `e2e.pages` in our IDE.
+
 ```ecmascript 6
 export const Pages = {
   Login,
@@ -49,17 +76,21 @@ export const Pages = {
 };
 
 ```
+
 Now that we have a `Page` called `Login` in our `Pages` const we can use that to add a selector in our html like shown below and now this really signals to future developers that it is part of an e2e test.
+
 ```jsx harmony
 <div>
-    <input type="text" className="gf-form-input login-form-input" aria-label={e2e.pages.Login.selectors.username}/>
+  <input type="text" className="gf-form-input login-form-input" aria-label={e2e.pages.Login.selectors.username} />
 </div>
 ```
 
 The last step in our example is to use our `Login` page as part of a test. The `pageFactory` function we used before gives us two things:
+
 - The `url` property is used whenever we call the `visit` function and is equivalent to the Cypress function [cy.visit()](https://docs.cypress.io/api/commands/visit.html#Syntax).
-> Best practice after calling `visit` is to always call `should` on a selector to prevent flaky tests when you try to access an element that isn't ready. For more information, refer to [Commands vs. assertions](https://docs.cypress.io/guides/core-concepts/retry-ability.html#Commands-vs-assertions).
+  > Best practice after calling `visit` is to always call `should` on a selector to prevent flaky tests when you try to access an element that isn't ready. For more information, refer to [Commands vs. assertions](https://docs.cypress.io/guides/core-concepts/retry-ability.html#Commands-vs-assertions).
 - Any defined selector in the `selectors` property can be accessed from the `Login` page by invoking it. This is equivalent to the result of the Cypress function [cy.get(...)](https://docs.cypress.io/api/commands/get.html#Syntax).
+
 ```ecmascript 6
 describe('Login test', () => {
   it('Should pass', () => {
@@ -73,6 +104,7 @@ describe('Login test', () => {
 ```
 
 ## Advanced example
+
 Let's take a look at an example that uses the same `selector` for multiple items in a list for instance. In this example app we have a list of data sources that we want to click on during an e2e test.
 
 ```jsx harmony
@@ -80,15 +112,14 @@ Let's take a look at an example that uses the same `selector` for multiple items
   {dataSources.map(dataSource => (
     <li className="card-item-wrapper" key={dataSource.id}>
       <a className="card-item" href={`datasources/edit/${dataSource.id}`}>
-        <div className="card-item-name">
-          {dataSource.name}
-        </div>
+        <div className="card-item-name">{dataSource.name}</div>
       </a>
     </li>
   ))}
 </ul>
 ```
-```
+
+````
 
 Just as before in the basic example we'll start by creating a page abstraction using the `pageFactory` function:
 ```typescript
@@ -98,12 +129,14 @@ export const DataSources = pageFactory({
     dataSources: (dataSourceName: string) => `Data source list item ${dataSourceName}`,
   },
 });
-```
+````
+
 You might have noticed that instead of a simple `string` as the `selector`, we're using a `function` that takes a string parameter as an argument and returns a formatted string using the argument.
 
 Just as before we need to add the `DataSources` page to the exported const `Pages` in `packages/grafana-e2e/src/pages/index.ts`.
 
 The next step is to use the `dataSources` selector function as in our example below:
+
 ```jsx harmony
 <ul>
   {dataSources.map(dataSource => (
@@ -119,6 +152,7 @@ The next step is to use the `dataSources` selector function as in our example be
 ```
 
 When this list is rendered with the data sources with names `A`, `B`, `C` the resulting html would become:
+
 ```jsx harmony
 <div class="card-item-name" aria-label="Data source list item A">
  A
@@ -134,7 +168,9 @@ When this list is rendered with the data sources with names `A`, `B`, `C` the re
 ```
 
 Now we can write our test. The one thing that differs from the `Basic example` is that we pass in which data source we want to click on as an argument to the selector function:
+
 > Best practice after calling `visit` is to always call `should` on a selector to prevent flaky tests when you try to access an element that isn't ready. For more information, refer to [Commands vs. assertions](https://docs.cypress.io/guides/core-concepts/retry-ability.html#Commands-vs-assertions).
+
 ```ecmascript 6
 describe('List test', () => {
   it('Clicking on data source named B', () => {
@@ -147,17 +183,18 @@ describe('List test', () => {
 });
 ```
 
-
 ## Debugging PhantomJS image rendering
 
 ### Common Error
+
 The most common error with PhantomJs image rendering is when a PR introduces an import that has functionality that's not supported by PhantomJs. To quickly identify which new import causes this you can use a tool like `es-check`.
-   
+
 1. Run > `npx es-check es5 './public/build/*.js'`
 2. Check the output for files that break es5 compatibility.
-3. Lazy load the failing imports if possible. 
+3. Lazy load the failing imports if possible.
 
 ### Debugging
+
 There is no easy or comprehensive way to debug PhantomJS smoke test (image rendering) failures. However, PhantomJS exposes remote debugging interface which can give you a sense of what is going wrong in the smoke test. Before performing the steps described below make sure your local Grafana instance is running:
 
 1. Go to `tools/phantomjs` directory

e2e/run-suite

@@ -8,8 +8,23 @@ timeout 60 bash -c 'until nc -z $0 $1; do sleep 1; done' localhost $PORT
 
 echo -e "Starting Cypress scenarios"
 
+CMD="start"
+PARAMS=""
+SLOWMO=0
+
+if [ "$1" == "debug" ]; then
+    echo -e "Debug mode"
+    SLOWMO=1
+    PARAMS="--headed --no-exit"
+fi
+
+if [ "$1" == "dev" ]; then
+    echo "Dev mode"
+    CMD="open"
+fi
+
 cd packages/grafana-e2e
 
-yarn start --env BASE_URL=http://localhost:$PORT,CIRCLE_SHA1=$CIRCLE_SHA1,SLOWMO=$SLOWMO \
+yarn $CMD --env BASE_URL=http://localhost:$PORT,CIRCLE_SHA1=$CIRCLE_SHA1,SLOWMO=$SLOWMO \
   --config integrationFolder=../../e2e/suite1/specs,screenshotsFolder=../../e2e/suite1/screenshots,videosFolder=../../e2e/suite1/videos,fileServerFolder=./cypress,viewportWidth=1920,viewportHeight=1080,trashAssetsBeforeRuns=false \
-  "$@"
+  $PARAMS

e2e/suite1/specs/bar-gauge.spec.ts

@@ -0,0 +1,18 @@
+import { e2e } from '@grafana/e2e';
+
+e2e.scenario({
+  describeName: 'Bar Gauge Panel',
+  itName: 'Bar Guage rendering e2e tests',
+  addScenarioDataSource: false,
+  addScenarioDashBoard: false,
+  skipScenario: false,
+  scenario: () => {
+    // open Panel Tests - Bar Gauge
+    e2e.flows.openDashboard('O6f11TZWk');
+
+    e2e()
+      .get('#panel-6 .bar-gauge__value')
+      .should('have.css', 'color', 'rgb(242, 73, 92)')
+      .contains('100');
+  },
+});

jest.config.e2e.js

@@ -1,15 +0,0 @@
-require('module-alias/register');
-
-module.exports = {
-  verbose: false,
-  transform: {
-    '^.+\\.(ts|tsx)$': 'ts-jest',
-  },
-  moduleDirectories: ['node_modules', 'public'],
-  roots: ['<rootDir>/public/e2e-test'],
-  testRegex: '(\\.|/)(test)\\.(jsx?|tsx?)$',
-  moduleFileExtensions: ['ts', 'tsx', 'js', 'jsx', 'json'],
-  setupFiles: [],
-  globals: { 'ts-jest': { isolatedModules: true } },
-  setupFilesAfterEnv: ['expect-puppeteer', '<rootDir>/packages/grafana-toolkit/src/e2e/install.ts'],
-};

package.json

@@ -155,7 +155,8 @@
     "build": "grunt build",
     "dev": "webpack --progress --colors --config scripts/webpack/webpack.dev.js",
     "e2e": "./e2e/start-and-run-suite",
-    "e2e:debug": "SLOWMO=1 && ./e2e/start-and-run-suite --headed --no-exit",
+    "e2e:debug": "./e2e/start-and-run-suite debug",
+    "e2e:dev": "./e2e/start-and-run-suite dev",
     "jest": "jest --notify --watch",
     "jest-ci": "mkdir -p reports/junit && export JEST_JUNIT_OUTPUT_DIR=reports/junit && jest --ci --reporters=default --reporters=jest-junit --maxWorkers 2",
     "lint": "eslint public/app e2e/suite1 public/test --ext=.js,.ts,.tsx",

packages/grafana-e2e/cypress/fixtures/example.json

@@ -0,0 +1,5 @@
+{
+  "name": "Using fixtures to represent data",
+  "email": "hello@cypress.io",
+  "body": "Fixtures are a great way to mock data for responses to routes"
+}

yarn.lock

@@ -18135,7 +18135,13 @@ openurl@1.1.1:
   resolved "https://registry.yarnpkg.com/openurl/-/openurl-1.1.1.tgz#3875b4b0ef7a52c156f0db41d4609dbb0f94b387"
   integrity sha1-OHW0sO96UsFW8NtB1GCduw+Us4c=
 
+<<<<<<< HEAD
+opn@^5.1.0:
+||||||| merged common ancestors
+opn@^5.1.0, opn@^5.3.0:
+=======
 opn@^5.5.0:
+>>>>>>> origin/master
   version "5.5.0"
   resolved "https://registry.yarnpkg.com/opn/-/opn-5.5.0.tgz#fc7164fab56d235904c51c3b27da6758ca3b9bfc"
   integrity sha512-PqHpggC9bLV0VeWcdKhkpxY+3JTzetLSqTCWL/z/tFIbI6G8JCjondXklT1JinczLz2Xib62sSp0T/gKT4KksA==