IntenseWebs/grafana
3
0
Fork 0
mirror of https://github.com/grafana/grafana.git synced 2024-12-30 10:47:30 -06:00
Code Issues Packages Projects Releases Wiki Activity

Docs: E2E (#27461)

Browse Source 
* Separated generalized E2E docs from that of core

… and rewrote some minor things to be more concise for faster reads.

* Added plugins E2E document
This commit is contained in:
Steven Vachon 2020-09-09 08:55:42 -04:00 committed by GitHub
parent b35af9eaa7
commit 2e3b6b6850
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 130 additions and 103 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

25
contribute/style-guides/e2e-core.md Normal file
View File

@ -0,0 +1,25 @@
# End-to-End Tests for core Grafana
This document is specific to the [Grafana repository](https://github.com/grafana/grafana). Be sure that you've read the [generalized E2E document](e2e.md).
## Commands
- `yarn e2e` Creates an isolated `grafana-server` home under _\<repo-root>/e2e/tmp_ with provisioned data sources and dashboards. This copies locally build binary and frontend assets from your repo root so you need to have a built backend and frontend for this to run locally. The server starts on port 3001 so it does not conflict with your normal dev server.
- `yarn e2e:debug` Same as above but runs the tests in chrome and does not shutdown after completion.
- `yarn e2e:dev` Same as above but does not run any tests on startup. It lets you pick a test first.
If you already have a Grafana instance running, you can provide a specific URL by setting the `BASE_URL` environment variable:
```shell
BASE_URL=http://172.0.10.2:3333 yarn e2e
```
The above commands use some utils scripts under [_\<repo-root>/e2e_](../../e2e) that can also be used for more control.
- `./e2e/start-server` This creates a fresh new grafana server working dir, setup's config and starts the server. It will also kill any previously started server that is still running using pid file at _\<repo-root>/e2e/tmp/pid_.
- `./e2e/wait-for-grafana` waits for `$HOST` and `$PORT` to be available. Per default localhost and 3001.
- `./e2e/run-suite <debug|dev|noarg>` Starts cypress in different modes.
## Test suites
All the integration tests are located at _\<repo-root>/e2e/suite\<x>/specs_. The page objects and reusable flows are in the [_\<repo-root>/packages/grafana-e2e_](../../packages/grafana-e2e) package.

28
contribute/style-guides/e2e-plugins.md Normal file
View File

@ -0,0 +1,28 @@
# End-to-End Tests for plugins
Be sure that you've read the [generalized E2E document](e2e.md).
## Commands
- `yarn test:e2e` will run [Grafana's E2E utility](../../packages/grafana-e2e) against an already running Grafana server.
- `yarn test:e2e:update` will run `test:e2e` but instead of asserting that screenshots match their expected fixtures, they'll be replaced with new ones.
Your running Grafana instance can be targeted by setting the `CYPRESS_BASE_URL`, `CYPRESS_USERNAME` and `CYPRESS_PASSWORD` environment variableS:
```shell
CYPRESS_BASE_URL=https://localhost:3000 CYPRESS_USERNAME=admin CYPRESS_PASSWORD=admin yarn test:e2e
```
## Test suites
All tests are located at _\<repo-root>/cypress/integration_ by default.
## Things to test
- Add data source (if applicable)
- Add panel
- Edit panel
- Annotations (if applicable)
- Aliases (if applicable)
- Template variables
- "Explore" view

180
contribute/style-guides/e2e.md
View File

@ -1,139 +1,122 @@
# End to end test framework # End-to-End tests
Grafana Labs uses a minimal home grown solution built on top of Cypress for our end to end (e2e) tests. Grafana Labs uses a minimal [homegrown solution](../../packages/grafana-e2e) built on top of [Cypress](https://cypress.io) for its end-to-end (E2E) tests.
## Commands Important notes:
- `yarn e2e` Creates an isolated grafana-server home under `<repo-root>/e2e/tmp` with provisioned data sources and dashboards. This - We generally store all element identifiers ([CSS selectors](https://mdn.io/docs/Web/CSS/CSS_Selectors)) within the framework for reuse and maintainability.
copies locally build binary and frontend assets from your repo root so you need to have a built backend and frontend - We generally do not use stubs or mocks as to fully simulate a real user.
for this to run locally. The server starts on port 3001 so it does not conflict with your normal dev server. - Cypress' promises [do not behave as you'd expect](https://docs.cypress.io/guides/core-concepts/introduction-to-cypress.html#Mixing-Async-and-Sync-code).
- `yarn e2e:debug` Same as above but runs the tests in chrome and does not shutdown after completion. - [Testing core Grafana](e2e-core.md) is slightly different than [testing plugins](e2e-plugins.md).
- `yarn e2e:dev` Same as above but does not run any tests on startup. It lets you pick a test first.
If you already have a Grafana instance running, you can provide a specific URL by setting the `BASE_URL` environment variable: ## Framework structure
``` Inspired by https://martinfowler.com/bliki/PageObject.html
BASE_URL=http://172.0.10.2:3333 yarn e2e
```
The above commands use some utils scripts under `<repo-root>/e2e` that can also be used for more control. - `Selector`: A unique identifier that is used from the E2E framework to retrieve an element from the Browser
- `./e2e/start-server` This creates a fresh new grafana server working dir, setup's config and starts the server. It
will also kill any previously started server that is still running using pid file at `<repo-root>/e2e/tmp/pid`.
- `./e2e/wait-for-grafana` waits for `$HOST` and `$PORT` to be available. Per default localhost and 3001.
- `./e2e/run-suite <debug|dev|noarg>` Starts cypress in different modes.
## Test Suites
All the integration tests are located at `e2e/suite<x>/specs`. The page objects and reusable flows are in the
`packages/grafana-e2e` package.
## Basic concepts
Here is a good introduction to e2e best practices: https://martinfowler.com/bliki/PageObject.html.
- `Selector`: A unique identifier that is used from the e2e framework to retrieve an element from the Browser
- `Page`: An abstraction for an object that contains one or more `Selectors` with `visit` function to navigate to the page. - `Page`: An abstraction for an object that contains one or more `Selectors` with `visit` function to navigate to the page.
- `Component`: An abstraction for an object that contains one or more `Selectors` but without `visit` function - `Component`: An abstraction for an object that contains one or more `Selectors` but without `visit` function
- `Flow`: An abstraction that contains a sequence of actions on one or more `Pages` that can be reused and shared between tests - `Flow`: An abstraction that contains a sequence of actions on one or more `Pages` that can be reused and shared between tests
## Basic example ## Basic example
Let's start with a simple example with a single selector. For simplicity, all examples are in JSX. Let's start with a simple [JSX](https://reactjs.org/docs/introducing-jsx.html) example containing a single input field that we want to populate during our E2E test:
In our example app, we have an input that we want to type some text into during our e2e test. ```jsx
<input
```jsx harmony className="gf-form-input login-form-input"
<div> type="text"
<input type="text" className="gf-form-input login-form-input" /> />
</div>
``` ```
We could define a selector using `JQuery` [type selectors](https://api.jquery.com/category/selectors/) with a string like `'.gf-form-input.login-form-input'` but that would be brittle as style changes occur frequently. Furthermore there is nothing that signals to future developers that this input is part of an e2e test. We _could_ target the field with a CSS selector like `.gf-form-input.login-form-input` but that would be brittle as style changes occur frequently. Furthermore there is nothing that signals to future developers that this input is part of an E2E test. At Grafana, we use `aria-label` attributes as our preferred way of defining selectors instead of [`data-*`](https://mdn.io/docs/Web/HTML/Global_attributes/data-*) as they also aid in [accessibility](https://mdn.io/docs/Learn/Accessibility/What_is_accessibility):
At Grafana, we use `aria-label` as our preferred way of defining selectors instead of `data-*` attributes. This also aids in accessibility. ```jsx
Let's add a descriptive `aria-label` to our simple example. <input
aria-label="Username input field"
```jsx harmony className="gf-form-input login-form-input"
<div> type="text"
<input type="text" className="gf-form-input login-form-input" aria-label="Username input field" /> />
</div>
``` ```
Now that we added the `aria-label` we suddenly get more information about this particular field. It's an input field that represents a username, but there it's still not really signaling that it's part of an e2e test. The next step is to create a `Page` representation in our E2E framework to glue the test with the real implementation using the `pageFactory` function. For that function we can supply a `url` and `selectors` like in the example below:
The next step is to create a `Page` representation in our e2e test framework to glue the test with the real implementation using the `pageFactory` function. For that function we can supply a `url` and `selectors` like in the example below:
```typescript ```typescript
export const Login = { export const Login = {
url: '/login', // used when called from Login.visit() // Called via `Login.visit()`
username: 'Username input field', // used when called from Login.username().type('Hello World') url: '/login',
// Called via `Login.username()`
username: 'Username input field',
}; };
``` ```
The next step is to add the `Login` page to the exported const `Pages` in `packages/grafana-e2e-selectors/src/selectors/pages.ts` so that it appears when we type `e2e.pages` in our IDE. The next step is to add the `Login` page to the `Pages` export within [_\<repo-root>/packages/grafana-e2e-selectors/src/selectors/pages.ts_](../../packages/grafana-e2e-selectors/src/selectors/pages.ts) so that it appears when we type `e2e.pages` in our IDE.
```ecmascript 6 ```typescript
export const Pages = { export const Pages = {
Login, Login,
..., …,
..., …,
..., …,
}; };
``` ```
Now that we have a `Page` called `Login` in our `Pages` const we can use that to add a selector in our html like shown below and now this really signals to future developers that it is part of an e2e test. Now that we have a `Page` called `Login` in our `Pages` const we can use that to add a selector in our html like shown below and now this really signals to future developers that it is part of an E2E test.
```jsx harmony ```jsx
<div> import { selectors } from '@grafana/e2e-selectors';
<input type="text" className="gf-form-input login-form-input" aria-label={selectors.pages.Login.username} />
</div> <input
aria-label={selectors.pages.Login.username}
className="gf-form-input login-form-input"
type="text"
/>
``` ```
The last step in our example is to use our `Login` page as part of a test. The last step in our example is to use our `Login` page as part of a test.
- The `url` property is used whenever we call the `visit` function and is equivalent to the Cypress function [cy.visit()](https://docs.cypress.io/api/commands/visit.html#Syntax). - The `url` property is used whenever we call the `visit` function and is equivalent to the Cypress' [`cy.visit()`](https://docs.cypress.io/api/commands/visit.html#Syntax).
> Best practice after calling `visit` is to always call `should` on a selector to prevent flaky tests when you try to access an element that isn't ready. For more information, refer to [Commands vs. assertions](https://docs.cypress.io/guides/core-concepts/retry-ability.html#Commands-vs-assertions).
- Any defined selector can be accessed from the `Login` page by invoking it. This is equivalent to the result of the Cypress function [cy.get(...)](https://docs.cypress.io/api/commands/get.html#Syntax).
```ecmascript 6 - Any defined selector can be accessed from the `Login` page by invoking it. This is equivalent to the result of the Cypress function [`cy.get(…)`](https://docs.cypress.io/api/commands/get.html#Syntax).
```typescript
describe('Login test', () => { describe('Login test', () => {
it('Should pass', () => { it('passes', () => {
e2e.pages.Login.visit(); e2e.pages.Login.visit();
// To prevent flaky tests, always do a .should on any selector that you expect to be in the DOM. // To prevent flaky tests, always do a `.should` on any selector that you expect to be in the DOM.
// Read more here: https://docs.cypress.io/guides/core-concepts/retry-ability.html#Commands-vs-assertions // Read more here: https://docs.cypress.io/guides/core-concepts/retry-ability.html#Commands-vs-assertions
e2e.pages.Login.username().should('be.visible'); e2e.pages.Login.username()
e2e.pages.Login.username().type('admin'); .should('be.visible')
.type('admin');
}); });
}); });
``` ```
## Advanced example ## Advanced example
Let's take a look at an example that uses the same `selector` for multiple items in a list for instance. In this example app we have a list of data sources that we want to click on during an e2e test. Let's take a look at an example that uses the same `selector` for multiple items in a list for instance. In this example app we have a list of data sources that we want to click on during an E2E test.
```jsx harmony ```jsx
<ul> <ul>
{dataSources.map(dataSource => ( {dataSources.map(({ id, name }) => (
<li className="card-item-wrapper" key={dataSource.id}> <li className="card-item-wrapper" key={id}>
<a className="card-item" href={`datasources/edit/${dataSource.id}`}> <a className="card-item" href={`datasources/edit/${id}`}>
<div className="card-item-name">{dataSource.name}</div> <div className="card-item-name">{name}</div>
</a> </a>
</li> </li>
))} ))}
</ul> </ul>
``` ```
````
Just as before in the basic example we'll start by creating a page abstraction using the `pageFactory` function: Just as before in the basic example we'll start by creating a page abstraction using the `pageFactory` function:
```typescript ```typescript
export const DataSources = { export const DataSources = {
url: '/datasources', url: '/datasources',
dataSources: (dataSourceName: string) => `Data source list item ${dataSourceName}`, dataSources: (dataSourceName: string) => `Data source list item ${dataSourceName}`,
}; };
```` ```
You might have noticed that instead of a simple `string` as the `selector`, we're using a `function` that takes a string parameter as an argument and returns a formatted string using the argument. You might have noticed that instead of a simple `string` as the `selector`, we're using a `function` that takes a string parameter as an argument and returns a formatted string using the argument.
@ -141,13 +124,13 @@ Just as before we need to add the `DataSources` page to the exported const `Page
The next step is to use the `dataSources` selector function as in our example below: The next step is to use the `dataSources` selector function as in our example below:
```jsx harmony ```jsx
<ul> <ul>
{dataSources.map(dataSource => ( {dataSources.map(({ id, name }) => (
<li className="card-item-wrapper" key={dataSource.id}> <li className="card-item-wrapper" key={id}>
<a className="card-item" href={`datasources/edit/${dataSource.id}`}> <a className="card-item" href={`datasources/edit/${id}`}>
<div className="card-item-name" aria-label={selectors.pages.DataSources.dataSources(dataSource.name)}> <div className="card-item-name" aria-label={selectors.pages.DataSources.dataSources(name)}>
{dataSource.name} {name}
</div> </div>
</a> </a>
</li> </li>
@ -155,34 +138,25 @@ The next step is to use the `dataSources` selector function as in our example be
</ul> </ul>
``` ```
When this list is rendered with the data sources with names `A`, `B`, `C` the resulting html would become: When this list is rendered with the data sources with names `A`, `B` and `C` ,the resulting HTML would look like:
```jsx harmony ```html
<div class="card-item-name" aria-label="Data source list item A"> <div class="card-item-name" aria-label="Data source list item A">A</div>
A <div class="card-item-name" aria-label="Data source list item B">B</div>
</div> <div class="card-item-name" aria-label="Data source list item C">C</div>
...
<div class="card-item-name" aria-label="Data source list item B">
B
</div>
...
<div class="card-item-name" aria-label="Data source list item C">
C
</div>
``` ```
Now we can write our test. The one thing that differs from the `Basic example` is that we pass in which data source we want to click on as an argument to the selector function: Now we can write our test. The one thing that differs from the [basic example](#basic-example) above is that we pass in which data source we want to click on as an argument to the selector function:
> Best practice after calling `visit` is to always call `should` on a selector to prevent flaky tests when you try to access an element that isn't ready. For more information, refer to [Commands vs. assertions](https://docs.cypress.io/guides/core-concepts/retry-ability.html#Commands-vs-assertions). ```typescript
```ecmascript 6
describe('List test', () => { describe('List test', () => {
it('Clicking on data source named B', () => { it('clicks on data source named B', () => {
e2e.pages.DataSources.visit(); e2e.pages.DataSources.visit();
// To prevent flaky tests, always do a .should on any selector that you expect to be in the DOM. // To prevent flaky tests, always do a .should on any selector that you expect to be in the DOM.
// Read more here: https://docs.cypress.io/guides/core-concepts/retry-ability.html#Commands-vs-assertions // Read more here: https://docs.cypress.io/guides/core-concepts/retry-ability.html#Commands-vs-assertions
e2e.pages.DataSources.dataSources('B').should('be.visible'); e2e.pages.DataSources.dataSources('B')
e2e.pages.DataSources.dataSources('B').click(); .should('be.visible')
.click();
}); });
}); });
``` ```