Azure Pipelines provide counters for running test jobs, these split into
System.StageAttempt and System.PhaseAttempt. Use them to make test
artifacts unique.
For XML test results we don't need to name them differently as they
aren't uploaded as artifacts but rather presented in a separate test
pane.
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Reviewed-By: Stanislav Levin <slev@altlinux.org>
Fedora nodejs builds were fixed, we don't need to limit ourselves
anymore.
Also, make sure python3-pyyaml is installed because pylint in Fedora 31
detects its use in contribs/
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Reviewed-By: Stanislav Levin <slev@altlinux.org>
The unit tests execution time within Azure Pipelines(AP) is not
balanced. One test job(Base) takes ~13min, while another(XMLRPC)
~28min. Fortunately, AP supports slicing:
> An agent job can be used to run a suite of tests in parallel. For
example, you can run a large suite of 1000 tests on a single agent.
Or, you can use two agents and run 500 tests on each one in parallel.
To leverage slicing, the tasks in the job should be smart enough to
understand the slice they belong to.
>The step that runs the tests in a job needs to know which test slice
should be run. The variables System.JobPositionInPhase and
System.TotalJobsInPhase can be used for this purpose.
Thus, to support this pytest should know how to split the test suite
into groups(slices). For this, a new internal pytest plugin was added.
About plugin.
- Tests within a slice are grouped by test modules because not all of
the tests within the module are independent from each other.
- Slices are balanced by the number of tests within test module.
- To run some module within its own environment there is a dedicated
slice option (could help with extremely slow tests)
Examples.
- To split `test_cmdline` tests into 2 slices and run the first one:
ipa-run-tests --slices=2 --slice-num=1 test_cmdline
- To split tests into 2 slices, then to move one module out to its own slice
and run the second one:
ipa-run-tests --slices=2 --slice-dedicated=test_cmdline/test_cli.py \
--slice-num=2 test_cmdline
Fixes: https://pagure.io/freeipa/issue/8008
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Display which dnf repositories were available at the
prepare-build step via metalink.
Also display the fastestmirror cache.
Signed-off-by: François Cami <fcami@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Sometimes the prepare-build step of azure pipelines fails
with download errors:
"configure: error: Package requirements (nspr) were not met:"
This can be due to fastestmirror not being used to check
mirror availability and sometimes speed. Combined with a
too-low default number of retries, and a high timeout this
can lead to download failures that could be avoided.
Activate fastestmirror, add more download workers, and tune
timeout/retries to make dnf more reliable.
Fixes: https://pagure.io/freeipa/issue/7999
Signed-off-by: François Cami <fcami@redhat.com>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
By default, the `last` exit code returned from Azure script will be
checked and, if non-zero, treated as a step failure. Luckily,
for Linux script is a shortcut for Bash. Hence errexit/e option
could be applied. But Azure pipelines doesn't set it by default:
https://github.com/microsoft/azure-pipelines-agent/issues/1803
For multiline script this is a problem, unless otherwise designed.
Some of benefits of checking the result of each subcommand:
- preventing subsequent issues (broken packages, container images, etc.)
- time saving (next steps will not run)
- good diagnostics (tells which part of script fails)
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Azure tests fail because we couldn't configure docker for IPv6 anymore.
This happened because we weren't able to copy our configuration file to
/etc/docker -- looks like the docker directory does not exist.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Rewrite templates to make test job declarations simpler and easier to
work with.
A test job template can be instantiated this way:
- template: templates/test-jobs.yml
parameters:
jobName: Base
jobTitle: Base tests
testsToRun:
- test_cmdline
- test_install
- test_ipaclient
- test_ipalib
- test_ipaplatform
- test_ipapython
testsToIgnore:
- test_integration
- test_webui
- test_ipapython/test_keyring.py
taskToRun: run-tests
Both 'testsToRun' and 'testsToIgnore' accept arrays of test matches.
Wildcards also supported:
....
testsToRun:
- test_xmlrpc/test_hbac*
....
'taskToRun' specifies a script ipatests/azure/azure-${taskToRun}.sh that
will be executed in the test environment to actually start tests.
Parameters 'testsToRun' and 'testsToIgnore' define TESTS_TO_{RUN,IGNORE}
variables that will be set in the environment of the test script. These
variables will have entries from the parameters separated by a space.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Sets up a pipeline to run FreeIPA build and tests in Azure Pipelines.
Azure Pipelines provides 10 parallel free runners for open source projects.
Use them to run following jobs:
- Build: build RPMs and Fedora 30 container with them
- Lint: run linting of the source code
- Tox: run py36,pypi,pylint tests using Tox
- Web UI unit tests: run Web UI unit tests with Grunt/QUnit/PhantomJS
- XMLRPC tests: install FreeIPA server and run XMLRPC tests against it
All jobs are running in Fedora 30 containers. Build, Lint, Tox, and Web
UI unit tests run inside f30/fedora-toolbox container. Build job
generates a container with pre-installed FreeIPA packages using official
fedora:30 container. All containers are picked up from
registry.fedoraproject.org.
Artifacts from the build job are pushed to a pipeline storage and reused
in the XMLRPC tests. They also are accessible in the 'Summary' tab to
download.
XUnit and QUnit outputs from the tests that produce it are reported in
the 'Tests' tab.
Logs from individual steps from each job are available for review in
the 'Logs' tab. They also can be downloaded.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Alexander Bokovoy
Stanislav Levin
François Cami
Alexander Bokovoy