'get_marker' was a compat shim for Pytest < 3.6.
Since the requred Pytest is 3.9.1+, the workaround can be
removed.
Fixes: https://pagure.io/freeipa/issue/8101
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Sergey Orlov <sorlov@redhat.com>
Azure provides Microsoft-hosted agents having tasty resources [0].
For now (Feb 2020),
- (Linux only) Run steps in a cgroup that offers 6 GB of physical memory and
13 GB of total memory
- Provide at least 10 GB of storage for your source and build outputs.
This is enough to set up IPA environments consisted of not only master but also
replicas and clients and thus, run IPA integration tests.
New Azure IPA tests workflow:
+ 1) Azure generate jobs using Matrix strategy
2) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ c) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ d) launch IPA tests on tests' controller
e) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
f) publish regular system logs as artifacts
[0] https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops
[1] https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
This Pytest plugin is intended to issue warnings on collecting
tests, which employ unittest/nose frameworks or xunit style.
For example, this may look like:
"""
test_a/test_xunit.py:25
test_a/test_xunit.py:25: PytestDeprecationWarning: xunit style is deprecated
def test_foo_bar(self):
test_b/test_unittest.py:7
test_b/test_unittest.py:7: PytestDeprecationWarning: unittest is deprecated
def test_foo_bar(self):
"""
To treat these warnings as errors it's enough to run Pytest with:
-W error:'xunit style is deprecated':pytest.PytestDeprecationWarning
Related: https://pagure.io/freeipa/issue/7989
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
The unit tests execution time within Azure Pipelines(AP) is not
balanced. One test job(Base) takes ~13min, while another(XMLRPC)
~28min. Fortunately, AP supports slicing:
> An agent job can be used to run a suite of tests in parallel. For
example, you can run a large suite of 1000 tests on a single agent.
Or, you can use two agents and run 500 tests on each one in parallel.
To leverage slicing, the tasks in the job should be smart enough to
understand the slice they belong to.
>The step that runs the tests in a job needs to know which test slice
should be run. The variables System.JobPositionInPhase and
System.TotalJobsInPhase can be used for this purpose.
Thus, to support this pytest should know how to split the test suite
into groups(slices). For this, a new internal pytest plugin was added.
About plugin.
- Tests within a slice are grouped by test modules because not all of
the tests within the module are independent from each other.
- Slices are balanced by the number of tests within test module.
- To run some module within its own environment there is a dedicated
slice option (could help with extremely slow tests)
Examples.
- To split `test_cmdline` tests into 2 slices and run the first one:
ipa-run-tests --slices=2 --slice-num=1 test_cmdline
- To split tests into 2 slices, then to move one module out to its own slice
and run the second one:
ipa-run-tests --slices=2 --slice-dedicated=test_cmdline/test_cli.py \
--slice-num=2 test_cmdline
Fixes: https://pagure.io/freeipa/issue/8008
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
This is a sort of rollback to the pre #93c158b05 state with
several improvements.
For now, the nodeids calculation by ipa-run-tests is not stable,
since it depends on current working directory. Nodeids (tests
addresses) are utilized by the other plugins, for example.
Unfortunately, the `pytest_load_initial_conftests` hook doesn't
correctly work with pytest internal paths, since it is called
after the calculation of rootdir was performed, for example.
Eventually, it's simpler to follow the default convention for
Python test discovery.
There is at least one drawback of new "old" implementation.
The ignore rules don't support globs, because pytest 4.3.0+
has the same facility via `--ignore-glob`:
> Add the `--ignore-glob` parameter to exclude test-modules with
> Unix shell-style wildcards. Add the collect_ignore_glob for
> conftest.py to exclude test-modules with Unix shell-style
> wildcards.
Upon switching to pytest4 it will be possible to utilize this.
Anyway, tests for checking current basic facilities of
ipa-run-tests were added.
Fixes: https://pagure.io/freeipa/issue/8007
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
config_replace_variables() can now also remove lines from a file.
Related: https://pagure.io/freeipa/issue/7860
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
pytest 3.6 has deprecated get_marker in 3.6. The method was removed in 4.x
and replaced with get_closest_marker.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Thomas Woerner <twoerner@redhat.com>
conftest uses the Function attribute of a pytest.Function object. Latest
pytest has deprecated the attribute:
_pytest.warning_types.RemovedInPytest4Warning: usage of Function.Function
is deprecated, please use pytest.Function instead
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Serhii Tsymbaliuk <stsymbal@redhat.com>
pytest 3.7.0 doesn't like ipatests.pytest_plugins package. The string
"pytest_plugins" is used as marker to load plugins. By populare vote and
to avoid future conflicts, we decided to rename the directory to pytest_ipa.
Fixes: https://pagure.io/freeipa/issue/7663
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Some fast tests depend on an api.env with realm, domain, and host. On
machines without ~/.ipa/default.conf, the settings are not available.
Provide dummy values to make tests pass.
Closes: https://pagure.io/freeipa/issue/7432
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Reviewed-By: Aleksei Slaikovskii <aslaikov@redhat.com>
The new marker needs_ipaapi is used to mark tests that needs an
initialized API (ipalib.api) or some sort of other API services (running
LDAP server) to work. Some packages use api.Command or api.Backend on
module level. They are not marked but rather skipped entirely.
A new option ``skip-ipaapi`` is added to skip all API based tests. With
the option, only simple unit tests are executed. As of now, freeIPA
contains more than 500 unit tests that can be executed in about 5
seconds.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Pytest can be executed from a machine that doesn't have IPA configured.
In this case, api can't be finalized because values such as basedn are
unknown and missing.
Fixes https://pagure.io/freeipa/issue/7046
Signed-off-by: Tomas Krizek <tkrizek@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Add tox infrastructure to test client wheel packages workflow:
* build client packages
* install client packages
* ipa-run-tests --ipaclient-unittests under Python 2 and 3
* pylint of client packages under Python 2 and 3
* placeholder packages work as expected
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Print api.env, uname, euid/egid, cwd and Python version when tests are
run with -v (e.g. ipa-run-tests -v).
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
1e06a5195b removed pytest.ini. Without the
ini file, pytest 3.x has suboptimal settings and no longer picks up all
test functions and test files.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Milan Kubik <mkubik@redhat.com>
Let's replace some ugly hacks with proper pytest conftest.py hooks.
Test initialization of ipalib.api is now handled in
pytest_cmdline_main(). Pytest plugins, markers and ignores are also
moved into conftest.py. Additional guards make it possible to run tests
without ipaserver installed.
I added confcutdir to ensure that pytest does not leave our project
space. Pytest used pytest.ini or setup.py before but pytest.ini is gone.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Milan Kubik <mkubik@redhat.com>
