The shell command line options and parameters used there are bash-
specific. This results in an error on attempting of running
'ipa-run-tests' on systems where '/bin/sh' is pointing to another
shell, for example, dash on Ubuntu.
Fixes: https://pagure.io/freeipa/issue/8101
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Sergey Orlov <sorlov@redhat.com>
Pytest 5.2+ warns if tests XML report is generated but its format (schema)
is not explicitly specified:
```
/root/.local/lib/python3/site-packages/_pytest/junitxml.py:417
/root/.local/lib/python3/site-packages/_pytest/junitxml.py:417: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0.
Add 'junit_family=xunit1' to your pytest.ini file to keep the current format in future versions of pytest and silence this warning.
_issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)
```
For example, xunit2 is used by jenkins and Pytest strictly conforms its
schema [0]. Pytest's xunit1, in turn, allows to attach user fields to
report.
The only known client of IPA tests results is Azure. Azure supports
[1] JUnit, which is likely the same as Pytest's xunit1, while Azure's
xUnit2 is actually xUnit.net v2. This means that Azure supports (in
one form or another) Pytest's both xunit1 and xunit2 as JUnit.
[0]: https://github.com/jenkinsci/xunit-plugin/blob/xunit-2.3.2/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd
[1]: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8101
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Sergey Orlov <sorlov@redhat.com>
Currently, such arguments are eaten by 'ipa-run-tests' script as they
are not quoted.
For example, running ipa-run-tests -k ''
results in the actual invocation would be like as:
['/bin/sh',
'--norc',
'--noprofile',
'-c',
'--',
"/usr/bin/python3 -c 'import sys,pytest;sys.exit(pytest.main())' -o "
'cache_dir=/tmp/pytest-of-root/pytest-12/test_ipa_run_tests_empty_expression0/.pytest_cache '
'--confcutdir=/usr/lib64/python3/site-packages/ipatests -k ']
Note: expressions or marks could be empty as a result of the building
of command line args by more high-level tools, scripts, etc.
So, a short-termed solution is the quotting of zero-length arguments.
Fixes: https://pagure.io/freeipa/issue/8173
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
This is a sort of rollback to the pre #93c158b05 state with
several improvements.
For now, the nodeids calculation by ipa-run-tests is not stable,
since it depends on current working directory. Nodeids (tests
addresses) are utilized by the other plugins, for example.
Unfortunately, the `pytest_load_initial_conftests` hook doesn't
correctly work with pytest internal paths, since it is called
after the calculation of rootdir was performed, for example.
Eventually, it's simpler to follow the default convention for
Python test discovery.
There is at least one drawback of new "old" implementation.
The ignore rules don't support globs, because pytest 4.3.0+
has the same facility via `--ignore-glob`:
> Add the `--ignore-glob` parameter to exclude test-modules with
> Unix shell-style wildcards. Add the collect_ignore_glob for
> conftest.py to exclude test-modules with Unix shell-style
> wildcards.
Upon switching to pytest4 it will be possible to utilize this.
Anyway, tests for checking current basic facilities of
ipa-run-tests were added.
Fixes: https://pagure.io/freeipa/issue/8007
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
ipa-run-tests expands arguments passed with their full paths. However,
it doesn't support expanding globs, so targets like 'test_ipa*' cannot
be specified.
Expand the code that replaces '--ignore foo' and 'foo' positional
arguments with support for '--ignore foo*' and 'foo*'.
This allows to reduce a number of additional steps in the CI pipeline
preparation.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
pytest removed copy() method from its Namespace class. Use the copy
module to make a copy of early options.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Stanislav Laznicka <slaznick@redhat.com>
The Python 3 refactoring effort is finishing, it should be safe
to turn all scripts to run in Python 3 by default.
https://pagure.io/freeipa/issue/4985
Reviewed-By: Christian Heimes <cheimes@redhat.com>
ipa-run-tests now applies the same logic to --ignore then to included
paths.
https://pagure.io/freeipa/issue/7355
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Michal Reznik <mreznik@redhat.com>
The ipa-run-tests command used os.chdir() to change into the ipatests/
directory. The approach works for simple cases but breaks some pytest
features. For example it makes it impossible to selects tests by their
fully qualified test name.
Further more, coverage statistics break because path and module names
get messed up by chdir.
A name plugin takes care of adjusting paths relative to ipatests and to
add ipatests as base. It's now possible to run tests with qualified test
names, e.g.
ipa-run-tests ipatests/test_ipalib/test_base.py::test_ReadOnly::test_lock
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Felipe Volpone <fbarreto@redhat.com>
Closes: https://fedorahosted.org/freeipa/ticket/6631
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Let's replace some ugly hacks with proper pytest conftest.py hooks.
Test initialization of ipalib.api is now handled in
pytest_cmdline_main(). Pytest plugins, markers and ignores are also
moved into conftest.py. Additional guards make it possible to run tests
without ipaserver installed.
I added confcutdir to ensure that pytest does not leave our project
space. Pytest used pytest.ini or setup.py before but pytest.ini is gone.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Milan Kubik <mkubik@redhat.com>
Logs from Beaker jobs are normally very brief, with the standard
output/error containing detailed information. Make ipa-run-tests
with BeakerLib plugin follow this convention.
Only include INFO and higher level messages in the Beaker logs.
Downgrade several message levels to DEBUG.
Log to console using Python logging instead of showing the Beaker logs.
Since ipa-run-tests sets up its own logging, Nose's own log
handling just causes duplicate messages. Disable it with --nologcapture.
Output from IPA's log manager is not captured by Nose's logcapture plugin.
Forward IPA logs to a regular Python logger so that they are shown
on failures.
IPA log messages are also shown on standard error.
Filter out Paramiko logs by default; these are too verbose.
Part of the work for: https://fedorahosted.org/freeipa/ticket/3621
Tests in test classes decorated by @ipatests.order_plugin.ordered
are sorted by the source line number instead of alphabetically,
if the plugin is enabled.
The ipa-run-tests helper now loads and enables the plugin.
This should make writing integration tests easier.
The plugin hooks into the Nose runner and IPA's logging infrastructure
and calls the appropriate BeakerLib functions (rl*).
IPA's log_manager is extended to accept custom Handler classes.
The ipa-run-tests helper now loads the plugin.
Patr of the work for: https://fedorahosted.org/freeipa/ticket/3621
