This is a sort of rollback to the pre #93c158b05 state with
several improvements.
For now, the nodeids calculation by ipa-run-tests is not stable,
since it depends on current working directory. Nodeids (tests
addresses) are utilized by the other plugins, for example.
Unfortunately, the `pytest_load_initial_conftests` hook doesn't
correctly work with pytest internal paths, since it is called
after the calculation of rootdir was performed, for example.
Eventually, it's simpler to follow the default convention for
Python test discovery.
There is at least one drawback of new "old" implementation.
The ignore rules don't support globs, because pytest 4.3.0+
has the same facility via `--ignore-glob`:
> Add the `--ignore-glob` parameter to exclude test-modules with
> Unix shell-style wildcards. Add the collect_ignore_glob for
> conftest.py to exclude test-modules with Unix shell-style
> wildcards.
Upon switching to pytest4 it will be possible to utilize this.
Anyway, tests for checking current basic facilities of
ipa-run-tests were added.
Fixes: https://pagure.io/freeipa/issue/8007
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Previously fastlint would only process .py files.
Detect which .in files are in fact Python files and add
them to the list of files to process during the fastlint
Makefile target.
Original change suggested by Alexander Bokovoy.
Fixes: https://pagure.io/freeipa/issue/7984
Signed-off-by: François Cami <fcami@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
The make target `pylint` hasn't a full list of its dependencies.
This leads to problems like:
- different build results
- PR tests just do not run pylint over some Python scripts.
The new build target (`python_scripts`) was implemented.
It's intended for building all Python scripts (files, containing
@PYTHONSHEBANG@ as a shebang placeholder).
The make `pylint` should require it.
Fixes: https://pagure.io/freeipa/issue/7921
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Build all scripts in install/tools/ to check them with pylint, so that
``make pylint`` always checks all scripts. The script files are
generated by make.
Please note that fastlint does not check script files.
See: https://pagure.io/freeipa/issue/7772
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Thomas Woerner <twoerner@redhat.com>
Add "make cov-scan" to automate Coverity scan builds. cov-build requires
extra quirks to work with recent versions of GCC on Fedora.
The make target requires a token and Coverity's build chain. Both are
available for privileged project owners on
https://scan.coverity.com/projects/freeipa-freeipa .
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Let's catch broken YAML files (Travis, PR-CI) and spec file early.
- Use rpmlint to detect syntax errors in spec file early
- Attempt to parse all YAML files with PyYAML
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Florence Blanc-Renaud <frenaud@redhat.com>
Remove Python 2 related code and configuration from spec file, autoconf
and CI infrastructure. From now on, FreeIPA 4.8 requires at least Python
3.6. Python 2 packages like python2-ipaserver or python2-ipaclient are
no longer available. PR-CI, lint, and tox aren't testing Python 2
compatibility either.
See: https://fedoraproject.org/wiki/Changes/FreeIPA_Python_2_Removal
Fixes: https://pagure.io/freeipa/issue/7568
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Reviewed-By: Tibor Dudlak <tdudlak@redhat.com>
Python scripts are now generated from templates. The scripts are marked
as nodist (no distribution) but install targets. The templates for the
scripts are extra distribution data, no installation (noinst).
Fixes: https://pagure.io/freeipa/issue/7680
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Move logic for installing just the Python packages out of the spec file
and into our root Makefile. It removes code duplication to simplify a
spec file that supports building without Python 2.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
when any file from FreeIPA tree has been deleted there was
a failure like:
pylint
------
************* Module ipaserver/install/ntpinstance.py
ipaserver/install/ntpinstance.py:1: [F0001(fatal), ] No module named ipaserver/install/ntpinstance.py)
Adding --diff-filter to fastlint will not list deleted files
in git diff --names-only output to not include not existing
files to checklist.
Reviewed-By: Stanislav Laznicka <slaznick@redhat.com>
`git diff` shows also context lines by default. When passed to pycodestyle
it can produce errors unrelated to changed lines. It prevents running of
subsequent checks.
Limiting context to 0 lines by `git diff -U0` enables to test only the
modified lines and allows to run subsequent checks.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Python 3's regular expression default to full range of unicode
characters. Restrict \w matches to ASCII and drop \b suffix check to fix
a problem with validation the Chinese translation zh_CN.
Co-Authored-By: Stanislav Laznicka <slaznick@redhat.com>
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Stanislav Laznicka <slaznick@redhat.com>
Make it possible to run API, ACI, and potests under Python 3.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
- Check pycodestyle before pylint. pycodestyle takes seconds while
pylint can easily take half a minute or more.
- Fix exit, needs two $
- Add some newlines to make output more readable
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Tibor Dudlak <tdudlak@redhat.com>
Fast linting only needs modified files with pylint and diff with
pycodestyle. It's good enough to detect most code errors very fast. It
typically takes less than 10 seconds. A complete full pylint run uses
all CPU cores for several minutes. PEP 8 violations are typically
reported after 30 minutes to several hours on Travis CI.
Fast lintings uses git diff and git merge-base to find all modified
files in a branch or working tree. There is no easy way to find the
branch source. On Travis the information is provided by Travis. For
local development it's a new variable IPA_GIT_BRANCH in VERSION.m4.
Fast testing execute all unit tests that do not depend on ipalib.api.
In total it takes about 30-40 seconds (!) to execute linting, PEP 8 checks
and unittests for both Python 2 and 3.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Instead of symlinks and build-time configuration the ipaplatform module
is now able to auto-detect platforms on import time. The meta importer
uses the platform 'ID' from /etc/os-releases. It falls back to 'ID_LIKE'
on platforms like CentOS, which has ID=centos and ID_LIKE="rhel fedora".
The meta importer is able to handle namespace packages and the
ipaplatform package has been turned into a namespace package in order to
support external platform specifications.
https://fedorahosted.org/freeipa/ticket/6474
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
ignore_import_errors was added in 9b534238 to build FreeIPA ACI/API with
some dependencies missing. It turns out that the import hook doesn't
play nice with other meta importers or Cython-generated code like lxml:
./makeaci: ipaserver/plugins/dogtag.py:246: ignoring ImportError: No module named lxml.re
Traceback (most recent call last):
File "./makeaci", line 134, in <module>
main(options)
File "./makeaci", line 107, in main
api.finalize()
File "ipalib/plugable.py", line 733, in finalize
self.__do_if_not_done('load_plugins')
File "ipalib/plugable.py", line 425, in __do_if_not_done
getattr(self, name)()
File "ipalib/plugable.py", line 614, in load_plugins
self.add_package(package)
File "ipalib/plugable.py", line 641, in add_package
module = importlib.import_module(name)
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "ipaserver/plugins/dogtag.py", line 246, in <module>
from lxml import etree
File "src/lxml/etree.pyx", line 93, in init lxml.etree
File "src/lxml/_elementpath.py", line 58, in init lxml._elementpath
AttributeError: 'FailedImport' object has no attribute 'compile'
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
This patchset returns back js directory with its symlinks into
the repository so it does not need to be created during build.
https://pagure.io/freeipa/issue/6447
Reviewed-By: Petr Vobornik <pvoborni@redhat.com>
pip install foo foo[more] does not install the extra dependencies 'more'
of foo. It's a known bug in pip, see
https://github.com/pypa/pip/issues/4391#issuecomment-290712930 and
https://github.com/pypa/pip/issues/988
The same bug applies to pip wheel. As a workaround pip wheel first
builds extra dependencies, then wheel dependencies. This ensures that
ipaclient[otptoken_yubikey] dependencies get built properly.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Add tox infrastructure to test client wheel packages workflow:
* build client packages
* install client packages
* ipa-run-tests --ipaclient-unittests under Python 2 and 3
* pylint of client packages under Python 2 and 3
* placeholder packages work as expected
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Parallel make or flags like IPA_OMIT_INSTALL and IPA_SERVER_WHEELS could
lead to bad packages for PyPI. Only build the packages we want with
correct flags.
Placeholder packages from 'make pypi_package' conflict with
'make wheel_bundle' packages. Use a separate destination directory for
PyPI packages.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
To create a wheel bundle with ipaserver and its dependencies:
make wheel_bundle IPA_SERVER_WHEELS=1
To include additional dependencies:
make wheel_bundle IPA_EXTRA_WHEELS=ipatests[webui]
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Ticket 6604 makes pylint and jsl optional dependencies. The change
is controversal, because some developers prefer that pylint and jsl
should be required unless explicitly disabled.
`make devcheck` is my answer to address the concerns. It's a superior
solution to `make lint` as pre-commit check. It combines several
additional checks under a single, easy rememberable and convenient make
target:
* build all
* acilint, apiclient, jslint, polint
* make check
* pylint under Python 2 and 3
* subset of unit test suite
https://fedorahosted.org/freeipa/ticket/6604
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Stanislav Laznicka <slaznick@redhat.com>
The presence of IPA packages on PyPI revealed an interesting issue with
make wheel_bundle. pip gives final releases a higher precedence than our
development packages. make wheel_bundle downloads ipa 4.5.0 from PyPI
instead of using our own wheels.
Use a constraint file to enforce correct versions.
https://pagure.io/freeipa/issue/6468
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
./configure no longer fails when pylint or jsl are not available. The
make targets for pylint and jsl are no longer defined without the tools.
Rational:
pylint and jsl are not required to build FreeIPA. Both are useful
developer tools. It's more user friendly to make both components
optionally with default config arguments. There is no reason to
fail building on a build system without development tools.
It's still possible to enforce dependency checks with --with-jslint and
--enable-pylint.
https://fedorahosted.org/freeipa/ticket/6604
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Reviewed-By: Tomas Krizek <tkrizek@redhat.com>
Add new pylint AST checker plugin which implements a check for imports
forbidden in IPA. Which imports are forbidden is configurable in pylintrc.
Provide default forbidden import configuration and disable the check for
existing forbidden imports in our code base.
Reviewed-By: Martin Basti <mbasti@redhat.com>
So that it will be nicely formatted on FreeIPA Pagure landing page.
https://pagure.io/freeipa
Some links were updated as other projects also moved to Pagure.io.
Reviewed-By: Stanislav Laznicka <slaznick@redhat.com>
pylint gets confused by duplicated package names, e.g. ipaplatform and
pypi/ipaplatform.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
I also renamed the base directory to pypi and added a new build target
pypi_packages.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
The ipa and freeipa packages are placeholders to prevent PyPI squashing
attacks and reserve the names for future use. `pip install ipa` installs
ipaclient.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
--without-ipatests skips building and installation of the ipatests
package. By default the ipatests package is always build and installed
by make install.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Simo Sorce <ssorce@redhat.com>
Reviewed-By: Tomas Krizek <tkrizek@redhat.com>
The new development server depends on werkzeug instead of paste. The
werkzeug WSGI server comes with some additional features, most
noticeable multi-processing server. The IPA framework is not compatible
with threaded servers. Werkzeug can serve static files easily and has a
fast auto-reloader.
The new lite-server implementation depends on PR 314 (privilege
separation). For Python 3 support, it additionally depends on PR 393.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Let's replace some ugly hacks with proper pytest conftest.py hooks.
Test initialization of ipalib.api is now handled in
pytest_cmdline_main(). Pytest plugins, markers and ignores are also
moved into conftest.py. Additional guards make it possible to run tests
without ipaserver installed.
I added confcutdir to ensure that pytest does not leave our project
space. Pytest used pytest.ini or setup.py before but pytest.ini is gone.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Milan Kubik <mkubik@redhat.com>
configure option --with/without-pylint was replaced by
--enable/disable-pylint. Pylint is always called as $(PYTHON) -m python.
If you need to override Pylint version, use command "make pylint PYTHON=xxx".
https://fedorahosted.org/freeipa/ticket/157
Reviewed-By: Martin Basti <mbasti@redhat.com>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
* make wheel_bundle no longer bundles ipaplatform
* ipaclient and ipalib use a consistent extra tag for the install
subpackage. `pip install ipalib[ipalib.install]` looks a bit silly.
https://fedorahosted.org/freeipa/ticket/6474
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Tomas Krizek <tkrizek@redhat.com>
AC_CONFIG_FILES in configure.ac works well only with Makefiles.
Other files have to be handled by Makefile.am so depedencies
are tracked properly.
There is a problem that Python sub-directories depend on ipasetup.py
which is one level above the sub-directory. This means that depedencies
are the other way around that expected. This is being worked around
using hack from
http://lists.gnu.org/archive/html/automake/2009-03/msg00011.htmlhttps://fedorahosted.org/freeipa/ticket/6498
Reviewed-By: Martin Basti <mbasti@redhat.com>
AC_CONFIG_FILES in configure.ac works well only with Makefiles.
Other files have to be handled by Makefile.am so depedencies
are tracked properly.
https://fedorahosted.org/freeipa/ticket/6498
Reviewed-By: Martin Basti <mbasti@redhat.com>
AC_CONFIG_FILES in configure.ac works well only with Makefiles.
Other files have to be handled by Makefile.am so depedencies
are tracked properly.
https://fedorahosted.org/freeipa/ticket/6498
Reviewed-By: Martin Basti <mbasti@redhat.com>
AC_CONFIG_FILES in configure.ac works well only with Makefiles.
Other files have to be handled by Makefile.am so depedencies
are tracked properly.
https://fedorahosted.org/freeipa/ticket/6498
Reviewed-By: Martin Basti <mbasti@redhat.com>
The Makefile test targets were lost when gettextize infrastructure was
introduced. Now it is re-added in its modernized form which counts with
generated .pot files.
ipatests/i18n.py is now explicitly setting character encoding in files
it generates. According to gettext manual chapter "Filling in the Header Entry"
the Content-Type header is language-specific so it does not make sense
to fill it in in .pot file.
https://fedorahosted.org/freeipa/ticket/6418
Reviewed-By: Martin Basti <mbasti@redhat.com>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Lukas Slebodnik <lslebodn@redhat.com>
The target was added to top-level Makefile.am as well so the maintainer
does not need to jump between directories when doing Zanata pull/push
and strip-po.
https://fedorahosted.org/freeipa/ticket/6418
Reviewed-By: Martin Basti <mbasti@redhat.com>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Lukas Slebodnik <lslebodn@redhat.com>
We now use standard framework generatedby "gettextize" utility.
It has two limitations which I do not consider sufficiently important
to invest into hand-made solution:
1. It can automatically gather strings only from files which have some
file extension like .c or .py. Right now we do not have any
translatable strings in Python files without extensions. Given that these
files will be removed from source tree and replaced with entry points
from setuptools I do not see a reason to invest into supporting this.
2. It does not automatically strip untranslated strings from po files.
This is a manual step in mainteiner's in workflow anyway so I will
add separate Makefile target for it later on.
This commit contains gettextize instrastructure + filled-in files
Makevars and POTFILES.in.
https://fedorahosted.org/freeipa/ticket/6418
Reviewed-By: Martin Basti <mbasti@redhat.com>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Lukas Slebodnik <lslebodn@redhat.com>
Exclude rpmbuild from pylint checks when make lint is executed.
Clean up the current find expression.
https://fedorahosted.org/freeipa/ticket/6418
Reviewed-By: Petr Spacek <pspacek@redhat.com>
This is a huge hack. rpms target will touch VERSION.m4 file. This change
is then detected by automake Makefiles which subsequently re-execute configure
and make.
We have to workaround fact that variables in new make targets
(executed after new configure) are different than original ones.
Also, we have to 'bake-in' precise snapshot version from Git to
VERSION.m4 inside of RPM tarball so the RPM does not depend on git
anymore.
All this magic slows build down a bit.
Do not enable IPA_VERSION_IS_GIT_SNAPSHOT if you want fastest possible builds.
The option IPA_VERSION_IS_GIT_SNAPSHOT is now enabled by default as it
was before we started the build system refactoring effort.
https://fedorahosted.org/freeipa/ticket/6418
Reviewed-By: Tomas Krizek <tkrizek@redhat.com>
