`printf` ignores excessive arguments unused in formatting.
This resulted in only the first file from two file lists was
linted/ stylechecked if both Python template files and Python
modules were changed.
Make use of formatting instead:
> The format is reused as necessary to consume all of the arguments
Fixes: https://pagure.io/freeipa/issue/9318
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
In the yamllint target of makefile, prci_checker result was being masked by echo statement.
Aditionally, prci_checker script has been added to the list of Python sources to be Pylinted.
Addressing comments of recently merged PR:
https://github.com/freeipa/freeipa/pull/6301#discussion_r923163970https://github.com/freeipa/freeipa/pull/6301#issuecomment-1187037261
Signed-off-by: David Pascual <davherna@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Reviewed-By: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Francisco Trivino <ftrivino@redhat.com>
This script allows developers to check if prci definition jobs have the correct format,
which is defined in prci_jobs_spec.yaml
Useful when adding new jobs to the definitions.
Signed-off-by: David Pascual <davherna@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Reviewed-By: Petr Vobornik <pvoborni@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Reviewed-By: Stanislav Levin <slev@altlinux.org>
Many of IPA Python scripts are shebang configurable scripts and
have special suffix '.in' for that. Pycodestyle by default check
only '*.py' files [0].
[0]: https://pycodestyle.pycqa.org/en/latest/intro.html
Fixes: https://pagure.io/freeipa/issue/8961
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Reviewed-By: François Cami <fcami@redhat.com>
- previously, fastlint make's target includes both the Pylint task
and pycodestyle one. The purpose of this target is a fast checking
only for changed Python files. This makes sense for pycodestyle, but
limits Pylint due to a context(file) checking. The clients which
call the code being linted are not checked at all. In Azure Pylint
(for the whole codebase) is run in the Lint task, this makes fastlint
extra for Azure.
- `Quick code style check` task used distro's Pylint, while `Lint`
task PyPI's one. This may cause different results and confuse a
user.
- `Build` task takes time longer than `Lint` one, so this change
doesn't lead to increased CI time.
- all Azure tests depend on Build and Lint tasks. Mostly it's no need
to run tests due to a probably broken code.
Fixes: https://pagure.io/freeipa/issue/8961
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Reviewed-By: François Cami <fcami@redhat.com>
New LDAP object class "ipaUserSubordinate" with four new fields:
- ipasubuidnumber / ipasubuidcount
- ipasubgidnumber / ipasgbuidcount
New self-service permission to add subids.
New command user-auto-subid to auto-assign subid
The code hard-codes counts to 65536, sets subgid equal to subuid, and
does not allow removal of subids. There is also a hack that emulates a
DNA plugin with step interval 65536 for testing.
Work around problem with older SSSD clients that fail with unknown
idrange type "ipa-local-subid", see: https://github.com/SSSD/sssd/issues/5571
Related: https://pagure.io/freeipa/issue/8361
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Francois Cami <fcami@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Lint task uses PyPI to get the latest Pylint, which may pull in
any other packages as dependencies. For distro isolation the Lint
job should not produce any meaningful artifacts or they should not
be used in subsequent jobs. So, this job have to be isolated from
the others.
Fixes: https://pagure.io/freeipa/issue/8772
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Distributions may want to run comprehensive fastcheck or lint tasks,
but rpmlint tool is mandatory for these targets while some platforms
don't have it at all.
With this change the rpmlint becomes optional for fastcheck, devcheck
and lint make targets.
Note: rpmlint option is disabled by default.
To enable: ./configure --enable-rpmlint
To explicitly disable: ./configure --disable-rpmlint
Fixes: https://pagure.io/freeipa/issue/8768
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
A new test case was not picking up ``ipa-run-tests`` script.
Fixes: https://pagure.io/freeipa/issue/8307
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Sphinx is extensible with plugins that can add new syntax, roles,
directives, domains, and more.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Run sphinx-builder with -W (fail on error), --keep-going, and -j auto.
Auto-job scaling speeds up sphinx-builder a LOT.
Add make lint target to doc/Makefile. The -E and -a option ensure that
all files are always re-read and rewritten.
Add option to run sphinx-builder from a virtual env that mimics RTD
builds closer than Fedora packages.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: François Cami <fcami@redhat.com>
Add doclint to fastcheck so that documentation syntax issues
are caught sooner (before they hit CI).
Signed-off-by: François Cami <fcami@redhat.com>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
'makeapi' and 'makeaci' has to be run in a particular environment that
forces IPA Python modules from the source tree used instead of what
might be installed system-wide.
Create 'make api' and 'make aci' targets to provide easy access to them.
Make sure we run Python interpreter with PYTHONPATH set to force use of
the source tree.
Signed-off-by: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Florence Blanc-Renaud <frenaud@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
The SELinux policy defines file contexts that are also used by clients,
e.g. /var/log/ipa/. Make freeipa-selinux a dependency of freeipa-common.
Related: https://pagure.io/freeipa/issue/6891
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Hook up the new policy to autoconf and automake.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
> Pylint is a tool that checks for errors in Python code, tries to enforce a
> coding standard and looks for code smells. It can also look for certain type
> errors, it can recommend suggestions about how particular blocks can be
> refactored and can offer you details about the code's complexity.
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
For now, a list of YAML files' paths is hardcoded (even after
globbing) into Makefile.am. Moreover, Azure templates are not
checked at all until Azure triggered.
With this change, the list of YAMLs is populated automatically
on yamllinting.
Jinja templates are not parseable by a regular yaml module, to
skip such the YAML_TEMPLATE_FILES is utilized.
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Removing Travis CI in favour of Azure Pipelines.
All tests previously tested by Travis are also configured in Azure.
Repos [1] and [2] were used to build Docker images for Travis, thus
they are no longer required for branches master and ipa-4-8.
Branches ipa-4-7 and ipa-4-6 don't have Azure pipelines configured,
so Travis will continue to be used by them.
1 - https://github.com/freeipa/ipa-docker-test-runner
2 - https://github.com/freeipa/freeipa-builder
Related: https://pagure.io/freeipa/issue/7323
Signed-off-by: Armando Neto <abiagion@redhat.com>
Reviewed-By: Francisco Trivino <ftrivino@redhat.com>
This is a sort of rollback to the pre #93c158b05 state with
several improvements.
For now, the nodeids calculation by ipa-run-tests is not stable,
since it depends on current working directory. Nodeids (tests
addresses) are utilized by the other plugins, for example.
Unfortunately, the `pytest_load_initial_conftests` hook doesn't
correctly work with pytest internal paths, since it is called
after the calculation of rootdir was performed, for example.
Eventually, it's simpler to follow the default convention for
Python test discovery.
There is at least one drawback of new "old" implementation.
The ignore rules don't support globs, because pytest 4.3.0+
has the same facility via `--ignore-glob`:
> Add the `--ignore-glob` parameter to exclude test-modules with
> Unix shell-style wildcards. Add the collect_ignore_glob for
> conftest.py to exclude test-modules with Unix shell-style
> wildcards.
Upon switching to pytest4 it will be possible to utilize this.
Anyway, tests for checking current basic facilities of
ipa-run-tests were added.
Fixes: https://pagure.io/freeipa/issue/8007
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Previously fastlint would only process .py files.
Detect which .in files are in fact Python files and add
them to the list of files to process during the fastlint
Makefile target.
Original change suggested by Alexander Bokovoy.
Fixes: https://pagure.io/freeipa/issue/7984
Signed-off-by: François Cami <fcami@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
The make target `pylint` hasn't a full list of its dependencies.
This leads to problems like:
- different build results
- PR tests just do not run pylint over some Python scripts.
The new build target (`python_scripts`) was implemented.
It's intended for building all Python scripts (files, containing
@PYTHONSHEBANG@ as a shebang placeholder).
The make `pylint` should require it.
Fixes: https://pagure.io/freeipa/issue/7921
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Build all scripts in install/tools/ to check them with pylint, so that
``make pylint`` always checks all scripts. The script files are
generated by make.
Please note that fastlint does not check script files.
See: https://pagure.io/freeipa/issue/7772
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Thomas Woerner <twoerner@redhat.com>
Add "make cov-scan" to automate Coverity scan builds. cov-build requires
extra quirks to work with recent versions of GCC on Fedora.
The make target requires a token and Coverity's build chain. Both are
available for privileged project owners on
https://scan.coverity.com/projects/freeipa-freeipa .
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Let's catch broken YAML files (Travis, PR-CI) and spec file early.
- Use rpmlint to detect syntax errors in spec file early
- Attempt to parse all YAML files with PyYAML
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Florence Blanc-Renaud <frenaud@redhat.com>
Remove Python 2 related code and configuration from spec file, autoconf
and CI infrastructure. From now on, FreeIPA 4.8 requires at least Python
3.6. Python 2 packages like python2-ipaserver or python2-ipaclient are
no longer available. PR-CI, lint, and tox aren't testing Python 2
compatibility either.
See: https://fedoraproject.org/wiki/Changes/FreeIPA_Python_2_Removal
Fixes: https://pagure.io/freeipa/issue/7568
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Reviewed-By: Tibor Dudlak <tdudlak@redhat.com>
Python scripts are now generated from templates. The scripts are marked
as nodist (no distribution) but install targets. The templates for the
scripts are extra distribution data, no installation (noinst).
Fixes: https://pagure.io/freeipa/issue/7680
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Move logic for installing just the Python packages out of the spec file
and into our root Makefile. It removes code duplication to simplify a
spec file that supports building without Python 2.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
when any file from FreeIPA tree has been deleted there was
a failure like:
pylint
------
************* Module ipaserver/install/ntpinstance.py
ipaserver/install/ntpinstance.py:1: [F0001(fatal), ] No module named ipaserver/install/ntpinstance.py)
Adding --diff-filter to fastlint will not list deleted files
in git diff --names-only output to not include not existing
files to checklist.
Reviewed-By: Stanislav Laznicka <slaznick@redhat.com>
`git diff` shows also context lines by default. When passed to pycodestyle
it can produce errors unrelated to changed lines. It prevents running of
subsequent checks.
Limiting context to 0 lines by `git diff -U0` enables to test only the
modified lines and allows to run subsequent checks.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Python 3's regular expression default to full range of unicode
characters. Restrict \w matches to ASCII and drop \b suffix check to fix
a problem with validation the Chinese translation zh_CN.
Co-Authored-By: Stanislav Laznicka <slaznick@redhat.com>
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Stanislav Laznicka <slaznick@redhat.com>
Make it possible to run API, ACI, and potests under Python 3.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
- Check pycodestyle before pylint. pycodestyle takes seconds while
pylint can easily take half a minute or more.
- Fix exit, needs two $
- Add some newlines to make output more readable
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Tibor Dudlak <tdudlak@redhat.com>
Fast linting only needs modified files with pylint and diff with
pycodestyle. It's good enough to detect most code errors very fast. It
typically takes less than 10 seconds. A complete full pylint run uses
all CPU cores for several minutes. PEP 8 violations are typically
reported after 30 minutes to several hours on Travis CI.
Fast lintings uses git diff and git merge-base to find all modified
files in a branch or working tree. There is no easy way to find the
branch source. On Travis the information is provided by Travis. For
local development it's a new variable IPA_GIT_BRANCH in VERSION.m4.
Fast testing execute all unit tests that do not depend on ipalib.api.
In total it takes about 30-40 seconds (!) to execute linting, PEP 8 checks
and unittests for both Python 2 and 3.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Instead of symlinks and build-time configuration the ipaplatform module
is now able to auto-detect platforms on import time. The meta importer
uses the platform 'ID' from /etc/os-releases. It falls back to 'ID_LIKE'
on platforms like CentOS, which has ID=centos and ID_LIKE="rhel fedora".
The meta importer is able to handle namespace packages and the
ipaplatform package has been turned into a namespace package in order to
support external platform specifications.
https://fedorahosted.org/freeipa/ticket/6474
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
ignore_import_errors was added in 9b534238 to build FreeIPA ACI/API with
some dependencies missing. It turns out that the import hook doesn't
play nice with other meta importers or Cython-generated code like lxml:
./makeaci: ipaserver/plugins/dogtag.py:246: ignoring ImportError: No module named lxml.re
Traceback (most recent call last):
File "./makeaci", line 134, in <module>
main(options)
File "./makeaci", line 107, in main
api.finalize()
File "ipalib/plugable.py", line 733, in finalize
self.__do_if_not_done('load_plugins')
File "ipalib/plugable.py", line 425, in __do_if_not_done
getattr(self, name)()
File "ipalib/plugable.py", line 614, in load_plugins
self.add_package(package)
File "ipalib/plugable.py", line 641, in add_package
module = importlib.import_module(name)
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "ipaserver/plugins/dogtag.py", line 246, in <module>
from lxml import etree
File "src/lxml/etree.pyx", line 93, in init lxml.etree
File "src/lxml/_elementpath.py", line 58, in init lxml._elementpath
AttributeError: 'FailedImport' object has no attribute 'compile'
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
This patchset returns back js directory with its symlinks into
the repository so it does not need to be created during build.
https://pagure.io/freeipa/issue/6447
Reviewed-By: Petr Vobornik <pvoborni@redhat.com>
pip install foo foo[more] does not install the extra dependencies 'more'
of foo. It's a known bug in pip, see
https://github.com/pypa/pip/issues/4391#issuecomment-290712930 and
https://github.com/pypa/pip/issues/988
The same bug applies to pip wheel. As a workaround pip wheel first
builds extra dependencies, then wheel dependencies. This ensures that
ipaclient[otptoken_yubikey] dependencies get built properly.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Add tox infrastructure to test client wheel packages workflow:
* build client packages
* install client packages
* ipa-run-tests --ipaclient-unittests under Python 2 and 3
* pylint of client packages under Python 2 and 3
* placeholder packages work as expected
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Parallel make or flags like IPA_OMIT_INSTALL and IPA_SERVER_WHEELS could
lead to bad packages for PyPI. Only build the packages we want with
correct flags.
Placeholder packages from 'make pypi_package' conflict with
'make wheel_bundle' packages. Use a separate destination directory for
PyPI packages.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
To create a wheel bundle with ipaserver and its dependencies:
make wheel_bundle IPA_SERVER_WHEELS=1
To include additional dependencies:
make wheel_bundle IPA_EXTRA_WHEELS=ipatests[webui]
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Ticket 6604 makes pylint and jsl optional dependencies. The change
is controversal, because some developers prefer that pylint and jsl
should be required unless explicitly disabled.
`make devcheck` is my answer to address the concerns. It's a superior
solution to `make lint` as pre-commit check. It combines several
additional checks under a single, easy rememberable and convenient make
target:
* build all
* acilint, apiclient, jslint, polint
* make check
* pylint under Python 2 and 3
* subset of unit test suite
https://fedorahosted.org/freeipa/ticket/6604
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Stanislav Laznicka <slaznick@redhat.com>
The presence of IPA packages on PyPI revealed an interesting issue with
make wheel_bundle. pip gives final releases a higher precedence than our
development packages. make wheel_bundle downloads ipa 4.5.0 from PyPI
instead of using our own wheels.
Use a constraint file to enforce correct versions.
https://pagure.io/freeipa/issue/6468
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
./configure no longer fails when pylint or jsl are not available. The
make targets for pylint and jsl are no longer defined without the tools.
Rational:
pylint and jsl are not required to build FreeIPA. Both are useful
developer tools. It's more user friendly to make both components
optionally with default config arguments. There is no reason to
fail building on a build system without development tools.
It's still possible to enforce dependency checks with --with-jslint and
--enable-pylint.
https://fedorahosted.org/freeipa/ticket/6604
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Reviewed-By: Tomas Krizek <tkrizek@redhat.com>
Add new pylint AST checker plugin which implements a check for imports
forbidden in IPA. Which imports are forbidden is configurable in pylintrc.
Provide default forbidden import configuration and disable the check for
existing forbidden imports in our code base.
Reviewed-By: Martin Basti <mbasti@redhat.com>
So that it will be nicely formatted on FreeIPA Pagure landing page.
https://pagure.io/freeipa
Some links were updated as other projects also moved to Pagure.io.
Reviewed-By: Stanislav Laznicka <slaznick@redhat.com>
pylint gets confused by duplicated package names, e.g. ipaplatform and
pypi/ipaplatform.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
I also renamed the base directory to pypi and added a new build target
pypi_packages.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
