Doctests of ipatests.util fail under Python 3.
The old test scenario does no longer work on Python 3 since u'how are you'
and 'how are you' have identical type, but u'how are you' != b'how are you'.
It works with int / float on all Python versions.
Python 2 has <type 'int'> while Python 3 uses <class 'int'>.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
In-memory cache causes problem in forking servers. A file based cache is
good enough. It's easier to understand and avoids performance regression
and synchronization issues when cert becomes out-of-date.
https://pagure.io/freeipa/issue/6787
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
pytest 3.x does no longer support plain pytest.skip() on module level.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
The presence of IPA packages on PyPI revealed an interesting issue with
make wheel_bundle. pip gives final releases a higher precedence than our
development packages. make wheel_bundle downloads ipa 4.5.0 from PyPI
instead of using our own wheels.
Use a constraint file to enforce correct versions.
https://pagure.io/freeipa/issue/6468
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
The changes made to ipa-run-tests script in fd1b4f6 broke the
ipa-test-config and ipa-test-task scripts which are not executed
via pytest.
To fix the issue, all helper code and dependencies of the integration
plugin are moved out of ipatests.test_integration and into the
integration plugin. As first step ipatests.pytest_plugins.integration
is turned into a package.
https://pagure.io/freeipa/issue/6798
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Milan Kubik <mkubik@redhat.com>
Apache has a default keep alive timeout of 5 seconds. That's too low for
interactive commands, e.g. password prompts. 30 seconds sounds like a
good compromise.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Tomas Krizek <tkrizek@redhat.com>
Do not forcefully close the connection after every request. This enables
HTTP connection keep-alive, also known as persistent TCP and TLS/SSL
connection. Keep-alive speed up consecutive HTTP requests by 15% (for
local, low-latency network connections to a fast server) to multiple
times (high latency connections or remote peers).
https://pagure.io/freeipa/issue/6641
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Tomas Krizek <tkrizek@redhat.com>
ctypes can only handle bytes, not text. Encode and decode all incoming
and outgoing text from UTF-8 to bytes.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Simo Sorce <ssorce@redhat.com>
************* Module ipaserver.install.ipa_kra_install
ipaserver/install/ipa_kra_install.py:25: [W0402(deprecated-module), ] Uses of a deprecated module 'optparse')
************* Module ipapython.install.core
ipapython/install/core.py:163: [E1101(no-member), _knob] Module 'types' has no 'TypeType' member)
************* Module ipatests.test_ipapython.test_dn
ipatests/test_ipapython/test_dn.py:1205: [W1505(deprecated-method), TestDN.test_x500_text] Using deprecated method assertEquals())
************* Module ipa-ca-install
install/tools/ipa-ca-install:228: [E1101(no-member), install_master] Instance of 'ValueError' has no 'message' member)
install/tools/ipa-ca-install:232: [E1101(no-member), install_master] Instance of 'ValueError' has no 'message' member)
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Simo Sorce <ssorce@redhat.com>
Reviewed-By: Stanislav Laznicka <slaznick@redhat.com>
Debian packages should be installed under dist-packages, not
site-packages. Debian has patched distutils and setuptools to add a new
flag '--install-layout'. For --with-ipaplatform=debian,
PYTHON_INSTALL_EXTRA_OPTIONS is set to '--install-layout=deb'.
https://pagure.io/freeipa/issue/6764
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Timo Aaltonen <tjaalton@debian.org>
Reviewed-By: Martin Basti <mbasti@redhat.com>
./configure no longer fails when pylint or jsl are not available. The
make targets for pylint and jsl are no longer defined without the tools.
Rational:
pylint and jsl are not required to build FreeIPA. Both are useful
developer tools. It's more user friendly to make both components
optionally with default config arguments. There is no reason to
fail building on a build system without development tools.
It's still possible to enforce dependency checks with --with-jslint and
--enable-pylint.
https://fedorahosted.org/freeipa/ticket/6604
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Reviewed-By: Tomas Krizek <tkrizek@redhat.com>
First try custom location, then csrgen subdir in confdir and finally
fall back to package data.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Ben Lipton <blipton@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
csrgen broke packaging of ipaclient for PyPI. All csrgen related
resources are now package data of ipaclient package. Package data is
accessed with Jinja's PackageLoader() or through pkg_resources.
https://pagure.io/freeipa/issue/6714
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Ben Lipton <blipton@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
* use with statement to open/close files
* prefer fchmod/fchown when a file descriptor is available
* set permission before data is written to file
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Stanislav Laznicka <slaznick@redhat.com>
automake was complaining about duplicated definitions of pkginstall. It
was defined to true in Makefile.python.am only to be overriden in some
Makefile.am.
Now we assume that pkginstall is implicit true and only skip
installation when pkginstall is explicitly set to false.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
pylint gets confused by duplicated package names, e.g. ipaplatform and
pypi/ipaplatform.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Some calls to setup.py specified a build base, some did not. This can
lead to issues, e.g. build, clean and install are using different build
directories.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
I also renamed the base directory to pypi and added a new build target
pypi_packages.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
The ipa and freeipa packages are placeholders to prevent PyPI squashing
attacks and reserve the names for future use. `pip install ipa` installs
ipaclient.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
test_ldap just imported nss.nss to call nss_init_nodb(). It should be
safe to remove the call. Let's see what CI has to say.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Babinsky <mbabinsk@redhat.com>
It's not necesary to restore the context of newly created files. SELinux
ensures that new files have the correct permission. An explicit
restore_context() is only required when either policies have changed or
the context was changed manually.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Reviewed-By: Stanislav Laznicka <slaznick@redhat.com>
* add missing default_backend
* unpad encrypted data
* use cryptography's hashes and HMAC construct
* remove hard dependency on python-nss from setup.py
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Stanislav Laznicka <slaznick@redhat.com>
The schema cache used a BytesIO buffer to read/write schema cache before
it got flushed to disk. Since the schema cache is now loaded in one go,
the temporary buffer is no longer needed.
File locking has been replaced with a temporary file and atomic rename.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: David Kupka <dkupka@redhat.com>
It's inefficient to open a zip file over and over again. By loading all
members of the schema cache file at once, the ipa CLI script starts
about 25 to 30% faster for simple cases like help and ping.
Before:
$ time for i in {1..20}; do ./ipa ping >/dev/null; done
real 0m13.608s
user 0m10.316s
sys 0m1.121s
After:
$ time for i in {1..20}; do ./ipa ping >/dev/null; done
real 0m9.330s
user 0m7.635s
sys 0m1.146s
https://fedorahosted.org/freeipa/ticket/6690
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: David Kupka <dkupka@redhat.com>
Fix "implicit declaration of function ‘strlen’" in ipa_pwd_ntlm.c,
credits to Lukas.
Add -Werror=implicit-function-declaration to CFLAGS to point developers
to missing includes. It causes compilation to fail when a developer
forgets to add a required include. The problem is no longer hidden in a
massive wall of text from make.
Silence a harmless error from 389-DS slapi.h until the bug is fixed in
downstream, https://pagure.io/389-ds-base/issue/48979
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Stanislav Laznicka <slaznick@redhat.com>
--without-ipatests skips building and installation of the ipatests
package. By default the ipatests package is always build and installed
by make install.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Simo Sorce <ssorce@redhat.com>
Reviewed-By: Tomas Krizek <tkrizek@redhat.com>
The new development server depends on werkzeug instead of paste. The
werkzeug WSGI server comes with some additional features, most
noticeable multi-processing server. The IPA framework is not compatible
with threaded servers. Werkzeug can serve static files easily and has a
fast auto-reloader.
The new lite-server implementation depends on PR 314 (privilege
separation). For Python 3 support, it additionally depends on PR 393.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Some tests assume that JSON deserializier returns tuples instead of
lists. I don't think it is necessary but let's pass the tests for now.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Improve performance of FreeIPA's JSON serializer and deserializer.
* Don't indent and sort keys. Both options trigger a slow path in
Python's json package. Without indention and sorting, encoding
mostly happens in optimized C code.
* Replace O(n) type checks with O(1) type lookup and eliminate
the use of isinstance().
* Check each client capability only once for every conversion.
* Use decoder's obj_hook feature to traverse the object tree once and
to eliminate calls to isinstance().
Closes: https://fedorahosted.org/freeipa/ticket/6655
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Avoid comparison of bytes with int in _is_null() check. b'' == 0
triggers a BytesWarning.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Print api.env, uname, euid/egid, cwd and Python version when tests are
run with -v (e.g. ipa-run-tests -v).
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
In tree runs of make check leave some artifacts around. The patch adds
them to make clean and .gitignore.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Babinsky <mbabinsk@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
install-exec and bdist_wheel also depend on version.py. Let's ensure
that version.py is correctly generated when installing or building
packages.
Yes, make is clever and correctly merges dependencies with rules from
included make files.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Milan Kubik <mkubik@redhat.com>
1e06a5195b removed pytest.ini. Without the
ini file, pytest 3.x has suboptimal settings and no longer picks up all
test functions and test files.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Milan Kubik <mkubik@redhat.com>
Let's replace some ugly hacks with proper pytest conftest.py hooks.
Test initialization of ipalib.api is now handled in
pytest_cmdline_main(). Pytest plugins, markers and ignores are also
moved into conftest.py. Additional guards make it possible to run tests
without ipaserver installed.
I added confcutdir to ensure that pytest does not leave our project
space. Pytest used pytest.ini or setup.py before but pytest.ini is gone.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Milan Kubik <mkubik@redhat.com>
Let's relax the check for .git from directory to exists in order to
support freeipa in a git submodule. Submodules have a .git file with
content like
gitdir: ../.git/modules/freeipa
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Petr Spacek <pspacek@redhat.com>
The environment variable IPA_CONFDIR overrides the default confdir path.
The value of the environment variable must be an absolute path to an existing
directory. The new variable makes it much simpler to use the 'ipa'
command and ipalib with a local configuration directory.
Some scripts (e.g. servers, installers, and upgrades) set the confdir
explicitly and do not support the env var.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Some API contexts are used to modify global state (e.g. files in /etc
and /var). These contexts do not support confdir overrides. Initialize
the API with an explicit confdir argument to paths.ETC_IPA.
The special contexts are:
* backup
* cli_installer
* installer
* ipctl
* renew
* restore
* server
* updates
The patch also corrects the context of the ipa-httpd-kdcproxy script to
'server'.
https://fedorahosted.org/freeipa/ticket/6389
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
python-cryptography versions < 1.3 no longer compile with recent OpenSSL
1.0.2 versions. In order to build wheels, a more recent version of
cryptography is required. 1.3.1 is the oldest well tested version (RHEL
7.3) that is known to work with FreeIPA.
Bump up in freeipa.spec is not required for technical reasons. The
problem only affects PyPI packages. It's policy to keep
requirements in sync.
https://fedorahosted.org/freeipa/ticket/6468
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Babinsky <mbabinsk@redhat.com>
* make wheel_bundle no longer bundles ipaplatform
* ipaclient and ipalib use a consistent extra tag for the install
subpackage. `pip install ipalib[ipalib.install]` looks a bit silly.
https://fedorahosted.org/freeipa/ticket/6474
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Tomas Krizek <tkrizek@redhat.com>
The PyPI package for python-gssapi 1.1.x has a packaging bug. It depends on
enum34 for Python 3 although it is only required for 2.7. 1.2.0 is the
oldest version that has been tested at length by QE. It's know to work.
Bump up in freeipa.spec is not required for technical reasons. The
packaging bug only affects PyPI packages. It's policy to keep
requirements in sync.
https://fedorahosted.org/freeipa/ticket/6468
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Babinsky <mbabinsk@redhat.com>
In Python 3 exception instances no longer have a message attribute.
For most exceptions, str(e) or string formatting give the same result.
Fix some renamed modules, module members and functions.
https://fedorahosted.org/freeipa/ticket/4985
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Here is an attempt to break the import cycle of hell between ipaplatform
and ipalib. All services now pass an ipalib.api object to
services.service(). RedHatServices.__init__() still needs to do a local
import because it initializes its wellknown service dict with service
instances.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
pylint is having a hard time with distutils.version in tox's virtual
envs. virtualenv uses some tricks to provide a virtual distutils
package, pylint can't cope with.
https://github.com/PyCQA/pylint/issues/73 suggests to use pkg_resources
instead. pkg_resources' version parser has some more benefits, e.g. PEP
440 conformity. But pkg_resources.parse_version() is a heavy weight solution
with reduced functionality, e.g. no access to major version.
For API_VERSION and plugin version we can use a much simpler and faster
approach.
https://fedorahosted.org/freeipa/ticket/6468
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Reviewed-By: Martin Babinsky <mbabinsk@redhat.com>
The install subpackages of ipaclient, ipalib and ipapython contain
helper code for installers such as ipa-client-install. They also depend
on external modules that are not available on PyPI, e.g. SSSDConfig.
Since PyPI wheel packages do not support client installation, the
install subpackages contain dead and unsupported code.
The custom build_py plugin removes the subpackages from bdist_wheel
builds. It's not enough to just remove 'ipaclient.install' from the
'packages' list. Surplus files have to be removed from build/lib, too.
https://fedorahosted.org/freeipa/ticket/6468
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Py3: classes with __eq__ must provide __hash__ function or set __hash__
to None.
Comparison function like __eq__ must signal unsupported types by
returning NotImplemented. Python turns this in a proper TypeError.
Make the version member read-only and cache _bytes represention.
https://fedorahosted.org/freeipa/ticket/6473
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
Pylint refuses to load extension modules from unsafe places. This
triggers import-error failures for pylint runs inside a tox virtualenv.
Any module or package in extension-pkg-whitelist is whitelisted and
pylint imports extension modules.
https://fedorahosted.org/freeipa/ticket/6468
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
With setuptools in place FreeIPA is able to register its Custodia
plugins. Custodia 0.1 ignores the plugins directives. Custodia 0.2 uses
the entry points to discover plugins.
https://fedorahosted.org/freeipa/ticket/6492
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Babinsky <mbabinsk@redhat.com>
The ipa-client-automount script used lxml.etree to modify
/etc/autofs_ldap_auth.conf.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
api.env.nss_dir is no longer hard-coded to paths.IPA_NSSDB_DIR. Instead
the path is calculated relatively to api.env.confdir. The default value
is still /etc/ipa/nssdb. The change makes it a bit easier to run
FreeIPA's API with a custom configuration directory.
See https://fedorahosted.org/freeipa/ticket/6386
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Tomas Krizek <tkrizek@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
ipaplatform.fedora.services used to modify the redhat_system_units dict.
It now creates a proper shallow copy.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Martin Babinsky <mbabinsk@redhat.com>
Christian Heimes