The ipautil.run function now returns an object with returncode and
output are accessible as attributes.
The stdout and stderr of all commands are logged (unless skip_output is given).
The stdout/stderr contents must be explicitly requested with a keyword
argument, otherwise they are None.
This is because in Python 3, the output needs to be decoded, and that can
fail if it's not decodable (human-readable) text.
The raw (bytes) output is always available from the result object,
as is "leniently" decoded output suitable for logging.
All calls are changed to reflect this.
A use of Popen in cainstance is changed to ipautil.run.
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
The client XML-RPC implementation is tied to rpclib internals,
so with a change in Python it needs to be updated. And rpclib
changed in Python 3.
Reviewed-By: Tomas Babej <tbabej@redhat.com>
StringIO was renamed in Python 3. The import was was unused,
so remove it.
Files need to be opened in binary mode if bytes are written to them.
(For Python 2: on Linux, there's no practical difference between
text and binary mode)
Reviewed-By: Tomas Babej <tbabej@redhat.com>
In Python 3, the "message" property was removed in favor of calling str().
Add it to PublicMessage, since IPA code depends on it.
Reviewed-By: Tomas Babej <tbabej@redhat.com>
In Python 3, the base64.b64decode function raises binascii.Error (a ValueError
subclass) when it finds incorrect padding. In Python 2 it raises TypeError.
Callers should usually handle ValueError; unless they are specifically
concerned with handling base64 padding issues).
In some cases, callers should handle ValueError:
- ipalib.pkcs10 (get_friendlyname, load_certificate_request): callers should
handle ValueError
- ipalib.x509 (load_certificate*, get_*): callers should handle ValueError
In other cases ValueError is handled:
- ipalib.parameters
- ipapython.ssh
- ipalib.rpc (json_decode_binary - callers already expect ValueError)
- ipaserver.install.ldapupdate
Elsewhere no error handling is done, because values come from trusted
sources, or are pre-validated:
- vault plugin
- ipaserver.install.cainstance
- ipaserver.install.certs
- ipaserver.install.ipa_otptoken_import
Reviewed-By: Tomas Babej <tbabej@redhat.com>
In Python 2, numbers prfixed with '0' are parsed as octal,
e.g. '020' -> 16. In Python 3, the prefix is '0o'.
Handle the old syntax for IPA's parameter conversion to keep
backwards compatibility.
Reviewed-By: Tomas Babej <tbabej@redhat.com>
Python 3 removes the "message" attribute from exceptions, in favor
of just calling str().
Add it back for IPA's own exception types.
Reviewed-By: Tomas Babej <tbabej@redhat.com>
- Don't encode under Python 3, where shlex would choke on bytes
- Sort the attrs dictionary in export_to_string, so the tests are
deterministic. (The iteration order of dicts was always unspecified,
but was always the same in practice under CPython 2.)
Reviewed-By: Tomas Babej <tbabej@redhat.com>
Python 3 uses plain function objects instead of unbound methods.
So, what was Class.method.__func__ is now just Class.method.
Reviewed-By: Tomas Babej <tbabej@redhat.com>
In Python 3, the variable with the currently handled exception is unset
at the end of the except block. (This is done to break reference
cycles, since exception instances now carry tracebacks, which contain
all locals.)
Fix this in baseldap's error handler.
Use a simpler structure for the ipatests.raises utility that only uses the
exception inside the except block.
Reviewed-By: Tomas Babej <tbabej@redhat.com>
In python 3 , `bytes` has the buffer interface, and `buffer` was removed.
Also, invalid padding in base64-encoded data raises a ValueError rather
than TypeError.
In tests, use pytest.assert_raises for more correct exception assertions.
Also, get rid of unused imports in the tests
Reviewed-By: Tomas Babej <tbabej@redhat.com>
In Python 3, different types are generally not comparable (except for equality),
and None can't be compared to None.
Fix cases of these comparisons.
In ipatest.util, give up on sorting lists if the sorting raises a TypeError.
Reviewed-By: Tomas Babej <tbabej@redhat.com>
Sort out the accepted types.
Handle Python 3's stricter separation between bytes and unicode.
Reviewed-By: David Kupka <dkupka@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
In Python 3, the types module no longer provide alternate names for
built-in types, e.g. `types.StringType` can just be spelled `str`.
NoneType is also removed; it needs to be replaced with type(None)
Reviewed-By: David Kupka <dkupka@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
The StringIO class was moved to the io module.
(In Python 2, io.StringIO is available, but is Unicode-only.)
Reviewed-By: David Kupka <dkupka@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
In Python 3, integers don't have a maximum. The number called
"sys.maxint" is now "sys.maxsize" (defined as larger than the
largest possible list/string index).
The new spelling is also available in Python 2.7.
Reviewed-By: David Kupka <dkupka@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
- Use PyLong instead of PyInt on Python 3
- Use PyBytes instead of PyString for binary strings
- Use Py_TYPE(o) instead of o->py_type, and PyVarObject_HEAD_INIT
instead of PyObject_HEAD_INIT
- Use a helper function instead of Py_BuildValue("s#"), so that
bytes are returned on Python 3
- Use new style module initialization on Python 3
- Use PyModule_AddIntConstant and PyModule_AddIntMacro for adding constants
Reviewed-By: David Kupka <dkupka@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Reviewed-By: Martin Basti <mbasti@redhat.com>
In Python 3, zip() returns an iterator. To get a list, it must
be explicitly converted.
In most cases, zip() result is iterated over so this is not
necessary.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
In Python 3, range() behaves like the old xrange().
The difference between range() and xrange() is usually not significant,
especially if the whole result is iterated over.
Convert xrange() usage to range() for small ranges.
Use modern idioms in a few other uses of range().
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
The three-argument raise is going away in Python 3. Use the six.reraise
helper instead.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
The form`raise Error, value` is deprecated in favor of `raise Error(value)`,
and will be removed in Python 3.
Use the new syntax.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
In Python 3, `print` is no longer a statement. Call it as a function
everywhere, and include the future import to remove the statement
in Python 2 code as well.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
In Python 3, next() for iterators is a function rather than method.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
In Python 2, map() returns a list; in Python 3 it returns an iterator.
Replace all uses by list comprehensions, generators, or for loops,
as required.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
In Python 3, raw_input() was renamed to input().
Import the function from six.moves to get the right version.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
In Python 3, filter() returns an iterator.
Use list comprehensions instead.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Python 2 has keys()/values()/items(), which return lists,
iterkeys()/itervalues()/iteritems(), which return iterators,
and viewkeys()/viewvalues()/viewitems() which return views.
Python 3 has only keys()/values()/items(), which return views.
To get iterators, one can use iter() or a for loop/comprehension;
for lists there's the list() constructor.
When iterating through the entire dict, without modifying the dict,
the difference between Python 2's items() and iteritems() is
negligible, especially on small dicts (the main overhead is
extra memory, not CPU time). In the interest of simpler code,
this patch changes many instances of iteritems() to items(),
iterkeys() to keys() etc.
In other cases, helpers like six.itervalues are used.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
This feature was removed in Python 3 to ease introspection.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
sys.exc_value is deprecated since Python 1.5, and was removed in Python 3.
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
The reduce function is no longer a built-in in Python 3.
Importing it from functools works on both py2 and py3.
Reviewed-By: Tomas Babej <tbabej@redhat.com>
The deprecated has_key method will be removed from dicts in Python 3.
For custom dict-like classes, has_key() is kept on Python 2,
but disabled for Python 3.
Reviewed-By: Tomas Babej <tbabej@redhat.com>
Python 3 uses double-underscored names for internal function attributes.
In Python 2.7, these names exist as aliases to the old 'func_*' and
'im_*' names.
Reviewed-By: Tomas Babej <tbabej@redhat.com>
Use Python-3 compatible syntax, without breaking compatibility with py 2.7
- Octals literals start with 0o to prevent confusion
- The "L" at the end of large int literals is not required as they use
long on Python 2 automatically.
- Using 'int' instead of 'long' for small numbers is OK in all cases except
strict type checking checking, e.g. type(0).
https://fedorahosted.org/freeipa/ticket/4985
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Make all DNs, RDNs and AVAs immutable.
Immutability makes reasoning about DN-handling code easier,
as value objects can't be changed once created.
Instead of mutable DNs, one can use a list (or even a generator)
of RDNs that's converted to a DN on output.
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
The racker object "remembers" expected state across several tests,
so only changes (rather than all expected state) need to be specified
in each test. Also, the tracker fixture will make it easy to use hosts
in other test modules.
This change makes the tests independent; any permutation of any subset
of these tests should now pass.
Reviewed-By: Tomas Babej <tbabej@redhat.com>
This has several advantages:
- Tests other than run-command/check-response can be added easily
- Tracebacks are meaningful (which means we'll be able to remove a lot of
test name/description/location tracking code)
- Individual tests can be selected/deselected using normal pytest mechanisms
(but for isolated tests, more changes will be needed)
Reviewed-By: Tomas Babej <tbabej@redhat.com>
Drop support for pylint < 1.0
Enable ignoring unknown attributes on modules (both nose and pytest
use advanced techniques, support for which only made it to pylint
recently)
Fix some bugs revealed by pylint
Do minor refactoring or add pylint:disable directives where the
linter complains.
Reviewed-By: Tomas Babej <tbabej@redhat.com>
The plugin to run tests within a class in the order they're defined
in the source was split into a separate project.
Use this project instead of a FreeIPA-specific copy.
Reviewed-By: Tomas Babej <tbabej@redhat.com>
The plugin for BeakerLib integration was split into a separate project.
If BeakerLib integration is desired, python-pytest-beakerlib shoule be
installed separately.
The IPA-specific beakerlib integration only sets up logging to BeakerLib,
if the plugin is active.
Reviewed-By: Tomas Babej <tbabej@redhat.com>
The core integration testing functionality was split into a separate
project. Use this project, and configure it for FreeIPA.
The "mh" (multihost) fixture is made available for integration tests.
Configuration based on environment variables is moved into a separate
module, to ease eventual deprecation.
Reviewed-By: Tomas Babej <tbabej@redhat.com>
Pytest imports all modules when running doctests.
The setup.py runs code on import, and raises an exception,
depending on globa connand-line arguments, so it needs to be ignored.
Also, pytest dislikes multiple top-level modules with the same name
("setup" in this case). Again ignoring is the way to go.
Move the IPA-specific log collection out of the Beakerlib plugin.
Add the --logfile-dir option to tests and ipa-test-task, so that logs
can be collected even if BeakerLib is not used.
https://fedorahosted.org/freeipa/ticket/4610
Reviewed-By: Tomas Babej <tbabej@redhat.com>
The --with-xunit option ihas the same behavior as in nosetests:
it's an alias for pytest's --junitxml=nosetests.py
The --logging-level option enables direct IPA logging to stdout.
https://fedorahosted.org/freeipa/ticket/4610
Reviewed-By: Tomas Babej <tbabej@redhat.com>
The unused capability to run on multiple Python versions is removed,
and needed arguments are now listed in pytest.ini,
leaving just a simple call to the actual test runner.
https://fedorahosted.org/freeipa/ticket/4610
Reviewed-By: Tomas Babej <tbabej@redhat.com>
The pytest.ini file needs to be in or above the directory py.test is called in.
When in IPA project root, this invocation will find ./ipatests/pytest.ini:
py.test ipatests/
but these will not (they're equivalent):
py.test .
py.test
So pytest.ini must be in the project root.
However, setupttols can't include files outside package directories,
so we also need this file to be under ipatests/
Solve the problem by symlinking ./pytest.ini to ipatests/pytest.ini.
https://fedorahosted.org/freeipa/ticket/4610
Reviewed-By: Tomas Babej <tbabej@redhat.com>
The setUp/dearDown names are used in the unittest module, but there is no reason
to use them in non-`unittest` test cases.
Nose supports both styles (but mixing them can cause trouble when
calling super()'s methods).
Pytest only supports the new ones.
https://fedorahosted.org/freeipa/ticket/4610
Reviewed-By: Tomas Babej <tbabej@redhat.com>
IPA only uses one instance of the directory server. When an instance
is not specified to a call to service.start/stop/restart/...,
use IPA's instance.
Stopping a systemd service is synchronous (bby default), but stopping
a target is not. This will change ensures that the directory server
is actually down when stop() finishes.
https://fedorahosted.org/freeipa/ticket/4709
Reviewed-By: Jan Cholasta <jcholast@redhat.com>
Petr Viktorin