* Convert every string specifying a DN into a DN object
* Every place a dn was manipulated in some fashion it was replaced by
the use of DN operators
* Add new DNParam parameter type for parameters which are DN's
* DN objects are used 100% of the time throughout the entire data
pipeline whenever something is logically a dn.
* Many classes now enforce DN usage for their attributes which are
dn's. This is implmented via ipautil.dn_attribute_property(). The
only permitted types for a class attribute specified to be a DN are
either None or a DN object.
* Require that every place a dn is used it must be a DN object.
This translates into lot of::
assert isinstance(dn, DN)
sprinkled through out the code. Maintaining these asserts is
valuable to preserve DN type enforcement. The asserts can be
disabled in production.
The goal of 100% DN usage 100% of the time has been realized, these
asserts are meant to preserve that.
The asserts also proved valuable in detecting functions which did
not obey their function signatures, such as the baseldap pre and
post callbacks.
* Moved ipalib.dn to ipapython.dn because DN class is shared with all
components, not just the server which uses ipalib.
* All API's now accept DN's natively, no need to convert to str (or
unicode).
* Removed ipalib.encoder and encode/decode decorators. Type conversion
is now explicitly performed in each IPASimpleLDAPObject method which
emulates a ldap.SimpleLDAPObject method.
* Entity & Entry classes now utilize DN's
* Removed __getattr__ in Entity & Entity clases. There were two
problems with it. It presented synthetic Python object attributes
based on the current LDAP data it contained. There is no way to
validate synthetic attributes using code checkers, you can't search
the code to find LDAP attribute accesses (because synthetic
attriutes look like Python attributes instead of LDAP data) and
error handling is circumscribed. Secondly __getattr__ was hiding
Python internal methods which broke class semantics.
* Replace use of methods inherited from ldap.SimpleLDAPObject via
IPAdmin class with IPAdmin methods. Directly using inherited methods
was causing us to bypass IPA logic. Mostly this meant replacing the
use of search_s() with getEntry() or getList(). Similarly direct
access of the LDAP data in classes using IPAdmin were replaced with
calls to getValue() or getValues().
* Objects returned by ldap2.find_entries() are now compatible with
either the python-ldap access methodology or the Entity/Entry access
methodology.
* All ldap operations now funnel through the common
IPASimpleLDAPObject giving us a single location where we interface
to python-ldap and perform conversions.
* The above 4 modifications means we've greatly reduced the
proliferation of multiple inconsistent ways to perform LDAP
operations. We are well on the way to having a single API in IPA for
doing LDAP (a long range goal).
* All certificate subject bases are now DN's
* DN objects were enhanced thusly:
- find, rfind, index, rindex, replace and insert methods were added
- AVA, RDN and DN classes were refactored in immutable and mutable
variants, the mutable variants are EditableAVA, EditableRDN and
EditableDN. By default we use the immutable variants preserving
important semantics. To edit a DN cast it to an EditableDN and
cast it back to DN when done editing. These issues are fully
described in other documentation.
- first_key_match was removed
- DN equalty comparison permits comparison to a basestring
* Fixed ldapupdate to work with DN's. This work included:
- Enhance test_updates.py to do more checking after applying
update. Add test for update_from_dict(). Convert code to use
unittest classes.
- Consolidated duplicate code.
- Moved code which should have been in the class into the class.
- Fix the handling of the 'deleteentry' update action. It's no longer
necessary to supply fake attributes to make it work. Detect case
where subsequent update applies a change to entry previously marked
for deletetion. General clean-up and simplification of the
'deleteentry' logic.
- Rewrote a couple of functions to be clearer and more Pythonic.
- Added documentation on the data structure being used.
- Simplfy the use of update_from_dict()
* Removed all usage of get_schema() which was being called prior to
accessing the .schema attribute of an object. If a class is using
internal lazy loading as an optimization it's not right to require
users of the interface to be aware of internal
optimization's. schema is now a property and when the schema
property is accessed it calls a private internal method to perform
the lazy loading.
* Added SchemaCache class to cache the schema's from individual
servers. This was done because of the observation we talk to
different LDAP servers, each of which may have it's own
schema. Previously we globally cached the schema from the first
server we connected to and returned that schema in all contexts. The
cache includes controls to invalidate it thus forcing a schema
refresh.
* Schema caching is now senstive to the run time context. During
install and upgrade the schema can change leading to errors due to
out-of-date cached schema. The schema cache is refreshed in these
contexts.
* We are aware of the LDAP syntax of all LDAP attributes. Every
attribute returned from an LDAP operation is passed through a
central table look-up based on it's LDAP syntax. The table key is
the LDAP syntax it's value is a Python callable that returns a
Python object matching the LDAP syntax. There are a handful of LDAP
attributes whose syntax is historically incorrect
(e.g. DistguishedNames that are defined as DirectoryStrings). The
table driven conversion mechanism is augmented with a table of
hard coded exceptions.
Currently only the following conversions occur via the table:
- dn's are converted to DN objects
- binary objects are converted to Python str objects (IPA
convention).
- everything else is converted to unicode using UTF-8 decoding (IPA
convention).
However, now that the table driven conversion mechanism is in place
it would be trivial to do things such as converting attributes
which have LDAP integer syntax into a Python integer, etc.
* Expected values in the unit tests which are a DN no longer need to
use lambda expressions to promote the returned value to a DN for
equality comparison. The return value is automatically promoted to
a DN. The lambda expressions have been removed making the code much
simpler and easier to read.
* Add class level logging to a number of classes which did not support
logging, less need for use of root_logger.
* Remove ipaserver/conn.py, it was unused.
* Consolidated duplicate code wherever it was found.
* Fixed many places that used string concatenation to form a new
string rather than string formatting operators. This is necessary
because string formatting converts it's arguments to a string prior
to building the result string. You can't concatenate a string and a
non-string.
* Simplify logic in rename_managed plugin. Use DN operators to edit
dn's.
* The live version of ipa-ldap-updater did not generate a log file.
The offline version did, now both do.
https://fedorahosted.org/freeipa/ticket/1670https://fedorahosted.org/freeipa/ticket/1671https://fedorahosted.org/freeipa/ticket/1672https://fedorahosted.org/freeipa/ticket/1673https://fedorahosted.org/freeipa/ticket/1674https://fedorahosted.org/freeipa/ticket/1392https://fedorahosted.org/freeipa/ticket/2872
All service start/restart currently go through ipapython/platform so
move the "wait for service to start" code there as well.
A dictionary of known services and ports to wait on is defined in base.py
This is referenced by the platforms by instance name to determine what
to wait for. For the case of dirsrv if we get that as a plain name
(no specific instance) it is assumed to be the main IPA service.
https://fedorahosted.org/freeipa/ticket/2375https://fedorahosted.org/freeipa/ticket/2610
python-ldap add_s method raises a NO_SUCH_OBJECT exception when
a parent entry of the entry being added does not exist. This may
not be an error, for example NIS entries are only added when NIS
is enabled and thus the NIS entry container exists.
The exception raised by python-ldap is also incorrectly processed
in ipaldap's addEntry function and an irrelevant exception is
re-raised instead.
Fix LDAP updater to just log an information when an object cannot
be added due to missing parent object. Also make sure that the
addEntry function exception processing provides the right exception
with a useful description.
https://fedorahosted.org/freeipa/ticket/2520https://fedorahosted.org/freeipa/ticket/2743
IPA has some unused code from abandoned features (Radius, ipa 1.x user
input, commant-line tab completion), as well as some duplicate utilities.
This patch cleans up the utility modules.
Duplicate code consolidated into ipapython.ipautil:
{ipalib.util,ipaserver.ipautil,ipapython.ipautil}.realm_to_suffix
{ipaserver,ipapython}.ipautil.CIDict
(with style improvements from the ipaserver version)
{ipapython.entity,ipaserver.ipautil}.utf8_encode_value
{ipapython.entity,ipaserver.ipautil}.utf8_encode_values
ipalib.util.get_fqdn was removed in favor of the same function in
ipaserver.install.installutils
Removed unused code:
ipalib.util:
load_plugins_in_dir
import_plugins_subpackage
make_repr (was imported but unused; also removed from tests)
ipapython.ipautil:
format_list
parse_key_value_pairs
read_pairs_file
read_items_file
user_input_plain
AttributeValueCompleter
ItemCompleter
ipaserver.ipautil:
get_gsserror (a different version exists in ipapython.ipautil)
ipaserver.ipautil ended up empty and is removed entirely.
https://fedorahosted.org/freeipa/ticket/2650
If minssf is set in configuration and this is not set then clients won't
be able to detect the available namingContexts, defaultNamingContext,
capabilities, etc.
https://fedorahosted.org/freeipa/ticket/2542
The dn value needs to be quoted otherwise it is interpreted to be a
multi-value.
This will replace whatever value is currently set.
https://fedorahosted.org/freeipa/ticket/2452
This is for the LDAP updater in particular. When adding new schema
order can be important when one objectclass depends on another via
SUP.
This calculation will preserve the order of changes in the update file.
Discovered trying to add SSH schema.
https://fedorahosted.org/freeipa/ticket/754
We have bind code that can handle the case where a server hasn't
come up yet. It needs to handle a real connection failure such
as the TLS hostname not matching. If we try to bind anyway we end
up with a segfault in openldap.
https://fedorahosted.org/freeipa/ticket/2301
Add new class "cachedproperty" for creating property-like attributes
that cache the return value of a method call.
Also fix few issues in the unit tests to enable them to succeed.
ticket 1959
Recover from connection failures in IPAdmin LDAP bind functions and
rather try reconnect in scope of a given timeout instead of giving
up after the first failed connection.
The recovery fixes ipa-ldap-updater on F-16 which always failed
because of a missing dirsrv socket.
https://fedorahosted.org/freeipa/ticket/2175
We use convenience types (classes) in IPA which make working with LDAP
easier and more robust. It would be really nice if the basic python-ldap
library understood our utility types and could accept them as parameters
to the basic ldap functions and/or the basic ldap functions returned our
utility types.
Normally such a requirement would trivially be handled in an object-
oriented language (which Python is) by subclassing to extend and modify
the functionality. For some reason we didn't do this with the python-ldap
classes.
python-ldap objects are primarily used in two different places in our
code, ipaserver.ipaldap.py for the IPAdmin class and in
ipaserver/plugins/ldap2.py for the ldap2 class's .conn member.
In IPAdmin we use a IPA utility class called Entry to make it easier to
use the results returned by LDAP. The IPAdmin class is derived from
python-ldap.SimpleLDAPObject. But for some reason when we added the
support for the use of the Entry class in SimpleLDAPObject we didn't
subclass SimpleLDAPObject and extend it for use with the Entry class as
would be the normal expected methodology in an object-oriented language,
rather we used an obscure feature of the Python language to override all
methods of the SimpleLDAPObject class by wrapping those class methods in
another function call. The reason why this isn't a good approach is:
* It violates object-oriented methodology.
* Other classes cannot be derived and inherit the customization (because
the method wrapping occurs in a class instance, not within the class
type).
* It's non-obvious and obscure
* It's inefficient.
Here is a summary of what the code was doing:
It iterated over every member of the SimpleLDAPObject class and if it was
callable it wrapped the method. The wrapper function tested the name of
the method being wrapped, if it was one of a handful of methods we wanted
to customize we modified a parameter and called the original method. If
the method wasn't of interest to use we still wrapped the method.
It was inefficient because every non-customized method (the majority)
executed a function call for the wrapper, the wrapper during run-time used
logic to determine if the method was being overridden and then called the
original method. So every call to ldap was doing extra function calls and
logic processing which for the majority of cases produced nothing useful
(and was non-obvious from brief code reading some methods were being
overridden).
Object-orientated languages have support built in for calling the right
method for a given class object that do not involve extra function call
overhead to realize customized class behaviour. Also when programmers look
for customized class behaviour they look for derived classes. They might
also want to utilize the customized class as the base class for their use.
Also the wrapper logic was fragile, it did things like: if the method name
begins with "add" I'll unconditionally modify the first and second
argument. It would be some much cleaner if the "add", "add_s", etc.
methods were overridden in a subclass where the logic could be seen and
where it would apply to only the explicit functions and parameters being
overridden.
Also we would really benefit if there were classes which could be used as
a base class which had specific ldap customization.
At the moment our ldap customization needs are:
1) Support DN objects being passed to ldap operations
2) Support Entry & Entity objects being passed into and returned from
ldap operations.
We want to subclass the ldap SimpleLDAPObject class, that is the base
ldap class with all the ldap methods we're using. IPASimpleLDAPObject
class would subclass SimpleLDAPObject class which knows about DN
objects (and possilby other IPA specific types that are universally
used in IPA). Then IPAEntrySimpleLDAPObject would subclass
IPASimpleLDAPObject which knows about Entry objects.
The reason for the suggested class hierarchy is because DN objects will be
used whenever we talk to LDAP (in the future we may want to add other IPA
specific classes which will always be used). We don't add Entry support to
the the IPASimpleLDAPObject class because Entry objects are (currently)
only used in IPAdmin.
What this patch does is:
* Introduce IPASimpleLDAPObject derived from
SimpleLDAPObject. IPASimpleLDAPObject is DN object aware.
* Introduce IPAEntryLDAPObject derived from
IPASimpleLDAPObject. IPAEntryLDAPObject is Entry object aware.
* Derive IPAdmin from IPAEntryLDAPObject and remove the funky method
wrapping from IPAdmin.
* Code which called add_s() with an Entry or Entity object now calls
addEntry(). addEntry() always existed, it just wasn't always
used. add_s() had been modified to accept Entry or Entity object
(why didn't we just call addEntry()?). The add*() ldap routine in
IPAEntryLDAPObject have been subclassed to accept Entry and Entity
objects, but that should proably be removed in the future and just
use addEntry().
* Replace the call to ldap.initialize() in ldap2.create_connection()
with a class constructor for IPASimpleLDAPObject. The
ldap.initialize() is a convenience function in python-ldap, but it
always returns a SimpleLDAPObject created via the SimpleLDAPObject
constructor, thus ldap.initialize() did not allow subclassing, yet
has no particular ease-of-use advantage thus we better off using the
obvious class constructor mechanism.
* Fix the use of _handle_errors(), it's not necessary to construct an
empty dict to pass to it.
If we follow the standard class derivation pattern for ldap we can make us
of our own ldap utilities in a far easier, cleaner and more efficient
manner.
The IPAdmin class in ipaserver/ipaldap.py has methods with anonymous
undefined parameter lists.
For example:
def getList(self,*args):
In Python syntax this means you can call getList with any positional
parameter list you want.
This is bad because:
1) It's not true, *args gets passed to an ldap function with a well
defined parameter list, so you really do have to call it with a
defined parameter list. *args will let you pass anything, but once it
gets passed to the ldap function it will blow up if the parameters do
not match (what parameters are those you're wondering? see item 2).
2) The programmer does not know what the valid parameters are unless
they are defined in the formal parameter list.
3) Without a formal parameter list automatic documentation generators
cannot produce API documentation (see item 2)
4) The Python interpreter cannot validate the parameters being passed
because there is no formal parameter list. Note, Python does not
validate the type of parameters, but it does validate the correct
number of postitional parameters are passed and only defined keyword
parameters are passed. Bypassing the language support facilities leads
to programming errors.
5) Without a formal parameter list program checkers such as pylint
cannot validate the program which leads to progamming errors.
6) Without a formal parameter list which includes default keyword
parameters it's not possible to use keyword arguments nor to know what
their default values are (see item 2). One is forced to pass a keyword
argument as a positional argument, plus you must then pass every
keyword argument between the end of the positional argument list and
keyword arg of interest even of the other keyword arguments are not of
interest. This also demands you know what the default value of the
intermediate keyword arguments are (see item 2) and hope they don't
change.
Also the *args anonymous tuple get passed into the error handling code
so it can report what the called values were. But because the tuple is
anonymous the error handler cannot not describe what it was passed. In
addition the error handling code makes assumptions about the possible
contents of the anonymous tuple based on current practice instead of
actual defined values. Things like "if the number of items in the
tuple is 2 or less then the first tuple item must be a dn
(Distinguished Name)" or "if the number of items in the tuple is
greater than 2 then the 3rd item must be an ldap search filter". These
are constructs which are not robust and will fail at some point in the
future.
This patch also fixes the use of IPAdmin.addEntry(). It was sometimes
being called with (dn, modlist), sometimes a Entry object, or
sometimes a Entity object. Now it's always called with either a Entry
or Entity object and IPAdmin.addEntry() validates the type of the
parameter passed.
The bug to fix updates, BZ 741744, isn't working. For the short
term add the attributes we want to update to the REPLACE
whitelist so rather than using an ADD and DEL operation it will
use a REPLACE.
https://fedorahosted.org/freeipa/ticket/1888
https://fedorahosted.org/freeipa/ticket/1259
Python code will see nsaccountlock as bool. JavaScript code will also see it as bool.
This allows native boolean operations with the lock field. Passes both CLI and WebUI tests.
Enforce that the remote hostname matches the remote SSL server certificate
when 389-ds operates as an SSL client.
Also add an update file to turn this off for existing installations.
This also changes the way the ldapupdater modlist is generated to be more
like the framework. Single-value attributes are done as replacements
and there is a list of force-replacement attributes.
ticket 1069
This was causing a replica DS instance to crash if the task was not
completed when we attempted a shutdown to do a restart.
In replication.py we were restarting the DS instance without waiting
for the ports to become available.
It is unlikely that the dn of the memberof task will change but just in
case I noted it in the two places it is referenced.
ticket 1188
Looking at the schema in 60basev2.ldif there were many attributes that did
not have an ORDERING matching rule specified correctly. There were also a
number of attributeTypes that should have been just SUP
distinguishedName that had a combination of SUP, SYNTAX, ORDERING, etc.
This requires 389-ds-base-1.2.8.0-1+
ticket 1153
By calling directly sasl_interactive_bind_s() we were not calling __lateinit()
This in turn resulted in some variables like dbdir not to be set on the
IPAadmin object.
Keep all bind types in the same place so the same common sbind steps can be
performed in each case.
Related to: https://fedorahosted.org/freeipa/ticket/1022
The changes include:
* Change license blobs in source files to mention GPLv3+ not GPLv2 only
* Add GPLv3+ license text
* Package COPYING not LICENSE as the license blobs (even the old ones)
mention COPYING specifically, it is also more common, I think
https://fedorahosted.org/freeipa/ticket/239
This disables all but the ldapi listener in DS so it will be quiet when
we perform our upgrades. It is expected that any other clients that
also use ldapi will be shut down by other already (krb5 and dns).
Add ldapi as an option in ipaldap and add the beginning of pure offline
support (e.g. direct editing of LDIF files).
Once this is committed we can start the process of renaming errors2 as errors.
I thought that combinig this into one commit would be more difficult to
review.
John Dennis
Rob Crittenden
Martin Kosek
Petr Viktorin
Alexander Bokovoy
Jan Cholasta
Simo Sorce
Jakub Hrozek
Pavel Zuna
Jason Gerard DeRose