IntenseWebs/freeipa
3
0
Fork 0
mirror of https://salsa.debian.org/freeipa-team/freeipa.git synced 2024-12-28 09:51:19 -06:00
Code Issues Packages Projects Releases Wiki Activity
2b3be21b25
freeipa/ipalib/plugins/migration.py

936 lines
38 KiB
Python
Raw Normal View History

Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
# Authors:
# Pavel Zuna <pzuna@redhat.com>
#
# Copyright (C) 2009 Red Hat
# see file 'COPYING' for use and warranty information
#
Change FreeIPA license to GPLv3+ The changes include: * Change license blobs in source files to mention GPLv3+ not GPLv2 only * Add GPLv3+ license text * Package COPYING not LICENSE as the license blobs (even the old ones) mention COPYING specifically, it is also more common, I think https://fedorahosted.org/freeipa/ticket/239
2010-12-09 06:59:11 -06:00
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
Change FreeIPA license to GPLv3+ The changes include: * Change license blobs in source files to mention GPLv3+ not GPLv2 only * Add GPLv3+ license text * Package COPYING not LICENSE as the license blobs (even the old ones) mention COPYING specifically, it is also more common, I think https://fedorahosted.org/freeipa/ticket/239
2010-12-09 06:59:11 -06:00
# along with this program. If not, see <http://www.gnu.org/licenses/>.
ticket 1669 - improve i18n docstring extraction This patch reverts the use of pygettext for i18n string extraction. It was originally introduced because the help documentation for commands are in the class docstring and module docstring. Docstrings are a Python construct whereby any string which immediately follows a class declaration, function/method declaration or appears first in a module is taken to be the documentation for that object. Python automatically assigns that string to the __doc__ variable associated with the object. Explicitly assigning to the __doc__ variable is equivalent and permitted. We mark strings in the source for i18n translation by embedding them in _() or ngettext(). Specialized extraction tools (e.g. xgettext) scan the source code looking for strings with those markers and extracts the string for inclusion in a translation catalog. It was mistakingly assumed one could not mark for translation Python docstrings. Since some docstrings are vital for our command help system some method had to be devised to extract docstrings for the translation catalog. pygettext has the ability to locate and extract docstrings and it was introduced to acquire the documentation for our commands located in module and class docstrings. However pygettext was too large a hammer for this task, it lacked any fined grained ability to extract only the docstrings we were interested in. In practice it extracted EVERY docstring in each file it was presented with. This caused a large number strings to be extracted for translation which had no reason to be translated, the string might have been internal code documentation never meant to be seen by users. Often the superfluous docstrings were long, complex and likely difficult to translate. This placed an unnecessary burden on our volunteer translators. Instead what is needed is some method to extract only those strings intended for translation. We already have such a mechanism and it is already widely used, namely wrapping strings intended for translation in calls to _() or _negettext(), i.e. marking a string for i18n translation. Thus the solution to the docstring translation problem is to mark the docstrings exactly as we have been doing, it only requires that instead of a bare Python docstring we instead assign the marked string to the __doc__ variable. Using the hypothetical class foo as an example. class foo(Command): ''' The foo command takes out the garbage. ''' Would become: class foo(Command): __doc__ = _('The foo command takes out the garbage.') But which docstrings need to be marked for translation? The makeapi tool knows how to iterate over every command in our public API. It was extended to validate every command's documentation and report if any documentation is missing or not marked for translation. That information was then used to identify each docstring in the code which needed to be transformed. In summary what this patch does is: * Remove the use of pygettext (modification to install/po/Makefile.in) * Replace every docstring with an explicit assignment to __doc__ where the rhs of the assignment is an i18n marking function. * Single line docstrings appearing in multi-line string literals (e.g. ''' or """) were replaced with single line string literals because the multi-line literals were introducing unnecessary whitespace and newlines in the string extracted for translation. For example: ''' The foo command takes out the garbage. ''' Would appear in the translation catalog as: "\n The foo command takes out the garbage.\n " The superfluous whitespace and newlines are confusing to translators and requires us to strip leading and trailing whitespace from the translation at run time. * Import statements were moved from below the docstring to above it. This was necessary because the i18n markers are imported functions and must be available before the the doc is parsed. Technically only the import of the i18n markers had to appear before the doc but stylistically it's better to keep all the imports together. * It was observed during the docstring editing process that the command documentation was inconsistent with respect to the use of periods to terminate a sentence. Some doc had a trailing period, others didn't. Consistency was enforced by adding a period to end of every docstring if one was missing.
2011-08-24 21:48:30 -05:00
import re
from ipalib import api, errors, output
migrate-ds added --ca-cert-file=FILE option FILE is used to specify CA certificate for DS connection when TLS is required (ldaps://...). Ticket: https://fedorahosted.org/freeipa/ticket/3243
2013-11-21 10:01:36 -06:00
from ipalib import Command, Password, Str, Flag, StrEnum, DNParam, File
ticket 1669 - improve i18n docstring extraction This patch reverts the use of pygettext for i18n string extraction. It was originally introduced because the help documentation for commands are in the class docstring and module docstring. Docstrings are a Python construct whereby any string which immediately follows a class declaration, function/method declaration or appears first in a module is taken to be the documentation for that object. Python automatically assigns that string to the __doc__ variable associated with the object. Explicitly assigning to the __doc__ variable is equivalent and permitted. We mark strings in the source for i18n translation by embedding them in _() or ngettext(). Specialized extraction tools (e.g. xgettext) scan the source code looking for strings with those markers and extracts the string for inclusion in a translation catalog. It was mistakingly assumed one could not mark for translation Python docstrings. Since some docstrings are vital for our command help system some method had to be devised to extract docstrings for the translation catalog. pygettext has the ability to locate and extract docstrings and it was introduced to acquire the documentation for our commands located in module and class docstrings. However pygettext was too large a hammer for this task, it lacked any fined grained ability to extract only the docstrings we were interested in. In practice it extracted EVERY docstring in each file it was presented with. This caused a large number strings to be extracted for translation which had no reason to be translated, the string might have been internal code documentation never meant to be seen by users. Often the superfluous docstrings were long, complex and likely difficult to translate. This placed an unnecessary burden on our volunteer translators. Instead what is needed is some method to extract only those strings intended for translation. We already have such a mechanism and it is already widely used, namely wrapping strings intended for translation in calls to _() or _negettext(), i.e. marking a string for i18n translation. Thus the solution to the docstring translation problem is to mark the docstrings exactly as we have been doing, it only requires that instead of a bare Python docstring we instead assign the marked string to the __doc__ variable. Using the hypothetical class foo as an example. class foo(Command): ''' The foo command takes out the garbage. ''' Would become: class foo(Command): __doc__ = _('The foo command takes out the garbage.') But which docstrings need to be marked for translation? The makeapi tool knows how to iterate over every command in our public API. It was extended to validate every command's documentation and report if any documentation is missing or not marked for translation. That information was then used to identify each docstring in the code which needed to be transformed. In summary what this patch does is: * Remove the use of pygettext (modification to install/po/Makefile.in) * Replace every docstring with an explicit assignment to __doc__ where the rhs of the assignment is an i18n marking function. * Single line docstrings appearing in multi-line string literals (e.g. ''' or """) were replaced with single line string literals because the multi-line literals were introducing unnecessary whitespace and newlines in the string extracted for translation. For example: ''' The foo command takes out the garbage. ''' Would appear in the translation catalog as: "\n The foo command takes out the garbage.\n " The superfluous whitespace and newlines are confusing to translators and requires us to strip leading and trailing whitespace from the translation at run time. * Import statements were moved from below the docstring to above it. This was necessary because the i18n markers are imported functions and must be available before the the doc is parsed. Technically only the import of the i18n markers had to appear before the doc but stylistically it's better to keep all the imports together. * It was observed during the docstring editing process that the command documentation was inconsistent with respect to the use of periods to terminate a sentence. Some doc had a trailing period, others didn't. Consistency was enforced by adding a period to end of every docstring if one was missing.
2011-08-24 21:48:30 -05:00
from ipalib.cli import to_cli
Update all remaining plugins to the new Registry API Reviewed-By: Petr Vobornik <pvoborni@redhat.com>
2014-06-10 10:27:51 -05:00
from ipalib.plugable import Registry
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
from ipalib.plugins.user import NO_UPG_MAGIC
ticket 1669 - improve i18n docstring extraction This patch reverts the use of pygettext for i18n string extraction. It was originally introduced because the help documentation for commands are in the class docstring and module docstring. Docstrings are a Python construct whereby any string which immediately follows a class declaration, function/method declaration or appears first in a module is taken to be the documentation for that object. Python automatically assigns that string to the __doc__ variable associated with the object. Explicitly assigning to the __doc__ variable is equivalent and permitted. We mark strings in the source for i18n translation by embedding them in _() or ngettext(). Specialized extraction tools (e.g. xgettext) scan the source code looking for strings with those markers and extracts the string for inclusion in a translation catalog. It was mistakingly assumed one could not mark for translation Python docstrings. Since some docstrings are vital for our command help system some method had to be devised to extract docstrings for the translation catalog. pygettext has the ability to locate and extract docstrings and it was introduced to acquire the documentation for our commands located in module and class docstrings. However pygettext was too large a hammer for this task, it lacked any fined grained ability to extract only the docstrings we were interested in. In practice it extracted EVERY docstring in each file it was presented with. This caused a large number strings to be extracted for translation which had no reason to be translated, the string might have been internal code documentation never meant to be seen by users. Often the superfluous docstrings were long, complex and likely difficult to translate. This placed an unnecessary burden on our volunteer translators. Instead what is needed is some method to extract only those strings intended for translation. We already have such a mechanism and it is already widely used, namely wrapping strings intended for translation in calls to _() or _negettext(), i.e. marking a string for i18n translation. Thus the solution to the docstring translation problem is to mark the docstrings exactly as we have been doing, it only requires that instead of a bare Python docstring we instead assign the marked string to the __doc__ variable. Using the hypothetical class foo as an example. class foo(Command): ''' The foo command takes out the garbage. ''' Would become: class foo(Command): __doc__ = _('The foo command takes out the garbage.') But which docstrings need to be marked for translation? The makeapi tool knows how to iterate over every command in our public API. It was extended to validate every command's documentation and report if any documentation is missing or not marked for translation. That information was then used to identify each docstring in the code which needed to be transformed. In summary what this patch does is: * Remove the use of pygettext (modification to install/po/Makefile.in) * Replace every docstring with an explicit assignment to __doc__ where the rhs of the assignment is an i18n marking function. * Single line docstrings appearing in multi-line string literals (e.g. ''' or """) were replaced with single line string literals because the multi-line literals were introducing unnecessary whitespace and newlines in the string extracted for translation. For example: ''' The foo command takes out the garbage. ''' Would appear in the translation catalog as: "\n The foo command takes out the garbage.\n " The superfluous whitespace and newlines are confusing to translators and requires us to strip leading and trailing whitespace from the translation at run time. * Import statements were moved from below the docstring to above it. This was necessary because the i18n markers are imported functions and must be available before the the doc is parsed. Technically only the import of the i18n markers had to appear before the doc but stylistically it's better to keep all the imports together. * It was observed during the docstring editing process that the command documentation was inconsistent with respect to the use of periods to terminate a sentence. Some doc had a trailing period, others didn't. Consistency was enforced by adding a period to end of every docstring if one was missing.
2011-08-24 21:48:30 -05:00
if api.env.in_server and api.env.context in ['lite', 'server']:
try:
from ipaserver.plugins.ldap2 import ldap2
except StandardError, e:
raise e
from ipalib import _
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
from ipapython.dn import DN
migrate-ds added --ca-cert-file=FILE option FILE is used to specify CA certificate for DS connection when TLS is required (ldaps://...). Ticket: https://fedorahosted.org/freeipa/ticket/3243
2013-11-21 10:01:36 -06:00
from ipapython.ipautil import write_tmp_file
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
import datetime
ipaplatform: Move all filesystem paths to ipaplatform.paths module https://fedorahosted.org/freeipa/ticket/4052 Reviewed-By: Petr Viktorin <pviktori@redhat.com>
2014-05-29 07:47:17 -05:00
from ipaplatform.paths import paths
ticket 1669 - improve i18n docstring extraction This patch reverts the use of pygettext for i18n string extraction. It was originally introduced because the help documentation for commands are in the class docstring and module docstring. Docstrings are a Python construct whereby any string which immediately follows a class declaration, function/method declaration or appears first in a module is taken to be the documentation for that object. Python automatically assigns that string to the __doc__ variable associated with the object. Explicitly assigning to the __doc__ variable is equivalent and permitted. We mark strings in the source for i18n translation by embedding them in _() or ngettext(). Specialized extraction tools (e.g. xgettext) scan the source code looking for strings with those markers and extracts the string for inclusion in a translation catalog. It was mistakingly assumed one could not mark for translation Python docstrings. Since some docstrings are vital for our command help system some method had to be devised to extract docstrings for the translation catalog. pygettext has the ability to locate and extract docstrings and it was introduced to acquire the documentation for our commands located in module and class docstrings. However pygettext was too large a hammer for this task, it lacked any fined grained ability to extract only the docstrings we were interested in. In practice it extracted EVERY docstring in each file it was presented with. This caused a large number strings to be extracted for translation which had no reason to be translated, the string might have been internal code documentation never meant to be seen by users. Often the superfluous docstrings were long, complex and likely difficult to translate. This placed an unnecessary burden on our volunteer translators. Instead what is needed is some method to extract only those strings intended for translation. We already have such a mechanism and it is already widely used, namely wrapping strings intended for translation in calls to _() or _negettext(), i.e. marking a string for i18n translation. Thus the solution to the docstring translation problem is to mark the docstrings exactly as we have been doing, it only requires that instead of a bare Python docstring we instead assign the marked string to the __doc__ variable. Using the hypothetical class foo as an example. class foo(Command): ''' The foo command takes out the garbage. ''' Would become: class foo(Command): __doc__ = _('The foo command takes out the garbage.') But which docstrings need to be marked for translation? The makeapi tool knows how to iterate over every command in our public API. It was extended to validate every command's documentation and report if any documentation is missing or not marked for translation. That information was then used to identify each docstring in the code which needed to be transformed. In summary what this patch does is: * Remove the use of pygettext (modification to install/po/Makefile.in) * Replace every docstring with an explicit assignment to __doc__ where the rhs of the assignment is an i18n marking function. * Single line docstrings appearing in multi-line string literals (e.g. ''' or """) were replaced with single line string literals because the multi-line literals were introducing unnecessary whitespace and newlines in the string extracted for translation. For example: ''' The foo command takes out the garbage. ''' Would appear in the translation catalog as: "\n The foo command takes out the garbage.\n " The superfluous whitespace and newlines are confusing to translators and requires us to strip leading and trailing whitespace from the translation at run time. * Import statements were moved from below the docstring to above it. This was necessary because the i18n markers are imported functions and must be available before the the doc is parsed. Technically only the import of the i18n markers had to appear before the doc but stylistically it's better to keep all the imports together. * It was observed during the docstring editing process that the command documentation was inconsistent with respect to the use of periods to terminate a sentence. Some doc had a trailing period, others didn't. Consistency was enforced by adding a period to end of every docstring if one was missing.
2011-08-24 21:48:30 -05:00
__doc__ = _("""
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
Migration to IPA
Add documentation to the migrate-ds command. ticket 539
2010-12-07 10:47:21 -06:00
Migrate users and groups from an LDAP server to IPA.
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
Add documentation to the migrate-ds command. ticket 539
2010-12-07 10:47:21 -06:00
This performs an LDAP query against the remote server searching for
users and groups in a container. In order to migrate passwords you need
to bind as a user that can read the userPassword attribute on the remote
server. This is generally restricted to high-level admins such as
cn=Directory Manager in 389-ds (this is the default bind user).
The default user container is ou=People.
The default group container is ou=Groups.
Users and groups that already exist on the IPA server are skipped.
Two LDAP schemas define how group members are stored: RFC2307 and
RFC2307bis. RFC2307bis uses member and uniquemember to specify group
members, RFC2307 uses memberUid. The default schema is RFC2307bis.
Migration warning when compat enabled Added check into migration plugin to warn user when compat is enabled. If compat is enabled, the migration fails and user is warned that he must turn the compat off or run the script with (the newly introduced) option '--with-compat'. '--with-compat' is new flag. If it is set, the compat status is ignored. https://fedorahosted.org/freeipa/ticket/2274
2012-03-01 04:41:53 -06:00
The schema compat feature allows IPA to reformat data for systems that
do not support RFC2307bis. It is recommended that this feature is disabled
during migration to reduce system overhead. It can be re-enabled after
migration. To migrate with it enabled use the "--with-compat" option.
Add documentation to the migrate-ds command. ticket 539
2010-12-07 10:47:21 -06:00
Migrated users do not have Kerberos credentials, they have only their
Fix style and grammatical issues in built-in command help. There is a rather large API.txt change but it is only due to changes in the doc string in parameters. ticket 729
2011-03-04 10:08:54 -06:00
LDAP password. To complete the migration process, users need to go
Add documentation to the migrate-ds command. ticket 539
2010-12-07 10:47:21 -06:00
to http://ipa.example.com/ipa/migration and authenticate using their
LDAP password in order to generate their Kerberos credentials.
Fix style and grammatical issues in built-in command help. There is a rather large API.txt change but it is only due to changes in the doc string in parameters. ticket 729
2011-03-04 10:08:54 -06:00
Migration is disabled by default. Use the command ipa config-mod to
enable it:
Add documentation to the migrate-ds command. ticket 539
2010-12-07 10:47:21 -06:00
ipa config-mod --enable-migration=TRUE
Add support defaultNamingContext and add --basedn to migrate-ds There are two sides to this, the server and client side. On the server side we attempt to add a defaultNamingContext on already installed servers. This will fail on older 389-ds instances but the failure is not fatal. New installations on versions of 389-ds that support this attribute will have it already defined. On the client side we need to look for both defaultNamingContext and namingContexts. We still need to check that the defaultNamingContext is an IPA server (info=IPAV2). The migration change also takes advantage of this and adds a new option which allows one to provide a basedn to use instead of trying to detect it. https://fedorahosted.org/freeipa/ticket/1919 https://fedorahosted.org/freeipa/ticket/2314
2012-01-30 15:29:32 -06:00
If a base DN is not provided with --basedn then IPA will use either
the value of defaultNamingContext if it is set or the first value
in namingContexts set in the root of the remote LDAP server.
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
Users are added as members to the default user group. This can be a
time-intensive task so during migration this is done in a batch
mode for every 100 users. As a result there will be a window in which
users will be added to IPA but will not be members of the default
user group.
Add documentation to the migrate-ds command. ticket 539
2010-12-07 10:47:21 -06:00
EXAMPLES:
Fix typo in migration documentation
2010-12-13 08:48:16 -06:00
The simplest migration, accepting all defaults:
Add documentation to the migrate-ds command. ticket 539
2010-12-07 10:47:21 -06:00
ipa migrate-ds ldap://ds.example.com:389
Improve migration help Improve migration help topic so that it easier understandable: - Add missing list of Topic commands - Add one more example to demonstrate migration abilities - Add breaks to too long lines to improve readibility https://fedorahosted.org/freeipa/ticket/2174
2012-02-03 02:38:16 -06:00
Specify the user and group container. This can be used to migrate user
and group data from an IPA v1 server:
ipa migrate-ds --user-container='cn=users,cn=accounts' \\
--group-container='cn=groups,cn=accounts' \\
ldap://ds.example.com:389
Improve handling of GIDs when migrating groups Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866
2011-10-03 09:01:01 -05:00
Since IPA v2 server already contain predefined groups that may collide with
Improve migration help Improve migration help topic so that it easier understandable: - Add missing list of Topic commands - Add one more example to demonstrate migration abilities - Add breaks to too long lines to improve readibility https://fedorahosted.org/freeipa/ticket/2174
2012-02-03 02:38:16 -06:00
groups in migrated (IPA v1) server (for example admins, ipausers), users
having colliding group as their primary group may happen to belong to
an unknown group on new IPA v2 server.
Improve handling of GIDs when migrating groups Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866
2011-10-03 09:01:01 -05:00
Use --group-overwrite-gid option to overwrite GID of already existing groups
to prevent this issue:
Improve migration help Improve migration help topic so that it easier understandable: - Add missing list of Topic commands - Add one more example to demonstrate migration abilities - Add breaks to too long lines to improve readibility https://fedorahosted.org/freeipa/ticket/2174
2012-02-03 02:38:16 -06:00
ipa migrate-ds --group-overwrite-gid \\
--user-container='cn=users,cn=accounts' \\
--group-container='cn=groups,cn=accounts' \\
ldap://ds.example.com:389
Migrated users or groups may have object class and accompanied attributes
unknown to the IPA v2 server. These object classes and attributes may be
left out of the migration process:
ipa migrate-ds --user-container='cn=users,cn=accounts' \\
--group-container='cn=groups,cn=accounts' \\
--user-ignore-objectclass=radiusprofile \\
--user-ignore-attribute=radiusgroupname \\
ldap://ds.example.com:389
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
LOGGING
Migration will log warnings and errors to the Apache error log. This
file should be evaluated post-migration to correct or investigate any
issues that were discovered.
For every 100 users migrated an info-level message will be displayed to
give the current progress and duration to make it possible to track
the progress of migration.
If the log level is debug, either by setting debug = True in
/etc/ipa/default.conf or /etc/ipa/server.conf, then an entry will be printed
for each user added plus a summary when the default user group is
updated.
ticket 1669 - improve i18n docstring extraction This patch reverts the use of pygettext for i18n string extraction. It was originally introduced because the help documentation for commands are in the class docstring and module docstring. Docstrings are a Python construct whereby any string which immediately follows a class declaration, function/method declaration or appears first in a module is taken to be the documentation for that object. Python automatically assigns that string to the __doc__ variable associated with the object. Explicitly assigning to the __doc__ variable is equivalent and permitted. We mark strings in the source for i18n translation by embedding them in _() or ngettext(). Specialized extraction tools (e.g. xgettext) scan the source code looking for strings with those markers and extracts the string for inclusion in a translation catalog. It was mistakingly assumed one could not mark for translation Python docstrings. Since some docstrings are vital for our command help system some method had to be devised to extract docstrings for the translation catalog. pygettext has the ability to locate and extract docstrings and it was introduced to acquire the documentation for our commands located in module and class docstrings. However pygettext was too large a hammer for this task, it lacked any fined grained ability to extract only the docstrings we were interested in. In practice it extracted EVERY docstring in each file it was presented with. This caused a large number strings to be extracted for translation which had no reason to be translated, the string might have been internal code documentation never meant to be seen by users. Often the superfluous docstrings were long, complex and likely difficult to translate. This placed an unnecessary burden on our volunteer translators. Instead what is needed is some method to extract only those strings intended for translation. We already have such a mechanism and it is already widely used, namely wrapping strings intended for translation in calls to _() or _negettext(), i.e. marking a string for i18n translation. Thus the solution to the docstring translation problem is to mark the docstrings exactly as we have been doing, it only requires that instead of a bare Python docstring we instead assign the marked string to the __doc__ variable. Using the hypothetical class foo as an example. class foo(Command): ''' The foo command takes out the garbage. ''' Would become: class foo(Command): __doc__ = _('The foo command takes out the garbage.') But which docstrings need to be marked for translation? The makeapi tool knows how to iterate over every command in our public API. It was extended to validate every command's documentation and report if any documentation is missing or not marked for translation. That information was then used to identify each docstring in the code which needed to be transformed. In summary what this patch does is: * Remove the use of pygettext (modification to install/po/Makefile.in) * Replace every docstring with an explicit assignment to __doc__ where the rhs of the assignment is an i18n marking function. * Single line docstrings appearing in multi-line string literals (e.g. ''' or """) were replaced with single line string literals because the multi-line literals were introducing unnecessary whitespace and newlines in the string extracted for translation. For example: ''' The foo command takes out the garbage. ''' Would appear in the translation catalog as: "\n The foo command takes out the garbage.\n " The superfluous whitespace and newlines are confusing to translators and requires us to strip leading and trailing whitespace from the translation at run time. * Import statements were moved from below the docstring to above it. This was necessary because the i18n markers are imported functions and must be available before the the doc is parsed. Technically only the import of the i18n markers had to appear before the doc but stylistically it's better to keep all the imports together. * It was observed during the docstring editing process that the command documentation was inconsistent with respect to the use of periods to terminate a sentence. Some doc had a trailing period, others didn't. Consistency was enforced by adding a period to end of every docstring if one was missing.
2011-08-24 21:48:30 -05:00
""")
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
Update all remaining plugins to the new Registry API Reviewed-By: Petr Vobornik <pvoborni@redhat.com>
2014-06-10 10:27:51 -05:00
register = Registry()
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
# USER MIGRATION CALLBACKS AND VARS
localize doc strings A number of doc strings were not localized, wrap them in _(). Some messages were not localized, wrap them in _() Fix a couple of failing tests: The method name in RPC should not be unicode. The doc attribute must use the .msg attribute for comparison. Also clean up imports of _() The import should come from ipalib or ipalib.text, not ugettext from request.
2010-03-05 15:11:21 -06:00
_krb_err_msg = _('Kerberos principal %s already exists. Use \'ipa user-mod\' to set it manually.')
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
_krb_failed_msg = _('Unable to determine if Kerberos principal %s already exists. Use \'ipa user-mod\' to set it manually.')
localize doc strings A number of doc strings were not localized, wrap them in _(). Some messages were not localized, wrap them in _() Fix a couple of failing tests: The method name in RPC should not be unicode. The doc attribute must use the .msg attribute for comparison. Also clean up imports of _() The import should come from ipalib or ipalib.text, not ugettext from request.
2010-03-05 15:11:21 -06:00
_grp_err_msg = _('Failed to add user to the default group. Use \'ipa group-add-member\' to add manually.')
Handle LDAP search references LDAP search operation may return a search reference pointing to an LDAP resource. As the framework does not handle search references, skip these results to prevent result processing failures. Migrate operation crashed when the migrated DS contained search references. Now, it correctly skips these records and prints the failed references to user. https://fedorahosted.org/freeipa/ticket/1209
2011-06-01 11:04:24 -05:00
_ref_err_msg = _('Migration of LDAP search reference is not supported.')
migrate process cannot handle multivalued pkey attribute When group/user is migrated, the attribute used for RDN may be multivalued. Make sure that we pick the value used in the RDN which should be the unique one and not just the first one. https://fedorahosted.org/freeipa/ticket/1892
2011-09-29 04:55:13 -05:00
_dn_err_msg = _('Malformed DN')
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
_supported_schemas = (u'RFC2307bis', u'RFC2307')
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
_compat_dn = DN(('cn', 'Schema Compatibility'), ('cn', 'plugins'), ('cn', 'config'))
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
def _pre_migrate_user(ldap, pkey, dn, entry_attrs, failed, config, ctx, **kwargs):
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
assert isinstance(dn, DN)
Fix migration to work between v2 servers and remove search/size limits. Migration from a v2 server would fail because of our fake memberofindirect attribute. This isn't in any objectclass so would cause entries to fail to migrate. We can safely just remove it. Also remove any limits on time/size when searching for entries on the remote server. Otherwise only the number of entries configured in the local IPA server can be migrated. ticket 1124
2011-04-08 09:19:42 -05:00
attr_blacklist = ['krbprincipalkey','memberofindirect','memberindirect']
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
attr_blacklist.extend(kwargs.get('attr_blacklist', []))
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
ds_ldap = ctx['ds_ldap']
has_upg = ctx['has_upg']
search_bases = kwargs.get('search_bases', None)
valid_gids = kwargs['valid_gids']
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
invalid_gids = kwargs['invalid_gids']
Do not migrate krbPrincipalKey https://fedorahosted.org/freeipa/ticket/455
2010-11-26 08:37:12 -06:00
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
if 'gidnumber' not in entry_attrs:
raise errors.NotFound(reason=_('%(user)s is not a POSIX user') % dict(user=pkey))
else:
# See if the gidNumber at least points to a valid group on the remote
# server.
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
if entry_attrs['gidnumber'][0] in invalid_gids:
api.log.warn('GID number %s of migrated user %s does not point to a known group.' \
% (entry_attrs['gidnumber'][0], pkey))
elif entry_attrs['gidnumber'][0] not in valid_gids:
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
try:
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
remote_entry = ds_ldap.find_entry_by_attr(
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
'gidnumber', entry_attrs['gidnumber'][0], 'posixgroup',
[''], search_bases['group']
)
valid_gids.append(entry_attrs['gidnumber'][0])
except errors.NotFound:
Do not fail migration because of duplicate groups When 2 groups in a remote LDAP server share the same GID number, the migration may fail entirely with incomprehensible message. This should not be taken as unrecoverable error - GID number check is just a sanity check, a warning is enough. This patch also makes sure that GID check warnings include a user name to make an investigation easier. https://fedorahosted.org/freeipa/ticket/2644
2012-04-17 13:39:32 -05:00
api.log.warn('GID number %s of migrated user %s does not point to a known group.' \
% (entry_attrs['gidnumber'][0], pkey))
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
invalid_gids.append(entry_attrs['gidnumber'][0])
Do not fail migration because of duplicate groups When 2 groups in a remote LDAP server share the same GID number, the migration may fail entirely with incomprehensible message. This should not be taken as unrecoverable error - GID number check is just a sanity check, a warning is enough. This patch also makes sure that GID check warnings include a user name to make an investigation easier. https://fedorahosted.org/freeipa/ticket/2644
2012-04-17 13:39:32 -05:00
except errors.SingleMatchExpected, e:
# GID number matched more groups, this should not happen
api.log.warn('GID number %s of migrated user %s should match 1 group, but it matched %d groups' \
% (entry_attrs['gidnumber'][0], pkey, e.found))
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
except errors.LimitsExceeded, e:
api.log.warn('Search limit exceeded searching for GID %s' % entry_attrs['gidnumber'][0])
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
# We don't want to create a UPG so set the magic value in description
# to let the DS plugin know.
entry_attrs.setdefault('description', [])
entry_attrs['description'].append(NO_UPG_MAGIC)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
# fill in required attributes by IPA
UUIDs: remove uuid python plugin and let DS always autogenerate merge in remove uuid
2010-10-26 09:26:06 -05:00
entry_attrs['ipauniqueid'] = 'autogenerate'
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
if 'homedirectory' not in entry_attrs:
ipaplatform: Move all filesystem paths to ipaplatform.paths module https://fedorahosted.org/freeipa/ticket/4052 Reviewed-By: Petr Viktorin <pviktori@redhat.com>
2014-05-29 07:47:17 -05:00
homes_root = config.get('ipahomesrootdir', (paths.HOME_DIR, ))[0]
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
home_dir = '%s/%s' % (homes_root, pkey)
home_dir = home_dir.replace('//', '/').rstrip('/')
entry_attrs['homedirectory'] = home_dir
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
if 'loginshell' not in entry_attrs:
ipaplatform: Move all filesystem paths to ipaplatform.paths module https://fedorahosted.org/freeipa/ticket/4052 Reviewed-By: Petr Viktorin <pviktori@redhat.com>
2014-05-29 07:47:17 -05:00
default_shell = config.get('ipadefaultloginshell', [paths.SH])[0]
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
entry_attrs.setdefault('loginshell', default_shell)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
Fix migration to work between v2 servers and remove search/size limits. Migration from a v2 server would fail because of our fake memberofindirect attribute. This isn't in any objectclass so would cause entries to fail to migrate. We can safely just remove it. Also remove any limits on time/size when searching for entries on the remote server. Otherwise only the number of entries configured in the local IPA server can be migrated. ticket 1124
2011-04-08 09:19:42 -05:00
# do not migrate all attributes
Do not migrate krbPrincipalKey https://fedorahosted.org/freeipa/ticket/455
2010-11-26 08:37:12 -06:00
for attr in entry_attrs.keys():
if attr in attr_blacklist:
del entry_attrs[attr]
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
# do not migrate all object classes
if 'objectclass' in entry_attrs:
for object_class in kwargs.get('oc_blacklist', []):
try:
entry_attrs['objectclass'].remove(object_class)
except ValueError: # object class not present
pass
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
# generate a principal name and check if it isn't already taken
Make error message in migration plugin unicode.
2010-02-17 11:32:58 -06:00
principal = u'%s@%s' % (pkey, api.env.realm)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
try:
ldap.find_entry_by_attr(
Use valid LDAP search base in migration plugin One find_entry_by_attr call did not set a search base leading to LDAP search call with zero search base. This leads to false negative results from LDAP.
2013-07-26 06:39:42 -05:00
'krbprincipalname', principal, 'krbprincipalaux', [''],
DN(api.env.container_user, api.env.basedn)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
)
except errors.NotFound:
entry_attrs['krbprincipalname'] = principal
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
except errors.LimitsExceeded:
failed[pkey] = unicode(_krb_failed_msg % principal)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
else:
Handle LDAP search references LDAP search operation may return a search reference pointing to an LDAP resource. As the framework does not handle search references, skip these results to prevent result processing failures. Migrate operation crashed when the migrated DS contained search references. Now, it correctly skips these records and prints the failed references to user. https://fedorahosted.org/freeipa/ticket/1209
2011-06-01 11:04:24 -05:00
failed[pkey] = unicode(_krb_err_msg % principal)
Translatable Param.label, Param.doc
2010-02-19 10:08:16 -06:00
Fix attributes that contain DNs when migrating. Some attributes, like secretary and manager, may point to other LDAP entries. We need to fix these during migration. https://fedorahosted.org/freeipa/ticket/2562
2012-03-20 21:37:27 -05:00
# Fix any attributes with DN syntax that point to entries in the old
# tree
for attr in entry_attrs.keys():
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
if ldap.has_dn_syntax(attr):
Fix attributes that contain DNs when migrating. Some attributes, like secretary and manager, may point to other LDAP entries. We need to fix these during migration. https://fedorahosted.org/freeipa/ticket/2562
2012-03-20 21:37:27 -05:00
for ind, value in enumerate(entry_attrs[attr]):
Fix migration for openldap DS openldap server does not store its schema in cn=schema entry, but rather in cn=subschema. Add a fallback to ldap2 plugin to read from this entry when cn=schema is not found. ldap2 plugin uses the schema when doing some of the automatic encoding, like an automatic encoding of DN object. IPA migration plugin DN attribute processing is now also more tolerant when it finds that some DN attribute was not autoencoded. It tries to convert it to DN on its own and report a warning and continue with user processing when the conversion fails instead of crashing with AssertionError and thus abandoning the whole migration run. https://fedorahosted.org/freeipa/ticket/3372
2013-01-30 02:46:02 -06:00
if not isinstance(value, DN):
# value is not DN instance, the automatic encoding may have
# failed due to missing schema or the remote attribute type OID was
# not detected as DN type. Try to work this around
api.log.debug('%s: value %s of type %s in attribute %s is not a DN'
', convert it', pkey, value, type(value), attr)
try:
value = DN(value)
except ValueError, e:
api.log.warn('%s: skipping normalization of value %s of type %s '
'in attribute %s which could not be converted to DN: %s',
pkey, value, type(value), attr, e)
continue
Fix attributes that contain DNs when migrating. Some attributes, like secretary and manager, may point to other LDAP entries. We need to fix these during migration. https://fedorahosted.org/freeipa/ticket/2562
2012-03-20 21:37:27 -05:00
try:
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
remote_entry = ds_ldap.get_entry(value, [api.Object.user.primary_key.name, api.Object.group.primary_key.name])
Fix attributes that contain DNs when migrating. Some attributes, like secretary and manager, may point to other LDAP entries. We need to fix these during migration. https://fedorahosted.org/freeipa/ticket/2562
2012-03-20 21:37:27 -05:00
except errors.NotFound:
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
api.log.warn('%s: attribute %s refers to non-existent entry %s' % (pkey, attr, value))
Fix attributes that contain DNs when migrating. Some attributes, like secretary and manager, may point to other LDAP entries. We need to fix these during migration. https://fedorahosted.org/freeipa/ticket/2562
2012-03-20 21:37:27 -05:00
continue
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
if value.endswith(search_bases['user']):
Fix attributes that contain DNs when migrating. Some attributes, like secretary and manager, may point to other LDAP entries. We need to fix these during migration. https://fedorahosted.org/freeipa/ticket/2562
2012-03-20 21:37:27 -05:00
primary_key = api.Object.user.primary_key.name
container = api.env.container_user
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
elif value.endswith(search_bases['group']):
Fix attributes that contain DNs when migrating. Some attributes, like secretary and manager, may point to other LDAP entries. We need to fix these during migration. https://fedorahosted.org/freeipa/ticket/2562
2012-03-20 21:37:27 -05:00
primary_key = api.Object.group.primary_key.name
container = api.env.container_group
else:
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
api.log.warn('%s: value %s in attribute %s does not belong into any known container' % (pkey, value, attr))
Fix attributes that contain DNs when migrating. Some attributes, like secretary and manager, may point to other LDAP entries. We need to fix these during migration. https://fedorahosted.org/freeipa/ticket/2562
2012-03-20 21:37:27 -05:00
continue
if not remote_entry.get(primary_key):
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
api.log.warn('%s: there is no primary key %s to migrate for %s' % (pkey, primary_key, attr))
Fix attributes that contain DNs when migrating. Some attributes, like secretary and manager, may point to other LDAP entries. We need to fix these during migration. https://fedorahosted.org/freeipa/ticket/2562
2012-03-20 21:37:27 -05:00
continue
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
api.log.debug('converting DN value %s for %s in %s' % (value, attr, dn))
Fix attributes that contain DNs when migrating. Some attributes, like secretary and manager, may point to other LDAP entries. We need to fix these during migration. https://fedorahosted.org/freeipa/ticket/2562
2012-03-20 21:37:27 -05:00
rdnval = remote_entry[primary_key][0].lower()
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
entry_attrs[attr][ind] = DN((primary_key, rdnval), container, api.env.basedn)
Fix attributes that contain DNs when migrating. Some attributes, like secretary and manager, may point to other LDAP entries. We need to fix these during migration. https://fedorahosted.org/freeipa/ticket/2562
2012-03-20 21:37:27 -05:00
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
return dn
def _post_migrate_user(ldap, pkey, dn, entry_attrs, failed, config, ctx):
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
assert isinstance(dn, DN)
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
_update_default_group(ldap, pkey, config, ctx, False)
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
if 'description' in entry_attrs and NO_UPG_MAGIC in entry_attrs['description']:
entry_attrs['description'].remove(NO_UPG_MAGIC)
try:
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
update_attrs = ldap.get_entry(dn, ['description'])
update_attrs['description'] = entry_attrs['description']
ldap.update_entry(update_attrs)
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
except (errors.EmptyModlist, errors.NotFound):
pass
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
def _update_default_group(ldap, pkey, config, ctx, force):
migrate_cnt = ctx['migrate_cnt']
group_dn = ctx['def_group_dn']
# Purposely let this fire when migrate_cnt == 0 so on re-running migration
# it can catch any users migrated but not added to the default group.
if force or migrate_cnt % 100 == 0:
s = datetime.datetime.now()
searchfilter = "(&(objectclass=posixAccount)(!(memberof=%s)))" % group_dn
try:
(result, truncated) = ldap.find_entries(searchfilter,
Migration does not add users to default group When users with missing default group were searched, IPA suffix was not passed so these users were searched in a wrong base DN. Thus, no user was detected and added to default group. https://fedorahosted.org/freeipa/ticket/4141 Reviewed-By: Petr Viktorin <pviktori@redhat.com>
2014-01-27 05:28:12 -06:00
[''], DN(api.env.container_user, api.env.basedn),
scope=ldap.SCOPE_SUBTREE, time_limit = -1)
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
except errors.NotFound:
Migration does not add users to default group When users with missing default group were searched, IPA suffix was not passed so these users were searched in a wrong base DN. Thus, no user was detected and added to default group. https://fedorahosted.org/freeipa/ticket/4141 Reviewed-By: Petr Viktorin <pviktori@redhat.com>
2014-01-27 05:28:12 -06:00
api.log.debug('All users have default group set')
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
return
new_members = []
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
group_entry_attrs = ldap.get_entry(group_dn, ['member'])
Migration does not add users to default group When users with missing default group were searched, IPA suffix was not passed so these users were searched in a wrong base DN. Thus, no user was detected and added to default group. https://fedorahosted.org/freeipa/ticket/4141 Reviewed-By: Petr Viktorin <pviktori@redhat.com>
2014-01-27 05:28:12 -06:00
existing_members = set(group_entry_attrs.get('member', []))
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
for m in result:
Migration does not add users to default group When users with missing default group were searched, IPA suffix was not passed so these users were searched in a wrong base DN. Thus, no user was detected and added to default group. https://fedorahosted.org/freeipa/ticket/4141 Reviewed-By: Petr Viktorin <pviktori@redhat.com>
2014-01-27 05:28:12 -06:00
if m.dn not in existing_members:
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
new_members.append(m.dn)
Migration does not add users to default group When users with missing default group were searched, IPA suffix was not passed so these users were searched in a wrong base DN. Thus, no user was detected and added to default group. https://fedorahosted.org/freeipa/ticket/4141 Reviewed-By: Petr Viktorin <pviktori@redhat.com>
2014-01-27 05:28:12 -06:00
if new_members:
members = group_entry_attrs.setdefault('member', [])
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
members.extend(new_members)
try:
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
ldap.update_entry(group_entry_attrs)
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
except errors.EmptyModlist:
pass
e = datetime.datetime.now()
d = e - s
mode = " (forced)" if force else ""
Migration does not add users to default group When users with missing default group were searched, IPA suffix was not passed so these users were searched in a wrong base DN. Thus, no user was detected and added to default group. https://fedorahosted.org/freeipa/ticket/4141 Reviewed-By: Petr Viktorin <pviktori@redhat.com>
2014-01-27 05:28:12 -06:00
api.log.debug('Adding %d users to group%s duration %s',
len(new_members), mode, d)
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
# GROUP MIGRATION CALLBACKS AND VARS
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
def _pre_migrate_group(ldap, pkey, dn, entry_attrs, failed, config, ctx, **kwargs):
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
def convert_members_rfc2307bis(member_attr, search_bases, overwrite=False):
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
"""
Convert DNs in member attributes to work in IPA.
"""
new_members = []
entry_attrs.setdefault(member_attr, [])
Translatable Param.label, Param.doc
2010-02-19 10:08:16 -06:00
for m in entry_attrs[member_attr]:
Convert uniqueMember members into DN objects. We were asserting that they should be DN objects but weren't converting them anywhere. https://fedorahosted.org/freeipa/ticket/3339
2013-01-10 14:31:11 -06:00
try:
m = DN(m)
except ValueError, e:
# This should be impossible unless the remote server
# doesn't enforce syntax checking.
api.log.error('Malformed DN %s: %s' % (m, e))
continue
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
try:
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
rdnval = m[0].value
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
except IndexError:
api.log.error('Malformed DN %s has no RDN?' % m)
continue
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
if m.endswith(search_bases['user']):
Convert uniqueMember members into DN objects. We were asserting that they should be DN objects but weren't converting them anywhere. https://fedorahosted.org/freeipa/ticket/3339
2013-01-10 14:31:11 -06:00
api.log.info('migrating %s user %s' % (member_attr, m))
Use full DNs in plugin code.
2013-02-04 02:47:00 -06:00
m = DN((api.Object.user.primary_key.name, rdnval),
api.env.container_user, api.env.basedn)
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
elif m.endswith(search_bases['group']):
Convert uniqueMember members into DN objects. We were asserting that they should be DN objects but weren't converting them anywhere. https://fedorahosted.org/freeipa/ticket/3339
2013-01-10 14:31:11 -06:00
api.log.info('migrating %s group %s' % (member_attr, m))
Use full DNs in plugin code.
2013-02-04 02:47:00 -06:00
m = DN((api.Object.group.primary_key.name, rdnval),
api.env.container_group, api.env.basedn)
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
else:
api.log.error('entry %s does not belong into any known container' % m)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
continue
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
new_members.append(m)
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
del entry_attrs[member_attr]
if overwrite:
entry_attrs['member'] = []
entry_attrs['member'] += new_members
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
def convert_members_rfc2307(member_attr):
"""
Convert usernames in member attributes to work in IPA.
"""
new_members = []
entry_attrs.setdefault(member_attr, [])
for m in entry_attrs[member_attr]:
Use full DNs in plugin code.
2013-02-04 02:47:00 -06:00
memberdn = DN((api.Object.user.primary_key.name, m),
api.env.container_user, api.env.basedn)
Remove support for DN normalization from LDAPClient.
2013-02-04 04:50:58 -06:00
new_members.append(memberdn)
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
entry_attrs['member'] = new_members
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
assert isinstance(dn, DN)
Fix migration to work between v2 servers and remove search/size limits. Migration from a v2 server would fail because of our fake memberofindirect attribute. This isn't in any objectclass so would cause entries to fail to migrate. We can safely just remove it. Also remove any limits on time/size when searching for entries on the remote server. Otherwise only the number of entries configured in the local IPA server can be migrated. ticket 1124
2011-04-08 09:19:42 -05:00
attr_blacklist = ['memberofindirect','memberindirect']
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
attr_blacklist.extend(kwargs.get('attr_blacklist', []))
Fix migration to work between v2 servers and remove search/size limits. Migration from a v2 server would fail because of our fake memberofindirect attribute. This isn't in any objectclass so would cause entries to fail to migrate. We can safely just remove it. Also remove any limits on time/size when searching for entries on the remote server. Otherwise only the number of entries configured in the local IPA server can be migrated. ticket 1124
2011-04-08 09:19:42 -05:00
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
schema = kwargs.get('schema', None)
UUIDs: remove uuid python plugin and let DS always autogenerate merge in remove uuid
2010-10-26 09:26:06 -05:00
entry_attrs['ipauniqueid'] = 'autogenerate'
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
if schema == 'RFC2307bis':
search_bases = kwargs.get('search_bases', None)
if not search_bases:
raise ValueError('Search bases not specified')
convert_members_rfc2307bis('member', search_bases, overwrite=True)
convert_members_rfc2307bis('uniquemember', search_bases)
elif schema == 'RFC2307':
convert_members_rfc2307('memberuid')
else:
raise ValueError('Schema %s not supported' % schema)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
Fix migration to work between v2 servers and remove search/size limits. Migration from a v2 server would fail because of our fake memberofindirect attribute. This isn't in any objectclass so would cause entries to fail to migrate. We can safely just remove it. Also remove any limits on time/size when searching for entries on the remote server. Otherwise only the number of entries configured in the local IPA server can be migrated. ticket 1124
2011-04-08 09:19:42 -05:00
# do not migrate all attributes
for attr in entry_attrs.keys():
if attr in attr_blacklist:
del entry_attrs[attr]
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
# do not migrate all object classes
if 'objectclass' in entry_attrs:
for object_class in kwargs.get('oc_blacklist', []):
try:
entry_attrs['objectclass'].remove(object_class)
except ValueError: # object class not present
pass
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
return dn
Improve handling of GIDs when migrating groups Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866
2011-10-03 09:01:01 -05:00
def _group_exc_callback(ldap, dn, entry_attrs, exc, options):
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
assert isinstance(dn, DN)
Improve handling of GIDs when migrating groups Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866
2011-10-03 09:01:01 -05:00
if isinstance(exc, errors.DuplicateEntry):
if options.get('groupoverwritegid', False) and \
entry_attrs.get('gidnumber') is not None:
try:
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
new_entry_attrs = ldap.get_entry(dn, ['gidnumber'])
new_entry_attrs['gidnumber'] = entry_attrs['gidnumber']
ldap.update_entry(new_entry_attrs)
Improve handling of GIDs when migrating groups Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866
2011-10-03 09:01:01 -05:00
except errors.EmptyModlist:
# no change to the GID
pass
# mark as success
return
elif not options.get('groupoverwritegid', False) and \
entry_attrs.get('gidnumber') is not None:
msg = unicode(exc)
# add information about possibility to overwrite GID
msg = msg + unicode(_('. Check GID of the existing group. ' \
'Use --group-overwrite-gid option to overwrite the GID'))
raise errors.DuplicateEntry(message=msg)
raise exc
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
# DS MIGRATION PLUGIN
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
def construct_filter(template, oc_list):
oc_subfilter = ''.join([ '(objectclass=%s)' % oc for oc in oc_list])
return template % oc_subfilter
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
def validate_ldapuri(ugettext, ldapuri):
m = re.match('^ldaps?://[-\w\.]+(:\d+)?$', ldapuri)
if not m:
Add labels for passwords, fix output of exceptions, fix passwd output. Passwords didn't have internationalizable labels. Exceptions that occured during required input weren't printed as unicode so weren't being translated properly. Don't use output_for_cli() directly in the passwd plugin, use output.Output. ticket 352
2010-12-02 15:31:42 -06:00
err_msg = _('Invalid LDAP URI.')
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
raise errors.ValidationError(name='ldap_uri', error=err_msg)
Update all remaining plugins to the new Registry API Reviewed-By: Petr Vobornik <pvoborni@redhat.com>
2014-06-10 10:27:51 -05:00
@register()
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
class migrate_ds(Command):
ticket 1669 - improve i18n docstring extraction This patch reverts the use of pygettext for i18n string extraction. It was originally introduced because the help documentation for commands are in the class docstring and module docstring. Docstrings are a Python construct whereby any string which immediately follows a class declaration, function/method declaration or appears first in a module is taken to be the documentation for that object. Python automatically assigns that string to the __doc__ variable associated with the object. Explicitly assigning to the __doc__ variable is equivalent and permitted. We mark strings in the source for i18n translation by embedding them in _() or ngettext(). Specialized extraction tools (e.g. xgettext) scan the source code looking for strings with those markers and extracts the string for inclusion in a translation catalog. It was mistakingly assumed one could not mark for translation Python docstrings. Since some docstrings are vital for our command help system some method had to be devised to extract docstrings for the translation catalog. pygettext has the ability to locate and extract docstrings and it was introduced to acquire the documentation for our commands located in module and class docstrings. However pygettext was too large a hammer for this task, it lacked any fined grained ability to extract only the docstrings we were interested in. In practice it extracted EVERY docstring in each file it was presented with. This caused a large number strings to be extracted for translation which had no reason to be translated, the string might have been internal code documentation never meant to be seen by users. Often the superfluous docstrings were long, complex and likely difficult to translate. This placed an unnecessary burden on our volunteer translators. Instead what is needed is some method to extract only those strings intended for translation. We already have such a mechanism and it is already widely used, namely wrapping strings intended for translation in calls to _() or _negettext(), i.e. marking a string for i18n translation. Thus the solution to the docstring translation problem is to mark the docstrings exactly as we have been doing, it only requires that instead of a bare Python docstring we instead assign the marked string to the __doc__ variable. Using the hypothetical class foo as an example. class foo(Command): ''' The foo command takes out the garbage. ''' Would become: class foo(Command): __doc__ = _('The foo command takes out the garbage.') But which docstrings need to be marked for translation? The makeapi tool knows how to iterate over every command in our public API. It was extended to validate every command's documentation and report if any documentation is missing or not marked for translation. That information was then used to identify each docstring in the code which needed to be transformed. In summary what this patch does is: * Remove the use of pygettext (modification to install/po/Makefile.in) * Replace every docstring with an explicit assignment to __doc__ where the rhs of the assignment is an i18n marking function. * Single line docstrings appearing in multi-line string literals (e.g. ''' or """) were replaced with single line string literals because the multi-line literals were introducing unnecessary whitespace and newlines in the string extracted for translation. For example: ''' The foo command takes out the garbage. ''' Would appear in the translation catalog as: "\n The foo command takes out the garbage.\n " The superfluous whitespace and newlines are confusing to translators and requires us to strip leading and trailing whitespace from the translation at run time. * Import statements were moved from below the docstring to above it. This was necessary because the i18n markers are imported functions and must be available before the the doc is parsed. Technically only the import of the i18n markers had to appear before the doc but stylistically it's better to keep all the imports together. * It was observed during the docstring editing process that the command documentation was inconsistent with respect to the use of periods to terminate a sentence. Some doc had a trailing period, others didn't. Consistency was enforced by adding a period to end of every docstring if one was missing.
2011-08-24 21:48:30 -05:00
__doc__ = _('Migrate users and groups from DS to IPA.')
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
migrate_objects = {
# OBJECT_NAME: (search_filter, pre_callback, post_callback)
#
# OBJECT_NAME - is the name of an LDAPObject subclass
# search_filter - is the filter to retrieve objects from DS
# pre_callback - is called for each object just after it was
# retrieved from DS and before being added to IPA
# post_callback - is called for each object after it was added to IPA
Improve handling of GIDs when migrating groups Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866
2011-10-03 09:01:01 -05:00
# exc_callback - is called when adding entry to IPA raises an exception
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
#
# {pre, post}_callback parameters:
# ldap - ldap2 instance connected to IPA
# pkey - primary key value of the object (uid for users, etc.)
# dn - dn of the object as it (will be/is) stored in IPA
# entry_attrs - attributes of the object
# failed - a list of so-far failed objects
# config - IPA config entry attributes
# ctx - object context, used to pass data between callbacks
#
# If pre_callback return value evaluates to False, migration
# of the current object is aborted.
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
'user': {
'filter_template' : '(&(|%s)(uid=*))',
'oc_option' : 'userobjectclass',
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
'oc_blacklist_option' : 'userignoreobjectclass',
'attr_blacklist_option' : 'userignoreattribute',
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
'pre_callback' : _pre_migrate_user,
Improve handling of GIDs when migrating groups Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866
2011-10-03 09:01:01 -05:00
'post_callback' : _post_migrate_user,
'exc_callback' : None
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
},
'group': {
'filter_template' : '(&(|%s)(cn=*))',
'oc_option' : 'groupobjectclass',
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
'oc_blacklist_option' : 'groupignoreobjectclass',
'attr_blacklist_option' : 'groupignoreattribute',
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
'pre_callback' : _pre_migrate_group,
Improve handling of GIDs when migrating groups Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866
2011-10-03 09:01:01 -05:00
'post_callback' : None,
'exc_callback' : _group_exc_callback,
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
},
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
}
migrate_order = ('user', 'group')
takes_args = (
Str('ldapuri', validate_ldapuri,
cli_name='ldap_uri',
Translatable Param.label, Param.doc
2010-02-19 10:08:16 -06:00
label=_('LDAP URI'),
doc=_('LDAP URI of DS server to migrate from'),
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
),
Password('bindpw',
cli_name='password',
Add labels for passwords, fix output of exceptions, fix passwd output. Passwords didn't have internationalizable labels. Exceptions that occured during required input weren't printed as unicode so weren't being translated properly. Don't use output_for_cli() directly in the passwd plugin, use output.Output. ticket 352
2010-12-02 15:31:42 -06:00
label=_('Password'),
Improve handling of GIDs when migrating groups Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866
2011-10-03 09:01:01 -05:00
confirm=False,
localize doc strings A number of doc strings were not localized, wrap them in _(). Some messages were not localized, wrap them in _() Fix a couple of failing tests: The method name in RPC should not be unicode. The doc attribute must use the .msg attribute for comparison. Also clean up imports of _() The import should come from ipalib or ipalib.text, not ugettext from request.
2010-03-05 15:11:21 -06:00
doc=_('bind password'),
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
),
)
takes_options = (
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
DNParam('binddn?',
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
cli_name='bind_dn',
Translatable Param.label, Param.doc
2010-02-19 10:08:16 -06:00
label=_('Bind DN'),
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
default=DN(('cn', 'directory manager')),
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
autofill=True,
),
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
DNParam('usercontainer',
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
cli_name='user_container',
Translatable Param.label, Param.doc
2010-02-19 10:08:16 -06:00
label=_('User container'),
Validate DN & RDN parameters for migrate command Ticket #2555 We were generating a traceback (server error) if a malformed RDN was passed as a parameter to the migrate command. * add parameter validation functions validate_dn_param() and validate_rdn_param() to ipalib.util. Those functions simply invoke the DN or RDN constructor from our dn module passing it the string representation. If the constructor does not throw an error it's valid. * Add the parameter validation function pointers to the Param objects in the migrate command. * Make the usercontainer and groupcontainer parameters required. passing --usercontainer= on the command line will produce ipa: ERROR: 'user_container' is required * Fix _get_search_bases() so if a container dn is empty it it just uses the base dn alone instead of faulting (currently bullet-proofing because now the containers are required). * Update the doc for usercontainer and groupcontainer to reflect the fact they are DN's not RDN's. A RDN can only be one level and it should be possible to have a container more than one RDN removed from the base.
2012-04-16 01:33:26 -05:00
doc=_('DN of container for users in DS relative to base DN'),
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
default=DN(('ou', 'people')),
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
autofill=True,
),
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
DNParam('groupcontainer',
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
cli_name='group_container',
Translatable Param.label, Param.doc
2010-02-19 10:08:16 -06:00
label=_('Group container'),
Validate DN & RDN parameters for migrate command Ticket #2555 We were generating a traceback (server error) if a malformed RDN was passed as a parameter to the migrate command. * add parameter validation functions validate_dn_param() and validate_rdn_param() to ipalib.util. Those functions simply invoke the DN or RDN constructor from our dn module passing it the string representation. If the constructor does not throw an error it's valid. * Add the parameter validation function pointers to the Param objects in the migrate command. * Make the usercontainer and groupcontainer parameters required. passing --usercontainer= on the command line will produce ipa: ERROR: 'user_container' is required * Fix _get_search_bases() so if a container dn is empty it it just uses the base dn alone instead of faulting (currently bullet-proofing because now the containers are required). * Update the doc for usercontainer and groupcontainer to reflect the fact they are DN's not RDN's. A RDN can only be one level and it should be possible to have a container more than one RDN removed from the base.
2012-04-16 01:33:26 -05:00
doc=_('DN of container for groups in DS relative to base DN'),
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
default=DN(('ou', 'groups')),
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
autofill=True,
),
Improve migration NotFound error When no user/group was found, migration plugin reported an ambiguous error about invalid container. But the root cause may be for example in a wrong list of user/group objectclasses. Report both in the error message to avoid user confusion. User/group objectclass attribute is now also marked as required. Without the list of objectclasses, an invalid LDAP search is produced. https://fedorahosted.org/freeipa/ticket/2206
2012-06-04 07:25:41 -05:00
Str('userobjectclass+',
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
cli_name='user_objectclass',
label=_('User object class'),
Update argument docs to reflect dropped CSV support https://fedorahosted.org/freeipa/ticket/3352
2013-02-14 11:01:16 -06:00
doc=_('Objectclasses used to search for user entries in DS'),
Parse comma-separated lists of values in all parameter types. This can be enabled for a specific parameter by setting the "csv" option to True. Remove "List" parameter type and replace all occurences of it with appropriate multi-valued parameter ("Str" in most cases) with csv enabled. Add new parameter type "Any", capable of holding values of any type. This is needed by the "batch" command, as "Str" is not suitable type for the "methods" parameter. ticket 2007
2011-11-21 09:50:27 -06:00
csv=True,
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
default=(u'person',),
autofill=True,
),
Improve migration NotFound error When no user/group was found, migration plugin reported an ambiguous error about invalid container. But the root cause may be for example in a wrong list of user/group objectclasses. Report both in the error message to avoid user confusion. User/group objectclass attribute is now also marked as required. Without the list of objectclasses, an invalid LDAP search is produced. https://fedorahosted.org/freeipa/ticket/2206
2012-06-04 07:25:41 -05:00
Str('groupobjectclass+',
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
cli_name='group_objectclass',
label=_('Group object class'),
Update argument docs to reflect dropped CSV support https://fedorahosted.org/freeipa/ticket/3352
2013-02-14 11:01:16 -06:00
doc=_('Objectclasses used to search for group entries in DS'),
Parse comma-separated lists of values in all parameter types. This can be enabled for a specific parameter by setting the "csv" option to True. Remove "List" parameter type and replace all occurences of it with appropriate multi-valued parameter ("Str" in most cases) with csv enabled. Add new parameter type "Any", capable of holding values of any type. This is needed by the "batch" command, as "Str" is not suitable type for the "methods" parameter. ticket 2007
2011-11-21 09:50:27 -06:00
csv=True,
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
default=(u'groupOfUniqueNames', u'groupOfNames'),
autofill=True,
),
Parse comma-separated lists of values in all parameter types. This can be enabled for a specific parameter by setting the "csv" option to True. Remove "List" parameter type and replace all occurences of it with appropriate multi-valued parameter ("Str" in most cases) with csv enabled. Add new parameter type "Any", capable of holding values of any type. This is needed by the "batch" command, as "Str" is not suitable type for the "methods" parameter. ticket 2007
2011-11-21 09:50:27 -06:00
Str('userignoreobjectclass*',
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
cli_name='user_ignore_objectclass',
label=_('Ignore user object class'),
Update argument docs to reflect dropped CSV support https://fedorahosted.org/freeipa/ticket/3352
2013-02-14 11:01:16 -06:00
doc=_('Objectclasses to be ignored for user entries in DS'),
Parse comma-separated lists of values in all parameter types. This can be enabled for a specific parameter by setting the "csv" option to True. Remove "List" parameter type and replace all occurences of it with appropriate multi-valued parameter ("Str" in most cases) with csv enabled. Add new parameter type "Any", capable of holding values of any type. This is needed by the "batch" command, as "Str" is not suitable type for the "methods" parameter. ticket 2007
2011-11-21 09:50:27 -06:00
csv=True,
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
default=tuple(),
autofill=True,
),
Parse comma-separated lists of values in all parameter types. This can be enabled for a specific parameter by setting the "csv" option to True. Remove "List" parameter type and replace all occurences of it with appropriate multi-valued parameter ("Str" in most cases) with csv enabled. Add new parameter type "Any", capable of holding values of any type. This is needed by the "batch" command, as "Str" is not suitable type for the "methods" parameter. ticket 2007
2011-11-21 09:50:27 -06:00
Str('userignoreattribute*',
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
cli_name='user_ignore_attribute',
label=_('Ignore user attribute'),
Update argument docs to reflect dropped CSV support https://fedorahosted.org/freeipa/ticket/3352
2013-02-14 11:01:16 -06:00
doc=_('Attributes to be ignored for user entries in DS'),
Parse comma-separated lists of values in all parameter types. This can be enabled for a specific parameter by setting the "csv" option to True. Remove "List" parameter type and replace all occurences of it with appropriate multi-valued parameter ("Str" in most cases) with csv enabled. Add new parameter type "Any", capable of holding values of any type. This is needed by the "batch" command, as "Str" is not suitable type for the "methods" parameter. ticket 2007
2011-11-21 09:50:27 -06:00
csv=True,
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
default=tuple(),
autofill=True,
),
Parse comma-separated lists of values in all parameter types. This can be enabled for a specific parameter by setting the "csv" option to True. Remove "List" parameter type and replace all occurences of it with appropriate multi-valued parameter ("Str" in most cases) with csv enabled. Add new parameter type "Any", capable of holding values of any type. This is needed by the "batch" command, as "Str" is not suitable type for the "methods" parameter. ticket 2007
2011-11-21 09:50:27 -06:00
Str('groupignoreobjectclass*',
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
cli_name='group_ignore_objectclass',
label=_('Ignore group object class'),
Update argument docs to reflect dropped CSV support https://fedorahosted.org/freeipa/ticket/3352
2013-02-14 11:01:16 -06:00
doc=_('Objectclasses to be ignored for group entries in DS'),
Parse comma-separated lists of values in all parameter types. This can be enabled for a specific parameter by setting the "csv" option to True. Remove "List" parameter type and replace all occurences of it with appropriate multi-valued parameter ("Str" in most cases) with csv enabled. Add new parameter type "Any", capable of holding values of any type. This is needed by the "batch" command, as "Str" is not suitable type for the "methods" parameter. ticket 2007
2011-11-21 09:50:27 -06:00
csv=True,
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
default=tuple(),
autofill=True,
),
Parse comma-separated lists of values in all parameter types. This can be enabled for a specific parameter by setting the "csv" option to True. Remove "List" parameter type and replace all occurences of it with appropriate multi-valued parameter ("Str" in most cases) with csv enabled. Add new parameter type "Any", capable of holding values of any type. This is needed by the "batch" command, as "Str" is not suitable type for the "methods" parameter. ticket 2007
2011-11-21 09:50:27 -06:00
Str('groupignoreattribute*',
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
cli_name='group_ignore_attribute',
label=_('Ignore group attribute'),
Update argument docs to reflect dropped CSV support https://fedorahosted.org/freeipa/ticket/3352
2013-02-14 11:01:16 -06:00
doc=_('Attributes to be ignored for group entries in DS'),
Parse comma-separated lists of values in all parameter types. This can be enabled for a specific parameter by setting the "csv" option to True. Remove "List" parameter type and replace all occurences of it with appropriate multi-valued parameter ("Str" in most cases) with csv enabled. Add new parameter type "Any", capable of holding values of any type. This is needed by the "batch" command, as "Str" is not suitable type for the "methods" parameter. ticket 2007
2011-11-21 09:50:27 -06:00
csv=True,
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
default=tuple(),
autofill=True,
),
Improve handling of GIDs when migrating groups Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866
2011-10-03 09:01:01 -05:00
Flag('groupoverwritegid',
cli_name='group_overwrite_gid',
label=_('Overwrite GID'),
doc=_('When migrating a group already existing in IPA domain overwrite the '\
'group GID and report as success'),
),
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
StrEnum('schema?',
cli_name='schema',
label=_('LDAP schema'),
doc=_('The schema used on the LDAP server. Supported values are RFC2307 and RFC2307bis. The default is RFC2307bis'),
values=_supported_schemas,
default=_supported_schemas[0],
autofill=True,
),
Handle an empty base_dn and no cn=ipaconfig in the ldap2 backend, fix migration. We lacked good error messages if the user/group container you used doesn't exist. Add a --continue option so things can continue if you use a bad user/group container. This has the side-effect of letting you migrate just users or groups by using a bad container for the one you don't want. Fix a Gettext() error when displaying the migrated password message. ticket 289
2010-09-27 12:50:54 -05:00
Flag('continue?',
Improve handling of GIDs when migrating groups Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866
2011-10-03 09:01:01 -05:00
label=_('Continue'),
Typos in freeIPA messages This patch fixes several reported typos in IPA messages and in comments. Contributors file has been updated + the original author of the patch reporting the typos was added. https://fedorahosted.org/freeipa/ticket/848
2011-02-02 10:15:35 -06:00
doc=_('Continuous operation mode. Errors are reported but the process continues'),
Handle an empty base_dn and no cn=ipaconfig in the ldap2 backend, fix migration. We lacked good error messages if the user/group container you used doesn't exist. Add a --continue option so things can continue if you use a bad user/group container. This has the side-effect of letting you migrate just users or groups by using a bad container for the one you don't want. Fix a Gettext() error when displaying the migrated password message. ticket 289
2010-09-27 12:50:54 -05:00
default=False,
),
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
DNParam('basedn?',
Add support defaultNamingContext and add --basedn to migrate-ds There are two sides to this, the server and client side. On the server side we attempt to add a defaultNamingContext on already installed servers. This will fail on older 389-ds instances but the failure is not fatal. New installations on versions of 389-ds that support this attribute will have it already defined. On the client side we need to look for both defaultNamingContext and namingContexts. We still need to check that the defaultNamingContext is an IPA server (info=IPAV2). The migration change also takes advantage of this and adds a new option which allows one to provide a basedn to use instead of trying to detect it. https://fedorahosted.org/freeipa/ticket/1919 https://fedorahosted.org/freeipa/ticket/2314
2012-01-30 15:29:32 -06:00
cli_name='base_dn',
label=_('Base DN'),
doc=_('Base DN on remote LDAP server'),
),
Migration warning when compat enabled Added check into migration plugin to warn user when compat is enabled. If compat is enabled, the migration fails and user is warned that he must turn the compat off or run the script with (the newly introduced) option '--with-compat'. '--with-compat' is new flag. If it is set, the compat status is ignored. https://fedorahosted.org/freeipa/ticket/2274
2012-03-01 04:41:53 -06:00
Flag('compat?',
cli_name='with_compat',
label=_('Ignore compat plugin'),
doc=_('Allows migration despite the usage of compat plugin'),
default=False,
),
migrate-ds added --ca-cert-file=FILE option FILE is used to specify CA certificate for DS connection when TLS is required (ldaps://...). Ticket: https://fedorahosted.org/freeipa/ticket/3243
2013-11-21 10:01:36 -06:00
File('cacertfile?',
cli_name='ca_cert_file',
label=_('CA certificate'),
doc=_('Load CA certificate of LDAP server from FILE'),
default=None
),
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
)
has_output = (
output.Output('result',
type=dict,
localize doc strings A number of doc strings were not localized, wrap them in _(). Some messages were not localized, wrap them in _() Fix a couple of failing tests: The method name in RPC should not be unicode. The doc attribute must use the .msg attribute for comparison. Also clean up imports of _() The import should come from ipalib or ipalib.text, not ugettext from request.
2010-03-05 15:11:21 -06:00
doc=_('Lists of objects migrated; categorized by type.'),
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
),
output.Output('failed',
type=dict,
localize doc strings A number of doc strings were not localized, wrap them in _(). Some messages were not localized, wrap them in _() Fix a couple of failing tests: The method name in RPC should not be unicode. The doc attribute must use the .msg attribute for comparison. Also clean up imports of _() The import should come from ipalib or ipalib.text, not ugettext from request.
2010-03-05 15:11:21 -06:00
doc=_('Lists of objects that could not be migrated; categorized by type.'),
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
),
output.Output('enabled',
type=bool,
localize doc strings A number of doc strings were not localized, wrap them in _(). Some messages were not localized, wrap them in _() Fix a couple of failing tests: The method name in RPC should not be unicode. The doc attribute must use the .msg attribute for comparison. Also clean up imports of _() The import should come from ipalib or ipalib.text, not ugettext from request.
2010-03-05 15:11:21 -06:00
doc=_('False if migration mode was disabled.'),
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
),
Migration warning when compat enabled Added check into migration plugin to warn user when compat is enabled. If compat is enabled, the migration fails and user is warned that he must turn the compat off or run the script with (the newly introduced) option '--with-compat'. '--with-compat' is new flag. If it is set, the compat status is ignored. https://fedorahosted.org/freeipa/ticket/2274
2012-03-01 04:41:53 -06:00
output.Output('compat',
type=bool,
doc=_('False if migration fails because the compatibility plug-in is enabled.'),
),
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
)
Update argument docs to reflect dropped CSV support https://fedorahosted.org/freeipa/ticket/3352
2013-02-14 11:01:16 -06:00
exclude_doc = _('%s to exclude from migration')
localize doc strings A number of doc strings were not localized, wrap them in _(). Some messages were not localized, wrap them in _() Fix a couple of failing tests: The method name in RPC should not be unicode. The doc attribute must use the .msg attribute for comparison. Also clean up imports of _() The import should come from ipalib or ipalib.text, not ugettext from request.
2010-03-05 15:11:21 -06:00
truncated_err_msg = _('''\
search results for objects to be migrated
have been truncated by the server;
Typos in freeIPA messages This patch fixes several reported typos in IPA messages and in comments. Contributors file has been updated + the original author of the patch reporting the typos was added. https://fedorahosted.org/freeipa/ticket/848
2011-02-02 10:15:35 -06:00
migration process might be incomplete\n''')
localize doc strings A number of doc strings were not localized, wrap them in _(). Some messages were not localized, wrap them in _() Fix a couple of failing tests: The method name in RPC should not be unicode. The doc attribute must use the .msg attribute for comparison. Also clean up imports of _() The import should come from ipalib or ipalib.text, not ugettext from request.
2010-03-05 15:11:21 -06:00
migration_disabled_msg = _('''\
Migration mode is disabled. Use \'ipa config-mod\' to enable it.''')
pwd_migration_msg = _('''\
Passwords have been migrated in pre-hashed format.
IPA is unable to generate Kerberos keys unless provided
with clear text passwords. All migrated users need to
login at https://your.domain/ipa/migration/ before they
can use their Kerberos accounts.''')
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
def get_options(self):
"""
Call get_options of the baseclass and add "exclude" options
for each type of object being migrated.
"""
for option in super(migrate_ds, self).get_options():
yield option
for ldap_obj_name in self.migrate_objects:
ldap_obj = self.api.Object[ldap_obj_name]
name = 'exclude_%ss' % to_cli(ldap_obj_name)
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
doc = self.exclude_doc % ldap_obj.object_name_plural
Parse comma-separated lists of values in all parameter types. This can be enabled for a specific parameter by setting the "csv" option to True. Remove "List" parameter type and replace all occurences of it with appropriate multi-valued parameter ("Str" in most cases) with csv enabled. Add new parameter type "Any", capable of holding values of any type. This is needed by the "batch" command, as "Str" is not suitable type for the "methods" parameter. ticket 2007
2011-11-21 09:50:27 -06:00
yield Str(
'%s*' % name, cli_name=name, doc=doc, csv=True,
default=tuple(), autofill=True
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
)
def normalize_options(self, options):
"""
Convert all "exclude" option values to lower-case.
Also, empty List parameters are converted to None, but the migration
plugin doesn't like that - convert back to empty lists.
"""
for p in self.params():
Parse comma-separated lists of values in all parameter types. This can be enabled for a specific parameter by setting the "csv" option to True. Remove "List" parameter type and replace all occurences of it with appropriate multi-valued parameter ("Str" in most cases) with csv enabled. Add new parameter type "Any", capable of holding values of any type. This is needed by the "batch" command, as "Str" is not suitable type for the "methods" parameter. ticket 2007
2011-11-21 09:50:27 -06:00
if p.csv:
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
if options[p.name]:
options[p.name] = tuple(
v.lower() for v in options[p.name]
)
else:
options[p.name] = tuple()
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
def _get_search_bases(self, options, ds_base_dn, migrate_order):
search_bases = dict()
for ldap_obj_name in migrate_order:
Validate DN & RDN parameters for migrate command Ticket #2555 We were generating a traceback (server error) if a malformed RDN was passed as a parameter to the migrate command. * add parameter validation functions validate_dn_param() and validate_rdn_param() to ipalib.util. Those functions simply invoke the DN or RDN constructor from our dn module passing it the string representation. If the constructor does not throw an error it's valid. * Add the parameter validation function pointers to the Param objects in the migrate command. * Make the usercontainer and groupcontainer parameters required. passing --usercontainer= on the command line will produce ipa: ERROR: 'user_container' is required * Fix _get_search_bases() so if a container dn is empty it it just uses the base dn alone instead of faulting (currently bullet-proofing because now the containers are required). * Update the doc for usercontainer and groupcontainer to reflect the fact they are DN's not RDN's. A RDN can only be one level and it should be possible to have a container more than one RDN removed from the base.
2012-04-16 01:33:26 -05:00
container = options.get('%scontainer' % to_cli(ldap_obj_name))
if container:
don't append basedn to container if it is included ticket #2566 When specifying a container to ds-migrate we should not automatically append the basedn if it is provided by the end-user. This is easy to detect using DN objects because DN objects have a endswith() method which can easily and correctly ascertain if a base already exists.
2012-04-16 17:48:57 -05:00
# Don't append base dn if user already appended it in the container dn
if container.endswith(ds_base_dn):
search_base = container
else:
search_base = DN(container, ds_base_dn)
Validate DN & RDN parameters for migrate command Ticket #2555 We were generating a traceback (server error) if a malformed RDN was passed as a parameter to the migrate command. * add parameter validation functions validate_dn_param() and validate_rdn_param() to ipalib.util. Those functions simply invoke the DN or RDN constructor from our dn module passing it the string representation. If the constructor does not throw an error it's valid. * Add the parameter validation function pointers to the Param objects in the migrate command. * Make the usercontainer and groupcontainer parameters required. passing --usercontainer= on the command line will produce ipa: ERROR: 'user_container' is required * Fix _get_search_bases() so if a container dn is empty it it just uses the base dn alone instead of faulting (currently bullet-proofing because now the containers are required). * Update the doc for usercontainer and groupcontainer to reflect the fact they are DN's not RDN's. A RDN can only be one level and it should be possible to have a container more than one RDN removed from the base.
2012-04-16 01:33:26 -05:00
else:
search_base = ds_base_dn
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
search_bases[ldap_obj_name] = search_base
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
return search_bases
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
def migrate(self, ldap, config, ds_ldap, ds_base_dn, options):
"""
Migrate objects from DS to LDAP.
"""
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
assert isinstance(ds_base_dn, DN)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
migrated = {} # {'OBJ': ['PKEY1', 'PKEY2', ...], ...}
failed = {} # {'OBJ': {'PKEY1': 'Failed 'cos blabla', ...}, ...}
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
search_bases = self._get_search_bases(options, ds_base_dn, self.migrate_order)
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
migration_start = datetime.datetime.now()
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
for ldap_obj_name in self.migrate_order:
ldap_obj = self.api.Object[ldap_obj_name]
Improve migration NotFound error When no user/group was found, migration plugin reported an ambiguous error about invalid container. But the root cause may be for example in a wrong list of user/group objectclasses. Report both in the error message to avoid user confusion. User/group objectclass attribute is now also marked as required. Without the list of objectclasses, an invalid LDAP search is produced. https://fedorahosted.org/freeipa/ticket/2206
2012-06-04 07:25:41 -05:00
template = self.migrate_objects[ldap_obj_name]['filter_template']
oc_list = options[to_cli(self.migrate_objects[ldap_obj_name]['oc_option'])]
search_filter = construct_filter(template, oc_list)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
exclude = options['exclude_%ss' % to_cli(ldap_obj_name)]
Fix attributes that contain DNs when migrating. Some attributes, like secretary and manager, may point to other LDAP entries. We need to fix these during migration. https://fedorahosted.org/freeipa/ticket/2562
2012-03-20 21:37:27 -05:00
context = dict(ds_ldap = ds_ldap)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
migrated[ldap_obj_name] = []
failed[ldap_obj_name] = {}
Handle an empty base_dn and no cn=ipaconfig in the ldap2 backend, fix migration. We lacked good error messages if the user/group container you used doesn't exist. Add a --continue option so things can continue if you use a bad user/group container. This has the side-effect of letting you migrate just users or groups by using a bad container for the one you don't want. Fix a Gettext() error when displaying the migrated password message. ticket 289
2010-09-27 12:50:54 -05:00
try:
Remove unneeded python-ldap imports Part of the work for: https://fedorahosted.org/freeipa/ticket/2660
2013-01-31 08:02:01 -06:00
entries, truncated = ds_ldap.find_entries(
Fix migration to work between v2 servers and remove search/size limits. Migration from a v2 server would fail because of our fake memberofindirect attribute. This isn't in any objectclass so would cause entries to fail to migrate. We can safely just remove it. Also remove any limits on time/size when searching for entries on the remote server. Otherwise only the number of entries configured in the local IPA server can be migrated. ticket 1124
2011-04-08 09:19:42 -05:00
search_filter, ['*'], search_bases[ldap_obj_name],
Remove unneeded python-ldap imports Part of the work for: https://fedorahosted.org/freeipa/ticket/2660
2013-01-31 08:02:01 -06:00
ds_ldap.SCOPE_ONELEVEL,
Handle LDAP search references LDAP search operation may return a search reference pointing to an LDAP resource. As the framework does not handle search references, skip these results to prevent result processing failures. Migrate operation crashed when the migrated DS contained search references. Now, it correctly skips these records and prints the failed references to user. https://fedorahosted.org/freeipa/ticket/1209
2011-06-01 11:04:24 -05:00
time_limit=0, size_limit=-1,
search_refs=True # migrated DS may contain search references
Handle an empty base_dn and no cn=ipaconfig in the ldap2 backend, fix migration. We lacked good error messages if the user/group container you used doesn't exist. Add a --continue option so things can continue if you use a bad user/group container. This has the side-effect of letting you migrate just users or groups by using a bad container for the one you don't want. Fix a Gettext() error when displaying the migrated password message. ticket 289
2010-09-27 12:50:54 -05:00
)
except errors.NotFound:
if not options.get('continue',False):
Fix translatable strings in ipalib plugins. Needed for xgettext/pygettext processing.
2011-02-23 15:47:49 -06:00
raise errors.NotFound(
Improve migration NotFound error When no user/group was found, migration plugin reported an ambiguous error about invalid container. But the root cause may be for example in a wrong list of user/group objectclasses. Report both in the error message to avoid user confusion. User/group objectclass attribute is now also marked as required. Without the list of objectclasses, an invalid LDAP search is produced. https://fedorahosted.org/freeipa/ticket/2206
2012-06-04 07:25:41 -05:00
reason=_('%(container)s LDAP search did not return any result '
'(search base: %(search_base)s, '
'objectclass: %(objectclass)s)')
% {'container': ldap_obj_name,
'search_base': search_bases[ldap_obj_name],
'objectclass': ', '.join(oc_list)}
Fix translatable strings in ipalib plugins. Needed for xgettext/pygettext processing.
2011-02-23 15:47:49 -06:00
)
Handle an empty base_dn and no cn=ipaconfig in the ldap2 backend, fix migration. We lacked good error messages if the user/group container you used doesn't exist. Add a --continue option so things can continue if you use a bad user/group container. This has the side-effect of letting you migrate just users or groups by using a bad container for the one you don't want. Fix a Gettext() error when displaying the migrated password message. ticket 289
2010-09-27 12:50:54 -05:00
else:
truncated = False
entries = []
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
if truncated:
self.log.error(
'%s: %s' % (
Fixed object_name and object_name_plural internationalization The object_name, object_name_plural and messages that use these attributes have been converted to support translation. The label attribute in the Param class has been modified to accept unicode string. Ticket #1435
2011-07-12 11:01:25 -05:00
ldap_obj.name, self.truncated_err_msg
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
)
)
Add ignore lists to migrate-ds command When user migrates users/groups from an old DS instance, the migration may fail on unsupported object classes and/or relevant LDAP object attributes. This patch implements a support for object class and attribute ignore lists that can be used to suppress these migration issues. Additionally, a redundant "dev/null" file is removed from git repo (originally added in 26b0e8fc9809a4cd9f2f9a2281f0894e2e0f8db2). https://fedorahosted.org/freeipa/ticket/1266
2011-06-03 07:21:43 -05:00
blacklists = {}
for blacklist in ('oc_blacklist', 'attr_blacklist'):
blacklist_option = self.migrate_objects[ldap_obj_name][blacklist+'_option']
if blacklist_option is not None:
blacklists[blacklist] = options.get(blacklist_option, tuple())
else:
blacklists[blacklist] = tuple()
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
# get default primary group for new users
if 'def_group_dn' not in context:
def_group = config.get('ipadefaultprimarygroup')
context['def_group_dn'] = api.Object.group.get_dn(def_group)
try:
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
g_attrs = ldap.get_entry(context['def_group_dn'], ['gidnumber', 'cn'])
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
except errors.NotFound:
error_msg = _('Default group for new users not found')
raise errors.NotFound(reason=error_msg)
if 'gidnumber' in g_attrs:
context['def_group_gid'] = g_attrs['gidnumber'][0]
context['has_upg'] = ldap.has_upg()
valid_gids = []
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
invalid_gids = []
migrate_cnt = 0
Prevent a crash when no entries are successfully migrated. It would fail in _update_default_group() because migrate_cnt wasn't defined in context. https://fedorahosted.org/freeipa/ticket/3386
2013-02-08 08:42:34 -06:00
context['migrate_cnt'] = 0
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
for entry_attrs in entries:
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
context['migrate_cnt'] = migrate_cnt
s = datetime.datetime.now()
Handle LDAP search references LDAP search operation may return a search reference pointing to an LDAP resource. As the framework does not handle search references, skip these results to prevent result processing failures. Migrate operation crashed when the migrated DS contained search references. Now, it correctly skips these records and prints the failed references to user. https://fedorahosted.org/freeipa/ticket/1209
2011-06-01 11:04:24 -05:00
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
ava = entry_attrs.dn[0][0]
migrate process cannot handle multivalued pkey attribute When group/user is migrated, the attribute used for RDN may be multivalued. Make sure that we pick the value used in the RDN which should be the unique one and not just the first one. https://fedorahosted.org/freeipa/ticket/1892
2011-09-29 04:55:13 -05:00
if ava.attr == ldap_obj.primary_key.name:
# In case if pkey attribute is in the migrated object DN
# and the original LDAP is multivalued, make sure that
# we pick the correct value (the unique one stored in DN)
pkey = ava.value.lower()
else:
pkey = entry_attrs[ldap_obj.primary_key.name][0].lower()
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
if pkey in exclude:
continue
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
entry_attrs.dn = ldap_obj.get_dn(pkey)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
entry_attrs['objectclass'] = list(
set(
config.get(
ldap_obj.object_class_config, ldap_obj.object_class
) + [o.lower() for o in entry_attrs['objectclass']]
)
)
Normalize the primary key value to lowercase during migration. https://bugzilla.redhat.com/show_bug.cgi?id=804609
2012-03-20 21:50:17 -05:00
entry_attrs[ldap_obj.primary_key.name][0] = entry_attrs[ldap_obj.primary_key.name][0].lower()
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
Make the migration plugin more configurable This patch adds new options to the migration plugin: * the option to fine-tune the objectclass of users or groups being imported * the option to select the LDAP schema (RFC2307 or RFC2307bis) Also makes the logic that decides whether an entry is a nested group or user (for RFC2307bis) smarter by looking at the DNS. Does not hardcode primary keys for migrated entries. https://fedorahosted.org/freeipa/ticket/429
2010-10-26 15:10:42 -05:00
callback = self.migrate_objects[ldap_obj_name]['pre_callback']
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
if callable(callback):
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
try:
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
entry_attrs.dn = callback(
ldap, pkey, entry_attrs.dn, entry_attrs,
failed[ldap_obj_name], config, context,
schema=options['schema'],
search_bases=search_bases,
valid_gids=valid_gids,
invalid_gids=invalid_gids,
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
**blacklists
)
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
if not entry_attrs.dn:
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
continue
except errors.NotFound, e:
failed[ldap_obj_name][pkey] = unicode(e.reason)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
continue
try:
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
ldap.add_entry(entry_attrs)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
except errors.ExecutionError, e:
Improve handling of GIDs when migrating groups Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866
2011-10-03 09:01:01 -05:00
callback = self.migrate_objects[ldap_obj_name]['exc_callback']
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
if callable(callback):
Improve handling of GIDs when migrating groups Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866
2011-10-03 09:01:01 -05:00
try:
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
callback(
ldap, entry_attrs.dn, entry_attrs, e, options)
Improve handling of GIDs when migrating groups Since IPA v2 server already contain predefined groups that may collide with groups in migrated (IPA v1) server (for example admins, ipausers), users having colliding group as their primary group may happen to belong to an unknown group on new IPA v2 server. Implement --group-overwrite-gid option to overwrite GID of already existing groups to prevent this issue. https://fedorahosted.org/freeipa/ticket/1866
2011-10-03 09:01:01 -05:00
except errors.ExecutionError, e:
failed[ldap_obj_name][pkey] = unicode(e)
continue
else:
failed[ldap_obj_name][pkey] = unicode(e)
continue
migrated[ldap_obj_name].append(pkey)
callback = self.migrate_objects[ldap_obj_name]['post_callback']
if callable(callback):
callback(
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
ldap, pkey, entry_attrs.dn, entry_attrs,
failed[ldap_obj_name], config, context)
Improve migration performance Add new users to the default users group in batches of 100. The biggest overhead of migration is in calculating the modlist when managing the default user's group and applying the changes. A significant amount of time can be saved by not doing this on every add operation. Some other minor improvements include: Add a negative cache for groups not found in the remote LDAP server. Replace call to user_mod with a direct LDAP update. Catch some occurances of LimitError and handle more gracefully. I also added some debug logging to report on migration status and performance. https://fedorahosted.org/freeipa/ticket/3386
2013-01-29 10:19:30 -06:00
e = datetime.datetime.now()
d = e - s
total_dur = e - migration_start
migrate_cnt += 1
if migrate_cnt > 0 and migrate_cnt % 100 == 0:
api.log.info("%d %ss migrated. %s elapsed." % (migrate_cnt, ldap_obj_name, total_dur))
api.log.debug("%d %ss migrated, duration: %s (total %s)" % (migrate_cnt, ldap_obj_name, d, total_dur))
_update_default_group(ldap, pkey, config, context, True)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
return (migrated, failed)
def execute(self, ldapuri, bindpw, **options):
ldap = self.api.Backend.ldap2
self.normalize_options(options)
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
config = ldap.get_ipa_config()
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
Add support defaultNamingContext and add --basedn to migrate-ds There are two sides to this, the server and client side. On the server side we attempt to add a defaultNamingContext on already installed servers. This will fail on older 389-ds instances but the failure is not fatal. New installations on versions of 389-ds that support this attribute will have it already defined. On the client side we need to look for both defaultNamingContext and namingContexts. We still need to check that the defaultNamingContext is an IPA server (info=IPAV2). The migration change also takes advantage of this and adds a new option which allows one to provide a basedn to use instead of trying to detect it. https://fedorahosted.org/freeipa/ticket/1919 https://fedorahosted.org/freeipa/ticket/2314
2012-01-30 15:29:32 -06:00
ds_base_dn = options.get('basedn')
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
if ds_base_dn is not None:
assert isinstance(ds_base_dn, DN)
Add support defaultNamingContext and add --basedn to migrate-ds There are two sides to this, the server and client side. On the server side we attempt to add a defaultNamingContext on already installed servers. This will fail on older 389-ds instances but the failure is not fatal. New installations on versions of 389-ds that support this attribute will have it already defined. On the client side we need to look for both defaultNamingContext and namingContexts. We still need to check that the defaultNamingContext is an IPA server (info=IPAV2). The migration change also takes advantage of this and adds a new option which allows one to provide a basedn to use instead of trying to detect it. https://fedorahosted.org/freeipa/ticket/1919 https://fedorahosted.org/freeipa/ticket/2314
2012-01-30 15:29:32 -06:00
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
# check if migration mode is enabled
if config.get('ipamigrationenabled', ('FALSE', ))[0] == 'FALSE':
Migration warning when compat enabled Added check into migration plugin to warn user when compat is enabled. If compat is enabled, the migration fails and user is warned that he must turn the compat off or run the script with (the newly introduced) option '--with-compat'. '--with-compat' is new flag. If it is set, the compat status is ignored. https://fedorahosted.org/freeipa/ticket/2274
2012-03-01 04:41:53 -06:00
return dict(result={}, failed={}, enabled=False, compat=True)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
# connect to DS
ds_ldap = ldap2(shared_instance=False, ldap_uri=ldapuri, base_dn='')
migrate-ds added --ca-cert-file=FILE option FILE is used to specify CA certificate for DS connection when TLS is required (ldaps://...). Ticket: https://fedorahosted.org/freeipa/ticket/3243
2013-11-21 10:01:36 -06:00
cacert = None
if options.get('cacertfile') is not None:
#store CA cert into file
tmp_ca_cert_f = write_tmp_file(options['cacertfile'])
cacert = tmp_ca_cert_f.name
#start TLS connection
ds_ldap.connect(bind_dn=options['binddn'], bind_pw=bindpw,
tls_cacertfile=cacert)
tmp_ca_cert_f.close()
else:
ds_ldap.connect(bind_dn=options['binddn'], bind_pw=bindpw)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
Migration warning when compat enabled Added check into migration plugin to warn user when compat is enabled. If compat is enabled, the migration fails and user is warned that he must turn the compat off or run the script with (the newly introduced) option '--with-compat'. '--with-compat' is new flag. If it is set, the compat status is ignored. https://fedorahosted.org/freeipa/ticket/2274
2012-03-01 04:41:53 -06:00
#check whether the compat plugin is enabled
if not options.get('compat'):
Fix migration plugin compat check Ticket #2274 implements a check for compat plugin and warns user if it is enabled. However, there are 2 issues connected with the plugin: 1) The check is performed against the remote (migrated) LDAP server and not the local LDAP server, which does not make much sense 2) When the compat plugin is missing in cn=plugins,cn=config, it raises an error and thus breaks the migration This patch fixes both issues. https://fedorahosted.org/freeipa/ticket/2508
2012-03-09 05:36:03 -06:00
try:
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
check_compat = ldap.get_entry(_compat_dn)
Fix migration plugin compat check Ticket #2274 implements a check for compat plugin and warns user if it is enabled. However, there are 2 issues connected with the plugin: 1) The check is performed against the remote (migrated) LDAP server and not the local LDAP server, which does not make much sense 2) When the compat plugin is missing in cn=plugins,cn=config, it raises an error and thus breaks the migration This patch fixes both issues. https://fedorahosted.org/freeipa/ticket/2508
2012-03-09 05:36:03 -06:00
if check_compat is not None and \
check_compat.get('nsslapd-pluginenabled', [''])[0].lower() == 'on':
Don't create private groups for migrated users, check for valid gidnumber Migrated users don't get a private group, there is no safe way to verify that the namespace is correct without redoing the uidnumber as well. Verify that the GID at least points to a valid group on the remote server and warn if it doesn't (this doesn't guarantee that the group gets migrated but at least we try). If the remote entry has no gidNumber then don't migrate that user. We don't know why that user is non-POSIX, it could be a special user used for auth, for example. Add a loginshell if the remote user doesn't have one. https://fedorahosted.org/freeipa/ticket/2562
2012-03-22 12:40:54 -05:00
return dict(result={}, failed={}, enabled=True, compat=False)
Fix migration plugin compat check Ticket #2274 implements a check for compat plugin and warns user if it is enabled. However, there are 2 issues connected with the plugin: 1) The check is performed against the remote (migrated) LDAP server and not the local LDAP server, which does not make much sense 2) When the compat plugin is missing in cn=plugins,cn=config, it raises an error and thus breaks the migration This patch fixes both issues. https://fedorahosted.org/freeipa/ticket/2508
2012-03-09 05:36:03 -06:00
except errors.NotFound:
pass
Migration warning when compat enabled Added check into migration plugin to warn user when compat is enabled. If compat is enabled, the migration fails and user is warned that he must turn the compat off or run the script with (the newly introduced) option '--with-compat'. '--with-compat' is new flag. If it is set, the compat status is ignored. https://fedorahosted.org/freeipa/ticket/2274
2012-03-01 04:41:53 -06:00
Add support defaultNamingContext and add --basedn to migrate-ds There are two sides to this, the server and client side. On the server side we attempt to add a defaultNamingContext on already installed servers. This will fail on older 389-ds instances but the failure is not fatal. New installations on versions of 389-ds that support this attribute will have it already defined. On the client side we need to look for both defaultNamingContext and namingContexts. We still need to check that the defaultNamingContext is an IPA server (info=IPAV2). The migration change also takes advantage of this and adds a new option which allows one to provide a basedn to use instead of trying to detect it. https://fedorahosted.org/freeipa/ticket/1919 https://fedorahosted.org/freeipa/ticket/2314
2012-01-30 15:29:32 -06:00
if not ds_base_dn:
# retrieve base DN from remote LDAP server
Remove unneeded python-ldap imports Part of the work for: https://fedorahosted.org/freeipa/ticket/2660
2013-01-31 08:02:01 -06:00
entries, truncated = ds_ldap.find_entries(
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
'', ['namingcontexts', 'defaultnamingcontext'], DN(''),
Remove unneeded python-ldap imports Part of the work for: https://fedorahosted.org/freeipa/ticket/2660
2013-01-31 08:02:01 -06:00
ds_ldap.SCOPE_BASE, size_limit=-1, time_limit=0,
Add support defaultNamingContext and add --basedn to migrate-ds There are two sides to this, the server and client side. On the server side we attempt to add a defaultNamingContext on already installed servers. This will fail on older 389-ds instances but the failure is not fatal. New installations on versions of 389-ds that support this attribute will have it already defined. On the client side we need to look for both defaultNamingContext and namingContexts. We still need to check that the defaultNamingContext is an IPA server (info=IPAV2). The migration change also takes advantage of this and adds a new option which allows one to provide a basedn to use instead of trying to detect it. https://fedorahosted.org/freeipa/ticket/1919 https://fedorahosted.org/freeipa/ticket/2314
2012-01-30 15:29:32 -06:00
)
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
if 'defaultnamingcontext' in entries[0]:
ds_base_dn = DN(entries[0]['defaultnamingcontext'][0])
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
assert isinstance(ds_base_dn, DN)
Add support defaultNamingContext and add --basedn to migrate-ds There are two sides to this, the server and client side. On the server side we attempt to add a defaultNamingContext on already installed servers. This will fail on older 389-ds instances but the failure is not fatal. New installations on versions of 389-ds that support this attribute will have it already defined. On the client side we need to look for both defaultNamingContext and namingContexts. We still need to check that the defaultNamingContext is an IPA server (info=IPAV2). The migration change also takes advantage of this and adds a new option which allows one to provide a basedn to use instead of trying to detect it. https://fedorahosted.org/freeipa/ticket/1919 https://fedorahosted.org/freeipa/ticket/2314
2012-01-30 15:29:32 -06:00
else:
try:
Convert remaining frontend code to LDAPEntry API.
2013-10-31 11:54:21 -05:00
ds_base_dn = DN(entries[0]['namingcontexts'][0])
Use DN objects instead of strings * Convert every string specifying a DN into a DN object * Every place a dn was manipulated in some fashion it was replaced by the use of DN operators * Add new DNParam parameter type for parameters which are DN's * DN objects are used 100% of the time throughout the entire data pipeline whenever something is logically a dn. * Many classes now enforce DN usage for their attributes which are dn's. This is implmented via ipautil.dn_attribute_property(). The only permitted types for a class attribute specified to be a DN are either None or a DN object. * Require that every place a dn is used it must be a DN object. This translates into lot of:: assert isinstance(dn, DN) sprinkled through out the code. Maintaining these asserts is valuable to preserve DN type enforcement. The asserts can be disabled in production. The goal of 100% DN usage 100% of the time has been realized, these asserts are meant to preserve that. The asserts also proved valuable in detecting functions which did not obey their function signatures, such as the baseldap pre and post callbacks. * Moved ipalib.dn to ipapython.dn because DN class is shared with all components, not just the server which uses ipalib. * All API's now accept DN's natively, no need to convert to str (or unicode). * Removed ipalib.encoder and encode/decode decorators. Type conversion is now explicitly performed in each IPASimpleLDAPObject method which emulates a ldap.SimpleLDAPObject method. * Entity & Entry classes now utilize DN's * Removed __getattr__ in Entity & Entity clases. There were two problems with it. It presented synthetic Python object attributes based on the current LDAP data it contained. There is no way to validate synthetic attributes using code checkers, you can't search the code to find LDAP attribute accesses (because synthetic attriutes look like Python attributes instead of LDAP data) and error handling is circumscribed. Secondly __getattr__ was hiding Python internal methods which broke class semantics. * Replace use of methods inherited from ldap.SimpleLDAPObject via IPAdmin class with IPAdmin methods. Directly using inherited methods was causing us to bypass IPA logic. Mostly this meant replacing the use of search_s() with getEntry() or getList(). Similarly direct access of the LDAP data in classes using IPAdmin were replaced with calls to getValue() or getValues(). * Objects returned by ldap2.find_entries() are now compatible with either the python-ldap access methodology or the Entity/Entry access methodology. * All ldap operations now funnel through the common IPASimpleLDAPObject giving us a single location where we interface to python-ldap and perform conversions. * The above 4 modifications means we've greatly reduced the proliferation of multiple inconsistent ways to perform LDAP operations. We are well on the way to having a single API in IPA for doing LDAP (a long range goal). * All certificate subject bases are now DN's * DN objects were enhanced thusly: - find, rfind, index, rindex, replace and insert methods were added - AVA, RDN and DN classes were refactored in immutable and mutable variants, the mutable variants are EditableAVA, EditableRDN and EditableDN. By default we use the immutable variants preserving important semantics. To edit a DN cast it to an EditableDN and cast it back to DN when done editing. These issues are fully described in other documentation. - first_key_match was removed - DN equalty comparison permits comparison to a basestring * Fixed ldapupdate to work with DN's. This work included: - Enhance test_updates.py to do more checking after applying update. Add test for update_from_dict(). Convert code to use unittest classes. - Consolidated duplicate code. - Moved code which should have been in the class into the class. - Fix the handling of the 'deleteentry' update action. It's no longer necessary to supply fake attributes to make it work. Detect case where subsequent update applies a change to entry previously marked for deletetion. General clean-up and simplification of the 'deleteentry' logic. - Rewrote a couple of functions to be clearer and more Pythonic. - Added documentation on the data structure being used. - Simplfy the use of update_from_dict() * Removed all usage of get_schema() which was being called prior to accessing the .schema attribute of an object. If a class is using internal lazy loading as an optimization it's not right to require users of the interface to be aware of internal optimization's. schema is now a property and when the schema property is accessed it calls a private internal method to perform the lazy loading. * Added SchemaCache class to cache the schema's from individual servers. This was done because of the observation we talk to different LDAP servers, each of which may have it's own schema. Previously we globally cached the schema from the first server we connected to and returned that schema in all contexts. The cache includes controls to invalidate it thus forcing a schema refresh. * Schema caching is now senstive to the run time context. During install and upgrade the schema can change leading to errors due to out-of-date cached schema. The schema cache is refreshed in these contexts. * We are aware of the LDAP syntax of all LDAP attributes. Every attribute returned from an LDAP operation is passed through a central table look-up based on it's LDAP syntax. The table key is the LDAP syntax it's value is a Python callable that returns a Python object matching the LDAP syntax. There are a handful of LDAP attributes whose syntax is historically incorrect (e.g. DistguishedNames that are defined as DirectoryStrings). The table driven conversion mechanism is augmented with a table of hard coded exceptions. Currently only the following conversions occur via the table: - dn's are converted to DN objects - binary objects are converted to Python str objects (IPA convention). - everything else is converted to unicode using UTF-8 decoding (IPA convention). However, now that the table driven conversion mechanism is in place it would be trivial to do things such as converting attributes which have LDAP integer syntax into a Python integer, etc. * Expected values in the unit tests which are a DN no longer need to use lambda expressions to promote the returned value to a DN for equality comparison. The return value is automatically promoted to a DN. The lambda expressions have been removed making the code much simpler and easier to read. * Add class level logging to a number of classes which did not support logging, less need for use of root_logger. * Remove ipaserver/conn.py, it was unused. * Consolidated duplicate code wherever it was found. * Fixed many places that used string concatenation to form a new string rather than string formatting operators. This is necessary because string formatting converts it's arguments to a string prior to building the result string. You can't concatenate a string and a non-string. * Simplify logic in rename_managed plugin. Use DN operators to edit dn's. * The live version of ipa-ldap-updater did not generate a log file. The offline version did, now both do. https://fedorahosted.org/freeipa/ticket/1670 https://fedorahosted.org/freeipa/ticket/1671 https://fedorahosted.org/freeipa/ticket/1672 https://fedorahosted.org/freeipa/ticket/1673 https://fedorahosted.org/freeipa/ticket/1674 https://fedorahosted.org/freeipa/ticket/1392 https://fedorahosted.org/freeipa/ticket/2872
2012-05-13 06:36:35 -05:00
assert isinstance(ds_base_dn, DN)
Add support defaultNamingContext and add --basedn to migrate-ds There are two sides to this, the server and client side. On the server side we attempt to add a defaultNamingContext on already installed servers. This will fail on older 389-ds instances but the failure is not fatal. New installations on versions of 389-ds that support this attribute will have it already defined. On the client side we need to look for both defaultNamingContext and namingContexts. We still need to check that the defaultNamingContext is an IPA server (info=IPAV2). The migration change also takes advantage of this and adds a new option which allows one to provide a basedn to use instead of trying to detect it. https://fedorahosted.org/freeipa/ticket/1919 https://fedorahosted.org/freeipa/ticket/2314
2012-01-30 15:29:32 -06:00
except (IndexError, KeyError), e:
raise StandardError(str(e))
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
# migrate!
(migrated, failed) = self.migrate(
ldap, config, ds_ldap, ds_base_dn, options
)
Migration warning when compat enabled Added check into migration plugin to warn user when compat is enabled. If compat is enabled, the migration fails and user is warned that he must turn the compat off or run the script with (the newly introduced) option '--with-compat'. '--with-compat' is new flag. If it is set, the compat status is ignored. https://fedorahosted.org/freeipa/ticket/2274
2012-03-01 04:41:53 -06:00
return dict(result=migrated, failed=failed, enabled=True, compat=True)
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
def output_for_cli(self, textui, result, ldapuri, bindpw, **options):
textui.print_name(self.name)
if not result['enabled']:
textui.print_plain(self.migration_disabled_msg)
return 1
Migration warning when compat enabled Added check into migration plugin to warn user when compat is enabled. If compat is enabled, the migration fails and user is warned that he must turn the compat off or run the script with (the newly introduced) option '--with-compat'. '--with-compat' is new flag. If it is set, the compat status is ignored. https://fedorahosted.org/freeipa/ticket/2274
2012-03-01 04:41:53 -06:00
if not result['compat']:
textui.print_plain("The compat plug-in is enabled. This can increase the memory requirements during migration. Disable the compat plug-in with \'ipa-compat-manage disable\' or re-run this script with \'--with-compat\' option.")
return 1
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
textui.print_plain('Migrated:')
textui.print_entry1(
result['result'], attr_order=self.migrate_order,
one_value_per_line=False
)
for ldap_obj_name in self.migrate_order:
textui.print_plain('Failed %s:' % ldap_obj_name)
textui.print_entry1(
result['failed'][ldap_obj_name], attr_order=self.migrate_order,
one_value_per_line=True,
)
textui.print_plain('-' * len(self.name))
Handle an empty base_dn and no cn=ipaconfig in the ldap2 backend, fix migration. We lacked good error messages if the user/group container you used doesn't exist. Add a --continue option so things can continue if you use a bad user/group container. This has the side-effect of letting you migrate just users or groups by using a bad container for the one you don't want. Fix a Gettext() error when displaying the migrated password message. ticket 289
2010-09-27 12:50:54 -05:00
textui.print_plain(unicode(self.pwd_migration_msg))
Add DS migration plugin and password migration page.
2010-01-12 09:40:09 -06:00
Reference in New Issue Copy Permalink