When long additional text should follow the error message, one can
supply instructions parameter to a class derived from PublicError.
This will cause following text added to the error message:
Additional instructions:
<additional text>
`instructions' optional parameter could be a list or anything that coerces
into unicode(). List entries will be joined with '\n'.
https://fedorahosted.org/freeipa/ticket/3167
Do not print list of possible values as "%r" but simply as a list
of quoted values which should make it easier to read for users.
Also add a special case when there is just one allowed value.
https://fedorahosted.org/freeipa/ticket/2869
* Convert every string specifying a DN into a DN object
* Every place a dn was manipulated in some fashion it was replaced by
the use of DN operators
* Add new DNParam parameter type for parameters which are DN's
* DN objects are used 100% of the time throughout the entire data
pipeline whenever something is logically a dn.
* Many classes now enforce DN usage for their attributes which are
dn's. This is implmented via ipautil.dn_attribute_property(). The
only permitted types for a class attribute specified to be a DN are
either None or a DN object.
* Require that every place a dn is used it must be a DN object.
This translates into lot of::
assert isinstance(dn, DN)
sprinkled through out the code. Maintaining these asserts is
valuable to preserve DN type enforcement. The asserts can be
disabled in production.
The goal of 100% DN usage 100% of the time has been realized, these
asserts are meant to preserve that.
The asserts also proved valuable in detecting functions which did
not obey their function signatures, such as the baseldap pre and
post callbacks.
* Moved ipalib.dn to ipapython.dn because DN class is shared with all
components, not just the server which uses ipalib.
* All API's now accept DN's natively, no need to convert to str (or
unicode).
* Removed ipalib.encoder and encode/decode decorators. Type conversion
is now explicitly performed in each IPASimpleLDAPObject method which
emulates a ldap.SimpleLDAPObject method.
* Entity & Entry classes now utilize DN's
* Removed __getattr__ in Entity & Entity clases. There were two
problems with it. It presented synthetic Python object attributes
based on the current LDAP data it contained. There is no way to
validate synthetic attributes using code checkers, you can't search
the code to find LDAP attribute accesses (because synthetic
attriutes look like Python attributes instead of LDAP data) and
error handling is circumscribed. Secondly __getattr__ was hiding
Python internal methods which broke class semantics.
* Replace use of methods inherited from ldap.SimpleLDAPObject via
IPAdmin class with IPAdmin methods. Directly using inherited methods
was causing us to bypass IPA logic. Mostly this meant replacing the
use of search_s() with getEntry() or getList(). Similarly direct
access of the LDAP data in classes using IPAdmin were replaced with
calls to getValue() or getValues().
* Objects returned by ldap2.find_entries() are now compatible with
either the python-ldap access methodology or the Entity/Entry access
methodology.
* All ldap operations now funnel through the common
IPASimpleLDAPObject giving us a single location where we interface
to python-ldap and perform conversions.
* The above 4 modifications means we've greatly reduced the
proliferation of multiple inconsistent ways to perform LDAP
operations. We are well on the way to having a single API in IPA for
doing LDAP (a long range goal).
* All certificate subject bases are now DN's
* DN objects were enhanced thusly:
- find, rfind, index, rindex, replace and insert methods were added
- AVA, RDN and DN classes were refactored in immutable and mutable
variants, the mutable variants are EditableAVA, EditableRDN and
EditableDN. By default we use the immutable variants preserving
important semantics. To edit a DN cast it to an EditableDN and
cast it back to DN when done editing. These issues are fully
described in other documentation.
- first_key_match was removed
- DN equalty comparison permits comparison to a basestring
* Fixed ldapupdate to work with DN's. This work included:
- Enhance test_updates.py to do more checking after applying
update. Add test for update_from_dict(). Convert code to use
unittest classes.
- Consolidated duplicate code.
- Moved code which should have been in the class into the class.
- Fix the handling of the 'deleteentry' update action. It's no longer
necessary to supply fake attributes to make it work. Detect case
where subsequent update applies a change to entry previously marked
for deletetion. General clean-up and simplification of the
'deleteentry' logic.
- Rewrote a couple of functions to be clearer and more Pythonic.
- Added documentation on the data structure being used.
- Simplfy the use of update_from_dict()
* Removed all usage of get_schema() which was being called prior to
accessing the .schema attribute of an object. If a class is using
internal lazy loading as an optimization it's not right to require
users of the interface to be aware of internal
optimization's. schema is now a property and when the schema
property is accessed it calls a private internal method to perform
the lazy loading.
* Added SchemaCache class to cache the schema's from individual
servers. This was done because of the observation we talk to
different LDAP servers, each of which may have it's own
schema. Previously we globally cached the schema from the first
server we connected to and returned that schema in all contexts. The
cache includes controls to invalidate it thus forcing a schema
refresh.
* Schema caching is now senstive to the run time context. During
install and upgrade the schema can change leading to errors due to
out-of-date cached schema. The schema cache is refreshed in these
contexts.
* We are aware of the LDAP syntax of all LDAP attributes. Every
attribute returned from an LDAP operation is passed through a
central table look-up based on it's LDAP syntax. The table key is
the LDAP syntax it's value is a Python callable that returns a
Python object matching the LDAP syntax. There are a handful of LDAP
attributes whose syntax is historically incorrect
(e.g. DistguishedNames that are defined as DirectoryStrings). The
table driven conversion mechanism is augmented with a table of
hard coded exceptions.
Currently only the following conversions occur via the table:
- dn's are converted to DN objects
- binary objects are converted to Python str objects (IPA
convention).
- everything else is converted to unicode using UTF-8 decoding (IPA
convention).
However, now that the table driven conversion mechanism is in place
it would be trivial to do things such as converting attributes
which have LDAP integer syntax into a Python integer, etc.
* Expected values in the unit tests which are a DN no longer need to
use lambda expressions to promote the returned value to a DN for
equality comparison. The return value is automatically promoted to
a DN. The lambda expressions have been removed making the code much
simpler and easier to read.
* Add class level logging to a number of classes which did not support
logging, less need for use of root_logger.
* Remove ipaserver/conn.py, it was unused.
* Consolidated duplicate code wherever it was found.
* Fixed many places that used string concatenation to form a new
string rather than string formatting operators. This is necessary
because string formatting converts it's arguments to a string prior
to building the result string. You can't concatenate a string and a
non-string.
* Simplify logic in rename_managed plugin. Use DN operators to edit
dn's.
* The live version of ipa-ldap-updater did not generate a log file.
The offline version did, now both do.
https://fedorahosted.org/freeipa/ticket/1670https://fedorahosted.org/freeipa/ticket/1671https://fedorahosted.org/freeipa/ticket/1672https://fedorahosted.org/freeipa/ticket/1673https://fedorahosted.org/freeipa/ticket/1674https://fedorahosted.org/freeipa/ticket/1392https://fedorahosted.org/freeipa/ticket/2872
We only checked the length of Command output dictionaries. A misspelled
key in would not be caught.
Fix the problem by checking if the sets of keys are equal.
Add a test. Split the test methods into more manageable pieces.
https://fedorahosted.org/freeipa/ticket/2860
When unknown keyword arguments are passed to a Command, raise an
error instead of ignoring them.
Options used when IPA calls its commands internally are listed
in a new Command attribute called internal_options, and allowed.
Previous patches (0b01751c, c45174d6, c5689e7f) made IPA not use
unknown keyword arguments in its own commands and tests, but since
that some violations were reintroduced in permission_find and tests.
Fix those.
Tests included; both a frontend unittest and a XML-RPC test via the
ping plugin (which was untested previously).
https://fedorahosted.org/freeipa/ticket/2509
Parameter Decimal does not have a sufficient value checks. Some values
cause Decimal parameter with a custom precision to crash with
an unhandled exception.
Improve parameter conversion and normalization operations to handle
decimal exceptions more gracefully. Decimal parameter now also has
new attributes enabling 2 new validation/normalization methods:
* exponential: when False, decimal number is normalized to its
non-exponential form
* numberclass: a set of allowed decimal number classes
(e.g. +Infinity, -Normal, ...) that are enforced
for every Decimal parameter value
https://fedorahosted.org/freeipa/ticket/2705
IPA has some unused code from abandoned features (Radius, ipa 1.x user
input, commant-line tab completion), as well as some duplicate utilities.
This patch cleans up the utility modules.
Duplicate code consolidated into ipapython.ipautil:
{ipalib.util,ipaserver.ipautil,ipapython.ipautil}.realm_to_suffix
{ipaserver,ipapython}.ipautil.CIDict
(with style improvements from the ipaserver version)
{ipapython.entity,ipaserver.ipautil}.utf8_encode_value
{ipapython.entity,ipaserver.ipautil}.utf8_encode_values
ipalib.util.get_fqdn was removed in favor of the same function in
ipaserver.install.installutils
Removed unused code:
ipalib.util:
load_plugins_in_dir
import_plugins_subpackage
make_repr (was imported but unused; also removed from tests)
ipapython.ipautil:
format_list
parse_key_value_pairs
read_pairs_file
read_items_file
user_input_plain
AttributeValueCompleter
ItemCompleter
ipaserver.ipautil:
get_gsserror (a different version exists in ipapython.ipautil)
ipaserver.ipautil ended up empty and is removed entirely.
https://fedorahosted.org/freeipa/ticket/2650
Move the code for encoding boolean values to LDAP boolean syntax from the
Parameter class to the Encoder class, where the rest of LDAP encoding takes
place. Remove encoding code from the Parameter class altogether, as all LDAP
encoding should be done in the Encoder class.
We use custom gettext classes (e.g. GettextFactory &
NGettextFactory). We should exercise those classes with an installed
binary mo file to demonstrate we are actually returning the expected
translated strings for all strings defined as being translatable.
The test logic in install/po/test_i18n.py was recently enhanced to
make this type of testing easier and more complete.
tests/test_ipalib/test_text.py should import the new i18n test support
and run it.
Previously tests/test_ipalib/test_text.py made a feeble but incomplete
attempt to do the above but even that was often not run because the
test would skip because the necessary test files were not available
unless they had been manually created in the install/po subdir. It is
now possible to correct those deficiencies in the test.
This patch does the following:
* Moves the location of i18n test code and adjust references to it.
install/po/test_i18n.py was moved to tests/i18n.py. This permits
tests/test_ipalib/test_text.py to import the i18n test utilities
in a clean fashion. The Makefile in install/po now calls this
same file.
* Modfies test function in test_i18n.py to accept function pointers
for retreiving a translation.
* Imports test_i18n.py from the install/po directory in the tree
* Creates a tmp directory for the test localedir
* Parses the current ipa.pot file in install/po and generates
a test po and mo file with special unicode markers. It installs
the test mo file in the tmp localedir. This is accomplished by
calling create_po() from the test_i18n.py file.
* If any of the above does not work it raises nose.SkipTest with
the reason, and skips the test.
* It sets up functions to get a translation and a plural translation
via our text.GettextFactory class and text.NGettextFactory class
respectively. This are the functions we use intenally to get
translations. It set the localdir and lang which are used by those
classes to match our test configuration. It then runs a validation
test on every translation and it's plural found in the test.po file
by calling po_file_iterate and passed it the function pointers to
our internal routines.
* At the conclusion of the test it cleans up after itself.
Note: extraneous files are not created in the tree, only a tmp
directory is utilized.
Validating msgid's in C code was insufficient.
* Make the discovery of format conversions much more robust by authoring
a new function parse_printf_fmt() that is able to discover each
format conversion in a string and break it into it's individual
subparts. One of those subparts is the argument selector index. In c
code we need to know if the argumenet selector index is present to
know if translator can reorder the substitution strings.
This replaces the simplistic python_anonymous_substitutions_regexp
which was insufficient to deal with other programming languages
(e.g. c).
* Add get_prog_langs() function to return the set of programming
languages a msgid appears in. This is necessar because the msdid
validation is programming language specific.
https://fedorahosted.org/freeipa/ticket/2582
Replace all occurences of create_default with equivalent default_from
and remove create_default from the framework. This is needed for
proper parameter validation, as there is no way to tell which
parameters to validate prior to calling create_default, because
create_default does not provide information about which parameters are
used for generating the default value.
Splitting on commas is not an idempotent operation:
'a,b\,c' -> ('a', 'b,c') -> ('a', 'b', 'c')
That means we can't do it when the call is forwarded, so this is only
done on the CLI. The UI already sends values as a tuple.
Replace escaping in the csv parser with quoting. Quoted strings can have
embedded commas instead of having to escape them. This prevents the csv
parser from eating all escape characters.
Also, document Param's csv arguments, and update tests.
https://fedorahosted.org/freeipa/ticket/2417https://fedorahosted.org/freeipa/ticket/2227
Having float type as a base type for floating point parameters in
ipalib introduces several issues, e.g. problem with representation
or value comparison. Python language provides a Decimal type which
help overcome these issues.
This patch replaces a float type and Float parameter with a
decimal.Decimal type in Decimal parameter. A precision attribute
was added to Decimal parameter that can be used to limit a number
of decimal places in parameter representation. This approach fixes
a problem with API.txt validation where comparison of float values
may fail on different architectures due to float representation error.
In order to safely transfer the parameter value over RPC it is
being converted to string which is then converted back to
decimal.Decimal number on a server side.
https://fedorahosted.org/freeipa/ticket/2260
Add new class "cachedproperty" for creating property-like attributes
that cache the return value of a method call.
Also fix few issues in the unit tests to enable them to succeed.
ticket 1959
Remove "List" parameter type and replace all occurences of it with appropriate
multi-valued parameter ("Str" in most cases) with csv enabled.
Add new parameter type "Any", capable of holding values of any type. This is
needed by the "batch" command, as "Str" is not suitable type for the "methods"
parameter.
ticket 2007
This patch changes the way plugins are initialized. Instead of
finalizing all the plugins at once, plugins are finalized only after
they are accessed (currently applies to Command, Object and
Attribute subclasses, only in CLI by default).
This change provides significant performance boost, as only the
plugins that are actually used are finalized.
ticket 1336
We have a larger goal of replacing all DN creation via string
formatting/concatenation with DN object operations because string
operations are not a safe way to form a DN nor to compare a DN. This
work needs to be broken into smaller chunks for easier review and
testing.
Addressing the unit tests first makes sense because we don't want to
be modifying both the core code and the tests used to verify the core
code simultaneously. If we modify the unittests first with existing
core code and no regressions are found then we can move on to
modifying parts of the core code with the belief the unittests can
validate the changes in the core code. Also by doing the unittests
first we also help to validate the DN objects are working correctly
(although they do have an extensive unittest).
The fundamental changes are:
* replace string substitution & concatenation with DN object
constructor
* when comparing dn's the comparision is done after promotion
to a DN object, then two DN objects are compared
* when a list of string dn's are to be compared a new list is
formed where each string dn is replaced by a DN object
* because the unittest framework accepts a complex data structure of
expected values where dn's are represeted as strings the unittest
needs to express the expected value of a dn as a callable object
(e.g. a lambda expression) which promotes the dn string to a DN
object in order to do the comparision.
The DN unittest was lacking a test for i18n. The unittest was
updated to store "Hello" in Arabic with both utf-8 and unicode
and verify the values could be properly retrieved and converted
to dn string syntax.
During the testing a few problems were discovered and corrected.
* passing in utf-8 caused an ASCII decode error becuase of Python's
silly default encoding of ASCII. The fix was to explictly use
the utf-8 codec.
* there were a couple of places where encode/decode were not
called correctly.
* the internal attr and value members of the AVA class were renamed
to explicitly show they are stored as unicode.
Of course the unittest was updated as well.
Replace deepcopy with constructor (i.e. type call)
Can now "clone" with configuration changes by passing object
of the same type to it's constructor, e.g.
dn1 = DN(('cn', 'foo'))
dn2 = DN(dn1)
dn2 = DN(dn1, first_key_match=False)
Remove pairwise grouping for RDN's. Had previously removed it
for DN's, left it in for RDN's because it seemed to make sense
because of the way RDN's work but consistency is a higher goal.
Add keyword constructor parameters to pass configuration options.
Make first_key_match a configuration keyword.
Updated documentation.
Updated unit test.
FWIW, I noticed the unittest is now running 2x faster, not sure why,
removal of deepcopy? Anyway, hard to argue with performance doubling.
The pairwise grouping used to form RDN's and AVA's proved to be
confusing in practice, this patch removes that functionality thus
requiring programmers to explicitly pair attr,value using a tuple or
list.
In addition it was discovered additional functionality was needed to
support some DN operations in freeipa. DN objects now support
startswith(), endswith() and the "in" membership test. These functions
and operators will accept either a DN or RDN.
The unittest was modified to remove the pairwise tests and add new
explicit tests. The unittest was augmented to test the new
functionality. In addition the unittest was cleaned up a bit to use
common utilty functions for improved readabilty and robustness.
The documentation was updated.
fix test_role_plugin use of DN to avoid pairwise grouping
Allow a long to get as far as the min/max constraints where we can
compare it to min/max int values and reject with a proper error message.
https://fedorahosted.org/freeipa/ticket/1494
The hostname is passed in during the server installation. We should use
this hostname for the resulting server as well. It was being discarded
and we always used the system hostname value.
Important changes:
- configure ipa_hostname in sssd on masters
- set PKI_HOSTNAME so the hostname is passed to dogtag installer
- set the hostname when doing ldapi binds
This also reorders some things in the dogtag installer to eliminate an
unnecessary restart. We were restarting the service twice in a row with
very little time in between and this could result in a slew of reported
errors, though the server installed ok.
ticket 1052
This patch removes some individual work-arounds of converting strings
to unicode, they only masked the problem. String values are not
passed to the validator or normalizers so things like adding the
realm automatically to services weren't happening.
ticket 941
This patch contains 2 parts.
The first part is a small utility to create and validate the current
API. To do this it needs to load ipalib which on a fresh system
introduces a few problems, namely that it relies on a python plugin
to set the default encoding to utf8. For our purposes we can skip that.
It is also important that any optional plugins be loadable so the
API can be examined.
The second part is a version exchange between the client and server.
The version has a major and a minor version. The major verion is
updated whenever existing API changes. The minor version is updated when
new API is added. A request will be rejected if either the major versions
don't match or if the client major version is higher than then server
major version (though by implication new API would return a command not
found if allowed to proceed).
To determine the API version of the server from a client use the ping
command.
ticket 584
The changes include:
* Change license blobs in source files to mention GPLv3+ not GPLv2 only
* Add GPLv3+ license text
* Package COPYING not LICENSE as the license blobs (even the old ones)
mention COPYING specifically, it is also more common, I think
https://fedorahosted.org/freeipa/ticket/239
When a Requirement fails we throw an exception including the name of the
field that is missing. To make the command-line friendlier we have a
cli_name defined which may or may not match the LDAP attribute. This can
be confusing if you are using ipalib directly because the attribute name
missing may not match what is actually required (desc vs description is
a good example).
If you use the context 'cli' then it will throw exceptions using cli_name.
If you use any other context it will use the name of the attribute.
ticket 187
This adds a new global option to the ipa command, -f/--no-fallback. If this
is included then just the server configured in /etc/ipa/default.conf is used.
Otherwise that is tried first then all servers in DNS with the ldap SRV record
are tried.
Create a new Local() Command class for local-only commands. The help
command is one of these. It shouldn't need a remote connection to execute.
ticket #15
It would work if you ran the test from its location in tests/test_ipalib
but this isn't the most common method. If you want to run it individually
you can do:
$ ./make-test tests/test_ipalib/test_text.py
This patch:
- bumps up the minimum version of python-nss
- will initialize NSS with nodb if a CSR is loaded and it isn't already
init'd
- will shutdown NSS if initialized in the RPC subsystem so we use right db
- updated and added a few more tests
Relying more on NSS introduces a bit of a problem. For NSS to work you
need to have initialized a database (either a real one or no_db). But once
you've initialized one and want to use another you have to close down the
first one. I've added some code to nsslib.py to do just that. This could
potentially have some bad side-effects at some point, it works ok now.
A number of doc strings were not localized, wrap them in _().
Some messages were not localized, wrap them in _()
Fix a couple of failing tests:
The method name in RPC should not be unicode.
The doc attribute must use the .msg attribute for comparison.
Also clean up imports of _() The import should come from
ipalib or ipalib.text, not ugettext from request.
