There were quite errors in es.po, it was difficult or impossible to
track down where they came from, Transifex does not have good revision
history.
I fixed about 20% of the msgstr's in the file that had obvious
problems which could be spotted by a non-Spanish speaking person.
Spurious backslashes and backslash-newlines had been introduced. I
tracked this particular problem down to a bug in polib. polib is a
Python library which can read/write po/mo files. In Fedora it's
packaged as python-polib. polib is used by the Transifex instance to
read/write po files. We don't currently use polib in IPA (that will
change soon though) but I wrote utilities using polib to help fix the
bad po file and analyze what had gone wrong. I discovered that if one
simply uses polib to read a po file into memory and they write that po
file back out from memory you don't end up with the same contents if
there are backslashed escapes in the file. I tracked this down to the
escape() and unescape() functions in polib. This caused me to look to
see if upstream polib had been fixed. It had. Therefore I think the
spurious backslashes were introduced when Transifex was using an older
broken version of polib. I filed this Fedora bug
https://bugzilla.redhat.com/show_bug.cgi?id=744419 to get the fixes
into python-polib. I manually corrected all the backslash errors.
I compared all 1329 translations from a known good version of es.po
with the current version and generated a new es.po by taking the
translation (e.g. msgstr) from the two po files which was obviously
correct. In those instances where neither msgstr was obviosuly correct
the deleted the translation entirely.
I also wrote utilities to validate any "substitution" variables
appearing in the text. I discovered a number of instances where the
substitution variable had been malformed by the translator such that
it was syntactically invalid. This is how we originally discovered
problems with the translation, it was throwing Python exceptions. I
fixed all those errors.
I also found approximately 80 translations where the leading
whitespace had been altered by the translator. Those also were fixed.
I cannot verify that the remaining translations are a correct Spanish
translation of the original text (in fact a number of them I looked at
seemed dubious to me, for example it omitted recongnizable
keywords). But I do believe that the obvious errors are fixed and we
shouldn't be throwing any more Python exceptions because of malformed
substitution variables.
This patch reverts the use of pygettext for i18n string extraction. It
was originally introduced because the help documentation for commands
are in the class docstring and module docstring.
Docstrings are a Python construct whereby any string which immediately
follows a class declaration, function/method declaration or appears
first in a module is taken to be the documentation for that
object. Python automatically assigns that string to the __doc__
variable associated with the object. Explicitly assigning to the
__doc__ variable is equivalent and permitted.
We mark strings in the source for i18n translation by embedding them
in _() or ngettext(). Specialized extraction tools (e.g. xgettext)
scan the source code looking for strings with those markers and
extracts the string for inclusion in a translation catalog.
It was mistakingly assumed one could not mark for translation Python
docstrings. Since some docstrings are vital for our command help
system some method had to be devised to extract docstrings for the
translation catalog. pygettext has the ability to locate and extract
docstrings and it was introduced to acquire the documentation for our
commands located in module and class docstrings.
However pygettext was too large a hammer for this task, it lacked any
fined grained ability to extract only the docstrings we were
interested in. In practice it extracted EVERY docstring in each file
it was presented with. This caused a large number strings to be
extracted for translation which had no reason to be translated, the
string might have been internal code documentation never meant to be
seen by users. Often the superfluous docstrings were long, complex and
likely difficult to translate. This placed an unnecessary burden on
our volunteer translators.
Instead what is needed is some method to extract only those strings
intended for translation. We already have such a mechanism and it is
already widely used, namely wrapping strings intended for translation
in calls to _() or _negettext(), i.e. marking a string for i18n
translation. Thus the solution to the docstring translation problem is
to mark the docstrings exactly as we have been doing, it only requires
that instead of a bare Python docstring we instead assign the marked
string to the __doc__ variable. Using the hypothetical class foo as
an example.
class foo(Command):
'''
The foo command takes out the garbage.
'''
Would become:
class foo(Command):
__doc__ = _('The foo command takes out the garbage.')
But which docstrings need to be marked for translation? The makeapi
tool knows how to iterate over every command in our public API. It was
extended to validate every command's documentation and report if any
documentation is missing or not marked for translation. That
information was then used to identify each docstring in the code which
needed to be transformed.
In summary what this patch does is:
* Remove the use of pygettext (modification to install/po/Makefile.in)
* Replace every docstring with an explicit assignment to __doc__ where
the rhs of the assignment is an i18n marking function.
* Single line docstrings appearing in multi-line string literals
(e.g. ''' or """) were replaced with single line string literals
because the multi-line literals were introducing unnecessary
whitespace and newlines in the string extracted for translation. For
example:
'''
The foo command takes out the garbage.
'''
Would appear in the translation catalog as:
"\n
The foo command takes out the garbage.\n
"
The superfluous whitespace and newlines are confusing to translators
and requires us to strip leading and trailing whitespace from the
translation at run time.
* Import statements were moved from below the docstring to above
it. This was necessary because the i18n markers are imported
functions and must be available before the the doc is
parsed. Technically only the import of the i18n markers had to
appear before the doc but stylistically it's better to keep all the
imports together.
* It was observed during the docstring editing process that the
command documentation was inconsistent with respect to the use of
periods to terminate a sentence. Some doc had a trailing period,
others didn't. Consistency was enforced by adding a period to end of
every docstring if one was missing.
In cli.py is a framework for printing out help information. The
command documentation being displayed is internationalized, however
the text generated by the help framework itself is not
internationalized.
The strings output by the help subsystem need to be internationalized.
* Wrap each topic description in _()
* Replace the use of if 'topic' in dir(module) with the more
Pythonic and efficient getattr(module, 'topic', None)
* Make sure to invoke unicode on the value returned from _()
otherwise you'll get a GettextFactory instance, not a string
* Clean up trailing whitespace errors
Iterate over all API commands and perform the following validation:
* Every command must have documentation
and it must be marked for international translation
* Every module hosting a command must have documentation
and it must be marked for international translation
* Every module topic must be marked for international translation
For every error found emit a diagnostic.
Emit a summary of total errors found.
Return error flag if errors found, zero otherwise.
ticket 1650 (https://fedorahosted.org/freeipa/ticket/1650) has
an extensive discussion of the issues, please refer to that.
This patch does the following:
* does not count fuzzy translations when computing translation
statistics via the "msg-stats" make target in install/po
* adds a new make target called "pull-po" which pulls updated po files
from Transifex (configure.ac includes some trailing whitespace fixes)
* turns off the generation of fuzzy translation suggestions during the
message merge phase.
Our LINGUAS file and the set of po files have diverged from what's on
Transifex. We should update the LINGUAS file to match the set of
translations on Transifex and add po files currently on Transifex but
not in our git repo to our git repo.
dns.py at line 976 has an invalid i18n string and cannot be processed
during message extraction causing message catalog generation to fail.
The format parameters are trapped inside the i18n string. Also it's
not necessary to promote the i18n string literal to unicode via the u
prefix because the _() function returns unicode.
We have a larger goal of replacing all DN creation via string
formatting/concatenation with DN object operations because string
operations are not a safe way to form a DN nor to compare a DN. This
work needs to be broken into smaller chunks for easier review and
testing.
Addressing the unit tests first makes sense because we don't want to
be modifying both the core code and the tests used to verify the core
code simultaneously. If we modify the unittests first with existing
core code and no regressions are found then we can move on to
modifying parts of the core code with the belief the unittests can
validate the changes in the core code. Also by doing the unittests
first we also help to validate the DN objects are working correctly
(although they do have an extensive unittest).
The fundamental changes are:
* replace string substitution & concatenation with DN object
constructor
* when comparing dn's the comparision is done after promotion
to a DN object, then two DN objects are compared
* when a list of string dn's are to be compared a new list is
formed where each string dn is replaced by a DN object
* because the unittest framework accepts a complex data structure of
expected values where dn's are represeted as strings the unittest
needs to express the expected value of a dn as a callable object
(e.g. a lambda expression) which promotes the dn string to a DN
object in order to do the comparision.
The DN unittest was lacking a test for i18n. The unittest was
updated to store "Hello" in Arabic with both utf-8 and unicode
and verify the values could be properly retrieved and converted
to dn string syntax.
During the testing a few problems were discovered and corrected.
* passing in utf-8 caused an ASCII decode error becuase of Python's
silly default encoding of ASCII. The fix was to explictly use
the utf-8 codec.
* there were a couple of places where encode/decode were not
called correctly.
* the internal attr and value members of the AVA class were renamed
to explicitly show they are stored as unicode.
Of course the unittest was updated as well.
Pull the new translations for Spanish (es) and Ukrainian (uk)
Update the LINGUAS file to add comment showing the friendly
name for the language abbreviation.
The make target msg-stats which produces a report about the state
of the translations no longer maintained it's column alignment
due to larger numbers so the formating was tweaked to maintain
column alignment.
Replace deepcopy with constructor (i.e. type call)
Can now "clone" with configuration changes by passing object
of the same type to it's constructor, e.g.
dn1 = DN(('cn', 'foo'))
dn2 = DN(dn1)
dn2 = DN(dn1, first_key_match=False)
Remove pairwise grouping for RDN's. Had previously removed it
for DN's, left it in for RDN's because it seemed to make sense
because of the way RDN's work but consistency is a higher goal.
Add keyword constructor parameters to pass configuration options.
Make first_key_match a configuration keyword.
Updated documentation.
Updated unit test.
FWIW, I noticed the unittest is now running 2x faster, not sure why,
removal of deepcopy? Anyway, hard to argue with performance doubling.
The pairwise grouping used to form RDN's and AVA's proved to be
confusing in practice, this patch removes that functionality thus
requiring programmers to explicitly pair attr,value using a tuple or
list.
In addition it was discovered additional functionality was needed to
support some DN operations in freeipa. DN objects now support
startswith(), endswith() and the "in" membership test. These functions
and operators will accept either a DN or RDN.
The unittest was modified to remove the pairwise tests and add new
explicit tests. The unittest was augmented to test the new
functionality. In addition the unittest was cleaned up a bit to use
common utilty functions for improved readabilty and robustness.
The documentation was updated.
fix test_role_plugin use of DN to avoid pairwise grouping
Introduce a comma into a privilege name to assure we can handle
commas.
Commas must be escaped for some parameters, add escape_comma() utility
and invoke it for the necessary parameters.
Utilize a DN object to properly construct a DN and most importantly to
allow equality testing beween the DN we expect and the one
returned. This is necessary because a DN can be encoded according to
different encoding syntaxes all of which are valid. DN objects always
decode from their input. DN objects can test for equality between DN's
without being affected by DN encoding.
Add a equality callback for the dn in the expected dict. When the test
framework tests for equality between the expected value and the
returned value it will call back into a function we provide which will
convert the returned dn into a DN object. An equality test is then
performed between two DN objects. This is the only way to properly
compare two dn's.
Do not call status after pkisilent, it will return non-zero.
Instead restart server after pkisilent so configuration
changes take effect, the check the status.
We want to manually make the .pot file, we shouldn't have anything
in the Makefile which will cause the .pot file to be rebuilt
because of dependencies.
A number of doc strings were not localized, wrap them in _().
Some messages were not localized, wrap them in _()
Fix a couple of failing tests:
The method name in RPC should not be unicode.
The doc attribute must use the .msg attribute for comparison.
Also clean up imports of _() The import should come from
ipalib or ipalib.text, not ugettext from request.
As a consequence of using doc=_('some message') the _()
method was returning a Gettext instance, thus when optparse
was handed the help text it received a Gettext instance instead
of a basestring. optparse tried to operate on the Gettext instance
as if it were a basestring and since it wasn't threw an exception.
The fix is to promote (e.g. cast) the option.doc to unicode.
If the option.doc was a str it becomes unicode, if it was unicode
nothing happens, if it was Gettext (or any other object implementing
the __unicode__() method) object is converted to unicode via the
objects rules.
By the way, while debugging this I discovered strings which were not
localized, sometimes option.doc would be a str and sometimes a Gettext
object. In a subsequent patch I'll fix all those unlocalized doc
strings, but I don't want to bury this fix along with a load of
string fixes.
Traverse the objects passed to JSON for encoding and decoding.
When binary data is seen during encode replace the binary
data with a dict {'__base64__' : base64_encoding_of_binary_value}.
On decode if a dict is seen whose single key is '__base64__' replace
that dict with the base64 decoded value of the key's value.
There was a typo in install/po/Makefile.in which caused (some) of
the .po files to be overwritten because the test to see if a po
file existed had a typo in it.
This patch also removes the unnecessary rebuilding of the pot which was
happening when using the "all" target (the default). The pot file now
must be manually remade, which is what we want.
Added a new target "mo-files" to manually generate the .mo files.
This is useful to run before checking in a new .po file just to
assure it "compiles" and we don't have to discover this during a
build.
Formerly running 'make update-pot' would write an insignificantly
different pot file even if the msgid's xgettext found were unmodfied.
Now the result of running xgettext is compared to the existing
pot file after adjusting for things like timestamps, and only
copies the result of xgettext to the new pot file if there
were differences.
This will help eliminate git commits on the pot file if all one
did was see if the pot file was up to date, if it was up to date
git won't see any modifications. It used to be that timestamps
would be different in the pot file just by virtue of checking
if the pot file was current.
fix exit status; replace POT with $(DOMAN).pot
Add new Russian translation.
Update the Polish translation.
Add count of how many po translations we have in msg-stats.
Current translation statistics with this patch:
ipa.pot has 133 messages. There are 5 po translation files.
bn_IN: 14/133 10.5% 106 po untranslated, 13 missing, 119 untranslated
id: 107/133 80.5% 13 po untranslated, 13 missing, 26 untranslated
kn: 4/133 3.0% 116 po untranslated, 13 missing, 129 untranslated
pl: 133/133 100.0% 0 po untranslated, 0 missing, 0 untranslated
ru: 120/133 90.2% 0 po untranslated, 13 missing, 13 untranslated
The goal is to get the statistics to line up in
columns and not exceed an 80 character line which might
cause wrapping. Removing .po suffix from the translation
name gives us 3 extra characters. Formatting problems were
observed when bn_IN.po was added.
The Makefile in install/po has a new target "msg-stats" which
prints out statistics concerning the current pot and po files.
Here is an example:
% make msg-stats
ipa.pot has 133 messages
id.po: 107/133 80.5% 13 po untranslated, 13 missing, 26 untranslated
kn.po: 4/133 3.0% 116 po untranslated, 13 missing, 129 untranslated
pl.po: 120/133 90.2% 0 po untranslated, 13 missing, 13 untranslated
Also update configure.ac to search for msgcmp, awk & sed programs.
Add 3 new translations for Polish, Indonesian, Kannada.
Remove the dummy Italian translation (it.po), it's only purpose was to
have some translation in the tarball so %{find_lang} wouldn't fail.
Add a new file contributing_translators.txt so we can track our
translators contributions and give them credit.
The %{find_lang} macro in the ipa.spec file will fail if there
are no language translations causing the build to fail.
This patch creates an Italian translation (in honor of Simo).
The Italian translation is initially fully untranslated.
msginit should have been passed the locale because the resulting
.po file is parameterized from the locale. Also, if the target
locale is not specified it defaults to the current locale.
If the target locale is Engish msgid's are copied to their msgstr's
resulting in a fully translated .po instead of a fully untranslated
.po.
Add some comments to better explain some of the cryptic sed commands.
A new directory install/po has been added which contains all
the translations for all files in IPA.
The build has been agumented to build these files. Also the
autogen.sh script was mostly replaced by autoreconf, the preferred
method. The old autogen.sh sript also had some serious bugs in the
way it compared versions which caused it to run old versions of some
of the tools, using standared autoreconf is much better.
This error message was producing a warning from xgettext
because there were multiple substations in the string.
In some languages it may be necessary to reorder the
substitutions for a proper translation, this is only
possible if the substitutions use named values.
Remove SAFE_STRING_PATTERN, safe_string_re, needs_base64(),
wrap_binary_data(), unwrap_binary_data() from both instances
of ipautil.py. This code is no longer in use and the
SAFE_STRING_PATTERN regular expression string was causing xgettext
to abort because it wasn't a valid ASCII string.
This modifies the original patch by including a unit test, handling floats
when passed as unicode, and handling large magnitude values beyond maxint.
The INT parameter class was not respecting any radix prefix (e.g. 0x) the user
may have supplied. This patch implements _convert_scalar method for the Int
class so that we can pass the special radix base of zero to the int constructor
telling it to determine the radix from the prefix (if present).
Signed-off-by: John Dennis <jdennis@redhat.com>
along with stdout,stderr also log the initial command
This implements better logging of external commands. Formerly we were
just outputting stdout and stderr without labeling which was which. We
also omitted the initial command and it's arguments. This made it
difficult when reviewing the logs to know what the command was and
what was stdout vs. stderr. This patch fixes that.
Signed-off-by: John Dennis <jdennis@redhat.com>
If plugin fails to load log the traceback
If a plugin fails to load due to some kind of error it would be nice
if the error log contained the traceback so you can examine what went
wrong rather than being left blind as to why it failed to load.
profile command line tools to select between shared
and per user profiles
modify AttributeValueCompleter so default values prefer
previously entered values in editing session
get_radius_profile_by_uid
add_radius_profile
update_radius_profile
delete_radius_profile
find_radius_profiles
Rewrite command line arg handling, now support pair entry, interactive
mode with auto completion, reading pairs from a file, better handling
of mandatory values, better help, long arg names now match attribute
name in pairs
Establish mappings for all attributes and names used in clients and
profiles
Add notion of containers to radius clients and profiles in LDAP
Move common code, variables, constants, and strings into the files
radius_client.py, radius_util.py, ipautil.py to eliminate redundant
elements which could get out of sync if modified and to provide access
to other code which might benefit from using these items in the
future.
Add utility functions:
format_list()
parse_key_value_pairs()
Add utility class:
AttributeValueCompleter
Unify attribute usage in radius ldap schema
John Dennis
jdennis@VAIO