The code split the permission string on commas, essentially doing
poor man's CSV parsing. So if a permission contained a
comma-separated list of valid permissions, validation would pass
but we'd get errors later.
https://fedorahosted.org/freeipa/ticket/3420
This is to prevent a fatal name clash wih the new common "messages" Output.
Since i18n_messages is an internal plugin, the change does not affect
our public API.
The API version the client sends can now be used to check what the client
expects or is capable of.
All version tests IPA does will be be named and listed in one module,
ipalib.capabilities, which includes a function to test a specific capability
against an API version.
Similarly to Python's __future__ module, capabilities.py also serves as
documentation of backwards-incompatible changes to the API.
The first capability to be defined is "messages". Recent enough clients can
accept a list of warnings or other info under the "messages" key in the
result dict.
If a JSON client does not send the API version, it is assumed this is a testing
client (e.g. curl from the command line). Such a client "has" all capabilities,
but it will always receive a warning mentioning that forward compatibility
is not guaranteed.
If a XML client does not send the API version, it is assumed it uses the API
version before capabilities were introduced. (This is to keep backwards
compatibility with clients containing bug https://fedorahosted.org/freeipa/ticket/3294)
Whenever a capability is added, the API version must be incremented.
To ensure that, capabilities are written to API.txt and checked by
`makeapi --validate`.
Design page: http://freeipa.org/page/V3/Messages
Ticket: https://fedorahosted.org/freeipa/ticket/2732
The messages module contains message classes that can be added
to a RPC response to provide additional information or warnings.
This patch adds only the module with a single public message,
VersionMissing, and unit tests.
Since message classes are very similar to public errors, some
functionality and unit tests were shared.
Design page: http://freeipa.org/page/V3/Messages
Ticket: https://fedorahosted.org/freeipa/ticket/2732
Several Commands were missing the 'version' option. Add it to those
that were missing it.
Do not remove the version option before calling commands. This means
methods such as execute(), forward(), run() receive it.
Several of these needed `**options` added to their signatures.
Commands in the Cert plugin passed any unknown options to the underlying
functions, these are changed to pass what's needed explicitly.
Some commands in DNS and Batch plugins now pass version to commands
they call.
When the option is not given, fill it in automatically. (In a subsequent
commit, a warning will be added in this case).
Note that the public API did not change: all RPC calls already accepted
a version option. There's no need for an API version bump (even though
API.txt changes substantially).
Design page: http://freeipa.org/page/V3/Messages
Tickets:
https://fedorahosted.org/freeipa/ticket/2732https://fedorahosted.org/freeipa/ticket/3294
Sudo commands created in the past have the sudocmd in their RDN, while
the new case-sensitive ones have ipaUniqueID. In order for permissions
to apply to both of these, use a targetfilter for objectclass=ipasudocmd
instead of sudocmd=* in the target.
Since it is not really possible to separate SSH errors from
errors of the called program, add a SSH check before
calling replica-conncheck on the master.
The check also adds the master to a temporary known_hosts file,
so suppressing SSH's warning about unknown host is no longer
necessary. If the "real" connection fails despite the check,
any SSH errors will be included in the output.
https://fedorahosted.org/freeipa/ticket/3402
Move the parser setup from bootstrap_with_global_options to bootstrap,
so all API objects have access to it.
Add some CLI tests for the help system.
Part of the effort for https://fedorahosted.org/freeipa/ticket/3060
This avoids the problem with ambiguous command/topic names.
No functionality is changed; `ipa help <COMMAND>` still works as before
if there's no topic with the same name.
https://fedorahosted.org/freeipa/ticket/3247
Make `ipa -h` and `ipa help` output the same message.
Since `ipa -h` output is generated by the OptionParser, we need to make
the parser available. Store it in `api.parser`.
Part of the effort for https://fedorahosted.org/freeipa/ticket/3060
Whenever a command is used incorrectly, it should output
an error message (and possibly additional help) to stderr.
This patch adds a parameter to a bunch of places to allow
selecting either stdout or stderr for help output, and makes
badly called commands output to stderr only.
Part of the effort for https://fedorahosted.org/freeipa/ticket/3060
Fix the usage string to match actual usage.
Add command description.
Put information about `ipa help topics` etc. to the epilog,
instead of using empty option groups. Use a custom formatter
to preserve newlines.
Add the -h/--help option manually to ensure consistent case
(capital S).
Part of the effort for https://fedorahosted.org/freeipa/ticket/3060
Modify update file to use default: rather than add: in
cn=replication,cn=etc,$SUFFIX.
Drop quotes around nsDS5ReplicaRoot because default: values
are not parsed as CSV.
https://fedorahosted.org/freeipa/ticket/3394
- Automatically add a "Logging and output options" group with the --quiet,
--verbose, --log-file options.
- Set up logging based on these options; details are in the setup_logging
docstring and in the design document.
- Don't bind log methods as individual methods of the class. This means one
less linter exception.
- Make the help for command line options consistent with optparse's --help and
--version options.
Design document: http://freeipa.org/page/V3/Logging_and_output
The DS is installed before the CA cert is generated. Trying to
add the cert to LDAP before it exists resulted in a nasty-looking
error message.
This moves the cert upload to after the CA cert is ready and the
certdb is created.
Move the cert upload to after thecertdb is generated.
https://fedorahosted.org/freeipa/ticket/3375
Python does not guarantee dict order. Our reliance on it in `makeapi`
is technically incorrect, even if it doesn't matter yet in any of our
developers' environments.
This patch sorts the options by name instead of relying on dict order.
As an added benefit, future API.txt diffs should be be neater.
The ipa-csreplica-manage tool often assumed that the port numbers are the
same on both sides of a replication agreement.
This assumption doesn't hold in a cluster with both old-style hosts and
ones with merged DBs.
When managing agreements, determine the port with the PKI (or merged) DS
on each master, and use it.
Also, in CSReplicationManager, always use starttls rather than ldaps://.
The ancient IPAdmin class used some heuristics to determine the protocol
to connect with (ldap, ldaps, or ldapi). In turn, some calling code used
questionable mechanisms to get the correct protocol: in
ipaserver/install/replication.py, the CA cert was either passed to the
constructor or added to the class afterwards, to get ldap:// or ldaps://.
Add an explicit protocol argument and only fall back to backwards-compatible
guessing if it is not given.
Stopping certificate tracking was done as part of the PKI DS uninstall.
Since with the merged DB, thePKI DS is not used any more, this step
was skipped.
Move certificate untracking to a separate step and call it separately.
Also, the post-uninstall check for tracked certificates used the wrong
set of Dogtag constants. Fix the issue.
This brings /etc/dirsrv/slapd-REALM/certmap.conf under IPA control.
The file is overwritten on upgrades.
This ensures that the cert for the ipaca user is recognized when
ipa-ca-install is run on older masters.
The new merged database will replicate with both the IPA and CA trees, so all
DS instances (IPA and CA on the existing master, and the merged one on the
replica) need to have the same schema.
Dogtag does all its schema modifications online. Those are replicated normally.
The basic IPA schema, however, is delivered in ldif files, which are not
replicated. The files are not present on old CA DS instances. Any schema
update that references objects in these files will fail.
The whole 99user.ldif (i.e. changes introduced dynamically over LDAP) is
replicated as a blob. If we updated the old master's CA schema dynamically
during replica install, it would conflict with updates done during the
installation: the one with the lower CSN would get lost.
Dogtag's spawn script recently grew a new flag, 'pki_clone_replicate_schema'.
Turning it off tells Dogtag to create its schema in the clone, where the IPA
modifications are taking place, so that it is not overwritten by the IPA schema
on replication.
The patch solves the problems by:
- In __spawn_instance, turning off the pki_clone_replicate_schema flag.
- Providing a script to copy the IPA schema files to the CA DS instance.
The script needs to be copied to old masters and run there.
- At replica CA install, checking if the schema is updated, and failing if not.
The --skip-schema-check option is added to ipa-{replica,ca}-install to
override the check.
All pre-3.1 CA servers in a domain will have to have the script run on them to
avoid schema replication errors.
https://fedorahosted.org/freeipa/ticket/3213
- Provide a function for determinig the CA status using Dogtag 10's new
getStatus endpoint.
This must be done over HTTPS, but since our client certificate may not be set
up yet, we need HTTPS without client authentication.
Rather than copying from the existing http_request and https_request
function, shared code is factored out to a common helper.
- Call the new function when restarting the CA service. Since our Service
can only be extended in platform-specific code, do this for Fedora only.
Also, the status is only checked with Dogtag 10+.
- When a restart call in cainstance failed, users were refered to the
installation log, but no info was actually logged. Log the exception.
https://fedorahosted.org/freeipa/ticket/3084
Previously, ipa-replica-install tried to check DNS resolution on the master
being cloned. If that master was not a DNS server, the check failed.
Change the check to query the first available configured DNS server.
Log about the check before actually running it.
Log in the case the check is skipped (no IPA DNS servers installed).
https://fedorahosted.org/freeipa/ticket/3194
When the user interrupts a long-running command, this ensures that
the command is logged. Also, when watching log files (or the -d
output), it's apparent what's being done.
https://fedorahosted.org/freeipa/ticket/3174
Dogtag opens not only the insecure port (8080 or 9180, for d10 and
d9 respectively), but also secure ports (8443 or 9443&9444).
Wait for them when starting.
Part of the fix for https://fedorahosted.org/freeipa/ticket/3084
Our installation added two final dots to the NS records,
so the records were invalid, Bind ignored the entire zone,
and name resolution didn't work.
Fix this error and add a check for empty DNS labels to the validator
Instead of `except:`, use `except Exception:`. This means that errors
like KeyboardInterrupt are not handled, letting them terminate the
script as expected.
https://fedorahosted.org/freeipa/ticket/2941
When there were no updated attrs when modifying a nonexistent DNS record,
the error was not handled and caused an internal server error later (old_entry
was used uninitialized).
https://fedorahosted.org/freeipa/ticket/3055
Forward and reverse resolution of the newly created replica is already
checked via get_host_name (which calls verify_fqdn).
Add the same check for the existing master.
Additionally, if DNS is installed on the remote host, check forward
and reverse resolution of both replicas using that DNS only
(ignoring /etc/hosts). These checks give only warnings and, in interactive
installs, a "Continue?" prompt.
https://fedorahosted.org/freeipa/ticket/2845
We've been stopping both DS instances (main and PKI) when upgrading.
This can happen while the CA is running. In some cases stopping the PKI
DS also killed the CA.
Only stop the specific instance for upgrades.
Also, wait for open ports after the upgrade is complete. The wait was
skipped previously. This can prevent bugs if scripts that need a DS are
run after the upgrade.
https://fedorahosted.org/freeipa/ticket/3083
Put the changes from Ade's dogtag 10 patch into namespaced constants in
dogtag.py, which are then referenced in the code.
Make ipaserver.install.CAInstance use the service name specified in the
configuration. Uninstallation, where config is removed before CA uninstall,
also uses the (previously) configured value.
This and Ade's patch address https://fedorahosted.org/freeipa/ticket/2846
When clients install, they use kinit to obtain a TGT, which uses DNS to find
the KDC to connect to. It might happen that the newly created principal
has not replicated to selected KDC yet, making kinit fail and aborting the
install.
The client sets a temporary krb5 config file while installing via $KRB5_CONFIG.
Modify this file so that the kerberos library only uses the specific server
we're installing under, and call kinit while it's still in place.
Clean up the configure_krb5_conf function to remove unused arguments. For
clarity, use keyword arguments when calling it.
https://fedorahosted.org/freeipa/ticket/2982
Generalize the fix_replica_memberof update plugin to allow updating more
replication attributes.
Add nsds5ReplicaStripAttrs to replication agreements on update and
replica install.
https://fedorahosted.org/freeipa/ticket/2534
Currently, we throw many public exceptions without proper i18n.
Wrap natural-language error messages in _() so they can be translated.
In the service plugin, raise NotFound errors using handle_not_found helper
so the error message contains the offending service.
Use ScriptError instead of NotFoundError in bindinstance install.
https://fedorahosted.org/freeipa/ticket/1953
When the --hostname option is given to ipa-client-install, we
write HOSTNAME to /etc/sysconfig/network. When that file didn't exist,
the installer crashed.
Create the file if it doesn't exist and we need to write to it.
https://fedorahosted.org/freeipa/ticket/2840
Update the pot file to match current source, and pull
translations from Transifex
The commands used to create this commit were:
tx pull -fa # needs an account and transifex client setup
cd install/po
make update-pot
(update LINGUAS to contain all languages with translations, and no empty ones)
make update-po
The .po files we use for translations have two shortcomings when used in Git:
- They include file locations, which change each time the source is updated.
This results in large, unreadable diffs that don't merge well.
- They include source strings for untranslated messages, wasting space
unnecessarily.
Update the Makefile so that the extraneous information is stripped when the
files are updated or pulled form Transifex, and empty translation files are
removed entirely.
Also, translations are normalized to a common style. This should help diffs
and merges.
The validator requires file location comments to identify the programming
language, and to produce good error reports.
To make this work, merge the comments in before validation.
First patch for: https://fedorahosted.org/freeipa/ticket/2435
Currently, FreeIPA's install/admin scripts are long pieces of code
that aren't very reusable, importable, or testable.
They have been extended over time with features such as logging and
error handling, but since each tool was extended individually, there
is much inconsistency and code duplication.
This patch starts a framework which the admin tools can use, and
converts ipa-ldap-updater to use the framework.
Common tasks the tools do -- option parsing, validation, logging
setup, error handling -- are represented as methods. Individual
tools can extend, override or reuse the defaults as they see fit.
The ipa-ldap-updater has two modes (normal and --upgrade) that
don't share much functionality. They are represented by separate
classes. Option parsing, and selecting which class to run, happens
before they're instantiated.
All code is moved to importable modules to aid future testing. The
only thing that remains in the ipa-ldap-updater script is a two-line
call to the library.
First part of the work for:
https://fedorahosted.org/freeipa/ticket/2652
The upgrade script set the "psearch" directive in some circumstances,
but did not remember that it was set, so later, when setting
minimum_connections, it assumed psearch is not set.
Also, the script did not set minimum_connections if the directive wasn't
already there. It should be set in that case.
Related to https://fedorahosted.org/freeipa/ticket/2554
The Batch command did not report errors correctly: it reported
the text of *all* errors, not just PublicError, used unicode(e)
instead of e.strerror (which results in incorrect i18n), and only
reported the text of error messages, not their type and code.
Fix these problems. Update tests.
https://fedorahosted.org/freeipa/ticket/2874https://fedorahosted.org/freeipa/ticket/2901
Make permission commands not pass options that the underlying ACI commands
do not understand.
Update tests.
Remove some extraneous imports of the `copy` module.
https://fedorahosted.org/freeipa/ticket/2885
We only checked the length of Command output dictionaries. A misspelled
key in would not be caught.
Fix the problem by checking if the sets of keys are equal.
Add a test. Split the test methods into more manageable pieces.
https://fedorahosted.org/freeipa/ticket/2860
When unknown keyword arguments are passed to a Command, raise an
error instead of ignoring them.
Options used when IPA calls its commands internally are listed
in a new Command attribute called internal_options, and allowed.
Previous patches (0b01751c, c45174d6, c5689e7f) made IPA not use
unknown keyword arguments in its own commands and tests, but since
that some violations were reintroduced in permission_find and tests.
Fix those.
Tests included; both a frontend unittest and a XML-RPC test via the
ping plugin (which was untested previously).
https://fedorahosted.org/freeipa/ticket/2509
Track the source of discovered values (e.g. from option, interactive,
retrieved from DNS), and show it in the log in the configuration
overview and on erorrs involving the value.
Add additional log messages explaining the autodiscovery process.
For domains the discovery tries to get LDAP SRV records from, log
reasons explaining why the domain was chosen. Also, prevent the
same domain from being searched multiple times.
Add names for error codes, and show them in the log.
Also, modernize the discovery code a bit: move away from the
Java-style accessors, don't needlessly pre-declare variables, make
IPADiscovery a new-style class.
https://fedorahosted.org/freeipa/ticket/2553
The client does a fair bit of work when trying to validate the hostnames,
do discovery and verify that the server it gets back is an IPA server.
The debug logging around this was horrid with very little state information,
duplicate log messages or just nothing at all.
In many cases errors were printed only to stderr/stdout.
This patch makes the logging and output go through the IPA log manager.
It sets up logging so that INFO, WARNING, and ERROR messages show up on the
console. If -d is given, DEBUG messages are also printed.
All messages also go to the log file.
The only exception is user input: prompts are only printed to the console,
but if the user provides any information it is echoed in a DEBUG-level
message.
https://fedorahosted.org/freeipa/ticket/2553
Fix several problems with the callback interface:
- Automatically registered callbacks (i.e. methods named
exc_callback, pre_callback etc) were registered on every
instantiation.
Fix: Do not register callbacks in __init__; instead return the
method when asked for it.
- The calling code had to distinguish between bound methods and
plain functions by checking the 'im_self' attribute.
Fix: Always return the "default" callback as an unbound method.
Registered callbacks now always take the extra `self` argument,
whether they happen to be bound methods or not.
Calling code now always needs to pass the `self` argument.
- Did not work well with inheritance: due to the fact that Python
looks up missing attributes in superclasses, callbacks could
get attached to a superclass if it was instantiated early enough. *
Fix: Instead of attribute lookup, use a dictionary with class keys.
- The interface included the callback types, which are LDAP-specific.
Fix: Create generic register_callback and get_callback mehods,
move LDAP-specific code to BaseLDAPCommand
Update code that calls the callbacks.
Add tests.
Remove lint exceptions for CallbackInterface.
* https://fedorahosted.org/freeipa/ticket/2674
Sort a filtered list in the update plugin ordering method.
Unlike the previous algorithm, this always gives a correct order.
It should also be faster and more readable.
https://fedorahosted.org/freeipa/ticket/2820
In ipa-client-install (which is also called from server/replica
installation), call `ipa-rmkeytab -k <keytab> -r $REALM` to be
sure that there aren't any remnants from a previous install of
IPA or another KDC altogether.
https://fedorahosted.org/freeipa/ticket/2698
Several plugins need restarting the DS (or they currently do
an external bind).
Rather than disabling plugins (possibly partially), refuse
to run them when run as an unprivileged user.
This means running ipa-ldap-updater as non-root requires specifying
a list of files, and omiting the --upgrade and --plugins options.
https://fedorahosted.org/freeipa/ticket/2621
Raise an error when trying to delete the last user in the
'admins' group, or remove the last member from the group,
or delete the group itself.
https://fedorahosted.org/freeipa/ticket/2564
This adds tests for the automountlocation_tofiles and
automountlocation_import commands, and to automountmap_add_indirect
with the --parentmap option.
The tofiles test checks not only the XML-RPC output, but also the
output_for_cli method.
The import tests load data from tofiles output to the directory
and check that tofiles output matches.
This only works when all maps are connected to auto.master.
Two minor touches to the automount plugin itself: remove an extra
space, and don't hide the traceback when re-raising an exception.
All of our install/admin scripts had a try/except block calling the
main function and handling common exceptions. These were copy-pasted
from each other and modified to various levels of sophistication.
This refactors them out of installers to a single function, which
includes a final pass/fail message for all of the scripts.
Non-install scripts that set up the same log handler levels for
stderr and log file are not changed, as it's not possible to log
to only the logfile without changing the logger configuration.
https://fedorahosted.org/freeipa/ticket/2071
If --delattr is used on an attribute that's not present on an entry,
and --{set,add}attr isn't being used on that same attribute,
say that there's "no such attribute" instead of "<attribute> does
not contain <value>".
https://fedorahosted.org/freeipa/ticket/2699
Make --{set,add,del}attr fail on parameters with the no_update/no_create
flag for the respective command.
For attributes that can be modified, but we just don't want to display
in the CLI, use the 'no_option' flag. These are "locking" attributes
(ipaenabledflag, nsaccountlock) and externalhost.
Document the 'no_option' flag. Add some tests.
https://fedorahosted.org/freeipa/ticket/2580
Always call convert_time_for_output so time gets reported correctly.
That method has its own checks for whether the attributes are present;
an additional check is unnecessary.
Use a key function for sorting; cmp is deprecated, slower and
more complicated.
Add a test
https://fedorahosted.org/freeipa/ticket/2726
Allowing Commands to be called with ignored unknown options opens the
door to problems, for example with misspelled option names.
Before we start rejecting them, we need to make sure IPA itself does
not use them when it calls commands internally.
This patch does that for ACI-related plugins.
Part of the work for https://fedorahosted.org/freeipa/ticket/2509
Change the externalhost attribute of hbacrule, netgroup
and sudorule into a full-fledged Parameter, and attach
a validator to it.
The validator is relaxed to allow underscores, so that
some hosts with nonstandard names can be added.
Tests included.
https://fedorahosted.org/freeipa/ticket/2649
IPA has some unused code from abandoned features (Radius, ipa 1.x user
input, commant-line tab completion), as well as some duplicate utilities.
This patch cleans up the utility modules.
Duplicate code consolidated into ipapython.ipautil:
{ipalib.util,ipaserver.ipautil,ipapython.ipautil}.realm_to_suffix
{ipaserver,ipapython}.ipautil.CIDict
(with style improvements from the ipaserver version)
{ipapython.entity,ipaserver.ipautil}.utf8_encode_value
{ipapython.entity,ipaserver.ipautil}.utf8_encode_values
ipalib.util.get_fqdn was removed in favor of the same function in
ipaserver.install.installutils
Removed unused code:
ipalib.util:
load_plugins_in_dir
import_plugins_subpackage
make_repr (was imported but unused; also removed from tests)
ipapython.ipautil:
format_list
parse_key_value_pairs
read_pairs_file
read_items_file
user_input_plain
AttributeValueCompleter
ItemCompleter
ipaserver.ipautil:
get_gsserror (a different version exists in ipapython.ipautil)
ipaserver.ipautil ended up empty and is removed entirely.
https://fedorahosted.org/freeipa/ticket/2650
The config plugin was adding together a list and a tuple, then
converting to a set.
Replace the operation with a set union.
Regression test included.
https://fedorahosted.org/freeipa/ticket/2706
Calling a LDAP{Add,Remove}ReverseMember with an empty reverse_member
caused an internal error, because empty values are converted to None,
which is then iterated.
Use an empty list instead of None (or other false falues, of which we
only use the empty list).
https://fedorahosted.org/freeipa/ticket/2681
Allowing Commands to be called with ignored unknown options opens the
door to problems, for example with misspelled option names.
Before we start rejecting them, we need to make sure IPA itself does
not use them when it calls commands internally.
This patch does that for the automount plugin and its tests.
Part of the work for https://fedorahosted.org/freeipa/ticket/2509
Test that `pwpolicy_find --pkey-only` works as expected
Test that deleting a group removes its password policy
Rename the test module to be consistent with other plugin tests.
Currently, our test script forwards a select few command line arguments
to nosetests.
This patch removes the filtering, passing all arguments through.
This allows things like disabling output redirection (--nocapture),
dropping into a debugger (--pdb, --pdb-failures), coverage reporting
(--with-cover, if installed), etc.
https://fedorahosted.org/freeipa/ticket/2135
Have the test suite check error messages.
Since XMLRPC doesn't give us structured error information, just
compare the resulting text.
Fix messages that tests expect to cause.
Minor changes:
Make netgroup-mod's NotFound message consistent with other objects
and methods.
In test_automember_plugin, test with nonexistent automember rules
of both types, instead of nonexistent users.
https://fedorahosted.org/freeipa/ticket/2549
Attribute Patrams marked no_update never get cloned to Update commands,
and thus never receive the `attribute` flag. This makes their `encode`
method a no-op, which meant they don't get properly encoded when used
with --setattr, making the --setattr fail.
Introduce a `force` argument to encode, which overrides checking
for the attribute flag. Use this in set/add/delattr normalization,
where we know we are dealing with attributes.
https://fedorahosted.org/freeipa/ticket/2616
The DN and ACI code doesn't always escape special characters properly.
Rather than trying to fix it, this patch takes the easy way out and
enforces that the names are safe.
https://fedorahosted.org/freeipa/ticket/2585
This adds tests for the batch plugin, and changes its output
declaration to allow results as tuples (this tripped validation).
The assert_deepequal function ignores the order of items in lists.
Document this in its docstring, and use a custom checker for the
batch plugin results.
Petr Viktorin