Currently, only one IPA environment is tested within Docker
containers. This is not efficient because Azure's agent gives
6 GB of physical memory and 13 GB of total memory (Feb 2020),
but limits CPU with 2 cores.
Next examples are for 'master-only' topologies.
Let's assume that only one member of github repo simultaneously
run CI. This allows to get the full strength of Azure.
Concurrency results for TestInstallMaster:
------------------------------------------
| job concurrency | time/jobs |
------------------------------------------
| 5 | 40/5 |
| 4 | 34/4 |
| 3 | 25/3 |
| 2 | 19/2 |
| 1 | 17/1 |
------------------------------------------
Results prove the limitation of 2 cores. So, in case of jobs'
number not exceeds the max capacity for parallel jobs(10) the
proposed method couldn't save time, but it reduces the used
jobs number up to 2 times. In other words, in this case CI
could pass 2 x tests.
But what if CI was triggered by several PRs? or jobs' number is
bigger than 10. For example, there are 20 tests to be run.
Concurrency results for TestInstallMaster and 20 input jobs:
------------------------------------------------------------------
| job concurrency | time | jobs used | jobs free |
------------------------------------------------------------------
| 5 | 40 | 4 | 6 |
| 4 | 34 | 5 | 5 |
| 3 | 25 | 7 | 3 |
| 2 | 19 | 10 | 0 |
| 1 | 34 | 20 | 0 |
------------------------------------------------------------------
So, in this case the optimal concurrency would be 4 since it
allows to run two CIs simultaneously (20 tasks on board) and get
results in 34 minutes for both. In other words, two people could
trigger CI from PR and don't wait for each other.
New Azure IPA tests workflow:
+ 1) generate-matrix.py script generates JSON from user's YAML [0]
2) Azure generate jobs using Matrix strategy
3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) GNU 'parallel' launch each IPA environment in parallel:
+ 1) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ 2) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ 3) launch IPA tests on tests' controller
c) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
d) publish regular system logs as artifacts
[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
This log file is collected by azure-run-tests.sh script and then by
Azure 'PublishPipelineArtifact' task. So, the same file gets into
logs artifact.
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
This allows to run IPA tests on Azure using any distro.
To achieve this, one has to do:
1) place a platform specific template on 'ipatests/azure/templates/'
and make a soft link from 'ipatests/azure/templates/variables.yml' to
the new template.
2) place a configuration templates on these paths
3) templates have to answer the questions such as:
a) which Docker image to use to build IPA packages (rpm, deb, etc.)
b) how to prepare Build environment
c) how to build IPA packages
d) how to prepare environment to run Tox tests
e) how to prepare environment to run WebUI unittests
f) which base Docker image to use to build the new image to run
IPA tests within it
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Azure provides Microsoft-hosted agents having tasty resources [0].
For now (Feb 2020),
- (Linux only) Run steps in a cgroup that offers 6 GB of physical memory and
13 GB of total memory
- Provide at least 10 GB of storage for your source and build outputs.
This is enough to set up IPA environments consisted of not only master but also
replicas and clients and thus, run IPA integration tests.
New Azure IPA tests workflow:
+ 1) Azure generate jobs using Matrix strategy
2) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ c) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ d) launch IPA tests on tests' controller
e) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
f) publish regular system logs as artifacts
[0] https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops
[1] https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
IPA integration tests utilize SSH as a transport to communicate
with IPA hosts. To run such tests Docker environments should
have configured SSH.
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
As for now, a list of tests which will be ignored by Pytest is
mandatory. But actually, a list of tests to run is explicitly set
in yaml config. And thus, 'ignore' list should be an optional field.
This simplifies tests definitions to drop extra stuff.
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
The labels for memberservice_hbacsvc and memberservice_hbacsvcgroup are
only "Services" and "Service Groups" but they should be "HBAC Services"
and "HBAC Service Groups".
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Add a little utility function to get the certmonger status
of a request id on a particular host and wait until it is either
failed on the CA or issued (or times out).
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
This is to ensure added HSM support for FreeIPA. This test adds
certificate with sofhsm token and checks if certmonger is tracking
it.
related : https://pagure.io/certmonger/issue/125
Signed-off-by: Mohammad Rizwan Yusuf <myusuf@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
AES SHA 256 and 384-bit enctypes supported by MIT kerberos but
was not enabled in IPA. This test is to check if these types are
enabled.
related: https://pagure.io/freeipa/issue/8110
Signed-off-by: Mohammad Rizwan Yusuf <myusuf@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Reviewed-By: Francois Cami <fcami@redhat.com>
There was a regression caused in nightly run of test
TestBackupReinstallRestoreWithDNS of test_backup_and_restore
test suite because of PR#3962.
Signed-off-by: Kaleemullah Siddiqui <ksiddiqu@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Tests for ipa-restore behaviour when dns or adtrust
rpm is missing which is required during ipa-restore
https://pagure.io/freeipa/issue/7630
Signed-off-by: Kaleemullah Siddiqui <ksiddiqu@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
The test suite test_trust was missing in nightly definitions
because PR-CI was not able to provision multi-AD topology.
Now that PR-CI is updated, we can start executing this test suite.
It is not reasonable to add it to gating as this suite is
time consuming like other tests requiring provisioning of AD instances.
Signed-off-by: Sergey Orlov <sorlov@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
When there are more external groups than default limit, then
SSSD should fetch all groups.
Related : https://pagure.io/SSSD/sssd/issue/4058
Signed-off-by: Anuja More <amore@redhat.com>
Reviewed-By: Florence Blanc-Renaud <frenaud@redhat.com>
Due to https://pagure.io/SSSD/sssd/issue/4149 ipa-client-automount
fails to remove the ipa_automount_location entry from sssd.conf.
Test that autofs_provider and ipa_automount_location are removed.
Fixes: https://pagure.io/freeipa/issue/8190
Signed-off-by: François Cami <fcami@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Currently, such arguments are eaten by 'ipa-run-tests' script as they
are not quoted.
For example, running ipa-run-tests -k ''
results in the actual invocation would be like as:
['/bin/sh',
'--norc',
'--noprofile',
'-c',
'--',
"/usr/bin/python3 -c 'import sys,pytest;sys.exit(pytest.main())' -o "
'cache_dir=/tmp/pytest-of-root/pytest-12/test_ipa_run_tests_empty_expression0/.pytest_cache '
'--confcutdir=/usr/lib64/python3/site-packages/ipatests -k ']
Note: expressions or marks could be empty as a result of the building
of command line args by more high-level tools, scripts, etc.
So, a short-termed solution is the quotting of zero-length arguments.
Fixes: https://pagure.io/freeipa/issue/8173
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Added changes in topology for test_sssd.py
As in test it needs client also.
Signed-off-by: Anuja More <amore@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Reviewed-By: Sergey Orlov <sorlov@redhat.com>
Reviewed-By: Mohammad Rizwan Yusuf <myusuf@redhat.com>
If group contains @ in group name on AD,
then it should fetch successfully on ipa-client.
Related to: https://bugzilla.redhat.com/1746951
Signed-off-by: Anuja More <amore@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Reviewed-By: Sergey Orlov <sorlov@redhat.com>
Reviewed-By: Mohammad Rizwan Yusuf <myusuf@redhat.com>
This is the first time running Pylint-2.4 over the whole IPA codebase.
```
Pylint on /usr/bin/python is running, please wait ...
internal error with sending report for module ['ipaserver/plugins/serverroles.py']
maximum recursion depth exceeded while calling a Python object
************* Module ipatests.test_integration.base
ipatests/test_integration/base.py:84: [W0125(using-constant-test), IntegrationTest.install] Using a conditional statement with a constant value)
************* Module ipaserver.install.ipa_cacert_manage
ipaserver/install/ipa_cacert_manage.py:522: [R1724(no-else-continue), CACertManage.delete] Unnecessary "elif" after "continue")
```
The latest Pylint (via the Tox task) checks only:
```
{envsitepackagesdir}/ipaclient \
{envsitepackagesdir}/ipalib \
{envsitepackagesdir}/ipapython
```
, while the distro-Pylint runs over all project but it is not fresh.
That's why these warnings/errors weren't exposed before now.
Concerning `internal error`: a fix was accepted by upstream:
https://github.com/PyCQA/pylint/issues/3245, but wasn't released yet.
Until that is done, Pylint just warns.
Related: https://pagure.io/freeipa/issue/8116
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Pylint is a static analysis tool and therefore, couldn't always
analyze dynamic stuff properly. Transformation plugins is a way
to teach Pylint how to handle such cases.
Particularly, with the help of FreeIPA own plugin, it is possible
to tell Pylint about instance fields having a duck-typing nature.
A drawback exposed here is that a static view (Pylint's) of code
should be consistent with an actual one, otherwise, codebase will
be polluted with various skips of pylint checks.
* added missing fields to ipatests.test_integration.base.IntegrationTest
* an attempt is made to clear `no-member` skips for ipatests
* removed no longer needed `pytest` module transformation
Related: https://pagure.io/freeipa/issue/8116
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
There is a race condition exposed in 'test_gpg_asymmetric'.
The teardown of 'tempdir' fixture and gpg-agent being called
from the teardown of 'gpgkey' fixture could simultaneously
remove the gnugpg's socket files.
This results in an error like:
```
================= ERRORS ===================
_ ERROR at teardown of test_gpg_asymmetric __
...
> os.unlink(entry.name, dir_fd=topfd)
E FileNotFoundError: [Errno 2] No such file or directory: 'S.gpg-agent.extra'
/usr/lib64/python3.7/shutil.py:450: FileNotFoundError
```
The problem is that the agent is not terminated properly.
Instead, gpgconf could be used to kill daemonized gpg-agent.
Related: https://pagure.io/freeipa/issue/7989
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
nodejs:12 requires libicu-65.1 while gdb (not direct dependency)
libicu-63.2. As a workaround gdb-minimal [0] could be used.
It's even better as requires less packages to be downloaded
and then installed.
[0] https://fedoraproject.org/wiki/Changes/Minimal_GDB_in_buildroot
Co-authored-by: Stanislav Levin <slev@altlinux.org>
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
This Pytest plugin is intended to issue warnings on collecting
tests, which employ unittest/nose frameworks or xunit style.
For example, this may look like:
"""
test_a/test_xunit.py:25
test_a/test_xunit.py:25: PytestDeprecationWarning: xunit style is deprecated
def test_foo_bar(self):
test_b/test_unittest.py:7
test_b/test_unittest.py:7: PytestDeprecationWarning: unittest is deprecated
def test_foo_bar(self):
"""
To treat these warnings as errors it's enough to run Pytest with:
-W error:'xunit style is deprecated':pytest.PytestDeprecationWarning
Related: https://pagure.io/freeipa/issue/7989
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Even though Pytest supports xunit style setups, unittest and nose
tests, this support is limited and may be dropped in the future
releases. Worst of all is that the mixing of various test
frameworks results in weird conflicts and of course, is not widely
tested.
This is a part of work to remove the mixing of test idioms in the
IPA's test suite:
1) replace unittest.TestCase subclasses
2) replace unittest test controls (SkipTest, fail, etc.)
3) replace unittest assertions
Related: https://pagure.io/freeipa/issue/7989
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Even though Pytest supports xunit style setups, unittest and nose
tests, this support is limited and may be dropped in the future
releases. Worst of all is that the mixing of various test
frameworks results in weird conflicts and of course, is not widely
tested.
This is a part of work to remove the mixing of test idioms in the
IPA's test suite:
1) replace xunit style
2) employ the fixtures' interdependencies
Related: https://pagure.io/freeipa/issue/7989
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
The method modify_sssd_conf() is copying a remote sssd.conf file
to the test controller then uses sssd python API to modify the
config file.
When the test controller does not have sssd-common package installed,
SSSDConfig() call fails because the API needs sssd schema in order
to properly parse the config file, and the schema files are provided
by sssd-common pkg.
The fix also downloads the files representing sssd schema and calls
SSSDConfig() with those files. Using the schema from the test machine
is ensuring that config is consistent with the schema (if the sssd
version differs between controller and test machine for instance).
Note: we currently don't see any issue in the nightly tests because
the test controller is installed with sssd-common package but if you
run the tests as specified in https://www.freeipa.org/page/Testing
with a controller missing sssd-common, you will see the issue.
Reviewed-By: Sergey Orlov <sorlov@redhat.com>
It checks if 'SSH public key fingerprint' is
not present in the output of the command
Related: https://pagure.io/freeipa/issue/8029
Signed-off-by: Sumedh Sidhaye <ssidhaye@redhat.com>
Reviewed-By: Florence Blanc-Renaud <frenaud@redhat.com>
The nightly tests for rawhide and updates_testing are expected
to set
update_packages: True
in all the job definitions to make sure that dnf/yum update is called
before starting the tests.
This tag was missing for some jobs, this commit fixes the issue.
Reviewed-By: Armando Neto <abiagion@redhat.com>
Add decorator to has_record method which repeats the check when an active facet is changed
(catch StaleElementReferenceException).
Ticket: https://pagure.io/freeipa/issue/8157
Signed-off-by: Serhii Tsymbaliuk <stsymbal@redhat.com>
Reviewed-By: Armando Neto <abiagion@redhat.com>
These new images have SELinux enabled in permissive mode. After
this all tests skipped because SELinux was disabled will be
executed again.
Signed-off-by: Armando Neto <abiagion@redhat.com>
Reviewed-By: Florence Blanc-Renaud <frenaud@redhat.com>
Add a small timeout (up to 5 seconds) which allows to prevent exceptions when
WebDriver attempts to click a button before it is rendered.
Ticket: https://pagure.io/freeipa/issue/8169
Signed-off-by: Serhii Tsymbaliuk <stsymbal@redhat.com>
Reviewed-By: Sergey Orlov <sorlov@redhat.com>
fedora-latest/temp_commit section was removed from
temp_commit.yaml file while working with PR4108, adding it back.
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
When there is directory mounted on the ipa-client
Then no "Unspecified GSS failure" should be in logs.
This is an integration test for :
https://bugzilla.redhat.com/show_bug.cgi?id=1759665
Signed-off-by: Anuja More <amore@redhat.com>
Reviewed-By: Francois Cami <fcami@redhat.com>
Reviewed-By: Sumedh Sidhaye <ssidhaye@redhat.com>
Use a consistent way to label the tests. As a result, replace external_ca_1 with test_external_ca_TestExternalCA and external_ca_2 with test_external_ca_TestSelfExternalSelf to better reflect which subtest is executed.
Issue : freeipa/freeipa-pr-ci#336
Signed-off-by: Gaurav Talreja <gtalreja@redhat.com>
Reviewed-By: Sumedh Sidhaye <ssidhaye@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Check that ipa-client-samba tool reports specific properties of domains:
name, netbios name, sid and id range
Related to https://pagure.io/freeipa/issue/8149
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
This tests the following cases:
- deletion without nickname (expect fail)
- deletion with an unknown nickname (expect fail)
- deletion of IPA CA (expect fail)
- deletion of a root CA needed by a subCA (expect fail)
- deletion of a root CA needed by a subCA with --force (ok)
- deletion of a subca (ok)
As a side-effect this also tests install by installing the LE
root and a sub-ca. The sub-ca expires in 2021 but I tested in
the future the ipa-cacert-manage install doesn't do date
validation so for now this is ok.
https://pagure.io/freeipa/issue/8124
Reviewed-By: Florence Blanc-Renaud <frenaud@redhat.com>
Rename job titles to match their test suites and how they are defined in nightly yamls.
Issue : https://github.com/freeipa/freeipa-pr-ci/issues/336
Signed-off-by: Gaurav Talreja <gtalreja@redhat.com>
Reviewed-By: Florence Blanc-Renaud <frenaud@redhat.com>
The test suite test_winsyncmigrate was missing in nightly definitions
because CI was lacking configuration needed for establishing winsync
agreement: the Certificate Authority needs to be configured on
Windows AD instance. Now that PR-CI is updated to include said changes, we
can start executing this test suite. It is not reasonable to add it to
gating as this suite is time consuming just like other tests requiring
provisioning of AD instances.
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
The re.Pattern class was introduced in Python 3.7. Use duck-typing to
distinguish between str and re pattern object.
Fixes: https://pagure.io/freeipa/issue/8179
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Sergey Orlov <sorlov@redhat.com>
The tests for backup_and_restore check that the ipa-backup command
compresses the tar file AFTER restarting IPA services by reading the
output and looking for a pattern with "gzip" before "Starting IPA service."
As the tar file name is randomly created, it sometimes happen that the
name contains gzip and in this case the test wrongly assumes that
the gzip cmd was called.
The fix makes a stricter comparison, looking for /bin/gzip.
Fixes: https://pagure.io/freeipa/issue/8170
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
This checks that valid/invalid inputs for subtypes of
authentication indicator kerberos ticket policy options.
Signed-off-by: Anuja More <amore@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Fix XMLRPC tests so that "Dynamic Update" and "Bind update policy"
can be displayed by default in many DNS commands' output.
Related to: https://pagure.io/freeipa/issue/7938
Signed-off-by: François Cami <fcami@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
This forces PR-CI to update the packages instead of using the versions
already included in the vagrant image.
Signed-off-by: Armando Neto <abiagion@redhat.com>
Reviewed-By: Florence Blanc-Renaud <frenaud@redhat.com>
Test on replica for ipa-ca-install with options
--no-host-dns,--skip-schema-check,done changes in
ipatests/pytest_ipa/integration/tasks.py because
wants to pass few arguments to install_ca method
Signed-off-by: Jayesh <jgarg@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Reviewed-By: Fraser Tweedale <ftweedal@redhat.com>
Test was failing in nightly_PR for ipa-4.7
As https://pagure.io/SSSD/sssd/issue/3978 is not available on
fedora-29
Signed-off-by: Anuja More <amore@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
This also exercises the Authentication Indicator Kerberos ticket
policy options by testing a otp indicator type.
Related: https://pagure.io/freeipa/issue/8001
Signed-off-by: Anuja More <amore@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
