Commit Graph

15 Commits

Author SHA1 Message Date
François Cami
baed2e6f91 Azure: temporarily disable problematic tests, #2
test_cert and test_SubCAkeyReplication are randomly failing.
The suspect for test_SubCAkeyReplication is an nss bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1985061

The reason for test_cert failures was not identified, the only
relevant line in the log contains:
2021-07-22T17:37:21.0873339Z tests: cert, result: 1, time: 30:08.98
2021-07-22T17:37:21.0874172Z Command exited with non-zero status 1

Disable these tests in gating until the NSS bug is fixed and
the related build is available in Fedora.

Related: https://pagure.io/freeipa/issue/8864
Signed-off-by: François Cami <fcami@redhat.com>
Reviewed-By: Michal Polovka <mpolovka@redhat.com>
Reviewed-By: Stanislav Levin <slev@altlinux.org>
2021-07-23 10:52:55 +02:00
François Cami
b71009b31a Azure: temporarily disable problematic tests, #1
test_installation.TestInstallMaster, test_advise,
and test_integration.test_commands.TestIPACommand rely on DNS
forwarders and hit a known BIND bug:
https://gitlab.isc.org/isc-projects/bind9/-/issues/2728
quite often.
This is blocking gating nearly completely.
Disable these tests in gating until the bug is fixed and
the related build is available in Fedora.

Related: https://pagure.io/freeipa/issue/8864
Signed-off-by: François Cami <fcami@redhat.com>
Reviewed-By: Michal Polovka <mpolovka@redhat.com>
Reviewed-By: Stanislav Levin <slev@altlinux.org>
2021-07-23 10:52:55 +02:00
Christian Heimes
470bb6ed45 Add Custodia tests
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
2021-06-16 10:28:17 -04:00
Stanislav Levin
10461b7091 azure: Make it possible to adjust Docker resources per test env
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2021-05-25 10:45:49 +03:00
Stanislav Levin
01553572d4 azure: Re-balance tests envs
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2021-05-25 10:45:49 +03:00
Stanislav Levin
fc0c6b44a8 azure: Run Base and XMLRPC tests is isolated network
The tests in these envs make DNS requests to wild(internet) NSs,
though usually tests assume the opposite making requests to
`test.` zone. This makes CI unstable and dependent on wild
resolvers and logically wrong.

In future there can be tests which may want to check BIND as
resolver(cache) for external networks. In this case such tests
should be placed on not isolated mode.

By default, a test env is not isolated from internet(as it was
before), but it may be a good idea to change this default in
future.

Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2021-05-25 10:45:49 +03:00
Stanislav Levin
3ac2cdfd43 Azure: Make it possible to pass additional Pytest args
Some tests require its specific Pytest args. With this change
they can be specified in tests definitions.

Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2021-03-30 09:58:42 +02:00
Florence Blanc-Renaud
a9b4ed4f52 ipatests: re-add test_dnssec.py::TestInstallDNSSECFirst in gating
The test was temporarily removed because of a known issue
but the issue is now fixed.

Related: https://pagure.io/freeipa/issue/8496
Signed-off-by: Florence Blanc-Renaud <flo@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Reviewed-By: Sergey Orlov <sorlov@redhat.com>
2021-03-17 16:28:19 +01:00
Florence Blanc-Renaud
a33530f2f6 ipatests: temporarily remove test_dnssec.py::TestInstallDNSSECFirst from gating
The test test_dnssec.py::TestInstallDNSSECFirst is failing due to known
issue https://pagure.io/freeipa/issue/8496
currently under investigation by 389ds team.

In the meantime, remove the test from gating to avoid blocking the PRs.

Related: https://pagure.io/freeipa/issue/8496
Reviewed-By: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-11-17 08:39:59 +01:00
Stanislav Levin
e5c09675f3 ipatests: Skip keyring tests on containerized platforms
The kernel keyrings are not namespaced yet.

Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Francois Cami <fcami@redhat.com>
2020-08-07 17:49:31 +03:00
Alexander Bokovoy
4ff972c23f azure: do not run test_commands due to failures in low memory cases
389-ds memory autotuning doesn't really work well in containerized
environment as it only looks into host-wide /proc/meminfo. It gets
fooled by 'missing' memory while there is still enough swap space.

This is in particular affects test_commands test suite where
ipa-adtrust-install cannot fully proceed and fails. We plan to rebalance
test containers' memory split but right now just disable test_commands
in Azure CI.

Signed-off-by: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Isaac Boukris <iboukris@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
2020-05-27 17:57:39 +03:00
Stanislav Levin
38e0a9f4c0 Azure: Rebalance tests
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00
Stanislav Levin
9203404cd5 Azure: Skip tests requiring external DNS
An external DNS is not supported yet, but it could be easily
implemented by adding another container with simple DNS server.

Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00
Stanislav Levin
6daf4d2e10 Azure: Sync Gating definitions to current PR-CI
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00
Stanislav Levin
31d05650fb Azure: Add support for testing multi IPA environments
Currently, only one IPA environment is tested within Docker
containers. This is not efficient because Azure's agent gives
6 GB of physical memory and 13 GB of total memory (Feb 2020),
but limits CPU with 2 cores.

Next examples are for 'master-only' topologies.

Let's assume that only one member of github repo simultaneously
run CI. This allows to get the full strength of Azure.

Concurrency results for TestInstallMaster:
------------------------------------------
|    job concurrency      |  time/jobs   |
------------------------------------------
|             5           |     40/5     |
|             4           |     34/4     |
|             3           |     25/3     |
|             2           |     19/2     |
|             1           |     17/1     |
------------------------------------------
Results prove the limitation of 2 cores. So, in case of jobs'
number not exceeds the max capacity for parallel jobs(10) the
proposed method couldn't save time, but it reduces the used
jobs number up to 2 times. In other words, in this case CI
could pass 2 x tests.

But what if CI was triggered by several PRs? or jobs' number is
bigger than 10. For example, there are 20 tests to be run.

Concurrency results for TestInstallMaster and 20 input jobs:
------------------------------------------------------------------
|    job concurrency      |     time     | jobs used | jobs free |
------------------------------------------------------------------
|             5           |      40      |      4    |     6     |
|             4           |      34      |      5    |     5     |
|             3           |      25      |      7    |     3     |
|             2           |      19      |     10    |     0     |
|             1           |      34      |     20    |     0     |
------------------------------------------------------------------
So, in this case the optimal concurrency would be 4 since it
allows to run two CIs simultaneously (20 tasks on board) and get
results in 34 minutes for both. In other words, two people could
trigger CI from PR and don't wait for each other.

New Azure IPA tests workflow:

+ 1) generate-matrix.py script generates JSON from user's YAML [0]
  2) Azure generate jobs using Matrix strategy
  3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
    a) downloads prepared Docker container image (artifact) from Azure cloud
       (built on Build Job) and loads the received image into local pool
  + b) GNU 'parallel' launch each IPA environment in parallel:
    + 1) docker-compose creates the Docker environment having a required number
         of replicas and/or clients
    + 2) setup_containers.py script does the needed container's changes (DNS,
         SSH, etc.)
    + 3) launch IPA tests on tests' controller
    c) publish tests results in JUnit format to provide a comprehensive test
       reporting and analytics experience via Azure WebUI [1]
    d) publish regular system logs as artifacts

[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml

Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00