Sometimes tar fails on changed in process files:
```
[2021-09-07 11:03:33] + tar --ignore-failed-read -czf ipaserver_install_logs.tar.gz --warning=no-failed-read /var/log/dirsrv /var/log/httpd2 /var/log/ipa /var/log/ipaclient-install.log /var/log/ipa-custodia.audit.log /var/log/ipaserver-install.log /var/log/krb5kdc.log /var/log/pki /var/log/samba /var/lib/bind/data systemd_journal.log
[2021-09-07 11:03:33] tar: Removing leading `/' from member names
[2021-09-07 11:03:33] tar: Removing leading `/' from hard link targets
[2021-09-07 11:03:33] tar: /var/log/dirsrv/slapd-IPA-TEST/access: file changed as we read it
[2021-09-07 11:03:33] + tests_result=1
```
This is expected failure since processes are not stopped during logs
collection and can flush their logs.
Fixes: https://pagure.io/freeipa/issue/8983
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
There are tons of useless warnings about missing files on collecting
logs, such as:
```
tar: /var/log/ipaserver-kra-install.log: Warning: Cannot stat: No such file or directory
tar: /var/log/ipaepn.log: Warning: Cannot stat: No such file or directory
tar: /etc/NetworkManager/NetworkManager.conf: Warning: Cannot stat: No such file or directory
tar: /var/log/ipabackup.log: Warning: Cannot stat: No such file or directory
tar: /var/log/iparestore.log: Warning: Cannot stat: No such file or directory
...
```
Since `--ignore-failed-read` option is passed to tar the caller
doesn't care about not readable(mostly missing) files and these warnings
may be filtered out.
This improves the readability of test logs.
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
By default pytest reports in summary section about tests failures and errors.
It will be helpful to see skipped, xfailed and xpassed tests.
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
The tests in these envs make DNS requests to wild(internet) NSs,
though usually tests assume the opposite making requests to
`test.` zone. This makes CI unstable and dependent on wild
resolvers and logically wrong.
In future there can be tests which may want to check BIND as
resolver(cache) for external networks. In this case such tests
should be placed on not isolated mode.
By default, a test env is not isolated from internet(as it was
before), but it may be a good idea to change this default in
future.
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Collect disk usage information may be helpful, for example, for
debugging code required free space such as healthcheck tests.
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Some tests require its specific Pytest args. With this change
they can be specified in tests definitions.
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Some applications remove their logs on uninstallation.
As a result of this, Azure lost `install` logs.
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
* use "developer" in Azure
* fix man page: "development" to "developer"
* list known modes in API bootstrap methods
Other values for mode are still supported to avoid breaking existing
installations.
Fixes: https://pagure.io/freeipa/issue/8313
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Azure host has 6 GB of physical memory + 7 GB of swap.
FreeIPA CI runs at least 5 masters on each Azure's host.
Thus, swap is intensively used.
Based on the available *physical* memory 389-ds performs db tweaks
and in future may fail to start in case of memory shortage.
Current memory limit for Azure Docker containers(master/replica):
- Physical
$ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
1610612736
- Physical + swap:
$ cat /sys/fs/cgroup/memory/memory.memsw.limit_in_bytes
3221225472
In the meantime, installation of master + ca + kra + dnssec requires:
$ cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
1856929792
Some test environments require more memory.
For example, 'ipatests.test_integration.test_commands.TestIPACommand':
$ cat /sys/fs/cgroup/memory/memory.memsw.max_usage_in_bytes
2232246272
$ cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
2232246272
Fixes: https://pagure.io/freeipa/issue/8264
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Currently, only one IPA environment is tested within Docker
containers. This is not efficient because Azure's agent gives
6 GB of physical memory and 13 GB of total memory (Feb 2020),
but limits CPU with 2 cores.
Next examples are for 'master-only' topologies.
Let's assume that only one member of github repo simultaneously
run CI. This allows to get the full strength of Azure.
Concurrency results for TestInstallMaster:
------------------------------------------
| job concurrency | time/jobs |
------------------------------------------
| 5 | 40/5 |
| 4 | 34/4 |
| 3 | 25/3 |
| 2 | 19/2 |
| 1 | 17/1 |
------------------------------------------
Results prove the limitation of 2 cores. So, in case of jobs'
number not exceeds the max capacity for parallel jobs(10) the
proposed method couldn't save time, but it reduces the used
jobs number up to 2 times. In other words, in this case CI
could pass 2 x tests.
But what if CI was triggered by several PRs? or jobs' number is
bigger than 10. For example, there are 20 tests to be run.
Concurrency results for TestInstallMaster and 20 input jobs:
------------------------------------------------------------------
| job concurrency | time | jobs used | jobs free |
------------------------------------------------------------------
| 5 | 40 | 4 | 6 |
| 4 | 34 | 5 | 5 |
| 3 | 25 | 7 | 3 |
| 2 | 19 | 10 | 0 |
| 1 | 34 | 20 | 0 |
------------------------------------------------------------------
So, in this case the optimal concurrency would be 4 since it
allows to run two CIs simultaneously (20 tasks on board) and get
results in 34 minutes for both. In other words, two people could
trigger CI from PR and don't wait for each other.
New Azure IPA tests workflow:
+ 1) generate-matrix.py script generates JSON from user's YAML [0]
2) Azure generate jobs using Matrix strategy
3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) GNU 'parallel' launch each IPA environment in parallel:
+ 1) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ 2) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ 3) launch IPA tests on tests' controller
c) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
d) publish regular system logs as artifacts
[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
