With fix for https://pagure.io/freeipa/issue/7951 we started to modify
RPM macros in Azure CI environment. Don't fail if the file does not
exist anymore like it happens now in Fedora.
Fixes: https://pagure.io/freeipa/issue/9347
Signed-off-by: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Francisco Trivino <ftrivino@redhat.com>
Use fedora-toolbox:36 image as it is prepared to work with systemd and
sudo
Signed-off-by: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Florence Blanc-Renaud <frenaud@redhat.com>
The list of installed packages may be useful for checking the
versions of packages for analysis. Previously, only the newly
installed packages can be observed on Build phase.
This is convenient for experienced users of PR-CI.
Note: the read-only access provided for non-master containers
to be able to execute Azure scripts. The logs are still collected
only on controller.
Only RPM-based collection is implemented for Fedora. By default
nothing is collected.
Users may want to override `installed_packages` function
in the corresponding `ipatests/azure/scripts/variables-DISTRO.sh`.
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
The tests in these envs make DNS requests to wild(internet) NSs,
though usually tests assume the opposite making requests to
`test.` zone. This makes CI unstable and dependent on wild
resolvers and logically wrong.
In future there can be tests which may want to check BIND as
resolver(cache) for external networks. In this case such tests
should be placed on not isolated mode.
By default, a test env is not isolated from internet(as it was
before), but it may be a good idea to change this default in
future.
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
The initial value of NS of resolv.conf is 127.0.0.11, this
is the embedded NS of docker-compose. The disabling of
this feature is not currently supported by Docker.
On startup systemd-resolved caches the /etc/resolv.conf
(docker-compose version), which is later modified by
setup_containers.py script.
This results in resolving error occurs:
```console
[root@replica1 /]# getent ahosts master1.ipa.test
... can't resolve
[root@replica1 /]# grep 'hosts:' /etc/nsswitch.conf
hosts: files myhostname resolve [!UNAVAIL=return] dns
[root@replica1 /]# resolvectl status
Global
LLMNR setting: resolve
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Current DNS Server: 127.0.0.11
DNS Servers: 127.0.0.11
Fallback DNS Servers: 1.1.1.1
8.8.8.8
1.0.0.1
8.8.4.4
2606:4700:4700::1111
2001:4860:4860::8888
2606:4700:4700::1001
2001:4860:4860::8844
```
According to docs:
https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html#/etc/resolv.conf
our case is 4(managed by other packages).
So, restart of systemd-resolved is enough for its re-initialization,
but not for services that already received DNS results. To speed up
the overall process and to no restart each service which wants
internet connection(or wait until service retries connection)
systemd-resolved is masked.
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
This may speed up boot process.
For example, 'fedora:34' set graphical.target as default,
while multi-user one will be more appropriate.
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Azure's VM distro is Ubuntu, which has enabled AppArmor.
This security module sometimes interferes with CI Docker containers,
but to be completely disabled it requires reboot(this is impossible,
at least for now). So, Azure will warn about AVC records in Host's
journal as a possible clue.
It will be equally important to be warned about SECCOMP records to
see possible blocked syscalls(requires SCMP_ACT_LOG as defaultAction in
seccomp profile).
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Linux 5.8 made faccessat2() system call available in August 2020. This
system call is used now by GNU libc to implement more precisely
faccessat() system call. GNU glibc does compile-time check for the
kernel version and uses faccessat2() unconditionally in case it is
available. If kernel responds with ENOSYS error code, GNU libc will
attempt to use older, less flexible, faccessat(() system call.
When running on a system where libseccomp does not know about the new
syscall, the default action in seccomp filters in Docker and other
container runtimes is to respond with EPERM error code. This breaks GNU
libc's implementation of the faccessat() function -- as well as other
newer syscall implementations (e.g. statx()).
libseccomp started to support faccessat2() in July 2020 with
5696c89640
(version 2.5.0: https://github.com/seccomp/libseccomp/releases/tag/v2.5.0)
With Ubuntu 20.04 as a host, use PPA abbra/freeipa-libseccomp which
provides libseccomp 2.5.0 rebuild from Debian Sid.
Signed-off-by: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
FreeIPA uses git in its build process. In the past git was automatically
pulled in. On Fedora 33 builds are failing because git is missing.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Fixes the following error in Azure Pipelines CI after upgrade of Docker
setup:
[2020-09-14 06:50:07] The Compose file './docker-compose.yml' is invalid because:
[2020-09-14 06:50:07] services.client.mem_limit contains an invalid type, it should be a string
Fixes: https://pagure.io/freeipa/issue/8494
Signed-off-by: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
- allow override variables template file with an externally
provided one. This allows to add new Azure Pipeline which will
point to a custom platform definition. Note: Azure's WebUI
variables are runtime variables and not available at parsing time,
that's why it's impossible to override template from WebUI in
this case.
- add Rawhide templates
- add Dockerfile for build Rawhie Docker image for tests phase
Note: 'fedora:rawhide' is too old, use for now
'registry.fedoraproject.org/fedora:rawhide'.
See, https://bugzilla.redhat.com/show_bug.cgi?id=1869612
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
`registry.fedoraproject.org/f32/fedora-toolbox` image is used to build
packages on Azure Pipelines.
registry.fedoraproject.org experiences an availability problem and makes
unstable FreeIPA CI.
Fedora also distributes its official images on https://hub.docker.com/_/fedora.
`fedora:32` is already used by FreeIPA CI to build the image for tests.
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Francois Cami <fcami@redhat.com>
Build container(image registry.fedoraproject.org/f32/fedora-toolbox)
has two more dnf repos enabled compared to Tests container(image
fedora:32). This results in the packages built within the Build
container can have dependencies which are unresolvable(missing)
within Tests container.
This enables updates-testing and updates-testing-modular,
disables fedora-cisco-openh264 for Tests container.
Fixes: https://pagure.io/freeipa/issue/8330
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
This allows to override the default seccomp profile.
Custom profile was generated from the default one [0] by adding one
allowed system call 'clock_adjtime'. This one is indirectly used by
chronyd with recent glibc2.31.
[0]: https://github.com/containers/libpod/blob/master/seccomp.json
Fixes: https://pagure.io/freeipa/issue/8316
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Azure host has 6 GB of physical memory + 7 GB of swap.
FreeIPA CI runs at least 5 masters on each Azure's host.
Thus, swap is intensively used.
Based on the available *physical* memory 389-ds performs db tweaks
and in future may fail to start in case of memory shortage.
Current memory limit for Azure Docker containers(master/replica):
- Physical
$ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
1610612736
- Physical + swap:
$ cat /sys/fs/cgroup/memory/memory.memsw.limit_in_bytes
3221225472
In the meantime, installation of master + ca + kra + dnssec requires:
$ cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
1856929792
Some test environments require more memory.
For example, 'ipatests.test_integration.test_commands.TestIPACommand':
$ cat /sys/fs/cgroup/memory/memory.memsw.max_usage_in_bytes
2232246272
$ cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
2232246272
Fixes: https://pagure.io/freeipa/issue/8264
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Currently, only one IPA environment is tested within Docker
containers. This is not efficient because Azure's agent gives
6 GB of physical memory and 13 GB of total memory (Feb 2020),
but limits CPU with 2 cores.
Next examples are for 'master-only' topologies.
Let's assume that only one member of github repo simultaneously
run CI. This allows to get the full strength of Azure.
Concurrency results for TestInstallMaster:
------------------------------------------
| job concurrency | time/jobs |
------------------------------------------
| 5 | 40/5 |
| 4 | 34/4 |
| 3 | 25/3 |
| 2 | 19/2 |
| 1 | 17/1 |
------------------------------------------
Results prove the limitation of 2 cores. So, in case of jobs'
number not exceeds the max capacity for parallel jobs(10) the
proposed method couldn't save time, but it reduces the used
jobs number up to 2 times. In other words, in this case CI
could pass 2 x tests.
But what if CI was triggered by several PRs? or jobs' number is
bigger than 10. For example, there are 20 tests to be run.
Concurrency results for TestInstallMaster and 20 input jobs:
------------------------------------------------------------------
| job concurrency | time | jobs used | jobs free |
------------------------------------------------------------------
| 5 | 40 | 4 | 6 |
| 4 | 34 | 5 | 5 |
| 3 | 25 | 7 | 3 |
| 2 | 19 | 10 | 0 |
| 1 | 34 | 20 | 0 |
------------------------------------------------------------------
So, in this case the optimal concurrency would be 4 since it
allows to run two CIs simultaneously (20 tasks on board) and get
results in 34 minutes for both. In other words, two people could
trigger CI from PR and don't wait for each other.
New Azure IPA tests workflow:
+ 1) generate-matrix.py script generates JSON from user's YAML [0]
2) Azure generate jobs using Matrix strategy
3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) GNU 'parallel' launch each IPA environment in parallel:
+ 1) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ 2) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ 3) launch IPA tests on tests' controller
c) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
d) publish regular system logs as artifacts
[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
