Fixes: https://pagure.io/freeipa/issue/9347
Signed-off-by: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Francisco Trivino <ftrivino@redhat.com>
While python-ldap is strict dependency of IPA in downstreams, it
is optional for IPA packages published on PyPI.
Openldap 2.6 no longer ships ldap_r-2, that makes
ipapython.dn_ctypes not working against such environments.
Thanks @abbra!
Fixes: https://pagure.io/freeipa/issue/9255
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
This daemon run clamav which is resource aggressive.
No point to run Windows virus scanner on Ubuntu in Linux-only
environment.
Fixes: https://pagure.io/freeipa/issue/9207
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Allow npm to install and use latest grunt that is compatible with newer
OpenSSL.
This, in turn, requires ATK interfaces to be present for the chromium
installed by puppeteer.
Signed-off-by: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Florence Blanc-Renaud <frenaud@redhat.com>
Use fedora-toolbox:36 image as it is prepared to work with systemd and
sudo
Signed-off-by: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Florence Blanc-Renaud <frenaud@redhat.com>
Documentation generator can be run inside Python virtual environment.
This allows to isolate from the system-wide changes and add Sphinx
extensions that aren't packaged in a distribution.
The only exception right now is plantuml package. We rely on plantuml to
generate diagrams and since it is written in Java, it cannot be
installed directly into the Python venv through 'pip' tool.
Fixes: https://pagure.io/freeipa/issue/9148
Signed-off-by: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Francisco Trivino <ftrivino@redhat.com>
Otherwise, 'Check for coredumps' task fails with:
```
Verifying : samba-debugsource-2:4.14.4-0.fc34.x86_64 20/20
[Errno 2] No such file or directory: '/var/lib/dnf/rpmdb_lock.pid'
Finishing: Check for coredumps
```
This is due to systemd-tmpfiles(not ready yet).
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
The list of installed packages may be useful for checking the
versions of packages for analysis. Previously, only the newly
installed packages can be observed on Build phase.
This is convenient for experienced users of PR-CI.
Note: the read-only access provided for non-master containers
to be able to execute Azure scripts. The logs are still collected
only on controller.
Only RPM-based collection is implemented for Fedora. By default
nothing is collected.
Users may want to override `installed_packages` function
in the corresponding `ipatests/azure/scripts/variables-DISTRO.sh`.
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
There are tons of useless warnings about missing files on collecting
logs, such as:
```
tar: /var/log/ipaserver-kra-install.log: Warning: Cannot stat: No such file or directory
tar: /var/log/ipaepn.log: Warning: Cannot stat: No such file or directory
tar: /etc/NetworkManager/NetworkManager.conf: Warning: Cannot stat: No such file or directory
tar: /var/log/ipabackup.log: Warning: Cannot stat: No such file or directory
tar: /var/log/iparestore.log: Warning: Cannot stat: No such file or directory
...
```
Since `--ignore-failed-read` option is passed to tar the caller
doesn't care about not readable(mostly missing) files and these warnings
may be filtered out.
This improves the readability of test logs.
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
The nonzero number of memory/memory+Swap usage hits limits may
indicate the possible env instability(crashes, random failures, etc.).
> memory.failcnt # show the number of memory usage hits limits
memory.memsw.failcnt # show the number of memory+Swap hits limits
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
If an error occured while containers setup phase then no logs will
be collected and it is hard(impossible?) to debug such issues on
remote Azure host. With this change in case of such error all the
container's journals will be collected in `systemd_boot_logs`.
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Two, three times a year PR-CI starts failing because tox tests pull in a
newer version of pylint with new warnings. To reduce breakage this
change constraints pylint (and indirectly astroid) to latest tested
minor version. The constraint should be updated when FreeIPA starts to
support a new Fedora version with more recent pylint.
Related: https://pagure.io/freeipa/issue/8818
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Lint task uses PyPI to get the latest Pylint, which may pull in
any other packages as dependencies. For distro isolation the Lint
job should not produce any meaningful artifacts or they should not
be used in subsequent jobs. So, this job have to be isolated from
the others.
Fixes: https://pagure.io/freeipa/issue/8772
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Ubuntu 20.04's docker meets the requirement(4+).
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
The security option 'apparmor:unconfined' tells Docker to not
apply AppArmor profiles for containers at all. This will not
replace or remove any existing profile. For example, this happens
on Ubuntu 20.04 which switched to chrony and brings its AppArmor
profile. Container's chronyd get blocked by AppArmor:
fv-az26-252 audit[11304]: AVC apparmor="DENIED" operation="capable" profile="/usr/sbin/chronyd" pid=11304 comm="chronyd" capability=2 capname="dac_read_search"
fv-az26-252 audit[11304]: AVC apparmor="DENIED" operation="capable" profile="/usr/sbin/chronyd" pid=11304 comm="chronyd" capability=1 capname="dac_override"
So, any of AppArmor profiles can block container's processes by
matching executable name. There are two ways:
1) prepare custom AppArmor unconfined profile, load it on Host and
reference it in container's configuration. This requires the
knowledge of profile syntax at least, not to difficult, but
potentially hard to maintain.
2) disable conflicting profile on Host;
Azure will warn about AVC in either case.
The second one was chosen as more simple.
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Azure's VM distro is Ubuntu, which has enabled AppArmor.
This security module sometimes interferes with CI Docker containers,
but to be completely disabled it requires reboot(this is impossible,
at least for now). So, Azure will warn about AVC records in Host's
journal as a possible clue.
It will be equally important to be warned about SECCOMP records to
see possible blocked syscalls(requires SCMP_ACT_LOG as defaultAction in
seccomp profile).
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
The journal of Host is useful for AVC/SECCOMP analyzing.
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
The distros may use different sphinx builder paths,
for example, by exporting of SPHINXBUILD env var.
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Template the autoconf phase.
Fixes: https://pagure.io/freeipa/issue/8768
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Linux 5.8 made faccessat2() system call available in August 2020. This
system call is used now by GNU libc to implement more precisely
faccessat() system call. GNU glibc does compile-time check for the
kernel version and uses faccessat2() unconditionally in case it is
available. If kernel responds with ENOSYS error code, GNU libc will
attempt to use older, less flexible, faccessat(() system call.
When running on a system where libseccomp does not know about the new
syscall, the default action in seccomp filters in Docker and other
container runtimes is to respond with EPERM error code. This breaks GNU
libc's implementation of the faccessat() function -- as well as other
newer syscall implementations (e.g. statx()).
libseccomp started to support faccessat2() in July 2020 with
5696c89640
(version 2.5.0: https://github.com/seccomp/libseccomp/releases/tag/v2.5.0)
With Ubuntu 20.04 as a host, use PPA abbra/freeipa-libseccomp which
provides libseccomp 2.5.0 rebuild from Debian Sid.
Signed-off-by: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
FreeIPA uses git in its build process. In the past git was automatically
pulled in. On Fedora 33 builds are failing because git is missing.
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Python is provided by the Docker container image and it's no
longer needed to bind mount host's Python into container.
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
- allow override variables template file with an externally
provided one. This allows to add new Azure Pipeline which will
point to a custom platform definition. Note: Azure's WebUI
variables are runtime variables and not available at parsing time,
that's why it's impossible to override template from WebUI in
this case.
- add Rawhide templates
- add Dockerfile for build Rawhie Docker image for tests phase
Note: 'fedora:rawhide' is too old, use for now
'registry.fedoraproject.org/fedora:rawhide'.
See, https://bugzilla.redhat.com/show_bug.cgi?id=1869612
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
`registry.fedoraproject.org/f32/fedora-toolbox` image is used to build
packages on Azure Pipelines.
registry.fedoraproject.org experiences an availability problem and makes
unstable FreeIPA CI.
Fedora also distributes its official images on https://hub.docker.com/_/fedora.
`fedora:32` is already used by FreeIPA CI to build the image for tests.
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Francois Cami <fcami@redhat.com>
Azure host has 6 GB of physical memory + 7 GB of swap.
FreeIPA CI runs at least 5 masters on each Azure's host.
Thus, swap is intensively used.
Based on the available *physical* memory 389-ds performs db tweaks
and in future may fail to start in case of memory shortage.
Current memory limit for Azure Docker containers(master/replica):
- Physical
$ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
1610612736
- Physical + swap:
$ cat /sys/fs/cgroup/memory/memory.memsw.limit_in_bytes
3221225472
In the meantime, installation of master + ca + kra + dnssec requires:
$ cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
1856929792
Some test environments require more memory.
For example, 'ipatests.test_integration.test_commands.TestIPACommand':
$ cat /sys/fs/cgroup/memory/memory.memsw.max_usage_in_bytes
2232246272
$ cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
2232246272
Fixes: https://pagure.io/freeipa/issue/8264
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Applications may crash.
If a crash happens on a remote system during CI run it's sometimes
very hard to understand the reason. The most important means to
analyze such is a stack trace. It's also very important to check
whether there was a core dump or not, even a test passed.
For Docker environment, the core dumps are collected by the host's
systemd-coredump, which knows nothing about such containers (for
now). To build an informative thread stack trace debuginfo packages
should be installed. But they can't be installed on the host OS
(ubuntu), That's why after all the tests completed an additional
container should be up and the host's core dumps and host's journal
should be passed into it.
Even if there weren't enough debuginfo packages at CI-runtime, the
core dump could be analyzed locally later.
Fixes: https://pagure.io/freeipa/issue/8251
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
The platforms may have different Pythons.
But due to [0] the Python installed via the 'UsePythonVersion@0'
task should be compatible with the container's 'libpythonxx.so'.
'AZURE_PYTHON_VERSION' platform variable is introduced to cover
this. So, if your distro has Python3.8, set the mentioned variable
to '3.8', later, this version will be installed by the
'UsePythonVersion@0' Azure task for 'WebUI_Unit_Tests' and 'Tox'
jobs.
To allow tox to run any Python3 environment the 'py3' one is used.
'py3' is the well-known Tox's environment, which utilizes 'python3'
executable.
[0]: https://github.com/microsoft/azure-pipelines-tasks/issues/11070
Fixes: https://pagure.io/freeipa/issue/8254
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abbra@users.noreply.github.com>
Fast mirror selection somehow stopped working. If disabled, the
difference is around 20 seconds for the 'Prepare build environment' step
(2:49 versus 3:09), so while we are saving, currently it is not a lot.
Also remove explicit nodejs stream choice, it seems to be not needed
anymore (again).
Signed-off-by: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Armando Neto <abiagion@redhat.com>
Currently, only one IPA environment is tested within Docker
containers. This is not efficient because Azure's agent gives
6 GB of physical memory and 13 GB of total memory (Feb 2020),
but limits CPU with 2 cores.
Next examples are for 'master-only' topologies.
Let's assume that only one member of github repo simultaneously
run CI. This allows to get the full strength of Azure.
Concurrency results for TestInstallMaster:
------------------------------------------
| job concurrency | time/jobs |
------------------------------------------
| 5 | 40/5 |
| 4 | 34/4 |
| 3 | 25/3 |
| 2 | 19/2 |
| 1 | 17/1 |
------------------------------------------
Results prove the limitation of 2 cores. So, in case of jobs'
number not exceeds the max capacity for parallel jobs(10) the
proposed method couldn't save time, but it reduces the used
jobs number up to 2 times. In other words, in this case CI
could pass 2 x tests.
But what if CI was triggered by several PRs? or jobs' number is
bigger than 10. For example, there are 20 tests to be run.
Concurrency results for TestInstallMaster and 20 input jobs:
------------------------------------------------------------------
| job concurrency | time | jobs used | jobs free |
------------------------------------------------------------------
| 5 | 40 | 4 | 6 |
| 4 | 34 | 5 | 5 |
| 3 | 25 | 7 | 3 |
| 2 | 19 | 10 | 0 |
| 1 | 34 | 20 | 0 |
------------------------------------------------------------------
So, in this case the optimal concurrency would be 4 since it
allows to run two CIs simultaneously (20 tasks on board) and get
results in 34 minutes for both. In other words, two people could
trigger CI from PR and don't wait for each other.
New Azure IPA tests workflow:
+ 1) generate-matrix.py script generates JSON from user's YAML [0]
2) Azure generate jobs using Matrix strategy
3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) GNU 'parallel' launch each IPA environment in parallel:
+ 1) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ 2) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ 3) launch IPA tests on tests' controller
c) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
d) publish regular system logs as artifacts
[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
This allows to run IPA tests on Azure using any distro.
To achieve this, one has to do:
1) place a platform specific template on 'ipatests/azure/templates/'
and make a soft link from 'ipatests/azure/templates/variables.yml' to
the new template.
2) place a configuration templates on these paths
3) templates have to answer the questions such as:
a) which Docker image to use to build IPA packages (rpm, deb, etc.)
b) how to prepare Build environment
c) how to build IPA packages
d) how to prepare environment to run Tox tests
e) how to prepare environment to run WebUI unittests
f) which base Docker image to use to build the new image to run
IPA tests within it
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Azure provides Microsoft-hosted agents having tasty resources [0].
For now (Feb 2020),
- (Linux only) Run steps in a cgroup that offers 6 GB of physical memory and
13 GB of total memory
- Provide at least 10 GB of storage for your source and build outputs.
This is enough to set up IPA environments consisted of not only master but also
replicas and clients and thus, run IPA integration tests.
New Azure IPA tests workflow:
+ 1) Azure generate jobs using Matrix strategy
2) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ c) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ d) launch IPA tests on tests' controller
e) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
f) publish regular system logs as artifacts
[0] https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops
[1] https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
nodejs:12 requires libicu-65.1 while gdb (not direct dependency)
libicu-63.2. As a workaround gdb-minimal [0] could be used.
It's even better as requires less packages to be downloaded
and then installed.
[0] https://fedoraproject.org/wiki/Changes/Minimal_GDB_in_buildroot
Co-authored-by: Stanislav Levin <slev@altlinux.org>
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
The azure pipeline defines a "prepare build" step that
installs the FreeIPA development dependencies but the
step does not report failures of the dnf builddep command.
As a consequence, subsequent steps may fail (for instance
because of components not installed such as tox) but are
hard to diagnose.
The fix reports the command failure.
Fixes: https://pagure.io/freeipa/issue/8022
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
Azure Pipelines provide counters for running test jobs, these split into
System.StageAttempt and System.PhaseAttempt. Use them to make test
artifacts unique.
For XML test results we don't need to name them differently as they
aren't uploaded as artifacts but rather presented in a separate test
pane.
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Reviewed-By: Stanislav Levin <slev@altlinux.org>
Fedora nodejs builds were fixed, we don't need to limit ourselves
anymore.
Also, make sure python3-pyyaml is installed because pylint in Fedora 31
detects its use in contribs/
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Reviewed-By: Stanislav Levin <slev@altlinux.org>
The unit tests execution time within Azure Pipelines(AP) is not
balanced. One test job(Base) takes ~13min, while another(XMLRPC)
~28min. Fortunately, AP supports slicing:
> An agent job can be used to run a suite of tests in parallel. For
example, you can run a large suite of 1000 tests on a single agent.
Or, you can use two agents and run 500 tests on each one in parallel.
To leverage slicing, the tasks in the job should be smart enough to
understand the slice they belong to.
>The step that runs the tests in a job needs to know which test slice
should be run. The variables System.JobPositionInPhase and
System.TotalJobsInPhase can be used for this purpose.
Thus, to support this pytest should know how to split the test suite
into groups(slices). For this, a new internal pytest plugin was added.
About plugin.
- Tests within a slice are grouped by test modules because not all of
the tests within the module are independent from each other.
- Slices are balanced by the number of tests within test module.
- To run some module within its own environment there is a dedicated
slice option (could help with extremely slow tests)
Examples.
- To split `test_cmdline` tests into 2 slices and run the first one:
ipa-run-tests --slices=2 --slice-num=1 test_cmdline
- To split tests into 2 slices, then to move one module out to its own slice
and run the second one:
ipa-run-tests --slices=2 --slice-dedicated=test_cmdline/test_cli.py \
--slice-num=2 test_cmdline
Fixes: https://pagure.io/freeipa/issue/8008
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Display which dnf repositories were available at the
prepare-build step via metalink.
Also display the fastestmirror cache.
Signed-off-by: François Cami <fcami@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
