Commit Graph

95 Commits

Author SHA1 Message Date
Stanislav Levin
26f96595b0 Azure: Make dnf repos consistent
Build container(image registry.fedoraproject.org/f32/fedora-toolbox)
has two more dnf repos enabled compared to Tests container(image
fedora:32). This results in the packages built within the Build
container can have dependencies which are unresolvable(missing)
within Tests container.

This enables updates-testing and updates-testing-modular,
disables fedora-cisco-openh264 for Tests container.

Fixes: https://pagure.io/freeipa/issue/8330
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-05-22 21:12:03 +03:00
Christian Heimes
82ba4db11e Make api.env.mode consistent
* use "developer" in Azure
* fix man page: "development" to "developer"
* list known modes in API bootstrap methods

Other values for mode are still supported to avoid breaking existing
installations.

Fixes: https://pagure.io/freeipa/issue/8313
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
2020-05-14 17:55:59 +02:00
Stanislav Levin
b6fbee53bc Azure: Always update apt cache
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
2020-05-13 11:04:45 +02:00
Stanislav Levin
8882fc49d0 Azure: Allow chronyd to sync time
Though time namespace support was added in Linux kernel 5.6, it
is not landed on Azure VM (Ubuntu) yet.

The syncing time stuff is required by IPA NTP tests. it's
acceptable for testing 1 IPA environment on 1 Azure VM for such
tests.

Fixes: https://pagure.io/freeipa/issue/8316
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-05-12 09:51:50 +02:00
Stanislav Levin
958e245813 Azure: Add custom seccomp profile
This allows to override the default seccomp profile.
Custom profile was generated from the default one [0] by adding one
allowed system call 'clock_adjtime'. This one is indirectly used by
chronyd with recent glibc2.31.

[0]: https://github.com/containers/libpod/blob/master/seccomp.json

Fixes: https://pagure.io/freeipa/issue/8316
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-05-12 09:51:50 +02:00
Alexander Bokovoy
f66ef8484d Azure Pipelines: switch to Fedora 32
Reviewed-By: Christian Heimes <cheimes@redhat.com>
2020-05-06 09:14:29 +02:00
Alexander Bokovoy
b8a1d130ad Azure Pipelines: Override services known to not work in containers
Chrony daemon tries to use adjtimex() which doesn't work in the
container we run in Docker environment on Azure Pipelines.

nis-domainname also tries to modify kernel-specific parameter that
doesn't really work in runc-based containers.

Use systemd container detection to avoid starting these services in the
containers.

Signed-off-by: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
2020-05-06 09:14:29 +02:00
Stanislav Levin
87408ee755 Azure: Increase memory limit
Azure host has 6 GB of physical memory + 7 GB of swap.
FreeIPA CI runs at least 5 masters on each Azure's host.
Thus, swap is intensively used.

Based on the available *physical* memory 389-ds performs db tweaks
and in future may fail to start in case of memory shortage.

Current memory limit for Azure Docker containers(master/replica):
- Physical
$ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
1610612736
- Physical + swap:
$ cat /sys/fs/cgroup/memory/memory.memsw.limit_in_bytes
3221225472

In the meantime, installation of master + ca + kra + dnssec requires:
$ cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
1856929792

Some test environments require more memory.
For example, 'ipatests.test_integration.test_commands.TestIPACommand':
$ cat /sys/fs/cgroup/memory/memory.memsw.max_usage_in_bytes
2232246272
$ cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
2232246272

Fixes: https://pagure.io/freeipa/issue/8264
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
2020-04-28 17:50:10 +02:00
François Cami
7558e1413d doc/Makefile: use sphinx-build -W by default
Use -W with sphinx-build by default to turn warnings into errors.

Signed-off-by: François Cami <fcami@redhat.com>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
2020-04-21 14:59:02 +02:00
Stanislav Levin
d1b53ded8b Azure: Gather coredumps
Applications may crash.
If a crash happens on a remote system during CI run it's sometimes
very hard to understand the reason. The most important means to
analyze such is a stack trace. It's also very important to check
whether there was a core dump or not, even a test passed.

For Docker environment, the core dumps are collected by the host's
systemd-coredump, which knows nothing about such containers (for
now). To build an informative thread stack trace debuginfo packages
should be installed. But they can't be installed on the host OS
(ubuntu), That's why after all the tests completed an additional
container should be up and the host's core dumps and host's journal
should be passed into it.

Even if there weren't enough debuginfo packages at CI-runtime, the
core dump could be analyzed locally later.

Fixes: https://pagure.io/freeipa/issue/8251
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
2020-04-08 11:27:45 +03:00
Stanislav Levin
aa5a3336a8 Azure: Allow distros to install Python they want
The platforms may have different Pythons.
But due to [0] the Python installed via the 'UsePythonVersion@0'
task should be compatible with the container's 'libpythonxx.so'.
'AZURE_PYTHON_VERSION' platform variable is introduced to cover
this. So, if your distro has Python3.8, set the mentioned variable
to '3.8', later, this version will be installed by the
'UsePythonVersion@0' Azure task for 'WebUI_Unit_Tests' and 'Tox'
jobs.

To allow tox to run any Python3 environment the 'py3' one is used.
'py3' is the well-known Tox's environment, which utilizes 'python3'
executable.

[0]: https://github.com/microsoft/azure-pipelines-tasks/issues/11070

Fixes: https://pagure.io/freeipa/issue/8254
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abbra@users.noreply.github.com>
2020-04-07 15:22:47 -04:00
Alexander Bokovoy
77ed0918a7 Remove Fedora repository fastmirror selection
Fast mirror selection somehow stopped working. If disabled, the
difference is around 20 seconds for the 'Prepare build environment' step
(2:49 versus 3:09), so while we are saving, currently it is not a lot.

Also remove explicit nodejs stream choice, it seems to be not needed
anymore (again).

Signed-off-by: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Armando Neto <abiagion@redhat.com>
2020-03-28 19:55:10 +01:00
Christian Heimes
a4efb3028b Test documentation builds in Azure 2020-03-21 07:40:33 +02:00
Stanislav Levin
0a1e98cdf0 Azure: Report elapsed time
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00
Stanislav Levin
38e0a9f4c0 Azure: Rebalance tests
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00
Stanislav Levin
9203404cd5 Azure: Skip tests requiring external DNS
An external DNS is not supported yet, but it could be easily
implemented by adding another container with simple DNS server.

Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00
Stanislav Levin
e925148ad9 Azure: Free Docker resources after usage
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00
Stanislav Levin
1fa033c32d Azure: Preliminary check for provided limits
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00
Stanislav Levin
6daf4d2e10 Azure: Sync Gating definitions to current PR-CI
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00
Stanislav Levin
31d05650fb Azure: Add support for testing multi IPA environments
Currently, only one IPA environment is tested within Docker
containers. This is not efficient because Azure's agent gives
6 GB of physical memory and 13 GB of total memory (Feb 2020),
but limits CPU with 2 cores.

Next examples are for 'master-only' topologies.

Let's assume that only one member of github repo simultaneously
run CI. This allows to get the full strength of Azure.

Concurrency results for TestInstallMaster:
------------------------------------------
|    job concurrency      |  time/jobs   |
------------------------------------------
|             5           |     40/5     |
|             4           |     34/4     |
|             3           |     25/3     |
|             2           |     19/2     |
|             1           |     17/1     |
------------------------------------------
Results prove the limitation of 2 cores. So, in case of jobs'
number not exceeds the max capacity for parallel jobs(10) the
proposed method couldn't save time, but it reduces the used
jobs number up to 2 times. In other words, in this case CI
could pass 2 x tests.

But what if CI was triggered by several PRs? or jobs' number is
bigger than 10. For example, there are 20 tests to be run.

Concurrency results for TestInstallMaster and 20 input jobs:
------------------------------------------------------------------
|    job concurrency      |     time     | jobs used | jobs free |
------------------------------------------------------------------
|             5           |      40      |      4    |     6     |
|             4           |      34      |      5    |     5     |
|             3           |      25      |      7    |     3     |
|             2           |      19      |     10    |     0     |
|             1           |      34      |     20    |     0     |
------------------------------------------------------------------
So, in this case the optimal concurrency would be 4 since it
allows to run two CIs simultaneously (20 tasks on board) and get
results in 34 minutes for both. In other words, two people could
trigger CI from PR and don't wait for each other.

New Azure IPA tests workflow:

+ 1) generate-matrix.py script generates JSON from user's YAML [0]
  2) Azure generate jobs using Matrix strategy
  3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
    a) downloads prepared Docker container image (artifact) from Azure cloud
       (built on Build Job) and loads the received image into local pool
  + b) GNU 'parallel' launch each IPA environment in parallel:
    + 1) docker-compose creates the Docker environment having a required number
         of replicas and/or clients
    + 2) setup_containers.py script does the needed container's changes (DNS,
         SSH, etc.)
    + 3) launch IPA tests on tests' controller
    c) publish tests results in JUnit format to provide a comprehensive test
       reporting and analytics experience via Azure WebUI [1]
    d) publish regular system logs as artifacts

[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml

Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00
Stanislav Levin
d3f1b9b43d Azure: Don't collect twice systemd_journal.log
This log file is collected by azure-run-tests.sh script and then by
Azure 'PublishPipelineArtifact' task. So, the same file gets into
logs artifact.

Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00
Stanislav Levin
b82515562a Azure: Make it possible to configure distro-specific stuff
This allows to run IPA tests on Azure using any distro.

To achieve this, one has to do:
1) place a platform specific template on 'ipatests/azure/templates/'
and make a soft link from 'ipatests/azure/templates/variables.yml' to
the new template.
2) place a configuration templates on these paths
3) templates have to answer the questions such as:
  a) which Docker image to use to build IPA packages (rpm, deb, etc.)
  b) how to prepare Build environment
  c) how to build IPA packages
  d) how to prepare environment to run Tox tests
  e) how to prepare environment to run WebUI unittests
  f) which base Docker image to use to build the new image to run
     IPA tests within it

Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00
Stanislav Levin
879855ce70 Azure: Allow to run integration tests
Azure provides Microsoft-hosted agents having tasty resources [0].
For now (Feb 2020),
- (Linux only) Run steps in a cgroup that offers 6 GB of physical memory and
13 GB of total memory
- Provide at least 10 GB of storage for your source and build outputs.

This is enough to set up IPA environments consisted of not only master but also
replicas and clients and thus, run IPA integration tests.

New Azure IPA tests workflow:

+ 1) Azure generate jobs using Matrix strategy
  2) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
    a) downloads prepared Docker container image (artifact) from Azure cloud
       (built on Build Job) and loads the received image into local pool
  + b) docker-compose creates the Docker environment having a required number
       of replicas and/or clients
  + c) setup_containers.py script does the needed container's changes (DNS,
       SSH, etc.)
  + d) launch IPA tests on tests' controller
    e) publish tests results in JUnit format to provide a comprehensive test
       reporting and analytics experience via Azure WebUI [1]
    f) publish regular system logs as artifacts

[0] https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops
[1] https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml

Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00
Stanislav Levin
157fa59e7c Azure: Allow SSH for Docker environments
IPA integration tests utilize SSH as a transport to communicate
with IPA hosts. To run such tests Docker environments should
have configured SSH.

Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00
Stanislav Levin
ecc398c409 Azure: Allow to not provide tests to be ignored
As for now, a list of tests which will be ignored by Pytest is
mandatory. But actually, a list of tests to run is explicitly set
in yaml config. And thus, 'ignore' list should be an optional field.

This simplifies tests definitions to drop extra stuff.

Fixes: https://pagure.io/freeipa/issue/8202

Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-25 18:02:12 +02:00
Alexander Bokovoy
ba904672a2 Azure Pipelines: re-enable nodejs:12 stream for Fedora 31+
Reviewed-By: Christian Heimes <cheimes@redhat.com>
2020-02-17 16:03:11 +02:00
Alexander Bokovoy
43a97082bb Update Azure Pipelines to use Fedora 31
nodejs:12 requires libicu-65.1 while gdb (not direct dependency)
libicu-63.2. As a workaround gdb-minimal [0] could be used.
It's even better as requires less packages to be downloaded
and then installed.

[0] https://fedoraproject.org/wiki/Changes/Minimal_GDB_in_buildroot

Co-authored-by: Stanislav Levin <slev@altlinux.org>
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
2020-02-12 18:08:32 +02:00
Stanislav Levin
1ed7dd4bf1 Install language packs for tests
* 'fr_FR' locale is utilized in
test_ipaserver/test_i18n_messages.py::test_i18n_messages::test_i18n_consequence_receive

* 'en_US' is a commonly used locale
AP warns regularly:

```
/bin/bash: warning: setlocale: LC_ALL: cannot change locale
(en_US.utf8): No such file or directory
```

Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
2019-10-17 08:29:06 +02:00
Stanislav Levin
16149831da Restore running of 'test_ipaserver' tests on Azure
`test_ipaserver` was lost on refactoring in #c8ef093e56.
Let's run that again.

Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
2019-10-17 08:29:06 +02:00
Stanislav Levin
feae9de73e Setup DNS for AP Docker container
Docker utilizes its own way to provide DNS (hostname, hosts, NS).
By default, they are almost the same as the host's ones.
For instance, below is from AP container:
```
cat /etc/hosts

127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.17.0.2	ipa.example.test ipa

cat /etc/resolv.conf
nameserver 168.63.129.16
search hqdv2iuiph0ufpcrhp4amkgzwf.fx.internal.cloudapp.net
```

As a result FreeIPA uses 168.63.129.16 (AP DNS NS [1]) as a DNS forwarder.
It's not desirable to rely on this.
Let's clear test environment.

[1] https://docs.microsoft.com/en-us/azure/virtual-network/what-is-ip-address-168-63-129-16

Related: https://pagure.io/freeipa/issue/8077
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
2019-09-25 20:14:06 +10:00
François Cami
70b96d76cb azure-pipelines.yml: switch to Python 3.7
* switch to Python 3.7 (UsePythonVersion@0 task)
* use "pip install --user"

Fixes: https://pagure.io/freeipa/issue/8030
Signed-off-by: François Cami <fcami@redhat.com>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2019-08-02 20:15:04 +02:00
Florence Blanc-Renaud
5e97e80069 Azure pipeline: report failure in prepare-build step
The azure pipeline defines a "prepare build" step that
installs the FreeIPA development dependencies but the
step does not report failures of the dnf builddep command.

As a consequence, subsequent steps may fail (for instance
because of components not installed such as tox) but are
hard to diagnose.

The fix reports the command failure.

Fixes: https://pagure.io/freeipa/issue/8022
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
2019-07-26 10:47:30 +02:00
Alexander Bokovoy
0187a746c9 Use stage and phase attempt counters when saving test artifacts
Azure Pipelines provide counters for running test jobs, these split into
System.StageAttempt and System.PhaseAttempt. Use them to make test
artifacts unique.

For XML test results we don't need to name them differently as they
aren't uploaded as artifacts but rather presented in a separate test
pane.

Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Reviewed-By: Stanislav Levin <slev@altlinux.org>
2019-07-17 13:10:34 +03:00
Alexander Bokovoy
d55c9b6d7f Use any nodejs version instead of forcing a version before nodejs 11
Fedora nodejs builds were fixed, we don't need to limit ourselves
anymore.

Also, make sure python3-pyyaml is installed because pylint in Fedora 31
detects its use in contribs/

Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
Reviewed-By: Stanislav Levin <slev@altlinux.org>
2019-07-17 13:10:34 +03:00
Stanislav Levin
a2d4e2a61f Make use of Azure Pipeline slicing
The unit tests execution time within Azure Pipelines(AP) is not
balanced. One test job(Base) takes ~13min, while another(XMLRPC)
~28min. Fortunately, AP supports slicing:

> An agent job can be used to run a suite of tests in parallel. For
example, you can run a large suite of 1000 tests on a single agent.
Or, you can use two agents and run 500 tests on each one in parallel.
To leverage slicing, the tasks in the job should be smart enough to
understand the slice they belong to.

>The step that runs the tests in a job needs to know which test slice
should be run. The variables System.JobPositionInPhase and
System.TotalJobsInPhase can be used for this purpose.

Thus, to support this pytest should know how to split the test suite
into groups(slices). For this, a new internal pytest plugin was added.

About plugin.
- Tests within a slice are grouped by test modules because not all of
the tests within the module are independent from each other.
- Slices are balanced by the number of tests within test module.
- To run some module within its own environment there is a dedicated
slice option (could help with extremely slow tests)

Examples.
- To split `test_cmdline` tests into 2 slices and run the first one:

ipa-run-tests --slices=2 --slice-num=1 test_cmdline

- To split tests into 2 slices, then to move one module out to its own slice
and run the second one:

ipa-run-tests --slices=2 --slice-dedicated=test_cmdline/test_cli.py \
    --slice-num=2 test_cmdline

Fixes: https://pagure.io/freeipa/issue/8008
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2019-07-16 13:23:21 +03:00
Stanislav Levin
2312b38a67 Simplify ipa-run-tests script
This is a sort of rollback to the pre #93c158b05 state with
several improvements.

For now, the nodeids calculation by ipa-run-tests is not stable,
since it depends on current working directory. Nodeids (tests
addresses) are utilized by the other plugins, for example.

Unfortunately, the `pytest_load_initial_conftests` hook doesn't
correctly work with pytest internal paths, since it is called
after the calculation of rootdir was performed, for example.

Eventually, it's simpler to follow the default convention for
Python test discovery.

There is at least one drawback of new "old" implementation.
The ignore rules don't support globs, because pytest 4.3.0+
has the same facility via `--ignore-glob`:

> Add the `--ignore-glob` parameter to exclude test-modules with
> Unix shell-style wildcards. Add the collect_ignore_glob for
> conftest.py to exclude test-modules with Unix shell-style
> wildcards.

Upon switching to pytest4 it will be possible to utilize this.
Anyway, tests for checking current basic facilities of
ipa-run-tests were added.

Fixes: https://pagure.io/freeipa/issue/8007
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2019-07-16 13:23:21 +03:00
François Cami
be7f54d40c ipatests/azure: display actual dnf repo URLs
Display which dnf repositories were available at the
prepare-build step via metalink.
Also display the fastestmirror cache.

Signed-off-by: François Cami <fcami@redhat.com>
Reviewed-By: Florence Blanc-Renaud <flo@redhat.com>
2019-07-08 17:30:24 +02:00
François Cami
7027f7914e Make dnf more robust and faster
Sometimes the prepare-build step of azure pipelines fails
with download errors:
"configure: error: Package requirements (nspr) were not met:"
This can be due to fastestmirror not being used to check
mirror availability and sometimes speed. Combined with a
too-low default number of retries, and a high timeout this
can lead to download failures that could be avoided.
Activate fastestmirror, add more download workers, and tune
timeout/retries to make dnf more reliable.

Fixes: https://pagure.io/freeipa/issue/7999
Signed-off-by: François Cami <fcami@redhat.com>
Reviewed-By: Christian Heimes <cheimes@redhat.com>
2019-07-03 08:13:39 +02:00
Stanislav Levin
b5bb436e9f Exit on fail in azure multiline script
By default, the `last` exit code returned from Azure script will be
checked and, if non-zero, treated as a step failure.  Luckily,
for Linux script is a shortcut for Bash. Hence errexit/e option
could be applied. But Azure pipelines doesn't set it by default:
https://github.com/microsoft/azure-pipelines-agent/issues/1803

For multiline script this is a problem, unless otherwise designed.
Some of benefits of checking the result of each subcommand:
- preventing subsequent issues (broken packages, container images, etc.)
- time saving (next steps will not run)
- good diagnostics (tells which part of script fails)

Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2019-06-26 20:47:58 +03:00
Alexander Bokovoy
8f4ca3957c azure tests: make sure /etc/docker folder exists
Azure tests fail because we couldn't configure docker for IPv6 anymore.
This happened because we weren't able to copy our configuration file to
/etc/docker -- looks like the docker directory does not exist.

Reviewed-By: Christian Heimes <cheimes@redhat.com>
2019-06-05 10:47:40 +03:00
Alexander Bokovoy
5230e2a12d Azure Pipelines: run fast linter in case of a pull request build
Reviewed-By: Christian Heimes <cheimes@redhat.com>
2019-05-28 09:55:51 +03:00
Alexander Bokovoy
c8ef093e56 Azure Pipelines: simplify test job definitions
Rewrite templates to make test job declarations simpler and easier to
work with.

A test job template can be instantiated this way:

- template: templates/test-jobs.yml
  parameters:
    jobName: Base
    jobTitle: Base tests
    testsToRun:
    - test_cmdline
    - test_install
    - test_ipaclient
    - test_ipalib
    - test_ipaplatform
    - test_ipapython
    testsToIgnore:
    - test_integration
    - test_webui
    - test_ipapython/test_keyring.py
    taskToRun: run-tests

Both 'testsToRun' and 'testsToIgnore' accept arrays of test matches.
Wildcards also supported:

    ....
    testsToRun:
    - test_xmlrpc/test_hbac*
    ....

'taskToRun' specifies a script ipatests/azure/azure-${taskToRun}.sh that
will be executed in the test environment to actually start tests.

Parameters 'testsToRun' and 'testsToIgnore' define TESTS_TO_{RUN,IGNORE}
variables that will be set in the environment of the test script. These
variables will have entries from the parameters separated by a space.

Reviewed-By: Christian Heimes <cheimes@redhat.com>
2019-05-28 09:55:51 +03:00
Alexander Bokovoy
74f3ca5db0 i18n_messages: get back a locale needed for testing
Commit f49fac7bda added a special
workaround to get fr-fr locale translations installed when running
tests in Travis CI.

Get it back to Azure Pipelines.

Fixes: https://pagure.io/freeipa/issue/7951
Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
2019-05-16 09:51:45 +03:00
Alexander Bokovoy
9cb6817b30 azure-run-tests: handle single unexpanded parameter too
If TESTS_TO_RUN contains a single parameter that cannot be expanded,
bash will not perform brace elimination. Remove braces manually.

For example, TESTS_TO_RUN='test_xmlrpc/test_*.py' will not expand
outside of ipatests and the script would generate

 tests_to_run=-k{test_xmlrpc/test_*.py}

Braces then will prevent actual ipa-run-tests execution from matching
any of XMLRPC tests.

Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
2019-05-16 09:51:45 +03:00
Alexander Bokovoy
58fe6fac61 Set up CI with Azure Pipelines
Sets up a pipeline to run FreeIPA build and tests in Azure Pipelines.

Azure Pipelines provides 10 parallel free runners for open source projects.
Use them to run following jobs:

 - Build: build RPMs and Fedora 30 container with them
 - Lint: run linting of the source code
 - Tox: run py36,pypi,pylint tests using Tox
 - Web UI unit tests: run Web UI unit tests with Grunt/QUnit/PhantomJS
 - XMLRPC tests: install FreeIPA server and run XMLRPC tests against it

All jobs are running in Fedora 30 containers. Build, Lint, Tox, and Web
UI unit tests run inside f30/fedora-toolbox container. Build job
generates a container with pre-installed FreeIPA packages using official
fedora:30 container. All containers are picked up from
registry.fedoraproject.org.

Artifacts from the build job are pushed to a pipeline storage and reused
in the XMLRPC tests. They also are accessible in the 'Summary' tab to
download.

XUnit and QUnit outputs from the tests that produce it are reported in
the 'Tests' tab.

Logs from individual steps from each job are available for review in
the 'Logs' tab. They also can be downloaded.

Reviewed-By: Christian Heimes <cheimes@redhat.com>
Reviewed-By: Rob Crittenden <rcritten@redhat.com>
2019-05-14 14:52:29 -04:00