2020-08-18 02:28:59 -05:00
|
|
|
parameters:
|
|
|
|
- name: VARIABLES_FILE
|
|
|
|
default: 'templates/variables.yml'
|
|
|
|
|
2019-02-23 14:23:17 -06:00
|
|
|
trigger:
|
|
|
|
- master
|
|
|
|
|
|
|
|
variables:
|
2020-02-10 05:00:33 -06:00
|
|
|
- template: templates/variables-common.yml
|
|
|
|
# platform specific variables, links to
|
2020-08-18 02:28:59 -05:00
|
|
|
- template: ${{ parameters.VARIABLES_FILE }}
|
2019-02-23 14:23:17 -06:00
|
|
|
|
|
|
|
jobs:
|
|
|
|
- job: Build
|
|
|
|
pool:
|
2020-02-10 03:33:32 -06:00
|
|
|
vmImage: $(VM_IMAGE)
|
2019-02-23 14:23:17 -06:00
|
|
|
container:
|
2020-02-10 05:00:33 -06:00
|
|
|
image: $(DOCKER_BUILD_IMAGE)
|
2020-08-07 03:27:28 -05:00
|
|
|
options: --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --privileged --env container=docker
|
2019-02-23 14:23:17 -06:00
|
|
|
steps:
|
2020-02-10 05:00:33 -06:00
|
|
|
- template: templates/${{ variables.PREPARE_BUILD_TEMPLATE }}
|
2021-02-17 09:29:02 -06:00
|
|
|
- template: templates/${{ variables.AUTOCONF_TEMPLATE }}
|
2019-05-17 07:40:03 -05:00
|
|
|
- script: |
|
2019-06-15 02:27:46 -05:00
|
|
|
set -e
|
2019-05-17 07:40:03 -05:00
|
|
|
git update-ref refs/heads/$(System.PullRequest.TargetBranch) origin/$(System.PullRequest.TargetBranch)
|
|
|
|
make V=0 "GIT_BRANCH=$(System.PullRequest.TargetBranch)" fastlint
|
|
|
|
displayName: Quick code style check
|
|
|
|
condition: eq(variables['Build.Reason'], 'PullRequest')
|
2020-02-10 05:00:33 -06:00
|
|
|
- template: templates/${{ variables.BUILD_TEMPLATE }}
|
Azure: Add support for testing multi IPA environments
Currently, only one IPA environment is tested within Docker
containers. This is not efficient because Azure's agent gives
6 GB of physical memory and 13 GB of total memory (Feb 2020),
but limits CPU with 2 cores.
Next examples are for 'master-only' topologies.
Let's assume that only one member of github repo simultaneously
run CI. This allows to get the full strength of Azure.
Concurrency results for TestInstallMaster:
------------------------------------------
| job concurrency | time/jobs |
------------------------------------------
| 5 | 40/5 |
| 4 | 34/4 |
| 3 | 25/3 |
| 2 | 19/2 |
| 1 | 17/1 |
------------------------------------------
Results prove the limitation of 2 cores. So, in case of jobs'
number not exceeds the max capacity for parallel jobs(10) the
proposed method couldn't save time, but it reduces the used
jobs number up to 2 times. In other words, in this case CI
could pass 2 x tests.
But what if CI was triggered by several PRs? or jobs' number is
bigger than 10. For example, there are 20 tests to be run.
Concurrency results for TestInstallMaster and 20 input jobs:
------------------------------------------------------------------
| job concurrency | time | jobs used | jobs free |
------------------------------------------------------------------
| 5 | 40 | 4 | 6 |
| 4 | 34 | 5 | 5 |
| 3 | 25 | 7 | 3 |
| 2 | 19 | 10 | 0 |
| 1 | 34 | 20 | 0 |
------------------------------------------------------------------
So, in this case the optimal concurrency would be 4 since it
allows to run two CIs simultaneously (20 tasks on board) and get
results in 34 minutes for both. In other words, two people could
trigger CI from PR and don't wait for each other.
New Azure IPA tests workflow:
+ 1) generate-matrix.py script generates JSON from user's YAML [0]
2) Azure generate jobs using Matrix strategy
3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) GNU 'parallel' launch each IPA environment in parallel:
+ 1) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ 2) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ 3) launch IPA tests on tests' controller
c) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
d) publish regular system logs as artifacts
[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
|
|
|
- template: templates/publish-build.yml
|
|
|
|
parameters:
|
2021-03-10 14:49:59 -06:00
|
|
|
artifactName: 'packages'
|
Azure: Add support for testing multi IPA environments
Currently, only one IPA environment is tested within Docker
containers. This is not efficient because Azure's agent gives
6 GB of physical memory and 13 GB of total memory (Feb 2020),
but limits CPU with 2 cores.
Next examples are for 'master-only' topologies.
Let's assume that only one member of github repo simultaneously
run CI. This allows to get the full strength of Azure.
Concurrency results for TestInstallMaster:
------------------------------------------
| job concurrency | time/jobs |
------------------------------------------
| 5 | 40/5 |
| 4 | 34/4 |
| 3 | 25/3 |
| 2 | 19/2 |
| 1 | 17/1 |
------------------------------------------
Results prove the limitation of 2 cores. So, in case of jobs'
number not exceeds the max capacity for parallel jobs(10) the
proposed method couldn't save time, but it reduces the used
jobs number up to 2 times. In other words, in this case CI
could pass 2 x tests.
But what if CI was triggered by several PRs? or jobs' number is
bigger than 10. For example, there are 20 tests to be run.
Concurrency results for TestInstallMaster and 20 input jobs:
------------------------------------------------------------------
| job concurrency | time | jobs used | jobs free |
------------------------------------------------------------------
| 5 | 40 | 4 | 6 |
| 4 | 34 | 5 | 5 |
| 3 | 25 | 7 | 3 |
| 2 | 19 | 10 | 0 |
| 1 | 34 | 20 | 0 |
------------------------------------------------------------------
So, in this case the optimal concurrency would be 4 since it
allows to run two CIs simultaneously (20 tasks on board) and get
results in 34 minutes for both. In other words, two people could
trigger CI from PR and don't wait for each other.
New Azure IPA tests workflow:
+ 1) generate-matrix.py script generates JSON from user's YAML [0]
2) Azure generate jobs using Matrix strategy
3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) GNU 'parallel' launch each IPA environment in parallel:
+ 1) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ 2) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ 3) launch IPA tests on tests' controller
c) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
d) publish regular system logs as artifacts
[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
|
|
|
targetPath: $(Build.Repository.LocalPath)/dist
|
|
|
|
displayName: Publish packages
|
|
|
|
|
2019-02-23 14:23:17 -06:00
|
|
|
- script: |
|
2019-06-15 02:27:46 -05:00
|
|
|
set -e
|
2019-02-23 14:23:17 -06:00
|
|
|
mkdir container
|
|
|
|
cp -pr dist container/
|
Azure: Add support for testing multi IPA environments
Currently, only one IPA environment is tested within Docker
containers. This is not efficient because Azure's agent gives
6 GB of physical memory and 13 GB of total memory (Feb 2020),
but limits CPU with 2 cores.
Next examples are for 'master-only' topologies.
Let's assume that only one member of github repo simultaneously
run CI. This allows to get the full strength of Azure.
Concurrency results for TestInstallMaster:
------------------------------------------
| job concurrency | time/jobs |
------------------------------------------
| 5 | 40/5 |
| 4 | 34/4 |
| 3 | 25/3 |
| 2 | 19/2 |
| 1 | 17/1 |
------------------------------------------
Results prove the limitation of 2 cores. So, in case of jobs'
number not exceeds the max capacity for parallel jobs(10) the
proposed method couldn't save time, but it reduces the used
jobs number up to 2 times. In other words, in this case CI
could pass 2 x tests.
But what if CI was triggered by several PRs? or jobs' number is
bigger than 10. For example, there are 20 tests to be run.
Concurrency results for TestInstallMaster and 20 input jobs:
------------------------------------------------------------------
| job concurrency | time | jobs used | jobs free |
------------------------------------------------------------------
| 5 | 40 | 4 | 6 |
| 4 | 34 | 5 | 5 |
| 3 | 25 | 7 | 3 |
| 2 | 19 | 10 | 0 |
| 1 | 34 | 20 | 0 |
------------------------------------------------------------------
So, in this case the optimal concurrency would be 4 since it
allows to run two CIs simultaneously (20 tasks on board) and get
results in 34 minutes for both. In other words, two people could
trigger CI from PR and don't wait for each other.
New Azure IPA tests workflow:
+ 1) generate-matrix.py script generates JSON from user's YAML [0]
2) Azure generate jobs using Matrix strategy
3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) GNU 'parallel' launch each IPA environment in parallel:
+ 1) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ 2) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ 3) launch IPA tests on tests' controller
c) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
d) publish regular system logs as artifacts
[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
|
|
|
cp $(IPA_TESTS_DOCKERFILES)/$(DOCKER_DOCKERFILE) container/Dockerfile
|
2019-02-23 14:23:17 -06:00
|
|
|
cd container
|
2020-02-10 05:00:33 -06:00
|
|
|
docker build -t freeipa-azure-builder .
|
|
|
|
docker save freeipa-azure-builder | gzip > '$(builddir)/freeipa-azure-builder-container.tar.gz'
|
2019-02-23 14:23:17 -06:00
|
|
|
displayName: Create container image for test
|
|
|
|
- template: templates/publish-build.yml
|
Azure: Add support for testing multi IPA environments
Currently, only one IPA environment is tested within Docker
containers. This is not efficient because Azure's agent gives
6 GB of physical memory and 13 GB of total memory (Feb 2020),
but limits CPU with 2 cores.
Next examples are for 'master-only' topologies.
Let's assume that only one member of github repo simultaneously
run CI. This allows to get the full strength of Azure.
Concurrency results for TestInstallMaster:
------------------------------------------
| job concurrency | time/jobs |
------------------------------------------
| 5 | 40/5 |
| 4 | 34/4 |
| 3 | 25/3 |
| 2 | 19/2 |
| 1 | 17/1 |
------------------------------------------
Results prove the limitation of 2 cores. So, in case of jobs'
number not exceeds the max capacity for parallel jobs(10) the
proposed method couldn't save time, but it reduces the used
jobs number up to 2 times. In other words, in this case CI
could pass 2 x tests.
But what if CI was triggered by several PRs? or jobs' number is
bigger than 10. For example, there are 20 tests to be run.
Concurrency results for TestInstallMaster and 20 input jobs:
------------------------------------------------------------------
| job concurrency | time | jobs used | jobs free |
------------------------------------------------------------------
| 5 | 40 | 4 | 6 |
| 4 | 34 | 5 | 5 |
| 3 | 25 | 7 | 3 |
| 2 | 19 | 10 | 0 |
| 1 | 34 | 20 | 0 |
------------------------------------------------------------------
So, in this case the optimal concurrency would be 4 since it
allows to run two CIs simultaneously (20 tasks on board) and get
results in 34 minutes for both. In other words, two people could
trigger CI from PR and don't wait for each other.
New Azure IPA tests workflow:
+ 1) generate-matrix.py script generates JSON from user's YAML [0]
2) Azure generate jobs using Matrix strategy
3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) GNU 'parallel' launch each IPA environment in parallel:
+ 1) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ 2) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ 3) launch IPA tests on tests' controller
c) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
d) publish regular system logs as artifacts
[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
|
|
|
parameters:
|
2021-03-10 14:49:59 -06:00
|
|
|
artifactName: 'image'
|
Azure: Add support for testing multi IPA environments
Currently, only one IPA environment is tested within Docker
containers. This is not efficient because Azure's agent gives
6 GB of physical memory and 13 GB of total memory (Feb 2020),
but limits CPU with 2 cores.
Next examples are for 'master-only' topologies.
Let's assume that only one member of github repo simultaneously
run CI. This allows to get the full strength of Azure.
Concurrency results for TestInstallMaster:
------------------------------------------
| job concurrency | time/jobs |
------------------------------------------
| 5 | 40/5 |
| 4 | 34/4 |
| 3 | 25/3 |
| 2 | 19/2 |
| 1 | 17/1 |
------------------------------------------
Results prove the limitation of 2 cores. So, in case of jobs'
number not exceeds the max capacity for parallel jobs(10) the
proposed method couldn't save time, but it reduces the used
jobs number up to 2 times. In other words, in this case CI
could pass 2 x tests.
But what if CI was triggered by several PRs? or jobs' number is
bigger than 10. For example, there are 20 tests to be run.
Concurrency results for TestInstallMaster and 20 input jobs:
------------------------------------------------------------------
| job concurrency | time | jobs used | jobs free |
------------------------------------------------------------------
| 5 | 40 | 4 | 6 |
| 4 | 34 | 5 | 5 |
| 3 | 25 | 7 | 3 |
| 2 | 19 | 10 | 0 |
| 1 | 34 | 20 | 0 |
------------------------------------------------------------------
So, in this case the optimal concurrency would be 4 since it
allows to run two CIs simultaneously (20 tasks on board) and get
results in 34 minutes for both. In other words, two people could
trigger CI from PR and don't wait for each other.
New Azure IPA tests workflow:
+ 1) generate-matrix.py script generates JSON from user's YAML [0]
2) Azure generate jobs using Matrix strategy
3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) GNU 'parallel' launch each IPA environment in parallel:
+ 1) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ 2) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ 3) launch IPA tests on tests' controller
c) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
d) publish regular system logs as artifacts
[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
|
|
|
targetPath: $(Build.Repository.LocalPath)/freeipa-azure-builder-container.tar.gz
|
|
|
|
displayName: Publish container image
|
|
|
|
- template: templates/generate-matrix.yml
|
|
|
|
parameters:
|
|
|
|
definition: 'ipatests/azure/azure_definitions/gating.yml'
|
|
|
|
displayName: Generate Matrix for Gating tests
|
|
|
|
name: gating_matrix
|
|
|
|
- template: templates/generate-matrix.yml
|
|
|
|
parameters:
|
|
|
|
definition: 'ipatests/azure/azure_definitions/base.yml'
|
|
|
|
displayName: Generate Matrix for Base tests
|
|
|
|
name: base_matrix
|
2019-02-23 14:23:17 -06:00
|
|
|
|
|
|
|
- job: Lint
|
|
|
|
pool:
|
2020-02-10 03:33:32 -06:00
|
|
|
vmImage: $(VM_IMAGE)
|
2019-02-23 14:23:17 -06:00
|
|
|
container:
|
2020-02-10 05:00:33 -06:00
|
|
|
image: $(DOCKER_BUILD_IMAGE)
|
2020-08-07 03:27:28 -05:00
|
|
|
options: --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --privileged --env container=docker
|
2019-02-23 14:23:17 -06:00
|
|
|
steps:
|
2020-02-10 05:00:33 -06:00
|
|
|
- template: templates/${{ variables.PREPARE_BUILD_TEMPLATE }}
|
2021-02-17 09:29:02 -06:00
|
|
|
- template: templates/${{ variables.AUTOCONF_TEMPLATE }}
|
2019-02-23 14:23:17 -06:00
|
|
|
- script: |
|
2019-06-15 02:27:46 -05:00
|
|
|
set -e
|
2019-02-23 14:23:17 -06:00
|
|
|
echo "Running make target 'lint'"
|
|
|
|
make V=0 lint
|
2021-03-10 14:49:59 -06:00
|
|
|
displayName: Lint sources
|
|
|
|
- template: templates/${{ variables.BUILD_DOCS_TEMPLATE }}
|
|
|
|
- template: templates/publish-build.yml
|
2020-03-18 07:19:55 -05:00
|
|
|
parameters:
|
2021-03-10 14:49:59 -06:00
|
|
|
artifactName: 'docs'
|
|
|
|
targetPath: doc/_build/html
|
|
|
|
displayName: Publish docs
|
2019-02-23 14:23:17 -06:00
|
|
|
|
|
|
|
- job: Tox
|
|
|
|
pool:
|
2020-02-10 03:33:32 -06:00
|
|
|
vmImage: $(VM_IMAGE)
|
2019-02-23 14:23:17 -06:00
|
|
|
container:
|
2020-02-10 05:00:33 -06:00
|
|
|
image: $(DOCKER_BUILD_IMAGE)
|
2020-08-07 03:27:28 -05:00
|
|
|
options: --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --privileged --env container=docker
|
2019-02-23 14:23:17 -06:00
|
|
|
steps:
|
2020-02-10 05:00:33 -06:00
|
|
|
- template: templates/${{ variables.PREPARE_BUILD_TEMPLATE }}
|
|
|
|
- template: templates/${{ variables.PREPARE_TOX_TEMPLATE }}
|
2019-02-23 14:23:17 -06:00
|
|
|
- script: |
|
2019-06-15 02:27:46 -05:00
|
|
|
set -e
|
2019-02-23 14:23:17 -06:00
|
|
|
echo "Running tox"
|
|
|
|
export LANG=en_US.utf8
|
|
|
|
export LC_CTYPE=en_US.utf8
|
|
|
|
locale
|
2020-08-27 07:59:32 -05:00
|
|
|
$(TOX_COMMAND) -e py3,pypi,pylint3 -vv
|
2019-02-23 14:23:17 -06:00
|
|
|
displayName: Tox
|
|
|
|
- task: PublishTestResults@2
|
|
|
|
inputs:
|
|
|
|
testResultsFiles: '.tox/**/junit-*.xml'
|
|
|
|
testRunTitle: 'Tox results'
|
|
|
|
condition: succeededOrFailed()
|
|
|
|
|
|
|
|
- job: WebUI_Unit_Tests
|
|
|
|
pool:
|
2020-02-10 03:33:32 -06:00
|
|
|
vmImage: $(VM_IMAGE)
|
2019-02-23 14:23:17 -06:00
|
|
|
container:
|
2020-02-10 05:00:33 -06:00
|
|
|
image: $(DOCKER_BUILD_IMAGE)
|
2020-08-07 03:27:28 -05:00
|
|
|
options: --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --privileged --env container=docker
|
2019-02-23 14:23:17 -06:00
|
|
|
steps:
|
2020-02-10 05:00:33 -06:00
|
|
|
- template: templates/${{ variables.PREPARE_BUILD_TEMPLATE }}
|
|
|
|
- template: templates/${{ variables.PREPARE_WEBUI_TEMPLATE }}
|
2021-02-17 09:29:02 -06:00
|
|
|
- template: templates/${{ variables.AUTOCONF_TEMPLATE }}
|
2019-02-23 14:23:17 -06:00
|
|
|
- script: |
|
2019-06-15 02:27:46 -05:00
|
|
|
set -e
|
2019-02-23 14:23:17 -06:00
|
|
|
echo "Running WebUI unit tests"
|
|
|
|
cd $(builddir)/install/ui/js/libs && make
|
|
|
|
cd $(builddir)/install/ui && npm install
|
|
|
|
cd $(builddir)/install/ui && node_modules/grunt/bin/grunt --verbose test
|
|
|
|
displayName: WebUI Unit Tests
|
|
|
|
- task: PublishTestResults@2
|
|
|
|
inputs:
|
|
|
|
testResultsFiles: 'install/ui/_build/test-reports/TEST-*.xml'
|
|
|
|
testRunTitle: 'Web UI unit test results'
|
|
|
|
condition: succeededOrFailed()
|
|
|
|
|
2020-02-10 03:33:32 -06:00
|
|
|
- job: BASE_XMLRPC
|
|
|
|
pool:
|
|
|
|
vmImage: $(VM_IMAGE)
|
|
|
|
dependsOn: Build
|
2021-03-10 14:49:59 -06:00
|
|
|
variables:
|
|
|
|
IPA_IMAGE_ARTIFACT: $[ dependencies.Build.outputs['artifacts_image.image'] ]
|
|
|
|
IPA_PACKAGES_ARTIFACT: $[ dependencies.Build.outputs['artifacts_packages.packages'] ]
|
2020-02-10 03:33:32 -06:00
|
|
|
condition: succeeded()
|
|
|
|
strategy:
|
Azure: Add support for testing multi IPA environments
Currently, only one IPA environment is tested within Docker
containers. This is not efficient because Azure's agent gives
6 GB of physical memory and 13 GB of total memory (Feb 2020),
but limits CPU with 2 cores.
Next examples are for 'master-only' topologies.
Let's assume that only one member of github repo simultaneously
run CI. This allows to get the full strength of Azure.
Concurrency results for TestInstallMaster:
------------------------------------------
| job concurrency | time/jobs |
------------------------------------------
| 5 | 40/5 |
| 4 | 34/4 |
| 3 | 25/3 |
| 2 | 19/2 |
| 1 | 17/1 |
------------------------------------------
Results prove the limitation of 2 cores. So, in case of jobs'
number not exceeds the max capacity for parallel jobs(10) the
proposed method couldn't save time, but it reduces the used
jobs number up to 2 times. In other words, in this case CI
could pass 2 x tests.
But what if CI was triggered by several PRs? or jobs' number is
bigger than 10. For example, there are 20 tests to be run.
Concurrency results for TestInstallMaster and 20 input jobs:
------------------------------------------------------------------
| job concurrency | time | jobs used | jobs free |
------------------------------------------------------------------
| 5 | 40 | 4 | 6 |
| 4 | 34 | 5 | 5 |
| 3 | 25 | 7 | 3 |
| 2 | 19 | 10 | 0 |
| 1 | 34 | 20 | 0 |
------------------------------------------------------------------
So, in this case the optimal concurrency would be 4 since it
allows to run two CIs simultaneously (20 tasks on board) and get
results in 34 minutes for both. In other words, two people could
trigger CI from PR and don't wait for each other.
New Azure IPA tests workflow:
+ 1) generate-matrix.py script generates JSON from user's YAML [0]
2) Azure generate jobs using Matrix strategy
3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) GNU 'parallel' launch each IPA environment in parallel:
+ 1) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ 2) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ 3) launch IPA tests on tests' controller
c) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
d) publish regular system logs as artifacts
[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
|
|
|
matrix: $[ dependencies.Build.outputs['base_matrix.matrix'] ]
|
2020-02-10 03:33:32 -06:00
|
|
|
steps:
|
Azure: Add support for testing multi IPA environments
Currently, only one IPA environment is tested within Docker
containers. This is not efficient because Azure's agent gives
6 GB of physical memory and 13 GB of total memory (Feb 2020),
but limits CPU with 2 cores.
Next examples are for 'master-only' topologies.
Let's assume that only one member of github repo simultaneously
run CI. This allows to get the full strength of Azure.
Concurrency results for TestInstallMaster:
------------------------------------------
| job concurrency | time/jobs |
------------------------------------------
| 5 | 40/5 |
| 4 | 34/4 |
| 3 | 25/3 |
| 2 | 19/2 |
| 1 | 17/1 |
------------------------------------------
Results prove the limitation of 2 cores. So, in case of jobs'
number not exceeds the max capacity for parallel jobs(10) the
proposed method couldn't save time, but it reduces the used
jobs number up to 2 times. In other words, in this case CI
could pass 2 x tests.
But what if CI was triggered by several PRs? or jobs' number is
bigger than 10. For example, there are 20 tests to be run.
Concurrency results for TestInstallMaster and 20 input jobs:
------------------------------------------------------------------
| job concurrency | time | jobs used | jobs free |
------------------------------------------------------------------
| 5 | 40 | 4 | 6 |
| 4 | 34 | 5 | 5 |
| 3 | 25 | 7 | 3 |
| 2 | 19 | 10 | 0 |
| 1 | 34 | 20 | 0 |
------------------------------------------------------------------
So, in this case the optimal concurrency would be 4 since it
allows to run two CIs simultaneously (20 tasks on board) and get
results in 34 minutes for both. In other words, two people could
trigger CI from PR and don't wait for each other.
New Azure IPA tests workflow:
+ 1) generate-matrix.py script generates JSON from user's YAML [0]
2) Azure generate jobs using Matrix strategy
3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) GNU 'parallel' launch each IPA environment in parallel:
+ 1) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ 2) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ 3) launch IPA tests on tests' controller
c) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
d) publish regular system logs as artifacts
[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
|
|
|
- template: templates/generate-job-variables.yml
|
|
|
|
- template: templates/test-jobs.yml
|
2020-02-10 03:33:32 -06:00
|
|
|
|
|
|
|
- job: GATING
|
|
|
|
pool:
|
|
|
|
vmImage: $(VM_IMAGE)
|
|
|
|
dependsOn: Build
|
2021-03-10 14:49:59 -06:00
|
|
|
variables:
|
|
|
|
IPA_IMAGE_ARTIFACT: $[ dependencies.Build.outputs['artifacts_image.image'] ]
|
|
|
|
IPA_PACKAGES_ARTIFACT: $[ dependencies.Build.outputs['artifacts_packages.packages'] ]
|
2020-02-10 03:33:32 -06:00
|
|
|
condition: succeeded()
|
|
|
|
strategy:
|
Azure: Add support for testing multi IPA environments
Currently, only one IPA environment is tested within Docker
containers. This is not efficient because Azure's agent gives
6 GB of physical memory and 13 GB of total memory (Feb 2020),
but limits CPU with 2 cores.
Next examples are for 'master-only' topologies.
Let's assume that only one member of github repo simultaneously
run CI. This allows to get the full strength of Azure.
Concurrency results for TestInstallMaster:
------------------------------------------
| job concurrency | time/jobs |
------------------------------------------
| 5 | 40/5 |
| 4 | 34/4 |
| 3 | 25/3 |
| 2 | 19/2 |
| 1 | 17/1 |
------------------------------------------
Results prove the limitation of 2 cores. So, in case of jobs'
number not exceeds the max capacity for parallel jobs(10) the
proposed method couldn't save time, but it reduces the used
jobs number up to 2 times. In other words, in this case CI
could pass 2 x tests.
But what if CI was triggered by several PRs? or jobs' number is
bigger than 10. For example, there are 20 tests to be run.
Concurrency results for TestInstallMaster and 20 input jobs:
------------------------------------------------------------------
| job concurrency | time | jobs used | jobs free |
------------------------------------------------------------------
| 5 | 40 | 4 | 6 |
| 4 | 34 | 5 | 5 |
| 3 | 25 | 7 | 3 |
| 2 | 19 | 10 | 0 |
| 1 | 34 | 20 | 0 |
------------------------------------------------------------------
So, in this case the optimal concurrency would be 4 since it
allows to run two CIs simultaneously (20 tasks on board) and get
results in 34 minutes for both. In other words, two people could
trigger CI from PR and don't wait for each other.
New Azure IPA tests workflow:
+ 1) generate-matrix.py script generates JSON from user's YAML [0]
2) Azure generate jobs using Matrix strategy
3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) GNU 'parallel' launch each IPA environment in parallel:
+ 1) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ 2) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ 3) launch IPA tests on tests' controller
c) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
d) publish regular system logs as artifacts
[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
|
|
|
matrix: $[ dependencies.Build.outputs['gating_matrix.matrix'] ]
|
2020-02-10 03:33:32 -06:00
|
|
|
timeoutInMinutes: 90
|
|
|
|
steps:
|
Azure: Add support for testing multi IPA environments
Currently, only one IPA environment is tested within Docker
containers. This is not efficient because Azure's agent gives
6 GB of physical memory and 13 GB of total memory (Feb 2020),
but limits CPU with 2 cores.
Next examples are for 'master-only' topologies.
Let's assume that only one member of github repo simultaneously
run CI. This allows to get the full strength of Azure.
Concurrency results for TestInstallMaster:
------------------------------------------
| job concurrency | time/jobs |
------------------------------------------
| 5 | 40/5 |
| 4 | 34/4 |
| 3 | 25/3 |
| 2 | 19/2 |
| 1 | 17/1 |
------------------------------------------
Results prove the limitation of 2 cores. So, in case of jobs'
number not exceeds the max capacity for parallel jobs(10) the
proposed method couldn't save time, but it reduces the used
jobs number up to 2 times. In other words, in this case CI
could pass 2 x tests.
But what if CI was triggered by several PRs? or jobs' number is
bigger than 10. For example, there are 20 tests to be run.
Concurrency results for TestInstallMaster and 20 input jobs:
------------------------------------------------------------------
| job concurrency | time | jobs used | jobs free |
------------------------------------------------------------------
| 5 | 40 | 4 | 6 |
| 4 | 34 | 5 | 5 |
| 3 | 25 | 7 | 3 |
| 2 | 19 | 10 | 0 |
| 1 | 34 | 20 | 0 |
------------------------------------------------------------------
So, in this case the optimal concurrency would be 4 since it
allows to run two CIs simultaneously (20 tasks on board) and get
results in 34 minutes for both. In other words, two people could
trigger CI from PR and don't wait for each other.
New Azure IPA tests workflow:
+ 1) generate-matrix.py script generates JSON from user's YAML [0]
2) Azure generate jobs using Matrix strategy
3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04):
a) downloads prepared Docker container image (artifact) from Azure cloud
(built on Build Job) and loads the received image into local pool
+ b) GNU 'parallel' launch each IPA environment in parallel:
+ 1) docker-compose creates the Docker environment having a required number
of replicas and/or clients
+ 2) setup_containers.py script does the needed container's changes (DNS,
SSH, etc.)
+ 3) launch IPA tests on tests' controller
c) publish tests results in JUnit format to provide a comprehensive test
reporting and analytics experience via Azure WebUI [1]
d) publish regular system logs as artifacts
[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Fixes: https://pagure.io/freeipa/issue/8202
Signed-off-by: Stanislav Levin <slev@altlinux.org>
Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
|
|
|
- template: templates/generate-job-variables.yml
|
|
|
|
- template: templates/test-jobs.yml
|