freeipa/ipatests/azure/azure-pipelines.yml

156 lines
5.3 KiB
YAML
Raw Normal View History

parameters:
- name: VARIABLES_FILE
default: 'templates/variables.yml'
trigger:
- master
variables:
- template: templates/variables-common.yml
# platform specific variables, links to
- template: ${{ parameters.VARIABLES_FILE }}
jobs:
- job: Build
pool:
2020-02-10 03:33:32 -06:00
vmImage: $(VM_IMAGE)
container:
image: $(DOCKER_BUILD_IMAGE)
options: --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --privileged --env container=docker
steps:
- template: templates/${{ variables.PREPARE_BUILD_TEMPLATE }}
- template: templates/${{ variables.AUTOCONF_TEMPLATE }}
- script: |
set -e
git update-ref refs/heads/$(System.PullRequest.TargetBranch) origin/$(System.PullRequest.TargetBranch)
make V=0 "GIT_BRANCH=$(System.PullRequest.TargetBranch)" fastlint
displayName: Quick code style check
condition: eq(variables['Build.Reason'], 'PullRequest')
- template: templates/${{ variables.BUILD_TEMPLATE }}
Azure: Add support for testing multi IPA environments Currently, only one IPA environment is tested within Docker containers. This is not efficient because Azure's agent gives 6 GB of physical memory and 13 GB of total memory (Feb 2020), but limits CPU with 2 cores. Next examples are for 'master-only' topologies. Let's assume that only one member of github repo simultaneously run CI. This allows to get the full strength of Azure. Concurrency results for TestInstallMaster: ------------------------------------------ | job concurrency | time/jobs | ------------------------------------------ | 5 | 40/5 | | 4 | 34/4 | | 3 | 25/3 | | 2 | 19/2 | | 1 | 17/1 | ------------------------------------------ Results prove the limitation of 2 cores. So, in case of jobs' number not exceeds the max capacity for parallel jobs(10) the proposed method couldn't save time, but it reduces the used jobs number up to 2 times. In other words, in this case CI could pass 2 x tests. But what if CI was triggered by several PRs? or jobs' number is bigger than 10. For example, there are 20 tests to be run. Concurrency results for TestInstallMaster and 20 input jobs: ------------------------------------------------------------------ | job concurrency | time | jobs used | jobs free | ------------------------------------------------------------------ | 5 | 40 | 4 | 6 | | 4 | 34 | 5 | 5 | | 3 | 25 | 7 | 3 | | 2 | 19 | 10 | 0 | | 1 | 34 | 20 | 0 | ------------------------------------------------------------------ So, in this case the optimal concurrency would be 4 since it allows to run two CIs simultaneously (20 tasks on board) and get results in 34 minutes for both. In other words, two people could trigger CI from PR and don't wait for each other. New Azure IPA tests workflow: + 1) generate-matrix.py script generates JSON from user's YAML [0] 2) Azure generate jobs using Matrix strategy 3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04): a) downloads prepared Docker container image (artifact) from Azure cloud (built on Build Job) and loads the received image into local pool + b) GNU 'parallel' launch each IPA environment in parallel: + 1) docker-compose creates the Docker environment having a required number of replicas and/or clients + 2) setup_containers.py script does the needed container's changes (DNS, SSH, etc.) + 3) launch IPA tests on tests' controller c) publish tests results in JUnit format to provide a comprehensive test reporting and analytics experience via Azure WebUI [1] d) publish regular system logs as artifacts [0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml Fixes: https://pagure.io/freeipa/issue/8202 Signed-off-by: Stanislav Levin <slev@altlinux.org> Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
- template: templates/publish-build.yml
parameters:
artifactName: 'packages'
Azure: Add support for testing multi IPA environments Currently, only one IPA environment is tested within Docker containers. This is not efficient because Azure's agent gives 6 GB of physical memory and 13 GB of total memory (Feb 2020), but limits CPU with 2 cores. Next examples are for 'master-only' topologies. Let's assume that only one member of github repo simultaneously run CI. This allows to get the full strength of Azure. Concurrency results for TestInstallMaster: ------------------------------------------ | job concurrency | time/jobs | ------------------------------------------ | 5 | 40/5 | | 4 | 34/4 | | 3 | 25/3 | | 2 | 19/2 | | 1 | 17/1 | ------------------------------------------ Results prove the limitation of 2 cores. So, in case of jobs' number not exceeds the max capacity for parallel jobs(10) the proposed method couldn't save time, but it reduces the used jobs number up to 2 times. In other words, in this case CI could pass 2 x tests. But what if CI was triggered by several PRs? or jobs' number is bigger than 10. For example, there are 20 tests to be run. Concurrency results for TestInstallMaster and 20 input jobs: ------------------------------------------------------------------ | job concurrency | time | jobs used | jobs free | ------------------------------------------------------------------ | 5 | 40 | 4 | 6 | | 4 | 34 | 5 | 5 | | 3 | 25 | 7 | 3 | | 2 | 19 | 10 | 0 | | 1 | 34 | 20 | 0 | ------------------------------------------------------------------ So, in this case the optimal concurrency would be 4 since it allows to run two CIs simultaneously (20 tasks on board) and get results in 34 minutes for both. In other words, two people could trigger CI from PR and don't wait for each other. New Azure IPA tests workflow: + 1) generate-matrix.py script generates JSON from user's YAML [0] 2) Azure generate jobs using Matrix strategy 3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04): a) downloads prepared Docker container image (artifact) from Azure cloud (built on Build Job) and loads the received image into local pool + b) GNU 'parallel' launch each IPA environment in parallel: + 1) docker-compose creates the Docker environment having a required number of replicas and/or clients + 2) setup_containers.py script does the needed container's changes (DNS, SSH, etc.) + 3) launch IPA tests on tests' controller c) publish tests results in JUnit format to provide a comprehensive test reporting and analytics experience via Azure WebUI [1] d) publish regular system logs as artifacts [0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml Fixes: https://pagure.io/freeipa/issue/8202 Signed-off-by: Stanislav Levin <slev@altlinux.org> Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
targetPath: $(Build.Repository.LocalPath)/dist
displayName: Publish packages
- script: |
set -e
mkdir container
cp -pr dist container/
Azure: Add support for testing multi IPA environments Currently, only one IPA environment is tested within Docker containers. This is not efficient because Azure's agent gives 6 GB of physical memory and 13 GB of total memory (Feb 2020), but limits CPU with 2 cores. Next examples are for 'master-only' topologies. Let's assume that only one member of github repo simultaneously run CI. This allows to get the full strength of Azure. Concurrency results for TestInstallMaster: ------------------------------------------ | job concurrency | time/jobs | ------------------------------------------ | 5 | 40/5 | | 4 | 34/4 | | 3 | 25/3 | | 2 | 19/2 | | 1 | 17/1 | ------------------------------------------ Results prove the limitation of 2 cores. So, in case of jobs' number not exceeds the max capacity for parallel jobs(10) the proposed method couldn't save time, but it reduces the used jobs number up to 2 times. In other words, in this case CI could pass 2 x tests. But what if CI was triggered by several PRs? or jobs' number is bigger than 10. For example, there are 20 tests to be run. Concurrency results for TestInstallMaster and 20 input jobs: ------------------------------------------------------------------ | job concurrency | time | jobs used | jobs free | ------------------------------------------------------------------ | 5 | 40 | 4 | 6 | | 4 | 34 | 5 | 5 | | 3 | 25 | 7 | 3 | | 2 | 19 | 10 | 0 | | 1 | 34 | 20 | 0 | ------------------------------------------------------------------ So, in this case the optimal concurrency would be 4 since it allows to run two CIs simultaneously (20 tasks on board) and get results in 34 minutes for both. In other words, two people could trigger CI from PR and don't wait for each other. New Azure IPA tests workflow: + 1) generate-matrix.py script generates JSON from user's YAML [0] 2) Azure generate jobs using Matrix strategy 3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04): a) downloads prepared Docker container image (artifact) from Azure cloud (built on Build Job) and loads the received image into local pool + b) GNU 'parallel' launch each IPA environment in parallel: + 1) docker-compose creates the Docker environment having a required number of replicas and/or clients + 2) setup_containers.py script does the needed container's changes (DNS, SSH, etc.) + 3) launch IPA tests on tests' controller c) publish tests results in JUnit format to provide a comprehensive test reporting and analytics experience via Azure WebUI [1] d) publish regular system logs as artifacts [0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml Fixes: https://pagure.io/freeipa/issue/8202 Signed-off-by: Stanislav Levin <slev@altlinux.org> Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
cp $(IPA_TESTS_DOCKERFILES)/$(DOCKER_DOCKERFILE) container/Dockerfile
cd container
docker build -t freeipa-azure-builder .
docker save freeipa-azure-builder | gzip > '$(builddir)/freeipa-azure-builder-container.tar.gz'
displayName: Create container image for test
- template: templates/publish-build.yml
Azure: Add support for testing multi IPA environments Currently, only one IPA environment is tested within Docker containers. This is not efficient because Azure's agent gives 6 GB of physical memory and 13 GB of total memory (Feb 2020), but limits CPU with 2 cores. Next examples are for 'master-only' topologies. Let's assume that only one member of github repo simultaneously run CI. This allows to get the full strength of Azure. Concurrency results for TestInstallMaster: ------------------------------------------ | job concurrency | time/jobs | ------------------------------------------ | 5 | 40/5 | | 4 | 34/4 | | 3 | 25/3 | | 2 | 19/2 | | 1 | 17/1 | ------------------------------------------ Results prove the limitation of 2 cores. So, in case of jobs' number not exceeds the max capacity for parallel jobs(10) the proposed method couldn't save time, but it reduces the used jobs number up to 2 times. In other words, in this case CI could pass 2 x tests. But what if CI was triggered by several PRs? or jobs' number is bigger than 10. For example, there are 20 tests to be run. Concurrency results for TestInstallMaster and 20 input jobs: ------------------------------------------------------------------ | job concurrency | time | jobs used | jobs free | ------------------------------------------------------------------ | 5 | 40 | 4 | 6 | | 4 | 34 | 5 | 5 | | 3 | 25 | 7 | 3 | | 2 | 19 | 10 | 0 | | 1 | 34 | 20 | 0 | ------------------------------------------------------------------ So, in this case the optimal concurrency would be 4 since it allows to run two CIs simultaneously (20 tasks on board) and get results in 34 minutes for both. In other words, two people could trigger CI from PR and don't wait for each other. New Azure IPA tests workflow: + 1) generate-matrix.py script generates JSON from user's YAML [0] 2) Azure generate jobs using Matrix strategy 3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04): a) downloads prepared Docker container image (artifact) from Azure cloud (built on Build Job) and loads the received image into local pool + b) GNU 'parallel' launch each IPA environment in parallel: + 1) docker-compose creates the Docker environment having a required number of replicas and/or clients + 2) setup_containers.py script does the needed container's changes (DNS, SSH, etc.) + 3) launch IPA tests on tests' controller c) publish tests results in JUnit format to provide a comprehensive test reporting and analytics experience via Azure WebUI [1] d) publish regular system logs as artifacts [0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml Fixes: https://pagure.io/freeipa/issue/8202 Signed-off-by: Stanislav Levin <slev@altlinux.org> Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
parameters:
artifactName: 'image'
Azure: Add support for testing multi IPA environments Currently, only one IPA environment is tested within Docker containers. This is not efficient because Azure's agent gives 6 GB of physical memory and 13 GB of total memory (Feb 2020), but limits CPU with 2 cores. Next examples are for 'master-only' topologies. Let's assume that only one member of github repo simultaneously run CI. This allows to get the full strength of Azure. Concurrency results for TestInstallMaster: ------------------------------------------ | job concurrency | time/jobs | ------------------------------------------ | 5 | 40/5 | | 4 | 34/4 | | 3 | 25/3 | | 2 | 19/2 | | 1 | 17/1 | ------------------------------------------ Results prove the limitation of 2 cores. So, in case of jobs' number not exceeds the max capacity for parallel jobs(10) the proposed method couldn't save time, but it reduces the used jobs number up to 2 times. In other words, in this case CI could pass 2 x tests. But what if CI was triggered by several PRs? or jobs' number is bigger than 10. For example, there are 20 tests to be run. Concurrency results for TestInstallMaster and 20 input jobs: ------------------------------------------------------------------ | job concurrency | time | jobs used | jobs free | ------------------------------------------------------------------ | 5 | 40 | 4 | 6 | | 4 | 34 | 5 | 5 | | 3 | 25 | 7 | 3 | | 2 | 19 | 10 | 0 | | 1 | 34 | 20 | 0 | ------------------------------------------------------------------ So, in this case the optimal concurrency would be 4 since it allows to run two CIs simultaneously (20 tasks on board) and get results in 34 minutes for both. In other words, two people could trigger CI from PR and don't wait for each other. New Azure IPA tests workflow: + 1) generate-matrix.py script generates JSON from user's YAML [0] 2) Azure generate jobs using Matrix strategy 3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04): a) downloads prepared Docker container image (artifact) from Azure cloud (built on Build Job) and loads the received image into local pool + b) GNU 'parallel' launch each IPA environment in parallel: + 1) docker-compose creates the Docker environment having a required number of replicas and/or clients + 2) setup_containers.py script does the needed container's changes (DNS, SSH, etc.) + 3) launch IPA tests on tests' controller c) publish tests results in JUnit format to provide a comprehensive test reporting and analytics experience via Azure WebUI [1] d) publish regular system logs as artifacts [0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml Fixes: https://pagure.io/freeipa/issue/8202 Signed-off-by: Stanislav Levin <slev@altlinux.org> Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
targetPath: $(Build.Repository.LocalPath)/freeipa-azure-builder-container.tar.gz
displayName: Publish container image
- template: templates/generate-matrix.yml
parameters:
definition: 'ipatests/azure/azure_definitions/gating.yml'
displayName: Generate Matrix for Gating tests
name: gating_matrix
- template: templates/generate-matrix.yml
parameters:
definition: 'ipatests/azure/azure_definitions/base.yml'
displayName: Generate Matrix for Base tests
name: base_matrix
- job: Lint
pool:
2020-02-10 03:33:32 -06:00
vmImage: $(VM_IMAGE)
container:
image: $(DOCKER_BUILD_IMAGE)
options: --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --privileged --env container=docker
steps:
- template: templates/${{ variables.PREPARE_BUILD_TEMPLATE }}
- template: templates/${{ variables.AUTOCONF_TEMPLATE }}
- script: |
set -e
echo "Running make target 'lint'"
make V=0 lint
displayName: Lint sources
- template: templates/${{ variables.BUILD_DOCS_TEMPLATE }}
- template: templates/publish-build.yml
2020-03-18 07:19:55 -05:00
parameters:
artifactName: 'docs'
targetPath: doc/_build/html
displayName: Publish docs
- job: Tox
pool:
2020-02-10 03:33:32 -06:00
vmImage: $(VM_IMAGE)
container:
image: $(DOCKER_BUILD_IMAGE)
options: --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --privileged --env container=docker
steps:
- template: templates/${{ variables.PREPARE_BUILD_TEMPLATE }}
- template: templates/${{ variables.PREPARE_TOX_TEMPLATE }}
- script: |
set -e
echo "Running tox"
export LANG=en_US.utf8
export LC_CTYPE=en_US.utf8
locale
$(TOX_COMMAND) -e py3,pypi,pylint3 -vv
displayName: Tox
- task: PublishTestResults@2
inputs:
testResultsFiles: '.tox/**/junit-*.xml'
testRunTitle: 'Tox results'
condition: succeededOrFailed()
- job: WebUI_Unit_Tests
pool:
2020-02-10 03:33:32 -06:00
vmImage: $(VM_IMAGE)
container:
image: $(DOCKER_BUILD_IMAGE)
options: --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --privileged --env container=docker
steps:
- template: templates/${{ variables.PREPARE_BUILD_TEMPLATE }}
- template: templates/${{ variables.PREPARE_WEBUI_TEMPLATE }}
- template: templates/${{ variables.AUTOCONF_TEMPLATE }}
- script: |
set -e
echo "Running WebUI unit tests"
cd $(builddir)/install/ui/js/libs && make
cd $(builddir)/install/ui && npm install
cd $(builddir)/install/ui && node_modules/grunt/bin/grunt --verbose test
displayName: WebUI Unit Tests
- task: PublishTestResults@2
inputs:
testResultsFiles: 'install/ui/_build/test-reports/TEST-*.xml'
testRunTitle: 'Web UI unit test results'
condition: succeededOrFailed()
2020-02-10 03:33:32 -06:00
- job: BASE_XMLRPC
pool:
vmImage: $(VM_IMAGE)
dependsOn: Build
variables:
IPA_IMAGE_ARTIFACT: $[ dependencies.Build.outputs['artifacts_image.image'] ]
IPA_PACKAGES_ARTIFACT: $[ dependencies.Build.outputs['artifacts_packages.packages'] ]
2020-02-10 03:33:32 -06:00
condition: succeeded()
strategy:
Azure: Add support for testing multi IPA environments Currently, only one IPA environment is tested within Docker containers. This is not efficient because Azure's agent gives 6 GB of physical memory and 13 GB of total memory (Feb 2020), but limits CPU with 2 cores. Next examples are for 'master-only' topologies. Let's assume that only one member of github repo simultaneously run CI. This allows to get the full strength of Azure. Concurrency results for TestInstallMaster: ------------------------------------------ | job concurrency | time/jobs | ------------------------------------------ | 5 | 40/5 | | 4 | 34/4 | | 3 | 25/3 | | 2 | 19/2 | | 1 | 17/1 | ------------------------------------------ Results prove the limitation of 2 cores. So, in case of jobs' number not exceeds the max capacity for parallel jobs(10) the proposed method couldn't save time, but it reduces the used jobs number up to 2 times. In other words, in this case CI could pass 2 x tests. But what if CI was triggered by several PRs? or jobs' number is bigger than 10. For example, there are 20 tests to be run. Concurrency results for TestInstallMaster and 20 input jobs: ------------------------------------------------------------------ | job concurrency | time | jobs used | jobs free | ------------------------------------------------------------------ | 5 | 40 | 4 | 6 | | 4 | 34 | 5 | 5 | | 3 | 25 | 7 | 3 | | 2 | 19 | 10 | 0 | | 1 | 34 | 20 | 0 | ------------------------------------------------------------------ So, in this case the optimal concurrency would be 4 since it allows to run two CIs simultaneously (20 tasks on board) and get results in 34 minutes for both. In other words, two people could trigger CI from PR and don't wait for each other. New Azure IPA tests workflow: + 1) generate-matrix.py script generates JSON from user's YAML [0] 2) Azure generate jobs using Matrix strategy 3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04): a) downloads prepared Docker container image (artifact) from Azure cloud (built on Build Job) and loads the received image into local pool + b) GNU 'parallel' launch each IPA environment in parallel: + 1) docker-compose creates the Docker environment having a required number of replicas and/or clients + 2) setup_containers.py script does the needed container's changes (DNS, SSH, etc.) + 3) launch IPA tests on tests' controller c) publish tests results in JUnit format to provide a comprehensive test reporting and analytics experience via Azure WebUI [1] d) publish regular system logs as artifacts [0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml Fixes: https://pagure.io/freeipa/issue/8202 Signed-off-by: Stanislav Levin <slev@altlinux.org> Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
matrix: $[ dependencies.Build.outputs['base_matrix.matrix'] ]
2020-02-10 03:33:32 -06:00
steps:
Azure: Add support for testing multi IPA environments Currently, only one IPA environment is tested within Docker containers. This is not efficient because Azure's agent gives 6 GB of physical memory and 13 GB of total memory (Feb 2020), but limits CPU with 2 cores. Next examples are for 'master-only' topologies. Let's assume that only one member of github repo simultaneously run CI. This allows to get the full strength of Azure. Concurrency results for TestInstallMaster: ------------------------------------------ | job concurrency | time/jobs | ------------------------------------------ | 5 | 40/5 | | 4 | 34/4 | | 3 | 25/3 | | 2 | 19/2 | | 1 | 17/1 | ------------------------------------------ Results prove the limitation of 2 cores. So, in case of jobs' number not exceeds the max capacity for parallel jobs(10) the proposed method couldn't save time, but it reduces the used jobs number up to 2 times. In other words, in this case CI could pass 2 x tests. But what if CI was triggered by several PRs? or jobs' number is bigger than 10. For example, there are 20 tests to be run. Concurrency results for TestInstallMaster and 20 input jobs: ------------------------------------------------------------------ | job concurrency | time | jobs used | jobs free | ------------------------------------------------------------------ | 5 | 40 | 4 | 6 | | 4 | 34 | 5 | 5 | | 3 | 25 | 7 | 3 | | 2 | 19 | 10 | 0 | | 1 | 34 | 20 | 0 | ------------------------------------------------------------------ So, in this case the optimal concurrency would be 4 since it allows to run two CIs simultaneously (20 tasks on board) and get results in 34 minutes for both. In other words, two people could trigger CI from PR and don't wait for each other. New Azure IPA tests workflow: + 1) generate-matrix.py script generates JSON from user's YAML [0] 2) Azure generate jobs using Matrix strategy 3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04): a) downloads prepared Docker container image (artifact) from Azure cloud (built on Build Job) and loads the received image into local pool + b) GNU 'parallel' launch each IPA environment in parallel: + 1) docker-compose creates the Docker environment having a required number of replicas and/or clients + 2) setup_containers.py script does the needed container's changes (DNS, SSH, etc.) + 3) launch IPA tests on tests' controller c) publish tests results in JUnit format to provide a comprehensive test reporting and analytics experience via Azure WebUI [1] d) publish regular system logs as artifacts [0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml Fixes: https://pagure.io/freeipa/issue/8202 Signed-off-by: Stanislav Levin <slev@altlinux.org> Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
- template: templates/generate-job-variables.yml
- template: templates/test-jobs.yml
2020-02-10 03:33:32 -06:00
- job: GATING
pool:
vmImage: $(VM_IMAGE)
dependsOn: Build
variables:
IPA_IMAGE_ARTIFACT: $[ dependencies.Build.outputs['artifacts_image.image'] ]
IPA_PACKAGES_ARTIFACT: $[ dependencies.Build.outputs['artifacts_packages.packages'] ]
2020-02-10 03:33:32 -06:00
condition: succeeded()
strategy:
Azure: Add support for testing multi IPA environments Currently, only one IPA environment is tested within Docker containers. This is not efficient because Azure's agent gives 6 GB of physical memory and 13 GB of total memory (Feb 2020), but limits CPU with 2 cores. Next examples are for 'master-only' topologies. Let's assume that only one member of github repo simultaneously run CI. This allows to get the full strength of Azure. Concurrency results for TestInstallMaster: ------------------------------------------ | job concurrency | time/jobs | ------------------------------------------ | 5 | 40/5 | | 4 | 34/4 | | 3 | 25/3 | | 2 | 19/2 | | 1 | 17/1 | ------------------------------------------ Results prove the limitation of 2 cores. So, in case of jobs' number not exceeds the max capacity for parallel jobs(10) the proposed method couldn't save time, but it reduces the used jobs number up to 2 times. In other words, in this case CI could pass 2 x tests. But what if CI was triggered by several PRs? or jobs' number is bigger than 10. For example, there are 20 tests to be run. Concurrency results for TestInstallMaster and 20 input jobs: ------------------------------------------------------------------ | job concurrency | time | jobs used | jobs free | ------------------------------------------------------------------ | 5 | 40 | 4 | 6 | | 4 | 34 | 5 | 5 | | 3 | 25 | 7 | 3 | | 2 | 19 | 10 | 0 | | 1 | 34 | 20 | 0 | ------------------------------------------------------------------ So, in this case the optimal concurrency would be 4 since it allows to run two CIs simultaneously (20 tasks on board) and get results in 34 minutes for both. In other words, two people could trigger CI from PR and don't wait for each other. New Azure IPA tests workflow: + 1) generate-matrix.py script generates JSON from user's YAML [0] 2) Azure generate jobs using Matrix strategy 3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04): a) downloads prepared Docker container image (artifact) from Azure cloud (built on Build Job) and loads the received image into local pool + b) GNU 'parallel' launch each IPA environment in parallel: + 1) docker-compose creates the Docker environment having a required number of replicas and/or clients + 2) setup_containers.py script does the needed container's changes (DNS, SSH, etc.) + 3) launch IPA tests on tests' controller c) publish tests results in JUnit format to provide a comprehensive test reporting and analytics experience via Azure WebUI [1] d) publish regular system logs as artifacts [0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml Fixes: https://pagure.io/freeipa/issue/8202 Signed-off-by: Stanislav Levin <slev@altlinux.org> Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
matrix: $[ dependencies.Build.outputs['gating_matrix.matrix'] ]
2020-02-10 03:33:32 -06:00
timeoutInMinutes: 90
steps:
Azure: Add support for testing multi IPA environments Currently, only one IPA environment is tested within Docker containers. This is not efficient because Azure's agent gives 6 GB of physical memory and 13 GB of total memory (Feb 2020), but limits CPU with 2 cores. Next examples are for 'master-only' topologies. Let's assume that only one member of github repo simultaneously run CI. This allows to get the full strength of Azure. Concurrency results for TestInstallMaster: ------------------------------------------ | job concurrency | time/jobs | ------------------------------------------ | 5 | 40/5 | | 4 | 34/4 | | 3 | 25/3 | | 2 | 19/2 | | 1 | 17/1 | ------------------------------------------ Results prove the limitation of 2 cores. So, in case of jobs' number not exceeds the max capacity for parallel jobs(10) the proposed method couldn't save time, but it reduces the used jobs number up to 2 times. In other words, in this case CI could pass 2 x tests. But what if CI was triggered by several PRs? or jobs' number is bigger than 10. For example, there are 20 tests to be run. Concurrency results for TestInstallMaster and 20 input jobs: ------------------------------------------------------------------ | job concurrency | time | jobs used | jobs free | ------------------------------------------------------------------ | 5 | 40 | 4 | 6 | | 4 | 34 | 5 | 5 | | 3 | 25 | 7 | 3 | | 2 | 19 | 10 | 0 | | 1 | 34 | 20 | 0 | ------------------------------------------------------------------ So, in this case the optimal concurrency would be 4 since it allows to run two CIs simultaneously (20 tasks on board) and get results in 34 minutes for both. In other words, two people could trigger CI from PR and don't wait for each other. New Azure IPA tests workflow: + 1) generate-matrix.py script generates JSON from user's YAML [0] 2) Azure generate jobs using Matrix strategy 3) each job is run in parallel (up to 10) within its own VM (Ubuntu-18.04): a) downloads prepared Docker container image (artifact) from Azure cloud (built on Build Job) and loads the received image into local pool + b) GNU 'parallel' launch each IPA environment in parallel: + 1) docker-compose creates the Docker environment having a required number of replicas and/or clients + 2) setup_containers.py script does the needed container's changes (DNS, SSH, etc.) + 3) launch IPA tests on tests' controller c) publish tests results in JUnit format to provide a comprehensive test reporting and analytics experience via Azure WebUI [1] d) publish regular system logs as artifacts [0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml Fixes: https://pagure.io/freeipa/issue/8202 Signed-off-by: Stanislav Levin <slev@altlinux.org> Reviewed-By: Alexander Bokovoy <abokovoy@redhat.com>
2020-02-10 10:12:23 -06:00
- template: templates/generate-job-variables.yml
- template: templates/test-jobs.yml