IntenseWebs/veilid
3
0
Fork 0
mirror of https://gitlab.com/veilid/veilid.git synced 2024-11-21 16:37:39 -06:00
Code Issues Packages Projects Releases Wiki Activity
59630dbe8c
veilid/.gitlab-ci.yml

419 lines
13 KiB
YAML
Raw Normal View History

cleanup and stats accounting organization
2021-11-26 08:54:38 -06:00
variables:
use new runner
2022-11-22 16:16:53 -06:00
NO_DOCKER: 1
FORCE_COLOR: 1
EARTHLY_EXEC_CMD: "/bin/sh"
Add submodule strategyt to build
2022-11-26 23:18:08 -06:00
GIT_SUBMODULE_STRATEGY: normal
cleanup and stats accounting organization
2021-11-26 08:54:38 -06:00
Add merge_requests as build target, split into two architectures
2023-02-20 13:14:05 -06:00
stages:
Just going to re-run the release but without pushing crates or python. [skip ci]
2024-08-02 10:13:25 -05:00
- prepare
- test
- build_packages
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 19:59:23 -05:00
- release
Update .gitlab-ci.yml file to the intermediary package distribution pipeline.
2023-06-20 21:46:05 -05:00
- distribute
Add pipeline failure job that destroys build machines [ci skip]
2024-09-15 15:20:17 -05:00
- failed
initial CI attempt
2021-11-24 15:49:13 -06:00
Add `format` job
2024-08-10 01:55:24 -05:00
format:
stage: test
image: rust:latest
before_script:
- rustup component add rustfmt
script:
- cargo fmt --all -- --check --verbose
Skip format job during release runs
2024-08-14 22:47:14 -05:00
rules:
- if: $CI_COMMIT_TAG
when: never
- when: always
Just going to re-run the release but without pushing crates or python. [skip ci]
2024-08-02 10:13:25 -05:00
# base earthly setup for jobs
.base:
tags: [ saas-linux-medium-amd64 ]
image: docker
services:
- docker:dind
Refactored Earthfile to improve caching Targets have been parallelized so that initial push of the container cache should build the whole build a little quicker, plus the container should now use the cache for more of the build and so speed up normal builds to just the compilation and test of the code that has changed The `build_cache` target now builds a `build-cache:latest` container that is stored in the GitLab project Container Registry, and then used (if it exists) by the `test_build` target. The `build_cache` task runs under 3 conditions, 1. the container does not exist, 2. if scheduled, 3. if run manually from the Pipelines page in the GitLab interface. It is recommended that the build is set up to run on a weekly schedule via the `Pipeline schedules` page in GitLab with the schedule of `0 2 * * 6`.
2023-10-31 06:17:13 -05:00
Just going to re-run the release but without pushing crates or python. [skip ci]
2024-08-02 10:13:25 -05:00
.earthly: &earthly_setup
- apk update && apk add git
- wget https://github.com/earthly/earthly/releases/download/v0.7.15/earthly-linux-amd64 -O /usr/local/bin/earthly
- chmod +x /usr/local/bin/earthly
- earthly bootstrap
- echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY
- test "$CI_PROJECT_PATH" != "veilid/veilid" && project_args="--CI_REGISTRY_IMAGE=$CI_REGISTRY_IMAGE"
# Create the build container if:
# - no container in the registry
# - run as part of a schedule
# - run manually from the pipelines web page
build_cache:
extends: .base
stage: prepare
script:
- apk update && apk add jq && apk add curl
- if ! docker manifest inspect $CI_REGISTRY_IMAGE/build-cache:latest > /dev/null; then
- CACHE_EPOCH=0
- else
- 'CONT_REPO_ID=$(curl "https://gitlab.com/api/graphql" --header "Content-Type: application/json" --request POST --data "{\"query\": \"query { project(fullPath: \\\"$CI_PROJECT_PATH\\\" ) { containerRepositories( name: \\\"build-cache\\\" ) { nodes { id }}}} \"}" | jq -r ".data.project.containerRepositories.nodes[0].id")'
- echo "CONTAINER REPO ID = $CONT_REPO_ID"
- 'CACHE_TS=$(curl "https://gitlab.com/api/graphql" --header "Content-Type: application/json" --request POST --data "{\"query\": \"query { containerRepository(id: \\\"$CONT_REPO_ID\\\") { tags( first: 1 name: \\\"latest\\\" ) { nodes { location publishedAt }}}} \"}" | jq -r ".data.containerRepository.tags.nodes[0].publishedAt" | cut -d "+" -f1 | sed "s/T/ /g" )'
- CACHE_EPOCH=$(date -d "$CACHE_TS" +%s)
- fi
- EARTHLY_EPOCH=$(git log -1 --format=%ct Earthfile)
- echo "CACHE EPOCH = $CACHE_EPOCH, EARTHLY EPOCH = $EARTHLY_EPOCH"
- if [[ $EARTHLY_EPOCH -gt $CACHE_EPOCH ]] || [[ "$CI_PIPELINE_SOURCE" == "schedule" ]] || [[ "$CI_PIPELINE_SOURCE" == "web" ]] ; then
- *earthly_setup
- earthly --use-inline-cache --save-inline-cache --strict --push -P +build-linux-cache $project_args
- else
- echo "No need to rebuild"
- fi
rules:
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
when: never
- if: $IS_NIGHTLY == "true"
when: never
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_PIPELINE_SOURCE == "push"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
# Runs a basic unit test build, this task will use the `build-cache:latest` as set up in the projects Container Registry
test_build:
extends: .base
stage: test
script:
- *earthly_setup
- earthly --use-inline-cache +unit-tests-linux --BASE=container $project_args
resource_group: test
rules:
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
when: never
- if: $IS_NIGHTLY == "true"
when: never
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_PIPELINE_SOURCE == "push"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
# Actual release -- triggered by pushing a new version tag
release_job:
stage: release
image: registry.gitlab.com/gitlab-org/release-cli:latest
script:
- echo "running release_job"
release: # See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
tag_name: '$CI_COMMIT_TAG'
description: '$CI_COMMIT_TAG'
rules:
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
create_build_machines:
stage: build_packages
tags:
- build-orchestration
script:
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create amd64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create arm64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create amd64-rpm
rules:
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
package_amd64_deb:
stage: build_packages
needs:
- create_build_machines
tags:
- build-amd64-deb
script:
- earthly bootstrap
- earthly +package-linux-amd64-deb
- bash scripts/cicd/build-machine/scp-amd64-debs-to-orchestrator.sh
rules:
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
package_arm64_deb:
stage: build_packages
needs:
- create_build_machines
tags:
- build-arm64-deb
script:
- earthly bootstrap
- earthly +package-linux-arm64-deb
- bash scripts/cicd/build-machine/scp-arm64-debs-to-orchestrator.sh
rules:
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
package_amd64_rpm:
stage: build_packages
needs:
- create_build_machines
tags:
- build-amd64-rpm
script:
- earthly bootstrap
- earthly +package-linux-amd64-rpm
- bash scripts/cicd/build-machine/scp-amd64-rpms-to-orchestrator.sh
rules:
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
# publish_crates:
# stage: build_packages
# needs:
# - package_amd64_deb
# tags:
# - build-amd64-deb
# script:
# - vlt login
# - vlt run --command="cargo publish -p veilid-tools --dry-run"
# - vlt run --command="cargo publish -p veilid-tools"
# - vlt run --command="cargo publish -p veilid-core --dry-run"
# - vlt run --command="cargo publish -p veilid-core"
# rules:
# - if: '$CI_COMMIT_TAG =~ /v\d.+/'
# publish_python:
# stage: build_packages
# needs:
# - package_amd64_deb
# tags:
# - build-amd64-deb
# script:
# - vlt login
# - cd veilid-python && /home/gitlab-runner/.local/bin/poetry build
# - vlt run --command="/home/gitlab-runner/.local/bin/poetry publish"
# rules:
# - if: '$CI_COMMIT_TAG =~ /v\d.+/'
Update .gitlab-ci.yml file
2023-07-16 14:18:35 -05:00
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 18:22:10 -05:00
build_repositories:
stage: distribute
Fixing order of jobs. `Needs` setting of build_repositories was causing stage leap frogging.
2024-08-01 21:58:24 -05:00
#needs:
# - publish_python
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 18:22:10 -05:00
tags:
- build-orchestration
Migrating CICD scripts to Veilid repo Copied CICD scripts into the repository so that the community can make contributions to the build system. Wrote a brief description of the build and distribute process. Modified the CICD config to use the repo hosted scripts. [ci skip]
2024-04-21 13:37:45 -05:00
variables:
Another tweak to GitLab secure storage
2024-07-19 15:49:29 -05:00
SECURE_FILES_DOWNLOAD_PATH: './'
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 18:22:10 -05:00
script:
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 14:08:25 -05:00
- curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash
- cp scripts/cicd/build-orchestration/rpm-repo-building/Dockerfile ~/rpm-build-container
- cp scripts/cicd/build-orchestration/rpm-repo-building/repobuild.sh ~/rpm-build-container
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 17:22:50 -05:00
- cp scripts/cicd/build-orchestration/generate-stable-release.sh ~
Fixed CICD missing params
2024-07-14 22:58:26 -05:00
- bash scripts/cicd/build-orchestration/distribute-stable-packages.sh
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 18:22:10 -05:00
rules:
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Just going to re-run the release but without pushing crates or python. [skip ci]
2024-08-02 10:13:25 -05:00
delete_build_machines:
Update .gitlab-ci.yml file
2023-07-16 14:18:35 -05:00
stage: distribute
needs:
Update .gitlab-ci.yml file [ci skip]
2024-08-02 10:18:13 -05:00
- build_repositories
Just going to re-run the release but without pushing crates or python. [skip ci]
2024-08-02 10:13:25 -05:00
tags:
- build-orchestration
script:
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete amd64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete arm64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete amd64-rpm
rules:
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
# Dryrun release -- triggered by changes in .gitlab-ci.yml, CICD scripts, or Earthfile
dryrun_create_build_machines:
stage: build_packages
tags:
- build-orchestration
script:
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create amd64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create arm64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create amd64-rpm
rules:
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
dryrun_package_amd64_deb:
stage: build_packages
needs:
- dryrun_create_build_machines
tags:
- build-amd64-deb
script:
- earthly bootstrap
- earthly +package-linux-amd64-deb
- bash scripts/cicd/build-machine/scp-amd64-debs-to-orchestrator.sh
rules:
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
dryrun_package_arm64_deb:
stage: build_packages
needs:
- dryrun_create_build_machines
tags:
- build-arm64-deb
script:
- earthly bootstrap
- earthly +package-linux-arm64-deb
- bash scripts/cicd/build-machine/scp-arm64-debs-to-orchestrator.sh
rules:
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
dryrun_package_amd64_rpm:
stage: build_packages
needs:
- dryrun_create_build_machines
tags:
- build-amd64-rpm
script:
- earthly bootstrap
- earthly +package-linux-amd64-rpm
- bash scripts/cicd/build-machine/scp-amd64-rpms-to-orchestrator.sh
rules:
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
dryrun_publish_crates:
stage: build_packages
needs:
- dryrun_create_build_machines
tags:
- build-amd64-deb
script:
- vlt login
- vlt run --command="cargo publish -p veilid-tools --dry-run"
- vlt run --command="cargo publish -p veilid-core --dry-run"
rules:
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
dryrun_publish_python:
stage: build_packages
needs:
- dryrun_create_build_machines
tags:
- build-amd64-deb
script:
- cd veilid-python && /home/gitlab-runner/.local/bin/poetry build
rules:
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
dryrun_build_repositories:
stage: distribute
tags:
- build-orchestration
variables:
SECURE_FILES_DOWNLOAD_PATH: './'
script:
- curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash
- cp scripts/cicd/build-orchestration/generate-release.sh ~
- bash scripts/cicd/build-orchestration/distribute-packages.sh
rules:
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
dryrun_deploy_repos:
stage: distribute
needs:
- dryrun_build_repositories
Update .gitlab-ci.yml file
2023-07-16 14:18:35 -05:00
tags:
- repo-server
script:
Just going to re-run the release but without pushing crates or python. [skip ci]
2024-08-02 10:13:25 -05:00
- ls -al $HOME/repo.tar
Update .gitlab-ci.yml file
2023-08-02 09:14:18 -05:00
rules:
Just going to re-run the release but without pushing crates or python. [skip ci]
2024-08-02 10:13:25 -05:00
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
dryrun_delete_build_machines:
stage: distribute
needs:
- dryrun_deploy_repos
tags:
- build-orchestration
script:
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete amd64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete arm64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete amd64-rpm
rules:
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
# Nightly build pipeline
nightly_create_build_machines:
stage: build_packages
tags:
- build-orchestration
script:
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create amd64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create arm64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create amd64-rpm
rules:
- if: $IS_NIGHTLY == "true"
nightly_package_amd64_deb:
stage: build_packages
needs:
- nightly_create_build_machines
tags:
- build-amd64-deb
script:
- earthly bootstrap
- earthly +package-linux-amd64-deb --IS_NIGHTLY="$IS_NIGHTLY"
- bash scripts/cicd/build-machine/scp-amd64-debs-to-orchestrator.sh
rules:
- if: $IS_NIGHTLY == "true"
Update .gitlab-ci.yml file
2023-08-02 09:14:18 -05:00
Just going to re-run the release but without pushing crates or python. [skip ci]
2024-08-02 10:13:25 -05:00
nightly_package_arm64_deb:
stage: build_packages
needs:
- nightly_create_build_machines
tags:
- build-arm64-deb
script:
- earthly bootstrap
- earthly +package-linux-arm64-deb --IS_NIGHTLY="$IS_NIGHTLY"
- bash scripts/cicd/build-machine/scp-arm64-debs-to-orchestrator.sh
rules:
- if: $IS_NIGHTLY == "true"
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 14:40:07 -05:00
Just going to re-run the release but without pushing crates or python. [skip ci]
2024-08-02 10:13:25 -05:00
nightly_package_amd64_rpm:
stage: build_packages
needs:
- nightly_create_build_machines
tags:
- build-amd64-rpm
script:
- earthly bootstrap
- earthly +package-linux-amd64-rpm --IS_NIGHTLY="$IS_NIGHTLY"
- bash scripts/cicd/build-machine/scp-amd64-rpms-to-orchestrator.sh
rules:
- if: $IS_NIGHTLY == "true"
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 14:40:07 -05:00
Just going to re-run the release but without pushing crates or python. [skip ci]
2024-08-02 10:13:25 -05:00
nightly_build_repositories:
stage: distribute
tags:
- build-orchestration
variables:
SECURE_FILES_DOWNLOAD_PATH: './'
script:
- curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash
- cp scripts/cicd/build-orchestration/generate-nightly-release.sh ~
- bash scripts/cicd/build-orchestration/distribute-nightly-packages.sh
rules:
- if: $IS_NIGHTLY == "true"
nightly_delete_build_machines:
stage: distribute
needs:
- nightly_build_repositories
tags:
- build-orchestration
script:
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete amd64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete arm64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete amd64-rpm
rules:
Add pipeline failure job that destroys build machines [ci skip]
2024-09-15 15:20:17 -05:00
- if: $IS_NIGHTLY == "true"
# If any steps fail this will run the build machine tear down scripts
failed_pipeline_actions:
stage: failed
tags:
- build-orchestration
script:
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete amd64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete arm64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete amd64-rpm
when: on_failure
Reference in New Issue Copy Permalink