IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[workflows/linux] Switch to sccache and Azure Blob Storage (#20484)

Browse Source 
* [workflows/linux] Switch to sccache and Azure Blob Storage

* Install curl

* Remove --show-config

* Add sccache to other Linux workflows

* sccache to  Android, curl to riscv and CC

* Use sccache action instead of manual install

* Oops, missed sccache manual installation in two places

* Use env vars instead of hardcoded CMAKE_C(XX)_COMPILER_LAUNCHER

* Forgot one more stage in Linux CC pipeline

* Temporarily disable Blob Storage for RISC-V

For some reason sccache has no effect on build time and show 0 hits
and 0 compilation requests despite being in CMake calls

* forgot to add sccache installation to Linux CC

* Revert "Temporarily disable Blob Storage for RISC-V"

This reverts commit b528f41dad583a38b9ef93121e38044b9dccb71b.

* Missing container option for CC build

* Remove curl installation

* Remove CCACHE* variables which have no effect on sccache

* Revert sccache changes for Linux RISC-V workflow
This commit is contained in:
Andrey Babushkin 2023-10-24 20:52:56 +01:00 committed by GitHub
parent 984e4dbf35
commit ee6263a141
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 90 additions and 60 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
.github/workflows/android_arm64.yml vendored
View File

@ -35,11 +35,12 @@ jobs:
image: openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
volumes:
- /mount/caches:/mount/caches
options: -e SCCACHE_AZURE_BLOB_CONTAINER -e SCCACHE_AZURE_CONNECTION_STRING
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
CMAKE_GENERATOR: 'Ninja'
CMAKE_CXX_COMPILER_LAUNCHER: ccache
CMAKE_C_COMPILER_LAUNCHER: ccache
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
OPENVINO_REPO: '/__w/openvino/openvino/openvino'
VCPKG_ROOT: '/__w/openvino/openvino/vcpkg'
BUILD_DIR: '/__w/openvino/openvino/build'
@ -49,10 +50,7 @@ jobs:
ANDROID_ABI_CONFIG: arm64-v8a
VCPKG_DEFAULT_BINARY_CACHE: '/mount/caches/ccache/android_arm64/vcpkg_cache'
VCPKG_FORCE_SYSTEM_BINARIES: '1'
CCACHE_DIR: '/mount/caches/ccache/android_arm64'
CCACHE_TEMPDIR: '/__w/openvino/openvino/ccache_temp'
CCACHE_COMPILERCHECK: 'content'
CCACHE_MAXSIZE: 50G
SCCACHE_AZURE_KEY_PREFIX: android_arm64
steps:
- name: Install git
run: apt-get update && apt-get install --assume-yes --no-install-recommends git ca-certificates
@ -102,6 +100,11 @@ jobs:
unzip commandlinetools-linux-7583922_latest.zip
echo "yes" | ./cmdline-tools/bin/sdkmanager --sdk_root=${ANDROID_TOOLS} --install "ndk-bundle" "platform-tools" "platforms;android-${{ env.ANDROID_SDK_VERSION }}"
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
with:
version: "v0.5.4"
#
# Build
#
@ -138,10 +141,10 @@ jobs:
-B ${BUILD_DIR}
- name: Clean ccache stats
run: ccache --zero-stats --show-config
run: sccache --zero-stats
- name: Cmake - build
run: cmake --build ${BUILD_DIR} --parallel
- name: Show ccache stats
run: ccache --show-stats
run: ${SCCACHE_PATH} --show-stats

18
.github/workflows/fedora.yml vendored
View File

@ -36,19 +36,18 @@ jobs:
image: fedora:33
volumes:
- /mount/caches:/mount/caches
options: -e SCCACHE_AZURE_BLOB_CONTAINER -e SCCACHE_AZURE_CONNECTION_STRING
env:
CMAKE_BUILD_TYPE: 'Release'
CMAKE_GENERATOR: 'Ninja'
CMAKE_CXX_COMPILER_LAUNCHER: ccache
CMAKE_C_COMPILER_LAUNCHER: ccache
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
GITHUB_WORKSPACE: '/__w/openvino/openvino'
OPENVINO_REPO: /__w/openvino/openvino/openvino
INSTALL_DIR: /__w/openvino/openvino/openvino_install
INSTALL_TEST_DIR: /__w/openvino/openvino/tests_install
BUILD_DIR: /__w/openvino/openvino/openvino_build
CCACHE_DIR: /mount/caches/ccache/fedora33_x86_64_Release
CCACHE_TEMPDIR: /__w/openvino/openvino/ccache_temp
CCACHE_MAXSIZE: 50G
SCCACHE_AZURE_KEY_PREFIX: fedora33_x86_64_Release
steps:
- name: Install git
run: yum update -y && yum install -y git
@ -66,6 +65,11 @@ jobs:
- name: Install build dependencies
run: bash ${OPENVINO_REPO}/install_build_dependencies.sh
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
with:
version: "v0.5.4"
- name: Install python dependencies
run: |
python3 -m pip install -U pip
@ -112,8 +116,8 @@ jobs:
- name: Cmake build - OpenVINO
run: cmake --build ${BUILD_DIR} --parallel --verbose
- name: Show ccache stats
run: ccache --show-stats
- name: Show sccache stats
run: ${SCCACHE_PATH} --show-stats
- name: Cmake install - OpenVINO
run: |

59
.github/workflows/linux.yml vendored
View File

@ -43,12 +43,13 @@ jobs:
image: openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
volumes:
- /mount/caches:/mount/caches
options: -e SCCACHE_AZURE_BLOB_CONTAINER -e SCCACHE_AZURE_CONNECTION_STRING
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
CMAKE_BUILD_TYPE: 'Release'
CMAKE_GENERATOR: 'Ninja Multi-Config'
CMAKE_CXX_COMPILER_LAUNCHER: ccache
CMAKE_C_COMPILER_LAUNCHER: ccache
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
GITHUB_WORKSPACE: '/__w/openvino/openvino'
OPENVINO_REPO: /__w/openvino/openvino/openvino
OPENVINO_CONTRIB_REPO: /__w/openvino/openvino/openvino_contrib
@ -56,9 +57,7 @@ jobs:
INSTALL_TEST_DIR: /__w/openvino/openvino/tests_install
DEVELOPER_PACKAGE_DIR: /__w/openvino/openvino/developer_package_install
BUILD_DIR: /__w/openvino/openvino/openvino_build
CCACHE_DIR: /mount/caches/ccache/ubuntu20_x86_64_Release
CCACHE_TEMPDIR: /__w/openvino/openvino/ccache_temp
CCACHE_MAXSIZE: 50G
SCCACHE_AZURE_KEY_PREFIX: ubuntu20_x86_64_Release
ONNX_RUNTIME_UTILS: /__w/openvino/openvino/openvino/.ci/azure/ci_utils/onnxruntime
steps:
@ -92,6 +91,11 @@ jobs:
# libssl1.1 - 'python3 -m pip' in self-hosted runner
apt install --assume-yes --no-install-recommends default-jdk libssl1.1
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
with:
version: "v0.5.4"
- uses: actions/setup-python@v4
with:
python-version: ${{ env.PYTHON_VERSION }}
@ -143,14 +147,14 @@ jobs:
-S ${OPENVINO_REPO} \
-B ${BUILD_DIR}
- name: Clean ccache stats
run: ccache --zero-stats --show-config
- name: Clean sccache stats
run: sccache --zero-stats
- name: Cmake build - OpenVINO
run: cmake --build ${BUILD_DIR} --parallel --config ${{ env.CMAKE_BUILD_TYPE }}
- name: Show ccache stats
run: ccache --show-stats
- name: Show sccache stats
run: ${SCCACHE_PATH} --show-stats
- name: Cmake install - OpenVINO
run: |
@ -497,16 +501,15 @@ jobs:
image: openvinogithubactions.azurecr.io/dockerhub/ubuntu:20.04
volumes:
- /mount/caches:/mount/caches
options: -e SCCACHE_AZURE_BLOB_CONTAINER -e SCCACHE_AZURE_CONNECTION_STRING
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
CMAKE_GENERATOR: 'Ninja Multi-Config'
CMAKE_CXX_COMPILER_LAUNCHER: ccache
CMAKE_C_COMPILER_LAUNCHER: ccache
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
OPENVINO_REPO: /__w/openvino/openvino/openvino
INSTALL_DIR: /__w/openvino/openvino/install
CCACHE_DIR: /mount/caches/ccache/ubuntu20_x86_64_onnxruntime
CCACHE_TEMPDIR: /__w/openvino/openvino/ccache_temp
CCACHE_MAXSIZE: 50G
SCCACHE_AZURE_KEY_PREFIX: ubuntu20_x86_64_onnxruntime
ONNX_RUNTIME_REPO: /__w/openvino/openvino/onnxruntime
ONNX_RUNTIME_UTILS: /__w/openvino/openvino/install/onnxruntime
ONNX_RUNTIME_BUILD_DIR: /__w/openvino/openvino/onnxruntime/build
@ -561,6 +564,11 @@ jobs:
- name: Install Build Dependencies
run: bash ${OPENVINO_REPO}/install_build_dependencies.sh
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
with:
version: "v0.5.4"
- name: Build Lin ONNX Runtime
run: |
source ${INSTALL_DIR}/setupvars.sh
@ -576,6 +584,9 @@ jobs:
env:
CXXFLAGS: "-Wno-error=deprecated-declarations"
- name: Show sccache stats
run: ${SCCACHE_PATH} --show-stats
- name: Run onnxruntime_test_all
run: |
source ${INSTALL_DIR}/setupvars.sh
@ -1346,21 +1357,20 @@ jobs:
image: openvinogithubactions.azurecr.io/dockerhub/nvidia/cuda:11.8.0-runtime-ubuntu20.04
volumes:
- /mount/caches:/mount/caches
options: -e SCCACHE_AZURE_BLOB_CONTAINER -e SCCACHE_AZURE_CONNECTION_STRING
env:
CMAKE_BUILD_TYPE: 'Release'
CMAKE_GENERATOR: 'Ninja Multi-Config'
CMAKE_CUDA_COMPILER_LAUNCHER: ccache
CMAKE_CXX_COMPILER_LAUNCHER: ccache
CMAKE_C_COMPILER_LAUNCHER: ccache
CMAKE_CUDA_COMPILER_LAUNCHER: sccache
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
INSTALL_DIR: /__w/openvino/openvino/install
OPENVINO_DEVELOPER_PACKAGE: /__w/openvino/openvino/install/developer_package
OPENVINO_REPO: /__w/openvino/openvino/openvino
OPENVINO_CONTRIB_REPO: /__w/openvino/openvino/openvino_contrib
NVIDIA_BUILD_DIR: /__w/openvino/openvino/nvidia_plugin_build
DEBIAN_FRONTEND: 'noninteractive'
CCACHE_DIR: /mount/caches/ccache/ubuntu20_x86_64_Release
CCACHE_TEMPDIR: /__w/openvino/openvino/ccache_temp
CCACHE_MAXSIZE: 50G
SCCACHE_AZURE_KEY_PREFIX: ubuntu20_x86_64_Release
steps:
- name: Install Prerequisites
@ -1417,7 +1427,12 @@ jobs:
- name: Install build dependencies
run: |
${OPENVINO_REPO}/install_build_dependencies.sh
apt -y --no-install-recommends install software-properties-common
apt -y --no-install-recommends install software-properties-common curl
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
with:
version: "v0.5.4"
- name: Install CUDA
run: |
@ -1452,4 +1467,4 @@ jobs:
cmake --build ${NVIDIA_BUILD_DIR} --parallel --config ${{ env.CMAKE_BUILD_TYPE }} --verbose -- ov_nvidia_func_tests ov_nvidia_unit_tests
- name: Show ccache stats
run: ccache --show-stats
run: ${SCCACHE_PATH} --show-stats

33
.github/workflows/linux_conditional_compilation.yml vendored
View File

@ -39,21 +39,20 @@ jobs:
image: openvinogithubactions.azurecr.io/dockerhub/ubuntu:22.04
volumes:
- /mount/caches:/mount/caches
options: -e SCCACHE_AZURE_BLOB_CONTAINER -e SCCACHE_AZURE_CONNECTION_STRING
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
CMAKE_BUILD_TYPE: 'Release'
CMAKE_GENERATOR: 'Ninja Multi-Config'
CMAKE_CXX_COMPILER_LAUNCHER: ccache
CMAKE_C_COMPILER_LAUNCHER: ccache
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
GITHUB_WORKSPACE: '/__w/openvino/openvino'
OPENVINO_REPO: /__w/openvino/openvino/openvino
INSTALL_DIR: /__w/openvino/openvino/openvino_install
BUILD_DIR: /__w/openvino/openvino/openvino_build
SELECTIVE_BUILD_STAT_DIR: /__w/openvino/openvino/selective_build_stat
MODELS_PATH: /__w/openvino/openvino/testdata
CCACHE_DIR: /mount/caches/ccache/ubuntu22_x86_64_itt_clang_Release
CCACHE_TEMPDIR: /__w/openvino/openvino/ccache_temp
CCACHE_MAXSIZE: 20G
SCCACHE_AZURE_KEY_PREFIX: ubuntu22_x86_64_itt_clang_Release
steps:
- name: Install git
@ -88,6 +87,11 @@ jobs:
update-alternatives --install /usr/bin/cc cc /usr/bin/clang 100
update-alternatives --install /usr/bin/c++ c++ /usr/bin/clang++ 100
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
with:
version: "v0.5.4"
- uses: actions/setup-python@v4
with:
python-version: ${{ env.PYTHON_VERSION }}
@ -141,8 +145,8 @@ jobs:
cmake --build ${BUILD_DIR} --parallel 8 --config ${{ env.CMAKE_BUILD_TYPE }}
cmake --build ${BUILD_DIR} --parallel --config ${{ env.CMAKE_BUILD_TYPE }} --target sea_itt_lib
- name: Show ccache stats
run: ccache --show-stats
- name: Show sccache stats
run: ${SCCACHE_PATH} --show-stats
- name: Cmake install - OpenVINO
run: cmake -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} -P ${BUILD_DIR}/cmake_install.cmake
@ -205,18 +209,17 @@ jobs:
image: openvinogithubactions.azurecr.io/dockerhub/ubuntu:22.04
volumes:
- /mount/caches:/mount/caches
options: -e SCCACHE_AZURE_BLOB_CONTAINER -e SCCACHE_AZURE_CONNECTION_STRING
env:
DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input
CMAKE_CXX_COMPILER_LAUNCHER: ccache
CMAKE_C_COMPILER_LAUNCHER: ccache
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
GITHUB_WORKSPACE: '/__w/openvino/openvino'
OPENVINO_REPO: /__w/openvino/openvino/openvino
BUILD_DIR: /__w/openvino/openvino/openvino_build
SELECTIVE_BUILD_STAT_DIR: /__w/openvino/openvino/selective_build_stat
MODELS_PATH: /__w/openvino/openvino/testdata
CCACHE_DIR: /mount/caches/ccache/ubuntu22_x86_64_cc_Release
CCACHE_TEMPDIR: /__w/openvino/openvino/ccache_temp
CCACHE_MAXSIZE: 20G
SCCACHE_AZURE_KEY_PREFIX: ubuntu22_x86_64_cc_Release
steps:
- name: Install git
@ -252,6 +255,10 @@ jobs:
- name: Install build dependencies
run: bash ${OPENVINO_REPO}/install_build_dependencies.sh
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
with:
version: "v0.5.4"
#
# Build
#
@ -281,7 +288,7 @@ jobs:
run: cmake --build ${BUILD_DIR} --parallel 8 --target benchmark_app
- name: Show ccache stats
run: ccache --show-stats
run: ${SCCACHE_PATH} --show-stats
- name: Run with CC-ed runtime
run: ${OPENVINO_REPO}/bin/intel64/Release/benchmark_app -niter 1 -nireq 1 -m ${MODELS_PATH}/models/test_model/test_model_fp32.xml -d CPU

21
.github/workflows/webassembly.yml vendored
View File

@ -36,15 +36,14 @@ jobs:
image: emscripten/emsdk
volumes:
- /mount/caches:/mount/caches
options: -e SCCACHE_AZURE_BLOB_CONTAINER -e SCCACHE_AZURE_CONNECTION_STRING
env:
CMAKE_BUILD_TYPE: 'Release'
CMAKE_CXX_COMPILER_LAUNCHER: ccache
CMAKE_C_COMPILER_LAUNCHER: ccache
CMAKE_CXX_COMPILER_LAUNCHER: sccache
CMAKE_C_COMPILER_LAUNCHER: sccache
OPENVINO_REPO: /__w/openvino/openvino/openvino
OPENVINO_BUILD_DIR: /__w/openvino/openvino/openvino_build
CCACHE_DIR: /mount/caches/ccache/webassembly_Release
CCACHE_TEMPDIR: /__w/openvino/openvino/ccache_temp
CCACHE_MAXSIZE: 50G
SCCACHE_AZURE_KEY_PREFIX: webassembly_Release
steps:
- name: Install git
run: apt-get update && apt-get install --assume-yes --no-install-recommends git ca-certificates
@ -55,8 +54,10 @@ jobs:
path: 'openvino'
submodules: 'true'
- name: Install ccache
run: apt-get install --assume-yes --no-install-recommends ccache
- name: Install sccache
uses: mozilla-actions/sccache-action@v0.0.3
with:
version: "v0.5.4"
- name: emcmake cmake - configure
run: |
@ -64,8 +65,8 @@ jobs:
-DCMAKE_CXX_FLAGS="-Wno-deprecated-declarations" \
-DCMAKE_C_FLAGS="-Wno-deprecated-declarations" \
-DCMAKE_COMPILE_WARNING_AS_ERROR=ON \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
-DCMAKE_CXX_COMPILER_LAUNCHER=${{ env.CMAKE_CXX_COMPILER_LAUNCHER }} \
-DCMAKE_C_COMPILER_LAUNCHER=${{ env.CMAKE_C_COMPILER_LAUNCHER }} \
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
-S ${OPENVINO_REPO} \
-B ${OPENVINO_BUILD_DIR}
@ -74,4 +75,4 @@ jobs:
run: emmake make -j$(nproc) hello_query_device -C ${OPENVINO_BUILD_DIR}
- name: Show ccache stats
run: ccache --show-stats
run: ${SCCACHE_PATH} --show-stats