IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

CMake based build for pyngraph module (#3080)

Browse Source 
* [MO] Add CMake install for Model Optimizer

* [MO] Update test for version.py

* [MO] Add CMake install for Model Optimizer

* [MO] Update test for version.py

* [MO] Add CMake install for Model Optimizer

* [MO] Update test for version.py

* [nGraph] Python API should be compiled and installed via CMake (41857)

* Refactored wheel setup script to build module using CMake

* Update build instructions

* Added USE_SOURCE_PERMISSIONS to cmake install

* Adjust CMake compiler flags conditions

* fix CPack issue for CI build pipeline

* case insensitive option check

* build only python API if ngraph_DIR provided

* fix lib extension for macOS

* -fixed style (flake8)

 -added paralllel build option & description

* fix flake8 B006 check

* add ngraph_DIR & remove unsed env. variables.

* Reworked build & test instructions to make it more straightforward

* remove unused CMake arguments for setup.py

* make source dir condition more general

* Update BUILDING.md

* Update BUILDING.md

* Update BUILDING.md

* beautified instructions wording

* fix wheel build issue after sourcing setupvars

* Extend user options to build, install and develop commands

Co-authored-by: Andrey Zaytsev <andrey.zaytsev@intel.com>
This commit is contained in:
Sergey Lyubimtsev 2020-12-24 16:57:58 +03:00 committed by GitHub
parent 4a62491927
commit 2e6ea1e290
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 300 additions and 495 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

131
ngraph/python/BUILDING.md
View File

@ -1,45 +1,44 @@
# Building the Python API for nGraph
# Building the nGraph Python* API
You can build the nGraph Python API from sources by following instructions in this document. A Python wheel is a
portable package which will allow you to install nGraph in your Python distribution, or dedicated virtual environment.
This document provides the instructions for building the nGraph Python API from source on Linux, macOS and Windows 10 platforms.
## Build nGraph Python Wheels on Linux or MacOS
For each platform, you can build and install the API as a part of OpenVINO™ Toolkit or as a Python wheel.
A Python wheel is a portable package that allows you to install nGraph in your Python distribution, or dedicated virtual environment.
## Linux* and macOS*
### Prerequisites
In order to build the nGraph Python wheel, you will need to install a few packages.
To build the nGraph Python API, you need to install a few additional packages.
On Ubuntu 20.04 LTS you can use the following instructions to install the required packages, including Python and Cython.
On Ubuntu* 20.04 LTS you can use the following instructions to install the required packages, including Python and Cython.
apt install git wget build-essential cmake
apt install python3 python3-dev python3-pip python3-virtualenv python-is-python3
You can see a full working example on an Ubuntu environment used in our continuous environment in this
[Dockerfile](https://github.com/openvinotoolkit/openvino/blob/master/.ci/openvino-onnx/Dockerfile).
On MacOS you can use [Homebrew](https://brew.sh) to install required packages:
On macOS, you can use [Homebrew](https://brew.sh) to install required packages:
brew install cmake
brew install automake
brew install libtool
brew install python3
Install Cython in the Python installation, or virtualenv which you are planning to use:
Install Cython in the Python installation, or virtualenv that you are planning to use:
pip3 install cython
### Configure, build and install OpenVINO
### Configure and Build as a part of OpenVINO™ Toolkit on Linux and macOS
The following section will illustrate how to download, build and install OpenVINO in a workspace directory specified
by the `${MY_OPENVINO_BASEDIR}` variable. Let's start by setting this variable to a directory of your choice:
The following section illustrates how to build and install OpenVINO™ in a workspace directory using CMake.
The workspace directory is specified by the `${OPENVINO_BASEDIR}` variable. Set this variable to a directory of your choice:
export MY_OPENVINO_BASEDIR=/path/to/my/workspace
export OPENVINO_BASEDIR=/path/to/my/workspace
Now we can clone OpenVINO, configure it using `cmake` and build using `make`. Please note that we're disabling
the building of a few modules by setting the `ENABLE_*` flag to `OFF`. In order to build the OpenVINO Python APIs
Now you can clone the OpenVINO™ repository, configure it using `cmake` and build using `make`. Please note that we're disabling
the building of a few modules by setting the `ENABLE_*` flag to `OFF`. In order to build the OpenVINO™ Python APIs
set the mentioned flags to `ON`. Note the `CMAKE_INSTALL_PREFIX`, which defaults to `/usr/local/` if not set.
cd "${MY_OPENVINO_BASEDIR}"
cd "${OPENVINO_BASEDIR}"
git clone --recursive https://github.com/openvinotoolkit/openvino.git
mkdir openvino/build
cd openvino/build
@ -51,64 +50,64 @@ set the mentioned flags to `ON`. Note the `CMAKE_INSTALL_PREFIX`, which defaults
-DENABLE_PYTHON=ON \
-DNGRAPH_PYTHON_BUILD_ENABLE=ON \
-DNGRAPH_ONNX_IMPORT_ENABLE=ON \
-DCMAKE_INSTALL_PREFIX="${MY_OPENVINO_BASEDIR}/openvino_dist"
-DCMAKE_INSTALL_PREFIX="${OPENVINO_BASEDIR}/openvino_dist"
make -j 4
make install
If you would like to use a specific version of Python, or use a virtual environment you can set the `PYTHON_EXECUTABLE`
variable. Examples:
The Python module is installed in the `${OPENVINO_BASEDIR}/openvino_dist/python/python<version>/` folder.
Set up the OpenVINO™ environment in order to add the module path to `PYTHONPATH`:
source ${OPENVINO_BASEDIR}/openvino_dist/bin/setupvars.sh
If you would like to use a specific version of Python, or use a virtual environment, you can set the `PYTHON_EXECUTABLE`
variable. For example:
```
-DPYTHON_EXECUTABLE=/path/to/venv/bin/python
-DPYTHON_EXECUTABLE=$(which python3.8)
```
```
### Build nGraph Python wheel
### Build an nGraph Python Wheel on Linux and macOS
When OpenVINO is built and installed, we can build the Python wheel by issuing the following command:
You can build the Python wheel running the following command:
make python_wheel
cd "${OPENVINO_BASEDIR}/openvino/ngraph/python"
python3 setup.py bdist_wheel
Once completed, the wheel package should be located under the following path:
$ ls "${MY_OPENVINO_BASEDIR}/openvino/ngraph/python/dist/"
$ ls "${OPENVINO_BASEDIR}/openvino/ngraph/python/dist/"
ngraph_core-0.0.0-cp38-cp38-linux_x86_64.whl
You can now install the wheel in your Python environment:
cd "${MY_OPENVINO_BASEDIR}/openvino/ngraph/python/dist/"
cd "${OPENVINO_BASEDIR}/openvino/ngraph/python/dist/"
pip3 install ngraph_core-0.0.0-cp38-cp38-linux_x86_64.whl
#### What does `make python_wheel` do?
The `python_wheel` target automates a few steps, required to build the wheel package. You can recreate the process
manually by issuing the following commands:
cd "${MY_OPENVINO_BASEDIR}/openvino/ngraph/python"
git clone --branch v2.5.0 https://github.com/pybind/pybind11.git
export NGRAPH_CPP_BUILD_PATH="${MY_OPENVINO_BASEDIR}/openvino_dist"
python3 setup.py bdist_wheel
## Build nGraph Python Wheels on Windows
## Windows* 10
### Prerequisites
In order to build OpenVINO and the nGraph Python wheel on Windows, you will need to install Visual Studio and Python.
In order to build OpenVINO™ and the nGraph Python wheel on Windows, you need to install Microsoft Visual Studio* and Python.
Once Python is installed, you will also need to install Cython using `pip install cython`.
Once Python is installed, you also need to install Cython using `pip install cython`.
### Configure, build and install OpenVINO
### Configure and Build as a Part of OpenVINO™ Toolkit on Windows
Configure the build with a `cmake` invocation similar to the following. Note that you'll need to set the `-G` and
`-DCMAKE_CXX_COMPILER` to match the version and location of your Visual Studio installation.
The following section illustrates how to build and install OpenVINO™ in a workspace directory using CMake.
The workspace directory is specified by the `OPENVINO_BASEDIR` variable. Set this variable to a directory of your choice:
set OPENVINO_BASEDIR=/path/to/my/workspace
Configure the build with a `cmake` invocation similar to the following. Note that need to set `-G` and
`-DCMAKE_CXX_COMPILER` to match the version and location of your Microsoft Visual Studio installation.
```
cmake .. ^
-G"Visual Studio 16 2019" ^
-DCMAKE_BUILD_TYPE=Release ^
-DCMAKE_INSTALL_PREFIX="C:\temporary_install_dir" ^
-DCMAKE_INSTALL_PREFIX="%OPENVINO_BASEDIR%/openvino_dist" ^
-DENABLE_CLDNN=OFF ^
-DENABLE_OPENCV=OFF ^
-DENABLE_VPU=OFF ^
@ -120,14 +119,9 @@ cmake .. ^
```
There are a couple of things to notice here. One is that the full path to the x64 version of
MSVC compiler has to be specified. This is because DNNL requires a 64-bit version and cmake may
MSVC compiler has to be specified. This is because DNNL requires a 64-bit version and `cmake` may
fail to detect it correctly.
The other equally important thing to note is that the temporary directory where the build is to be installed can be specified.
If the installation directory is not specified, the default location is `C:\Program Files\OpenVINO`.
This examples uses `C:\temporary_install_dir` however, a subdirectory of `openvino\build` works as well.
The final Python wheel will contain the contents of this temporary directory so it's important to set it.
If you want to specify an exact Python version, use the following options:
```
-DPYTHON_EXECUTABLE="C:\Program Files\Python37\python.exe" ^
@ -135,26 +129,32 @@ If you want to specify an exact Python version, use the following options:
-DPYTHON_INCLUDE_DIR="C:\Program Files\Python37\include" ^
```
In order to build and install OpenVINO, build the `install` target:
In order to build and install OpenVINO™, build the `install` target:
cmake --build . --target install --config Release -j 8
In this step OpenVINO will be built and installed to the directory specified above. You can
In this step, OpenVINO™ is built and installed to the directory specified above. You can
adjust the number of threads used in the building process to your machine's capabilities.
Set up the OpenVINO™ environment in order to add a module path to `PYTHONPATH`:
%OPENVINO_BASEDIR%\openvino_dist\bin\setupvars.bat
### Build an nGraph Python Wheel on Windows
Build the Python wheel package:
cmake --build . --target python_wheel --config Release -j 8
cd "%OPENVINO_BASEDIR%/openvino/ngraph/python"
python setup.py bdist_wheel
The final wheel should be located in `ngraph\python\dist` directory.
The final wheel should be located in the `ngraph\python\dist` directory.
dir openvino\ngraph\python\dist\
10/09/2020 04:06 PM 4,010,943 ngraph_core-0.0.0-cp38-cp38-win_amd64.whl
## Run Tests
## Run tests
### Using a virtualenv (optional)
### Use a virtualenv (Optional)
You may wish to use a virutualenv for your installation.
@ -162,25 +162,20 @@ You may wish to use a virutualenv for your installation.
$ source venv/bin/activate
(venv) $
### Install the nGraph wheel and other requirements
### Install the nGraph Wheel and Other Requirements
(venv) $ cd "${MY_OPENVINO_BASEDIR}/openvino/ngraph/python"
(venv) $ cd "${OPENVINO_BASEDIR}/openvino/ngraph/python"
(venv) $ pip3 install -r requirements.txt
(venv) $ pip3 install -r requirements_test.txt
(venv) $ pip3 install dist/ngraph_core-0.0.0-cp38-cp38-linux_x86_64.whl
### Run tests
### Run Tests
You should now be able to run tests.
You may need to run the `setupvars` script from OpenVINO to set paths to OpenVINO components.
You may need to run the `setupvars` script from the OpenVINO™ Toolkit to set paths to OpenVINO™ components.
source ${MY_OPENVINO_BASEDIR}/openvino/scripts/setupvars/setupvars.sh
The minimum requirement is to set the `PYTHONPATH` to include the Inference Engine Python API:
export PYTHONPATH="${MY_OPENVINO_BASEDIR}/openvino/bin/intel64/Release/lib/python_api/python3.8/":${PYTHONPATH}
pytest tests/
source ${OPENVINO_BASEDIR}/openvino_dist/bin/setupvars.sh
Now you can run tests using `pytest`:

79
ngraph/python/CMakeLists.txt
View File

@ -18,28 +18,67 @@ cmake_minimum_required (VERSION 3.13)
project (pyngraph)
include(ExternalProject)
if(NOT DEFINED OpenVINO_MAIN_SOURCE_DIR)
find_package(InferenceEngineDeveloperPackage)
find_package(ngraph REQUIRED)
endif()
ExternalProject_Add(
pybind11
GIT_REPOSITORY "https://github.com/pybind/pybind11.git"
GIT_TAG "v2.5.0"
SOURCE_DIR "${CMAKE_CURRENT_BINARY_DIR}/pybind11"
CONFIGURE_COMMAND ""
BUILD_COMMAND ""
INSTALL_COMMAND ""
if(ngraph_FOUND)
message("ngraph version = {${ngraph_VERSION}}")
endif()
include(FetchContent)
FetchContent_Declare(
pybind11
GIT_REPOSITORY "https://github.com/pybind/pybind11.git"
GIT_TAG "v2.5.0"
)
set(BUILD_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/build_wheel.py.in")
set(BUILD_PY "${CMAKE_CURRENT_BINARY_DIR}/build_wheel.py")
configure_file(${BUILD_PY_IN} ${BUILD_PY} @ONLY)
FetchContent_GetProperties(pybind11)
if(NOT pybind11_POPULATED)
FetchContent_Populate(pybind11)
add_subdirectory(${pybind11_SOURCE_DIR} ${pybind11_BINARY_DIR})
endif()
add_custom_command(
DEPENDS pybind11
OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/dist/
POST_BUILD
WORKING_DIR ${CMAKE_CURRENT_BINARY_DIR}
COMMAND python ${BUILD_PY}
)
find_package(Python COMPONENTS Interpreter Development REQUIRED)
add_custom_target(python_wheel DEPENDS ngraph ${CMAKE_CURRENT_BINARY_DIR}/dist/)
if(PYTHON_FOUND)
set(PYTHON_VERSION python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR})
message("Python version={${Python_VERSION}}")
else()
message(FATAL_ERROR "Python was not found!")
endif()
if(CMAKE_COMPILER_IS_GNUCXX)
add_compile_options(-fPIC)
endif()
if (APPLE)
add_link_options(-stdlib=libc++)
endif()
if (CMAKE_CXX_COMPILER_ID STREQUAL "MSVC")
#disable warning: This operator was deprecated and will be removed with v0 operation.
add_compile_options(/wd4996)
endif()
file(GLOB_RECURSE SOURCES src/pyngraph/*.cpp)
pybind11_add_module(_${PROJECT_NAME} MODULE ${SOURCES})
target_include_directories(_${PROJECT_NAME} PRIVATE src)
target_link_libraries(_${PROJECT_NAME} PRIVATE ngraph::ngraph ngraph::onnx_importer)
if(OpenVINO_MAIN_SOURCE_DIR OR InferenceEngineDeveloperPackage_FOUND)
ie_cpack_add_component(pyngraph_${PYTHON_VERSION})
install(TARGETS _${PROJECT_NAME}
DESTINATION python/${PYTHON_VERSION}
COMPONENT pyngraph_${PYTHON_VERSION})
install(DIRECTORY ${CMAKE_CURRENT_LIST_DIR}/src/ngraph
DESTINATION python/${PYTHON_VERSION}
COMPONENT pyngraph_${PYTHON_VERSION}
USE_SOURCE_PERMISSIONS)
ie_cpack(pyngraph_${PYTHON_VERSION})
endif()

81
ngraph/python/build_wheel.py.in
View File

@ -1,81 +0,0 @@
# ******************************************************************************
# Copyright 2017-2020 Intel Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ******************************************************************************
import os
import subprocess
import sys
import venv
print("Building ngraph wheel for Python {}".format(sys.version_info.major))
PYBIND_HEADERS_PATH = "@CMAKE_CURRENT_BINARY_DIR@/pybind11"
NGRAPH_CPP_BUILD_PATH = "@CMAKE_INSTALL_PREFIX@/@NGRAPH_COMPONENT_PREFIX@"
NGRAPH_ONNX_IMPORT_ENABLE = "@NGRAPH_ONNX_IMPORT_ENABLE@"
NGRAPH_VERSION = "@NGRAPH_WHEEL_VERSION@"
PYTHON_API_SOURCE_DIR = "@CMAKE_CURRENT_SOURCE_DIR@"
BUILD_DIR = "@CMAKE_CURRENT_BINARY_DIR@"
BUILD_DEPS = ["setuptools", "wheel", "pip"]
try:
venv_dir = os.path.join(os.path.curdir, "whl_build_venv")
print("Creating a virtualenv to build the wheel in: ", os.path.abspath(venv_dir))
venv.create(venv_dir, with_pip=True)
venv_python = (
os.path.abspath(os.path.join(venv_dir, "Scripts", "python"))
if os.name == "nt"
else os.path.abspath(os.path.join(venv_dir, "bin", "python"))
)
print("Installing build dependencies...")
pip_install_cmd = [venv_python, "-m", "pip", "install", "-U"]
pip_install_cmd.extend(BUILD_DEPS)
subprocess.check_call(pip_install_cmd)
build_env_variables = {
"PYBIND_HEADERS_PATH": PYBIND_HEADERS_PATH,
"NGRAPH_CPP_BUILD_PATH": NGRAPH_CPP_BUILD_PATH,
"NGRAPH_ONNX_IMPORT_ENABLE": NGRAPH_ONNX_IMPORT_ENABLE,
"NGRAPH_VERSION": NGRAPH_VERSION,
}
env = os.environ
env.update(build_env_variables)
print("Running setup.py bdist_wheel")
build_log = subprocess.Popen(
[venv_python, os.path.join(PYTHON_API_SOURCE_DIR, "setup.py"), "bdist_wheel"],
stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env, universal_newlines=True,
)
for line in build_log.stdout:
sys.stdout.write(line)
print("Running setup.py sdist")
subprocess.check_call([venv_python, os.path.join(PYTHON_API_SOURCE_DIR, "setup.py"), "sdist"])
output_dir = os.path.join(PYTHON_API_SOURCE_DIR, "dist")
print("\n>>> NOTE: nGraph Python packages created in ", output_dir)
print("\n".join(os.listdir(output_dir)))
except subprocess.CalledProcessError as err:
print("Could not complete the wheel building process")
print("Command that failed: ", err.cmd)
if err.stdout is not None:
print("Command std output: ", err.stdout.decode("utf-8"))
if err.stderr is not None:
print("Command err output: ", err.stderr.decode("utf-8"))
sys.exit(1)

498
ngraph/python/setup.py
View File

@ -14,214 +14,29 @@
# limitations under the License.
# ******************************************************************************
import distutils.ccompiler
import os
import re
import pathlib
import shutil
import glob
import sysconfig
import sys
import multiprocessing
import setuptools
from setuptools import Extension, setup
from setuptools.command.build_ext import build_ext
from setuptools.command.install_lib import install_lib
from setuptools.command.install import install as _install
from setuptools.command.develop import develop as _develop
from distutils.command.build import build as _build
__version__ = os.environ.get("NGRAPH_VERSION", "0.0.0.dev0")
PYNGRAPH_ROOT_DIR = os.path.abspath(os.path.dirname(__file__))
PYNGRAPH_SRC_DIR = os.path.join(PYNGRAPH_ROOT_DIR, "src")
NGRAPH_DEFAULT_INSTALL_DIR = os.environ.get("HOME")
NGRAPH_PYTHON_DEBUG = os.environ.get("NGRAPH_PYTHON_DEBUG")
NGRAPH_ROOT_DIR = os.path.normpath(os.path.join(PYNGRAPH_ROOT_DIR, ".."))
OPENVINO_ROOT_DIR = os.path.normpath(os.path.join(PYNGRAPH_ROOT_DIR, "../.."))
# Change current working dircectory to ngraph/python
os.chdir(PYNGRAPH_ROOT_DIR)
debug_optimization_flags = [
"O1", "O2", "O3", "O4", "Ofast", "Os", "Oz", "Og", "O", "DNDEBUG"
]
def find_ngraph_dist_dir():
"""Return location of compiled ngraph library home."""
if os.environ.get("NGRAPH_CPP_BUILD_PATH"):
ngraph_dist_dir = os.environ.get("NGRAPH_CPP_BUILD_PATH")
else:
ngraph_dist_dir = os.path.join(NGRAPH_DEFAULT_INSTALL_DIR, "ngraph_dist")
found = os.path.exists(os.path.join(ngraph_dist_dir, "include/ngraph"))
if not found:
print(
"Cannot find nGraph library in {} make sure that "
"NGRAPH_CPP_BUILD_PATH is set correctly".format(ngraph_dist_dir)
)
sys.exit(1)
else:
print("nGraph library found in {}".format(ngraph_dist_dir))
return ngraph_dist_dir
def find_pybind_headers_dir():
"""Return location of pybind11 headers."""
if os.environ.get("PYBIND_HEADERS_PATH"):
pybind_headers_dir = os.environ.get("PYBIND_HEADERS_PATH")
else:
pybind_headers_dir = os.path.join(PYNGRAPH_ROOT_DIR, "pybind11")
found = os.path.exists(os.path.join(pybind_headers_dir, "include/pybind11"))
if not found:
print(
"Cannot find pybind11 library in {} make sure that "
"PYBIND_HEADERS_PATH is set correctly".format(pybind_headers_dir)
)
sys.exit(1)
else:
print("pybind11 library found in {}".format(pybind_headers_dir))
return pybind_headers_dir
NGRAPH_CPP_DIST_DIR = find_ngraph_dist_dir()
PYBIND11_INCLUDE_DIR = find_pybind_headers_dir() + "/include"
NGRAPH_CPP_INCLUDE_DIR = NGRAPH_CPP_DIST_DIR + "/include"
if os.path.exists(os.path.join(NGRAPH_CPP_DIST_DIR, "lib")):
NGRAPH_CPP_LIBRARY_DIR = os.path.join(NGRAPH_CPP_DIST_DIR, "lib")
elif os.path.exists(os.path.join(NGRAPH_CPP_DIST_DIR, "lib64")):
NGRAPH_CPP_LIBRARY_DIR = os.path.join(NGRAPH_CPP_DIST_DIR, "lib64")
else:
print(
"Cannot find library directory in {}, make sure that nGraph is installed "
"correctly".format(NGRAPH_CPP_DIST_DIR)
)
sys.exit(1)
if sys.platform == "win32":
NGRAPH_CPP_DIST_DIR = os.path.normpath(NGRAPH_CPP_DIST_DIR)
PYBIND11_INCLUDE_DIR = os.path.normpath(PYBIND11_INCLUDE_DIR)
NGRAPH_CPP_INCLUDE_DIR = os.path.normpath(NGRAPH_CPP_INCLUDE_DIR)
NGRAPH_CPP_LIBRARY_DIR = os.path.normpath(NGRAPH_CPP_LIBRARY_DIR)
NGRAPH_CPP_LIBRARY_NAME = "ngraph"
"""For some platforms OpenVINO adds 'd' suffix to library names in debug configuration"""
if len([fn for fn in os.listdir(NGRAPH_CPP_LIBRARY_DIR) if re.search("ngraphd", fn)]):
NGRAPH_CPP_LIBRARY_NAME = "ngraphd"
ONNX_IMPORTER_CPP_LIBRARY_NAME = "onnx_importer"
if len([fn for fn in os.listdir(NGRAPH_CPP_LIBRARY_DIR) if re.search("onnx_importerd", fn)]):
ONNX_IMPORTER_CPP_LIBRARY_NAME = "onnx_importerd"
def _remove_compiler_flags(obj):
"""Make pybind11 more verbose in debug builds."""
for flag in debug_optimization_flags:
try:
if sys.platform == "win32":
obj.compiler.compile_options.remove("/{}".format(flag))
else:
obj.compiler.compiler_so.remove("-{}".format(flag))
obj.compiler.compiler.remove("-{}".format(flag))
except (AttributeError, ValueError):
pass
def parallelCCompile(
self,
sources,
output_dir=None,
macros=None,
include_dirs=None,
debug=0,
extra_preargs=None,
extra_postargs=None,
depends=None,
):
"""Build sources in parallel.
Reference link:
http://stackoverflow.com/questions/11013851/speeding-up-build-process-with-distutils
Monkey-patch for parallel compilation.
"""
# those lines are copied from distutils.ccompiler.CCompiler directly
macros, objects, extra_postargs, pp_opts, build = self._setup_compile(
output_dir, macros, include_dirs, sources, depends, extra_postargs
)
cc_args = self._get_cc_args(pp_opts, debug, extra_preargs)
# parallel code
import multiprocessing.pool
def _single_compile(obj):
try:
src, ext = build[obj]
except KeyError:
return
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
# convert to list, imap is evaluated on-demand
pool = multiprocessing.pool.ThreadPool()
list(pool.imap(_single_compile, objects))
return objects
distutils.ccompiler.CCompiler.compile = parallelCCompile
def has_flag(compiler, flagname):
"""Check whether a flag is supported by the specified compiler.
As of Python 3.6, CCompiler has a `has_flag` method.
cf http://bugs.python.org/issue26689
"""
import tempfile
with tempfile.NamedTemporaryFile("w", suffix=".cpp") as f:
f.write("int main (int argc, char **argv) { return 0; }")
try:
compiler.compile([f.name], extra_postargs=[flagname])
except setuptools.distutils.errors.CompileError:
return False
return True
def cpp_flag(compiler):
"""Check and return the -std=c++11 compiler flag."""
if sys.platform == "win32":
return "" # C++11 is on by default in MSVC
elif has_flag(compiler, "-std=c++11"):
return "-std=c++11"
else:
raise RuntimeError("Unsupported compiler -- C++11 support is needed!")
sources = [
"pyngraph/axis_set.cpp",
"pyngraph/axis_vector.cpp",
"pyngraph/coordinate.cpp",
"pyngraph/coordinate_diff.cpp",
"pyngraph/dict_attribute_visitor.cpp",
"pyngraph/dimension.cpp",
"pyngraph/function.cpp",
"pyngraph/node.cpp",
"pyngraph/node_input.cpp",
"pyngraph/node_output.cpp",
"pyngraph/node_factory.cpp",
"pyngraph/ops/constant.cpp",
"pyngraph/ops/parameter.cpp",
"pyngraph/ops/result.cpp",
"pyngraph/ops/util/arithmetic_reduction.cpp",
"pyngraph/ops/util/binary_elementwise_arithmetic.cpp",
"pyngraph/ops/util/binary_elementwise_comparison.cpp",
"pyngraph/ops/util/binary_elementwise_logical.cpp",
"pyngraph/ops/util/index_reduction.cpp",
"pyngraph/ops/util/op_annotations.cpp",
"pyngraph/ops/util/regmodule_pyngraph_op_util.cpp",
"pyngraph/ops/util/unary_elementwise_arithmetic.cpp",
"pyngraph/passes/manager.cpp",
"pyngraph/passes/regmodule_pyngraph_passes.cpp",
"pyngraph/partial_shape.cpp",
"pyngraph/pyngraph.cpp",
"pyngraph/shape.cpp",
"pyngraph/strides.cpp",
"pyngraph/tensor_iterator_builder.cpp",
"pyngraph/types/element_type.cpp",
"pyngraph/types/regmodule_pyngraph_types.cpp",
"pyngraph/util.cpp",
"pyngraph/variant.cpp",
"pyngraph/rt_map.cpp",
]
NGRAPH_LIBS = ["ngraph", "onnx_importer"]
packages = [
"ngraph",
@ -237,131 +52,172 @@ packages = [
"ngraph.impl.passes",
]
sources = [PYNGRAPH_SRC_DIR + "/" + source for source in sources]
include_dirs = [PYNGRAPH_SRC_DIR, NGRAPH_CPP_INCLUDE_DIR, PYBIND11_INCLUDE_DIR]
library_dirs = [NGRAPH_CPP_LIBRARY_DIR]
libraries = [NGRAPH_CPP_LIBRARY_NAME, ONNX_IMPORTER_CPP_LIBRARY_NAME]
extra_compile_args = []
extra_link_args = []
data_files = [
(
"lib",
[
os.path.join(NGRAPH_CPP_LIBRARY_DIR, library)
for library in os.listdir(NGRAPH_CPP_LIBRARY_DIR)
if os.path.isfile(os.path.join(NGRAPH_CPP_LIBRARY_DIR, library))
],
),
]
ext_modules = [
Extension(
"_pyngraph",
sources=sources,
include_dirs=include_dirs,
define_macros=[("VERSION_INFO", __version__)],
library_dirs=library_dirs,
libraries=libraries,
extra_compile_args=extra_compile_args,
extra_link_args=extra_link_args,
language="c++",
),
]
def add_platform_specific_link_args(link_args):
"""Add linker flags specific for the OS detected during the build."""
if sys.platform.startswith("linux"):
link_args += ["-Wl,-rpath,$ORIGIN/../.."]
link_args += ["-z", "noexecstack"]
link_args += ["-z", "relro"]
link_args += ["-z", "now"]
elif sys.platform == "darwin":
link_args += ["-Wl,-rpath,@loader_path/../.."]
link_args += ["-stdlib=libc++"]
elif sys.platform == "win32":
link_args += ["/LTCG"]
class BuildExt(build_ext):
"""A custom build extension for adding compiler-specific options."""
def _add_extra_compile_arg(self, flag, compile_args):
"""Return True if successfully added given flag to compiler args."""
if has_flag(self.compiler, flag):
compile_args += [flag]
return True
return False
def _add_debug_or_release_flags(self):
"""Return compiler flags for Release and Debug build types."""
if NGRAPH_PYTHON_DEBUG in ["TRUE", "ON", True]:
if sys.platform == "win32":
return ["/Od", "/Zi", "/RTC1"]
else:
return ["-O0", "-g"]
else:
if sys.platform == "win32":
return ["/O2"]
else:
return ["-O2", "-D_FORTIFY_SOURCE=2"]
def _add_win_compiler_flags(self, ext):
self._add_extra_compile_arg("/GL", ext.extra_compile_args) # Whole Program Optimization
self._add_extra_compile_arg("/analyze", ext.extra_compile_args)
def _add_unix_compiler_flags(self, ext):
if not self._add_extra_compile_arg("-fstack-protector-strong", ext.extra_compile_args):
self._add_extra_compile_arg("-fstack-protector", ext.extra_compile_args)
self._add_extra_compile_arg("-fvisibility=hidden", ext.extra_compile_args)
self._add_extra_compile_arg("-flto", ext.extra_compile_args)
self._add_extra_compile_arg("-fPIC", ext.extra_compile_args)
ext.extra_compile_args += ["-Wformat", "-Wformat-security"]
def _customize_compiler_flags(self):
"""Modify standard compiler flags."""
try:
# -Wstrict-prototypes is not a valid option for c++
self.compiler.compiler_so.remove("-Wstrict-prototypes")
except (AttributeError, ValueError):
pass
def build_extensions(self):
"""Build extension providing extra compiler flags."""
self._customize_compiler_flags()
for ext in self.extensions:
ext.extra_compile_args += [cpp_flag(self.compiler)]
if sys.platform == "win32":
self._add_win_compiler_flags(ext)
else:
self._add_unix_compiler_flags(ext)
add_platform_specific_link_args(ext.extra_link_args)
ext.extra_compile_args += self._add_debug_or_release_flags()
if sys.platform == "darwin":
ext.extra_compile_args += ["-stdlib=libc++"]
if NGRAPH_PYTHON_DEBUG in ["TRUE", "ON", True]:
_remove_compiler_flags(self)
build_ext.build_extensions(self)
data_files = []
with open(os.path.join(PYNGRAPH_ROOT_DIR, "requirements.txt")) as req:
requirements = req.read().splitlines()
cmdclass = {}
for super_class in [_build, _install, _develop]:
class command(super_class):
"""Add user options for build, install and develop commands."""
cmake_build_types = ["Release", "Debug", "RelWithDebInfo", "MinSizeRel"]
user_options = super_class.user_options + [
("config=", None, "Build configuration [{}].".format("|".join(cmake_build_types))),
("jobs=", None, "Specifies the number of jobs to use with make."),
("cmake-args=", None, "Additional options to be passed to CMake.")
]
def initialize_options(self):
"""Set default values for all the options that this command supports."""
super().initialize_options()
self.config = None
self.jobs = None
self.cmake_args = None
cmdclass[super_class.__name__] = command
class CMakeExtension(Extension):
"""Build extension stub."""
def __init__(self, name, sources=None):
if sources is None:
sources = []
super().__init__(name=name, sources=sources)
class BuildCMakeExt(build_ext):
"""Builds module using cmake instead of the python setuptools implicit build."""
cmake_build_types = ["Release", "Debug", "RelWithDebInfo", "MinSizeRel"]
user_options = [
("config=", None, "Build configuration [{}].".format("|".join(cmake_build_types))),
("jobs=", None, "Specifies the number of jobs to use with make."),
("cmake-args=", None, "Additional options to be passed to CMake.")
]
def initialize_options(self):
"""Set default values for all the options that this command supports."""
super().initialize_options()
self.build_base = "build"
self.config = None
self.jobs = None
self.cmake_args = None
def finalize_options(self):
"""Set final values for all the options that this command supports."""
super().finalize_options()
for cmd in ["build", "install", "develop"]:
self.set_undefined_options(cmd, ("config", "config"),
("jobs", "jobs"),
("cmake_args", "cmake_args"))
if not self.config:
if self.debug:
self.config = "Debug"
else:
self.announce("Set default value for CMAKE_BUILD_TYPE = Release.", level=4)
self.config = "Release"
else:
build_types = [item.lower() for item in self.cmake_build_types]
try:
i = build_types.index(str(self.config).lower())
self.config = self.cmake_build_types[i]
self.debug = True if "Debug" == self.config else False
except ValueError:
self.announce("Unsupported CMAKE_BUILD_TYPE value: " + self.config, level=4)
self.announce("Supported values: {}".format(", ".join(self.cmake_build_types)), level=4)
sys.exit(1)
if self.jobs is None and os.getenv("MAX_JOBS") is not None:
self.jobs = os.getenv("MAX_JOBS")
self.jobs = multiprocessing.cpu_count() if self.jobs is None else int(self.jobs)
def run(self):
"""Run CMake build for modules."""
for extension in self.extensions:
if extension.name == "_pyngraph":
self.build_cmake(extension)
def build_cmake(self, extension: Extension):
"""Cmake configure and build steps."""
self.announce("Preparing the build environment", level=3)
plat_specifier = ".%s-%d.%d" % (self.plat_name, *sys.version_info[:2])
self.build_temp = os.path.join(self.build_base, "temp" + plat_specifier, self.config)
build_dir = pathlib.Path(self.build_temp)
extension_path = pathlib.Path(self.get_ext_fullpath(extension.name))
os.makedirs(build_dir, exist_ok=True)
os.makedirs(extension_path.parent.absolute(), exist_ok=True)
# If ngraph_DIR is not set try to build from OpenVINO root
root_dir = OPENVINO_ROOT_DIR
bin_dir = os.path.join(OPENVINO_ROOT_DIR, "bin")
if os.environ.get("ngraph_DIR") is not None:
root_dir = PYNGRAPH_ROOT_DIR
bin_dir = build_dir
self.announce("Configuring cmake project", level=3)
ext_args = self.cmake_args.split() if self.cmake_args else []
self.spawn(["cmake", "-H" + root_dir, "-B" + self.build_temp,
"-DCMAKE_BUILD_TYPE={}".format(self.config),
"-DNGRAPH_PYTHON_BUILD_ENABLE=ON",
"-DNGRAPH_ONNX_IMPORT_ENABLE=ON"] + ext_args)
self.announce("Building binaries", level=3)
self.spawn(["cmake", "--build", self.build_temp, "--target", extension.name,
"--config", self.config, "-j", str(self.jobs)])
self.announce("Moving built python module to " + str(extension_path), level=3)
pyds = list(glob.iglob("{0}/**/{1}*{2}".format(bin_dir,
extension.name,
sysconfig.get_config_var("EXT_SUFFIX")), recursive=True))
for name in pyds:
self.announce("copy " + os.path.join(name), level=3)
shutil.copy(name, extension_path)
class InstallCMakeLibs(install_lib):
"""Finds and installs NGraph libraries to a package location."""
def run(self):
"""Copy libraries from the bin directory and place them as appropriate."""
self.announce("Adding library files", level=3)
root_dir = os.path.join(OPENVINO_ROOT_DIR, "bin")
if os.environ.get("ngraph_DIR") is not None:
root_dir = pathlib.Path(os.environ["ngraph_DIR"]) / ".."
lib_ext = ""
if "linux" in sys.platform:
lib_ext = ".so"
elif sys.platform == "darwin":
lib_ext = ".dylib"
elif sys.platform == "win32":
lib_ext = ".dll"
libs = []
for ngraph_lib in NGRAPH_LIBS:
libs.extend(list(glob.iglob("{0}/**/*{1}*{2}".format(root_dir,
ngraph_lib, lib_ext), recursive=True)))
if not libs:
raise Exception("NGraph libs not found.")
self.announce("Adding library files" + str(libs), level=3)
self.distribution.data_files.extend([("lib", [os.path.normpath(lib) for lib in libs])])
self.distribution.run_command("install_data")
super().run()
cmdclass["build_ext"] = BuildCMakeExt
cmdclass["install_lib"] = InstallCMakeLibs
setup(
name="ngraph-core",
description="nGraph - Intel's graph compiler and runtime for Neural Networks",
@ -369,12 +225,12 @@ setup(
author="Intel Corporation",
url="https://github.com/openvinotoolkit/openvino",
license="License :: OSI Approved :: Apache Software License",
ext_modules=ext_modules,
ext_modules=[CMakeExtension(name="_pyngraph")],
package_dir={"": "src"},
packages=packages,
cmdclass={"build_ext": BuildExt},
data_files=data_files,
install_requires=requirements,
data_files=data_files,
zip_safe=False,
extras_require={},
cmdclass=cmdclass
)

6
ngraph/python/tox.ini
View File

@ -11,13 +11,9 @@ deps =
flake8-bugbear
pytest-xdist
setenv =
NGRAPH_CPP_BUILD_PATH = {env:NGRAPH_CPP_BUILD_PATH:{homedir}/ngraph_dist}
NGRAPH_ONNX_IMPORT_ENABLE = {env:NGRAPH_ONNX_IMPORT_ENABLE:"FALSE"}
LD_LIBRARY_PATH = {env:LD_LIBRARY_PATH:{homedir}/ngraph_dist/lib}
DYLD_LIBRARY_PATH = {env:DYLD_LIBRARY_PATH:{homedir}/ngraph_dist/lib}
PYBIND_HEADERS_PATH = {env:PYBIND_HEADERS_PATH:}
NGRAPH_BACKEND = {env:NGRAPH_BACKEND:"CPU"}
PYTHONPATH = {env:PYTHONPATH}
ngraph_DIR = {env:NGRAPH_CPP_BUILD_PATH}
passenv =
http_proxy
https_proxy