IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/master' into itikhono/ts/slice

Browse Source
This commit is contained in:
Ivan 2023-03-15 16:25:41 +04:00
commit ce84ce24ec
152 changed files with 4693 additions and 2720 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
.ci/azure/linux.yml
View File

@ -447,16 +447,10 @@ jobs:
- script: |
$(RUN_PREFIX) $(INSTALL_TEST_DIR)/InferenceEngineCAPITests --gtest_output=xml:$(INSTALL_TEST_DIR)/TEST-InferenceEngineCAPITests.xml
env:
DATA_PATH: $(MODELS_PATH)
MODELS_PATH: $(MODELS_PATH)
displayName: 'IE CAPITests'
- script: |
$(RUN_PREFIX) $(INSTALL_TEST_DIR)/ov_capi_test --gtest_output=xml:$(INSTALL_TEST_DIR)/TEST-ov_capi_test.xml
env:
DATA_PATH: $(MODELS_PATH)
MODELS_PATH: $(MODELS_PATH)
displayName: 'OV CAPITests'
- task: CMake@1

6
.ci/azure/windows.yml
View File

@ -315,16 +315,10 @@ jobs:
- script: |
call $(SETUPVARS) && $(INSTALL_TEST_DIR)\InferenceEngineCAPITests --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-InferenceEngineCAPITests.xml
env:
DATA_PATH: $(MODELS_PATH)
MODELS_PATH: $(MODELS_PATH)
displayName: 'IE CAPITests'
- script: |
call $(SETUPVARS) && $(INSTALL_TEST_DIR)\ov_capi_test --gtest_output=xml:$(INSTALL_TEST_DIR)\TEST-ov_capi_test.xml
env:
DATA_PATH: $(MODELS_PATH)
MODELS_PATH: $(MODELS_PATH)
displayName: 'OV CAPITests'
- task: PublishTestResults@2

91
docs/Documentation/openvino_ecosystem.md
View File

@ -12,74 +12,97 @@
openvino_inference_engine_tools_compile_tool_README
openvino_docs_tuning_utilities
@endsphinxdirective
OpenVINO™ is not just one tool. It is an expansive ecosystem of utilities, providing a comprehensive workflow for deep learning solution development. Learn more about each of them to reach the full potential of OpenVINO™ Toolkit.
### Neural Network Compression Framework (NNCF)
Neural Network Compression Framework (NNCF)
###########################################
A suite of advanced algorithms for Neural Network inference optimization with minimal accuracy drop. NNCF applies quantization, filter pruning, binarization and sparsity algorithms to PyTorch and TensorFlow models during training.
More resources:
* [Documentation](@ref tmo_introduction)
* [GitHub](https://github.com/openvinotoolkit/nncf)
* [PyPI](https://pypi.org/project/nncf/)
### OpenVINO™ Training Extensions
* :doc:`Documentation <tmo_introduction>`
* `GitHub <https://github.com/openvinotoolkit/nncf>`__
* `PyPI <https://pypi.org/project/nncf/>`__
OpenVINO™ Training Extensions
#############################
A convenient environment to train Deep Learning models and convert them using the OpenVINO™ toolkit for optimized inference.
More resources:
* [Overview](@ref ote_documentation)
* [GitHub](https://github.com/openvinotoolkit/training_extensions)
* [Documentation](https://openvinotoolkit.github.io/training_extensions/stable/guide/get_started/introduction.html)
* :doc:`Overview <ote_documentation>`
* `GitHub <https://github.com/openvinotoolkit/training_extensions>`__
* `Documentation <https://openvinotoolkit.github.io/training_extensions/stable/guide/get_started/introduction.html>`__
OpenVINO™ Security Add-on
#########################
### OpenVINO™ Security Add-on
A solution for Model Developers and Independent Software Vendors to use secure packaging and secure model execution.
More resources:
* [documentation](https://docs.openvino.ai/latest/ovsa_get_started.html)
* [GitHub](https://github.com/openvinotoolkit/security_addon)
* `Documentation <https://docs.openvino.ai/latest/ovsa_get_started.html>`__
* `GitHub <https://github.com/openvinotoolkit/security_addon>`__
### OpenVINO™ integration with TensorFlow (OVTF)
OpenVINO™ integration with TensorFlow (OVTF)
############################################
A solution empowering TensorFlow developers with OpenVINO's optimization capabilities. With just two lines of code in your application, you can offload inference to OpenVINO, while keeping the TensorFlow API.
More resources:
* [documentation](https://github.com/openvinotoolkit/openvino_tensorflow)
* [PyPI](https://pypi.org/project/openvino-tensorflow/)
* [GitHub](https://github.com/openvinotoolkit/openvino_tensorflow)
### DL Streamer
* `Documentation <https://github.com/openvinotoolkit/openvino_tensorflow>`__
* `PyPI <https://pypi.org/project/openvino-tensorflow/>`__
* `GitHub <https://github.com/openvinotoolkit/openvino_tensorflow>`__
DL Streamer
###########
A streaming media analytics framework, based on the GStreamer multimedia framework, for creating complex media analytics pipelines.
More resources:
* [documentation on GitHub](https://dlstreamer.github.io/index.html)
* [installation Guide on GitHub](https://github.com/openvinotoolkit/dlstreamer_gst/wiki/Install-Guide)
* `Documentation on GitHub <https://dlstreamer.github.io/index.html>`__
* `Installation Guide on GitHub <https://github.com/openvinotoolkit/dlstreamer_gst/wiki/Install-Guide>`__
### DL Workbench
A web-based tool for deploying deep learning models. Built on the core of OpenVINO and equipped with a graphics user interface, DL Workbench is a great way to explore the possibilities of the OpenVINO workflow, import, analyze, optimize, and build your pre-trained models. You can do all that by visiting [Intel® Developer Cloud](https://software.intel.com/content/www/us/en/develop/tools/devcloud.html) and launching DL Workbench online.
DL Workbench
############
A web-based tool for deploying deep learning models. Built on the core of OpenVINO and equipped with a graphics user interface, DL Workbench is a great way to explore the possibilities of the OpenVINO workflow, import, analyze, optimize, and build your pre-trained models. You can do all that by visiting `Intel® Developer Cloud <https://software.intel.com/content/www/us/en/develop/tools/devcloud.html>`__ and launching DL Workbench online.
More resources:
* [Documentation](https://docs.openvino.ai/2022.3/workbench_docs_Workbench_DG_Introduction.html)
* [Docker Hub](https://hub.docker.com/r/openvino/workbench)
* [PyPI](https://pypi.org/project/openvino-workbench/)
### Computer Vision Annotation Tool (CVAT)
* `Documentation <https://docs.openvino.ai/2022.3/workbench_docs_Workbench_DG_Introduction.html>`__
* `Docker Hub <https://hub.docker.com/r/openvino/workbench>`__
* `PyPI <https://pypi.org/project/openvino-workbench/>`__
Computer Vision Annotation Tool (CVAT)
######################################
An online, interactive video and image annotation tool for computer vision purposes.
More resources:
* [documentation on GitHub](https://opencv.github.io/cvat/docs/)
* [web application](https://cvat.org/)
* [Docker Hub](https://hub.docker.com/r/openvino/cvat_server)
* [GitHub](https://github.com/openvinotoolkit/cvat)
### Dataset Management Framework (Datumaro)
* `Documentation on GitHub <https://opencv.github.io/cvat/docs/>`__
* `Web application <https://www.cvat.ai/>`__
* `Docker Hub <https://hub.docker.com/r/openvino/cvat_server>`__
* `GitHub <https://github.com/openvinotoolkit/cvat>`__
Dataset Management Framework (Datumaro)
#######################################
A framework and CLI tool to build, transform, and analyze datasets.
More resources:
* [documentation on GitHub](https://openvinotoolkit.github.io/datumaro/docs/)
* [PyPI](https://pypi.org/project/datumaro/)
* [GitHub](https://github.com/openvinotoolkit/datumaro)
* `Documentation on GitHub <https://openvinotoolkit.github.io/datumaro/docs/>`__
* `PyPI <https://pypi.org/project/datumaro/>`__
* `GitHub <https://github.com/openvinotoolkit/datumaro>`__
@endsphinxdirective

8
docs/MO_DG/prepare_model/convert_model/Converting_Model.md
View File

@ -2,7 +2,13 @@
With Model Optimizer you can increase your model's efficiency by providing an additional shape definition, with these two parameters: `--input_shape` and `--static_shape`.
@anchor when_to_specify_input_shapes
@sphinxdirective
.. _when_to_specify_input_shapes:
@endsphinxdirective
## Specifying --input_shape Command-line Parameter
Model Optimizer supports conversion of models with dynamic input shapes that contain undefined dimensions.
However, if the shape of data is not going to change from one inference request to another,

275
docs/OV_Runtime_UG/ShapeInference.md
View File

@ -8,169 +8,212 @@
troubleshooting_reshape_errors
@endsphinxdirective
OpenVINO™ enables you to change model input shape during the application runtime.
It may be useful when you want to feed the model an input that has different size than the model input shape.
The following instructions are for cases where you need to change the model input shape repeatedly.
.. note::
If you need to do this only once, prepare a model with updated shapes via
:doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`.
For more information, refer to the :ref:`Specifying --input_shape Command-line Parameter <when_to_specify_input_shapes>` article.
OpenVINO™ enables you to change model input shape during the application runtime. It may be useful when you want to feed the model an input that has different size than the model input shape. The following instructions are for cases where you need to change the model input shape repeatedly.
The reshape method
++++++++++++++++++++
> **NOTE**: If you need to do this only once, prepare a model with updated shapes via [Model Optimizer](@ref openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide). For more information, refer to the [Specifying --input_shape Command-line Parameter](@ref when_to_specify_input_shapes) article.
The reshape method is used as ``ov::Model::reshape`` in C++ and
`Model.reshape <api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.reshape>`__
in Python. The method updates input shapes and propagates them down to the outputs
of the model through all intermediate layers. The code below is an example of how
to set a new batch size with the ``reshape`` method:
### The reshape method
.. tab-set::
The reshape method is used as `ov::Model::reshape` in C++ and [Model.reshape](api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.reshape) in Python. The method updates input shapes and propagates them down to the outputs of the model through all intermediate layers.
The code below is an example of how to set a new batch size with the `reshape` method:
.. tab-item:: C++
:sync: cpp
@sphinxtabset
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: picture_snippet
@sphinxtab{C++}
.. tab-item:: Python
:sync: py
@snippet snippets/ShapeInference.cpp picture_snippet
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: Python
:fragment: picture_snippet
@endsphinxtab
The diagram below presents the results of using the method, where the size of
model input is changed with an image input:
@sphinxtab{Python}
.. image:: _static/images/original_vs_reshaped_model.svg
@snippet docs/snippets/ShapeInference.py picture_snippet
When using the ``reshape`` method, you may take one of the approaches:
@endsphinxtab
.. _usage_of_reshape_method:
@endsphinxtabset
The diagram below presents the results of using the method, where the size of model input is changed with an image input:
1. You can pass a new shape to the method in order to change the input shape of
the model with a single input. See the example of adjusting spatial dimensions to the input image:
![shape_inference_explained](./img/original_vs_reshaped_model.svg)
.. tab-set::
When using the `reshape` method, you may take one of the approaches:
.. tab-item:: C++
:sync: cpp
@anchor usage_of_reshape_method
@sphinxdirective
#. You can pass a new shape to the method in order to change the input shape of the model with a single input. See the example of adjusting spatial dimensions to the input image:
.. tab:: C++
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: spatial_reshape
.. tab:: Python
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: simple_spatials_change
To do the opposite - to resize input image to match the input shapes of the model, use the :doc:`pre-processing API <openvino_docs_OV_UG_Preprocessing_Overview>`.
To do the opposite - to resize input image to match the input shapes of the model,
use the :doc:`pre-processing API <openvino_docs_OV_UG_Preprocessing_Overview>`.
#. You can express a reshape plan, specifying the input by the port, the index, and the tensor name:
2. You can express a reshape plan, specifying the input by the port, the index, and the tensor name:
.. tab:: Port
.. tab:: C++
map<ov::Output<ov::Node>, ov::PartialShape specifies input by passing actual input port:
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: [obj_to_shape]
.. tab:: Python
`openvino.runtime.Output` dictionary key specifies input by passing actual input object.
Dictionary values representing new shapes could be `PartialShape`:
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: [obj_to_shape]
.. tab:: Index
.. tab:: C++
map<size_t, ov::PartialShape> specifies input by its index:
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: [idx_to_shape]
.. tab:: Python
`int` dictionary key specifies input by its index.
Dictionary values representing new shapes could be `tuple`:
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: [idx_to_shape]
.. tab:: Tensor Name
.. tab:: C++
map<string, ov::PartialShape> specifies input by its name:
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: [name_to_shape]
.. tab:: Python
`str` dictionary key specifies input by its name.
Dictionary values representing new shapes could be `str`:
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: [name_to_shape]
.. tab-set::
@endsphinxdirective
.. tab-item:: Port
.. tab-set::
.. tab-item:: C++
:sync: cpp
``map<ov::Output<ov::Node>, ov::PartialShape`` specifies input by passing actual input port:
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: [obj_to_shape]
.. tab-item:: Python
:sync: py
``openvino.runtime.Output`` dictionary key specifies input by passing actual input object.
Dictionary values representing new shapes could be ``PartialShape``:
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: [obj_to_shape]
.. tab-item:: Index
.. tab-set::
.. tab-item:: C++
:sync: cpp
``map<size_t, ov::PartialShape>`` specifies input by its index:
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: [idx_to_shape]
.. tab-item:: Python
:sync: py
``int`` dictionary key specifies input by its index.
Dictionary values representing new shapes could be ``tuple``:
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: [idx_to_shape]
.. tab-item:: Tensor Name
.. tab-set::
.. tab-item:: C++
:sync: cpp
``map<string, ov::PartialShape>`` specifies input by its name:
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: [name_to_shape]
.. tab-item:: Python
:sync: py
``str`` dictionary key specifies input by its name.
Dictionary values representing new shapes could be ``str``:
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: [name_to_shape]
You can find the usage scenarios of the `reshape` method in [Hello Reshape SSD Samples](@ref openvino_inference_engine_samples_hello_reshape_ssd_README).
You can find the usage scenarios of the ``reshape`` method in
:doc:`Hello Reshape SSD Samples <openvino_inference_engine_samples_hello_reshape_ssd_README>`.
> **NOTE**: In some cases, models may not be ready to be reshaped. Therefore, a new input shape cannot be set neither with [Model Optimizer](@ref openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide) nor the `reshape` method.
.. note::
### The set_batch method
In some cases, models may not be ready to be reshaped. Therefore, a new input
shape cannot be set neither with :doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`
nor the ``reshape`` method.
The set_batch method
++++++++++++++++++++
The meaning of the model batch may vary depending on the model design.
To change the batch dimension of the model, [set the layout](@ref declare_model_s_layout) and call the `set_batch` method.
To change the batch dimension of the model, :ref:`set the layout <declare_model_s_layout>` and call the ``set_batch`` method.
@sphinxtabset
.. tab-set::
@sphinxtab{C++}
.. tab-item:: C++
:sync: cpp
@snippet snippets/ShapeInference.cpp set_batch
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: set_batch
@endsphinxtab
.. tab-item:: Python
:sync: py
@sphinxtab{Python}
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: Python
:fragment: set_batch
@snippet docs/snippets/ShapeInference.py set_batch
@endsphinxtab
The ``set_batch`` method is a high-level API of the reshape functionality, so all
information about the ``reshape`` method implications are applicable for ``set_batch``
too, including the troubleshooting section.
@endsphinxtabset
Once you set the input shape of the model, call the ``compile_model`` method to
get a ``CompiledModel`` object for inference with updated shapes.
The `set_batch` method is a high-level API of the reshape functionality, so all information about the `reshape` method implications are applicable for `set_batch` too, including the troubleshooting section.
There are other approaches to change model input shapes during the stage of
:ref:`IR generation <when_to_specify_input_shapes>` or :ref:`model representation <openvino_docs_OV_UG_Model_Representation>` in OpenVINO Runtime.
Once you set the input shape of the model, call the `compile_model` method to get a `CompiledModel` object for inference with updated shapes.
There are other approaches to change model input shapes during the stage of [IR generation](@ref when_to_specify_input_shapes) or [model representation](@ref openvino_docs_OV_UG_Model_Representation) in OpenVINO Runtime.
@sphinxdirective
.. important::
Shape-changing functionality could be used to turn dynamic model input into a static one and vice versa. Always set static shapes when the shape of data is NOT going to change from one inference to another. Setting static shapes can avoid memory and runtime overheads for dynamic shapes which may vary depending on hardware plugin and model used. For more information, refer to the :doc:`Dynamic Shapes <openvino_docs_OV_UG_DynamicShapes>`.
Shape-changing functionality could be used to turn dynamic model input into a
static one and vice versa. Always set static shapes when the shape of data is
NOT going to change from one inference to another. Setting static shapes can
avoid memory and runtime overheads for dynamic shapes which may vary depending
on hardware plugin and model used. For more information, refer to the
:doc:`Dynamic Shapes <openvino_docs_OV_UG_DynamicShapes>`.
Additional Resources
####################
* :doc:`Extensibility documentation <openvino_docs_Extensibility_UG_Intro>` - describes a special mechanism in OpenVINO that allows adding support of shape inference for custom operations.
* `ov::Model::reshape <classov_1_1Model.html#doxid-classov-1-1-model-1aa21aff80598d5089d591888a4c7f33ae>`__ - in OpenVINO Runtime C++ API
* `Model.reshape <api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.reshape>`__ - in OpenVINO Runtime Python API.
* :doc:`Dynamic Shapes <openvino_docs_OV_UG_DynamicShapes>`
* :doc:`OpenVINO samples <openvino_docs_OV_UG_Samples_Overview>`
* :doc:`Preprocessing API <openvino_docs_OV_UG_Preprocessing_Overview>`
@endsphinxdirective
## Additional Resources
* [Extensibility documentation](@ref openvino_docs_Extensibility_UG_Intro) - describes a special mechanism in OpenVINO that allows adding support of shape inference for custom operations.
* `ov::Model::reshape` - in OpenVINO Runtime C++ API
* [Model.reshape](api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.reshape) - in OpenVINO Runtime Python API.
* [Dynamic Shapes](@ref openvino_docs_OV_UG_DynamicShapes)
* [OpenVINO samples](@ref openvino_docs_OV_UG_Samples_Overview)
* [Preprocessing API](@ref openvino_docs_OV_UG_Preprocessing_Overview)

518
docs/OV_Runtime_UG/migration_ov_2_0/configure_devices.md
View File

@ -1,391 +1,409 @@
# Configuring Devices {#openvino_2_0_configure_devices}
Inference Engine API provides the [ability to configure devices](https://docs.openvino.ai/2021.4/openvino_docs_IE_DG_InferenceEngine_QueryAPI.html) via configuration keys and [get device specific metrics](https://docs.openvino.ai/2021.4/openvino_docs_IE_DG_InferenceEngine_QueryAPI.html#getmetric). The values taken from `InferenceEngine::Core::GetConfig` are requested by the string name, while the return type is `InferenceEngine::Parameter`, making users lost on what the actual type is stored in this parameter.
@sphinxdirective
API 2.0 solves these issues by introducing [properties](../supported_plugins/config_properties.md), which unify metrics and configuration key concepts. The main advantage is that they have the C++ type:
```
static constexpr Property<std::string> full_name{"FULL_DEVICE_NAME"};
```
Inference Engine API provides the `ability to configure devices <https://docs.openvino.ai/2021.4/openvino_docs_IE_DG_InferenceEngine_QueryAPI.html>`__ via configuration keys and `get device specific metrics <https://docs.openvino.ai/2021.4/openvino_docs_IE_DG_InferenceEngine_QueryAPI.html#getmetric>`__. The values taken from `InferenceEngine::Core::GetConfig <namespaceInferenceEngine.html#doxid-namespace-inference-engine-1aff2231f886c9f8fc9c226fd343026789>`__ are requested by the string name, while the return type is `InferenceEngine::Parameter <namespaceInferenceEngine.html#doxid-namespace-inference-engine-1aff2231f886c9f8fc9c226fd343026789>`__, making users lost on what the actual type is stored in this parameter.
API 2.0 solves these issues by introducing :doc:`properties <openvino_docs_OV_UG_query_api>`, which unify metrics and configuration key concepts. The main advantage is that they have the C++ type:
.. code-block::
static constexpr Property<std::string> full_name{"FULL_DEVICE_NAME"};
where the property can be requested from an inference device as:
@snippet ov_properties_migration.cpp core_get_ro_property
The snippets in the following sections demostrate the device configurations for migrating from Inference Engine to API 2.0.
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
:language: cpp
:fragment: core_get_ro_property
## Setting Configuration Values
The snippets in the following sections demonstrate the device configurations for migrating from Inference Engine to API 2.0.
Setting Configuration Values
############################
**Inference Engine API**
@sphinxtabset
@sphinxtab{C++}
.. tab-set::
@sphinxtabset
.. tab-item:: C++
:sync: cpp
@sphinxtab{Devices}
.. tab-set::
@snippet docs/snippets/ov_properties_migration.cpp core_set_config
.. tab-item:: Devices
:sync: devices
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
:language: cpp
:fragment: core_set_config
@sphinxtab{Model Loading}
.. tab-item:: Model Loading
:sync: model-loading
@snippet docs/snippets/ov_properties_migration.cpp core_load_network
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
:language: cpp
:fragment: core_load_network
@endsphinxtab
.. tab-item:: Execution
:sync: execution
@sphinxtab{Execution}
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
:language: cpp
:fragment: executable_network_set_config
@snippet docs/snippets/ov_properties_migration.cpp executable_network_set_config
.. tab-item:: Python
:sync: py
@endsphinxtab
.. tab-set::
@endsphinxtabset
.. tab-item:: Devices
:sync: devices
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_properties_migration.py
:language: python
:fragment: core_set_config
@sphinxtab{Python}
.. tab-item:: Model Loading
:sync: model-loading
@sphinxtabset
.. doxygensnippet:: docs/snippets/ov_properties_migration.py
:language: python
:fragment: core_load_network
@sphinxtab{Devices}
.. tab-item:: Execution
:sync: execution
@snippet docs/snippets/ov_properties_migration.py core_set_config
.. doxygensnippet:: docs/snippets/ov_properties_migration.py
:language: python
:fragment: executable_network_set_config
@endsphinxtab
.. tab-item:: C
:sync: c
@sphinxtab{Model Loading}
.. tab-set::
@snippet docs/snippets/ov_properties_migration.py core_load_network
.. tab-item:: Devices
:sync: devices
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_properties_migration.c
:language: c
:fragment: core_set_config
@sphinxtab{Execution}
.. tab-item:: Model Loading
:sync: model-loading
@snippet docs/snippets/ov_properties_migration.py executable_network_set_config
.. doxygensnippet:: docs/snippets/ov_properties_migration.c
:language: c
:fragment: core_load_network
@endsphinxtab
.. tab-item:: Execution
:sync: execution
@endsphinxtabset
.. doxygensnippet:: docs/snippets/ov_properties_migration.c
:language: c
:fragment: executable_network_set_config
@endsphinxtab
@sphinxtab{C}
@sphinxtabset
@sphinxtab{Devices}
@snippet docs/snippets/ov_properties_migration.c core_set_config
@endsphinxtab
@sphinxtab{Model Loading}
@snippet docs/snippets/ov_properties_migration.c core_load_network
@endsphinxtab
@sphinxtab{Execution}
@snippet docs/snippets/ov_properties_migration.c executable_network_set_config
@endsphinxtab
@endsphinxtabset
@endsphinxtab
@endsphinxtabset
**API 2.0**
@sphinxtabset
@sphinxtab{C++}
.. tab-set::
@sphinxtabset
.. tab-item:: C++
:sync: cpp
@sphinxtab{Devices}
.. tab-set::
@snippet docs/snippets/ov_properties_migration.cpp core_set_property
.. tab-item:: Devices
:sync: devices
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
:language: cpp
:fragment: core_set_property
@sphinxtab{Model Loading}
.. tab-item:: Model Loading
:sync: model-loading
@snippet docs/snippets/ov_properties_migration.cpp core_compile_model
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
:language: cpp
:fragment: core_compile_model
@endsphinxtab
.. tab-item:: Execution
:sync: execution
@sphinxtab{Execution}
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
:language: cpp
:fragment: compiled_model_set_property
@snippet docs/snippets/ov_properties_migration.cpp compiled_model_set_property
.. tab-item:: Python
:sync: py
@endsphinxtab
.. tab-set::
@endsphinxtabset
.. tab-item:: Devices
:sync: devices
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_properties_migration.py
:language: python
:fragment: core_set_property
@sphinxtab{Python}
.. tab-item:: Model Loading
:sync: model-loading
@sphinxtabset
.. doxygensnippet:: docs/snippets/ov_properties_migration.py
:language: python
:fragment: core_compile_model
@sphinxtab{Devices}
.. tab-item:: Execution
:sync: execution
@snippet docs/snippets/ov_properties_migration.py core_set_property
.. doxygensnippet:: docs/snippets/ov_properties_migration.py
:language: python
:fragment: compiled_model_set_property
@endsphinxtab
.. tab-item:: C
:sync: c
@sphinxtab{Model Loading}
.. tab-set::
@snippet docs/snippets/ov_properties_migration.py core_compile_model
.. tab-item:: Devices
:sync: devices
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_properties_migration.c
:language: c
:fragment: core_set_property
@sphinxtab{Execution}
.. tab-item:: Model Loading
:sync: model-loading
@snippet docs/snippets/ov_properties_migration.py compiled_model_set_property
.. doxygensnippet:: docs/snippets/ov_properties_migration.c
:language: c
:fragment: core_compile_model
@endsphinxtab
.. tab-item:: Execution
:sync: execution
@endsphinxtabset
.. doxygensnippet:: docs/snippets/ov_properties_migration.c
:language: c
:fragment: compiled_model_set_property
@endsphinxtab
@sphinxtab{C}
@sphinxtabset
@sphinxtab{Devices}
@snippet docs/snippets/ov_properties_migration.c core_set_property
@endsphinxtab
@sphinxtab{Model Loading}
@snippet docs/snippets/ov_properties_migration.c core_compile_model
@endsphinxtab
@sphinxtab{Execution}
@snippet docs/snippets/ov_properties_migration.c compiled_model_set_property
@endsphinxtab
@endsphinxtabset
@endsphinxtab
@endsphinxtabset
## Getting Information
**Inference Engine API**
@sphinxtabset
@sphinxtab{C++}
.. tab-set::
@sphinxtabset
.. tab-item:: C++
:sync: cpp
@sphinxtab{Device Configuration}
.. tab-set::
@snippet docs/snippets/ov_properties_migration.cpp core_get_config
.. tab-item:: Device Configuration
:sync: device-config
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
:language: cpp
:fragment: core_get_config
@sphinxtab{Device metrics}
.. tab-item:: Device metrics
:sync: device-metrics
@snippet docs/snippets/ov_properties_migration.cpp core_get_metric
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
:language: cpp
:fragment: core_get_metric
@endsphinxtab
.. tab-item:: Execution config
:sync: execution-config
@sphinxtab{Execution config}
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
:language: cpp
:fragment: executable_network_set_config
@snippet docs/snippets/ov_properties_migration.cpp executable_network_get_config
.. tab-item:: Execution metrics
:sync: execution-metrics
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
:language: cpp
:fragment: executable_network_get_metric
@sphinxtab{Execution metrics}
.. tab-item:: Python
:sync: py
@snippet docs/snippets/ov_properties_migration.cpp executable_network_get_metric
.. tab-set::
@endsphinxtab
.. tab-item:: Device Configuration
:sync: device-config
@endsphinxtabset
.. doxygensnippet:: docs/snippets/ov_properties_migration.py
:language: python
:fragment: core_get_config
@endsphinxtab
.. tab-item:: Device metrics
:sync: device-metrics
@sphinxtab{Python}
.. doxygensnippet:: docs/snippets/ov_properties_migration.py
:language: python
:fragment: core_get_metric
@sphinxtabset
.. tab-item:: Execution config
:sync: execution-config
@sphinxtab{Device Configuration}
.. doxygensnippet:: docs/snippets/ov_properties_migration.py
:language: python
:fragment: executable_network_set_config
@snippet docs/snippets/ov_properties_migration.py core_get_config
.. tab-item:: Execution metrics
:sync: execution-metrics
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_properties_migration.py
:language: python
:fragment: executable_network_get_metric
@sphinxtab{Device metrics}
.. tab-item:: C
:sync: c
@snippet docs/snippets/ov_properties_migration.py core_get_metric
.. tab-set::
@endsphinxtab
.. tab-item:: Device Configuration
:sync: device-config
@sphinxtab{Execution config}
.. doxygensnippet:: docs/snippets/ov_properties_migration.c
:language: c
:fragment: core_get_config
@snippet docs/snippets/ov_properties_migration.py executable_network_get_config
.. tab-item:: Device metrics
:sync: device-metrics
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_properties_migration.c
:language: c
:fragment: core_get_metric
@sphinxtab{Execution metrics}
.. tab-item:: Execution config
:sync: execution-config
@snippet docs/snippets/ov_properties_migration.py executable_network_get_metric
.. doxygensnippet:: docs/snippets/ov_properties_migration.c
:language: c
:fragment: executable_network_set_config
@endsphinxtab
.. tab-item:: Execution metrics
:sync: execution-metrics
@endsphinxtabset
.. doxygensnippet:: docs/snippets/ov_properties_migration.c
:language: c
:fragment: executable_network_get_metric
@endsphinxtab
@sphinxtab{C}
@sphinxtabset
@sphinxtab{Device Configuration}
@snippet docs/snippets/ov_properties_migration.c core_get_config
@endsphinxtab
@sphinxtab{Device metrics}
@snippet docs/snippets/ov_properties_migration.c core_get_metric
@endsphinxtab
@sphinxtab{Execution config}
@snippet docs/snippets/ov_properties_migration.c executable_network_get_config
@endsphinxtab
@sphinxtab{Execution metrics}
@snippet docs/snippets/ov_properties_migration.c executable_network_get_metric
@endsphinxtab
@endsphinxtabset
@endsphinxtab
@endsphinxtabset
**API 2.0**
@sphinxtabset
@sphinxtab{C++}
.. tab-set::
@sphinxtabset
.. tab-item:: C++
:sync: cpp
@sphinxtab{Device Configuration}
.. tab-set::
@snippet docs/snippets/ov_properties_migration.cpp core_get_rw_property
.. tab-item:: Device Configuration
:sync: device-config
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
:language: cpp
:fragment: core_get_rw_property
@sphinxtab{Device metrics}
.. tab-item:: Device metrics
:sync: device-metrics
@snippet docs/snippets/ov_properties_migration.cpp core_get_ro_property
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
:language: cpp
:fragment: core_get_ro_property
@endsphinxtab
.. tab-item:: Execution config
:sync: execution-config
@sphinxtab{Execution config}
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
:language: cpp
:fragment: compiled_model_get_rw_property
@snippet docs/snippets/ov_properties_migration.cpp compiled_model_get_rw_property
.. tab-item:: Execution metrics
:sync: execution-metrics
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
:language: cpp
:fragment: compiled_model_get_ro_property
@sphinxtab{Execution metrics}
.. tab-item:: Python
:sync: py
@snippet docs/snippets/ov_properties_migration.cpp compiled_model_get_ro_property
.. tab-set::
@endsphinxtab
.. tab-item:: Device Configuration
:sync: device-config
@endsphinxtabset
.. doxygensnippet:: docs/snippets/ov_properties_migration.py
:language: python
:fragment: core_get_rw_property
@endsphinxtab
.. tab-item:: Device metrics
:sync: device-metrics
@sphinxtab{Python}
.. doxygensnippet:: docs/snippets/ov_properties_migration.py
:language: python
:fragment: core_get_ro_property
@sphinxtabset
.. tab-item:: Execution config
:sync: execution-config
@sphinxtab{Device Configuration}
.. doxygensnippet:: docs/snippets/ov_properties_migration.py
:language: python
:fragment: compiled_model_get_rw_property
@snippet docs/snippets/ov_properties_migration.py core_get_rw_property
.. tab-item:: Execution metrics
:sync: execution-metrics
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_properties_migration.py
:language: python
:fragment: compiled_model_get_ro_property
@sphinxtab{Device metrics}
.. tab-item:: C
:sync: c
@snippet docs/snippets/ov_properties_migration.py core_get_ro_property
.. tab-set::
@endsphinxtab
.. tab-item:: Device Configuration
:sync: device-config
@sphinxtab{Execution config}
.. doxygensnippet:: docs/snippets/ov_properties_migration.c
:language: c
:fragment: core_get_rw_property
@snippet docs/snippets/ov_properties_migration.py compiled_model_get_rw_property
.. tab-item:: Device metrics
:sync: device-metrics
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_properties_migration.c
:language: c
:fragment: core_get_ro_property
@sphinxtab{Execution metrics}
.. tab-item:: Execution config
:sync: execution-config
@snippet docs/snippets/ov_properties_migration.py compiled_model_get_ro_property
.. doxygensnippet:: docs/snippets/ov_properties_migration.c
:language: c
:fragment: compiled_model_get_rw_property
@endsphinxtab
.. tab-item:: Execution metrics
:sync: execution-metrics
@endsphinxtabset
.. doxygensnippet:: docs/snippets/ov_properties_migration.c
:language: c
:fragment: compiled_model_get_ro_property
@endsphinxtab
@sphinxtab{C}
@sphinxtabset
@sphinxtab{Device Configuration}
@snippet docs/snippets/ov_properties_migration.c core_get_rw_property
@endsphinxtab
@sphinxtab{Device metrics}
@snippet docs/snippets/ov_properties_migration.c core_get_ro_property
@endsphinxtab
@sphinxtab{Execution config}
@snippet docs/snippets/ov_properties_migration.c compiled_model_get_rw_property
@endsphinxtab
@sphinxtab{Execution metrics}
@snippet docs/snippets/ov_properties_migration.c compiled_model_get_ro_property
@endsphinxtab
@endsphinxtabset
@endsphinxtab
@endsphinxtabset
@endsphinxdirective

62
docs/OV_Runtime_UG/migration_ov_2_0/graph_construction.md
View File

@ -1,38 +1,56 @@
# Model Creation in OpenVINO™ Runtime {#openvino_2_0_model_creation}
OpenVINO™ Runtime with API 2.0 includes the nGraph engine as a common part. The `ngraph` namespace has been changed to `ov`, but all other parts of the ngraph API have been preserved.
@sphinxdirective
OpenVINO™ Runtime with API 2.0 includes the nGraph engine as a common part. The ``ngraph`` namespace has been changed to ``ov``, but all other parts of the ngraph API have been preserved.
The code snippets below show how to change the application code for migration to API 2.0.
## nGraph API
nGraph API
####################
@sphinxtabset
.. tab-set::
@sphinxtab{C++}
@snippet docs/snippets/ngraph.cpp ngraph:graph
@endsphinxtab
.. tab-item:: C++
:sync: cpp
@sphinxtab{Python}
@snippet docs/snippets/ngraph.py ngraph:graph
@endsphinxtab
.. doxygensnippet:: docs/snippets/ngraph.cpp
:language: cpp
:fragment: ngraph:graph
@endsphinxtabset
.. tab-item:: Python
:sync: py
## API 2.0
.. doxygensnippet:: docs/snippets/ngraph.py
:language: Python
:fragment: ngraph:graph
@sphinxtabset
@sphinxtab{C++}
@snippet docs/snippets/ov_graph.cpp ov:graph
@endsphinxtab
API 2.0
####################
@sphinxtab{Python}
@snippet docs/snippets/ov_graph.py ov:graph
@endsphinxtab
@endsphinxtabset
.. tab-set::
## Additional Resources
.. tab-item:: C++
:sync: cpp
- [Hello Model Creation C++ Sample](../../../samples/cpp/model_creation_sample/README.md)
- [Hello Model Creation Python Sample](../../../samples/python/model_creation_sample/README.md)
.. doxygensnippet:: docs/snippets/ov_graph.cpp
:language: cpp
:fragment: ov:graph
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_graph.py
:language: Python
:fragment: ov:graph
Additional Resources
####################
* :doc:`Hello Model Creation C++ Sample <openvino_inference_engine_samples_model_creation_sample_README>`
* :doc:`Hello Model Creation Python Sample <openvino_inference_engine_ie_bridges_python_sample_model_creation_sample_README>`
@endsphinxdirective

230
docs/OV_Runtime_UG/preprocessing_overview.md
View File

@ -10,162 +10,188 @@
openvino_docs_OV_UG_Layout_Overview
openvino_docs_OV_UG_Preprocess_Usecase_save
@endsphinxdirective
## Introduction
Introduction
####################
When input data does not fit the model input tensor perfectly, additional operations/steps are needed to transform the data to the format expected by the model. These operations are known as "preprocessing".
### Example
Consider the following standard example: deep learning model expects input with the `{1, 3, 224, 224}` shape, `FP32` precision, `RGB` color channels order, and it requires data normalization (subtract mean and divide by scale factor). However, there is just a `640x480` `BGR` image (data is `{480, 640, 3}`). This means that the following operations must be performed:
- Convert `U8` buffer to `FP32`.
- Transform to `planar` format: from `{1, 480, 640, 3}` to `{1, 3, 480, 640}`.
- Resize image from 640x480 to 224x224.
- Make `BGR->RGB` conversion as model expects `RGB`.
- For each pixel, subtract mean values and divide by scale factor.
Example
++++++++++++++++++++
Consider the following standard example: deep learning model expects input with the ``{1, 3, 224, 224}`` shape, ``FP32`` precision, ``RGB`` color channels order, and it requires data normalization (subtract mean and divide by scale factor). However, there is just a ``640x480 BGR`` image (data is ``{480, 640, 3}``). This means that the following operations must be performed:
* Convert ``U8`` buffer to ``FP32``.
* Transform to ``planar`` format: from ``{1, 480, 640, 3}`` to ``{1, 3, 480, 640}``.
* Resize image from 640x480 to 224x224.
* Make ``BGR->RGB`` conversion as model expects ``RGB``.
* For each pixel, subtract mean values and divide by scale factor.
![](img/preprocess_not_fit.png)
.. image:: _static/images/preprocess_not_fit.png
Even though it is relatively easy to implement all these steps in the application code manually, before actual inference, it is also possible with the use of Preprocessing API. Advantages of using the API are:
- Preprocessing API is easy to use.
- Preprocessing steps will be integrated into execution graph and will be performed on selected device (CPU/GPU/etc.) rather than always being executed on CPU. This will improve selected device utilization which is always good.
## Preprocessing API
* Preprocessing API is easy to use.
* Preprocessing steps will be integrated into execution graph and will be performed on selected device (CPU/GPU/etc.) rather than always being executed on CPU. This will improve selected device utilization which is always good.
Preprocessing API
####################
Intuitively, preprocessing API consists of the following parts:
1. **Tensor** - declares user data format, like shape, [layout](./layout_overview.md), precision, color format from actual user's data.
2. **Steps** - describes sequence of preprocessing steps which need to be applied to user data.
3. **Model** - specifies model data format. Usually, precision and shape are already known for model, only additional information, like [layout](./layout_overview.md) can be specified.
> **NOTE**: Graph modifications of a model shall be performed after the model is read from a drive and **before** it is loaded on the actual device.
1. **Tensor** - declares user data format, like shape, :doc:`layout <openvino_docs_OV_UG_Layout_Overview>`, precision, color format from actual user's data.
2. **Steps** - describes sequence of preprocessing steps which need to be applied to user data.
3. **Model** - specifies model data format. Usually, precision and shape are already known for model, only additional information, like :doc:`layout <openvino_docs_OV_UG_Layout_Overview>` can be specified.
### PrePostProcessor Object
.. note::
The `ov::preprocess::PrePostProcessor` class allows specifying preprocessing and postprocessing steps for a model read from disk.
Graph modifications of a model shall be performed after the model is read from a drive and **before** it is loaded on the actual device.
@sphinxtabset
PrePostProcessor Object
+++++++++++++++++++++++
@sphinxtab{C++}
The `ov::preprocess::PrePostProcessor <classov_1_1preprocess_1_1PrePostProcessor.html#doxid-classov-1-1preprocess-1-1-pre-post-processor>`__ class allows specifying preprocessing and postprocessing steps for a model read from disk.
@snippet docs/snippets/ov_preprocessing.cpp ov:preprocess:create
.. tab-set::
@endsphinxtab
.. tab-item:: C++
:sync: cpp
@sphinxtab{Python}
.. doxygensnippet:: docs/snippets/ov_preprocessing.cpp
:language: cpp
:fragment: ov:preprocess:create
@snippet docs/snippets/ov_preprocessing.py ov:preprocess:create
.. tab-item:: Python
:sync: py
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_preprocessing.py
:language: python
:fragment: ov:preprocess:create
@endsphinxtabset
### Declare User's Data Format
Declare User's Data Format
++++++++++++++++++++++++++
To address particular input of a model/preprocessor, use the `ov::preprocess::PrePostProcessor::input(input_name)` method.
To address particular input of a model/preprocessor, use the ``ov::preprocess::PrePostProcessor::input(input_name)`` method.
@sphinxtabset
@sphinxtab{C++}
.. tab-set::
@snippet docs/snippets/ov_preprocessing.cpp ov:preprocess:tensor
.. tab-item:: C++
:sync: cpp
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_preprocessing.cpp
:language: cpp
:fragment: ov:preprocess:tensor
@sphinxtab{Python}
.. tab-item:: Python
:sync: py
@snippet docs/snippets/ov_preprocessing.py ov:preprocess:tensor
.. doxygensnippet:: docs/snippets/ov_preprocessing.py
:language: python
:fragment: ov:preprocess:tensor
@endsphinxtab
@endsphinxtabset
Below is all the specified input information:
- Precision is `U8` (unsigned 8-bit integer).
- Data represents tensor with the `{1,480,640,3}` shape.
- [Layout](./layout_overview.md) is "NHWC". It means: `height=480`, `width=640`, `channels=3`'.
- Color format is `BGR`.
@anchor declare_model_s_layout
### Declaring Model Layout
Model input already has information about precision and shape. Preprocessing API is not intended to modify this. The only thing that may be specified is input data [layout](./layout_overview.md)
@sphinxtabset
@sphinxtab{C++}
@snippet docs/snippets/ov_preprocessing.cpp ov:preprocess:model
@endsphinxtab
@sphinxtab{Python}
@snippet docs/snippets/ov_preprocessing.py ov:preprocess:model
@endsphinxtab
@endsphinxtabset
* Precision is ``U8`` (unsigned 8-bit integer).
* Data represents tensor with the ``{1,480,640,3}`` shape.
* :doc:`Layout <openvino_docs_OV_UG_Layout_Overview>` is "NHWC". It means: ``height=480``, ``width=640``, ``channels=3``.
* Color format is ``BGR``.
Now, if the model input has `{1,3,224,224}` shape, preprocessing will be able to identify the `height=224`, `width=224`, and `channels=3` of that model. The `height`/`width` information is necessary for `resize`, and `channels` is needed for mean/scale normalization.
.. _declare_model_s_layout:
### Preprocessing Steps
Declaring Model Layout
++++++++++++++++++++++
Model input already has information about precision and shape. Preprocessing API is not intended to modify this. The only thing that may be specified is input data :doc:`layout <openvino_docs_OV_UG_Layout_Overview>`
.. tab-set::
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_preprocessing.cpp
:language: cpp
:fragment: ov:preprocess:model
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_preprocessing.py
:language: python
:fragment: ov:preprocess:model
Now, if the model input has ``{1,3,224,224}`` shape, preprocessing will be able to identify the ``height=224``, ``width=224``, and ``channels=3`` of that model. The ``height``/ ``width`` information is necessary for ``resize``, and ``channels`` is needed for mean/scale normalization.
Preprocessing Steps
++++++++++++++++++++
Now, the sequence of preprocessing steps can be defined:
@sphinxtabset
@sphinxtab{C++}
.. tab-set::
@snippet docs/snippets/ov_preprocessing.cpp ov:preprocess:steps
.. tab-item:: C++
:sync: cpp
@endsphinxtab
.. doxygensnippet:: docs/snippets/ov_preprocessing.cpp
:language: cpp
:fragment: ov:preprocess:steps
@sphinxtab{Python}
.. tab-item:: Python
:sync: py
@snippet docs/snippets/ov_preprocessing.py ov:preprocess:steps
.. doxygensnippet:: docs/snippets/ov_preprocessing.py
:language: python
:fragment: ov:preprocess:steps
@endsphinxtab
@endsphinxtabset
Perform the following:
1. Convert `U8` to `FP32` precision.
2. Convert current color format from `BGR` to `RGB`.
3. Resize to `height`/`width` of a model. Be aware that if a model accepts dynamic size e.g., `{?, 3, ?, ?}`, `resize` will not know how to resize the picture. Therefore, in this case, target `height`/`width` should be specified. For more details, see also the `ov::preprocess::PreProcessSteps::resize()`.
4. Subtract mean from each channel. In this step, color format is already `RGB`, so `100.5` will be subtracted from each `Red` component, and `101.5` will be subtracted from each `Blue` one.
5. Divide each pixel data to appropriate scale value. In this example, each `Red` component will be divided by 50, `Green` by 51, and `Blue` by 52 respectively.
6. Keep in mind that the last `convert_layout` step is commented out as it is not necessary to specify the last layout conversion. The `PrePostProcessor` will do such conversion automatically.
1. Convert ``U8`` to ``FP32`` precision.
2. Convert current color format from ``BGR`` to ``RGB``.
3. Resize to ``height``/ ``width`` of a model. Be aware that if a model accepts dynamic size e.g., ``{?, 3, ?, ?}``, ``resize`` will not know how to resize the picture. Therefore, in this case, target ``height``/ ``width`` should be specified. For more details, see also the `ov::preprocess::PreProcessSteps::resize() <classov_1_1preprocess_1_1PreProcessSteps.html#doxid-classov-1-1preprocess-1-1-pre-process-steps-1a40dab78be1222fee505ed6a13400efe6>`__.
4. Subtract mean from each channel. In this step, color format is already ``RGB``, so ``100.5`` will be subtracted from each ``Red`` component, and ``101.5`` will be subtracted from each ``Blue`` one.
5. Divide each pixel data to appropriate scale value. In this example, each ``Red`` component will be divided by 50, ``Green`` by 51, and ``Blue`` by 52 respectively.
6. Keep in mind that the last ``convert_layout`` step is commented out as it is not necessary to specify the last layout conversion. The ``PrePostProcessor`` will do such conversion automatically.
### Integrating Steps into a Model
Integrating Steps into a Model
++++++++++++++++++++++++++++++
Once the preprocessing steps have been finished the model can be finally built. It is possible to display `PrePostProcessor` configuration for debugging purposes:
@sphinxtabset
@sphinxtab{C++}
@snippet docs/snippets/ov_preprocessing.cpp ov:preprocess:build
@endsphinxtab
@sphinxtab{Python}
@snippet docs/snippets/ov_preprocessing.py ov:preprocess:build
@endsphinxtab
@endsphinxtabset
Once the preprocessing steps have been finished the model can be finally built. It is possible to display ``PrePostProcessor`` configuration for debugging purposes:
The `model` will accept `U8` input with the shape of `{1, 480, 640, 3}` and the `BGR` channel order. All conversion steps will be integrated into the execution graph. Now, model can be loaded on the device and the image can be passed to the model without any data manipulation in the application.
.. tab-set::
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_preprocessing.cpp
:language: cpp
:fragment: ov:preprocess:build
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_preprocessing.py
:language: python
:fragment: ov:preprocess:build
## Additional Resources
The ``model`` will accept ``U8`` input with the shape of ``{1, 480, 640, 3}`` and the ``BGR`` channel order. All conversion steps will be integrated into the execution graph. Now, model can be loaded on the device and the image can be passed to the model without any data manipulation in the application.
* [Preprocessing Details](@ref openvino_docs_OV_UG_Preprocessing_Details)
* [Layout API overview](@ref openvino_docs_OV_UG_Layout_Overview)
* <code>ov::preprocess::PrePostProcessor</code> C++ class documentation
Additional Resources
####################
* :doc:`Preprocessing Details <openvino_docs_OV_UG_Preprocessing_Details>`
* :doc:`Layout API overview <openvino_docs_OV_UG_Layout_Overview>`
* `ov::preprocess::PrePostProcessor <classov_1_1preprocess_1_1PrePostProcessor.html#doxid-classov-1-1preprocess-1-1-pre-post-processor>`__ C++ class documentation
@endsphinxdirective

159
docs/OV_Runtime_UG/preprocessing_usecase_save.md
View File

@ -1,84 +1,111 @@
# Use Case - Integrate and Save Preprocessing Steps Into IR {#openvino_docs_OV_UG_Preprocess_Usecase_save}
@sphinxdirective
Previous sections covered the topic of the [preprocessing steps](@ref openvino_docs_OV_UG_Preprocessing_Details) and the overview of [Layout](@ref openvino_docs_OV_UG_Layout_Overview) API.
Previous sections covered the topic of the :doc:`preprocessing steps <openvino_docs_OV_UG_Preprocessing_Details>`
and the overview of :doc:`Layout <openvino_docs_OV_UG_Layout_Overview>` API.
For many applications, it is also important to minimize read/load time of a model. Therefore, performing integration of preprocessing steps every time on application startup, after `ov::runtime::Core::read_model`, may seem inconvenient. In such cases, once pre and postprocessing steps have been added, it can be useful to store new execution model to OpenVINO Intermediate Representation (OpenVINO IR, `.xml` format).
For many applications, it is also important to minimize read/load time of a model.
Therefore, performing integration of preprocessing steps every time on application
startup, after ``ov::runtime::Core::read_model``, may seem inconvenient. In such cases,
once pre and postprocessing steps have been added, it can be useful to store new execution
model to OpenVINO Intermediate Representation (OpenVINO IR, `.xml` format).
Most available preprocessing steps can also be performed via command-line options, using Model Optimizer. For details on such command-line options, refer to the [Optimizing Preprocessing Computation](../MO_DG/prepare_model/Additional_Optimizations.md).
Most available preprocessing steps can also be performed via command-line options,
using Model Optimizer. For details on such command-line options, refer to the
:doc:`Optimizing Preprocessing Computation <openvino_docs_MO_DG_Additional_Optimization_Use_Cases>`.
## Code example - Saving Model with Preprocessing to OpenVINO IR
Code example - Saving Model with Preprocessing to OpenVINO IR
#############################################################
When some preprocessing steps cannot be integrated into the execution graph using Model Optimizer command-line options (for example, `YUV`->`RGB` color space conversion, `Resize`, etc.), it is possible to write a simple code which:
- Reads the original model (OpenVINO IR, TensorFlow, ONNX, PaddlePaddle).
- Adds the preprocessing/postprocessing steps.
- Saves resulting model as IR (`.xml` and `.bin`).
When some preprocessing steps cannot be integrated into the execution graph using
Model Optimizer command-line options (for example, ``YUV``->``RGB`` color space conversion,
``Resize``, etc.), it is possible to write a simple code which:
Consider the example, where an original ONNX model takes one `float32` input with the `{1, 3, 224, 224}` shape, the `RGB` channel order, and mean/scale values applied. In contrast, the application provides `BGR` image buffer with a non-fixed size and input images as batches of two. Below is the model conversion code that can be applied in the model preparation script for such a case.
* Reads the original model (OpenVINO IR, TensorFlow, ONNX, PaddlePaddle).
* Adds the preprocessing/postprocessing steps.
* Saves resulting model as IR (``.xml`` and ``.bin``).
- Includes / Imports
Consider the example, where an original ONNX model takes one ``float32`` input with the
``{1, 3, 224, 224}`` shape, the ``RGB`` channel order, and mean/scale values applied.
In contrast, the application provides ``BGR`` image buffer with a non-fixed size and
input images as batches of two. Below is the model conversion code that can be applied
in the model preparation script for such a case.
@sphinxtabset
@sphinxtab{C++}
@snippet docs/snippets/ov_preprocessing.cpp ov:preprocess:save_headers
@endsphinxtab
@sphinxtab{Python}
@snippet docs/snippets/ov_preprocessing.py ov:preprocess:save_headers
@endsphinxtab
@endsphinxtabset
- Preprocessing & Saving to the OpenVINO IR code.
@sphinxtabset
@sphinxtab{C++}
@snippet docs/snippets/ov_preprocessing.cpp ov:preprocess:save
@endsphinxtab
@sphinxtab{Python}
@snippet docs/snippets/ov_preprocessing.py ov:preprocess:save
@endsphinxtab
@endsphinxtabset
* Includes / Imports
## Application Code - Load Model to Target Device
.. tab-set::
After this, the application code can load a saved file and stop preprocessing. In this case, enable [model caching](./Model_caching_overview.md) to minimize load time when the cached model is available.
.. tab-item:: C++
:sync: cpp
@sphinxtabset
.. doxygensnippet:: docs/snippets/ov_preprocessing.cpp
:language: cpp
:fragment: ov:preprocess:save_headers
@sphinxtab{C++}
.. tab-item:: Python
:sync: py
@snippet docs/snippets/ov_preprocessing.cpp ov:preprocess:save_load
@endsphinxtab
@sphinxtab{Python}
@snippet docs/snippets/ov_preprocessing.py ov:preprocess:save_load
@endsphinxtab
@endsphinxtabset
.. doxygensnippet:: docs/snippets/ov_preprocessing.py
:language: Python
:fragment: ov:preprocess:save_headers
## Additional Resources
* [Preprocessing Details](@ref openvino_docs_OV_UG_Preprocessing_Details)
* [Layout API overview](@ref openvino_docs_OV_UG_Layout_Overview)
* [Model Optimizer - Optimize Preprocessing Computation](../MO_DG/prepare_model/Additional_Optimizations.md)
* [Model Caching Overview](./Model_caching_overview.md)
* The `ov::preprocess::PrePostProcessor` C++ class documentation
* The `ov::pass::Serialize` - pass to serialize model to XML/BIN
* The `ov::set_batch` - update batch dimension for a given model
* Preprocessing & Saving to the OpenVINO IR code.
.. tab-set::
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_preprocessing.cpp
:language: cpp
:fragment: ov:preprocess:save
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_preprocessing.py
:language: Python
:fragment: ov:preprocess:save
Application Code - Load Model to Target Device
##############################################
After this, the application code can load a saved file and stop preprocessing. In this case, enable
:doc:`model caching <openvino_docs_OV_UG_Model_caching_overview>` to minimize load
time when the cached model is available.
.. tab-set::
.. tab-item:: C++
:sync: cpp
.. doxygensnippet:: docs/snippets/ov_preprocessing.cpp
:language: cpp
:fragment: ov:preprocess:save_load
.. tab-item:: Python
:sync: py
.. doxygensnippet:: docs/snippets/ov_preprocessing.py
:language: Python
:fragment: ov:preprocess:save_load
Additional Resources
####################
* :doc:`Preprocessing Details <openvino_docs_OV_UG_Preprocessing_Details>`
* :doc:`Layout API overview <openvino_docs_OV_UG_Layout_Overview>`
* :doc:`Model Optimizer - Optimize Preprocessing Computation <openvino_docs_MO_DG_Additional_Optimization_Use_Cases>`
* :doc:`Model Caching Overview<openvino_docs_OV_UG_Model_caching_overview>`
* The `ov::preprocess::PrePostProcessor <classov_1_1preprocess_1_1PrePostProcessor.html#doxid-classov-1-1preprocess-1-1-pre-post-processor>` C++ class documentation
* The `ov::pass::Serialize <classov_1_1pass_1_1Serialize.html#doxid-classov-1-1pass-1-1-serialize>` - pass to serialize model to XML/BIN
* The `ov::set_batch <namespaceov.html#doxid-namespaceov-1a3314e2ff91fcc9ffec05b1a77c37862b>` - update batch dimension for a given model
@endsphinxdirective

73
docs/OV_Runtime_UG/protecting_model_guide.md
View File

@ -1,59 +1,50 @@
# Using Encrypted Models with OpenVINO {#openvino_docs_OV_UG_protecting_model_guide}
Deploying deep-learning capabilities to edge devices can present security
challenges like ensuring inference integrity, or providing copyright
protection of your deep-learning models.
@sphinxdirective
One possible solution is to use cryptography to protect models as they are
deployed and stored on edge devices. Model encryption, decryption and
authentication are not provided by OpenVINO but can be implemented with
third-party tools (i.e., OpenSSL). While implementing encryption, ensure that
the latest versions of tools are used and follow cryptography best practices.
Deploying deep-learning capabilities to edge devices can present security challenges like ensuring inference integrity, or providing copyright protection of your deep-learning models.
One possible solution is to use cryptography to protect models as they are deployed and stored on edge devices. Model encryption, decryption and authentication are not provided by OpenVINO but can be implemented with third-party tools (i.e., OpenSSL). While implementing encryption, ensure that the latest versions of tools are used and follow cryptography best practices.
This guide presents how to use OpenVINO securely with protected models.
## Secure Model Deployment
Secure Model Deployment
#######################
After a model is optimized by the OpenVINO Model Optimizer, it's deployed
to target devices in the OpenVINO Intermediate Representation (OpenVINO IR) format. An optimized
model is stored on edge device and is executed by the OpenVINO Runtime.
TensorFlow, ONNX and PaddlePaddle models can be read natively by OpenVINO Runtime as well.
After a model is optimized by the OpenVINO Model Optimizer, it's deployednto target devices in the OpenVINO Intermediate Representation (OpenVINO IR) format. An optimized model is stored on edge device and is executed by the OpenVINO Runtime. TensorFlow, ONNX and PaddlePaddle models can be read natively by OpenVINO Runtime as well.
Encrypting and optimizing model before deploying it to the edge device can be
used to protect deep-learning models. The edge device should keep the stored model
protected all the time and have the model decrypted **in runtime only** for use
by the OpenVINO Runtime.
Encrypting and optimizing model before deploying it to the edge device can be used to protect deep-learning models. The edge device should keep the stored model protected all the time and have the model decrypted **in runtime only** for use by the OpenVINO Runtime.
![deploy_encrypted_model](./img/deploy_encrypted_model.svg)
.. image:: _static/images/deploy_encrypted_model.svg
## Loading Encrypted Models
Loading Encrypted Models
########################
The OpenVINO Runtime requires model decryption before loading. Allocate
a temporary memory block for model decryption and use the
`ov::Core::read_model` method to load the model from a memory buffer.
For more information, see the `ov::Core` Class Reference Documentation.
The OpenVINO Runtime requires model decryption before loading. Allocate a temporary memory block for model decryption and use the ``ov::Core::read_model`` method to load the model from a memory buffer. For more information, see the ``ov::Core`` Class Reference Documentation.
@snippet snippets/protecting_model_guide.cpp part0
.. doxygensnippet:: docs/snippets/protecting_model_guide.cpp
:language: cpp
:fragment: part0
Hardware-based protection such as Intel Software Guard Extensions
(Intel SGX) can be used to protect decryption operation secrets and
bind them to a device. For more information, see the [Intel Software Guard
Extensions](https://software.intel.com/en-us/sgx).
Hardware-based protection such as Intel Software Guard Extensions (Intel SGX) can be used to protect decryption operation secrets and bind them to a device. For more information, see the `Intel Software Guard Extensions <https://software.intel.com/en-us/sgx>`__.
Use the `ov::Core::read_model` to set model representations and
weights respectively.
Use the ``ov::Core::read_model`` to set model representations and weights respectively.
Currently there is no way to read external weights from memory for ONNX models.
The `ov::Core::read_model(const std::string& model, const Tensor& weights)` method
should be called with `weights` passed as an empty `ov::Tensor`.
Currently there is no way to read external weights from memory for ONNX models. The ``ov::Core::read_model(const std::string& model, const Tensor& weights)`` method should be called with ``weights`` passed as an empty ``ov::Tensor``.
@snippet snippets/protecting_model_guide.cpp part1
.. doxygensnippet:: docs/snippets/protecting_model_guide.cpp
:language: cpp
:fragment: part1
## Additional Resources
Additional Resources
####################
- Intel® Distribution of OpenVINO™ toolkit `home page <https://software.intel.com/en-us/openvino-toolkit>`__.
- Model Optimizer :doc:`Developer Guide <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`.
- :doc:`OpenVINO™ Runtime User Guide <openvino_docs_OV_UG_OV_Runtime_User_Guide>`.
- For more information on Sample Applications, see the :doc:`OpenVINO Samples Overview <openvino_docs_OV_UG_Samples_Overview>`
- For information on a set of pre-trained models, see the `Overview of OpenVINO™ Toolkit Pre-Trained Models <https://docs.openvino.ai/latest/omz_models_group_intel.html#doxid-omz-models-group-intel>`__.
- For IoT Libraries and Code Samples, see the `Intel® IoT Developer Kit <https://github.com/intel-iot-devkit>`__.
@endsphinxdirective
- Intel® Distribution of OpenVINO™ toolkit [home page](https://software.intel.com/en-us/openvino-toolkit).
- Model Optimizer [Developer Guide](../MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md).
- [OpenVINO™ Runtime User Guide](openvino_intro.md).
- For more information on Sample Applications, see the [OpenVINO Samples Overview](Samples_Overview.md)
- For information on a set of pre-trained models, see the [Overview of OpenVINO™ Toolkit Pre-Trained Models](@ref omz_models_group_intel).
- For IoT Libraries and Code Samples, see the [Intel® IoT Developer Kit](https://github.com/intel-iot-devkit).

0
docs/OV_Runtime_UG/img/deploy_encrypted_model.svg → docs/_static/images/deploy_encrypted_model.svg vendored
View File

0
docs/OV_Runtime_UG/img/original_vs_reshaped_model.svg → docs/_static/images/original_vs_reshaped_model.svg vendored
View File

0
docs/OV_Runtime_UG/img/preprocess_not_fit.png → docs/_static/images/preprocess_not_fit.png vendored
View File

11
docs/security_guide/introduction.md
View File

@ -1,4 +1,4 @@
# OpenVINO&trade; Security {#openvino_docs_security_guide_introduction}
# OpenVINO Security {#openvino_docs_security_guide_introduction}
@sphinxdirective
@ -8,12 +8,13 @@
openvino_docs_OV_UG_protecting_model_guide
@endsphinxdirective
Deploying deep learning models for OpenVINO&trade; may raise security and privacy issues.
Deploying deep learning models for OpenVINO may raise security and privacy issues.
Trained models are often valuable intellectual property and you may choose to protect them with encryption or other security tools.
Actual security and privacy requirements depend on your unique deployment scenario.
This section provides general guidance on using OpenVINO tools and libraries securely.
The main security measure for OpenVINO is its [Security Add-on](../ovsa/ovsa_get_started.md). You can find its description in the Ecosystem section.
The main security measure for OpenVINO is its :doc:`Security Add-on <ovsa_get_started>`. You can find its description in the Ecosystem section.
@endsphinxdirective

35
samples/cpp/benchmark_app/inputs_filling.cpp
View File

@ -15,7 +15,6 @@
#include "format_reader_ptr.h"
#include "samples/slog.hpp"
#include "shared_tensor_allocator.hpp"
#include "utils.hpp"
template <typename T>
@ -31,10 +30,8 @@ ov::Tensor create_tensor_from_image(const std::vector<std::string>& files,
const benchmark_app::InputInfo& inputInfo,
const std::string& inputName,
std::string* filenames_used = nullptr) {
size_t tensor_size =
std::accumulate(inputInfo.dataShape.begin(), inputInfo.dataShape.end(), 1, std::multiplies<size_t>());
auto allocator = std::make_shared<SharedTensorAllocator>(tensor_size * sizeof(T));
auto data = reinterpret_cast<T*>(allocator->get_buffer());
auto tensor = ov::Tensor(inputInfo.type, inputInfo.dataShape);
auto data = tensor.data<T>();
/** Collect images data ptrs **/
std::vector<std::shared_ptr<uint8_t>> vreader;
@ -90,7 +87,6 @@ ov::Tensor create_tensor_from_image(const std::vector<std::string>& files,
}
}
auto tensor = ov::Tensor(inputInfo.type, inputInfo.dataShape, ov::Allocator(allocator));
return tensor;
}
@ -103,8 +99,8 @@ ov::Tensor create_tensor_from_numpy(const std::vector<std::string>& files,
std::string* filenames_used = nullptr) {
size_t tensor_size =
std::accumulate(inputInfo.dataShape.begin(), inputInfo.dataShape.end(), 1, std::multiplies<size_t>());
auto allocator = std::make_shared<SharedTensorAllocator>(tensor_size * sizeof(T));
auto data = reinterpret_cast<T*>(allocator->get_buffer());
auto tensor = ov::Tensor(inputInfo.type, inputInfo.dataShape);
auto data = tensor.data<T>();
std::vector<std::shared_ptr<unsigned char>> numpy_array_pointers;
numpy_array_pointers.reserve(batchSize);
@ -150,7 +146,7 @@ ov::Tensor create_tensor_from_numpy(const std::vector<std::string>& files,
}
}
return ov::Tensor(inputInfo.type, inputInfo.dataShape, ov::Allocator(allocator));
return tensor;
}
template <typename T>
@ -160,8 +156,8 @@ ov::Tensor create_tensor_im_info(const std::pair<size_t, size_t>& image_size,
const std::string& inputName) {
size_t tensor_size =
std::accumulate(inputInfo.dataShape.begin(), inputInfo.dataShape.end(), 1, std::multiplies<size_t>());
auto allocator = std::make_shared<SharedTensorAllocator>(tensor_size * sizeof(T));
auto data = reinterpret_cast<T*>(allocator->get_buffer());
auto tensor = ov::Tensor(inputInfo.type, inputInfo.dataShape);
char* data = static_cast<char*>(tensor.data());
size_t infoBatchSize = 1;
if (!inputInfo.layout.empty() && ov::layout::has_batch(inputInfo.layout)) {
@ -176,15 +172,14 @@ ov::Tensor create_tensor_im_info(const std::pair<size_t, size_t>& image_size,
for (size_t i = 0; i < iminfoSize; i++) {
size_t index = b * iminfoSize + i;
if (0 == i)
data[index] = static_cast<T>(image_size.first);
data[index] = static_cast<char>(image_size.first);
else if (1 == i)
data[index] = static_cast<T>(image_size.second);
data[index] = static_cast<char>(image_size.second);
else
data[index] = 1;
data[index] = static_cast<char>(1);
}
}
auto tensor = ov::Tensor(inputInfo.type, inputInfo.dataShape, ov::Allocator(allocator));
return tensor;
}
@ -197,8 +192,8 @@ ov::Tensor create_tensor_from_binary(const std::vector<std::string>& files,
std::string* filenames_used = nullptr) {
size_t tensor_size =
std::accumulate(inputInfo.dataShape.begin(), inputInfo.dataShape.end(), 1, std::multiplies<size_t>());
auto allocator = std::make_shared<SharedTensorAllocator>(tensor_size * sizeof(T));
char* data = allocator->get_buffer();
auto tensor = ov::Tensor(inputInfo.type, inputInfo.dataShape);
char* data = static_cast<char*>(tensor.data());
size_t binaryBatchSize = 1;
if (!inputInfo.layout.empty() && ov::layout::has_batch(inputInfo.layout)) {
binaryBatchSize = batchSize;
@ -245,7 +240,6 @@ ov::Tensor create_tensor_from_binary(const std::vector<std::string>& files,
}
}
auto tensor = ov::Tensor(inputInfo.type, inputInfo.dataShape, ov::Allocator(allocator));
return tensor;
}
@ -255,8 +249,8 @@ ov::Tensor create_tensor_random(const benchmark_app::InputInfo& inputInfo,
T rand_max = std::numeric_limits<uint8_t>::max()) {
size_t tensor_size =
std::accumulate(inputInfo.dataShape.begin(), inputInfo.dataShape.end(), 1, std::multiplies<size_t>());
auto allocator = std::make_shared<SharedTensorAllocator>(tensor_size * sizeof(T));
auto data = reinterpret_cast<T*>(allocator->get_buffer());
auto tensor = ov::Tensor(inputInfo.type, inputInfo.dataShape);
auto data = tensor.data<T>();
std::mt19937 gen(0);
uniformDistribution<T2> distribution(rand_min, rand_max);
@ -264,7 +258,6 @@ ov::Tensor create_tensor_random(const benchmark_app::InputInfo& inputInfo,
data[i] = static_cast<T>(distribution(gen));
}
auto tensor = ov::Tensor(inputInfo.type, inputInfo.dataShape, ov::Allocator(allocator));
return tensor;
}

42
samples/cpp/benchmark_app/shared_tensor_allocator.hpp
View File

@ -1,42 +0,0 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#pragma once
#include "openvino/runtime/allocator.hpp"
class SharedTensorAllocator : public ov::AllocatorImpl {
public:
SharedTensorAllocator(size_t sizeBytes) : size(sizeBytes) {
data = new char[size];
}
~SharedTensorAllocator() {
delete[] data;
}
virtual void* allocate(const size_t bytes, const size_t) override {
return bytes <= this->size ? (void*)data : nullptr;
}
void deallocate(void* handle, const size_t bytes, const size_t) override {
if (handle == data) {
delete[] data;
data = nullptr;
}
}
bool is_equal(const AllocatorImpl& other) const override {
auto other_tensor_allocator = dynamic_cast<const SharedTensorAllocator*>(&other);
return other_tensor_allocator != nullptr && other_tensor_allocator == this;
}
char* get_buffer() {
return data;
}
private:
char* data;
size_t size;
};

9
src/bindings/c/tests/CMakeLists.txt
View File

@ -11,9 +11,7 @@ target_link_libraries(${TARGET_NAME} PRIVATE openvino_c commonTestUtils gtest_ma
target_compile_definitions(${TARGET_NAME}
PRIVATE
$<$<BOOL:${ENABLE_GAPI_PREPROCESSING}>:ENABLE_GAPI_PREPROCESSING>
DATA_PATH=\"${DATA_PATH}\"
MODELS_PATH=\"${MODELS_PATH}\")
$<$<BOOL:${ENABLE_GAPI_PREPROCESSING}>:ENABLE_GAPI_PREPROCESSING>)
if(ENABLE_AUTO OR ENABLE_MULTI)
add_dependencies(${TARGET_NAME} openvino_auto_plugin)
@ -55,11 +53,6 @@ target_link_libraries(${TARGET_NAME} PRIVATE openvino_c openvino::util
target_include_directories(${TARGET_NAME} PUBLIC
$<BUILD_INTERFACE:${OPENVINO_API_SOURCE_DIR}/include>)
target_compile_definitions(${TARGET_NAME}
PRIVATE
DATA_PATH=\"${DATA_PATH}\"
MODELS_PATH=\"${MODELS_PATH}\")
if(TARGET OpenCL::OpenCL)
target_link_libraries(${TARGET_NAME} PRIVATE OpenCL::OpenCL)
endif()

574
src/bindings/c/tests/ie_c_api_test.cpp
View File

File diff suppressed because it is too large Load Diff

74
src/bindings/c/tests/ov_compiled_model_test.cpp
View File

@ -6,18 +6,26 @@
namespace {
class ov_compiled_model : public ::testing::TestWithParam<std::string> {};
class ov_compiled_model_test : public ov_capi_test_base {
void SetUp() override {
ov_capi_test_base::SetUp();
}
INSTANTIATE_TEST_SUITE_P(device_name, ov_compiled_model, ::testing::Values("CPU"));
void TearDown() override {
ov_capi_test_base::TearDown();
}
};
TEST_P(ov_compiled_model, ov_compiled_model_inputs_size) {
INSTANTIATE_TEST_SUITE_P(device_name, ov_compiled_model_test, ::testing::Values("CPU"));
TEST_P(ov_compiled_model_test, ov_compiled_model_inputs_size) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -33,14 +41,14 @@ TEST_P(ov_compiled_model, ov_compiled_model_inputs_size) {
ov_core_free(core);
}
TEST_P(ov_compiled_model, ov_compiled_model_input) {
TEST_P(ov_compiled_model_test, ov_compiled_model_input) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -57,14 +65,14 @@ TEST_P(ov_compiled_model, ov_compiled_model_input) {
ov_core_free(core);
}
TEST_P(ov_compiled_model, ov_compiled_model_input_by_index) {
TEST_P(ov_compiled_model_test, ov_compiled_model_input_by_index) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -85,14 +93,14 @@ TEST_P(ov_compiled_model, ov_compiled_model_input_by_index) {
ov_core_free(core);
}
TEST_P(ov_compiled_model, ov_compiled_model_input_by_name) {
TEST_P(ov_compiled_model_test, ov_compiled_model_input_by_name) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -113,7 +121,7 @@ TEST_P(ov_compiled_model, ov_compiled_model_input_by_name) {
ov_core_free(core);
}
TEST_P(ov_compiled_model, set_and_get_property) {
TEST_P(ov_compiled_model_test, set_and_get_property) {
// It seems that all set_property() for CPU plugin are not implement in compiled_model.
auto device_name = "MULTI:GPU,CPU";
ov_core_t* core = nullptr;
@ -128,7 +136,7 @@ TEST_P(ov_compiled_model, set_and_get_property) {
}
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -152,14 +160,14 @@ TEST_P(ov_compiled_model, set_and_get_property) {
ov_core_free(core);
}
TEST_P(ov_compiled_model, get_property) {
TEST_P(ov_compiled_model_test, get_property) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -176,14 +184,14 @@ TEST_P(ov_compiled_model, get_property) {
ov_core_free(core);
}
TEST_P(ov_compiled_model, create_compiled_model_with_property) {
TEST_P(ov_compiled_model_test, create_compiled_model_with_property) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
const char* key = ov_property_key_hint_performance_mode;
@ -201,14 +209,14 @@ TEST_P(ov_compiled_model, create_compiled_model_with_property) {
ov_core_free(core);
}
TEST_P(ov_compiled_model, ov_compiled_model_outputs_size) {
TEST_P(ov_compiled_model_test, ov_compiled_model_outputs_size) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -224,14 +232,14 @@ TEST_P(ov_compiled_model, ov_compiled_model_outputs_size) {
ov_core_free(core);
}
TEST_P(ov_compiled_model, ov_compiled_model_output) {
TEST_P(ov_compiled_model_test, ov_compiled_model_output) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -248,14 +256,14 @@ TEST_P(ov_compiled_model, ov_compiled_model_output) {
ov_core_free(core);
}
TEST_P(ov_compiled_model, ov_compiled_model_output_by_index) {
TEST_P(ov_compiled_model_test, ov_compiled_model_output_by_index) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -276,14 +284,14 @@ TEST_P(ov_compiled_model, ov_compiled_model_output_by_index) {
ov_core_free(core);
}
TEST_P(ov_compiled_model, ov_compiled_model_output_by_name) {
TEST_P(ov_compiled_model_test, ov_compiled_model_output_by_name) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -291,7 +299,7 @@ TEST_P(ov_compiled_model, ov_compiled_model_output_by_name) {
EXPECT_NE(nullptr, compiled_model);
ov_output_const_port_t* output_port = nullptr;
OV_EXPECT_OK(ov_compiled_model_output_by_name(compiled_model, "fc_out", &output_port));
OV_EXPECT_OK(ov_compiled_model_output_by_name(compiled_model, "relu", &output_port));
EXPECT_NE(nullptr, output_port);
ov_shape_t shape;
@ -304,14 +312,14 @@ TEST_P(ov_compiled_model, ov_compiled_model_output_by_name) {
ov_core_free(core);
}
TEST_P(ov_compiled_model, get_runtime_model) {
TEST_P(ov_compiled_model_test, get_runtime_model) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -328,14 +336,14 @@ TEST_P(ov_compiled_model, get_runtime_model) {
ov_core_free(core);
}
TEST_P(ov_compiled_model, get_runtime_model_error_handling) {
TEST_P(ov_compiled_model_test, get_runtime_model_error_handling) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -352,14 +360,14 @@ TEST_P(ov_compiled_model, get_runtime_model_error_handling) {
ov_core_free(core);
}
TEST_P(ov_compiled_model, create_infer_request) {
TEST_P(ov_compiled_model_test, create_infer_request) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -376,14 +384,14 @@ TEST_P(ov_compiled_model, create_infer_request) {
ov_core_free(core);
}
TEST_P(ov_compiled_model, create_infer_request_error_handling) {
TEST_P(ov_compiled_model_test, create_infer_request_error_handling) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;

111
src/bindings/c/tests/ov_core_test.cpp
View File

@ -22,10 +22,19 @@ TEST(ov_util, ov_get_error_info_check) {
EXPECT_STREQ(res, str);
}
class ov_core : public ::testing::TestWithParam<std::string> {};
INSTANTIATE_TEST_SUITE_P(device_name, ov_core, ::testing::Values("CPU"));
class ov_core_test : public ov_capi_test_base {
public:
void SetUp() override {
ov_capi_test_base::SetUp();
}
TEST(ov_core, ov_core_create_with_config) {
void TearDown() override {
ov_capi_test_base::TearDown();
}
};
INSTANTIATE_TEST_SUITE_P(device_name, ov_core_test, ::testing::Values("CPU"));
TEST_P(ov_core_test, ov_core_create_with_config) {
std::string plugins_xml = TestDataHelpers::generate_test_xml_file();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create_with_config(plugins_xml.c_str(), &core));
@ -34,45 +43,45 @@ TEST(ov_core, ov_core_create_with_config) {
TestDataHelpers::delete_test_xml_file();
}
TEST(ov_core, ov_core_create_with_no_config) {
TEST_P(ov_core_test, ov_core_create_with_no_config) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_core_free(core);
}
TEST(ov_core, ov_core_read_model) {
TEST_P(ov_core_test, ov_core_read_model) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_model_free(model);
ov_core_free(core);
}
TEST(ov_core, ov_core_read_model_no_bin) {
TEST_P(ov_core_test, ov_core_read_model_no_bin) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, nullptr, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), nullptr, &model));
EXPECT_NE(nullptr, model);
ov_model_free(model);
ov_core_free(core);
}
TEST(ov_core, ov_core_read_model_from_memory) {
TEST_P(ov_core_test, ov_core_read_model_from_memory) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
std::vector<uint8_t> weights_content(content_from_file(bin, true));
std::vector<uint8_t> weights_content(content_from_file(bin_file_name.c_str(), true));
ov_tensor_t* tensor = nullptr;
ov_shape_t shape;
@ -81,7 +90,7 @@ TEST(ov_core, ov_core_read_model_from_memory) {
OV_EXPECT_OK(ov_tensor_create_from_host_ptr(ov_element_type_e::U8, shape, weights_content.data(), &tensor));
EXPECT_NE(nullptr, tensor);
std::vector<uint8_t> xml_content(content_from_file(xml, false));
std::vector<uint8_t> xml_content(content_from_file(xml_file_name.c_str(), false));
ov_model_t* model = nullptr;
OV_EXPECT_OK(
ov_core_read_model_from_memory(core, reinterpret_cast<const char*>(xml_content.data()), tensor, &model));
@ -93,14 +102,14 @@ TEST(ov_core, ov_core_read_model_from_memory) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_compile_model) {
TEST_P(ov_core_test, ov_core_compile_model) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, nullptr, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), nullptr, &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -112,14 +121,14 @@ TEST_P(ov_core, ov_core_compile_model) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_compile_model_with_property) {
TEST_P(ov_core_test, ov_core_compile_model_with_property) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, nullptr, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), nullptr, &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -138,14 +147,14 @@ TEST_P(ov_core, ov_core_compile_model_with_property) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_compile_model_with_property_invalid) {
TEST_P(ov_core_test, ov_core_compile_model_with_property_invalid) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, nullptr, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), nullptr, &model));
EXPECT_NE(nullptr, model);
ov_compiled_model_t* compiled_model = nullptr;
@ -159,21 +168,21 @@ TEST_P(ov_core, ov_core_compile_model_with_property_invalid) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_compile_model_from_file) {
TEST_P(ov_core_test, ov_core_compile_model_from_file) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_compiled_model_t* compiled_model = nullptr;
OV_EXPECT_OK(ov_core_compile_model_from_file(core, xml, device_name.c_str(), 0, &compiled_model));
OV_EXPECT_OK(ov_core_compile_model_from_file(core, xml_file_name.c_str(), device_name.c_str(), 0, &compiled_model));
EXPECT_NE(nullptr, compiled_model);
ov_compiled_model_free(compiled_model);
ov_core_free(core);
}
TEST_P(ov_core, ov_core_set_property_enum) {
TEST_P(ov_core_test, ov_core_set_property_enum) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
@ -186,7 +195,7 @@ TEST_P(ov_core, ov_core_set_property_enum) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_set_property_invalid_number_property_arguments) {
TEST_P(ov_core_test, ov_core_set_property_invalid_number_property_arguments) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
@ -209,7 +218,7 @@ TEST_P(ov_core, ov_core_set_property_invalid_number_property_arguments) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_set_property_enum_invalid) {
TEST_P(ov_core_test, ov_core_set_property_enum_invalid) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
@ -232,7 +241,7 @@ TEST_P(ov_core, ov_core_set_property_enum_invalid) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_set_and_get_property_enum) {
TEST_P(ov_core_test, ov_core_set_and_get_property_enum) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
@ -249,7 +258,7 @@ TEST_P(ov_core, ov_core_set_and_get_property_enum) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_set_and_get_property_bool) {
TEST_P(ov_core_test, ov_core_set_and_get_property_bool) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
@ -266,7 +275,7 @@ TEST_P(ov_core, ov_core_set_and_get_property_bool) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_set_and_get_property_bool_invalid) {
TEST_P(ov_core_test, ov_core_set_and_get_property_bool_invalid) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
@ -284,7 +293,7 @@ TEST_P(ov_core, ov_core_set_and_get_property_bool_invalid) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_get_property) {
TEST_P(ov_core_test, ov_core_get_property) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
@ -297,7 +306,7 @@ TEST_P(ov_core, ov_core_get_property) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_set_get_property_str) {
TEST_P(ov_core_test, ov_core_set_get_property_str) {
#ifdef __aarch64__
GTEST_SKIP() << "Skip this test for ARM CPU for now, cause no string property supported";
#endif
@ -319,7 +328,7 @@ TEST_P(ov_core, ov_core_set_get_property_str) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_set_get_property_int) {
TEST_P(ov_core_test, ov_core_set_get_property_int) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
@ -337,7 +346,7 @@ TEST_P(ov_core, ov_core_set_get_property_int) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_set_property_int_invalid) {
TEST_P(ov_core_test, ov_core_set_property_int_invalid) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
@ -353,7 +362,7 @@ TEST_P(ov_core, ov_core_set_property_int_invalid) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_set_multiple_common_properties) {
TEST_P(ov_core_test, ov_core_set_multiple_common_properties) {
#ifdef __aarch64__
GTEST_SKIP() << "Skip this test for ARM CPU for now, cause no string property supported";
#endif
@ -404,7 +413,7 @@ TEST_P(ov_core, ov_core_set_multiple_common_properties) {
ov_core_free(core);
}
TEST(ov_core, ov_core_get_available_devices) {
TEST_P(ov_core_test, ov_core_get_available_devices) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
@ -416,7 +425,7 @@ TEST(ov_core, ov_core_get_available_devices) {
ov_core_free(core);
}
TEST_P(ov_core, ov_compiled_model_export_model) {
TEST_P(ov_core_test, ov_compiled_model_export_model) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
@ -429,17 +438,17 @@ TEST_P(ov_core, ov_compiled_model_export_model) {
}
ov_compiled_model_t* compiled_model = nullptr;
OV_EXPECT_OK(ov_core_compile_model_from_file(core, xml, device_name.c_str(), 0, &compiled_model));
OV_EXPECT_OK(ov_core_compile_model_from_file(core, xml_file_name.c_str(), device_name.c_str(), 0, &compiled_model));
EXPECT_NE(nullptr, compiled_model);
std::string export_path = TestDataHelpers::generate_model_path("test_model", "exported_model.blob");
std::string export_path = TestDataHelpers::get_exported_blob_file_name();
OV_EXPECT_OK(ov_compiled_model_export_model(compiled_model, export_path.c_str()));
ov_compiled_model_free(compiled_model);
ov_core_free(core);
}
TEST_P(ov_core, ov_core_import_model) {
TEST_P(ov_core_test, ov_core_import_model) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
@ -453,10 +462,10 @@ TEST_P(ov_core, ov_core_import_model) {
}
ov_compiled_model_t* compiled_model = nullptr;
OV_EXPECT_OK(ov_core_compile_model_from_file(core, xml, device_name.c_str(), 0, &compiled_model));
OV_EXPECT_OK(ov_core_compile_model_from_file(core, xml_file_name.c_str(), device_name.c_str(), 0, &compiled_model));
EXPECT_NE(nullptr, compiled_model);
std::string export_path = TestDataHelpers::generate_model_path("test_model", "exported_model.blob");
std::string export_path = TestDataHelpers::get_exported_blob_file_name();
OV_EXPECT_OK(ov_compiled_model_export_model(compiled_model, export_path.c_str()));
ov_compiled_model_free(compiled_model);
@ -472,7 +481,7 @@ TEST_P(ov_core, ov_core_import_model) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_get_versions_by_device_name) {
TEST_P(ov_core_test, ov_core_get_versions_by_device_name) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
@ -495,14 +504,14 @@ const std::vector<std::wstring> test_unicode_postfix_vector = {L"unicode_Яㅎ
L"그것이정당하다",
L"АБВГДЕЁЖЗИЙ",
L"СТУФХЦЧШЩЬЮЯ"};
TEST(ov_core, ov_core_create_with_config_unicode) {
TEST_P(ov_core_test, ov_core_create_with_config_unicode) {
std::string plugins_xml = TestDataHelpers::generate_test_xml_file();
ov_core_t* core = nullptr;
for (std::size_t index = 0; index < test_unicode_postfix_vector.size(); index++) {
std::wstring postfix = L"_" + test_unicode_postfix_vector[index];
std::wstring plugins_xml_ws = add_unicode_postfix_to_path(plugins_xml, postfix);
ASSERT_EQ(true, copy_file(plugins_xml, plugins_xml_ws));
std::wstring plugins_xml_ws = add_unicode_postfix_to_path(plugins_xml.c_str(), postfix);
ASSERT_EQ(true, copy_file(plugins_xml.c_str(), plugins_xml_ws));
OV_EXPECT_OK(ov_core_create_with_config_unicode(plugins_xml_ws.c_str(), &core));
EXPECT_NE(nullptr, core);
@ -512,7 +521,7 @@ TEST(ov_core, ov_core_create_with_config_unicode) {
TestDataHelpers::delete_test_xml_file();
}
TEST(ov_core, ov_core_read_model_unicode) {
TEST_P(ov_core_test, ov_core_read_model_unicode) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
@ -520,11 +529,11 @@ TEST(ov_core, ov_core_read_model_unicode) {
ov_model_t* model = nullptr;
for (std::size_t index = 0; index < test_unicode_postfix_vector.size(); index++) {
std::wstring postfix = L"_" + test_unicode_postfix_vector[index];
std::wstring xml_ws = add_unicode_postfix_to_path(xml, postfix);
std::wstring bin_ws = add_unicode_postfix_to_path(bin, postfix);
std::wstring xml_ws = add_unicode_postfix_to_path(xml_file_name.c_str(), postfix);
std::wstring bin_ws = add_unicode_postfix_to_path(bin_file_name.c_str(), postfix);
ASSERT_EQ(true, copy_file(xml, xml_ws));
ASSERT_EQ(true, copy_file(bin, bin_ws));
ASSERT_EQ(true, copy_file(xml_file_name.c_str(), xml_ws));
ASSERT_EQ(true, copy_file(bin_file_name.c_str(), bin_ws));
OV_EXPECT_OK(ov_core_read_model_unicode(core, xml_ws.c_str(), bin_ws.c_str(), &model));
EXPECT_NE(nullptr, model);
@ -537,7 +546,7 @@ TEST(ov_core, ov_core_read_model_unicode) {
ov_core_free(core);
}
TEST_P(ov_core, ov_core_compile_model_from_file_unicode) {
TEST_P(ov_core_test, ov_core_compile_model_from_file_unicode) {
auto device_name = GetParam();
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
@ -546,10 +555,10 @@ TEST_P(ov_core, ov_core_compile_model_from_file_unicode) {
ov_compiled_model_t* compiled_model = nullptr;
for (std::size_t index = 0; index < test_unicode_postfix_vector.size(); index++) {
std::wstring postfix = L"_" + test_unicode_postfix_vector[index];
std::wstring xml_ws = add_unicode_postfix_to_path(xml, postfix);
std::wstring bin_ws = add_unicode_postfix_to_path(bin, postfix);
ASSERT_EQ(true, copy_file(xml, xml_ws));
ASSERT_EQ(true, copy_file(bin, bin_ws));
std::wstring xml_ws = add_unicode_postfix_to_path(xml_file_name.c_str(), postfix);
std::wstring bin_ws = add_unicode_postfix_to_path(bin_file_name.c_str(), postfix);
ASSERT_EQ(true, copy_file(xml_file_name.c_str(), xml_ws));
ASSERT_EQ(true, copy_file(bin_file_name.c_str(), bin_ws));
OV_EXPECT_OK(
ov_core_compile_model_from_file_unicode(core, xml_ws.c_str(), device_name.c_str(), 0, &compiled_model));

78
src/bindings/c/tests/ov_infer_request_test.cpp
View File

@ -30,7 +30,7 @@ inline void get_tensor_info(ov_model_t* model, bool input, char** name, ov_shape
ov_output_const_port_free(port);
}
class ov_infer_request : public ::testing::TestWithParam<std::string> {
class ov_infer_request_test : public ov_capi_test_base {
protected:
void SetUp() override {
auto device_name = GetParam();
@ -43,11 +43,12 @@ protected:
infer_request = nullptr;
input_const_port = nullptr;
input_port = nullptr;
ov_capi_test_base::SetUp();
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
OV_EXPECT_OK(ov_model_const_input(model, &input_const_port));
@ -81,6 +82,7 @@ protected:
ov_compiled_model_free(compiled_model);
ov_model_free(model);
ov_core_free(core);
ov_capi_test_base::TearDown();
}
public:
@ -97,11 +99,11 @@ public:
static bool ready;
static std::condition_variable condVar;
};
bool ov_infer_request::ready = false;
std::mutex ov_infer_request::m;
std::condition_variable ov_infer_request::condVar;
bool ov_infer_request_test::ready = false;
std::mutex ov_infer_request_test::m;
std::condition_variable ov_infer_request_test::condVar;
class ov_infer_request_ppp : public ::testing::TestWithParam<std::string> {
class ov_infer_request_ppp : public ov_capi_test_base {
protected:
void SetUp() override {
auto device_name = GetParam();
@ -118,11 +120,12 @@ protected:
ov_layout_t* model_layout = nullptr;
compiled_model = nullptr;
infer_request = nullptr;
ov_capi_test_base::SetUp();
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
@ -182,6 +185,7 @@ protected:
ov_preprocess_prepostprocessor_free(preprocess);
ov_model_free(model);
ov_core_free(core);
ov_capi_test_base::TearDown();
}
public:
@ -198,83 +202,83 @@ public:
ov_preprocess_input_model_info_t* input_model;
};
INSTANTIATE_TEST_SUITE_P(device_name, ov_infer_request, ::testing::Values("CPU"));
INSTANTIATE_TEST_SUITE_P(device_name, ov_infer_request_test, ::testing::Values("CPU"));
INSTANTIATE_TEST_SUITE_P(device_name, ov_infer_request_ppp, ::testing::Values("CPU"));
TEST_P(ov_infer_request, set_tensor) {
TEST_P(ov_infer_request_test, set_tensor) {
OV_EXPECT_OK(ov_infer_request_set_tensor(infer_request, in_tensor_name, input_tensor));
}
TEST_P(ov_infer_request, set_input_tensor_by_index) {
TEST_P(ov_infer_request_test, set_input_tensor_by_index) {
OV_EXPECT_OK(ov_infer_request_set_input_tensor_by_index(infer_request, 0, input_tensor));
}
TEST_P(ov_infer_request, set_tensor_by_port) {
TEST_P(ov_infer_request_test, set_tensor_by_port) {
OV_EXPECT_OK(ov_infer_request_set_tensor_by_port(infer_request, input_port, input_tensor));
}
TEST_P(ov_infer_request, set_tensor_by_const_port) {
TEST_P(ov_infer_request_test, set_tensor_by_const_port) {
OV_EXPECT_OK(ov_infer_request_set_tensor_by_const_port(infer_request, input_const_port, input_tensor));
}
TEST_P(ov_infer_request, set_input_tensor) {
TEST_P(ov_infer_request_test, set_input_tensor) {
OV_EXPECT_OK(ov_infer_request_set_input_tensor(infer_request, input_tensor));
}
TEST_P(ov_infer_request, set_output_tensor_by_index) {
TEST_P(ov_infer_request_test, set_output_tensor_by_index) {
OV_EXPECT_OK(ov_infer_request_get_output_tensor_by_index(infer_request, 0, &output_tensor));
EXPECT_NE(nullptr, output_tensor);
OV_EXPECT_OK(ov_infer_request_set_output_tensor_by_index(infer_request, 0, output_tensor));
}
TEST_P(ov_infer_request, set_output_tensor) {
TEST_P(ov_infer_request_test, set_output_tensor) {
OV_EXPECT_OK(ov_infer_request_get_output_tensor_by_index(infer_request, 0, &output_tensor));
EXPECT_NE(nullptr, output_tensor);
OV_EXPECT_OK(ov_infer_request_set_output_tensor(infer_request, output_tensor));
}
TEST_P(ov_infer_request, set_tensor_error_handling) {
TEST_P(ov_infer_request_test, set_tensor_error_handling) {
OV_EXPECT_NOT_OK(ov_infer_request_set_tensor(nullptr, in_tensor_name, input_tensor));
OV_EXPECT_NOT_OK(ov_infer_request_set_tensor(infer_request, nullptr, input_tensor));
OV_EXPECT_NOT_OK(ov_infer_request_set_tensor(infer_request, in_tensor_name, nullptr));
}
TEST_P(ov_infer_request, get_tensor) {
TEST_P(ov_infer_request_test, get_tensor) {
OV_EXPECT_OK(ov_infer_request_get_tensor(infer_request, in_tensor_name, &input_tensor));
EXPECT_NE(nullptr, input_tensor);
}
TEST_P(ov_infer_request, get_input_tensor_by_index) {
TEST_P(ov_infer_request_test, get_input_tensor_by_index) {
OV_EXPECT_OK(ov_infer_request_get_input_tensor_by_index(infer_request, 0, &output_tensor));
}
TEST_P(ov_infer_request, get_tensor_by_const_port) {
TEST_P(ov_infer_request_test, get_tensor_by_const_port) {
OV_EXPECT_OK(ov_infer_request_get_tensor_by_const_port(infer_request, input_const_port, &output_tensor));
}
TEST_P(ov_infer_request, get_tensor_by_port) {
TEST_P(ov_infer_request_test, get_tensor_by_port) {
OV_EXPECT_OK(ov_infer_request_get_tensor_by_port(infer_request, input_port, &output_tensor));
}
TEST_P(ov_infer_request, get_input_tensor) {
TEST_P(ov_infer_request_test, get_input_tensor) {
OV_EXPECT_OK(ov_infer_request_get_input_tensor(infer_request, &output_tensor));
}
TEST_P(ov_infer_request, get_output_tensor_by_index) {
TEST_P(ov_infer_request_test, get_output_tensor_by_index) {
OV_EXPECT_OK(ov_infer_request_get_output_tensor_by_index(infer_request, 0, &output_tensor));
}
TEST_P(ov_infer_request, get_output_tensor) {
TEST_P(ov_infer_request_test, get_output_tensor) {
OV_EXPECT_OK(ov_infer_request_get_output_tensor(infer_request, &output_tensor));
}
TEST_P(ov_infer_request, get_tensor_error_handling) {
TEST_P(ov_infer_request_test, get_tensor_error_handling) {
OV_EXPECT_NOT_OK(ov_infer_request_get_tensor(nullptr, in_tensor_name, &input_tensor));
OV_EXPECT_NOT_OK(ov_infer_request_get_tensor(infer_request, nullptr, &input_tensor));
OV_EXPECT_NOT_OK(ov_infer_request_get_tensor(infer_request, in_tensor_name, nullptr));
}
TEST_P(ov_infer_request, infer) {
TEST_P(ov_infer_request_test, infer) {
OV_EXPECT_OK(ov_infer_request_set_tensor(infer_request, in_tensor_name, input_tensor));
OV_ASSERT_OK(ov_infer_request_infer(infer_request));
@ -291,7 +295,7 @@ TEST_P(ov_infer_request, infer) {
ov_free(out_tensor_name);
}
TEST_P(ov_infer_request, cancel) {
TEST_P(ov_infer_request_test, cancel) {
OV_EXPECT_OK(ov_infer_request_set_tensor(infer_request, in_tensor_name, input_tensor));
OV_EXPECT_OK(ov_infer_request_cancel(infer_request));
@ -306,11 +310,11 @@ TEST_P(ov_infer_request_ppp, infer_ppp) {
EXPECT_NE(nullptr, output_tensor);
}
TEST(ov_infer_request, infer_error_handling) {
TEST_P(ov_infer_request_test, infer_error_handling) {
OV_EXPECT_NOT_OK(ov_infer_request_infer(nullptr));
}
TEST_P(ov_infer_request, infer_async) {
TEST_P(ov_infer_request_test, infer_async) {
OV_EXPECT_OK(ov_infer_request_set_input_tensor_by_index(infer_request, 0, input_tensor));
OV_ASSERT_OK(ov_infer_request_start_async(infer_request));
@ -323,7 +327,7 @@ TEST_P(ov_infer_request, infer_async) {
}
}
TEST_P(ov_infer_request, infer_async_wait_for) {
TEST_P(ov_infer_request_test, infer_async_wait_for) {
OV_EXPECT_OK(ov_infer_request_set_input_tensor_by_index(infer_request, 0, input_tensor));
OV_ASSERT_OK(ov_infer_request_start_async(infer_request));
@ -358,12 +362,12 @@ inline void infer_request_callback(void* args) {
ov_tensor_free(out_tensor);
std::lock_guard<std::mutex> lock(ov_infer_request::m);
ov_infer_request::ready = true;
ov_infer_request::condVar.notify_one();
std::lock_guard<std::mutex> lock(ov_infer_request_test::m);
ov_infer_request_test::ready = true;
ov_infer_request_test::condVar.notify_one();
}
TEST_P(ov_infer_request, infer_request_set_callback) {
TEST_P(ov_infer_request_test, infer_request_set_callback) {
OV_EXPECT_OK(ov_infer_request_set_input_tensor_by_index(infer_request, 0, input_tensor));
ov_callback_t callback;
@ -375,14 +379,14 @@ TEST_P(ov_infer_request, infer_request_set_callback) {
OV_ASSERT_OK(ov_infer_request_start_async(infer_request));
if (!HasFatalFailure()) {
std::unique_lock<std::mutex> lock(ov_infer_request::m);
ov_infer_request::condVar.wait(lock, [] {
return ov_infer_request::ready;
std::unique_lock<std::mutex> lock(ov_infer_request_test::m);
ov_infer_request_test::condVar.wait(lock, [] {
return ov_infer_request_test::ready;
});
}
}
TEST_P(ov_infer_request, get_profiling_info) {
TEST_P(ov_infer_request_test, get_profiling_info) {
auto device_name = GetParam();
OV_EXPECT_OK(ov_infer_request_set_tensor(infer_request, in_tensor_name, input_tensor));

100
src/bindings/c/tests/ov_model_test.cpp
View File

@ -3,13 +3,25 @@
//
#include "ov_test.hpp"
TEST(ov_model, ov_model_const_input) {
class ov_model_test : public ov_capi_test_base {
void SetUp() override {
ov_capi_test_base::SetUp();
}
void TearDown() override {
ov_capi_test_base::TearDown();
}
};
INSTANTIATE_TEST_SUITE_P(device_name, ov_model_test, ::testing::Values(""));
TEST_P(ov_model_test, ov_model_const_input) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_output_const_port_t* input_port = nullptr;
@ -21,13 +33,13 @@ TEST(ov_model, ov_model_const_input) {
ov_core_free(core);
}
TEST(ov_model, ov_model_const_input_by_name) {
TEST_P(ov_model_test, ov_model_const_input_by_name) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_output_const_port_t* input_port = nullptr;
@ -43,13 +55,13 @@ TEST(ov_model, ov_model_const_input_by_name) {
ov_core_free(core);
}
TEST(ov_model, ov_model_const_input_by_index) {
TEST_P(ov_model_test, ov_model_const_input_by_index) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_output_const_port_t* input_port = nullptr;
@ -65,13 +77,13 @@ TEST(ov_model, ov_model_const_input_by_index) {
ov_core_free(core);
}
TEST(ov_model, ov_model_input) {
TEST_P(ov_model_test, ov_model_input) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_output_port_t* input_port = nullptr;
@ -83,13 +95,13 @@ TEST(ov_model, ov_model_input) {
ov_core_free(core);
}
TEST(ov_model, ov_model_input_by_name) {
TEST_P(ov_model_test, ov_model_input_by_name) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_output_port_t* input_port = nullptr;
@ -105,13 +117,13 @@ TEST(ov_model, ov_model_input_by_name) {
ov_core_free(core);
}
TEST(ov_model, ov_model_input_by_index) {
TEST_P(ov_model_test, ov_model_input_by_index) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_output_port_t* input_port = nullptr;
@ -127,13 +139,13 @@ TEST(ov_model, ov_model_input_by_index) {
ov_core_free(core);
}
TEST(ov_model, ov_model_const_output) {
TEST_P(ov_model_test, ov_model_const_output) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_output_const_port_t* output_port = nullptr;
@ -145,13 +157,13 @@ TEST(ov_model, ov_model_const_output) {
ov_core_free(core);
}
TEST(ov_model, ov_model_const_output_by_index) {
TEST_P(ov_model_test, ov_model_const_output_by_index) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_output_const_port_t* output_port = nullptr;
@ -167,17 +179,17 @@ TEST(ov_model, ov_model_const_output_by_index) {
ov_core_free(core);
}
TEST(ov_model, ov_model_const_output_by_name) {
TEST_P(ov_model_test, ov_model_const_output_by_name) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_output_const_port_t* output_port = nullptr;
OV_EXPECT_OK(ov_model_const_output_by_name(model, "fc_out", &output_port));
OV_EXPECT_OK(ov_model_const_output_by_name(model, "relu", &output_port));
EXPECT_NE(nullptr, output_port);
ov_shape_t shape;
@ -189,13 +201,13 @@ TEST(ov_model, ov_model_const_output_by_name) {
ov_core_free(core);
}
TEST(ov_model, ov_model_output) {
TEST_P(ov_model_test, ov_model_output) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_output_port_t* output_port = nullptr;
@ -207,13 +219,13 @@ TEST(ov_model, ov_model_output) {
ov_core_free(core);
}
TEST(ov_model, ov_model_output_by_index) {
TEST_P(ov_model_test, ov_model_output_by_index) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_output_port_t* output_port = nullptr;
@ -229,17 +241,17 @@ TEST(ov_model, ov_model_output_by_index) {
ov_core_free(core);
}
TEST(ov_model, ov_model_output_by_name) {
TEST_P(ov_model_test, ov_model_output_by_name) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_output_port_t* output_port = nullptr;
OV_EXPECT_OK(ov_model_output_by_name(model, "fc_out", &output_port));
OV_EXPECT_OK(ov_model_output_by_name(model, "relu", &output_port));
EXPECT_NE(nullptr, output_port);
ov_shape_t shape;
@ -251,13 +263,13 @@ TEST(ov_model, ov_model_output_by_name) {
ov_core_free(core);
}
TEST(ov_model, ov_model_inputs_size) {
TEST_P(ov_model_test, ov_model_inputs_size) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
size_t input_size;
@ -268,13 +280,13 @@ TEST(ov_model, ov_model_inputs_size) {
ov_core_free(core);
}
TEST(ov_model, ov_model_outputs_size) {
TEST_P(ov_model_test, ov_model_outputs_size) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
size_t output_size;
@ -285,13 +297,13 @@ TEST(ov_model, ov_model_outputs_size) {
ov_core_free(core);
}
TEST(ov_model, ov_model_is_dynamic) {
TEST_P(ov_model_test, ov_model_is_dynamic) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
EXPECT_NO_THROW(ov_model_is_dynamic(model));
@ -300,13 +312,13 @@ TEST(ov_model, ov_model_is_dynamic) {
ov_core_free(core);
}
TEST(ov_model, ov_model_reshape_input_by_name) {
TEST_P(ov_model_test, ov_model_reshape_input_by_name) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_output_const_port_t* input_port_1 = nullptr;
@ -339,13 +351,13 @@ TEST(ov_model, ov_model_reshape_input_by_name) {
ov_core_free(core);
}
TEST(ov_model, ov_model_reshape) {
TEST_P(ov_model_test, ov_model_reshape) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_output_const_port_t* input_port_1 = nullptr;
@ -379,13 +391,13 @@ TEST(ov_model, ov_model_reshape) {
ov_core_free(core);
}
TEST(ov_model, ov_model_reshape_by_port_indexes) {
TEST_P(ov_model_test, ov_model_reshape_by_port_indexes) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
size_t port_indexs[] = {0};
@ -409,13 +421,13 @@ TEST(ov_model, ov_model_reshape_by_port_indexes) {
ov_core_free(core);
}
TEST(ov_model, ov_model_reshape_single_input) {
TEST_P(ov_model_test, ov_model_reshape_single_input) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_shape_t shape = {0, nullptr};
@ -437,13 +449,13 @@ TEST(ov_model, ov_model_reshape_single_input) {
ov_core_free(core);
}
TEST(ov_model, ov_model_reshape_by_ports) {
TEST_P(ov_model_test, ov_model_reshape_by_ports) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
ov_output_port_t* input_port_1 = nullptr;
@ -471,13 +483,13 @@ TEST(ov_model, ov_model_reshape_by_ports) {
ov_core_free(core);
}
TEST(ov_model, ov_model_get_friendly_name) {
TEST_P(ov_model_test, ov_model_get_friendly_name) {
ov_core_t* core = nullptr;
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
ov_model_t* model = nullptr;
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
char* friendly_name = nullptr;

72
src/bindings/c/tests/ov_preprocess_test.cpp
View File

@ -3,7 +3,7 @@
//
#include "ov_test.hpp"
class ov_preprocess : public ::testing::Test {
class ov_preprocess_test : public ::testing::Test {
protected:
void SetUp() override {
core = nullptr;
@ -17,10 +17,14 @@ protected:
output_tensor_info = nullptr;
input_model = nullptr;
TestDataHelpers::generate_test_model();
xml_file_name = TestDataHelpers::get_model_xml_file_name();
bin_file_name = TestDataHelpers::get_model_bin_file_name();
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
}
void TearDown() override {
@ -34,6 +38,7 @@ protected:
ov_preprocess_prepostprocessor_free(preprocess);
ov_model_free(model);
ov_core_free(core);
TestDataHelpers::release_test_model();
}
public:
@ -47,14 +52,15 @@ public:
ov_preprocess_output_info_t* output_info;
ov_preprocess_output_tensor_info_t* output_tensor_info;
ov_preprocess_input_model_info_t* input_model;
std::string xml_file_name, bin_file_name;
};
TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_create) {
TEST_F(ov_preprocess_test, ov_preprocess_prepostprocessor_create) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
}
TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_get_input_info) {
TEST_F(ov_preprocess_test, ov_preprocess_prepostprocessor_get_input_info) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -62,7 +68,7 @@ TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_get_input_info) {
EXPECT_NE(nullptr, input_info);
}
TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_get_input_info_by_name) {
TEST_F(ov_preprocess_test, ov_preprocess_prepostprocessor_get_input_info_by_name) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -70,7 +76,7 @@ TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_get_input_info_by_name) {
EXPECT_NE(nullptr, input_info);
}
TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_get_input_info_by_index) {
TEST_F(ov_preprocess_test, ov_preprocess_prepostprocessor_get_input_info_by_index) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -78,7 +84,7 @@ TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_get_input_info_by_index) {
EXPECT_NE(nullptr, input_info);
}
TEST_F(ov_preprocess, ov_preprocess_input_info_get_tensor_info) {
TEST_F(ov_preprocess_test, ov_preprocess_input_info_get_tensor_info) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -89,7 +95,7 @@ TEST_F(ov_preprocess, ov_preprocess_input_info_get_tensor_info) {
EXPECT_NE(nullptr, input_tensor_info);
}
TEST_F(ov_preprocess, ov_preprocess_input_info_get_preprocess_steps) {
TEST_F(ov_preprocess_test, ov_preprocess_input_info_get_preprocess_steps) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -100,7 +106,7 @@ TEST_F(ov_preprocess, ov_preprocess_input_info_get_preprocess_steps) {
EXPECT_NE(nullptr, input_process);
}
TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_resize) {
TEST_F(ov_preprocess_test, ov_preprocess_preprocess_steps_resize) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -113,7 +119,7 @@ TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_resize) {
OV_EXPECT_OK(ov_preprocess_preprocess_steps_resize(input_process, ov_preprocess_resize_algorithm_e::RESIZE_LINEAR));
}
TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_scale) {
TEST_F(ov_preprocess_test, ov_preprocess_preprocess_steps_scale) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -126,7 +132,7 @@ TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_scale) {
OV_EXPECT_OK(ov_preprocess_preprocess_steps_scale(input_process, 2.0f));
}
TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_mean) {
TEST_F(ov_preprocess_test, ov_preprocess_preprocess_steps_mean) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -139,7 +145,7 @@ TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_mean) {
OV_EXPECT_OK(ov_preprocess_preprocess_steps_mean(input_process, 2.0f));
}
TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_crop) {
TEST_F(ov_preprocess_test, ov_preprocess_preprocess_steps_crop) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -154,7 +160,7 @@ TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_crop) {
OV_EXPECT_OK(ov_preprocess_preprocess_steps_crop(input_process, begin, 4, end, 4));
}
TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_convert_layout) {
TEST_F(ov_preprocess_test, ov_preprocess_preprocess_steps_convert_layout) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -172,7 +178,7 @@ TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_convert_layout) {
ov_layout_free(layout);
}
TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_reverse_channels) {
TEST_F(ov_preprocess_test, ov_preprocess_preprocess_steps_reverse_channels) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -185,7 +191,7 @@ TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_reverse_channels) {
OV_EXPECT_OK(ov_preprocess_preprocess_steps_reverse_channels(input_process));
}
TEST_F(ov_preprocess, ov_preprocess_input_tensor_info_set_element_type) {
TEST_F(ov_preprocess_test, ov_preprocess_input_tensor_info_set_element_type) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -198,7 +204,7 @@ TEST_F(ov_preprocess, ov_preprocess_input_tensor_info_set_element_type) {
OV_EXPECT_OK(ov_preprocess_input_tensor_info_set_element_type(input_tensor_info, ov_element_type_e::F32));
}
TEST_F(ov_preprocess, ov_preprocess_input_tensor_info_set_from) {
TEST_F(ov_preprocess_test, ov_preprocess_input_tensor_info_set_from) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -218,7 +224,7 @@ TEST_F(ov_preprocess, ov_preprocess_input_tensor_info_set_from) {
ov_shape_free(&shape);
}
TEST_F(ov_preprocess, ov_preprocess_input_tensor_info_set_layout) {
TEST_F(ov_preprocess_test, ov_preprocess_input_tensor_info_set_layout) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -236,7 +242,7 @@ TEST_F(ov_preprocess, ov_preprocess_input_tensor_info_set_layout) {
ov_layout_free(layout);
}
TEST_F(ov_preprocess, ov_preprocess_input_tensor_info_set_color_format) {
TEST_F(ov_preprocess_test, ov_preprocess_input_tensor_info_set_color_format) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -250,7 +256,7 @@ TEST_F(ov_preprocess, ov_preprocess_input_tensor_info_set_color_format) {
ov_preprocess_input_tensor_info_set_color_format(input_tensor_info, ov_color_format_e::NV12_SINGLE_PLANE));
}
TEST_F(ov_preprocess, ov_preprocess_input_tensor_info_set_spatial_static_shape) {
TEST_F(ov_preprocess_test, ov_preprocess_input_tensor_info_set_spatial_static_shape) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -266,7 +272,7 @@ TEST_F(ov_preprocess, ov_preprocess_input_tensor_info_set_spatial_static_shape)
ov_preprocess_input_tensor_info_set_spatial_static_shape(input_tensor_info, input_height, input_width));
}
TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_convert_element_type) {
TEST_F(ov_preprocess_test, ov_preprocess_preprocess_steps_convert_element_type) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -283,7 +289,7 @@ TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_convert_element_type) {
OV_EXPECT_OK(ov_preprocess_preprocess_steps_convert_element_type(input_process, ov_element_type_e::F32));
}
TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_convert_color) {
TEST_F(ov_preprocess_test, ov_preprocess_preprocess_steps_convert_color) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -304,7 +310,7 @@ TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_convert_color) {
OV_EXPECT_OK(ov_preprocess_preprocess_steps_convert_color(input_process, ov_color_format_e::BGR));
}
TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_convert_color_rgb_to_gray) {
TEST_F(ov_preprocess_test, ov_preprocess_preprocess_steps_convert_color_rgb_to_gray) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -321,7 +327,7 @@ TEST_F(ov_preprocess, ov_preprocess_preprocess_steps_convert_color_rgb_to_gray)
OV_EXPECT_OK(ov_preprocess_preprocess_steps_convert_color(input_process, ov_color_format_e::GRAY));
}
TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_get_output_info) {
TEST_F(ov_preprocess_test, ov_preprocess_prepostprocessor_get_output_info) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -329,7 +335,7 @@ TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_get_output_info) {
EXPECT_NE(nullptr, output_info);
}
TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_get_output_info_by_index) {
TEST_F(ov_preprocess_test, ov_preprocess_prepostprocessor_get_output_info_by_index) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -337,15 +343,15 @@ TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_get_output_info_by_index) {
EXPECT_NE(nullptr, output_info);
}
TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_get_output_info_by_name) {
TEST_F(ov_preprocess_test, ov_preprocess_prepostprocessor_get_output_info_by_name) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
OV_EXPECT_OK(ov_preprocess_prepostprocessor_get_output_info_by_name(preprocess, "fc_out", &output_info));
OV_EXPECT_OK(ov_preprocess_prepostprocessor_get_output_info_by_name(preprocess, "relu", &output_info));
EXPECT_NE(nullptr, output_info);
}
TEST_F(ov_preprocess, ov_preprocess_output_info_get_tensor_info) {
TEST_F(ov_preprocess_test, ov_preprocess_output_info_get_tensor_info) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -356,7 +362,7 @@ TEST_F(ov_preprocess, ov_preprocess_output_info_get_tensor_info) {
EXPECT_NE(nullptr, output_tensor_info);
}
TEST_F(ov_preprocess, ov_preprocess_output_set_element_type) {
TEST_F(ov_preprocess_test, ov_preprocess_output_set_element_type) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -369,7 +375,7 @@ TEST_F(ov_preprocess, ov_preprocess_output_set_element_type) {
OV_EXPECT_OK(ov_preprocess_output_set_element_type(output_tensor_info, ov_element_type_e::F32));
}
TEST_F(ov_preprocess, ov_preprocess_input_info_get_model_info) {
TEST_F(ov_preprocess_test, ov_preprocess_input_info_get_model_info) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -380,7 +386,7 @@ TEST_F(ov_preprocess, ov_preprocess_input_info_get_model_info) {
EXPECT_NE(nullptr, input_model);
}
TEST_F(ov_preprocess, ov_preprocess_input_model_info_set_layout) {
TEST_F(ov_preprocess_test, ov_preprocess_input_model_info_set_layout) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -398,7 +404,7 @@ TEST_F(ov_preprocess, ov_preprocess_input_model_info_set_layout) {
ov_layout_free(layout);
}
TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_build) {
TEST_F(ov_preprocess_test, ov_preprocess_prepostprocessor_build) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -409,7 +415,7 @@ TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_build) {
ov_model_free(new_model);
}
TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_build_apply) {
TEST_F(ov_preprocess_test, ov_preprocess_prepostprocessor_build_apply) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);
@ -459,7 +465,7 @@ TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_build_apply) {
ov_model_free(new_model);
}
TEST_F(ov_preprocess, ov_preprocess_prepostprocessor_for_nv12_input) {
TEST_F(ov_preprocess_test, ov_preprocess_prepostprocessor_for_nv12_input) {
OV_EXPECT_OK(ov_preprocess_prepostprocessor_create(model, &preprocess));
EXPECT_NE(nullptr, preprocess);

6
src/bindings/c/tests/ov_remote_context_test.cpp
View File

@ -5,7 +5,7 @@
#include "openvino/runtime/intel_gpu/ocl/ocl_wrapper.hpp"
#include "ov_test.hpp"
class ov_remote_context_ocl : public ::testing::TestWithParam<std::string> {
class ov_remote_context_ocl : public ov_capi_test_base {
protected:
void SetUp() override {
core = nullptr;
@ -16,11 +16,12 @@ protected:
remote_tensor = nullptr;
out_tensor_name = nullptr;
in_tensor_name = nullptr;
ov_capi_test_base::SetUp();
OV_EXPECT_OK(ov_core_create(&core));
EXPECT_NE(nullptr, core);
OV_EXPECT_OK(ov_core_read_model(core, xml, bin, &model));
OV_EXPECT_OK(ov_core_read_model(core, xml_file_name.c_str(), bin_file_name.c_str(), &model));
EXPECT_NE(nullptr, model);
char* info = nullptr;
@ -68,6 +69,7 @@ protected:
ov_free(in_tensor_name);
ov_remote_context_free(context);
ov_core_free(core);
ov_capi_test_base::TearDown();
}
public:

8
src/bindings/c/tests/ov_test.cpp
View File

@ -3,14 +3,6 @@
//
#include "ov_test.hpp"
#include "test_model_repo.hpp"
std::string xml_std = TestDataHelpers::generate_model_path("test_model", "test_model_fp32.xml");
std::string bin_std = TestDataHelpers::generate_model_path("test_model", "test_model_fp32.bin");
const char* xml = xml_std.c_str();
const char* bin = bin_std.c_str();
std::map<ov_element_type_e, size_t> element_type_size_map = {{ov_element_type_e::BOOLEAN, 8},
{ov_element_type_e::BF16, 16},
{ov_element_type_e::F16, 16},

22
src/bindings/c/tests/ov_test.hpp
View File

@ -11,11 +11,7 @@
#include "openvino/c/openvino.h"
#include "openvino/openvino.hpp"
extern const char* xml;
extern const char* bin;
extern const char* input_image;
extern const char* input_image_nv12;
#include "test_model_repo.hpp"
#define OV_EXPECT_OK(...) EXPECT_EQ(ov_status_e::OK, __VA_ARGS__)
#define OV_ASSERT_OK(...) ASSERT_EQ(ov_status_e::OK, __VA_ARGS__)
@ -40,6 +36,22 @@ extern const char* input_image_nv12;
extern std::map<ov_element_type_e, size_t> element_type_size_map;
#define GET_ELEMENT_TYPE_SIZE(a) element_type_size_map[a]
class ov_capi_test_base : public ::testing::TestWithParam<std::string> {
public:
void SetUp() override {
TestDataHelpers::generate_test_model();
xml_file_name = TestDataHelpers::get_model_xml_file_name();
bin_file_name = TestDataHelpers::get_model_bin_file_name();
}
void TearDown() override {
TestDataHelpers::release_test_model();
}
public:
std::string xml_file_name, bin_file_name;
};
inline size_t find_device(ov_available_devices_t avai_devices, const char* device_name) {
for (size_t i = 0; i < avai_devices.size; ++i) {
if (strstr(avai_devices.devices[i], device_name))

81
src/bindings/c/tests/test_model_repo.hpp
View File

@ -1,51 +1,54 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#pragma once
#include <fstream>
#include <random>
#include "ngraph_functions/builders.hpp"
#include "ngraph_functions/subgraph_builders.hpp"
#include "openvino/pass/manager.hpp"
namespace TestDataHelpers {
static const char kPathSeparator =
#if defined _WIN32 || defined __CYGWIN__
'\\';
#else
'/';
#endif
static const std::string model_bin_name = "test_model.bin";
static const std::string model_xml_name = "test_model.xml";
static const std::string model_exported_name = "test_exported_model.blob";
inline std::string getModelPathNonFatal() noexcept {
if (const auto envVar = std::getenv("MODELS_PATH")) {
return envVar;
inline void generate_test_model() {
ov::pass::Manager manager;
manager.register_pass<ov::pass::Serialize>(model_xml_name, model_bin_name);
auto function = ngraph::builder::subgraph::makeConvPoolReluNoReshapes({1, 3, 227, 227});
manager.run_passes(function);
}
inline std::string get_model_xml_file_name() {
return model_xml_name;
}
inline std::string get_model_bin_file_name() {
return model_bin_name;
}
inline std::string get_exported_blob_file_name() {
return model_exported_name;
}
inline void release_test_model() {
std::remove(model_xml_name.c_str());
std::remove(model_bin_name.c_str());
}
inline void fill_random_input_nv12_data(uint8_t* data, const size_t w, const size_t h) {
size_t size = w * h * 3 / 2;
std::mt19937 gen(0);
std::uniform_int_distribution<> distribution(0, 255);
for (size_t i = 0; i < size; i++) {
data[i] = static_cast<uint8_t>(distribution(gen));
}
#ifdef MODELS_PATH
return MODELS_PATH;
#else
return "";
#endif
}
inline std::string get_models_path() {
return getModelPathNonFatal() + kPathSeparator + std::string("models");
};
inline std::string get_data_path() {
if (const auto envVar = std::getenv("DATA_PATH")) {
return envVar;
}
#ifdef DATA_PATH
return DATA_PATH;
#else
return "";
#endif
}
inline std::string generate_model_path(std::string dir, std::string filename) {
return get_models_path() + kPathSeparator + dir + kPathSeparator + filename;
}
inline std::string generate_image_path(std::string dir, std::string filename) {
return get_data_path() + kPathSeparator + "validation_set" + kPathSeparator + dir + kPathSeparator + filename;
return;
}
inline std::string generate_test_xml_file() {

10
src/bindings/python/tests/test_runtime/test_tensor.py
View File

@ -241,7 +241,7 @@ def test_cannot_set_bigger_shape_on_preallocated_memory():
assert np.shares_memory(ones_arr, ov_tensor.data)
with pytest.raises(RuntimeError) as e:
ov_tensor.shape = ref_shape
assert "Cannot call setShape for Blobs created on top of preallocated memory" in str(e.value)
assert "failed" in str(e.value)
@pytest.mark.skip(reason="no support yet")
@ -258,11 +258,11 @@ def test_can_reset_shape_after_decreasing_on_preallocated_memory():
assert list(ov_tensor.shape) == ref_shape_2
def test_cannot_set_shape_incorrect_dims():
def test_can_set_shape_other_dims():
ov_tensor = Tensor(np.float32, [1, 3, 48, 48])
with pytest.raises(RuntimeError) as e:
ov_tensor.shape = [3, 28, 28]
assert "Dims and format are inconsistent" in str(e.value)
ref_shape_1 = [3, 28, 28]
ov_tensor.shape = ref_shape_1
assert list(ov_tensor.shape) == ref_shape_1
@pytest.mark.parametrize("ov_type", [

10
src/common/transformations/include/transformations/common_optimizations/transpose_sinking_binary.hpp
View File

@ -17,12 +17,22 @@ class TRANSFORMATIONS_API TransposeSinkingBinaryBackward;
} // namespace pass
} // namespace ov
/**
* @ingroup ie_transformation_common_api
* @brief TransposeSinkingBinaryForward transformation sinks Transpose through BinaryElementwiseArithmetic,
* BinaryElementwiseComparison, BinaryElementwiseLogical and PRelu operations in the forward direction.
*/
class ov::pass::TransposeSinkingBinaryForward : public ov::pass::MatcherPass {
public:
OPENVINO_RTTI("ov::pass::TransposeSinkingBinaryForward", "0");
TransposeSinkingBinaryForward();
};
/**
* @ingroup ie_transformation_common_api
* @brief TransposeSinkingBinaryBackward transformation sinks Transpose through BinaryElementwiseArithmetic,
* BinaryElementwiseComparison, BinaryElementwiseLogical and PRelu operations in the backward direction.
*/
class ov::pass::TransposeSinkingBinaryBackward : public ov::pass::MatcherPass {
public:
OPENVINO_RTTI("ov::pass::TransposeSinkingBinaryBackward", "0");

10
src/common/transformations/include/transformations/common_optimizations/transpose_sinking_concat.hpp
View File

@ -17,12 +17,22 @@ class TRANSFORMATIONS_API TransposeSinkingConcatBackward;
} // namespace pass
} // namespace ov
/**
* @ingroup ie_transformation_common_api
* @brief TransposeSinkingConcatForward transformation sinks Transpose through Concat operation
* in the forward direction.
*/
class ov::pass::TransposeSinkingConcatForward : public ov::pass::MatcherPass {
public:
OPENVINO_RTTI("ov::pass::TransposeSinkingConcatForward", "0");
TransposeSinkingConcatForward();
};
/**
* @ingroup ie_transformation_common_api
* @brief TransposeSinkingConcatBackward transformation sinks Transpose through Concat operation
* in the backward direction.
*/
class ov::pass::TransposeSinkingConcatBackward : public ov::pass::MatcherPass {
public:
OPENVINO_RTTI("ov::pass::TransposeSinkingConcatBackward", "0");

12
src/common/transformations/include/transformations/common_optimizations/transpose_sinking_data_movement.hpp
View File

@ -17,12 +17,24 @@ class TRANSFORMATIONS_API TransposeSinkingDataMovementBackward;
} // namespace pass
} // namespace ov
/**
* @ingroup ie_transformation_common_api
* @brief TransposeSinkingDataMovementForward transformation sinks Transpose through BatchToSpace, SpaceToBatch
* and Pad operations in the forward direction.
* These operations are categorized as "DataMovement" and are handled in a similar way in this transformation.
*/
class ov::pass::TransposeSinkingDataMovementForward : public ov::pass::MatcherPass {
public:
OPENVINO_RTTI("ov::pass::TransposeSinkingDataMovementForward", "0");
TransposeSinkingDataMovementForward();
};
/**
* @ingroup ie_transformation_common_api
* @brief TransposeSinkingDataMovementBackward transformation sinks Transpose through BatchToSpace, SpaceToBatch
* and Pad operations in the backward direction.
* These operations are categorized as "DataMovement" and are handled in a similar way in this transformation.
*/
class ov::pass::TransposeSinkingDataMovementBackward : public ov::pass::MatcherPass {
public:
OPENVINO_RTTI("ov::pass::TransposeSinkingDataMovementBackward", "0");

16
src/common/transformations/include/transformations/common_optimizations/transpose_sinking_general.hpp
View File

@ -17,18 +17,34 @@ class TRANSFORMATIONS_API TransposeSinkingGeneral;
} // namespace pass
} // namespace ov
/**
* @ingroup ie_transformation_common_api
* @brief TransposeSinkingGeneralForward transformation combines all TransposeSinkingForward* transformations into
* single GraphRewrite pass.
*/
class ov::pass::TransposeSinkingGeneralForward : public ov::pass::GraphRewrite {
public:
OPENVINO_RTTI("TransposeSinkingGeneralForward", "0");
TransposeSinkingGeneralForward();
};
/**
* @ingroup ie_transformation_common_api
* @brief TransposeSinkingGeneralBackward transformation combines all TransposeSinkingBackward* transformations into
* single GraphRewrite pass.
*/
class ov::pass::TransposeSinkingGeneralBackward : public ov::pass::GraphRewrite {
public:
OPENVINO_RTTI("TransposeSinkingGeneralBackward", "0");
TransposeSinkingGeneralBackward();
};
/**
* @ingroup ie_transformation_common_api
* @brief TransposeSinkingGeneral transformation combines TransposeSinkingGeneralForward and
* TransposeSinkingGeneralBackward transformations into single ModelPass pass and inserts
* ConstantFolding pass after them.
*/
class ov::pass::TransposeSinkingGeneral : public ov::pass::ModelPass {
public:
OPENVINO_RTTI("TransposeSinkingGeneral", "0");

10
src/common/transformations/include/transformations/common_optimizations/transpose_sinking_interpolate.hpp
View File

@ -17,12 +17,22 @@ class TRANSFORMATIONS_API TransposeSinkingInterpolateBackward;
} // namespace pass
} // namespace ov
/**
* @ingroup ie_transformation_common_api
* @brief TransposeSinkingInterpolateForward transformation sinks Transpose through Interpolate operation
* in the forward direction.
*/
class ov::pass::TransposeSinkingInterpolateForward : public ov::pass::MatcherPass {
public:
OPENVINO_RTTI("ov::pass::TransposeSinkingInterpolateForward", "0");
TransposeSinkingInterpolateForward();
};
/**
* @ingroup ie_transformation_common_api
* @brief TransposeSinkingInterpolateBackward transformation sinks Transpose through Interpolate operation
* in the backward direction.
*/
class ov::pass::TransposeSinkingInterpolateBackward : public ov::pass::MatcherPass {
public:
OPENVINO_RTTI("ov::pass::TransposeSinkingInterpolateBackward", "0");

6
src/common/transformations/include/transformations/common_optimizations/transpose_sinking_reduction.hpp
View File

@ -19,7 +19,8 @@ class TRANSFORMATIONS_API TransposeSinkingReductionBackward;
/**
* @ingroup ie_transformation_common_api
* @brief TransposeReductionForward transformation sinks Transpose through Reduce operations
* @brief TransposeReductionForward transformation sinks Transpose through Reduce, Squeeze, Unsqueeze operations
* in the forward direction.
*/
class ov::pass::TransposeSinkingReductionForward : public ov::pass::MatcherPass {
public:
@ -29,7 +30,8 @@ public:
/**
* @ingroup ie_transformation_common_api
* @brief TransposeReductionBackward transformation sinks Transpose through Reduce operations
* @brief TransposeReductionBackward transformation sinks Transpose through Reduce, Squeeze, Unsqueeze operations
* in the backward direction.
*/
class ov::pass::TransposeSinkingReductionBackward : public ov::pass::MatcherPass {
public:

10
src/common/transformations/include/transformations/common_optimizations/transpose_sinking_split.hpp
View File

@ -17,12 +17,22 @@ class TRANSFORMATIONS_API TransposeSinkingSplitForward;
} // namespace pass
} // namespace ov
/**
* @ingroup ie_transformation_common_api
* @brief TransposeSinkingSplitForward transformation sinks Transpose through Split, VariadicSplit operations
* in the forward direction.
*/
class ov::pass::TransposeSinkingSplitForward : public ov::pass::MatcherPass {
public:
OPENVINO_RTTI("ov::pass::TransposeSinkingSplitForward", "0");
TransposeSinkingSplitForward();
};
/**
* @ingroup ie_transformation_common_api
* @brief TransposeSinkingSplitBackward transformation sinks Transpose through Split, VariadicSplit operations
* in the backward direction.
*/
class ov::pass::TransposeSinkingSplitBackward : public ov::pass::MatcherPass {
public:
OPENVINO_RTTI("ov::pass::TransposeSinkingSplitBackward", "0");

10
src/common/transformations/include/transformations/common_optimizations/transpose_sinking_unary.hpp
View File

@ -16,12 +16,22 @@ class TRANSFORMATIONS_API TransposeSinkingUnaryBackward;
} // namespace pass
} // namespace ov
/**
* @ingroup ie_transformation_common_api
* @brief TransposeSinkingUnaryForward transformation sinks Transpose through UnaryElementwiseArithmetic, Clamp, Elu,
* SoftPlus, LogicalNot, Convert, IsInf, IsNaN, IsFinite operations in the forward direction.
*/
class ov::pass::TransposeSinkingUnaryForward : public ov::pass::MatcherPass {
public:
OPENVINO_RTTI("TransposeSinkingUnaryForward", "0");
TransposeSinkingUnaryForward();
};
/**
* @ingroup ie_transformation_common_api
* @brief TransposeSinkingUnaryBackward transformation sinks Transpose through UnaryElementwiseArithmetic, Clamp, Elu,
* SoftPlus, LogicalNot, Convert, IsInf, IsNaN, IsFinite in the backward direction.
*/
class ov::pass::TransposeSinkingUnaryBackward : public ov::pass::MatcherPass {
public:
OPENVINO_RTTI("TransposeSinkingUnaryBackwardMultiConsumers", "0");

15
src/common/transformations/include/transformations/common_optimizations/transpose_sinking_utils.hpp
View File

@ -4,23 +4,18 @@
#pragma once
#include <openvino/pass/pattern/op/or.hpp>
#include <transformations/utils/utils.hpp>
#include <utility>
#include "openvino/op/util/op_types.hpp"
#include "openvino/opsets/opset9.hpp"
#include "openvino/pass/graph_rewrite.hpp"
#include "openvino/pass/pattern/op/label.hpp"
#include "openvino/pass/pattern/op/wrap_type.hpp"
#include "openvino/opsets/opset10.hpp"
#include "openvino/util/common_util.hpp"
#include "openvino/util/log.hpp"
namespace transpose_sinking {
struct TransposeInputsInfo {
std::shared_ptr<ov::opset9::Transpose> transpose;
std::shared_ptr<ov::opset9::Constant> transpose_const;
std::shared_ptr<ov::opset10::Transpose> transpose;
std::shared_ptr<ov::opset10::Constant> transpose_const;
size_t input_idx;
bool isEmpty() const {
@ -87,7 +82,7 @@ namespace sink_backward {
* transposes for all inputs.
*/
ov::NodeVector InsertTransposeBeforeNode(const std::shared_ptr<ov::Node>& main_node,
const std::shared_ptr<ov::opset9::Constant>& transpose_const,
const std::shared_ptr<ov::opset10::Constant>& transpose_const,
std::vector<int> input_indexes = {});
} // namespace sink_backward
@ -109,6 +104,6 @@ void RemoveSingleOutputConsumers(const std::shared_ptr<ov::Node>&);
*/
ov::Output<ov::Node> ChangeValuesOrder(const ov::Output<ov::Node>& input,
const ov::AxisVector& transpose_axis_order,
const std::shared_ptr<ov::opset9::Constant>& axis);
const std::shared_ptr<ov::opset10::Constant>& axis);
} // namespace transpose_sinking

18
src/common/transformations/include/transformations/convert_precision.hpp
View File

@ -64,10 +64,16 @@ class TRANSFORMATIONS_API ConvertPrecision;
* LessEqual
*/
struct EnumClassHash {
template <class T>
std::size_t operator()(T t) const {
return static_cast<size_t>(t);
}
};
using precisions_map = std::unordered_map<ov::element::Type_t, ov::element::Type, EnumClassHash>;
using type_to_fuse_map =
std::unordered_map<ov::NodeTypeInfo,
std::function<bool(const std::shared_ptr<ov::Node>&, ov::element::Type, size_t idx)>>;
using precisions_array = std::vector<std::pair<ov::element::Type, ov::element::Type>>;
std::unordered_map<ov::NodeTypeInfo, std::function<bool(const std::shared_ptr<ov::Node>&, const precisions_map&)>>;
class ov::pass::ConvertPrecision : public ov::pass::ModelPass {
public:
@ -76,11 +82,11 @@ public:
ov::element::Type_t to,
type_to_fuse_map additional_type_to_fuse_map = {},
bool keep_precision_sensitive_in_fp32 = false)
: m_precisions(precisions_array{{from, to}}),
: m_precisions(precisions_map{{from, to}}),
m_additional_type_to_fuse_map(additional_type_to_fuse_map),
m_keep_precision_sensitive_in_fp32(keep_precision_sensitive_in_fp32) {}
ConvertPrecision(const precisions_array& precisions,
ConvertPrecision(const precisions_map& precisions,
const type_to_fuse_map& additional_type_to_fuse_map = {},
bool keep_precision_sensitive_in_fp32 = false)
: m_precisions(precisions),
@ -90,7 +96,7 @@ public:
bool run_on_model(const std::shared_ptr<ov::Model>& m) override;
private:
precisions_array m_precisions;
precisions_map m_precisions;
type_to_fuse_map m_additional_type_to_fuse_map;
bool m_keep_precision_sensitive_in_fp32;
};

4
src/common/transformations/src/transformations/common_optimizations/convert_compression_only_to_legacy.cpp
View File

@ -20,8 +20,8 @@ bool ov::pass::ConvertCompressedOnlyToLegacy::run_on_model(const std::shared_ptr
if (ov::op::util::has_decompression_converts(f)) {
Manager manager(get_pass_config());
const precisions_array convert_precision_list{{ov::element::f32, ov::element::f16}};
manager.register_pass<ConvertPrecision>(convert_precision_list);
const precisions_map convert_precision_map{{ov::element::f32, ov::element::f16}};
manager.register_pass<ConvertPrecision>(convert_precision_map);
using namespace ov::pass;
REGISTER_PASS(manager, EnableDecompressionConvertConstantFolding)
REGISTER_PASS(manager, ConstantFolding)

17
src/common/transformations/src/transformations/common_optimizations/transpose_sinking_concat.cpp
View File

@ -81,18 +81,23 @@ ov::pass::TransposeSinkingConcatBackward::TransposeSinkingConcatBackward() {
if (concat_axis < 0) {
return false;
}
for (auto& new_node : sink_backward::InsertTransposeBeforeNode(main_node, transpose_const)) {
register_new_node(new_node);
}
const auto transpose_axis_order = transpose_const->get_axis_vector_val();
const auto reversed_transpose_axis_order = ReverseTransposeOrder(transpose_axis_order);
if (static_cast<int64_t>(reversed_transpose_axis_order.size()) <= concat_axis) {
return false;
}
const auto transposed_concat_axis = reversed_transpose_axis_order[concat_axis];
concat_node->set_axis(static_cast<int64_t>(transposed_concat_axis));
concat_node->set_concatenation_axis(-1);
concat_node->validate_and_infer_types();
// remove output transposes
RemoveSingleOutputConsumers(main_node);
for (auto& new_node : sink_backward::InsertTransposeBeforeNode(main_node, transpose_const)) {
register_new_node(new_node);
}
concat_node->validate_and_infer_types();
RemoveSingleOutputConsumers(main_node);
SwapNames(transpose, main_node);
return true;
};

5
src/common/transformations/src/transformations/common_optimizations/transpose_sinking_fuse.cpp
View File

@ -41,6 +41,9 @@ ov::pass::TransposeSinkingFuse::TransposeSinkingFuse() {
bool is_ordered = true;
for (size_t i = 0; i < order1.size(); i++) {
if (static_cast<int64_t>(order1.size()) <= order2[i]) {
return false;
}
order2[i] = order1[order2[i]];
if (order2[i] != static_cast<int64_t>(i))
is_ordered = false;
@ -61,7 +64,7 @@ ov::pass::TransposeSinkingFuse::TransposeSinkingFuse() {
new_transpose->set_friendly_name(m.get_match_root()->get_friendly_name());
transpose_sinking::RemoveSingleOutputConsumers(transpose1);
copy_runtime_info(transpose1, new_transpose);
ngraph::replace_node(transpose1, new_transpose);
ov::replace_node(transpose1, new_transpose);
transpose_sinking::UpdateForwardSinkingAbility(new_transpose);
}

2
src/common/transformations/src/transformations/common_optimizations/transpose_sinking_interpolate.cpp
View File

@ -40,7 +40,7 @@ ov::pass::TransposeSinkingInterpolateForward::TransposeSinkingInterpolateForward
}
// remove Transpose on 1st input:
auto transpose_parent = main_node->input_value(0).get_node()->input_value(0);
auto transpose_parent = transpose->input_value(0);
main_node->input(0).replace_source_output(transpose_parent);
const auto transpose_axis_order = transpose_const->get_axis_vector_val();

637
src/common/transformations/src/transformations/convert_precision.cpp
View File

@ -28,37 +28,40 @@
using namespace ov;
bool fuse_type_to_constant(const std::shared_ptr<ngraph::Node>& node,
ngraph::element::Type to,
const precisions_map& precisions,
const std::vector<ngraph::Input<ngraph::Node>>& consumers);
bool fuse_type_to_shapeof(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_shapeof_v0(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_random_uniform_v8(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_unique_v10(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_range_v4(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_parameter(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_convert(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_nms3(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_nms4(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_nms5(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_nms9(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_matrix_nms(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_multiclass_nms(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_generate_proposals(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_topk(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_maxpool(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_nonzero(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_bucketize(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool fuse_type_to_shapeof(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_shapeof_v0(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_random_uniform_v8(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_unique_v10(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_range_v4(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_parameter(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_convert(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_nms3(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_nms4(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_nms5(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_nms9(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_matrix_nms(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_multiclass_nms(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_generate_proposals(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_topk(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_maxpool(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_nonzero(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_bucketize(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool fuse_type_to_ctc_greedy_decoder_seq_len(const std::shared_ptr<ngraph::Node>& node,
ngraph::element::Type to,
size_t idx);
const precisions_map& precisions);
bool fuse_type_to_random_uniform_v8(const std::shared_ptr<ngraph::Node>& node, ov::element::Type to, size_t idx);
bool fuse_type_to_random_uniform_v8(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool extend_select_type(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool extend_reverse_type(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx);
bool extend_select_type(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
bool extend_reverse_type(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
template <typename T>
bool fuse_type_to_binary_comparision(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool fuse_type_to_binary_comparision(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
auto it = precisions.find(node->get_output_element_type(0));
if (it == precisions.end())
return false;
const auto& to = it->second;
if (auto type_relaxed = std::dynamic_pointer_cast<ov::op::TypeRelaxedBase>(node)) {
type_relaxed->set_overridden_output_type(to);
return true;
@ -72,7 +75,11 @@ bool fuse_type_to_binary_comparision(const std::shared_ptr<ngraph::Node>& node,
}
template <typename T>
bool fuse_type_to_logical(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool fuse_type_to_logical(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
auto it = precisions.find(node->get_output_element_type(0));
if (it == precisions.end())
return false;
const auto& to = it->second;
if (auto type_relaxed = std::dynamic_pointer_cast<ov::op::TypeRelaxedBase>(node)) {
type_relaxed->set_overridden_output_type(to);
type_relaxed->set_origin_input_type(ov::element::boolean, 0);
@ -90,7 +97,11 @@ bool fuse_type_to_logical(const std::shared_ptr<ngraph::Node>& node, ngraph::ele
}
template <class T>
bool fuse_type_to_reduce_logical(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool fuse_type_to_reduce_logical(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
auto it = precisions.find(node->get_output_element_type(0));
if (it == precisions.end())
return false;
const auto& to = it->second;
if (auto type_relaxed = std::dynamic_pointer_cast<ov::op::TypeRelaxedBase>(node)) {
type_relaxed->set_overridden_output_type(to);
type_relaxed->set_origin_input_type(ov::element::boolean, 0);
@ -107,23 +118,75 @@ bool fuse_type_to_reduce_logical(const std::shared_ptr<ngraph::Node>& node, ngra
namespace {
void validate_nodes_and_infer_types(const std::vector<std::shared_ptr<Node>>& ops) {
for (auto& node : ops) {
node->revalidate_and_infer_types();
}
bool node_is_replaced(const std::shared_ptr<Node>& node) {
const auto outputs = node->outputs();
bool has_consumers = std::all_of(outputs.begin(), outputs.end(), [](const Output<Node>& output) {
return output.get_target_inputs().size() == 0;
});
return has_consumers && !(is_type<op::v0::Result>(node) || is_type<op::Sink>(node));
}
bool convert_precision(ov::pass::PassBase& pass,
const std::shared_ptr<ngraph::Function>& f,
const type_to_fuse_map& type_to_fuse,
const type_to_fuse_map& type_to_extend,
ov::element::Type from,
ov::element::Type to,
bool skip_precision_sensitive = false) {
// As Constant operations can be shared between multiple nGraph Functions so before
// changing precision we need to understand which Constant consumers belongs
// to the current nGraph Function
std::unordered_map<const ngraph::Node*, std::vector<Input<Node>>> const_to_internal_output;
bool convert_node_output_precision(
const std::shared_ptr<ngraph::Node>& node,
const precisions_map& precisions,
const type_to_fuse_map& type_to_fuse,
const std::unordered_map<const ngraph::Node*, std::vector<Input<Node>>>& const_to_internal_output,
bool function_changed) {
bool node_changed = false;
// Handle case with Constants as they can have consumers from other nGraph Function object
const auto constant = ov::as_type_ptr<opset10::Constant>(node);
const auto it = const_to_internal_output.find(node.get());
if (constant && it != const_to_internal_output.end()) {
return fuse_type_to_constant(node, precisions, it->second);
}
// Check that node type exists in map and we can fuse type into node
const auto t2f_it = type_to_fuse.find(node->get_type_info());
if (t2f_it != type_to_fuse.end()) {
node_changed = t2f_it->second(node, precisions);
}
if ((function_changed || node_changed) && !node_is_replaced(node)) {
node->revalidate_and_infer_types();
}
return node_changed;
}
bool convert_node_input_precision(const std::shared_ptr<ngraph::Node>& node,
const precisions_map& precisions,
const type_to_fuse_map& type_to_extend) {
// For some operations we need to extend their input types to support new type
auto it = type_to_extend.find(node->get_type_info());
if (it != type_to_extend.end()) {
return it->second(node, precisions);
}
return false;
}
bool convert_function_precision(
const std::shared_ptr<Model>& f,
const type_to_fuse_map& type_to_fuse,
const type_to_fuse_map& type_to_extend,
const precisions_map& precisions,
std::unordered_map<const ngraph::Node*, std::vector<Input<Node>>>& const_to_internal_output,
bool has_fp16_compression,
bool skip_precision_sensitive,
bool is_changed,
bool is_subgraph) {
bool is_output_precision_changed = false;
auto ops = f->get_ordered_ops();
// Iterate over all nodes in topological order and then iterate over node outputs.
// If output type mismatch given type we try to fuse type into this operation
// otherwise we insert Convert operation.
for (auto& node : ops) {
if (skip_precision_sensitive && fp16_compression_is_disabled(node) && has_fp16_compression)
continue;
is_changed |= convert_node_input_precision(node, precisions, type_to_extend);
}
if (is_changed)
ops = f->get_ordered_ops();
auto register_constants = [&const_to_internal_output](const std::vector<std::shared_ptr<Node>>& ops) {
for (auto& node : ops) {
@ -136,117 +199,81 @@ bool convert_precision(ov::pass::PassBase& pass,
}
};
auto convert_node_output_precision = [&](const std::shared_ptr<ngraph::Node>& node) {
bool res = false;
// Handle case with Constants as they can have consumers from other nGraph Function object
const auto constant = ov::as_type_ptr<opset10::Constant>(node);
const auto it = const_to_internal_output.find(node.get());
if (constant && constant->get_output_element_type(0) == from && it != const_to_internal_output.end()) {
return fuse_type_to_constant(node, to, it->second);
}
// Register internal constants only after fixing input type that could lead to nodes
// replacement
register_constants(ops);
for (const auto& output : node->outputs()) {
if (output.get_element_type() == from) {
// Check that node type exists in map and we can fuse type into node
const auto t2f_it = type_to_fuse.find(node->get_type_info());
if (t2f_it != type_to_fuse.end()) {
res |= t2f_it->second(node, to, output.get_index());
}
for (auto& node : ops) {
// skip precision sensitive nodes
if (skip_precision_sensitive && fp16_compression_is_disabled(node) && has_fp16_compression)
continue;
// Recursively apply transformation for sub-graph based operations
if (auto sub_graph_node = std::dynamic_pointer_cast<op::util::MultiSubGraphOp>(node)) {
size_t sub_graphs_num = sub_graph_node->get_internal_subgraphs_size();
for (size_t sub_graph_ind = 0; sub_graph_ind < sub_graphs_num; ++sub_graph_ind) {
is_changed |= convert_function_precision(sub_graph_node->get_function(static_cast<int>(sub_graph_ind)),
type_to_fuse,
type_to_extend,
precisions,
const_to_internal_output,
has_fp16_compression,
skip_precision_sensitive,
is_changed || is_output_precision_changed,
true);
}
}
return res;
};
is_output_precision_changed |= convert_node_output_precision(node,
precisions,
type_to_fuse,
const_to_internal_output,
is_changed || is_output_precision_changed);
}
auto convert_node_input_precision = [&](const std::shared_ptr<ngraph::Node>& node) {
for (auto input : node->inputs()) {
if (input.get_element_type() == from) {
// For some operations we need to extend their input types to support new type
auto it = type_to_extend.find(node->get_type_info());
if (it != type_to_extend.end() && it->second(node, to, input.get_index())) {
return true;
}
}
}
return false;
};
if (is_output_precision_changed) {
ops = f->get_ordered_ops();
is_changed |= is_output_precision_changed;
}
std::function<bool(const std::shared_ptr<Model>&, bool)> convert_function_precision =
[&](const std::shared_ptr<Model>& f, bool is_subgraph) {
bool is_changed = false;
auto ops = f->get_ordered_ops();
// Iterate over all nodes in topological order and then iterate over node outputs.
// If output type mismatch given type we try to fuse type into this operation
// otherwise we insert Convert operation.
for (auto& node : ops) {
if (skip_precision_sensitive && fp16_compression_is_disabled(node) && to == element::f16)
if (!is_subgraph) {
// TODO: we need to split NopElimination pass to separate MatcherPasses and call
// Convert elimination here
for (auto& node : ops) {
if (auto convert = std::dynamic_pointer_cast<opset4::Convert>(node)) {
if (pass::constant_folding_is_disabled(node))
continue;
// Recursively apply transformation for sub-graph based operations
if (auto sub_graph_node = std::dynamic_pointer_cast<op::util::MultiSubGraphOp>(node)) {
size_t sub_graphs_num = sub_graph_node->get_internal_subgraphs_size();
for (size_t sub_graph_ind = 0; sub_graph_ind < sub_graphs_num; ++sub_graph_ind) {
is_changed |=
convert_function_precision(sub_graph_node->get_function(static_cast<int>(sub_graph_ind)),
true);
}
}
is_changed |= convert_node_input_precision(node);
}
if (is_changed)
ops = f->get_ordered_ops();
// Register internal constants only after fixing input type that could lead to nodes
// replacement
register_constants(ops);
bool is_output_precision_changed = false;
for (auto& node : ops) {
// skip precision sensitive nodes
if (skip_precision_sensitive && fp16_compression_is_disabled(node) && to == element::f16)
continue;
is_output_precision_changed |= convert_node_output_precision(node);
}
if (is_output_precision_changed) {
ops = f->get_ordered_ops();
is_changed |= is_output_precision_changed;
}
if (!is_subgraph) {
if (is_changed)
validate_nodes_and_infer_types(ops);
// TODO: we need to split NopElimination pass to separate MatcherPasses and call
// Convert elimination here
for (auto& node : ops) {
if (auto convert = std::dynamic_pointer_cast<opset4::Convert>(node)) {
if (pass::constant_folding_is_disabled(node))
continue;
// WA for topK, dont remove fake convert
if (convert->input(0).get_element_type() == convert->get_convert_element_type() &&
convert->input_value(0).get_node_shared_ptr()->get_output_size() == 1) {
replace_output_update_name(convert->output(0), convert->input_value(0));
}
}
// WA for topK, dont remove fake convert
if (convert->input(0).get_element_type() == convert->get_convert_element_type() &&
convert->input_value(0).get_node_shared_ptr()->get_output_size() == 1) {
replace_output_update_name(convert->output(0), convert->input_value(0));
}
}
}
}
return is_changed;
};
return convert_function_precision(f, false);
return is_changed;
}
struct EnumClassHash {
template <class T>
std::size_t operator()(T t) const {
return static_cast<size_t>(t);
}
};
bool convert_precision(ov::pass::PassBase& pass,
const std::shared_ptr<ngraph::Function>& f,
const type_to_fuse_map& type_to_fuse,
const type_to_fuse_map& type_to_extend,
const precisions_map& precisions,
bool has_fp16_compression,
bool skip_precision_sensitive = false) {
// As Constant operations can be shared between multiple nGraph Functions so before
// changing precision we need to understand which Constant consumers belongs
// to the current nGraph Function
std::unordered_map<const ngraph::Node*, std::vector<Input<Node>>> const_to_internal_output;
return convert_function_precision(f,
type_to_fuse,
type_to_extend,
precisions,
const_to_internal_output,
has_fp16_compression,
skip_precision_sensitive,
false,
false);
}
using precisions_set_t = std::unordered_set<ngraph::element::Type_t, EnumClassHash>;
@ -273,6 +300,27 @@ precisions_set_t find_all_used_precisions(const std::shared_ptr<ngraph::Function
} // namespace
bool ov::pass::ConvertPrecision::run_on_model(const std::shared_ptr<ngraph::Function>& f) {
const auto used_precisions_set = find_all_used_precisions(f);
precisions_map used_precisions;
for (const auto& p : used_precisions_set) {
auto it = m_precisions.find(p);
if (it != m_precisions.end())
used_precisions.insert(*it);
}
if (used_precisions.empty())
return false;
bool has_fp16_compression = m_precisions.count(element::f32) > 0 && m_precisions[element::f32] == element::f16;
if (m_keep_precision_sensitive_in_fp32 && has_fp16_compression) {
pass::Manager manager(get_pass_config());
// Mark subgraphs with disable_fp16_compression to keep them in FP32
manager.register_pass<pass::MarkSugraphsToKeepInMixedPrecision>();
manager.register_pass<pass::AlignMixedFP32FP16Types>();
manager.run_passes(f);
}
type_to_fuse_map type_to_fuse{
{opset4::Parameter::get_type_info_static(), fuse_type_to_parameter},
{opset4::Convert::get_type_info_static(), fuse_type_to_convert},
@ -308,17 +356,6 @@ bool ov::pass::ConvertPrecision::run_on_model(const std::shared_ptr<ngraph::Func
{opset10::Unique::get_type_info_static(), fuse_type_to_unique_v10},
{opset8::RandomUniform::get_type_info_static(), fuse_type_to_random_uniform_v8}};
std::pair<ov::element::Type, ov::element::Type> compress_f16_pair = {ov::element::f32, ov::element::f16};
bool has_compress_f16 = std::count(m_precisions.begin(), m_precisions.end(), compress_f16_pair) > 0;
if (m_keep_precision_sensitive_in_fp32 && has_compress_f16) {
pass::Manager manager(get_pass_config());
// Mark subgraphs with disable_fp16_compression to keep them in FP32
manager.register_pass<pass::MarkSugraphsToKeepInMixedPrecision>();
manager.register_pass<pass::AlignMixedFP32FP16Types>();
manager.run_passes(f);
}
for (const auto& it : m_additional_type_to_fuse_map) {
type_to_fuse[it.first] = it.second;
}
@ -330,20 +367,13 @@ bool ov::pass::ConvertPrecision::run_on_model(const std::shared_ptr<ngraph::Func
{opset1::Reverse::get_type_info_static(), extend_reverse_type},
};
bool is_changed = false;
auto const used_precisions = find_all_used_precisions(f);
for (auto const& p : m_precisions) {
if (used_precisions.count(p.first))
is_changed = is_changed | convert_precision(*this,
f,
type_to_fuse,
type_to_extend,
p.first,
p.second,
m_keep_precision_sensitive_in_fp32);
}
bool is_changed = convert_precision(*this,
f,
type_to_fuse,
type_to_extend,
used_precisions,
has_fp16_compression,
m_keep_precision_sensitive_in_fp32);
// to remove extra converts
if (m_keep_precision_sensitive_in_fp32) {
@ -361,7 +391,11 @@ bool ov::pass::ConvertPrecision::run_on_model(const std::shared_ptr<ngraph::Func
return false;
}
bool fuse_type_to_shapeof(const std::shared_ptr<ngraph::Node>& node, ov::element::Type to, size_t idx) {
bool fuse_type_to_shapeof(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
auto it = precisions.find(node->get_output_element_type(0));
if (it == precisions.end())
return false;
const auto& to = it->second;
if (auto shapeof = ov::as_type_ptr<opset4::ShapeOf>(node)) {
if (to == ov::element::i32 || to == ov::element::i64) {
shapeof->set_output_type(to);
@ -371,7 +405,11 @@ bool fuse_type_to_shapeof(const std::shared_ptr<ngraph::Node>& node, ov::element
return false;
}
bool fuse_type_to_random_uniform_v8(const std::shared_ptr<ngraph::Node>& node, ov::element::Type to, size_t idx) {
bool fuse_type_to_random_uniform_v8(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
auto it = precisions.find(node->get_output_element_type(0));
if (it == precisions.end())
return false;
const auto& to = it->second;
if (auto random_uniform = ov::as_type_ptr<opset8::RandomUniform>(node)) {
if (to.is_integral_number() || to.is_real()) {
random_uniform->set_out_type(to);
@ -381,23 +419,28 @@ bool fuse_type_to_random_uniform_v8(const std::shared_ptr<ngraph::Node>& node, o
return false;
}
bool fuse_type_to_unique_v10(const std::shared_ptr<Node>& node, ov::element::Type to, size_t idx) {
bool fuse_type_to_unique_v10(const std::shared_ptr<Node>& node, const precisions_map& precisions) {
bool res = false;
if (auto unique = ov::as_type_ptr<opset10::Unique>(node)) {
if (to == ov::element::i32 || to == ov::element::i64) {
if (idx == 1 || idx == 2) {
unique->set_index_element_type(to);
res = true;
} else if (idx == 3) {
unique->set_count_element_type(to);
res = true;
}
auto it = precisions.find(node->get_output_element_type(1));
if (it != precisions.end()) {
unique->set_index_element_type(it->second);
res = true;
}
it = precisions.find(node->get_output_element_type(3));
if (it != precisions.end()) {
unique->set_count_element_type(it->second);
res = true;
}
}
return res;
}
bool fuse_type_to_range_v4(const std::shared_ptr<ngraph::Node>& node, ov::element::Type to, size_t idx) {
bool fuse_type_to_range_v4(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
auto it = precisions.find(node->get_output_element_type(0));
if (it == precisions.end())
return false;
const auto& to = it->second;
if (auto range = ov::as_type_ptr<opset4::Range>(node)) {
if (to.is_integral_number() || to.is_real()) {
range->set_output_type(to);
@ -407,7 +450,11 @@ bool fuse_type_to_range_v4(const std::shared_ptr<ngraph::Node>& node, ov::elemen
return false;
}
bool fuse_type_to_parameter(const std::shared_ptr<ngraph::Node>& node, ov::element::Type to, size_t idx) {
bool fuse_type_to_parameter(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
auto it = precisions.find(node->get_output_element_type(0));
if (it == precisions.end())
return false;
const auto& to = it->second;
if (auto param = ov::as_type_ptr<opset4::Parameter>(node)) {
param->set_element_type(to);
param->validate_and_infer_types();
@ -416,7 +463,11 @@ bool fuse_type_to_parameter(const std::shared_ptr<ngraph::Node>& node, ov::eleme
return false;
}
bool fuse_type_to_convert(const std::shared_ptr<ngraph::Node>& node, ov::element::Type to, size_t idx) {
bool fuse_type_to_convert(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
auto it = precisions.find(node->get_output_element_type(0));
if (it == precisions.end())
return false;
const auto& to = it->second;
if (auto convert = ov::as_type_ptr<opset4::Convert>(node)) {
convert->set_convert_element_type(to);
return true;
@ -424,7 +475,11 @@ bool fuse_type_to_convert(const std::shared_ptr<ngraph::Node>& node, ov::element
return false;
}
bool fuse_type_to_nms3(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool fuse_type_to_nms3(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
auto it = precisions.find(node->get_output_element_type(0));
if (it == precisions.end())
return false;
const auto& to = it->second;
if (auto nms = ov::as_type_ptr<opset3::NonMaxSuppression>(node)) {
if (to == ov::element::i32 || to == ov::element::i64) {
nms->set_output_type(to);
@ -436,7 +491,11 @@ bool fuse_type_to_nms3(const std::shared_ptr<ngraph::Node>& node, ngraph::elemen
return false;
}
bool fuse_type_to_nms4(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool fuse_type_to_nms4(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
auto it = precisions.find(node->get_output_element_type(0));
if (it == precisions.end())
return false;
const auto& to = it->second;
if (auto nms = ov::as_type_ptr<opset4::NonMaxSuppression>(node)) {
if (to == ov::element::i32 || to == ov::element::i64) {
nms->set_output_type(to);
@ -448,75 +507,125 @@ bool fuse_type_to_nms4(const std::shared_ptr<ngraph::Node>& node, ngraph::elemen
return false;
}
bool fuse_type_to_nms5(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool fuse_type_to_nms5(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
auto nms = ov::as_type_ptr<opset5::NonMaxSuppression>(node);
if (!nms) {
return false;
}
if ((idx == 0 || idx == 2) && (to == ov::element::i32 || to == ov::element::i64)) {
nms->set_output_type(to);
return true;
}
if (auto type_relaxed = std::dynamic_pointer_cast<ov::op::TypeRelaxedBase>(node)) {
type_relaxed->set_overridden_output_type(to, idx);
return true;
bool res = false;
auto it = precisions.find(node->get_output_element_type(0));
if (it != precisions.end()) {
const auto& to = it->second;
if (to == ov::element::i32 || to == ov::element::i64) {
nms->set_output_type(to);
res = true;
if (precisions.count(node->get_output_element_type(1)) == 0) {
return res;
}
}
}
auto type_relaxed = std::dynamic_pointer_cast<ov::op::TypeRelaxedBase>(node);
ov::element::TypeVector output_types;
for (const auto& output : nms->outputs()) {
output_types.emplace_back(output.get_element_type());
for (size_t i = 0; i < node->get_output_size(); i++) {
it = precisions.find(node->get_output_element_type(i));
if (it == precisions.end()) {
output_types.push_back(node->get_output_element_type(i));
continue;
}
const auto& to = it->second;
if (type_relaxed) {
type_relaxed->set_overridden_output_type(to, i);
res = true;
}
output_types.push_back(to);
}
output_types[idx] = to;
auto relaxed_op =
std::make_shared<ov::op::TypeRelaxed<opset5::NonMaxSuppression>>(*nms, ov::element::TypeVector{}, output_types);
replace_node(node, relaxed_op);
return true;
if (!type_relaxed) {
auto relaxed_op = std::make_shared<ov::op::TypeRelaxed<opset5::NonMaxSuppression>>(*nms,
ov::element::TypeVector{},
output_types);
replace_node(node, relaxed_op);
res = true;
}
return res;
}
bool fuse_type_to_nms9(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool fuse_type_to_nms9(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
auto nms = ov::as_type_ptr<opset9::NonMaxSuppression>(node);
if (!nms) {
return false;
}
if ((idx == 0 || idx == 2) && (to == ov::element::i32 || to == ov::element::i64)) {
nms->set_output_type(to);
return true;
}
if (auto type_relaxed = std::dynamic_pointer_cast<ov::op::TypeRelaxedBase>(node)) {
type_relaxed->set_overridden_output_type(to, idx);
return true;
bool res = false;
auto it = precisions.find(node->get_output_element_type(0));
if (it != precisions.end()) {
const auto& to = it->second;
if (to == ov::element::i32 || to == ov::element::i64) {
nms->set_output_type(to);
res = true;
if (precisions.count(node->get_output_element_type(1)) == 0) {
return res;
}
}
}
auto type_relaxed = std::dynamic_pointer_cast<ov::op::TypeRelaxedBase>(node);
ov::element::TypeVector output_types;
for (const auto& output : nms->outputs()) {
output_types.emplace_back(output.get_element_type());
for (size_t i = 0; i < node->get_output_size(); i++) {
it = precisions.find(node->get_output_element_type(i));
if (it == precisions.end()) {
output_types.push_back(node->get_output_element_type(i));
continue;
}
const auto& to = it->second;
if (type_relaxed) {
type_relaxed->set_overridden_output_type(to, i);
res = true;
}
output_types.push_back(to);
}
output_types[idx] = to;
auto relaxed_op =
std::make_shared<ov::op::TypeRelaxed<opset9::NonMaxSuppression>>(*nms, ov::element::TypeVector{}, output_types);
replace_node(node, relaxed_op);
return true;
if (!type_relaxed) {
auto relaxed_op = std::make_shared<ov::op::TypeRelaxed<opset9::NonMaxSuppression>>(*nms,
ov::element::TypeVector{},
output_types);
replace_node(node, relaxed_op);
res = true;
}
return res;
}
bool fuse_type_to_matrix_nms(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool update_type(size_t idx,
const std::shared_ptr<ngraph::Node>& node,
const precisions_map& precisions,
std::function<void(const element::Type&)> update_method) {
auto it = precisions.find(node->get_output_element_type(idx));
if (it != precisions.end()) {
const auto& to = it->second;
if (to == ov::element::i32 || to == ov::element::i64) {
update_method(to);
return true;
}
}
return false;
}
bool fuse_type_to_matrix_nms(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
auto nms = ov::as_type_ptr<opset8::MatrixNms>(node);
if (!nms) {
return false;
}
if ((idx == 1 || idx == 2) && (to == ov::element::i32 || to == ov::element::i64)) {
return update_type(1, node, precisions, [&](const element::Type& to) {
nms->set_output_type(to);
return true;
}
return false;
});
}
bool fuse_type_to_multiclass_nms(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool fuse_type_to_multiclass_nms(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
std::shared_ptr<ov::op::util::MulticlassNmsBase> nms;
if (ov::is_type<ov::op::v8::MulticlassNms>(node)) {
nms = ov::as_type_ptr<opset8::MulticlassNms>(node);
@ -527,85 +636,81 @@ bool fuse_type_to_multiclass_nms(const std::shared_ptr<ngraph::Node>& node, ngra
return false;
}
if ((idx == 1 || idx == 2) && (to == ov::element::i32 || to == ov::element::i64)) {
return update_type(1, node, precisions, [&](const element::Type& to) {
nms->set_output_type(to);
return true;
}
return false;
});
}
bool fuse_type_to_generate_proposals(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool fuse_type_to_generate_proposals(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
auto generate_proposals = ov::as_type_ptr<opset9::GenerateProposals>(node);
if (!generate_proposals) {
return false;
}
if ((idx == 2) && (to == ov::element::i32 || to == ov::element::i64)) {
return update_type(2, node, precisions, [&](const element::Type& to) {
generate_proposals->set_roi_num_type(to);
return true;
}
return false;
});
}
bool fuse_type_to_topk(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool fuse_type_to_topk(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
if (auto topk = ov::as_type_ptr<opset4::TopK>(node)) {
if (idx == 1 && (to == ov::element::i32 || to == ov::element::i64)) {
return update_type(1, node, precisions, [&](const element::Type& to) {
topk->set_index_element_type(to);
return true;
}
});
}
return false;
}
bool fuse_type_to_maxpool(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool fuse_type_to_maxpool(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
if (auto maxpool = ov::as_type_ptr<opset8::MaxPool>(node)) {
if (idx == 1 && (to == ov::element::i32 || to == ov::element::i64)) {
return update_type(1, node, precisions, [&](const element::Type& to) {
maxpool->set_index_element_type(to);
return true;
}
});
}
return false;
}
bool fuse_type_to_ctc_greedy_decoder_seq_len(const std::shared_ptr<ngraph::Node>& node,
ngraph::element::Type to,
size_t idx) {
const precisions_map& precisions) {
bool res = false;
if (auto ctc_decoder = ov::as_type_ptr<opset6::CTCGreedyDecoderSeqLen>(node)) {
if (idx == 0 && (to == ov::element::i32 || to == ov::element::i64)) {
res = update_type(0, node, precisions, [&](const element::Type& to) {
ctc_decoder->set_classes_index_type(to);
return true;
}
if (idx == 1 && (to == ov::element::i32 || to == ov::element::i64)) {
ctc_decoder->set_sequence_length_type(to);
return true;
}
});
res = update_type(1,
node,
precisions,
[&](const element::Type& to) {
ctc_decoder->set_sequence_length_type(to);
}) ||
res;
}
return false;
return res;
}
bool fuse_type_to_nonzero(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool fuse_type_to_nonzero(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
if (auto nonzero = ov::as_type_ptr<opset4::NonZero>(node)) {
if (to == ov::element::i32 || to == ov::element::i64) {
return update_type(0, node, precisions, [&](const element::Type& to) {
nonzero->set_output_type(to);
return true;
}
});
}
return false;
}
bool fuse_type_to_bucketize(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool fuse_type_to_bucketize(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
if (auto b = ov::as_type_ptr<opset4::Bucketize>(node)) {
if (to == ov::element::i32 || to == ov::element::i64) {
return update_type(0, node, precisions, [&](const element::Type& to) {
b->set_output_type(to);
return true;
}
});
}
return false;
}
bool fuse_type_to_shapeof_v0(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool fuse_type_to_shapeof_v0(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
auto it = precisions.find(node->get_output_element_type(0));
if (it == precisions.end())
return false;
const auto& to = it->second;
if (auto type_relaxed = std::dynamic_pointer_cast<ov::op::TypeRelaxedBase>(node)) {
type_relaxed->set_overridden_output_type(to);
return true;
@ -619,7 +724,7 @@ bool fuse_type_to_shapeof_v0(const std::shared_ptr<ngraph::Node>& node, ngraph::
return false;
}
bool extend_select_type(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool extend_select_type(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
if (auto type_relaxed = std::dynamic_pointer_cast<ov::op::TypeRelaxedBase>(node)) {
type_relaxed->set_origin_input_type(ov::element::boolean, 0);
return true;
@ -634,7 +739,7 @@ bool extend_select_type(const std::shared_ptr<ngraph::Node>& node, ngraph::eleme
return false;
}
bool extend_reverse_type(const std::shared_ptr<ngraph::Node>& node, ngraph::element::Type to, size_t idx) {
bool extend_reverse_type(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
if (const auto casted = std::dynamic_pointer_cast<opset1::Reverse>(node)) {
if (casted->get_mode() == ov::op::v1::Reverse::Mode::MASK) {
auto relaxed_op = std::make_shared<op::TypeRelaxed<opset1::Reverse>>(
@ -903,10 +1008,14 @@ std::shared_ptr<Node> convert_low_precisions_int(std::shared_ptr<opset4::Constan
} // namespace
bool fuse_type_to_constant(const std::shared_ptr<ngraph::Node>& node,
ov::element::Type to,
const precisions_map& precisions,
const std::vector<Input<Node>>& consumers) {
auto from = node->get_element_type();
auto it = precisions.find(from);
if (it == precisions.end())
return false;
const auto& to = it->second;
if (auto constant = ov::as_type_ptr<opset4::Constant>(node)) {
auto from = constant->get_element_type();
std::shared_ptr<ngraph::Node> new_const;
if (from == ov::element::u64 && to == ov::element::i32) {
new_const = change_constant_precision<ov::element::Type_t::u64, ov::element::Type_t::i32>(constant);

100
src/common/transformations/tests/utils/convert_precision.cpp
View File

@ -64,7 +64,7 @@ TEST(TransformationTests, ConvertPrecision_NMS3) {
pass::Manager manager;
static const precisions_array precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -94,7 +94,7 @@ TEST(TransformationTests, ConvertPrecision_NMS4) {
pass::Manager manager;
static const precisions_array precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -127,7 +127,7 @@ TEST(TransformationTests, ConvertPrecision_NMS5) {
}
pass::Manager manager;
static const precisions_array precisions = {{element::i64, element::i32}, {element::f32, element::f16}};
static const precisions_map precisions = {{element::i64, element::i32}, {element::f32, element::f16}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
ASSERT_FALSE(has_type<element::Type_t::i64>(f));
@ -150,7 +150,7 @@ TEST(TransformationTests, ConvertPrecision_MatrixNms) {
}
pass::Manager manager;
static const precisions_array precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
ASSERT_FALSE(has_type<element::Type_t::i64>(f));
@ -173,7 +173,7 @@ TEST(TransformationTests, ConvertPrecision_MulticlassNms) {
}
pass::Manager manager;
static const precisions_array precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
ASSERT_FALSE(has_type<element::Type_t::i64>(f));
@ -190,7 +190,7 @@ TEST(TransformationTests, ConvertPrecision_ShapeOf) {
pass::Manager manager;
static const precisions_array precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -212,7 +212,7 @@ TEST(TransformationTests, ConvertPrecision_Range) {
pass::Manager manager;
static const precisions_array precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -233,7 +233,7 @@ TEST(TransformationTests, ConvertPrecision_ConstantRelu) {
pass::Manager manager;
static const precisions_array precisions = {{element::f16, element::f32}};
static const precisions_map precisions = {{element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -253,7 +253,7 @@ TEST(TransformationTests, ConvertPrecision_Convert) {
pass::Manager manager;
static const precisions_array precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -273,7 +273,7 @@ TEST(TransformationTests, ConvertPrecision_ConvertElimination) {
f = std::make_shared<Model>(NodeVector{convert}, ParameterVector{input});
pass::Manager manager;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f16, element::f32}});
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f16, element::f32}});
manager.run_passes(f);
ASSERT_FALSE(has_type<element::Type_t::f16>(f));
}
@ -300,7 +300,7 @@ TEST(TransformationTests, ConvertPrecision_TopK) {
pass::Manager manager;
static const precisions_array precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -320,7 +320,7 @@ TEST(TransformationTests, ConvertPrecision_Unique10) {
pass::Manager manager;
static const precisions_array precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
manager.register_pass<ov::pass::ConvertPrecision>(precisions);
manager.run_passes(model);
@ -348,7 +348,7 @@ TEST(TransformationTests, ConvertPrecision_NonZero) {
pass::Manager manager;
static const precisions_array precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -369,7 +369,7 @@ TEST(TransformationTests, ConvertPrecision_Bucketize) {
pass::Manager manager;
static const precisions_array precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -399,7 +399,7 @@ TEST(TransformationTests, ConvertPrecision_Roundings) {
pass::Manager manager;
static const precisions_array precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -456,7 +456,7 @@ TEST(TransformationTests, ConvertPrecision_TIBody) {
pass::Manager manager;
static const precisions_array precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::i64, element::i32}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -479,7 +479,7 @@ TEST(TransformationTests, ConvertPrecision_Equal) {
pass::Manager manager;
static const precisions_array precisions = {{element::boolean, element::u8}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::boolean, element::u8}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -501,7 +501,7 @@ TEST(TransformationTests, ConvertPrecision_NotEqual) {
pass::Manager manager;
static const precisions_array precisions = {{element::boolean, element::u8}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::boolean, element::u8}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -523,7 +523,7 @@ TEST(TransformationTests, ConvertPrecision_Greater) {
pass::Manager manager;
static const precisions_array precisions = {{element::boolean, element::u8}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::boolean, element::u8}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -545,7 +545,7 @@ TEST(TransformationTests, ConvertPrecision_GreaterEqual) {
pass::Manager manager;
static const precisions_array precisions = {{element::boolean, element::u8}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::boolean, element::u8}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -567,7 +567,7 @@ TEST(TransformationTests, ConvertPrecision_Less) {
pass::Manager manager;
static const precisions_array precisions = {{element::boolean, element::u8}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::boolean, element::u8}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -589,7 +589,7 @@ TEST(TransformationTests, ConvertPrecision_LessEqual) {
pass::Manager manager;
static const precisions_array precisions = {{element::boolean, element::u8}, {element::f16, element::f32}};
static const precisions_map precisions = {{element::boolean, element::u8}, {element::f16, element::f32}};
manager.register_pass<pass::ConvertPrecision>(precisions);
manager.run_passes(f);
@ -610,7 +610,7 @@ TEST(TransformationTests, ConvertPrecision_LogicalAnd) {
f = std::make_shared<Model>(OutputVector{node}, ParameterVector{input1, input2});
pass::Manager manager;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::boolean, element::u8}});
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::boolean, element::u8}});
manager.run_passes(f);
}
@ -628,7 +628,7 @@ TEST(TransformationTests, ConvertPrecision_LogicalOr) {
f = std::make_shared<Model>(OutputVector{node}, ParameterVector{input1, input2});
pass::Manager manager;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::boolean, element::u8}});
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::boolean, element::u8}});
manager.run_passes(f);
}
@ -646,7 +646,7 @@ TEST(TransformationTests, ConvertPrecision_LogicalXor) {
f = std::make_shared<Model>(OutputVector{node}, ParameterVector{input1, input2});
pass::Manager manager;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::boolean, element::u8}});
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::boolean, element::u8}});
manager.run_passes(f);
}
@ -663,7 +663,7 @@ TEST(TransformationTests, ConvertPrecision_LogicalNot) {
f = std::make_shared<Model>(OutputVector{node}, ParameterVector{input1});
pass::Manager manager;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::boolean, element::u8}});
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::boolean, element::u8}});
manager.run_passes(f);
}
@ -681,7 +681,7 @@ TEST(TransformationTests, ConvertPrecision_Select) {
f = std::make_shared<Model>(OutputVector{select}, ParameterVector{input1});
pass::Manager manager;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::boolean, element::u8}});
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::boolean, element::u8}});
manager.run_passes(f);
}
@ -699,8 +699,8 @@ TEST(TransformationTests, ConvertPrecision_TypeRelaxedWithSelect) {
f = std::make_shared<Model>(OutputVector{select}, ParameterVector{input1});
pass::Manager manager;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::boolean, element::i32}});
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::i32, element::i64}});
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::boolean, element::i32}});
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::i32, element::i64}});
manager.run_passes(f);
}
@ -721,8 +721,8 @@ TEST(TransformationTests, ConvertPrecision_TypeRelaxed) {
f = std::make_shared<Model>(OutputVector{type_relaxed}, ParameterVector{input1});
pass::Manager manager;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::boolean, element::i32}});
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::i32, element::i64}});
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::boolean, element::i32}});
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::i32, element::i64}});
manager.run_passes(f);
ASSERT_FALSE(has_type<element::Type_t::boolean>(f));
@ -747,7 +747,7 @@ TEST(TransformationTests, ConvertPrecision_Variables) {
f = std::make_shared<Model>(NodeVector{mul}, ParameterVector{inp});
pass::Manager manager;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f16, element::f32}});
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f16, element::f32}});
manager.run_passes(f);
}
@ -778,7 +778,7 @@ TEST(TransformationTests, ConvertPrecision_skip_precision_sensitive) {
pass::Manager manager;
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = true;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);
@ -812,7 +812,7 @@ TEST(TransformationTests, ConvertPrecision_without_keep_precision_sensitive_in_f
pass::Manager manager;
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = false;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);
@ -834,7 +834,7 @@ TEST(TransformationTests, ConvertPrecision_check_marking_does_not_leak_in_trivia
pass::Manager manager;
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = true;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);
@ -873,7 +873,7 @@ TEST(TransformationTests, ConvertPrecision_whole_shape_subgraph_is_marked_1) {
pass::Manager manager;
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = true;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);
@ -926,7 +926,7 @@ TEST(TransformationTests, ConvertPrecision_whole_shape_subgraph_is_marked_2) {
pass::Manager manager;
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = true;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);
@ -1005,7 +1005,7 @@ TEST(TransformationTests, ConvertPrecision_whole_shape_subgraph_is_marked_3) {
pass::Manager manager;
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = true;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);
@ -1083,7 +1083,7 @@ TEST(TransformationTests, ConvertCompressedToMixedPrecission_do_not_keep_in_fp32
pass::Manager manager;
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = false; // didn't keep in FP32 intentionally
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);
@ -1108,7 +1108,7 @@ void constant_convert_test(element::Type type_from,
f = std::make_shared<Model>(NodeVector{c}, ParameterVector{});
pass::Manager manager;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{type_from, type_to}});
manager.register_pass<pass::ConvertPrecision>(precisions_map{{type_from, type_to}});
manager.run_passes(f);
}
auto ops = f->get_ordered_ops();
@ -1138,7 +1138,7 @@ void constant_convert_test(element::Type_t type_from, element::Type_t type_to, F
f = std::make_shared<Model>(NodeVector{c}, ParameterVector{});
pass::Manager manager;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{type_from, type_to}});
manager.register_pass<pass::ConvertPrecision>(precisions_map{{type_from, type_to}});
manager.run_passes(f);
}
auto ops = f->get_ordered_ops();
@ -1336,7 +1336,7 @@ TEST(TransformationTests, ConvertPrecision_keep_precission_sensitive_fp32_with_e
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = true;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);
@ -1383,7 +1383,7 @@ TEST(TransformationTests, ConvertPrecision_keep_precission_sensitive_fp32_with_r
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = true;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);
@ -1431,7 +1431,7 @@ TEST(TransformationTests, ConvertPrecision_reducesum_without_exp) {
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = true;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);
@ -1470,7 +1470,7 @@ TEST(TransformationTests, ConvertPrecision_MarkNormalizationOps_1) {
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = true;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);
@ -1509,7 +1509,7 @@ TEST(TransformationTests, ConvertPrecision_MarkNormalizationOps_2) {
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = true;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);
@ -1587,7 +1587,7 @@ TEST(TransformationTests, ConvertPrecision_keep_precission_sensitive_fp32_t2t_su
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = true;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);
@ -1667,7 +1667,7 @@ TEST(TransformationTests, ConvertPrecision_DivisionByZeroMinimalPattern) {
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = true;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);
@ -1708,7 +1708,7 @@ TEST(TransformationTests, ConvertPrecision_PowWithNegativeExponent) {
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = true;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);
@ -1756,7 +1756,7 @@ TEST(TransformationTests, ConvertPrecision_exp_through_unsqueeze) {
type_to_fuse_map empty_type_to_fuse_map = {};
bool keep_precision_sensitive_in_fp32 = true;
manager.register_pass<pass::ConvertPrecision>(precisions_array{{element::f32, element::f16}},
manager.register_pass<pass::ConvertPrecision>(precisions_map{{element::f32, element::f16}},
empty_type_to_fuse_map,
keep_precision_sensitive_in_fp32);
manager.run_passes(model);

1
src/core/CMakeLists.txt
View File

@ -23,6 +23,7 @@ add_subdirectory(shape_inference)
set(MIXED_SRC
"${CMAKE_CURRENT_SOURCE_DIR}/src/runtime/allocator.cpp"
"${CMAKE_CURRENT_SOURCE_DIR}/src/runtime/itensor.cpp"
"${CMAKE_CURRENT_SOURCE_DIR}/src/runtime/ov_tensor.cpp")
set_property(SOURCE ${MIXED_SRC}

76
src/core/dev_api/openvino/runtime/itensor.hpp Normal file
View File

@ -0,0 +1,76 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#pragma once
#include <memory>
#include "openvino/core/coordinate.hpp"
#include "openvino/core/core_visibility.hpp"
#include "openvino/core/shape.hpp"
#include "openvino/core/type/element_type.hpp"
#include "openvino/runtime/allocator.hpp"
namespace ov {
class OPENVINO_API ITensor : public std::enable_shared_from_this<ITensor> {
public:
/**
* @brief Set new shape for tensor
* @note Memory allocation may happen
* @param shape A new shape
*/
virtual void set_shape(ov::Shape shape) = 0;
/**
* @return A tensor element type
*/
virtual const element::Type& get_element_type() const = 0;
/**
* @return A tensor shape
*/
virtual const Shape& get_shape() const = 0;
/**
* @brief Returns the total number of elements (a product of all the dims or 1 for scalar)
* @return The total number of elements
*/
virtual size_t get_size() const;
/**
* @brief Returns the size of the current Tensor in bytes.
* @return Tensor's size in bytes
*/
virtual size_t get_byte_size() const;
/**
* @return Tensor's strides in bytes
*/
virtual const Strides& get_strides() const = 0;
/**
* @brief Provides an access to the underlaying host memory
* @param type Optional type parameter.
* @note If type parameter is specified, the method throws an exception
* if specified type's fundamental type does not match with tensor element type's fundamental type
* @return A host pointer to tensor memory
*/
virtual void* data(const element::Type& type = {}) const = 0;
/**
* @brief Provides an access to the underlaying host memory casted to type `T`
* @return A host pointer to tensor memory casted to specified type `T`.
* @note Throws exception if specified type does not match with tensor element type
*/
template <typename T, typename datatype = typename std::decay<T>::type>
T* data() const {
return static_cast<T*>(data(element::from<datatype>()));
}
protected:
virtual ~ITensor();
};
} // namespace ov

11
src/core/include/openvino/core/any.hpp
View File

@ -29,6 +29,9 @@ class Plugin;
/** @cond INTERNAL */
class Any;
namespace util {
OPENVINO_API bool equal(std::type_index lhs, std::type_index rhs);
template <typename T, typename = void>
struct Read;
@ -416,8 +419,6 @@ class OPENVINO_API Any {
}
};
static bool equal(std::type_index lhs, std::type_index rhs);
class OPENVINO_API Base : public std::enable_shared_from_this<Base> {
public:
void type_check(const std::type_info&) const;
@ -731,7 +732,7 @@ public:
return true;
}
for (const auto& type_index : _impl->base_type_info()) {
if (equal(type_index, typeid(decay_t<T>))) {
if (util::equal(type_index, typeid(decay_t<T>))) {
return true;
}
}
@ -797,7 +798,7 @@ public:
return *static_cast<decay_t<T>*>(_temp->addressof());
}
for (const auto& type_index : _impl->base_type_info()) {
if (equal(type_index, typeid(decay_t<T>))) {
if (util::equal(type_index, typeid(decay_t<T>))) {
return *static_cast<decay_t<T>*>(_impl->addressof());
}
}
@ -820,7 +821,7 @@ public:
return *static_cast<decay_t<T>*>(_impl->addressof());
}
for (const auto& type_index : _impl->base_type_info()) {
if (equal(type_index, typeid(decay_t<T>))) {
if (util::equal(type_index, typeid(decay_t<T>))) {
return *static_cast<decay_t<T>*>(_impl->addressof());
}
}

5
src/core/include/openvino/op/binary_convolution.hpp
View File

@ -88,9 +88,14 @@ public:
const CoordinateDiff& get_pads_end() const {
return m_pads_end;
}
OPENVINO_DEPRECATED("This method is deprecated and will be removed soon. Please use set_pads_end instead.")
void set_adding_above(const CoordinateDiff& pads_end) {
m_pads_end = pads_end;
}
void set_pads_end(const CoordinateDiff& pads_end) {
m_pads_end = pads_end;
}
/// \return The pad type for convolution.
const PadType& get_auto_pad() const {
return m_auto_pad;

9
src/core/include/openvino/op/topk.hpp
View File

@ -121,7 +121,7 @@ public:
/// \param axis The axis along which the TopK operation should be executed
/// \param mode Specifies whether TopK selects the largest or the smallest elements from each slice
/// \param sort Specifies the order of corresponding elements of the output tensor
/// \param index_element_type Specifies the data type type of of the elements in the 'indices' output tensor.
/// \param index_element_type Specifies the data type of the elements in the 'indices' output tensor.
/// \param stable Specifies whether the equivalent elements should maintain their relative order
/// from the input tensor during sorting.
TopK(const Output<Node>& data,
@ -139,7 +139,7 @@ public:
/// \param axis The axis along which the TopK operation should be executed
/// \param mode Specifies whether TopK selects the largest or the smallest elements from each slice
/// \param sort Specifies the order of corresponding elements of the output tensor
/// \param index_element_type Specifies the data type type of of the elements in the 'indices' output tensor.
/// \param index_element_type Specifies the data type of the elements in the 'indices' output tensor.
/// \param stable Specifies whether the equivalent elements should maintain their relative order
/// from the input tensor during sorting.
TopK(const Output<Node>& data,
@ -153,6 +153,11 @@ public:
bool visit_attributes(AttributeVisitor& visitor) override;
std::shared_ptr<Node> clone_with_new_inputs(const OutputVector& new_args) const override;
OPENVINO_SUPPRESS_DEPRECATED_START
bool evaluate(const HostTensorVector& outputs, const HostTensorVector& inputs) const override;
OPENVINO_SUPPRESS_DEPRECATED_END
bool has_evaluate() const override;
bool get_stable() const {
return m_stable;
}

79
src/core/include/openvino/runtime/allocator.hpp
View File

@ -12,15 +12,20 @@
#include <cstddef>
#include <memory>
#include "openvino/core/any.hpp"
#include "openvino/core/core_visibility.hpp"
#include "openvino/core/deprecated.hpp"
namespace ov {
/**
* @interface AllocatorImpl
* @deprecated This class will be removed in 2024.0 release
* @brief Tries to act like [std::pmr::memory_resource](https://en.cppreference.com/w/cpp/memory/memory_resource)
*/
struct AllocatorImpl : public std::enable_shared_from_this<AllocatorImpl> {
struct OPENVINO_DEPRECATED("Do not inherit from AllocatorImpl. This class will be removed in 2024.0 release. Pass "
"std::pmr::memory_resource like object directly to ov::Allocator") AllocatorImpl
: public std::enable_shared_from_this<AllocatorImpl> {
/**
* @brief A smart pointer containing AllocatorImpl object
*/
@ -61,22 +66,63 @@ class Tensor;
/**
* @brief Wraps allocator implementation to provide safe way to store allocater loaded from shared library
* And constructs default based on `new` `delete` c++ calls allocator if created without parameters
* Accepts any [std::pmr::memory_resource](https://en.cppreference.com/w/cpp/memory/memory_resource) like
* allocator
* @ingroup ov_runtime_cpp_api
*/
class OPENVINO_API Allocator {
AllocatorImpl::Ptr _impl;
std::shared_ptr<void> _so;
/**
* @brief Constructs Tensor from the initialized std::shared_ptr
* @param impl Initialized shared pointer
* @param other Initialized allocator
* @param so Plugin to use. This is required to ensure that Allocator can work properly even if plugin object is
* destroyed.
*/
Allocator(const AllocatorImpl::Ptr& impl, const std::shared_ptr<void>& so);
Allocator(const Allocator& other, const std::shared_ptr<void>& so);
friend class ov::Tensor;
struct Base : public std::enable_shared_from_this<Base> {
virtual void* addressof() = 0;
const void* addressof() const {
return const_cast<Base*>(this)->addressof();
}
virtual const std::type_info& type_info() const = 0;
virtual void* allocate(const size_t bytes, const size_t alignment = alignof(max_align_t)) = 0;
virtual void deallocate(void* handle, const size_t bytes, size_t alignment = alignof(max_align_t)) = 0;
virtual bool is_equal(const Base& other) const = 0;
protected:
~Base() = default;
};
template <typename A>
struct Impl : public Base {
template <typename... Args>
explicit Impl(Args&&... args) : a(std::forward<Args>(args)...) {}
void* addressof() override {
return &a;
}
const std::type_info& type_info() const override {
return typeid(a);
}
void* allocate(const size_t bytes, const size_t alignment = alignof(max_align_t)) override {
return a.allocate(bytes, alignment);
}
void deallocate(void* handle, const size_t bytes, size_t alignment = alignof(max_align_t)) override {
a.deallocate(handle, bytes, alignment);
}
bool is_equal(const Base& other) const override {
if (util::equal(type_info(), other.type_info())) {
return a.is_equal(*static_cast<const A*>(other.addressof()));
}
return false;
}
A a;
};
std::shared_ptr<Base> _impl;
std::shared_ptr<void> _so;
public:
/**
* @brief Destructor preserves unloading order of implementation object and reference to library
@ -104,11 +150,26 @@ public:
/// @return reference to the current object
Allocator& operator=(Allocator&& other) = default;
OPENVINO_SUPPRESS_DEPRECATED_START
/**
* @brief Constructs Allocator from the initialized std::shared_ptr
* @param impl Initialized shared pointer
*/
Allocator(const AllocatorImpl::Ptr& impl);
/**
* @brief Initialize allocator using any allocator like object
* @tparam A Type of allocator
* @param a allocator object
*/
template <
typename A,
typename std::enable_if<!std::is_convertible<A, AllocatorImpl::Ptr>::value &&
!std::is_same<typename std::decay<A>::type, Allocator>::value &&
!std::is_abstract<typename std::decay<A>::type>::value &&
!std::is_convertible<typename std::decay<A>::type, std::shared_ptr<Base>>::value,
bool>::type = true>
Allocator(A&& a) : _impl{std::make_shared<Impl<typename std::decay<A>::type>>(std::forward<A>(a))} {}
OPENVINO_SUPPRESS_DEPRECATED_END
/**
* @brief Allocates memory
@ -129,9 +190,9 @@ public:
void deallocate(void* ptr, const size_t bytes = 0, const size_t alignment = alignof(max_align_t));
/**
* @brief Compares with other AllocatorImpl
* @brief Compares with other Allocator
* @param other Other instance of allocator
* @return `true` if and only if memory allocated from one AllocatorImpl can be deallocated from the other and vice
* @return `true` if and only if memory allocated from one Allocator can be deallocated from the other and vice
* versa
*/
bool operator==(const Allocator& other) const;
@ -149,9 +210,11 @@ public:
explicit operator bool() const noexcept;
};
OPENVINO_SUPPRESS_DEPRECATED_START
namespace runtime {
using ov::Allocator;
using ov::AllocatorImpl;
} // namespace runtime
OPENVINO_SUPPRESS_DEPRECATED_END
} // namespace ov

20
src/core/include/openvino/runtime/tensor.hpp
View File

@ -18,7 +18,6 @@
#include "openvino/runtime/allocator.hpp"
namespace InferenceEngine {
class Blob;
class IAsyncInferRequestWrapper;
class IVariableStateWrapper;
} // namespace InferenceEngine
@ -33,6 +32,8 @@ class VariableState;
class ISyncInferRequest;
class IInferRequestInternalWrapper;
class IVariableStateInternalWrapper;
class ITensor;
class RemoteTensor;
/**
* @brief Tensor API holding host memory
@ -41,8 +42,8 @@ class IVariableStateInternalWrapper;
*/
class OPENVINO_API Tensor {
protected:
std::shared_ptr<InferenceEngine::Blob> _impl; //!< Shared pointer to internal tensor representation
std::vector<std::shared_ptr<void>> _so; //!< Reference to dynamically loaded library
std::shared_ptr<ITensor> _impl; //!< Shared pointer to internal tensor representation
std::vector<std::shared_ptr<void>> _so; //!< Reference to dynamically loaded library
/**
* @brief Constructs Tensor from the initialized std::shared_ptr
@ -50,11 +51,12 @@ protected:
* @param so Plugin to use. This is required to ensure that Tensor can work properly even if plugin object is
* destroyed.
*/
Tensor(const std::shared_ptr<InferenceEngine::Blob>& impl, const std::vector<std::shared_ptr<void>>& so);
Tensor(const std::shared_ptr<ITensor>& impl, const std::vector<std::shared_ptr<void>>& so);
friend class ov::Core;
friend class ov::CoreImpl;
friend class ov::InferRequest;
friend class ov::RemoteTensor;
friend class ov::RemoteContext;
friend class ov::VariableState;
friend class ov::ISyncInferRequest;
@ -103,7 +105,7 @@ public:
* @param shape Tensor shape
* @param allocator allocates memory for internal tensor storage
*/
Tensor(const element::Type type, const Shape& shape, const Allocator& allocator = {});
Tensor(const element::Type& type, const Shape& shape, const Allocator& allocator = {});
/**
* @brief Constructs Tensor using element type and shape. Wraps allocated host memory.
@ -114,7 +116,7 @@ public:
* @param strides Optional strides parameters in bytes. Strides are supposed to be computed automatically based
* on shape and element size
*/
Tensor(const element::Type type, const Shape& shape, void* host_ptr, const Strides& strides = {});
Tensor(const element::Type& type, const Shape& shape, void* host_ptr, const Strides& strides = {});
/**
* @brief Constructs Tensor using port from node. Allocate internal host storage using default allocator
@ -153,12 +155,12 @@ public:
/**
* @return A tensor element type
*/
element::Type get_element_type() const;
const element::Type& get_element_type() const;
/**
* @return A tensor shape
*/
Shape get_shape() const;
const Shape& get_shape() const;
/**
* @brief Copy tensor, destination tensor should have the same element type and shape
@ -198,7 +200,7 @@ public:
* if specified type's fundamental type does not match with tensor element type's fundamental type
* @return A host pointer to tensor memory
*/
void* data(const element::Type type = {}) const;
void* data(const element::Type& type = {}) const;
/**
* @brief Provides an access to the underlaying host memory casted to type `T`

39
src/core/reference/include/ngraph/runtime/reference/topk.hpp
View File

@ -14,9 +14,8 @@
namespace ngraph {
namespace runtime {
namespace reference {
// Had to split out these two functions. They used to be lambda expressions but
// MSVC had difficulty compiling. This way is more explicit.
template <typename T, typename U>
// This used to be lambda expressions but MSVC had difficulty compiling it. This way is more explicit.
template <bool D, typename T, typename U>
inline bool compare_max(const std::tuple<T, U>& a, const std::tuple<T, U>& b) {
// this is intentional to be able to compare floats directly
// without using relative or absolute tolerance
@ -30,19 +29,19 @@ inline bool compare_max(const std::tuple<T, U>& a, const std::tuple<T, U>& b) {
#if defined(__GNUC__)
# pragma GCC diagnostic pop
#endif
return a > b;
if (D)
return std::get<0>(a) > std::get<0>(b);
else
return std::get<0>(a) < std::get<0>(b);
}
template <typename T, typename U>
inline bool compare_min(const std::tuple<T, U>& a, const std::tuple<T, U>& b) {
return a < b;
}
template <typename T, typename U>
inline bool sort_indices_ascending(const std::tuple<T, U>& a, const std::tuple<T, U>& b) {
inline bool compare_indices_ascending(const std::tuple<T, U>& a, const std::tuple<T, U>& b) {
return std::get<1>(a) < std::get<1>(b);
}
// TopK reference implementation provides stable indices output
template <typename T, typename U>
void topk(const T* arg,
U* out_indices,
@ -52,7 +51,7 @@ void topk(const T* arg,
size_t axis,
size_t k,
bool compute_max,
op::v1::TopK::SortType sort = op::v1::TopK::SortType::NONE) {
op::TopKSortType sort = op::TopKSortType::NONE) {
NGRAPH_SUPPRESS_DEPRECATED_START
using namespace std;
// reorder source axis visit order and make "axis" inner most
@ -87,25 +86,25 @@ void topk(const T* arg,
}
// Sort the temp vector
if (compute_max) {
nth_element(workspace.begin(), workspace.begin() + k, workspace.end(), compare_max<T, U>);
nth_element(workspace.begin(), workspace.begin() + k, workspace.end(), compare_max<true, T, U>);
} else {
nth_element(workspace.begin(), workspace.begin() + k, workspace.end(), compare_min<T, U>);
nth_element(workspace.begin(), workspace.begin() + k, workspace.end(), compare_max<false, T, U>);
}
// Write temp vector to output
switch (sort) {
case op::v1::TopK::SortType::NONE:
case op::TopKSortType::NONE:
break;
case op::v1::TopK::SortType::SORT_INDICES:
std::sort(workspace.begin(), workspace.begin() + k, sort_indices_ascending<T, U>);
case op::TopKSortType::SORT_INDICES:
std::sort(workspace.begin(), workspace.begin() + k, compare_indices_ascending<T, U>);
break;
case op::v1::TopK::SortType::SORT_VALUES:
case op::TopKSortType::SORT_VALUES:
if (compute_max)
std::sort(workspace.begin(), workspace.begin() + k, compare_max<T, U>);
std::sort(workspace.begin(), workspace.begin() + k, compare_max<true, T, U>);
else
std::sort(workspace.begin(), workspace.begin() + k, compare_min<T, U>);
std::sort(workspace.begin(), workspace.begin() + k, compare_max<false, T, U>);
}
for (size_t j = 0; j < k; j++) {
tuple<T, U> entry = workspace[j];
const auto& entry = workspace[j];
out_values[out_index] = get<0>(entry);
out_indices[out_index] = get<1>(entry);
out_index += out_axis_stride;

4
src/core/src/any.cpp
View File

@ -9,7 +9,7 @@
namespace ov {
bool Any::equal(std::type_index lhs, std::type_index rhs) {
bool util::equal(std::type_index lhs, std::type_index rhs) {
auto result = lhs == rhs;
#if (defined(__ANDROID__) || defined(__APPLE__)) && defined(__clang__)
if (!result) {
@ -20,7 +20,7 @@ bool Any::equal(std::type_index lhs, std::type_index rhs) {
}
bool Any::Base::is(const std::type_info& other) const {
return Any::equal(type_info(), other);
return util::equal(type_info(), other);
}
void Any::Base::type_check(const std::type_info& type_info_) const {

111
src/core/src/op/topk.cpp
View File

@ -103,6 +103,37 @@ bool evaluate_topk(const HostTensorPtr& arg,
}
return rc;
}
bool TopK_evaluate(const ov::op::util::TopKBase* const node,
const HostTensorVector& outputs,
const HostTensorVector& inputs) {
const auto& arg_shape = inputs[0]->get_shape();
const auto axis = normalize_axis(node, node->get_provided_axis(), arg_shape.size());
const auto compute_max = node->get_mode() == ov::op::TopKMode::MAX;
const auto sort_type = node->get_sort_type();
const auto input_shapes = vector<PartialShape>{inputs[0]->get_partial_shape(), inputs[1]->get_partial_shape()};
const auto constant_data = map<size_t, HostTensorPtr>{{1, inputs[1]}};
auto output_shape = shape_infer(node, input_shapes, constant_data).front().to_shape();
if (output_shape[axis] == 0) {
// the kernel can't handle K (output_shape[axis]) equal 0, use arg_shape[axis] instead.
output_shape[axis] = arg_shape[axis];
}
const size_t k = output_shape[axis];
OPENVINO_ASSERT(k <= arg_shape[axis], "'K' exceeds the dimension of top_k_axis");
// TopK reference implementation provides stable indices output so this parameter is not passed on
return evaluate_topk(inputs[0],
outputs[1],
outputs[0],
output_shape,
axis,
k,
compute_max,
sort_type,
node->get_index_element_type());
}
} // namespace
} // namespace topk
@ -145,34 +176,7 @@ shared_ptr<Node> op::v1::TopK::clone_with_new_inputs(const OutputVector& new_arg
bool op::v1::TopK::evaluate(const HostTensorVector& outputs, const HostTensorVector& inputs) const {
OV_OP_SCOPE(v1_TopK_evaluate);
const auto& arg_shape = inputs[0]->get_shape();
// 1. get axis, mode (max/min), sort_type
auto axis = ngraph::normalize_axis(this, m_axis, arg_shape.size());
auto compute_max = get_mode() == TopKMode::MAX;
auto sort_type = get_sort_type();
const auto input_shapes = std::vector<PartialShape>{inputs[0]->get_partial_shape(), inputs[1]->get_partial_shape()};
const auto constant_data = std::map<size_t, HostTensorPtr>{{1, inputs[1]}};
auto output_shape = shape_infer(this, input_shapes, constant_data).front().to_shape();
if (output_shape[axis] == 0) {
// the kernel can't handle K (output_shape[axis]) equal 0, use arg_shape[axis] instead.
output_shape[axis] = arg_shape[axis];
}
// 2. get value of k
size_t k = output_shape[axis];
OPENVINO_ASSERT(k <= arg_shape[axis], "'K' exceeds the dimension of top_k_axis");
return topk::evaluate_topk(inputs[0],
outputs[1],
outputs[0],
output_shape,
axis,
k,
compute_max,
sort_type,
get_index_element_type());
return topk::TopK_evaluate(this, outputs, inputs);
}
bool op::v1::TopK::has_evaluate() const {
@ -245,34 +249,7 @@ shared_ptr<Node> op::v3::TopK::clone_with_new_inputs(const OutputVector& new_arg
bool op::v3::TopK::evaluate(const HostTensorVector& outputs, const HostTensorVector& inputs) const {
OV_OP_SCOPE(v3_TopK_evaluate);
const auto& arg_shape = inputs[0]->get_shape();
// 1. get axis, mode (max/min), sort_type
auto axis = ngraph::normalize_axis(this, m_axis, arg_shape.size());
auto compute_max = get_mode() == TopKMode::MAX;
auto sort_type = get_sort_type();
const auto input_shapes = std::vector<PartialShape>{inputs[0]->get_partial_shape(), inputs[1]->get_partial_shape()};
const auto constant_data = std::map<size_t, HostTensorPtr>{{1, inputs[1]}};
auto output_shape = shape_infer(this, input_shapes, constant_data).front().to_shape();
if (output_shape[axis] == 0) {
// the kernel can't handle K (output_shape[axis]) equal 0, use arg_shape[axis] instead.
output_shape[axis] = arg_shape[axis];
}
// 2. get value of k
size_t k = output_shape[axis];
OPENVINO_ASSERT(k <= arg_shape[axis], "'K' exceeds the dimension of top_k_axis");
return topk::evaluate_topk(inputs[0],
outputs[1],
outputs[0],
output_shape,
axis,
k,
compute_max,
sort_type,
get_index_element_type());
return topk::TopK_evaluate(this, outputs, inputs);
}
bool op::v3::TopK::has_evaluate() const {
@ -372,3 +349,25 @@ std::shared_ptr<Node> ov::op::v11::TopK::clone_with_new_inputs(const OutputVecto
m_index_element_type,
m_stable);
}
bool ov::op::v11::TopK::evaluate(const HostTensorVector& outputs, const HostTensorVector& inputs) const {
OV_OP_SCOPE(v11_TopK_evaluate);
return topk::TopK_evaluate(this, outputs, inputs);
}
bool ov::op::v11::TopK::has_evaluate() const {
OV_OP_SCOPE(v11_TopK_has_evaluate);
switch (get_input_element_type(0)) {
case ngraph::element::i32:
case ngraph::element::i64:
case ngraph::element::u32:
case ngraph::element::u64:
case ngraph::element::f16:
case ngraph::element::f32:
break;
default:
return false;
}
return true;
}

2
src/core/src/pass/convert_fp32_to_fp16.cpp
View File

@ -15,7 +15,7 @@ using namespace std;
bool ov::pass::ConvertFP32ToFP16::run_on_model(const std::shared_ptr<ov::Model>& f) {
RUN_ON_MODEL_SCOPE(ConvertFP32ToFP16);
ov::pass::Manager m(get_pass_config());
m.register_pass<ov::pass::ConvertPrecision>(precisions_array{{ngraph::element::f32, ngraph::element::f16}});
m.register_pass<ov::pass::ConvertPrecision>(precisions_map{{ngraph::element::f32, ngraph::element::f16}});
m.run_passes(f);
return false;
}

386
src/core/src/pass/serialize.cpp
View File

@ -15,16 +15,17 @@
#include "meta_data.hpp"
#include "ngraph/ops.hpp"
#include "ngraph/opsets/opset.hpp"
#include "ngraph/opsets/opset1.hpp"
#include "openvino/core/coordinate_diff.hpp"
#include "openvino/core/except.hpp"
#include "openvino/core/model.hpp"
#include "openvino/op/util/framework_node.hpp"
#include "openvino/opsets/opset1.hpp"
#include "openvino/pass/constant_folding.hpp"
#include "openvino/util/file_util.hpp"
#include "pugixml.hpp"
#include "transformations/hash.hpp"
#include "transformations/rt_info/primitives_priority_attribute.hpp"
using namespace ngraph;
namespace { // helpers
template <typename Container>
std::string join(const Container& c, const char* glue = ", ") {
@ -44,9 +45,9 @@ struct Edge {
int to_port = 0;
};
// Here operation type names are translated from ngraph convention to IR
// Here operation type names are translated from OpenVINO Model convention to IR
// convention. Most of them are the same, but there are exceptions, e.g
// Constant (ngraph name) and Const (IR name). If there will be more
// Constant (OpenVINO Model name) and Const (IR name). If there will be more
// discrepancies discovered, translations needs to be added here.
const std::unordered_map<std::string, std::string> translate_type_name_translator = {{"Constant", "Const"},
{"PRelu", "PReLU"},
@ -121,7 +122,7 @@ private:
};
void ngfunction_2_ir(pugi::xml_node& node,
const ngraph::Function& f,
const ov::Model& model,
const std::map<std::string, ngraph::OpSet>& custom_opsets,
ConstantWriter& constant_write_handler,
int64_t version,
@ -137,7 +138,7 @@ class XmlSerializer {
public:
explicit XmlSerializer(pugi::xml_node& xml_node) : m_xml_node(xml_node) {}
void serialize(const ngraph::Node::RTMap& rt_info) {
void serialize(const ov::Node::RTMap& rt_info) {
for (const auto& rt_info_name : list_of_names) {
const auto& found_rt_info = rt_info.find(rt_info_name);
if (found_rt_info != rt_info.end()) {
@ -152,85 +153,85 @@ private:
pugi::xml_node& m_xml_node;
};
class RTInfoSerializer : public ngraph::AttributeVisitor {
class RTInfoSerializer : public ov::AttributeVisitor {
pugi::xml_node m_node;
public:
RTInfoSerializer(const pugi::xml_node node) : m_node(node) {}
void on_adapter(const std::string& name, ngraph::ValueAccessor<void>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<void>& adapter) override {
check_attribute_name(name);
if (auto a = ov::as_type<ov::AttributeAdapter<std::set<std::string>>>(&adapter)) {
const auto& value = join(a->get());
m_node.append_attribute(name.c_str()).set_value(value.c_str());
} else {
throw ngraph_error("Unsupported attribute type for serialization: " + name);
OPENVINO_THROW("Unsupported attribute type for serialization: ", name);
}
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<bool>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<bool>& adapter) override {
check_attribute_name(name);
m_node.append_attribute(name.c_str()).set_value(adapter.get());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<std::string>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<std::string>& adapter) override {
check_attribute_name(name);
m_node.append_attribute(name.c_str()).set_value(adapter.get().c_str());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<int64_t>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<int64_t>& adapter) override {
check_attribute_name(name);
m_node.append_attribute(name.c_str()).set_value(static_cast<long long>(adapter.get()));
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<double>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<double>& adapter) override {
check_attribute_name(name);
m_node.append_attribute(name.c_str()).set_value(adapter.get());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<std::vector<int>>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<std::vector<int>>& adapter) override {
check_attribute_name(name);
const auto& value = join(adapter.get());
m_node.append_attribute(name.c_str()).set_value(value.c_str());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<std::vector<int64_t>>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<std::vector<int64_t>>& adapter) override {
check_attribute_name(name);
const auto& value = join(adapter.get());
m_node.append_attribute(name.c_str()).set_value(value.c_str());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<std::vector<uint64_t>>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<std::vector<uint64_t>>& adapter) override {
check_attribute_name(name);
const auto& value = join(adapter.get());
m_node.append_attribute(name.c_str()).set_value(value.c_str());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<std::vector<float>>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<std::vector<float>>& adapter) override {
check_attribute_name(name);
const auto& value = join(adapter.get());
m_node.append_attribute(name.c_str()).set_value(value.c_str());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<std::vector<std::string>>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<std::vector<std::string>>& adapter) override {
check_attribute_name(name);
const auto& value = join(adapter.get());
m_node.append_attribute(name.c_str()).set_value(value.c_str());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<std::shared_ptr<Function>>& adapter) override {
throw ngraph_error("Model type is unsupported for rt info serialization");
void on_adapter(const std::string& name, ov::ValueAccessor<std::shared_ptr<ov::Model>>& adapter) override {
OPENVINO_THROW("Model type is unsupported for rt info serialization");
}
void check_attribute_name(const std::string& name) const {
if (name == "name" || name == "version") {
throw ngraph_error("Attribute key with name: " + name + " is not allowed. Please use another name");
OPENVINO_THROW("Attribute key with name: ", name, " is not allowed. Please use another name");
}
}
};
} // namespace rt_info
class XmlSerializer : public ngraph::AttributeVisitor {
class XmlSerializer : public ov::AttributeVisitor {
pugi::xml_node& m_xml_node;
const std::string& m_node_type_name;
const std::map<std::string, ngraph::OpSet>& m_custom_opsets;
@ -239,7 +240,7 @@ class XmlSerializer : public ngraph::AttributeVisitor {
bool m_deterministic;
template <typename T>
std::string create_atribute_list(ngraph::ValueAccessor<std::vector<T>>& adapter) {
std::string create_atribute_list(ov::ValueAccessor<std::vector<T>>& adapter) {
return join(adapter.get());
}
@ -263,7 +264,7 @@ class XmlSerializer : public ngraph::AttributeVisitor {
}
void input_descriptions_on_adapter(
const std::vector<std::shared_ptr<ngraph::op::util::MultiSubGraphOp::InputDescription>>& input_descriptions,
const std::vector<std::shared_ptr<ov::op::util::MultiSubGraphOp::InputDescription>>& input_descriptions,
const std::vector<std::string>& parameter_mapping,
const std::vector<std::string>& result_mapping,
pugi::xml_node& port_map,
@ -280,14 +281,14 @@ class XmlSerializer : public ngraph::AttributeVisitor {
.set_value(parameter_mapping[input_description->m_body_parameter_index].c_str());
if (auto slice_input =
ov::as_type_ptr<ngraph::op::util::SubGraphOp::SliceInputDescription>(input_description)) {
ov::as_type_ptr<ov::op::util::SubGraphOp::SliceInputDescription>(input_description)) {
input.prepend_attribute("axis").set_value(static_cast<long long>(slice_input->m_axis));
input.append_attribute("start").set_value(static_cast<long long>(slice_input->m_start));
input.append_attribute("end").set_value(static_cast<long long>(slice_input->m_end));
input.append_attribute("stride").set_value(static_cast<long long>(slice_input->m_stride));
input.append_attribute("part_size").set_value(static_cast<long long>(slice_input->m_part_size));
} else if (auto merged_input =
ov::as_type_ptr<ngraph::op::util::SubGraphOp::MergedInputDescription>(input_description)) {
ov::as_type_ptr<ov::op::util::SubGraphOp::MergedInputDescription>(input_description)) {
pugi::xml_node back_edges = m_xml_node.parent().child("back_edges");
if (!back_edges) {
back_edges = m_xml_node.parent().insert_child_after("back_edges", port_map);
@ -301,12 +302,12 @@ class XmlSerializer : public ngraph::AttributeVisitor {
}
void output_descriptions_on_adapter(
const std::vector<std::shared_ptr<ngraph::op::util::MultiSubGraphOp::OutputDescription>>& output_descriptions,
const std::vector<std::shared_ptr<ov::op::util::MultiSubGraphOp::OutputDescription>>& output_descriptions,
const uint32_t& input_count,
const std::vector<std::string>& result_mapping,
pugi::xml_node& port_map,
const std::string& portmap_name) {
NGRAPH_CHECK(!result_mapping.empty(), "No results found in body Model.");
OPENVINO_ASSERT(!result_mapping.empty(), "No results found in body Model.");
if (!port_map) {
port_map = m_xml_node.parent().insert_child_before(portmap_name.c_str(), m_xml_node.parent().first_child());
@ -320,7 +321,7 @@ class XmlSerializer : public ngraph::AttributeVisitor {
.set_value(result_mapping[output_description->m_body_value_index].c_str());
if (auto concat_output =
ov::as_type_ptr<ngraph::op::util::SubGraphOp::ConcatOutputDescription>(output_description)) {
ov::as_type_ptr<ov::op::util::SubGraphOp::ConcatOutputDescription>(output_description)) {
output.prepend_attribute("axis").set_value(static_cast<long long>(concat_output->m_axis));
output.append_attribute("start").set_value(static_cast<long long>(concat_output->m_start));
output.append_attribute("end").set_value(static_cast<long long>(concat_output->m_end));
@ -330,11 +331,11 @@ class XmlSerializer : public ngraph::AttributeVisitor {
}
}
void special_body_ports_on_adapter(const ngraph::op::v5::Loop::SpecialBodyPorts& special_body_ports,
void special_body_ports_on_adapter(const ov::op::v5::Loop::SpecialBodyPorts& special_body_ports,
const std::vector<std::string>& parameter_mapping,
const std::vector<std::string>& result_mapping,
pugi::xml_node& port_map) {
NGRAPH_CHECK(port_map, "port_map section not found, purpose attribute cannot be added.");
OPENVINO_ASSERT(port_map, "port_map section not found, purpose attribute cannot be added.");
if (special_body_ports.current_iteration_input_idx != -1) {
pugi::xml_node iter_input = port_map.append_child("input");
@ -367,7 +368,7 @@ public:
m_version(version),
m_deterministic(deterministic) {}
void on_adapter(const std::string& name, ngraph::ValueAccessor<void>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<void>& adapter) override {
using BodyTargetNames = std::tuple<std::string, std::string, std::vector<std::string>>;
const std::vector<BodyTargetNames> body_names = {
@ -424,32 +425,28 @@ public:
pugi::xml_node port_map = m_xml_node.parent().child(portmap_name.c_str());
// Bodies can be without parameters(dependig on constants), but can not be without results
NGRAPH_CHECK(!result_mapping.empty(), "No results found in body Model.");
OPENVINO_ASSERT(!result_mapping.empty(), "No results found in body Model.");
// TI, Loop do not have attributtes as regular ops, it is necessary to append "port_map" and
// "back_edges" to layer above (m_xml_node.parent()) as in ngfunction_2_ir() layer (here "m_xml_node")
// with empty attributes is removed.
if (const auto& a = ngraph::as_type<ngraph::AttributeAdapter<
std::vector<std::shared_ptr<ngraph::op::util::MultiSubGraphOp::InputDescription>>>>(&adapter)) {
if (const auto& a = ov::as_type<ov::AttributeAdapter<
std::vector<std::shared_ptr<ov::op::util::MultiSubGraphOp::InputDescription>>>>(&adapter)) {
input_descriptions_on_adapter(a->get(), parameter_mapping, result_mapping, port_map, portmap_name);
} else if (const auto& a = ngraph::as_type<ngraph::AttributeAdapter<
std::vector<std::shared_ptr<ngraph::op::util::MultiSubGraphOp::OutputDescription>>>>(
&adapter)) {
} else if (const auto& a = ov::as_type<ov::AttributeAdapter<
std::vector<std::shared_ptr<ov::op::util::MultiSubGraphOp::OutputDescription>>>>(&adapter)) {
uint32_t op_input_count = 0;
for (auto c = m_xml_node.parent().child("input").first_child(); !c.empty(); c = c.next_sibling()) {
op_input_count++;
}
output_descriptions_on_adapter(a->get(), op_input_count, result_mapping, port_map, portmap_name);
} else if (const auto& a =
ngraph::as_type<ngraph::AttributeAdapter<ngraph::op::v5::Loop::SpecialBodyPorts>>(
&adapter)) {
ov::as_type<ov::AttributeAdapter<ov::op::v5::Loop::SpecialBodyPorts>>(&adapter)) {
special_body_ports_on_adapter(a->get(), parameter_mapping, result_mapping, port_map);
}
} else if (const auto& a =
ngraph::as_type<ngraph::AttributeAdapter<std::shared_ptr<ngraph::Variable>>>(&adapter)) {
} else if (const auto& a = ov::as_type<ov::AttributeAdapter<std::shared_ptr<ngraph::Variable>>>(&adapter)) {
m_xml_node.append_attribute(name.c_str()).set_value(a->get()->get_info().variable_id.c_str());
} else if (const auto& a =
ngraph::as_type<ngraph::AttributeAdapter<std::shared_ptr<ngraph::runtime::AlignedBuffer>>>(
&adapter)) {
ov::as_type<ov::AttributeAdapter<std::shared_ptr<ngraph::runtime::AlignedBuffer>>>(&adapter)) {
if (name == "value" && translate_type_name(m_node_type_name) == "Const") {
const int64_t size = a->get()->size();
int64_t offset = m_constant_write_handler.write(static_cast<const char*>(a->get()->get_ptr()), size);
@ -457,8 +454,7 @@ public:
m_xml_node.append_attribute("offset").set_value(static_cast<unsigned long long>(offset));
m_xml_node.append_attribute("size").set_value(static_cast<unsigned long long>(size));
}
} else if (const auto& a =
ngraph::as_type<ngraph::AttributeAdapter<ov::op::util::FrameworkNodeAttrs>>(&adapter)) {
} else if (const auto& a = ov::as_type<ov::AttributeAdapter<ov::op::util::FrameworkNodeAttrs>>(&adapter)) {
const auto& attrs = a->get();
// Update type and version attributes
@ -479,16 +475,16 @@ public:
for (const auto& attr : attrs) {
m_xml_node.append_attribute(attr.first.c_str()).set_value(attr.second.c_str());
}
} else if (const auto& a = ngraph::as_type<ngraph::AttributeAdapter<ngraph::element::TypeVector>>(&adapter)) {
} else if (const auto& a = ov::as_type<ov::AttributeAdapter<ov::element::TypeVector>>(&adapter)) {
const auto& attrs = a->get();
m_xml_node.append_attribute(name.c_str()).set_value(join(attrs).c_str());
} else if (const auto& a = ngraph::as_type<ngraph::AttributeAdapter<ov::PartialShape>>(&adapter)) {
} else if (const auto& a = ov::as_type<ov::AttributeAdapter<ov::PartialShape>>(&adapter)) {
const auto& attrs = a->get();
auto shape_str = attrs.to_string();
if (shape_str[0] == '[' && shape_str[shape_str.size() - 1] == ']')
shape_str = shape_str.substr(1, shape_str.size() - 2);
m_xml_node.append_attribute(name.c_str()).set_value(shape_str.c_str());
} else if (const auto& a = ngraph::as_type<ngraph::AttributeAdapter<ov::Dimension>>(&adapter)) {
} else if (const auto& a = ov::as_type<ov::AttributeAdapter<ov::Dimension>>(&adapter)) {
const auto& attrs = a->get();
std::stringstream dim_str_stream;
dim_str_stream << attrs;
@ -497,38 +493,38 @@ public:
dim_str = dim_str.substr(1, dim_str.size() - 2);
m_xml_node.append_attribute(name.c_str()).set_value(dim_str.c_str());
} else {
throw ngraph_error("Unsupported attribute type for serialization: " + name);
OPENVINO_THROW("Unsupported attribute type for serialization: ", name);
}
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<bool>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<bool>& adapter) override {
m_xml_node.append_attribute(name.c_str()).set_value(adapter.get());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<std::string>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<std::string>& adapter) override {
m_xml_node.append_attribute(name.c_str()).set_value(adapter.get().c_str());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<int64_t>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<int64_t>& adapter) override {
m_xml_node.append_attribute(name.c_str()).set_value(static_cast<long long>(adapter.get()));
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<double>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<double>& adapter) override {
m_xml_node.append_attribute(name.c_str()).set_value(adapter.get());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<std::vector<int>>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<std::vector<int>>& adapter) override {
m_xml_node.append_attribute(name.c_str()).set_value(create_atribute_list(adapter).c_str());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<std::vector<int64_t>>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<std::vector<int64_t>>& adapter) override {
m_xml_node.append_attribute(name.c_str()).set_value(create_atribute_list(adapter).c_str());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<std::vector<uint64_t>>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<std::vector<uint64_t>>& adapter) override {
m_xml_node.append_attribute(name.c_str()).set_value(create_atribute_list(adapter).c_str());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<std::vector<float>>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<std::vector<float>>& adapter) override {
m_xml_node.append_attribute(name.c_str()).set_value(create_atribute_list(adapter).c_str());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<std::vector<std::string>>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<std::vector<std::string>>& adapter) override {
m_xml_node.append_attribute(name.c_str()).set_value(create_atribute_list(adapter).c_str());
}
void on_adapter(const std::string& name, ngraph::ValueAccessor<std::shared_ptr<Function>>& adapter) override {
void on_adapter(const std::string& name, ov::ValueAccessor<std::shared_ptr<ov::Model>>& adapter) override {
if (name.find("body") != std::string::npos) {
// name that contains subgraphs: body{n}, then_body, else_body
// TI, Loop do not have attributtes as regular ops, it is necessary to append "body"
@ -551,25 +547,25 @@ public:
m_version,
m_deterministic);
} else {
NGRAPH_CHECK(false, "Unsupported Model name.");
OPENVINO_THROW("Unsupported Model name.");
}
}
};
const std::unordered_map<ngraph::Node*, int> create_layer_ids(const ngraph::Function& f) {
std::unordered_map<ngraph::Node*, int> layer_ids;
const std::unordered_map<ov::Node*, int> create_layer_ids(const ov::Model& model) {
std::unordered_map<ov::Node*, int> layer_ids;
int id = 0;
for (const auto& node : f.get_ordered_ops()) {
for (const auto& node : model.get_ordered_ops()) {
layer_ids[node.get()] = id++;
}
return layer_ids;
}
const std::vector<Edge> create_edge_mapping(const std::unordered_map<ngraph::Node*, int>& layer_ids,
const ngraph::Function& f) {
const std::vector<Edge> create_edge_mapping(const std::unordered_map<ov::Node*, int>& layer_ids,
const ov::Model& model) {
std::vector<Edge> edges;
for (const auto& node : f.get_ordered_ops()) {
if (ngraph::op::is_parameter(node)) {
for (const auto& node : model.get_ordered_ops()) {
if (ov::op::util::is_parameter(node)) {
continue;
}
@ -578,8 +574,8 @@ const std::vector<Edge> create_edge_mapping(const std::unordered_map<ngraph::Nod
auto source_node = source_output.get_node();
auto current_node = i.get_node();
NGRAPH_CHECK(layer_ids.find(source_node) != layer_ids.end(), "Internal error");
NGRAPH_CHECK(layer_ids.find(current_node) != layer_ids.end(), "Internal error");
OPENVINO_ASSERT(layer_ids.find(source_node) != layer_ids.end(), "Internal error");
OPENVINO_ASSERT(layer_ids.find(current_node) != layer_ids.end(), "Internal error");
Edge e{};
e.from_layer = layer_ids.find(source_node)->second;
@ -595,7 +591,7 @@ const std::vector<Edge> create_edge_mapping(const std::unordered_map<ngraph::Nod
return edges;
}
std::string get_opset_name(const ngraph::Node* n, const std::map<std::string, ngraph::OpSet>& custom_opsets) {
std::string get_opset_name(const ov::Node* n, const std::map<std::string, ngraph::OpSet>& custom_opsets) {
OPENVINO_ASSERT(n != nullptr);
// Try to find opset name from RT info
@ -624,47 +620,45 @@ std::string get_opset_name(const ngraph::Node* n, const std::map<std::string, ng
return "experimental";
}
std::string get_precision_name(const ngraph::element::Type& elem_type) {
std::string get_precision_name(const ov::element::Type& elem_type) {
switch (elem_type) {
case ::ngraph::element::Type_t::undefined:
case ::ngraph::element::Type_t::dynamic:
case ::ov::element::Type_t::undefined:
case ::ov::element::Type_t::dynamic:
return "UNSPECIFIED";
case ::ngraph::element::Type_t::f16:
case ::ov::element::Type_t::f16:
return "FP16";
case ::ngraph::element::Type_t::f32:
case ::ov::element::Type_t::f32:
return "FP32";
case ::ngraph::element::Type_t::bf16:
case ::ov::element::Type_t::bf16:
return "BF16";
case ::ngraph::element::Type_t::f64:
case ::ov::element::Type_t::f64:
return "FP64";
case ::ngraph::element::Type_t::i4:
case ::ov::element::Type_t::i4:
return "I4";
case ::ngraph::element::Type_t::i8:
case ::ov::element::Type_t::i8:
return "I8";
case ::ngraph::element::Type_t::i16:
case ::ov::element::Type_t::i16:
return "I16";
case ::ngraph::element::Type_t::i32:
case ::ov::element::Type_t::i32:
return "I32";
case ::ngraph::element::Type_t::i64:
case ::ov::element::Type_t::i64:
return "I64";
case ::ngraph::element::Type_t::u4:
case ::ov::element::Type_t::u4:
return "U4";
case ::ngraph::element::Type_t::u8:
case ::ov::element::Type_t::u8:
return "U8";
case ::ngraph::element::Type_t::u16:
case ::ov::element::Type_t::u16:
return "U16";
case ::ngraph::element::Type_t::u32:
case ::ov::element::Type_t::u32:
return "U32";
case ::ngraph::element::Type_t::u64:
case ::ov::element::Type_t::u64:
return "U64";
case ::ngraph::element::Type_t::u1:
case ::ov::element::Type_t::u1:
return "BIN";
case ::ngraph::element::Type_t::boolean:
case ::ov::element::Type_t::boolean:
return "BOOL";
default:
std::stringstream msg;
msg << "Unsupported precision: " << elem_type;
throw ngraph_error(msg.str());
OPENVINO_THROW("Unsupported precision: ", elem_type);
}
}
@ -697,7 +691,7 @@ bool is_name_auto_generated(const T& n) {
}
// TODO: remove when CNNNetwork will be supporting not-unique names
std::string get_node_unique_name(std::unordered_set<std::string>& unique_names, const ngraph::Node* n) {
std::string get_node_unique_name(std::unordered_set<std::string>& unique_names, const ov::Node* n) {
std::string name = n->get_friendly_name();
if (unique_names.find(name) != unique_names.end()) {
name = generate_unique_name(unique_names, name, 0);
@ -706,7 +700,7 @@ std::string get_node_unique_name(std::unordered_set<std::string>& unique_names,
return name;
}
void visit_exec_graph_node(pugi::xml_node& layer, const ngraph::Node* n) {
void visit_exec_graph_node(pugi::xml_node& layer, const ov::Node* n) {
auto data = layer.child("data");
for (const auto& param : n->get_rt_info()) {
if (param.second.is<std::string>()) {
@ -723,9 +717,9 @@ void visit_exec_graph_node(pugi::xml_node& layer, const ngraph::Node* n) {
}
}
bool is_exec_graph(const ngraph::Function& f) {
bool is_exec_graph(const ov::Model& model) {
// go over all operations and check whether performance stat is set
for (const auto& op : f.get_ops()) {
for (const auto& op : model.get_ops()) {
const auto& rtInfo = op->get_rt_info();
if (rtInfo.find("execTimeMcs") != rtInfo.end()) {
return true;
@ -734,55 +728,75 @@ bool is_exec_graph(const ngraph::Function& f) {
return false;
}
void auto_pad_resolving(ov::Node* node) {
class PaddingsFixer {
private:
ov::Node* m_node;
ov::OutputVector m_parameters;
std::shared_ptr<ov::Node> m_cloned_node;
const std::set<ov::op::PadType> pad_agnostic_types = {
ov::op::PadType::SAME_LOWER,
ov::op::PadType::SAME_UPPER,
ov::op::PadType::VALID,
ov::op::PadType::AUTO,
};
if (auto op = as_type<opset1::Convolution>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
op->set_pads_begin(CoordinateDiff(op->get_pads_begin().size(), 0));
op->set_pads_end(CoordinateDiff(op->get_pads_end().size(), 0));
template <class T, class P>
void clone_op_and_fix_paddings(const T* op) {
for (const auto& input : op->inputs()) {
m_parameters.emplace_back(
std::make_shared<ov::opset1::Parameter>(input.get_element_type(), input.get_partial_shape()));
}
} else if (auto op = as_type<opset1::GroupConvolution>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
op->set_pads_begin(CoordinateDiff(op->get_pads_begin().size(), 0));
op->set_pads_end(CoordinateDiff(op->get_pads_end().size(), 0));
}
} else if (auto op = as_type<opset1::ConvolutionBackpropData>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
op->set_pads_begin(CoordinateDiff(op->get_pads_begin().size(), 0));
op->set_pads_end(CoordinateDiff(op->get_pads_end().size(), 0));
}
} else if (auto op = as_type<opset1::GroupConvolutionBackpropData>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
op->set_pads_begin(CoordinateDiff(op->get_pads_begin().size(), 0));
op->set_pads_end(CoordinateDiff(op->get_pads_end().size(), 0));
}
} else if (auto op = as_type<ngraph::op::util::DeformableConvolutionBase>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
op->set_pads_begin(CoordinateDiff(op->get_pads_begin().size(), 0));
op->set_pads_end(CoordinateDiff(op->get_pads_end().size(), 0));
}
} else if (auto op = as_type<opset1::BinaryConvolution>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
op->set_pads_begin(CoordinateDiff(op->get_pads_begin().size(), 0));
op->set_adding_above(CoordinateDiff(op->get_pads_end().size(), 0));
}
} else if (auto op = as_type<opset1::AvgPool>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
op->set_pads_begin(Shape(op->get_pads_begin().size(), 0));
op->set_pads_end(Shape(op->get_pads_end().size(), 0));
}
} else if (auto op = as_type<ngraph::op::util::MaxPoolBase>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
op->set_pads_begin(Shape(op->get_pads_begin().size(), 0));
op->set_pads_end(Shape(op->get_pads_end().size(), 0));
m_cloned_node = op->clone_with_new_inputs(m_parameters);
auto typed_cloned_node = std::dynamic_pointer_cast<T>(m_cloned_node);
OPENVINO_ASSERT(typed_cloned_node);
typed_cloned_node->set_pads_begin(P(op->get_pads_begin().size(), 0));
typed_cloned_node->set_pads_end(P(op->get_pads_end().size(), 0));
m_node = m_cloned_node.get();
}
public:
ov::Node* get_node() {
return m_node;
}
explicit PaddingsFixer(ov::Node* node) : m_node(node) {
if (auto op = ov::as_type<ov::opset1::Convolution>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
clone_op_and_fix_paddings<ov::opset1::Convolution, ov::CoordinateDiff>(op);
}
} else if (auto op = ov::as_type<ov::opset1::GroupConvolution>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
clone_op_and_fix_paddings<ov::opset1::GroupConvolution, ov::CoordinateDiff>(op);
}
} else if (auto op = ov::as_type<ov::opset1::ConvolutionBackpropData>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
clone_op_and_fix_paddings<ov::opset1::ConvolutionBackpropData, ov::CoordinateDiff>(op);
}
} else if (auto op = ov::as_type<ov::opset1::GroupConvolutionBackpropData>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
clone_op_and_fix_paddings<ov::opset1::GroupConvolutionBackpropData, ov::CoordinateDiff>(op);
}
} else if (auto op = ov::as_type<ov::op::util::DeformableConvolutionBase>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
clone_op_and_fix_paddings<ov::op::util::DeformableConvolutionBase, ov::CoordinateDiff>(op);
}
} else if (auto op = ov::as_type<ov::opset1::BinaryConvolution>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
clone_op_and_fix_paddings<ov::opset1::BinaryConvolution, ov::CoordinateDiff>(op);
}
} else if (auto op = ov::as_type<ov::opset1::AvgPool>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
clone_op_and_fix_paddings<ov::opset1::AvgPool, ov::Shape>(op);
}
} else if (auto op = ov::as_type<ov::op::util::MaxPoolBase>(node)) {
if (pad_agnostic_types.count(op->get_auto_pad())) {
clone_op_and_fix_paddings<ov::op::util::MaxPoolBase, ov::Shape>(op);
}
}
}
}
};
void serialize_rt_info(pugi::xml_node& root, const std::string& name, const ov::Any& data) {
auto child = root.append_child(name.c_str());
@ -804,48 +818,48 @@ void serialize_rt_info(pugi::xml_node& root, const std::string& name, const ov::
}
void ngfunction_2_ir(pugi::xml_node& netXml,
const ngraph::Function& f,
const ov::Model& model,
const std::map<std::string, ngraph::OpSet>& custom_opsets,
ConstantWriter& constant_node_write_handler,
int64_t version,
bool deterministic) {
// If determinism is not required, include auto-generated names into xml
if (!deterministic || !is_name_auto_generated(f)) {
netXml.append_attribute("name").set_value(f.get_friendly_name().c_str());
if (!deterministic || !is_name_auto_generated(model)) {
netXml.append_attribute("name").set_value(model.get_friendly_name().c_str());
}
netXml.append_attribute("version").set_value(static_cast<long long>(version));
pugi::xml_node layers = netXml.append_child("layers");
const std::unordered_map<ngraph::Node*, int> layer_ids = create_layer_ids(f);
const std::unordered_map<ov::Node*, int> layer_ids = create_layer_ids(model);
std::unordered_set<std::string> unique_names;
const bool exec_graph = is_exec_graph(f);
const bool exec_graph = is_exec_graph(model);
auto sorted_ops = f.get_ordered_ops();
auto sorted_ops = model.get_ordered_ops();
if (version >= 11) {
std::vector<std::shared_ptr<ov::Node>> result;
result.reserve(sorted_ops.size());
for (const auto& param : f.get_parameters()) {
for (const auto& param : model.get_parameters()) {
result.emplace_back(param);
}
for (auto&& node : sorted_ops) {
if (!ov::op::util::is_parameter(node) && !ov::op::util::is_output(node) && !ov::op::util::is_sink(node))
result.emplace_back(node);
}
for (const auto& sink : f.get_sinks()) {
for (const auto& sink : model.get_sinks()) {
result.emplace_back(sink);
}
for (const auto& res : f.get_results()) {
for (const auto& res : model.get_results()) {
result.emplace_back(res);
}
sorted_ops = result;
}
for (const auto& n : sorted_ops) {
ngraph::Node* node = n.get();
ov::Node* node = n.get();
const std::string& node_type_name{node->get_type_name()};
NGRAPH_CHECK(layer_ids.find(node) != layer_ids.end(), "Internal error");
OPENVINO_ASSERT(layer_ids.find(node) != layer_ids.end(), "Internal error");
// <layers>
pugi::xml_node layer = layers.append_child("layer");
layer.append_attribute("id").set_value(layer_ids.find(node)->second);
@ -894,7 +908,7 @@ void ngfunction_2_ir(pugi::xml_node& netXml,
pugi::xml_node input = layer.append_child("input");
for (auto& i : node->inputs()) {
// WA for LSTMCellv0, peephole input shall not be serialized
if (i.get_index() == 6 && dynamic_cast<opset1::LSTMCell*>(node)) {
if (i.get_index() == 6 && dynamic_cast<ov::opset1::LSTMCell*>(node)) {
port_id++;
continue;
}
@ -920,7 +934,7 @@ void ngfunction_2_ir(pugi::xml_node& netXml,
}
}
// <layers/output>
if ((node->get_output_size() > 0) && !ngraph::op::is_output(node)) {
if ((node->get_output_size() > 0) && !ov::op::util::is_output(node)) {
pugi::xml_node output = layer.append_child("output");
for (auto& o : node->outputs()) {
pugi::xml_node port = output.append_child("port");
@ -960,9 +974,17 @@ void ngfunction_2_ir(pugi::xml_node& netXml,
}
// fill <data> general attributes
auto_pad_resolving(node); // Backward compatibility: clear padding values for nodes with auto_pad
XmlSerializer visitor(data, node_type_name, custom_opsets, constant_node_write_handler, version, deterministic);
NGRAPH_CHECK(node->visit_attributes(visitor), "Visitor API is not supported in ", node);
{
// Backward compatibility: clear padding values for nodes with auto_pad
PaddingsFixer fixed_node(node);
XmlSerializer visitor(data,
node_type_name,
custom_opsets,
constant_node_write_handler,
version,
deterministic);
OPENVINO_ASSERT(fixed_node.get_node()->visit_attributes(visitor), "Visitor API is not supported in ", node);
}
rt_info::XmlSerializer{data}.serialize(node->get_rt_info());
if (exec_graph) {
@ -975,12 +997,13 @@ void ngfunction_2_ir(pugi::xml_node& netXml,
}
}
// <edges>
const std::vector<Edge> edge_mapping = create_edge_mapping(layer_ids, f);
const std::vector<Edge> edge_mapping = create_edge_mapping(layer_ids, model);
pugi::xml_node edges = netXml.append_child("edges");
auto ordered_ops = model.get_ordered_ops();
for (auto e : edge_mapping) {
// WA for LSTMCellv0, peephole input shall not be serialized
if (e.to_port == 6) {
auto type_info = f.get_ordered_ops()[e.to_layer]->get_type_info();
const auto& type_info = ordered_ops[e.to_layer]->get_type_info();
OPENVINO_SUPPRESS_DEPRECATED_START
if (!strcmp(type_info.name, "LSTMCell") && type_info.version == 0) {
continue;
@ -996,7 +1019,7 @@ void ngfunction_2_ir(pugi::xml_node& netXml,
// Serialize rt info
pugi::xml_node rt_info_node = netXml.append_child("rt_info");
for (const auto& it : f.get_rt_info()) {
for (const auto& it : model.get_rt_info()) {
// Skip IR version
if (it.first == "version")
continue;
@ -1005,12 +1028,12 @@ void ngfunction_2_ir(pugi::xml_node& netXml,
}
std::string valid_xml_path(const std::string& path) {
NGRAPH_CHECK(path.length() > 4, "Path for xml file is to short: \"" + path + "\"");
OPENVINO_ASSERT(path.length() > 4, "Path for xml file is to short: \"" + path + "\"");
const char* const extension = ".xml";
const bool has_xml_extension = path.rfind(extension) == path.size() - std::strlen(extension);
NGRAPH_CHECK(has_xml_extension,
"Path for xml file doesn't contains file name with 'xml' extension: \"" + path + "\"");
OPENVINO_ASSERT(has_xml_extension,
"Path for xml file doesn't contains file name with 'xml' extension: \"" + path + "\"");
return path;
}
@ -1028,33 +1051,33 @@ std::string provide_bin_path(const std::string& xmlPath, const std::string& binP
void serializeFunc(std::ostream& xml_file,
std::ostream& bin_file,
std::shared_ptr<ov::Model> f,
std::shared_ptr<ov::Model> model,
ov::pass::Serialize::Version ver,
const std::map<std::string, ngraph::OpSet>& custom_opsets,
bool deterministic = false) {
auto version = static_cast<int64_t>(ver);
auto& rt_info = f->get_rt_info();
auto& rt_info = model->get_rt_info();
if (rt_info.count("version")) {
version = rt_info.at("version").as<int64_t>();
}
if (version != static_cast<int64_t>(ver) && ver != ov::pass::Serialize::Version::UNSPECIFIED)
throw ngraph_error("Cannot serialize Model to incompatible IR version");
OPENVINO_THROW("Cannot serialize Model to incompatible IR version");
if (version == static_cast<int64_t>(ov::pass::Serialize::Version::UNSPECIFIED))
version = static_cast<int64_t>(ov::pass::Serialize::Version::IR_V11);
if (version != static_cast<int64_t>(ov::pass::Serialize::Version::IR_V10) &&
version != static_cast<int64_t>(ov::pass::Serialize::Version::IR_V11)) {
throw ngraph_error("Unsupported version");
OPENVINO_THROW("Unsupported version");
}
std::string name = "net";
pugi::xml_document xml_doc;
pugi::xml_node net_node = xml_doc.append_child(name.c_str());
ConstantWriter constant_write_handler(bin_file);
XmlSerializer visitor(net_node, name, custom_opsets, constant_write_handler, version, deterministic);
visitor.on_attribute(name, f);
visitor.on_attribute(name, model);
xml_doc.save(xml_file);
xml_file.flush();
@ -1064,26 +1087,25 @@ void serializeFunc(std::ostream& xml_file,
} // namespace
namespace ov {
bool pass::Serialize::run_on_model(const std::shared_ptr<ngraph::Function>& f_orig) {
bool pass::Serialize::run_on_model(const std::shared_ptr<ov::Model>& model) {
RUN_ON_FUNCTION_SCOPE(Serialize);
auto f = f_orig->clone();
if (m_xmlFile && m_binFile) {
serializeFunc(*m_xmlFile, *m_binFile, f, m_version, m_custom_opsets);
serializeFunc(*m_xmlFile, *m_binFile, model, m_version, m_custom_opsets);
} else {
auto xmlDir = ov::util::get_directory(m_xmlPath);
if (xmlDir != m_xmlPath)
ov::util::create_directory_recursive(xmlDir);
std::ofstream bin_file(m_binPath, std::ios::out | std::ios::binary);
NGRAPH_CHECK(bin_file, "Can't open bin file: \"" + m_binPath + "\"");
OPENVINO_ASSERT(bin_file, "Can't open bin file: \"" + m_binPath + "\"");
// create xml file
std::ofstream xml_file(m_xmlPath, std::ios::out);
NGRAPH_CHECK(xml_file, "Can't open xml file: \"" + m_xmlPath + "\"");
OPENVINO_ASSERT(xml_file, "Can't open xml file: \"" + m_xmlPath + "\"");
try {
serializeFunc(xml_file, bin_file, f, m_version, m_custom_opsets);
} catch (const ngraph::CheckFailure&) {
serializeFunc(xml_file, bin_file, model, m_version, m_custom_opsets);
} catch (const ov::AssertFailure&) {
// optimization decision was made to create .bin file upfront and
// write to it directly instead of buffering its content in memory,
// hence we need to delete it here in case of failure
@ -1095,7 +1117,7 @@ bool pass::Serialize::run_on_model(const std::shared_ptr<ngraph::Function>& f_or
}
}
// Return false because we didn't change nGraph Function
// Return false because we didn't change ov Model
return false;
}
@ -1140,7 +1162,7 @@ pass::StreamSerialize::StreamSerialize(std::ostream& stream,
m_version(version) {
if (version != Serialize::Version::UNSPECIFIED && version != Serialize::Version::IR_V10 &&
version != Serialize::Version::IR_V11) {
throw ngraph_error("Unsupported version");
OPENVINO_THROW("Unsupported version");
}
}
@ -1150,7 +1172,7 @@ pass::StreamSerialize::StreamSerialize(std::ostream& stream,
: StreamSerialize(stream, {}, custom_data_serializer, version) {}
OPENVINO_SUPPRESS_DEPRECATED_END
bool pass::StreamSerialize::run_on_model(const std::shared_ptr<ngraph::Function>& f) {
bool pass::StreamSerialize::run_on_model(const std::shared_ptr<ov::Model>& model) {
RUN_ON_MODEL_SCOPE(StreamSerialize);
/*
Format:
@ -1165,13 +1187,13 @@ bool pass::StreamSerialize::run_on_model(const std::shared_ptr<ngraph::Function>
m_stream.write((const char*)&hdr, sizeof hdr);
};
auto version = static_cast<int64_t>(m_version);
auto& rt_info = f->get_rt_info();
auto& rt_info = model->get_rt_info();
if (rt_info.count("version")) {
version = rt_info.at("version").as<int64_t>();
}
if (version != static_cast<int64_t>(m_version) && m_version != Serialize::Version::UNSPECIFIED)
throw ngraph_error("Cannot serialize model to incompatible IR version");
OPENVINO_THROW("Cannot serialize model to incompatible IR version");
if (version == static_cast<int64_t>(Serialize::Version::UNSPECIFIED)) {
version = static_cast<int64_t>(Serialize::Version::IR_V11);
@ -1194,7 +1216,7 @@ bool pass::StreamSerialize::run_on_model(const std::shared_ptr<ngraph::Function>
pugi::xml_node net_node = xml_doc.append_child(name.c_str());
ConstantWriter constant_write_handler(m_stream);
XmlSerializer visitor(net_node, name, m_custom_opsets, constant_write_handler, version);
std::shared_ptr<ov::Model> fun = f;
std::shared_ptr<ov::Model> fun = model;
visitor.on_attribute(name, fun);
// IR
@ -1213,7 +1235,7 @@ bool pass::StreamSerialize::run_on_model(const std::shared_ptr<ngraph::Function>
m_stream.seekp(file_size);
// Return false because we didn't change nGraph Function
// Return false because we didn't change ov Model
return false;
}
@ -1252,7 +1274,7 @@ public:
};
} // namespace
bool pass::Hash::run_on_model(const std::shared_ptr<ov::Model>& f) {
bool pass::Hash::run_on_model(const std::shared_ptr<ov::Model>& model) {
RUN_ON_MODEL_SCOPE(Hash);
OstreamHashWrapper xmlHash;
OstreamHashWrapper binHash;
@ -1260,14 +1282,14 @@ bool pass::Hash::run_on_model(const std::shared_ptr<ov::Model>& f) {
std::ostream bin(&binHash);
// Determinism is important for hash calculation
serializeFunc(xml, bin, f, Serialize::Version::UNSPECIFIED, {}, true);
serializeFunc(xml, bin, model, Serialize::Version::UNSPECIFIED, {}, true);
uint64_t seed = 0;
seed = hash_combine(seed, xmlHash.getResult());
seed = hash_combine(seed, binHash.getResult());
m_hash = seed;
// Return false because we didn't change nGraph Function
// Return false because we didn't change OpenVINO Model
return false;
}

65
src/core/src/runtime/allocator.cpp
View File

@ -11,19 +11,68 @@
namespace ov {
Allocator::Allocator() : _impl{std::make_shared<BlobAllocator>()} {}
struct DefaultAllocator {
void* allocate(const size_t bytes, const size_t alignment) {
if (alignment == alignof(max_align_t)) {
return ::operator new(bytes);
} else {
OPENVINO_ASSERT(alignment && !static_cast<bool>(alignment & (alignment - static_cast<size_t>(1))),
"Alignment is not power of 2: ",
alignment);
#if defined(_WIN32)
return _aligned_malloc(bytes, alignment);
#else
void* result = nullptr;
if (posix_memalign(&result, std::max(sizeof(void*), alignment), bytes) != 0) {
OPENVINO_THROW("posix_memalign failed");
}
return result;
#endif
}
}
void deallocate(void* handle, const size_t bytes, const size_t alignment) {
if (alignment == alignof(max_align_t)) {
::operator delete(handle);
} else {
#if defined(_WIN32)
return _aligned_free(handle);
#else
return free(handle);
#endif
}
}
bool is_equal(const DefaultAllocator&) const {
return true;
}
};
Allocator::Allocator() : Allocator{DefaultAllocator{}} {}
OPENVINO_SUPPRESS_DEPRECATED_START
struct AllocatorImplWrapper {
AllocatorImplWrapper(const AllocatorImpl::Ptr& impl_) : impl{impl_} {}
void* allocate(const size_t bytes, const size_t alignment) {
return impl->allocate(bytes, alignment);
}
void deallocate(void* handle, const size_t bytes, const size_t alignment) {
impl->deallocate(handle, bytes, alignment);
}
bool is_equal(const AllocatorImplWrapper& other) const {
return impl->is_equal(*other.impl);
}
AllocatorImpl::Ptr impl;
};
Allocator::Allocator(const AllocatorImpl::Ptr& allocator_impl) : Allocator{AllocatorImplWrapper{allocator_impl}} {}
OPENVINO_SUPPRESS_DEPRECATED_END
Allocator::~Allocator() {
_impl = {};
}
Allocator::Allocator(const std::shared_ptr<AllocatorImpl>& impl, const std::shared_ptr<void>& so)
: _impl{impl},
_so{so} {
OPENVINO_ASSERT(_impl != nullptr, "Allocator was not initialized.");
}
Allocator::Allocator(const std::shared_ptr<AllocatorImpl>& impl) : _impl{impl} {
Allocator::Allocator(const Allocator& other, const std::shared_ptr<void>& so) : _impl{other._impl}, _so{so} {
OPENVINO_ASSERT(_impl != nullptr, "Allocator was not initialized.");
}

27
src/core/src/runtime/blob_allocator.hpp
View File

@ -12,7 +12,7 @@
namespace InferenceEngine {
struct BlobAllocator : public IAllocator {
BlobAllocator(const std::shared_ptr<ov::AllocatorImpl>& impl) : _impl{impl} {}
BlobAllocator(const ov::Allocator& impl) : _impl{impl} {}
void* lock(void* handle, LockOp) noexcept override {
return handle;
@ -22,7 +22,7 @@ struct BlobAllocator : public IAllocator {
void* alloc(const size_t size) noexcept override {
try {
return size_map.emplace(_impl->allocate(size), size).first->first;
return size_map.emplace(_impl.allocate(size), size).first->first;
} catch (...) {
return nullptr;
}
@ -32,24 +32,23 @@ struct BlobAllocator : public IAllocator {
try {
auto size = size_map.at(handle);
size_map.erase(handle);
_impl->deallocate(handle, size);
_impl.deallocate(handle, size);
return true;
} catch (...) {
return false;
}
}
std::shared_ptr<ov::AllocatorImpl> _impl;
ov::Allocator _impl;
std::unordered_map<void*, size_t> size_map;
};
} // namespace InferenceEngine
namespace ov {
struct BlobAllocator : public runtime::AllocatorImpl {
BlobAllocator(const std::shared_ptr<ie::IAllocator>& impl = std::make_shared<ie::SystemMemoryAllocator>())
: _impl{impl} {}
struct BlobAllocator {
BlobAllocator() : _impl{std::make_shared<ie::SystemMemoryAllocator>()} {}
void* allocate(const size_t bytes, const size_t alignment) override {
void* allocate(const size_t bytes, const size_t alignment) {
OPENVINO_ASSERT(alignment == alignof(max_align_t),
"Aligned deallocation is not implemented. alignment: ",
alignment);
@ -58,7 +57,7 @@ struct BlobAllocator : public runtime::AllocatorImpl {
return handle;
}
void deallocate(void* handle, const size_t bytes, const size_t alignment) override {
void deallocate(void* handle, const size_t bytes, const size_t alignment) {
OPENVINO_ASSERT(bytes == 0, "Sized deallocation is not implemented. bytes: ", bytes);
OPENVINO_ASSERT(alignment == alignof(max_align_t),
"Aligned deallocation is not implemented. alignment: ",
@ -67,14 +66,10 @@ struct BlobAllocator : public runtime::AllocatorImpl {
OPENVINO_ASSERT(res != false, "Can not deallocate storage");
}
bool is_equal(const AllocatorImpl& other) const override {
auto other_blob_allocator = dynamic_cast<const BlobAllocator*>(&other);
if (other_blob_allocator == nullptr)
return false;
if (other_blob_allocator->_impl == _impl)
bool is_equal(const BlobAllocator& other) const {
if (other._impl == _impl)
return true;
auto other_system_memory_allocator =
dynamic_cast<const ie::SystemMemoryAllocator*>(other_blob_allocator->_impl.get());
auto other_system_memory_allocator = dynamic_cast<const ie::SystemMemoryAllocator*>(other._impl.get());
auto system_allocator = dynamic_cast<const ie::SystemMemoryAllocator*>(_impl.get());
if (system_allocator != nullptr && other_system_memory_allocator != nullptr)
return true;

281
src/core/src/runtime/itensor.cpp Normal file
View File

@ -0,0 +1,281 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include "openvino/runtime/itensor.hpp"
#include "dev/make_tensor.hpp"
#include "openvino/core/except.hpp"
#include "openvino/runtime/allocator.hpp"
#include "openvino/runtime/properties.hpp"
namespace ov {
ITensor::~ITensor() = default;
size_t ITensor::get_size() const {
return shape_size(get_shape());
}
size_t ITensor::get_byte_size() const {
return (get_size() * get_element_type().bitwidth() + 8 - 1) / 8;
}
/**
* @brief View tensor to external memory
* The tensor doesn't own the external memory
*/
class ViewTensor : public ITensor {
public:
ViewTensor(const element::Type element_type, const Shape& shape, void* ptr)
: m_element_type{element_type},
m_shape{shape},
m_capacity{shape},
m_ptr{ptr} {
OPENVINO_ASSERT(m_ptr != nullptr);
OPENVINO_ASSERT(m_element_type != element::undefined && m_element_type != element::dynamic);
update_strides();
}
void* data(const element::Type& element_type) const override {
if (element_type != element::undefined && element_type != element::dynamic) {
OPENVINO_ASSERT(element_type == get_element_type(),
"Tensor data with element type ",
get_element_type(),
", is not representable as pointer to ",
element_type);
}
return m_ptr;
}
const element::Type& get_element_type() const override {
return m_element_type;
}
const Shape& get_shape() const override {
return m_shape;
}
void set_shape(ov::Shape new_shape) override {
OPENVINO_ASSERT(shape_size(new_shape) <= ov::shape_size(m_capacity), "Could set new shape: ", new_shape);
m_shape = std::move(new_shape);
update_strides();
}
const Strides& get_strides() const override {
OPENVINO_ASSERT(m_element_type.bitwidth() >= 8,
"Could not get strides for types with bitwidths less then 8 bit. Tensor type: ",
m_element_type);
return m_strides;
}
protected:
void update_strides() {
if (m_element_type.bitwidth() < 8)
return;
auto& shape = get_shape();
m_strides.clear();
if (!shape.empty()) {
m_strides.resize(shape.size());
m_strides.back() = m_element_type.size();
std::copy(shape.rbegin(), shape.rend() - 1, m_strides.rbegin() + 1);
std::partial_sum(m_strides.rbegin(), m_strides.rend(), m_strides.rbegin(), std::multiplies<size_t>());
}
}
element::Type m_element_type;
Shape m_shape;
Shape m_capacity;
Strides m_strides;
void* m_ptr;
};
/**
* @brief View tensor on external memory with strides
*/
class StridedViewTensor : public ViewTensor {
public:
StridedViewTensor(const element::Type element_type, const Shape& shape, void* ptr, const Strides& strides)
: ViewTensor{element_type, shape, ptr} {
OPENVINO_ASSERT(
get_element_type().bitwidth() >= 8,
"Could not create strided access tensor for types with bitwidths less then 8 bit. Tensor type: ",
get_element_type());
// Save default strides
auto shape_strides = m_strides;
// Change strides
m_strides = strides;
OPENVINO_ASSERT(m_shape.size() == m_strides.size());
for (size_t i = 0; i < m_strides.size(); ++i) {
OPENVINO_ASSERT(shape_strides[i] <= m_strides[i],
"shape stride: ",
shape_strides[i],
", stride: ",
m_strides[i]);
OPENVINO_ASSERT((m_strides[i] % get_element_type().size()) == 0,
"shape stride: ",
shape_strides[i],
", stride: ",
m_strides[i]);
if (i) {
OPENVINO_ASSERT(m_strides[i - 1] >= m_strides[i] * shape[i],
"Strides: ",
m_strides,
" are incompatible with shapes: ",
m_shape);
}
}
}
void set_shape(ov::Shape new_shape) override {
OPENVINO_ASSERT(m_capacity.size() == new_shape.size(),
"Cannot set new shape: ",
new_shape,
" for tensor with strides! Shapes are not compatible.");
for (size_t i = 0; i < new_shape.size(); i++) {
OPENVINO_ASSERT(m_capacity[i] >= new_shape[i],
"Cannot set new shape: ",
new_shape,
" for tensor with strides! Dimension: ",
i,
" is not compatible.");
}
m_shape = std::move(new_shape);
}
};
/**
* @brief Creates view tensor on external memory
*
* @param element_type Tensor element type
* @param shape Tensor shape
* @param ptr pointer to external memoty
* @param byte_strides Tensor strides
*
* @return Shared pointer to tensor interface
*/
std::shared_ptr<ITensor> make_tensor(const element::Type element_type,
const Shape& shape,
void* ptr,
const Strides& byte_strides) {
return byte_strides.empty() ? std::make_shared<ViewTensor>(element_type, shape, ptr)
: std::make_shared<StridedViewTensor>(element_type, shape, ptr, byte_strides);
}
/**
* @brief Tensor with allocated memory
* Tensor owns the memory
*/
class AllocatedTensor : public ViewTensor {
public:
AllocatedTensor(const element::Type element_type, const Shape& shape, const Allocator& allocator)
: ViewTensor{element_type,
shape,
[&] {
OPENVINO_ASSERT(allocator, "Allocator was not initialized");
return const_cast<Allocator&>(allocator).allocate(element_type.size() * shape_size(shape));
}()},
m_allocator{allocator} {}
~AllocatedTensor() {
m_allocator.deallocate(m_ptr, get_byte_size());
}
void set_shape(ov::Shape new_shape) override {
auto old_byte_size = get_byte_size();
m_shape = std::move(new_shape);
if (get_byte_size() > old_byte_size) {
m_allocator.deallocate(m_ptr, old_byte_size);
m_ptr = m_allocator.allocate(get_byte_size());
}
update_strides();
}
private:
Allocator m_allocator;
};
/**
* @brief Creates allocated tensor
*
* @param element_type Tensor element type
* @param shape Tensor shape
* @param allocator Tensor allocator
*
* @return Shared pointer to tensor interface
*/
std::shared_ptr<ITensor> make_tensor(const element::Type element_type, const Shape& shape, const Allocator& allocator) {
return std::make_shared<AllocatedTensor>(element_type, shape, allocator);
}
/**
* @brief ROI tensor on other tensor
* ROI tensor holds the owner
*/
class RoiTensor : public ITensor {
public:
RoiTensor(const std::shared_ptr<ITensor>& owner, const Coordinate& begin, const Coordinate& end)
: m_owner{owner},
m_offsets{begin} {
OPENVINO_ASSERT(owner->get_element_type().bitwidth() >= 8,
"ROI Tensor for types with bitwidths less then 8 bit is not implemented. Tensor type: ",
owner->get_element_type());
auto owner_shape = owner->get_shape();
OPENVINO_ASSERT(owner_shape.size() == begin.size());
OPENVINO_ASSERT(begin.size() == end.size());
m_shape.resize(begin.size());
for (size_t i = 0; i < begin.size(); ++i) {
OPENVINO_ASSERT(begin[i] <= owner_shape[i]);
OPENVINO_ASSERT(end[i] <= owner_shape[i]);
m_shape[i] = end[i] - begin[i];
OPENVINO_ASSERT(m_shape[i] <= owner_shape[i]);
}
}
const element::Type& get_element_type() const override {
return m_owner->get_element_type();
}
const Strides& get_strides() const override {
return m_owner->get_strides();
}
const Shape& get_shape() const override {
return m_shape;
}
void set_shape(ov::Shape new_shape) override {
OPENVINO_THROW("Shapes cannot be changed for ROI Tensor");
}
void* data(const element::Type& element_type) const override {
auto owner_data = m_owner->data(element_type);
auto& strides = get_strides();
size_t byte_offset =
std::inner_product(m_offsets.begin(), m_offsets.end(), strides.begin(), static_cast<size_t>(0));
return static_cast<uint8_t*>(owner_data) + byte_offset;
}
private:
std::shared_ptr<ITensor> m_owner;
Coordinate m_offsets;
Shape m_shape;
};
/**
* @brief Creates ROI tensor
*
* @param other Tensor what owns the memory
* @param begin Begin coordinates
* @param end End coordinates
*
* @return Shared pointer to tensor interface
*/
std::shared_ptr<ITensor> make_tensor(const std::shared_ptr<ITensor>& other,
const Coordinate& begin,
const Coordinate& end) {
return std::make_shared<RoiTensor>(other, begin, end);
}
} // namespace ov

134
src/core/src/runtime/ov_tensor.cpp
View File

@ -4,14 +4,14 @@
#include <numeric>
#include "blob_factory.hpp" // IE private header
#include "ie_ngraph_utils.hpp" // IE private header
#include "dev/make_tensor.hpp"
#include "openvino/core/except.hpp"
#include "openvino/core/node_output.hpp"
#include "openvino/core/shape.hpp"
#include "openvino/core/strides.hpp"
#include "openvino/runtime/itensor.hpp"
#include "openvino/runtime/remote_tensor.hpp"
#include "openvino/runtime/tensor.hpp"
#include "runtime/blob_allocator.hpp"
#include "shape_util.hpp"
namespace ov {
@ -32,70 +32,21 @@ Tensor::~Tensor() {
_impl = {};
}
Tensor::Tensor(const std::shared_ptr<ie::Blob>& impl, const std::vector<std::shared_ptr<void>>& so)
Tensor::Tensor(const std::shared_ptr<ITensor>& impl, const std::vector<std::shared_ptr<void>>& so)
: _impl{impl},
_so{so} {
OPENVINO_ASSERT(_impl != nullptr, "Tensor was not initialized.");
}
Tensor::Tensor(const element::Type element_type, const Shape& shape, const Allocator& allocator) {
OPENVINO_ASSERT(allocator, "Allocator was not initialized");
auto allocator_impl = dynamic_cast<const BlobAllocator*>(allocator._impl.get());
auto blob_allocator =
(allocator_impl != nullptr) ? allocator_impl->_impl : std::make_shared<ie::BlobAllocator>(allocator._impl);
_impl = make_blob_with_precision(
{ie::details::convertPrecision(element_type), shape, ie::TensorDesc::getLayoutByDims(shape)},
blob_allocator);
_impl->allocate();
}
Tensor::Tensor(const element::Type& element_type, const Shape& shape, const Allocator& allocator)
: _impl{make_tensor(element_type, shape, allocator)} {}
Tensor::Tensor(const element::Type element_type, const Shape& shape, void* host_ptr, const Strides& byte_strides) {
ie::SizeVector blk_order(shape.size());
std::iota(blk_order.begin(), blk_order.end(), 0);
ie::SizeVector dim_offset(shape.size(), 0);
ie::SizeVector blk_strides;
if (byte_strides.empty()) {
blk_strides = ov::row_major_strides(shape);
} else {
blk_strides.resize(byte_strides.size());
std::transform(byte_strides.begin(),
byte_strides.end(),
blk_strides.begin(),
[&element_type](size_t byte_stride) {
OPENVINO_ASSERT(byte_stride % element_type.size() == 0,
"Limitation: Stride in bytes ",
byte_stride,
" should be divisible by size of element ",
element_type.size());
return byte_stride / element_type.size();
});
}
Tensor::Tensor(const element::Type& element_type, const Shape& shape, void* host_ptr, const Strides& byte_strides)
: _impl{make_tensor(element_type, shape, host_ptr, byte_strides)} {}
try {
_impl = make_blob_with_precision(ie::details::convertPrecision(element_type),
ie::TensorDesc{ie::details::convertPrecision(element_type),
shape,
ie::BlockingDesc{shape, blk_order, 0, dim_offset, blk_strides}},
host_ptr);
} catch (const std::exception& ex) {
OPENVINO_THROW(ex.what());
} catch (...) {
OPENVINO_ASSERT(false, "Unexpected exception");
}
}
Tensor::Tensor(const Tensor& owner, const Coordinate& begin, const Coordinate& end) : _so{owner._so} {
OPENVINO_ASSERT(owner.get_element_type().bitwidth() >= 8,
"ROI Tensor for types with bitwidths less then 8 bit is not implemented. Tensor type: ",
owner.get_element_type());
try {
_impl = owner._impl->createROI(begin, end);
} catch (const std::exception& ex) {
OPENVINO_THROW(ex.what());
} catch (...) {
OPENVINO_ASSERT(false, "Unexpected exception");
}
}
Tensor::Tensor(const Tensor& owner, const Coordinate& begin, const Coordinate& end)
: _impl{make_tensor(owner._impl, begin, end)},
_so{owner._so} {}
Tensor::Tensor(const ov::Output<const ov::Node>& port, const Allocator& allocator)
: Tensor(port.get_element_type(),
@ -108,23 +59,16 @@ Tensor::Tensor(const ov::Output<const ov::Node>& port, void* host_ptr, const Str
host_ptr,
byte_strides) {}
element::Type Tensor::get_element_type() const {
OV_TENSOR_STATEMENT(return ie::details::convertPrecision(_impl->getTensorDesc().getPrecision()));
const element::Type& Tensor::get_element_type() const {
OV_TENSOR_STATEMENT(return _impl->get_element_type());
}
void Tensor::set_shape(const ov::Shape& shape) {
// WA for tensor conversion from host tensor with dynamic shape.
if (util::is_dynamic_shape(get_shape())) {
_impl = make_blob_with_precision(
{_impl->getTensorDesc().getPrecision(), shape, ie::TensorDesc::getLayoutByRank(shape.size())});
_impl->allocate();
} else {
OV_TENSOR_STATEMENT(_impl->setShape({shape.begin(), shape.end()}));
}
OV_TENSOR_STATEMENT(_impl->set_shape(shape));
}
Shape Tensor::get_shape() const {
OV_TENSOR_STATEMENT({ return _impl->getTensorDesc().getBlockingDesc().getBlockDims(); });
const Shape& Tensor::get_shape() const {
OV_TENSOR_STATEMENT(return _impl->get_shape());
}
void Tensor::copy_to(ov::Tensor& dst) const {
@ -258,55 +202,19 @@ void Tensor::copy_to(ov::Tensor& dst) const {
}
Strides Tensor::get_strides() const {
OPENVINO_ASSERT(get_element_type().bitwidth() >= 8,
"Could not get strides for types with bitwidths less then 8 bit. Tensor type: ",
get_element_type());
OV_TENSOR_STATEMENT({
const auto& element_strides = _impl->getTensorDesc().getBlockingDesc().getStrides();
const size_t elem_size = get_element_type().size();
Strides byte_strides;
byte_strides.resize(element_strides.size());
std::transform(element_strides.begin(),
element_strides.end(),
byte_strides.begin(),
[&elem_size](size_t stride) {
return stride * elem_size;
});
return byte_strides;
});
OV_TENSOR_STATEMENT(return _impl->get_strides(););
}
size_t Tensor::get_size() const {
OV_TENSOR_STATEMENT(return _impl->size());
OV_TENSOR_STATEMENT(return _impl->get_size());
}
size_t Tensor::get_byte_size() const {
OV_TENSOR_STATEMENT(return _impl->byteSize(););
OV_TENSOR_STATEMENT(return _impl->get_byte_size(););
}
void* Tensor::data(const element::Type element_type) const {
OPENVINO_ASSERT(_impl != nullptr, "Tensor was not initialized.");
#define TYPE_CHECK(TYPE) (dynamic_cast<const ie::TBlob<TYPE>*>(_impl.get()) != nullptr)
auto host_accesable_implementation = TYPE_CHECK(bool) || TYPE_CHECK(int8_t) || TYPE_CHECK(uint8_t) ||
TYPE_CHECK(int16_t) || TYPE_CHECK(uint16_t) || TYPE_CHECK(int32_t) ||
TYPE_CHECK(uint32_t) || TYPE_CHECK(int64_t) || TYPE_CHECK(uint64_t) ||
TYPE_CHECK(float) || TYPE_CHECK(double);
#undef TYPE_CHECK
OPENVINO_ASSERT(host_accesable_implementation, "Tensor implementation type dose not contains host accessable data");
if (element_type != element::undefined) {
OPENVINO_ASSERT(element_type == get_element_type(),
"Tensor data with element type ",
get_element_type(),
", is not representable as pointer to ",
element_type);
}
// since we don't use byte offsets, we need to explicitly multiply by element_size
auto byte_offset = _impl->getTensorDesc().getBlockingDesc().getOffsetPadding() * get_element_type().size();
OPENVINO_ASSERT((get_element_type().bitwidth() >= 8) || (byte_offset == 0),
"ROI access for types with bitwidths less then 8 bit is not implemented. Tensor type: ",
get_element_type());
OV_TENSOR_STATEMENT(
{ return byte_offset + InferenceEngine::as<InferenceEngine::MemoryBlob>(_impl)->rmap().as<uint8_t*>(); });
void* Tensor::data(const element::Type& element_type) const {
OV_TENSOR_STATEMENT(return _impl->data(element_type));
}
bool Tensor::operator!() const noexcept {

12
src/core/tests/ov_default_allocator_test.cpp
View File

@ -25,18 +25,18 @@ TEST_F(OVDefaultAllocatorTest, canAllocateAndDeallocate) {
ASSERT_NO_THROW(allocator.deallocate(ptr));
}
TEST_F(OVDefaultAllocatorTest, alignedAllocationIsNotImplemented) {
TEST_F(OVDefaultAllocatorTest, alignedAllocationNotThrow) {
ov::Allocator allocator;
ASSERT_THROW(allocator.allocate(64, 64), ov::Exception);
ASSERT_NO_THROW(allocator.allocate(64, 64));
}
TEST_F(OVDefaultAllocatorTest, sizedAndAlignedDeallocationAreNotImplemented) {
TEST_F(OVDefaultAllocatorTest, sizedAndAlignedDeallocationNotThrow) {
ov::Allocator allocator;
void* ptr = nullptr;
ASSERT_NO_THROW(ptr = allocator.allocate(64));
ASSERT_THROW(allocator.deallocate(ptr, 64), ov::Exception);
ASSERT_THROW(allocator.deallocate(ptr, 0, 64), ov::Exception);
ASSERT_NO_THROW(allocator.deallocate(ptr));
ASSERT_NO_THROW(allocator.deallocate(ptr, 64));
ASSERT_NO_THROW(ptr = allocator.allocate(64, 64));
ASSERT_NO_THROW(allocator.deallocate(ptr, 64, 64));
}
TEST_F(OVDefaultAllocatorTest, defaultAllocatorsAreEqual) {

54
src/core/tests/ov_tensor_test.cpp
View File

@ -87,16 +87,18 @@ TEST_F(OVTensorTest, operators) {
ASSERT_TRUE(!t);
}
class OVMockAllocator : public ov::AllocatorImpl {
OPENVINO_SUPPRESS_DEPRECATED_START
class OVMockAllocatorImpl : public ov::AllocatorImpl {
public:
MOCK_METHOD(void*, allocate, (size_t, size_t), ());
MOCK_METHOD(void, deallocate, (void*, size_t, size_t), ()); // NOLINT(readability/casting)
MOCK_METHOD(bool, is_equal, (const ov::AllocatorImpl&), (const, noexcept)); // NOLINT(readability/casting)
};
TEST_F(OVTensorTest, canCreateTensorUsingMockAllocator) {
OPENVINO_SUPPRESS_DEPRECATED_START
TEST_F(OVTensorTest, canCreateTensorUsingMockAllocatorImpl) {
ov::Shape shape = {1, 2, 3};
auto allocator = std::make_shared<OVMockAllocator>();
auto allocator = std::make_shared<OVMockAllocatorImpl>();
EXPECT_CALL(*allocator, allocate(::testing::_, ::testing::_))
.WillRepeatedly(testing::Return(reinterpret_cast<void*>(1)));
@ -104,6 +106,40 @@ TEST_F(OVTensorTest, canCreateTensorUsingMockAllocator) {
{ ov::Tensor t{ov::element::f32, shape, ov::Allocator{allocator}}; }
}
OPENVINO_SUPPRESS_DEPRECATED_END
struct OVMockAllocator {
struct Impl {
MOCK_METHOD(void*, allocate, (size_t, size_t), ());
MOCK_METHOD(void, deallocate, (void*, size_t, size_t), ());
MOCK_METHOD(bool, is_equal, (const Impl&), (const, noexcept));
};
OVMockAllocator() : impl{std::make_shared<Impl>()} {}
void* allocate(size_t b, size_t a) {
return impl->allocate(b, a);
}
void deallocate(void* ptr, size_t b, size_t a) {
impl->deallocate(ptr, b, a);
}
bool is_equal(const OVMockAllocator& other) const {
return impl->is_equal(*other.impl);
}
std::shared_ptr<Impl> impl;
};
TEST_F(OVTensorTest, canCreateTensorUsingMockAllocator) {
ov::Shape shape = {1, 2, 3};
OVMockAllocator allocator;
EXPECT_CALL(*allocator.impl, allocate(::testing::_, ::testing::_))
.WillRepeatedly(testing::Return(reinterpret_cast<void*>(1)));
EXPECT_CALL(*allocator.impl, deallocate(::testing::_, ::testing::_, ::testing::_)).Times(1);
{ ov::Tensor t{ov::element::f32, shape, allocator}; }
}
TEST_F(OVTensorTest, canAccessExternalData) {
ov::Shape shape = {1, 1, 3};
@ -235,7 +271,7 @@ TEST_F(OVTensorTest, canSetShapeOfSameSizeOnPreallocatedMemory) {
ASSERT_NO_THROW(t.set_shape(newShape));
}
TEST_F(OVTensorTest, DISABLED_canSetShapeOfOriginalSizeAfterDecreasingOnPreallocatedMemory) {
TEST_F(OVTensorTest, canSetShapeOfOriginalSizeAfterDecreasingOnPreallocatedMemory) {
float data[4 * 5 * 6 * 2];
ov::Tensor t{ov::element::f32, {4, 5, 6}, data};
const ov::Shape smallerShape({1, 2, 3});
@ -245,6 +281,16 @@ TEST_F(OVTensorTest, DISABLED_canSetShapeOfOriginalSizeAfterDecreasingOnPrealloc
ASSERT_NO_THROW(t.set_shape(originalShape));
}
TEST_F(OVTensorTest, canChangeShapeOnStridedTensor) {
float data[64 * 4];
ov::Tensor t{ov::element::f32, {4, 2, 2}, data, {64, 16, 4}};
const ov::Shape incorrect_shape({2, 4, 2});
const ov::Shape correct_shape({1, 1, 2});
ASSERT_THROW(t.set_shape(incorrect_shape), ov::Exception);
ASSERT_NO_THROW(t.set_shape(correct_shape));
}
TEST_F(OVTensorTest, makeRangeRoiTensor) {
ov::Tensor t{ov::element::i32, {1, 3, 6, 5}}; // RGBp picture of size (WxH) = 5x6
ov::Tensor roi_tensor{t, {0, 0, 1, 2}, {1, 3, 5, 4}};

6
src/frontends/tensorflow_common/src/op/ctc_loss.cpp
View File

@ -47,9 +47,9 @@ OutputVector translate_ctc_loss_op(const NodeContext& node) {
auto logits_shape = make_shared<ShapeOf>(logits, ov::element::i64);
auto dense_shape = make_shared<Slice>(logits_shape,
make_shared<Constant>(ov::element::i64, ov::Shape{}, 0),
make_shared<Constant>(ov::element::i64, ov::Shape{}, 2),
make_shared<Constant>(ov::element::i64, ov::Shape{}, 1));
make_shared<Constant>(ov::element::i64, ov::Shape{1}, 0),
make_shared<Constant>(ov::element::i64, ov::Shape{1}, 2),
make_shared<Constant>(ov::element::i64, ov::Shape{1}, 1));
auto minus_one_value = make_shared<Constant>(decoded_values.get_element_type(), ov::Shape{}, -1);
auto init_decoded_values = make_shared<Broadcast>(minus_one_value, dense_shape);
auto decoded_values_dense = make_shared<ScatterNDUpdate>(init_decoded_values, decoded_indices, decoded_values);

37
src/inference/dev_api/openvino/runtime/iremote_tensor.hpp Normal file
View File

@ -0,0 +1,37 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
/**
* @brief OpenVINO Runtime IRemoteTensor interface
* @file openvino/runtime/iremote_tensor.hpp
*/
#pragma once
#include "openvino/core/except.hpp"
#include "openvino/runtime/common.hpp"
#include "openvino/runtime/itensor.hpp"
namespace ov {
class OPENVINO_RUNTIME_API IRemoteTensor : public ITensor {
public:
void* data(const element::Type& type = {}) const final {
OPENVINO_NOT_IMPLEMENTED;
}
~IRemoteTensor() override;
/**
* @brief Returns additional information associated with tensor
* @return Map of property names to properties
*/
virtual const AnyMap& get_properties() const = 0;
/**
* @brief Returns device name
* @return Device name
*/
virtual const std::string& get_device_name() const = 0;
};
} // namespace ov

2
src/inference/include/ie/ie_blob.h
View File

@ -192,7 +192,7 @@ public:
*
* @param dims new shape
*/
void setShape(const SizeVector& dims);
virtual void setShape(const SizeVector& dims);
/**
* @deprecated Cast to MemoryBlob and use new wlock/rwlock API instead.

6
src/inference/src/cpp/ie_remote_context.cpp
View File

@ -7,9 +7,11 @@
#include <exception>
#include "any_copy.hpp"
#include "dev/make_tensor.hpp"
#include "ie_ngraph_utils.hpp"
#include "ie_remote_blob.hpp"
#include "openvino/core/except.hpp"
#include "openvino/runtime/itensor.hpp"
#include "openvino/runtime/remote_context.hpp"
#define OV_REMOTE_CONTEXT_STATEMENT(...) \
@ -67,7 +69,7 @@ RemoteTensor RemoteContext::create_tensor(const element::Type& type, const Shape
{ie::details::convertPrecision(type), shape, ie::TensorDesc::getLayoutByRank(shape.size())},
params);
blob->allocate();
return {blob, {_so}};
return {ov::make_tensor(blob), {_so}};
});
}
@ -76,7 +78,7 @@ Tensor RemoteContext::create_host_tensor(const element::Type element_type, const
auto blob = _impl->CreateHostBlob(
{ie::details::convertPrecision(element_type), shape, ie::TensorDesc::getLayoutByRank(shape.size())});
blob->allocate();
return {blob, {_so}};
return {ov::make_tensor(blob), {_so}};
});
}

27
src/inference/src/dev/converter_utils.cpp
View File

@ -13,6 +13,7 @@
#include "cpp_interfaces/interface/ie_iexecutable_network_internal.hpp"
#include "cpp_interfaces/interface/ie_iplugin_internal.hpp"
#include "cpp_interfaces/interface/ie_ivariable_state_internal.hpp"
#include "dev/make_tensor.hpp"
#include "icompiled_model_wrapper.hpp"
#include "ie_blob.h"
#include "ie_common.h"
@ -30,6 +31,7 @@
#include "openvino/runtime/icompiled_model.hpp"
#include "openvino/runtime/iinfer_request.hpp"
#include "openvino/runtime/iplugin.hpp"
#include "openvino/runtime/itensor.hpp"
#include "openvino/runtime/ivariable_state.hpp"
#include "openvino/runtime/profiling_info.hpp"
#include "openvino/runtime/remote_context.hpp"
@ -206,11 +208,11 @@ public:
}
void SetState(const InferenceEngine::Blob::Ptr& newState) override {
m_state->set_state(ov::Tensor(newState, {}));
m_state->set_state(ov::Tensor(ov::make_tensor(newState), {}));
}
InferenceEngine::Blob::CPtr GetState() const override {
return m_state->get_state()._impl;
return tensor_to_blob(m_state->get_state()._impl);
}
};
@ -499,7 +501,7 @@ public:
void SetBlob(const std::string& name, const InferenceEngine::Blob::Ptr& data) override {
try {
m_request->set_tensor(find_port(name), ov::Tensor{data, {}});
m_request->set_tensor(find_port(name), ov::Tensor{ov::make_tensor(data), {}});
} catch (const ov::Exception& ex) {
const std::string what = ex.what();
if (what.find("Failed to set tensor") != std::string::npos) {
@ -513,7 +515,7 @@ public:
try {
std::vector<ov::Tensor> tensors;
for (const auto& blob : blobs) {
tensors.emplace_back(ov::Tensor{blob, {}});
tensors.emplace_back(ov::Tensor{ov::make_tensor(blob), {}});
}
m_request->set_tensors(find_port(name), tensors);
} catch (const ov::Exception& ex) {
@ -522,14 +524,14 @@ public:
}
InferenceEngine::Blob::Ptr GetBlob(const std::string& name) override {
return m_request->get_tensor(find_port(name))._impl;
return tensor_to_blob(m_request->get_tensor(find_port(name))._impl);
}
InferenceEngine::BatchedBlob::Ptr GetBlobs(const std::string& name) override {
auto tensors = m_request->get_tensors(find_port(name));
std::vector<InferenceEngine::Blob::Ptr> blobs;
for (const auto& tensor : tensors) {
blobs.emplace_back(tensor._impl);
blobs.emplace_back(tensor_to_blob(tensor._impl));
}
return std::make_shared<InferenceEngine::BatchedBlob>(blobs);
}
@ -604,11 +606,12 @@ public:
}
void set_state(const ov::Tensor& state) override {
m_state->SetState(state._impl);
m_state->SetState(ov::tensor_to_blob(state._impl));
}
const ov::Tensor& get_state() const override {
m_converted_state = ov::Tensor(std::const_pointer_cast<InferenceEngine::Blob>(m_state->GetState()), {});
m_converted_state =
ov::Tensor(ov::make_tensor(std::const_pointer_cast<InferenceEngine::Blob>(m_state->GetState())), {});
return m_converted_state;
}
};
@ -706,11 +709,11 @@ public:
name,
"'");
auto blob = m_request->GetBlob(name);
ov::Tensor tensor = {blob, {m_request->getPointerToSo()}};
ov::Tensor tensor = {ov::make_tensor(blob), {m_request->getPointerToSo()}};
return tensor;
}
void set_tensor(const ov::Output<const ov::Node>& port, const ov::Tensor& tensor) override {
m_request->SetBlob(get_legacy_name_from_port(port), tensor._impl);
m_request->SetBlob(get_legacy_name_from_port(port), ov::tensor_to_blob(tensor._impl));
}
std::vector<ov::Tensor> get_tensors(const ov::Output<const ov::Node>& port) const override {
@ -719,14 +722,14 @@ public:
if (!blobs)
return ret;
for (size_t i = 0; i < blobs->size(); i++) {
ret.emplace_back(ov::Tensor{blobs->getBlob(i), {m_request->getPointerToSo()}});
ret.emplace_back(ov::Tensor{ov::make_tensor(blobs->getBlob(i)), {m_request->getPointerToSo()}});
}
return ret;
}
void set_tensors(const ov::Output<const ov::Node>& port, const std::vector<ov::Tensor>& tensors) override {
std::vector<InferenceEngine::Blob::Ptr> blobs;
for (const auto& tensor : tensors) {
blobs.emplace_back(tensor._impl);
blobs.emplace_back(ov::tensor_to_blob(tensor._impl));
}
m_request->SetBlobs(get_legacy_name_from_port(port), blobs);
}

4
src/inference/src/dev/core_impl.cpp
View File

@ -13,6 +13,7 @@
#include "cpp_interfaces/interface/ie_internal_plugin_config.hpp"
#include "cpp_interfaces/interface/ie_iplugin_internal.hpp"
#include "dev/converter_utils.hpp"
#include "dev/make_tensor.hpp"
#include "file_utils.h"
#include "ie_itt.hpp"
#include "ie_network_reader.hpp"
@ -27,6 +28,7 @@
#include "openvino/core/version.hpp"
#include "openvino/pass/manager.hpp"
#include "openvino/runtime/icompiled_model.hpp"
#include "openvino/runtime/itensor.hpp"
#include "openvino/runtime/remote_context.hpp"
#include "openvino/runtime/threading/executor_manager.hpp"
#include "openvino/util/common_util.hpp"
@ -1097,7 +1099,7 @@ std::shared_ptr<ov::Model> ov::CoreImpl::read_model(const std::string& model,
bool frontendMode) const {
InferenceEngine::Blob::Ptr blob;
if (weights) {
blob = weights._impl;
blob = tensor_to_blob(weights._impl);
}
OV_ITT_SCOPE(FIRST_INFERENCE, ov::itt::domains::IE_RT, "CoreImpl::read_model from memory");
return ReadNetwork(model, blob, frontendMode).getFunction();

11
src/inference/src/dev/core_impl_ie.cpp
View File

@ -10,6 +10,7 @@
#include "cpp_interfaces/interface/ie_internal_plugin_config.hpp"
#include "cpp_interfaces/interface/ie_iplugin_internal.hpp"
#include "dev/converter_utils.hpp"
#include "dev/make_tensor.hpp"
#include "ie_itt.hpp"
#include "ie_network_reader.hpp"
#include "iplugin_wrapper.hpp"
@ -17,6 +18,7 @@
#include "ngraph/pass/constant_folding.hpp"
#include "openvino/itt.hpp"
#include "openvino/runtime/icompiled_model.hpp"
#include "openvino/runtime/itensor.hpp"
#include "openvino/util/common_util.hpp"
bool ov::CoreImpl::isNewAPI() const {
@ -113,10 +115,11 @@ InferenceEngine::SoExecutableNetworkInternal ov::CoreImpl::LoadNetwork(
const std::function<void(const InferenceEngine::CNNNetwork&)>& val) {
OV_ITT_SCOPE(FIRST_INFERENCE, InferenceEngine::itt::domains::IE_LT, "Core::LoadNetwork::Memory");
auto compiled_model = compile_model(modelStr,
ov::Tensor{std::const_pointer_cast<InferenceEngine::Blob>(weights), {}},
deviceName,
ov::any_copy(config));
auto compiled_model =
compile_model(modelStr,
ov::Tensor{ov::make_tensor(std::const_pointer_cast<InferenceEngine::Blob>(weights)), {}},
deviceName,
ov::any_copy(config));
return {ov::legacy_convert::convert_compiled_model(compiled_model._ptr), compiled_model._so};
}

351
src/inference/src/dev/iremote_tensor.cpp Normal file
View File

@ -0,0 +1,351 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include "openvino/runtime/iremote_tensor.hpp"
#include <memory>
#include "dev/make_tensor.hpp"
#include "ie_blob.h"
#include "ie_ngraph_utils.hpp"
#include "ie_remote_blob.hpp"
#include "openvino/runtime/properties.hpp"
namespace ov {
IRemoteTensor::~IRemoteTensor() = default;
/**
* @brief Tensor what contains InferenceEngine::Blob inside
* Blob owns the memory
*/
class BlobTensor : public ITensor {
mutable element::Type m_type;
mutable Shape m_shape;
mutable Strides m_strides;
public:
std::shared_ptr<ie::Blob> blob;
BlobTensor(const InferenceEngine::Blob::Ptr& blob) : blob{blob} {
auto remote_impl = dynamic_cast<InferenceEngine::RemoteBlob*>(blob.get());
OPENVINO_ASSERT(!remote_impl);
OPENVINO_ASSERT(blob);
m_shape = blob->getTensorDesc().getBlockingDesc().getBlockDims();
}
const element::Type& get_element_type() const override {
m_type = InferenceEngine::details::convertPrecision(blob->getTensorDesc().getPrecision());
return m_type;
}
void set_shape(ov::Shape shape) override {
blob->setShape({shape.begin(), shape.end()});
}
const Shape& get_shape() const override {
m_shape = blob->getTensorDesc().getBlockingDesc().getBlockDims();
return m_shape;
}
const Strides& get_strides() const override {
OPENVINO_ASSERT(get_element_type().bitwidth() >= 8,
"Could not get strides for types with bitwidths less then 8 bit. Tensor type: ",
get_element_type());
const auto& element_strides = blob->getTensorDesc().getBlockingDesc().getStrides();
const size_t elem_size = get_element_type().size();
m_strides.clear();
m_strides.resize(element_strides.size());
std::transform(element_strides.begin(), element_strides.end(), m_strides.begin(), [&elem_size](size_t stride) {
return stride * elem_size;
});
return m_strides;
}
size_t get_size() const override {
return blob->size();
}
size_t get_byte_size() const override {
return blob->byteSize();
}
void* data(const element::Type& element_type) const override {
OPENVINO_ASSERT(blob != nullptr, "Tensor was not initialized.");
#define TYPE_CHECK(TYPE) (dynamic_cast<const ie::TBlob<TYPE>*>(blob.get()) != nullptr)
auto host_accesable_implementation = TYPE_CHECK(bool) || TYPE_CHECK(int8_t) || TYPE_CHECK(uint8_t) ||
TYPE_CHECK(int16_t) || TYPE_CHECK(uint16_t) || TYPE_CHECK(int32_t) ||
TYPE_CHECK(uint32_t) || TYPE_CHECK(int64_t) || TYPE_CHECK(uint64_t) ||
TYPE_CHECK(float) || TYPE_CHECK(double);
#undef TYPE_CHECK
OPENVINO_ASSERT(host_accesable_implementation,
"Tensor implementation type dose not contains host accessable data");
if (element_type != element::undefined) {
OPENVINO_ASSERT(element_type == get_element_type(),
"Tensor data with element type ",
get_element_type(),
", is not representable as pointer to ",
element_type);
}
// since we don't use byte offsets, we need to explicitly multiply by element_size
auto byte_offset = blob->getTensorDesc().getBlockingDesc().getOffsetPadding() * get_element_type().size();
OPENVINO_ASSERT((get_element_type().bitwidth() >= 8) || (byte_offset == 0),
"ROI access for types with bitwidths less then 8 bit is not implemented. Tensor type: ",
get_element_type());
return byte_offset + InferenceEngine::as<InferenceEngine::MemoryBlob>(blob)->rmap().as<uint8_t*>();
}
};
/**
* @brief Tensor what contains InferenceEngine::RemoteBlob inside
* Blob owns the memory
*/
class RemoteBlobTensor : public IRemoteTensor {
mutable element::Type m_type;
mutable Shape m_shape;
mutable Strides m_strides;
mutable ov::AnyMap m_properties;
mutable std::string m_dev_name;
public:
std::shared_ptr<ie::RemoteBlob> blob;
RemoteBlobTensor(const InferenceEngine::RemoteBlob::Ptr& blob) : blob{blob} {
OPENVINO_ASSERT(blob);
m_shape = blob->getTensorDesc().getBlockingDesc().getBlockDims();
}
const element::Type& get_element_type() const override {
m_type = InferenceEngine::details::convertPrecision(blob->getTensorDesc().getPrecision());
return m_type;
}
void set_shape(ov::Shape shape) override {
blob->setShape({shape.begin(), shape.end()});
}
const Shape& get_shape() const override {
m_shape = blob->getTensorDesc().getBlockingDesc().getBlockDims();
return m_shape;
}
const Strides& get_strides() const override {
OPENVINO_ASSERT(get_element_type().bitwidth() >= 8,
"Could not get strides for types with bitwidths less then 8 bit. Tensor type: ",
get_element_type());
const auto& element_strides = blob->getTensorDesc().getBlockingDesc().getStrides();
const size_t elem_size = get_element_type().size();
m_strides.clear();
m_strides.resize(element_strides.size());
std::transform(element_strides.begin(), element_strides.end(), m_strides.begin(), [&elem_size](size_t stride) {
return stride * elem_size;
});
return m_strides;
}
size_t get_size() const override {
return blob->size();
}
size_t get_byte_size() const override {
return blob->byteSize();
}
const AnyMap& get_properties() const override {
m_properties = blob->getParams();
return m_properties;
}
const std::string& get_device_name() const override {
m_dev_name = blob->getDeviceName();
return m_dev_name;
}
};
/**
* @brief Create InferenceEngine::RemoteBlob from the Tensor
*/
class TensorRemoteBlob : public ie::RemoteBlob {
public:
TensorRemoteBlob(const std::shared_ptr<ITensor>& tensor)
: ie::RemoteBlob{ie::TensorDesc{ie::details::convertPrecision(tensor->get_element_type()),
tensor->get_shape(),
ie::TensorDesc::getLayoutByRank(tensor->get_shape().size())}},
tensor{std::dynamic_pointer_cast<ov::IRemoteTensor>(tensor)} {
OPENVINO_ASSERT(this->tensor);
}
AnyMap getParams() const override {
return tensor->get_properties();
}
std::string getDeviceName() const noexcept override {
try {
return tensor->get_device_name();
} catch (...) {
return {};
}
}
std::shared_ptr<ie::RemoteContext> getContext() const noexcept override {
return {};
}
void allocate() noexcept override {}
bool deallocate() noexcept override {
return true;
}
ie::LockedMemory<void> buffer() noexcept override {
return {nullptr, nullptr, 0};
}
ie::LockedMemory<const void> cbuffer() const noexcept override {
return {nullptr, nullptr, 0};
}
ie::LockedMemory<void> rwmap() noexcept override {
return {nullptr, nullptr, 0};
}
ie::LockedMemory<const void> rmap() const noexcept override {
return {nullptr, nullptr, 0};
}
ie::LockedMemory<void> wmap() noexcept override {
return {nullptr, nullptr, 0};
}
const std::shared_ptr<ie::IAllocator>& getAllocator() const noexcept override {
return m_allocator;
}
void* getHandle() const noexcept override {
return nullptr;
}
std::shared_ptr<IRemoteTensor> tensor;
private:
std::shared_ptr<ie::IAllocator> m_allocator;
};
/**
* @brief Create InferenceEngine::TBlob<T> from the tensor
*
* @tparam T Blob data type
*/
template <typename T>
class TensorMemoryBlob : public ie::TBlob<T> {
public:
~TensorMemoryBlob() override = default;
explicit TensorMemoryBlob(const std::shared_ptr<ITensor>& tensor_) try : ie
::TBlob<T>{[&] {
auto element_type = tensor_->get_element_type();
auto shape = tensor_->get_shape();
ie::SizeVector blk_order(shape.size());
std::iota(blk_order.begin(), blk_order.end(), 0);
ie::SizeVector dim_offset(shape.size(), 0);
ie::SizeVector blk_strides;
auto byte_strides = element_type.bitwidth() >= 8 ? tensor_->get_strides() : Strides{};
if (byte_strides.empty()) {
blk_strides = ov::row_major_strides(shape);
} else {
blk_strides.resize(byte_strides.size());
std::transform(byte_strides.begin(),
byte_strides.end(),
blk_strides.begin(),
[&element_type](size_t byte_stride) {
OPENVINO_ASSERT(byte_stride % element_type.size() == 0,
"Limitation: Stride in bytes ",
byte_stride,
" should be divisible by size of element ",
element_type.size());
return byte_stride / element_type.size();
});
}
return ie::TensorDesc{ie::details::convertPrecision(element_type),
shape,
ie::BlockingDesc{shape, blk_order, 0, dim_offset, blk_strides}};
}(),
static_cast<T*>(tensor_->data()),
tensor_->get_byte_size()},
tensor{tensor_} {
OPENVINO_ASSERT(!std::dynamic_pointer_cast<ov::IRemoteTensor>(tensor));
}
catch (const std::exception& ex) {
throw ov::Exception(ex.what());
}
void setShape(const ie::SizeVector& dims) override {
tensor->set_shape(dims);
ie::TBlob<T>::setShape(dims);
}
std::shared_ptr<ITensor> tensor;
};
std::shared_ptr<ITensor> make_tensor(const std::shared_ptr<ie::Blob>& blob) {
#define ELSE_IF(type) \
else if (auto tblob = dynamic_cast<const TensorMemoryBlob<type>*>(blob.get())) { \
return tblob->tensor; \
}
if (blob == nullptr) {
return {};
} else if (auto remote_blob = std::dynamic_pointer_cast<TensorRemoteBlob>(blob)) {
return remote_blob->tensor;
} else if (auto remote_blob = std::dynamic_pointer_cast<InferenceEngine::RemoteBlob>(blob)) {
return std::make_shared<RemoteBlobTensor>(remote_blob);
}
ELSE_IF(float)
ELSE_IF(double)
ELSE_IF(int8_t)
ELSE_IF(int8_t)
ELSE_IF(int16_t)
ELSE_IF(int32_t)
ELSE_IF(int64_t)
ELSE_IF(uint8_t)
ELSE_IF(uint8_t)
ELSE_IF(uint16_t)
ELSE_IF(uint32_t)
ELSE_IF(uint64_t)
ELSE_IF(int8_t)
ELSE_IF(bool) else {
return std::make_shared<BlobTensor>(blob);
}
#undef IF
}
ie::Blob::Ptr tensor_to_blob(const std::shared_ptr<ITensor>& tensor) {
if (tensor == nullptr) {
return {};
} else if (auto blob_tensor = std::dynamic_pointer_cast<BlobTensor>(tensor)) {
return blob_tensor->blob;
} else if (auto blob_tensor = std::dynamic_pointer_cast<RemoteBlobTensor>(tensor)) {
return blob_tensor->blob;
} else if (auto blob_tensor = dynamic_cast<const BlobTensor*>(tensor.get())) {
return blob_tensor->blob;
} else if (std::dynamic_pointer_cast<ov::IRemoteTensor>(tensor)) {
return std::make_shared<TensorRemoteBlob>(tensor);
} else {
#define CASE(precision, T) \
case element::precision: \
return std::make_shared<TensorMemoryBlob<T>>(tensor);
switch (tensor->get_element_type()) {
CASE(f32, float);
CASE(f64, double);
CASE(i4, int8_t);
CASE(i8, int8_t);
CASE(i16, int16_t);
CASE(i32, int32_t);
CASE(i64, int64_t);
CASE(u4, uint8_t);
CASE(u8, uint8_t);
CASE(u16, uint16_t);
CASE(u32, uint32_t);
CASE(u64, uint64_t);
CASE(u1, int8_t);
CASE(boolean, bool);
case element::f16:
return std::make_shared<TensorMemoryBlob<int16_t>>(tensor);
case element::bf16:
return std::make_shared<TensorMemoryBlob<int16_t>>(tensor);
default:
OPENVINO_THROW("Unsupported element type");
}
#undef CASE
}
OPENVINO_THROW("Cannot convert tensor to blob!");
}
} // namespace ov

8
src/inference/src/dev/isync_infer_request.cpp
View File

@ -7,7 +7,6 @@
#include <unordered_map>
#include "cpp_interfaces/plugin_itt.hpp"
#include "ie_blob.h"
#include "openvino/core/except.hpp"
#include "openvino/core/layout.hpp"
#include "openvino/core/parallel.hpp"
@ -248,16 +247,11 @@ void ov::ISyncInferRequest::check_tensor(const ov::Output<const ov::Node>& port,
" tensor size is not equal to the model ",
tensor_type,
" type: got ",
tensor.get_size(),
tensor.get_shape(),
" expecting ",
port.get_shape(),
".");
OPENVINO_ASSERT(tensor.data() != nullptr, "Tensor data equal nullptr!");
// FIXME: SyncInferRequest is a friend only to check that blob is correct
OPENVINO_ASSERT(ov::shape_size(tensor._impl->getTensorDesc().getDims()) ==
ov::shape_size(tensor._impl->getTensorDesc().getBlockingDesc().getBlockDims()),
"Tensor is corrupted!");
}
void ov::ISyncInferRequest::allocate_tensor(const ov::Output<const ov::Node>& port,

52
src/inference/src/dev/make_tensor.hpp Normal file
View File

@ -0,0 +1,52 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#pragma once
#include "ie_blob.h"
#include "openvino/runtime/itensor.hpp"
namespace ov {
/**
* @brief Constructs Tensor using element type and shape. Allocate internal host storage using default allocator
* @param type Tensor element type
* @param shape Tensor shape
* @param allocator allocates memory for internal tensor storage
*/
std::shared_ptr<ITensor> make_tensor(const element::Type type, const Shape& shape, const Allocator& allocator = {});
/**
* @brief Constructs Tensor using element type and shape. Wraps allocated host memory.
* @note Does not perform memory allocation internally
* @param type Tensor element type
* @param shape Tensor shape
* @param host_ptr Pointer to pre-allocated host memory
* @param strides Optional strides parameters in bytes. Strides are supposed to be computed automatically based
* on shape and element size
*/
std::shared_ptr<ITensor> make_tensor(const element::Type type,
const Shape& shape,
void* host_ptr,
const Strides& strides = {});
/**
* @brief Constructs region of interest (ROI) tensor form another tensor.
* @note Does not perform memory allocation internally
* @param other original tensor
* @param begin start coordinate of ROI object inside of the original object.
* @param end end coordinate of ROI object inside of the original object.
* @note A Number of dimensions in `begin` and `end` must match number of dimensions in `other.get_shape()`
*/
std::shared_ptr<ITensor> make_tensor(const std::shared_ptr<ITensor>& other,
const Coordinate& begin,
const Coordinate& end);
/** @cond INTERNAL */
std::shared_ptr<ITensor> make_tensor(const std::shared_ptr<InferenceEngine::Blob>& tensor);
std::shared_ptr<InferenceEngine::Blob> tensor_to_blob(const std::shared_ptr<ITensor>& tensor);
/** @endcond */
} // namespace ov

35
src/inference/src/remote_tensor.cpp
View File

@ -4,22 +4,26 @@
#include "openvino/runtime/remote_tensor.hpp"
#include "any_copy.hpp"
#include "ie_ngraph_utils.hpp"
#include "ie_remote_blob.hpp"
#include <memory>
#include "openvino/runtime/iremote_tensor.hpp"
#include "openvino/runtime/itensor.hpp"
#include "openvino/runtime/properties.hpp"
namespace ov {
void RemoteTensor::type_check(const Tensor& tensor, const std::map<std::string, std::vector<std::string>>& type_info) {
OPENVINO_ASSERT(tensor, "Could not check empty tensor type");
auto remote_tensor = static_cast<const RemoteTensor*>(&tensor);
auto remote_impl = dynamic_cast<ie::RemoteBlob*>(remote_tensor->_impl.get());
OPENVINO_ASSERT(remote_impl != nullptr, "Tensor was not initialized using remote implementation");
auto remote_tensor = std::dynamic_pointer_cast<ov::IRemoteTensor>(tensor._impl);
OPENVINO_ASSERT(remote_tensor, "Tensor is not remote.");
if (!type_info.empty()) {
auto params = remote_impl->getParams();
auto remote_properties = remote_tensor->get_properties();
for (auto&& type_info_value : type_info) {
auto it_param = params.find(type_info_value.first);
OPENVINO_ASSERT(it_param != params.end(), "Parameter with key ", type_info_value.first, " not found");
auto it_param = remote_properties.find(type_info_value.first);
OPENVINO_ASSERT(it_param != remote_properties.end(),
"Parameter with key ",
type_info_value.first,
" not found");
if (!type_info_value.second.empty()) {
auto param_value = it_param->second.as<std::string>();
auto param_found = std::any_of(type_info_value.second.begin(),
@ -34,12 +38,12 @@ void RemoteTensor::type_check(const Tensor& tensor, const std::map<std::string,
}
AnyMap RemoteTensor::get_params() const {
OPENVINO_ASSERT(_impl != nullptr, "Remote tensor was not initialized.");
OPENVINO_ASSERT(_impl != nullptr, "Tensor was not initialized.");
type_check(*this);
auto remote_impl = static_cast<ie::RemoteBlob*>(_impl.get());
auto remote_tensor = std::dynamic_pointer_cast<ov::IRemoteTensor>(_impl);
try {
AnyMap paramMap;
for (auto&& param : remote_impl->getParams()) {
for (auto&& param : remote_tensor->get_properties()) {
paramMap.emplace(param.first, Any{param.second, _so});
}
return paramMap;
@ -51,11 +55,12 @@ AnyMap RemoteTensor::get_params() const {
}
std::string RemoteTensor::get_device_name() const {
OPENVINO_ASSERT(_impl != nullptr, "Remote tensor was not initialized.");
auto remote_impl = static_cast<ie::RemoteBlob*>(_impl.get());
OPENVINO_ASSERT(_impl != nullptr, "Tensor was not initialized.");
auto remote_tensor = std::dynamic_pointer_cast<ov::IRemoteTensor>(_impl);
OPENVINO_ASSERT(remote_tensor, "Tensor is not remote.");
type_check(*this);
try {
return remote_impl->getDeviceName();
return remote_tensor->get_device_name();
} catch (const std::exception& ex) {
OPENVINO_THROW(ex.what());
} catch (...) {

2
src/inference/tests/unit/query_model_test.cpp
View File

@ -145,7 +145,7 @@ TEST_F(GetSupportedNodesTest, SupportedCompressedConstantNop) {
[&](std::shared_ptr<ov::Model>& model) {
ov::pass::Manager m;
m.register_pass<ov::pass::InitNodeInfo>();
m.register_pass<ov::pass::ConvertPrecision>(precisions_array{{ngraph::element::f16, ngraph::element::f32}});
m.register_pass<ov::pass::ConvertPrecision>(precisions_map{{ngraph::element::f16, ngraph::element::f32}});
m.register_pass<ov::pass::NopElimination>();
m.run_passes(model);
},

2
src/plugins/intel_cpu/src/ngraph_transformations/convert_to_cpu_specific_opset.hpp
View File

@ -44,7 +44,7 @@ inline void ConvertToCPUSpecificOpset(std::shared_ptr<ngraph::Function> &nGraphF
// after transformation "MoveEltwiseUpThroughDataMov" there can be Reshape sequences that should be eliminated or fused
manager.register_pass<ov::pass::ReshapeSequenceFusion>();
manager.register_pass<ngraph::pass::ConstantFolding>();
manager.register_pass<ov::pass::ConvertPrecision>(precisions_array {{ ngraph::element::i64, ngraph::element::i32 }});
manager.register_pass<ov::pass::ConvertPrecision>(precisions_map {{ ngraph::element::i64, ngraph::element::i32 }});
manager.register_pass<ov::pass::Validate>();
manager.run_passes(nGraphFunc);

13
src/plugins/intel_cpu/src/transformation_pipeline.cpp
View File

@ -112,7 +112,12 @@ namespace intel_cpu {
using const_node_ptr = const std::shared_ptr<const ov::Node>;
bool Transformations::fuse_type_to_convert(const std::shared_ptr<ngraph::Node>& node, ov::element::Type to, size_t idx) {
bool Transformations::fuse_type_to_convert(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions) {
const auto& from = node->get_output_element_type(0);
auto it = precisions.find(from);
if (it == precisions.end())
return false;
const auto& to = it->second;
if (auto convert = ov::as_type_ptr<ov::opset10::Convert>(node)) {
// For Convert node, converting precision from floating point to boolean will lead to mathematical
// error, because here the output precision boolean is replaced by u8. E.g. floating point value 0.01
@ -187,7 +192,7 @@ void Transformations::PreLpt(const std::vector<ov::element::Type>& defaultPrecis
}
auto get_convert_precisions = []() {
precisions_array array = {
precisions_map map = {
{ov::element::i64, ov::element::i32},
{ov::element::u64, ov::element::i32},
{ov::element::i16, ov::element::i32},
@ -201,9 +206,9 @@ void Transformations::PreLpt(const std::vector<ov::element::Type>& defaultPrecis
};
if (!dnnl::impl::cpu::x64::mayiuse(dnnl::impl::cpu::x64::avx512_core))
array.push_back({ov::element::bf16, ov::element::f32});
map.insert({ov::element::bf16, ov::element::f32});
return array;
return map;
};
static const auto precisions = get_convert_precisions();
type_to_fuse_map type_to_fuse = {{ov::opset10::Convert::get_type_info_static(), fuse_type_to_convert}};

3
src/plugins/intel_cpu/src/transformation_pipeline.h
View File

@ -8,6 +8,7 @@
#include "utils/debug_capabilities.h"
#include "low_precision/low_precision.hpp"
#include "config.h"
#include "transformations/convert_precision.hpp"
#include "itt.h"
@ -61,7 +62,7 @@ private:
void Snippets(void);
static bool fuse_type_to_convert(const std::shared_ptr<ngraph::Node>& node, ov::element::Type to, size_t idx);
static bool fuse_type_to_convert(const std::shared_ptr<ngraph::Node>& node, const precisions_map& precisions);
};
} // namespace intel_cpu

6
src/plugins/intel_gna/legacy/tests/convert_ngraph_to_cnn_network_tests.cpp
View File

@ -180,7 +180,7 @@ TEST(ConvertFunctionToCNNNetworkTests, ConvertTopKWithOneInput) {
manager.register_pass<ov::pass::ConvertOpSet3ToOpSet2>();
manager.register_pass<ov::pass::ConvertOpSet2ToOpSet1>();
static const precisions_array convert_precision_list{
static const precisions_map convert_precision_map{
{ngraph::element::i64, ngraph::element::i32},
{ngraph::element::u64, ngraph::element::i32},
{ngraph::element::u16, ngraph::element::i32},
@ -189,9 +189,9 @@ TEST(ConvertFunctionToCNNNetworkTests, ConvertTopKWithOneInput) {
{ngraph::element::boolean, ngraph::element::u8},
};
manager.register_pass<ov::pass::ConvertPrecision>(convert_precision_list);
manager.register_pass<ov::pass::ConvertPrecision>(convert_precision_map);
manager.register_pass<ngraph::pass::ConvertOpSet1ToLegacy>();
manager.register_pass<ov::pass::ConvertPrecision>(precisions_array{{ngraph::element::i64, ngraph::element::i32}});
manager.register_pass<ov::pass::ConvertPrecision>(precisions_map{{ngraph::element::i64, ngraph::element::i32}});
manager.run_passes(f);

8
src/plugins/intel_gna/src/gna_transformations_pipeline.cpp
View File

@ -67,7 +67,7 @@ void TransformationsPipeline::apply(const std::shared_ptr<ov::Model>& model) {
// In OV API 2.0(IRv10) default convertion to fp32 (inputs, outputs and weights) is disabled
// and we need to run the ConvertPrecision transformation to support old networks.
manager.register_pass<ov::pass::ConvertPrecision>(precisions_array{{ngraph::element::f16, ngraph::element::f32}});
manager.register_pass<ov::pass::ConvertPrecision>(precisions_map{{ngraph::element::f16, ngraph::element::f32}});
manager.register_pass<ov::pass::ConvertMVN1ToMVN6>();
manager.register_pass<ov::intel_gna::pass::DecomposeMVN>();
manager.register_pass<ov::pass::CommonOptimizations>();
@ -148,9 +148,9 @@ void TransformationsPipeline::apply(const std::shared_ptr<ov::Model>& model) {
manager.register_pass<ov::intel_gna::pass::InsertCopyBeforeConcatLayer>();
manager.register_pass<ov::intel_gna::pass::HandleMultiConnectedLayerToConcatAndMemory>();
manager.register_pass<ov::intel_gna::pass::HandleNonFunctionalSubgraphs>();
manager.register_pass<ov::pass::ConvertPrecision>(precisions_array{{ov::element::i64, ov::element::i32},
{ov::element::u64, ov::element::i32},
{ov::element::u32, ov::element::i32}});
manager.register_pass<ov::pass::ConvertPrecision>(precisions_map{{ov::element::i64, ov::element::i32},
{ov::element::u64, ov::element::i32},
{ov::element::u32, ov::element::i32}});
const auto& pass_config = manager.get_pass_config();
// Allowing FP16 Converts to be folded and FP16 constants to upgrade to FP32 data type

55
src/plugins/intel_gpu/include/intel_gpu/graph/kernel_impl_params.hpp
View File

@ -24,6 +24,12 @@ struct program;
struct kernel_impl_params {
struct Hasher {
size_t operator()(const kernel_impl_params &k) const {
return k.hash();
}
};
bool has_runtime_layouts = false;
const program *prog;
std::shared_ptr<const primitive> desc;
@ -116,53 +122,8 @@ struct kernel_impl_params {
return *prog;
}
size_t hash() const {
size_t seed = desc->hash();
const size_t prime_number = 2654435761; // magic number to reduce hash collision rate.
for (auto& in : input_layouts) {
seed = hash_combine(seed, in.hash() * prime_number);
}
for (auto& out : output_layouts) {
seed = hash_combine(seed, out.hash() * prime_number);
}
// hash for fused prims
for (auto& fd : fused_desc) {
seed = hash_combine(seed, fd.desc->hash());
}
return seed;
}
bool operator==(const kernel_impl_params& rhs) const {
if (*desc != *rhs.desc)
return false;
if (rhs.input_layouts.size() != input_layouts.size())
return false;
if (rhs.output_layouts.size() != output_layouts.size())
return false;
for (size_t i = 0; i < input_layouts.size(); i++) {
if (input_layouts[i] != rhs.input_layouts[i])
return false;
}
for (size_t i = 0; i < output_layouts.size(); i++) {
if (output_layouts[i] != rhs.output_layouts[i])
return false;
}
if (fused_desc.size() != rhs.fused_desc.size())
return false;
for (size_t i = 0; i < rhs.fused_desc.size(); i++) {
if (fused_desc[i] != rhs.fused_desc[i])
return false;
}
return true;
}
size_t hash() const;
bool operator==(const kernel_impl_params& rhs) const;
};
} // namespace cldnn

12
src/plugins/intel_gpu/include/intel_gpu/graph/program.hpp
View File

@ -225,7 +225,6 @@ public:
void add_optimized_primitive_info(primitive_id optimized_primitive_id, std::vector<primitive_id> replaced_with_ids = {});
void reset_program();
uint32_t get_id() const { return prog_id; }
static ptr build_program(engine& engine,
@ -240,8 +239,6 @@ public:
std::shared_ptr<InferenceEngine::CPUStreamsExecutor> task_executor,
bool is_internal);
static void init_primitives();
void compile();
void init_kernels();
kernel_id add_kernel(const std::shared_ptr<kernel_string>& kernel_sring);
kernel::ptr get_kernel(kernel_id id);
kernels_cache& get_kernels_cache() const;
@ -251,13 +248,7 @@ public:
void remove_kernel(kernel_id id);
struct ImplHasher {
size_t operator()(const kernel_impl_params &k) const {
return k.hash();
}
};
using ImplementationsCache = cldnn::LruCacheThreadSafe<kernel_impl_params, std::shared_ptr<primitive_impl>, ImplHasher>;
using ImplementationsCache = cldnn::LruCacheThreadSafe<kernel_impl_params, std::shared_ptr<primitive_impl>, kernel_impl_params::Hasher>;
ImplementationsCache& get_implementations_cache() const { return *_impls_cache; }
ICompilationContext& get_compilation_context() const { return *_compilation_context; }
void cancel_compilation_context();
@ -314,7 +305,6 @@ private:
void run_graph_compilation();
void pre_optimize_graph(bool is_internal);
void post_optimize_graph(bool is_internal);
void cleanup();
void transfer_memory_to_device();
InferenceEngine::CPUStreamsExecutor::Config make_task_executor_config(const ExecutionConfig& config, std::string tags = "") const;

6
src/plugins/intel_gpu/include/intel_gpu/plugin/compiled_model.hpp
View File

@ -24,8 +24,9 @@ class CompiledModel : public InferenceEngine::ExecutableNetworkThreadSafeDefault
public:
typedef std::shared_ptr<CompiledModel> Ptr;
CompiledModel(InferenceEngine::CNNNetwork &network, InferenceEngine::RemoteContext::Ptr context, const ExecutionConfig& config);
CompiledModel(std::istream& networkModel, InferenceEngine::RemoteContext::Ptr context, const ExecutionConfig& config);
CompiledModel(InferenceEngine::CNNNetwork &network, InferenceEngine::RemoteContext::Ptr context, const ExecutionConfig& config,
InferenceEngine::InputsDataMap* inputs = nullptr, InferenceEngine::OutputsDataMap* outputs = nullptr);
CompiledModel(cldnn::BinaryInputBuffer& ib, InferenceEngine::RemoteContext::Ptr context, const ExecutionConfig& config);
void Export(std::ostream& networkModel) override;
std::shared_ptr<ngraph::Function> GetExecGraphInfo() override;
@ -46,6 +47,7 @@ public:
ExecutionConfig m_config;
InferenceEngine::ITaskExecutor::Ptr m_taskExecutor;
InferenceEngine::ITaskExecutor::Ptr m_waitExecutor;
InferenceEngine::CNNNetwork m_network;
};
} // namespace intel_gpu

4
src/plugins/intel_gpu/include/intel_gpu/plugin/graph.hpp
View File

@ -43,7 +43,9 @@ public:
Graph(InferenceEngine::CNNNetwork& network,
RemoteContextImpl::Ptr context,
const ExecutionConfig& config,
uint16_t stream_id = 0);
uint16_t stream_id = 0,
InferenceEngine::InputsDataMap* inputs = nullptr,
InferenceEngine::OutputsDataMap* outputs = nullptr);
Graph(cldnn::BinaryInputBuffer& ib, RemoteContextImpl::Ptr context, const ExecutionConfig& config, uint16_t stream_id = 0);
explicit Graph(std::shared_ptr<Graph> graph, uint16_t stream_id = 0);
void Export(cldnn::BinaryOutputBuffer &ob);

2
src/plugins/intel_gpu/include/intel_gpu/plugin/legacy_api_helper.hpp
View File

@ -16,7 +16,7 @@ public:
static bool is_legacy_property(const std::pair<std::string, ov::Any>& property, bool is_new_api);
static bool is_new_api_property(const std::pair<std::string, ov::Any>& property);
static std::vector<std::string> get_supported_configs();
static std::vector<std::string> get_supported_metrics(bool model_caching_enabled);
static std::vector<std::string> get_supported_metrics();
};
} // namespace intel_gpu

1
src/plugins/intel_gpu/include/intel_gpu/plugin/plugin.hpp
View File

@ -18,7 +18,6 @@ namespace intel_gpu {
class Plugin : public InferenceEngine::IInferencePlugin {
struct impl;
std::shared_ptr<impl> _impl;
bool isModelCachingEnabled = true;
std::string default_device_id = "0";
// key: device_id, value: cldnn device

3
src/plugins/intel_gpu/include/intel_gpu/plugin/program.hpp
View File

@ -82,7 +82,8 @@ public:
class Program {
public:
Program(InferenceEngine::CNNNetwork& network, cldnn::engine& engine, const ExecutionConfig& config,
bool createTopologyOnly = false, bool partialBuild = false);
bool createTopologyOnly = false, bool partialBuild = false,
InferenceEngine::InputsDataMap* inputs = nullptr, InferenceEngine::OutputsDataMap* outputs = nullptr);
Program(cldnn::engine& engine, const ExecutionConfig& config)
: m_max_batch(1)
, m_curBatch(-1)

27
src/plugins/intel_gpu/src/graph/graph_optimizer/build_implementations.cpp Normal file
View File

@ -0,0 +1,27 @@
// Copyright (C) 2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include "pass_manager.h"
#include "program_helpers.h"
#include "intel_gpu/runtime/itt.hpp"
using namespace cldnn;
void build_implementations::run(program& p) {
OV_ITT_SCOPED_TASK(ov::intel_gpu::itt::domains::intel_gpu_plugin, "pass::build_implementations");
if (p.get_config().get_property(ov::intel_gpu::partial_build_program)) {
return;
}
auto& cache = p.get_kernels_cache();
cache.build_all();
for (auto& n : p.get_processing_order()) {
if (n->get_selected_impl()) {
n->get_selected_impl()->init_kernels(cache);
n->get_selected_impl()->reset_kernels_source();
}
}
cache.reset();
}

31
src/plugins/intel_gpu/src/graph/impls/ocl/concatenation.cpp
View File

@ -54,9 +54,9 @@ struct concatenation_impl : typed_primitive_impl_ocl<concatenation> {
}
public:
static kernel_params_t get_kernel_params(const kernel_impl_params& impl_param) {
static kernel_params_t get_kernel_params(const kernel_impl_params& impl_param, bool is_shape_agnostic = false) {
const auto& primitive = impl_param.typed_desc<concatenation>();
auto params = get_default_params<kernel_selector::concatenation_params>(impl_param);
auto params = get_default_params<kernel_selector::concatenation_params>(impl_param, is_shape_agnostic);
auto optional_params = get_default_optional_params<kernel_selector::concatenation_optional_params>(impl_param.get_program());
auto axis = primitive->axis;
@ -72,11 +72,38 @@ public:
return {params, optional_params};
}
void update_dispatch_data(const kernel_impl_params& impl_param) override {
auto kernel_params = get_kernel_params(impl_param, true);
(_kernel_data.update_dispatch_data_func)(kernel_params.first, _kernel_data);
update_kernels_list_to_skip();
}
};
namespace detail {
attach_concatenation_impl::attach_concatenation_impl() {
auto dyn_types = {
data_types::i8,
data_types::u8,
data_types::f16,
data_types::f32,
data_types::i32,
data_types::i64
};
auto dyn_formats = {
format::bfyx,
format::bfzyx,
format::bfwzyx
};
implementation_map<concatenation>::add(impl_types::ocl,
shape_types::dynamic_shape,
typed_primitive_impl_ocl<concatenation>::create<concatenation_impl>,
dyn_types,
dyn_formats);
implementation_map<concatenation>::add(impl_types::ocl, typed_primitive_impl_ocl<concatenation>::create<concatenation_impl>, {
std::make_tuple(data_types::f32, format::yxfb),
std::make_tuple(data_types::f16, format::yxfb),

126
src/plugins/intel_gpu/src/graph/impls/ocl/kernel_selector_helper.cpp
View File

@ -1246,130 +1246,4 @@ void set_optional_params(const program& program, kernel_selector::optional_param
params.allowOutputReordering = false;
}
void kernel_impl_params::save(BinaryOutputBuffer& ob) const {
ob << desc;
ob << has_runtime_layouts;
ob << unique_id;
ob << input_layouts;
ob << output_layouts;
ob << input_offsets.size();
for (size_t i = 0; i < input_offsets.size(); i++) {
ob << input_offsets[i].sizes();
}
if (weights_layout.has_value()) {
ob << true;
ob << weights_layout.value();
} else {
ob << false;
}
if (bias_layout.has_value()) {
ob << true;
ob << bias_layout.value();
} else {
ob << false;
}
if (weights_zero_points_layout.has_value()) {
ob << true;
ob << weights_zero_points_layout.value();
} else {
ob << false;
}
if (activations_zero_points_layout.has_value()) {
ob << true;
ob << activations_zero_points_layout.value();
} else {
ob << false;
}
if (compensation_layout.has_value()) {
ob << true;
ob << compensation_layout.value();
} else {
ob << false;
}
ob << fused_desc.size();
#ifdef ENABLE_ONEDNN_FOR_GPU
size_t num_fused_prims = fused_desc_onednn.size();
ob << num_fused_prims;
for (auto fused_prim : fused_desc_onednn) {
ob << make_data(&fused_prim, sizeof(fused_primitive_desc_onednn));
}
#endif // ENABLE_ONEDNN_FOR_GPU
ob << primary_input_idx;
}
void kernel_impl_params::load(BinaryInputBuffer& ib) {
prog = nullptr;
ib >> desc;
ib >> has_runtime_layouts;
ib >> unique_id;
ib >> input_layouts;
ib >> output_layouts;
{
size_t num_input_offsets;
ib >> num_input_offsets;
input_offsets.resize(num_input_offsets);
for (size_t i = 0; i < num_input_offsets; i++) {
std::vector<cldnn::tensor::value_type> sizes;
ib >> sizes;
input_offsets[i] = cldnn::tensor(sizes);
}
}
bool has_value = false;
layout layout_buf;
ib >> has_value;
if (has_value) {
ib >> layout_buf;
weights_layout = layout_buf;
}
ib >> has_value;
if (has_value) {
ib >> layout_buf;
bias_layout = layout_buf;
}
ib >> has_value;
if (has_value) {
ib >> layout_buf;
weights_zero_points_layout = layout_buf;
}
ib >> has_value;
if (has_value) {
ib >> layout_buf;
activations_zero_points_layout = layout_buf;
}
ib >> has_value;
if (has_value) {
ib >> layout_buf;
compensation_layout = layout_buf;
}
{
// Fake fused_desc just for has_fused_primitives()
size_t num_fused_desc;
ib >> num_fused_desc;
if (num_fused_desc > 0) {
fused_desc.emplace_back(cldnn::fused_primitive_desc(nullptr));
}
}
#ifdef ENABLE_ONEDNN_FOR_GPU
size_t num_fused_prims;
ib >> num_fused_prims;
fused_desc_onednn.resize(num_fused_prims);
for (size_t idx = 0; idx < num_fused_prims; ++idx) {
ib >> make_data(&fused_desc_onednn[idx], sizeof(fused_primitive_desc_onednn));
}
#endif // ENABLE_ONEDNN_FOR_GPU
ib >> primary_input_idx;
}
} // namespace cldnn

52
src/plugins/intel_gpu/src/graph/impls/ocl/kernel_selector_helper.h
View File

@ -239,4 +239,56 @@ template <typename optional_params_t>
inline optional_params_t get_default_weights_bias_optional_params(const program& program) {
return get_default_optional_params<optional_params_t>(program);
}
inline kernel_selector::eltwise_mode convert_to_eltwise_mode(eltwise_mode mode) {
switch (mode) {
case eltwise_mode::sum:
return kernel_selector::eltwise_mode::ADD;
case eltwise_mode::sub:
return kernel_selector::eltwise_mode::SUB;
case eltwise_mode::max:
return kernel_selector::eltwise_mode::MAX;
case eltwise_mode::prod:
return kernel_selector::eltwise_mode::MUL;
case eltwise_mode::div:
return kernel_selector::eltwise_mode::DIV;
case eltwise_mode::min:
return kernel_selector::eltwise_mode::MIN;
case eltwise_mode::pow:
return kernel_selector::eltwise_mode::POW;
case eltwise_mode::mod:
return kernel_selector::eltwise_mode::MODULU;
case eltwise_mode::eq:
return kernel_selector::eltwise_mode::EQ;
case eltwise_mode::ne:
return kernel_selector::eltwise_mode::NE;
case eltwise_mode::lt:
return kernel_selector::eltwise_mode::LT;
case eltwise_mode::le:
return kernel_selector::eltwise_mode::LE;
case eltwise_mode::gt:
return kernel_selector::eltwise_mode::GT;
case eltwise_mode::ge:
return kernel_selector::eltwise_mode::GE;
case eltwise_mode::logic_and:
return kernel_selector::eltwise_mode::LOGIC_AND;
case eltwise_mode::logic_or:
return kernel_selector::eltwise_mode::LOGIC_OR;
case eltwise_mode::logic_xor:
return kernel_selector::eltwise_mode::LOGIC_XOR;
case eltwise_mode::squared_diff:
return kernel_selector::eltwise_mode::SQUARED_DIFF;
case eltwise_mode::floor_mod:
return kernel_selector::eltwise_mode::FLOOR_MOD;
case eltwise_mode::is_finite:
return kernel_selector::eltwise_mode::IS_FINITE;
case eltwise_mode::is_inf:
return kernel_selector::eltwise_mode::IS_INF;
case eltwise_mode::is_nan:
return kernel_selector::eltwise_mode::IS_NAN;
default:
return kernel_selector::eltwise_mode::ADD;
}
}
} // namespace cldnn

23
src/plugins/intel_gpu/src/graph/impls/ocl/primitive_base.hpp
View File

@ -174,8 +174,12 @@ protected:
return;
}
OPENVINO_ASSERT(_kernels.size() == _kernel_data.kernels.size(), "[GPU] Mismatch between compiled kernels count and expected kernels data\n",
"[GPU] Compiled kernels count: ", _kernels.size(), "\n",
"[GPU] KernelData count: ", _kernel_data.kernels.size(), "\n",
"[GPU] Likely some issue with empty tensors hanlding happened");
stream& stream = instance.get_network().get_stream();
size_t k_idx = 0;
for (size_t kd_idx = 0; kd_idx < _kernel_data.kernels.size(); ++kd_idx) {
if (_kernel_data.kernels[kd_idx].skip_execution) {
continue;
@ -188,7 +192,7 @@ protected:
args.intermediates.push_back(m);
}
stream.set_arguments(*_kernels[k_idx++], _kernel_data.kernels[kd_idx].params, args);
stream.set_arguments(*_kernels[kd_idx], _kernel_data.kernels[kd_idx].params, args);
}
}
@ -216,7 +220,10 @@ protected:
}
std::vector<event::ptr> tmp_events(events);
std::vector<event::ptr> all_events;
size_t k_idx = 0;
OPENVINO_ASSERT(_kernels.size() == _kernel_data.kernels.size(), "[GPU] Mismatch between compiled kernels count and expected kernels data\n",
"[GPU] Compiled kernels count: ", _kernels.size(), "\n",
"[GPU] KernelData count: ", _kernel_data.kernels.size(), "\n",
"[GPU] Likely some issue with empty tensors hanlding happened");
for (size_t kd_idx = 0; kd_idx < _kernel_data.kernels.size(); ++kd_idx) {
if (_kernel_data.kernels[kd_idx].skip_execution)
continue;
@ -237,7 +244,7 @@ protected:
args.intermediates.push_back(m);
}
auto ev = stream.enqueue_kernel(*_kernels[k_idx++], _kernel_data.kernels[kd_idx].params, args, tmp_events, is_output_event);
auto ev = stream.enqueue_kernel(*_kernels[kd_idx], _kernel_data.kernels[kd_idx].params, args, tmp_events, is_output_event);
new_events.push_back(ev);
all_events.push_back(ev);
@ -258,8 +265,7 @@ protected:
std::vector<std::shared_ptr<cldnn::kernel_string>> get_kernels_source() override {
std::vector<std::shared_ptr<cldnn::kernel_string>> kernel_strings;
for (size_t i = 0; i < _kernel_data.kernels.size(); ++i) {
if (!_kernel_data.kernels[i].skip_execution)
kernel_strings.push_back(_kernel_data.kernels[i].code.kernelString);
kernel_strings.push_back(_kernel_data.kernels[i].code.kernelString);
}
return kernel_strings;
}
@ -272,9 +278,8 @@ protected:
void update_kernels_list_to_skip() {
for (size_t i = 0; i < _kernel_data.kernels.size(); ++i) {
auto gws = _kernel_data.kernels[0].params.workGroups.global;
_kernel_data.kernels[0].skip_execution =
(std::accumulate(gws.begin(), gws.end(), 1, std::multiplies<size_t>()) == 0);
auto gws = _kernel_data.kernels[i].params.workGroups.global;
_kernel_data.kernels[i].skip_execution = (std::accumulate(gws.begin(), gws.end(), 1, std::multiplies<size_t>()) == 0);
}
}

51
src/plugins/intel_gpu/src/graph/include/eltwise_inst.h
View File

@ -11,57 +11,6 @@
namespace cldnn {
inline kernel_selector::eltwise_mode convert_to_eltwise_mode(eltwise_mode mode) {
switch (mode) {
case eltwise_mode::sum:
return kernel_selector::eltwise_mode::ADD;
case eltwise_mode::sub:
return kernel_selector::eltwise_mode::SUB;
case eltwise_mode::max:
return kernel_selector::eltwise_mode::MAX;
case eltwise_mode::prod:
return kernel_selector::eltwise_mode::MUL;
case eltwise_mode::div:
return kernel_selector::eltwise_mode::DIV;
case eltwise_mode::min:
return kernel_selector::eltwise_mode::MIN;
case eltwise_mode::pow:
return kernel_selector::eltwise_mode::POW;
case eltwise_mode::mod:
return kernel_selector::eltwise_mode::MODULU;
case eltwise_mode::eq:
return kernel_selector::eltwise_mode::EQ;
case eltwise_mode::ne:
return kernel_selector::eltwise_mode::NE;
case eltwise_mode::lt:
return kernel_selector::eltwise_mode::LT;
case eltwise_mode::le:
return kernel_selector::eltwise_mode::LE;
case eltwise_mode::gt:
return kernel_selector::eltwise_mode::GT;
case eltwise_mode::ge:
return kernel_selector::eltwise_mode::GE;
case eltwise_mode::logic_and:
return kernel_selector::eltwise_mode::LOGIC_AND;
case eltwise_mode::logic_or:
return kernel_selector::eltwise_mode::LOGIC_OR;
case eltwise_mode::logic_xor:
return kernel_selector::eltwise_mode::LOGIC_XOR;
case eltwise_mode::squared_diff:
return kernel_selector::eltwise_mode::SQUARED_DIFF;
case eltwise_mode::floor_mod:
return kernel_selector::eltwise_mode::FLOOR_MOD;
case eltwise_mode::is_finite:
return kernel_selector::eltwise_mode::IS_FINITE;
case eltwise_mode::is_inf:
return kernel_selector::eltwise_mode::IS_INF;
case eltwise_mode::is_nan:
return kernel_selector::eltwise_mode::IS_NAN;
default:
return kernel_selector::eltwise_mode::ADD;
}
}
class EltwiseFuseParams : public NodeFuseParams {
public:
EltwiseFuseParams(std::shared_ptr<eltwise> desc) : NodeFuseParams(eltwise::type_id()), _desc(desc) {}

6
src/plugins/intel_gpu/src/graph/include/pass_manager.h
View File

@ -415,4 +415,10 @@ public:
void run(program& p) override;
};
class build_implementations : public base_pass {
public:
build_implementations() : base_pass("build_implementations") {}
void run(program& p) override;
};
} // namespace cldnn

191
src/plugins/intel_gpu/src/graph/kernel_impl_params.cpp Normal file
View File

@ -0,0 +1,191 @@
// Copyright (C) 2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include "intel_gpu/graph/kernel_impl_params.hpp"
#include "intel_gpu/graph/program.hpp"
#include "intel_gpu/graph/serialization/polymorphic_serializer.hpp"
#include "intel_gpu/graph/serialization/layout_serializer.hpp"
#include "intel_gpu/graph/serialization/string_serializer.hpp"
#include "intel_gpu/graph/serialization/vector_serializer.hpp"
#include <string>
#include <vector>
namespace cldnn {
size_t kernel_impl_params::hash() const {
size_t seed = desc->hash();
const size_t prime_number = 2654435761; // magic number to reduce hash collision rate.
for (auto& in : input_layouts) {
seed = hash_combine(seed, in.hash() * prime_number);
}
for (auto& out : output_layouts) {
seed = hash_combine(seed, out.hash() * prime_number);
}
// hash for fused prims
for (auto& fd : fused_desc) {
seed = hash_combine(seed, fd.desc->hash());
}
return seed;
}
bool kernel_impl_params::operator==(const kernel_impl_params& rhs) const {
if (*desc != *rhs.desc)
return false;
if (rhs.input_layouts.size() != input_layouts.size())
return false;
if (rhs.output_layouts.size() != output_layouts.size())
return false;
for (size_t i = 0; i < input_layouts.size(); i++) {
if (input_layouts[i] != rhs.input_layouts[i])
return false;
}
for (size_t i = 0; i < output_layouts.size(); i++) {
if (output_layouts[i] != rhs.output_layouts[i])
return false;
}
if (fused_desc.size() != rhs.fused_desc.size())
return false;
for (size_t i = 0; i < rhs.fused_desc.size(); i++) {
if (fused_desc[i] != rhs.fused_desc[i])
return false;
}
return true;
}
void kernel_impl_params::save(BinaryOutputBuffer& ob) const {
ob << desc;
ob << has_runtime_layouts;
ob << unique_id;
ob << input_layouts;
ob << output_layouts;
ob << input_offsets.size();
for (size_t i = 0; i < input_offsets.size(); i++) {
ob << input_offsets[i].sizes();
}
if (weights_layout.has_value()) {
ob << true;
ob << weights_layout.value();
} else {
ob << false;
}
if (bias_layout.has_value()) {
ob << true;
ob << bias_layout.value();
} else {
ob << false;
}
if (weights_zero_points_layout.has_value()) {
ob << true;
ob << weights_zero_points_layout.value();
} else {
ob << false;
}
if (activations_zero_points_layout.has_value()) {
ob << true;
ob << activations_zero_points_layout.value();
} else {
ob << false;
}
if (compensation_layout.has_value()) {
ob << true;
ob << compensation_layout.value();
} else {
ob << false;
}
ob << fused_desc.size();
#ifdef ENABLE_ONEDNN_FOR_GPU
size_t num_fused_prims = fused_desc_onednn.size();
ob << num_fused_prims;
for (auto fused_prim : fused_desc_onednn) {
ob << make_data(&fused_prim, sizeof(fused_primitive_desc_onednn));
}
#endif // ENABLE_ONEDNN_FOR_GPU
ob << primary_input_idx;
}
void kernel_impl_params::load(BinaryInputBuffer& ib) {
prog = nullptr;
ib >> desc;
ib >> has_runtime_layouts;
ib >> unique_id;
ib >> input_layouts;
ib >> output_layouts;
{
size_t num_input_offsets;
ib >> num_input_offsets;
input_offsets.resize(num_input_offsets);
for (size_t i = 0; i < num_input_offsets; i++) {
std::vector<cldnn::tensor::value_type> sizes;
ib >> sizes;
input_offsets[i] = cldnn::tensor(sizes);
}
}
bool has_value = false;
layout layout_buf;
ib >> has_value;
if (has_value) {
ib >> layout_buf;
weights_layout = layout_buf;
}
ib >> has_value;
if (has_value) {
ib >> layout_buf;
bias_layout = layout_buf;
}
ib >> has_value;
if (has_value) {
ib >> layout_buf;
weights_zero_points_layout = layout_buf;
}
ib >> has_value;
if (has_value) {
ib >> layout_buf;
activations_zero_points_layout = layout_buf;
}
ib >> has_value;
if (has_value) {
ib >> layout_buf;
compensation_layout = layout_buf;
}
{
// Fake fused_desc just for has_fused_primitives()
size_t num_fused_desc;
ib >> num_fused_desc;
if (num_fused_desc > 0) {
fused_desc.emplace_back(cldnn::fused_primitive_desc(nullptr));
}
}
#ifdef ENABLE_ONEDNN_FOR_GPU
size_t num_fused_prims;
ib >> num_fused_prims;
fused_desc_onednn.resize(num_fused_prims);
for (size_t idx = 0; idx < num_fused_prims; ++idx) {
ib >> make_data(&fused_desc_onednn[idx], sizeof(fused_primitive_desc_onednn));
}
#endif // ENABLE_ONEDNN_FOR_GPU
ib >> primary_input_idx;
}
} // namespace cldnn

13
src/plugins/intel_gpu/src/graph/layout_optimizer.cpp
View File

@ -1824,13 +1824,18 @@ void layout_optimizer::select_preferred_formats_for_onednn(program_node& node, d
for (size_t idx = 0 ; idx < node.get_dependencies().size() ; idx++) {
if (node.get_dependency(idx).is_constant())
continue;
node.set_preferred_input_fmt(idx, cldnn::format::bfyx);
size_t out_rank = node.get_output_layout().get_rank();
auto target_format = format::get_default_format(out_rank);
node.set_preferred_input_fmt(idx, target_format);
if (node.get_preferred_output_fmt() == format::any) {
for (size_t usr = 0; usr < std::max<size_t>(1, node.get_users().size()); usr++)
node.set_preferred_output_fmt(usr, cldnn::format::bfyx);
for (size_t usr = 0; usr < std::max<size_t>(1, node.get_users().size()); usr++) {
node.set_preferred_output_fmt(usr, target_format);
}
}
GPU_DEBUG_LOG << "select_preferred_formats:" << node.id() << ": " << fmt_to_str(cldnn::format::bfyx) << " --> " << fmt_to_str(cldnn::format::bfyx)
GPU_DEBUG_LOG << "select_preferred_formats:" << node.id() << ": " << fmt_to_str(target_format) << " --> " << fmt_to_str(target_format)
<< " For index : " << idx << std::endl;
}
}

41
src/plugins/intel_gpu/src/graph/program.cpp
View File

@ -229,21 +229,6 @@ std::shared_ptr<InferenceEngine::CPUStreamsExecutor> program::make_task_executor
return std::make_shared<InferenceEngine::CPUStreamsExecutor>(task_executor_config);
}
void program::compile() {
GPU_DEBUG_DEFINE_MEM_LOGGER("compile");
_kernels_cache->build_all();
}
void program::init_kernels() {
GPU_DEBUG_DEFINE_MEM_LOGGER("init_kernels");
for (auto& n : get_processing_order()) {
if (n->get_selected_impl()) {
n->get_selected_impl()->init_kernels(*_kernels_cache);
n->get_selected_impl()->reset_kernels_source();
}
}
}
kernel_id program::add_kernel(const std::shared_ptr<kernel_string>& kernelSring) {
return _kernels_cache->set_kernel_source(kernelSring, false);
}
@ -516,13 +501,7 @@ void program::build_program(bool is_internal) {
{
#endif
prepare_memory_dependencies();
if (_config.get_property(ov::intel_gpu::partial_build_program)) {
return;
}
compile();
init_kernels();
apply_opt_pass<build_implementations>();
}
if (!is_internal) {
@ -530,12 +509,10 @@ void program::build_program(bool is_internal) {
if (get_engine().get_device_info().dev_type == device_type::discrete_gpu)
transfer_memory_to_device();
}
cleanup();
}
void program::init_graph() {
OV_ITT_SCOPED_TASK(ov::intel_gpu::itt::domains::intel_gpu_plugin, "ProgramImpl::InitGraph");
OV_ITT_SCOPED_TASK(ov::intel_gpu::itt::domains::intel_gpu_plugin, "Program::init_graph");
apply_opt_pass<graph_initializations>();
apply_opt_pass<calculate_prior_boxes>();
@ -546,7 +523,7 @@ void program::init_graph() {
void program::run_graph_compilation() { apply_opt_pass<compile_graph>(); }
void program::pre_optimize_graph(bool is_internal) {
OV_ITT_SCOPED_TASK(ov::intel_gpu::itt::domains::intel_gpu_plugin, "ProgramImpl::PreOptimizeGraph");
OV_ITT_SCOPED_TASK(ov::intel_gpu::itt::domains::intel_gpu_plugin, "Program::pre_optimize_graph");
// trim to outputs
apply_opt_pass<trim_to_outputs>(); // ToDo remove hidden dependencies from trimm pass
@ -626,7 +603,7 @@ void program::pre_optimize_graph(bool is_internal) {
}
void program::post_optimize_graph(bool is_internal) {
OV_ITT_SCOPED_TASK(ov::intel_gpu::itt::domains::intel_gpu_plugin, "ProgramImpl::PostOptimizeGraph");
OV_ITT_SCOPED_TASK(ov::intel_gpu::itt::domains::intel_gpu_plugin, "Program::post_optimize_graph");
// input reorder for fully connected if necessary
apply_opt_pass<post_input_reorder>();
@ -689,7 +666,7 @@ void program::mark_if_data_flow(program_node& node) {
void program::transfer_memory_to_device() {
GPU_DEBUG_DEFINE_MEM_LOGGER("transfer_memory_to_device");
OV_ITT_SCOPED_TASK(ov::intel_gpu::itt::domains::intel_gpu_plugin, "ProgramImpl::TransferMemory");
OV_ITT_SCOPED_TASK(ov::intel_gpu::itt::domains::intel_gpu_plugin, "Program::transfer_memory_to_device");
if (!get_engine().supports_allocation(allocation_type::usm_device))
return;
@ -724,14 +701,6 @@ void program::transfer_memory_to_device() {
}
}
void program::cleanup() {
GPU_DEBUG_DEFINE_MEM_LOGGER("cleanup");
for (auto& node : processing_order)
node->get_output_layout();
_kernels_cache->reset();
}
void program::add_split_outputs() {
auto itr = nodes_map.begin();
while (itr != nodes_map.end()) {

10
src/plugins/intel_gpu/src/kernel_selector/cl_kernels/concatenation_gpu_simple_ref.cl
View File

@ -4,7 +4,11 @@
#include "include/fetch_utils.cl"
KERNEL (concatenation_gpu_ref)(__global INPUT0_TYPE* input, __global OUTPUT_TYPE* output, uint output_offset_in_concat_axis)
KERNEL (concatenation_gpu_ref)(
OPTIONAL_SHAPE_INFO_ARG
__global INPUT0_TYPE* input,
__global OUTPUT_TYPE* output,
uint output_offset_in_concat_axis)
{
const uint x = (uint)get_global_id(0) % INPUT0_SIZE_X;
const uint y = (uint)get_global_id(0) / INPUT0_SIZE_X;
@ -36,8 +40,8 @@ KERNEL (concatenation_gpu_ref)(__global INPUT0_TYPE* input, __global OUTPUT_TYPE
# error concatenation_gpu_bfzyx_ref.cl: Unrecognized concat axis.
#endif
uint input_offset = FUNC_CALL(get_input_index)(b, f, w, z, y, x);
uint output_offset = FUNC_CALL(get_output_index)(out_b, out_f, out_w, out_z, out_y, out_x);
uint input_offset = FUNC_CALL(get_input_index)(OPTIONAL_SHAPE_INFO_TENSOR b, f, w, z, y, x);
uint output_offset = FUNC_CALL(get_output_index)(OPTIONAL_SHAPE_INFO_TENSOR out_b, out_f, out_w, out_z, out_y, out_x);
output[output_offset] = TO_OUTPUT_TYPE(ACTIVATION(input[input_offset], ACTIVATION_PARAMS));
}

60
src/plugins/intel_gpu/src/kernel_selector/kernel_base.cpp
View File

@ -71,7 +71,7 @@ Datatype KernelBase::GetUnitType(const base_params& params) const {
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// MakeBaseParamsJitConstants
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
JitConstants KernelBase::MakeBaseParamsJitConstants(const base_params& params) const {
JitConstants KernelBase::MakeBaseParamsJitConstants(const base_params& params, bool add_tensor_definitions) const {
auto unitType = GetUnitType(params);
JitConstants jit{
@ -89,40 +89,42 @@ JitConstants KernelBase::MakeBaseParamsJitConstants(const base_params& params) c
jit.Merge(MakeUnitTypeJitConstants(unitType));
jit.Merge(MakeActivationJitConstants(params.activations, unitType));
size_t dyn_tensor_idx = 0;
if (add_tensor_definitions) {
size_t dyn_tensor_idx = 0;
for (size_t i = 0; i < params.inputs.size(); i++) {
jit.AddConstant(MakeJitConstant("INPUT" + toCodeString(i), params.inputs[i], dyn_tensor_idx));
if (params.inputs[i].is_dynamic())
dyn_tensor_idx++;
}
for (size_t i = 0; i < params.fused_ops.size(); i++) {
auto& fused_op_inputs = params.fused_ops[i].tensors;
for (auto& t : fused_op_inputs) {
if (t.is_dynamic())
for (size_t i = 0; i < params.inputs.size(); i++) {
jit.AddConstant(MakeJitConstant("INPUT" + toCodeString(i), params.inputs[i], dyn_tensor_idx));
if (params.inputs[i].is_dynamic())
dyn_tensor_idx++;
}
}
// NOTE : until all cl kernels legacy is resolved, the outputs are to be OUTPUT, OUTPUT1, OUTPUT2, ...
jit.AddConstant(MakeJitConstant("OUTPUT", params.outputs[0], dyn_tensor_idx));
if (params.outputs[0].is_dynamic())
dyn_tensor_idx++;
for (size_t i = 1; i < params.outputs.size(); i++) {
jit.AddConstant(MakeJitConstant("OUTPUT" + toCodeString(i), params.outputs[i], dyn_tensor_idx));
for (size_t i = 0; i < params.fused_ops.size(); i++) {
auto& fused_op_inputs = params.fused_ops[i].tensors;
for (auto& t : fused_op_inputs) {
if (t.is_dynamic())
dyn_tensor_idx++;
}
}
// NOTE : until all cl kernels legacy is resolved, the outputs are to be OUTPUT, OUTPUT1, OUTPUT2, ...
jit.AddConstant(MakeJitConstant("OUTPUT", params.outputs[0], dyn_tensor_idx));
if (params.outputs[0].is_dynamic())
dyn_tensor_idx++;
}
dyn_tensor_idx++;
for (size_t i = 1; i < params.outputs.size(); i++) {
jit.AddConstant(MakeJitConstant("OUTPUT" + toCodeString(i), params.outputs[i], dyn_tensor_idx));
if (params.outputs[0].is_dynamic())
dyn_tensor_idx++;
}
if (dyn_tensor_idx > 0) {
jit.AddConstant(MakeJitConstant("IS_DYNAMIC", 1));
jit.AddConstant(MakeJitConstant("OPTIONAL_SHAPE_INFO_ARG", "__global const int* shape_info,"));
jit.AddConstant(MakeJitConstant("OPTIONAL_SHAPE_INFO_TENSOR", "shape_info,"));
} else {
jit.AddConstant(MakeJitConstant("OPTIONAL_SHAPE_INFO_ARG", ""));
jit.AddConstant(MakeJitConstant("OPTIONAL_SHAPE_INFO_TENSOR", ""));
if (dyn_tensor_idx > 0) {
jit.AddConstant(MakeJitConstant("IS_DYNAMIC", 1));
jit.AddConstant(MakeJitConstant("OPTIONAL_SHAPE_INFO_ARG", "__global const int* shape_info,"));
jit.AddConstant(MakeJitConstant("OPTIONAL_SHAPE_INFO_TENSOR", "shape_info,"));
} else {
jit.AddConstant(MakeJitConstant("OPTIONAL_SHAPE_INFO_ARG", ""));
jit.AddConstant(MakeJitConstant("OPTIONAL_SHAPE_INFO_TENSOR", ""));
}
}
#ifndef NDEBUG

2
src/plugins/intel_gpu/src/kernel_selector/kernel_base.h
View File

@ -70,7 +70,7 @@ protected:
bool IsFusedPrimitiveSupported(const fused_operation_desc& fused_op) const;
bool IsSIMDSizeSupported(const EngineInfo& info, size_t simd_size) const;
JitConstants MakeBaseParamsJitConstants(const base_params& params) const;
JitConstants MakeBaseParamsJitConstants(const base_params& params, bool add_tensor_definitions = true) const;
virtual std::vector<FusedOpType> GetSupportedFusedOps() const;
virtual JitConstants MakeFusedOpsJitConstants(const base_params &params, const std::vector<FusedOpsConfiguration> &conf) const;
virtual JitConstants MakeFusedOpsDeclsJitConstants(const base_params &params, const std::vector<FusedOpsConfiguration> &conf) const;

66
src/plugins/intel_gpu/src/kernel_selector/kernels/concatenation/concatenation_kernel_base.cpp
View File

@ -47,12 +47,38 @@ bool ConcatenationKernelBase::Validate(const Params& p, const optional_params&)
}
JitConstants ConcatenationKernelBase::GetJitConstants(const concatenation_params& params) const {
JitConstants jit = MakeBaseParamsJitConstants(params);
auto& inputs = params.original_input_layouts;
bool is_dynamic = std::any_of(inputs.begin(), inputs.end(), [](const DataTensor& t) { return t.is_dynamic(); }) ||
std::any_of(params.outputs.begin(), params.outputs.end(), [](const DataTensor& t) { return t.is_dynamic(); });
JitConstants jit = MakeBaseParamsJitConstants(params, !is_dynamic);
jit.AddConstants({
MakeJitConstant("CONCAT_" + toString(params.axis), 1),
});
if (is_dynamic) {
// shape info is supposed to contain shapes for
// in0, in1, ..., inN, out0
// So each dynamic kernel requires some custom offsets for shape_info access
size_t in_offset = 0;
for (size_t i = 0; i < params.kernel_split_id; i++) {
if (params.original_input_layouts[i].is_dynamic() || params.original_input_layouts[i].LogicalSize() == 0)
in_offset++;
}
size_t out_offset = 0;
for (size_t i = 0; i < params.original_input_layouts.size(); i++) {
if (params.original_input_layouts[i].is_dynamic() || params.original_input_layouts[i].LogicalSize() == 0)
out_offset++;
}
jit.AddConstant(MakeJitConstant("INPUT0", params.inputs[0], in_offset));
jit.AddConstant(MakeJitConstant("OUTPUT", params.outputs[0], out_offset));
jit.AddConstant(MakeJitConstant("IS_DYNAMIC", 1));
jit.AddConstant(MakeJitConstant("OPTIONAL_SHAPE_INFO_ARG", "__global const int* shape_info,"));
jit.AddConstant(MakeJitConstant("OPTIONAL_SHAPE_INFO_TENSOR", "shape_info,"));
}
jit.AddConstant(MakeJitConstant("CONCAT_AXIS_INDEX", GetConcatChannelIndex(params)));
return jit;
}
@ -91,6 +117,38 @@ KernelsData ConcatenationKernelBase::GetCommonKernelsData(const Params& params,
const concatenation_params& orgParams = static_cast<const concatenation_params&>(params);
KernelData kd = KernelData::Default<concatenation_params>(params, orgParams.inputs.size());
kd.update_dispatch_data_func = [this](const Params& params, KernelData& kd) {
const auto& prim_params = static_cast<const concatenation_params&>(params);
uint32_t lastOffset = 0;
for (size_t i = 0; i < prim_params.inputs.size(); i++) {
size_t ifm_offset = 0;
const auto& input = prim_params.inputs[i];
auto newParams = prim_params;
newParams.inputs.resize(1);
newParams.inputs[0] = input;
size_t ifm = input.Feature().v;
newParams.isAligned = ifm_offset % GetAlignment(newParams) == 0;
newParams.misalignment = ifm_offset % GetAlignment(newParams);
ifm_offset += ifm;
auto& kernel = kd.kernels[i];
DispatchData dispatchData = SetDefault(newParams);
kernel.params.workGroups.global = dispatchData.gws;
kernel.params.workGroups.local = dispatchData.lws;
ScalarDescriptor s;
s.t = ScalarDescriptor::Types::UINT32;
s.v.u32 = lastOffset;
kernel.params.scalars.resize(1);
kernel.params.scalars[0] = s;
auto concatChannelIndex = DataTensor::Channelndex(input.GetLayout(), GetConcatChannel(prim_params));
lastOffset += (uint32_t)input.GetDims()[concatChannelIndex].v;
}
};
bool is_dynamic = orgParams.has_dynamic_tensors();
uint32_t lastOffset = 0;
size_t ifm_offset = 0;
for (size_t i = 0; i < orgParams.inputs.size(); i++) {
@ -103,6 +161,9 @@ KernelsData ConcatenationKernelBase::GetCommonKernelsData(const Params& params,
newParams.misalignment = ifm_offset % GetAlignment(newParams);
ifm_offset += ifm;
newParams.kernel_split_id = i;
newParams.original_input_layouts = orgParams.inputs;
auto& kernel = kd.kernels[i];
DispatchData dispatchData = SetDefault(newParams);
auto cldnnJit = GetJitConstants(newParams);
@ -112,6 +173,9 @@ KernelsData ConcatenationKernelBase::GetCommonKernelsData(const Params& params,
kernel.code.kernelString = GetKernelString(kernelName, jit, entryPoint, params.engineInfo);
kernel.params.workGroups.global = dispatchData.gws;
kernel.params.workGroups.local = dispatchData.lws;
if (is_dynamic) {
kernel.params.arguments.push_back({ArgumentDescriptor::Types::SHAPE_INFO, 0});
}
kernel.params.arguments.push_back({ArgumentDescriptor::Types::INPUT, (uint32_t) i});
kernel.params.arguments.push_back({ArgumentDescriptor::Types::OUTPUT, 0});

Some files were not shown because too many files have changed in this diff Show More