Docs: complete migration guide (#10652)
* Updated glossary * Removed references to OpenVX * Moved migration_ov_2_0 to OpenVINO User guide * Replaced IE with OV runtime * Complete migration guide * Migration 2.0 * Self-review * Added property migration guide * Fixed table * Added preprocessing migration * Update docs/OV_Runtime_UG/migration_ov_2_0/preprocessing.md Co-authored-by: Mikhail Nosov <mikhail.nosov@intel.com> * Update docs/OV_Runtime_UG/migration_ov_2_0/preprocessing.md Co-authored-by: Mikhail Nosov <mikhail.nosov@intel.com> * Update docs/snippets/ov_preprocessing_migration.cpp Co-authored-by: Mikhail Nosov <mikhail.nosov@intel.com> * reivew fixes * Preprocessing intro updated * Updated config migration guide * Updates * Fixes Co-authored-by: Mikhail Nosov <mikhail.nosov@intel.com>
This commit is contained in:
Ilya Lavrenov
GitHub
parent
d1bcb6d0fc
commit
86b175534a
200
docs/OV_Runtime_UG/migration_ov_2_0/common_inference_pipeline.md
Normal file
200
docs/OV_Runtime_UG/migration_ov_2_0/common_inference_pipeline.md
Normal file
@ -0,0 +1,200 @@
|
||||
# Inference Pipeline {#openvino_2_0_inference_pipeline}
|
||||
|
||||
Usually to inference model with the OpenVINO™ Runtime an user needs to do the following steps in the application pipeline:
|
||||
- 1. Create Core object
|
||||
- 2. Read model from the disk
|
||||
- 2.1. (Optional) Model preprocessing
|
||||
- 3. Load the model to the device
|
||||
- 4. Create an inference request
|
||||
- 5. Fill input tensors with data
|
||||
- 6. Start inference
|
||||
- 7. Process the inference results
|
||||
|
||||
Code snippets below cover these steps and show how application code should be changed for migration to OpenVINO™ Runtime 2.0.
|
||||
|
||||
## 1. Create Core
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet docs/snippets/ie_common.cpp ie:create_core
|
||||
|
||||
OpenVINO™ Runtime API 2.0:
|
||||
|
||||
@snippet docs/snippets/ov_common.cpp ov_api_2_0:create_core
|
||||
|
||||
## 2. Read model from the disk
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet docs/snippets/ie_common.cpp ie:read_model
|
||||
|
||||
OpenVINO™ Runtime API 2.0:
|
||||
|
||||
@snippet docs/snippets/ov_common.cpp ov_api_2_0:read_model
|
||||
|
||||
Read model has the same structure as in the example from [Model Creation](./graph_construction.md) migration guide.
|
||||
|
||||
Note, you can combine read and compile model stages into a single call `ov::Core::compile_model(filename, devicename)`.
|
||||
|
||||
### 2.1 (Optional) Model preprocessing
|
||||
|
||||
When application's input data doesn't perfectly match with model's input format, preprocessing steps may need to be added.
|
||||
See detailed guide [how to migrate preprocessing in OpenVINO Runtime API 2.0](./preprocessing.md)
|
||||
|
||||
## 3. Load the Model to the Device
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet docs/snippets/ie_common.cpp ie:compile_model
|
||||
|
||||
OpenVINO™ Runtime API 2.0:
|
||||
|
||||
@snippet docs/snippets/ov_common.cpp ov_api_2_0:compile_model
|
||||
|
||||
If you need to configure OpenVINO Runtime devices with additional configuration parameters, please, refer to the migration [Configure devices](./configure_devices.md) guide.
|
||||
|
||||
## 4. Create an Inference Request
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet docs/snippets/ie_common.cpp ie:create_infer_request
|
||||
|
||||
OpenVINO™ Runtime API 2.0:
|
||||
|
||||
@snippet docs/snippets/ov_common.cpp ov_api_2_0:create_infer_request
|
||||
|
||||
## 5. Fill input tensors
|
||||
|
||||
Inference Engine API fills inputs as `I32` precision (**not** aligned with the original model):
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: IR v10
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ie_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ie:get_input_tensor]
|
||||
|
||||
.. tab:: IR v11
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ie_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ie:get_input_tensor]
|
||||
|
||||
.. tab:: ONNX
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ie_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ie:get_input_tensor]
|
||||
|
||||
.. tab:: Model created in code
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ie_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ie:get_input_tensor]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
OpenVINO™ Runtime API 2.0 fills inputs as `I64` precision (aligned with the original model)::
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: IR v10
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ov_api_2_0:get_input_tensor_v10]
|
||||
|
||||
.. tab:: IR v11
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ov_api_2_0:get_input_tensor_aligned]
|
||||
|
||||
.. tab:: ONNX
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ov_api_2_0:get_input_tensor_aligned]
|
||||
|
||||
.. tab:: Model created in code
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ov_api_2_0:get_input_tensor_aligned]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
## 6. Start Inference
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet docs/snippets/ie_common.cpp ie:inference
|
||||
|
||||
OpenVINO™ Runtime API 2.0:
|
||||
|
||||
@snippet docs/snippets/ov_common.cpp ov_api_2_0:inference
|
||||
|
||||
## 7. Process the Inference Results
|
||||
|
||||
Inference Engine API processes outputs as `I32` precision (**not** aligned with the original model):
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: IR v10
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ie_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ie:get_output_tensor]
|
||||
|
||||
.. tab:: IR v11
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ie_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ie:get_output_tensor]
|
||||
|
||||
.. tab:: ONNX
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ie_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ie:get_output_tensor]
|
||||
|
||||
.. tab:: Model created in code
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ie_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ie:get_output_tensor]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
OpenVINO™ Runtime API 2.0 processes outputs:
|
||||
- For IR v10 as `I32` precision (**not** aligned with the original model) to match **old** behavior
|
||||
- For IR v11, ONNX, ov::Model, Paddle as `I64` precision (aligned with the original model) to match **new** behavior
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: IR v10
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ov_api_2_0:get_output_tensor_v10]
|
||||
|
||||
.. tab:: IR v11
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ov_api_2_0:get_output_tensor_aligned]
|
||||
|
||||
.. tab:: ONNX
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ov_api_2_0:get_output_tensor_aligned]
|
||||
|
||||
.. tab:: Model created in code
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_common.cpp
|
||||
:language: cpp
|
||||
:fragment: [ov_api_2_0:get_output_tensor_aligned]
|
||||
|
||||
@endsphinxdirective
|
||||
129
docs/OV_Runtime_UG/migration_ov_2_0/configure_devices.md
Normal file
129
docs/OV_Runtime_UG/migration_ov_2_0/configure_devices.md
Normal file
@ -0,0 +1,129 @@
|
||||
# Configure devices {#openvino_2_0_configure_devices}
|
||||
|
||||
### Introduction
|
||||
|
||||
Inference Engine API provides an [ability to configure devices](https://docs.openvino.ai/2021.4/openvino_docs_IE_DG_InferenceEngine_QueryAPI.html) via configuration keys and [get device specific metrics](https://docs.openvino.ai/2021.4/openvino_docs_IE_DG_InferenceEngine_QueryAPI.html#getmetric). The values taken from `InferenceEngine::Core::GetConfig` are requested by its string name, while return type is `InferenceEngine::Parameter` and users don't know what is the actual type is stored in this parameter.
|
||||
|
||||
OpenVINO Runtime API 2.0 solves these issues by introducing [properties](../supported_plugins/config_properties.md), which unify metrics and configuration key concepts, but the main advantage of properties - they have C++ type:
|
||||
|
||||
```
|
||||
static constexpr Property<std::string> full_name{"FULL_DEVICE_NAME"};
|
||||
```
|
||||
|
||||
And the property can be requested from an inference device as:
|
||||
|
||||
@snippet ov_properties_migration.cpp core_get_ro_property
|
||||
|
||||
The snippets below show how to migrate from Inference Engine device configuration to OpenVINO Runtime API 2.0 steps.
|
||||
|
||||
### Set configuration values
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: Devices
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
|
||||
:language: cpp
|
||||
:fragment: [core_set_config]
|
||||
|
||||
.. tab:: Model Loading
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
|
||||
:language: cpp
|
||||
:fragment: [core_load_network]
|
||||
|
||||
.. tab:: Execution
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
|
||||
:language: cpp
|
||||
:fragment: [executable_network_set_config]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
OpenVINO Runtime API 2.0:
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: Devices
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
|
||||
:language: cpp
|
||||
:fragment: [core_set_property]
|
||||
|
||||
.. tab:: Model Loading
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
|
||||
:language: cpp
|
||||
:fragment: [core_compile_model]
|
||||
|
||||
.. tab:: Execution
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
|
||||
:language: cpp
|
||||
:fragment: [compiled_model_set_property]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
### Get information
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: Device configuration
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
|
||||
:language: cpp
|
||||
:fragment: [core_get_config]
|
||||
|
||||
.. tab:: Device metrics
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
|
||||
:language: cpp
|
||||
:fragment: [core_get_metric]
|
||||
|
||||
.. tab:: Execution config
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
|
||||
:language: cpp
|
||||
:fragment: [executable_network_get_metric]
|
||||
|
||||
.. tab:: Execution metrics
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
|
||||
:language: cpp
|
||||
:fragment: [executable_network_get_config]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
OpenVINO Runtime API 2.0:
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: Device configuration
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
|
||||
:language: cpp
|
||||
:fragment: [core_get_rw_property]
|
||||
|
||||
.. tab:: Device metrics
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
|
||||
:language: cpp
|
||||
:fragment: [core_get_ro_property]
|
||||
|
||||
.. tab:: Execution config
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
|
||||
:language: cpp
|
||||
:fragment: [compiled_model_get_rw_property]
|
||||
|
||||
.. tab:: Execution metrics
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_properties_migration.cpp
|
||||
:language: cpp
|
||||
:fragment: [compiled_model_get_ro_property]
|
||||
|
||||
@endsphinxdirective
|
||||
@ -1,161 +0,0 @@
|
||||
# OpenVINO™ Inference Pipeline {#openvino_inference_pipeline}
|
||||
|
||||
Usually to inference network with the OpenVINO™ toolkit users need to do next steps:
|
||||
1. Create Core
|
||||
2. (Optional) Read model from the disk
|
||||
2.1. Configure Input and Output of the Model
|
||||
3. Load the Model to the Device
|
||||
4. Create an Inference Request
|
||||
5. Prepare Input
|
||||
6. Start Inference
|
||||
7. Process the Inference Results
|
||||
|
||||
Code snippets below cover these steps and show how application code should be changed for migration to OpenVINO™ 2.0.
|
||||
|
||||
## 1. Create Core
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet snippets/ie_common.cpp ie:create_core
|
||||
|
||||
OpenVINO™ 2.0 API:
|
||||
|
||||
@snippet snippets/ov_common.cpp ov_api_2_0:create_core
|
||||
|
||||
## 2. (Optional) Read model from the disk
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet snippets/ie_common.cpp ie:read_model
|
||||
|
||||
OpenVINO™ 2.0 API:
|
||||
|
||||
@snippet snippets/ov_common.cpp ov_api_2_0:read_model
|
||||
|
||||
Read model has the same structure as in the example from [OpenVINO™ Graph Construction](@ref openvino_graph_construction) guide.
|
||||
|
||||
### 2.1 Configure Input and Output of the Model
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet snippets/ie_common.cpp ie:get_inputs_outputs
|
||||
|
||||
OpenVINO™ 2.0 API:
|
||||
|
||||
@snippet snippets/ov_common.cpp ov_api_2_0:get_inputs_outputs
|
||||
|
||||
## 3. Load the Model to the Device
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet snippets/ie_common.cpp ie:compile_model
|
||||
|
||||
OpenVINO™ 2.0 API:
|
||||
|
||||
@snippet snippets/ov_common.cpp ov_api_2_0:compile_model
|
||||
|
||||
## 4. Create an Inference Request
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet snippets/ie_common.cpp ie:create_infer_request
|
||||
|
||||
OpenVINO™ 2.0 API:
|
||||
|
||||
@snippet snippets/ov_common.cpp ov_api_2_0:create_infer_request
|
||||
|
||||
## 5. Prepare input
|
||||
|
||||
### IR v10
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet snippets/ie_common.cpp ie:get_input_tensor
|
||||
|
||||
OpenVINO™ 2.0 API:
|
||||
|
||||
@snippet snippets/ov_common.cpp ov_api_2_0:get_input_tensor_v10
|
||||
|
||||
### IR v11
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet snippets/ie_common.cpp ie:get_input_tensor
|
||||
|
||||
OpenVINO™ 2.0 API:
|
||||
|
||||
@snippet snippets/ov_common.cpp ov_api_2_0:get_input_tensor_aligned
|
||||
|
||||
### ONNX
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet snippets/ie_common.cpp ie:get_input_tensor
|
||||
|
||||
OpenVINO™ 2.0 API:
|
||||
|
||||
@snippet snippets/ov_common.cpp ov_api_2_0:get_input_tensor_aligned
|
||||
|
||||
### From Function
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet snippets/ie_common.cpp ie:get_input_tensor
|
||||
|
||||
OpenVINO™ 2.0 API:
|
||||
|
||||
@snippet snippets/ov_common.cpp ov_api_2_0:get_input_tensor_aligned
|
||||
|
||||
## 6. Start Inference
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet snippets/ie_common.cpp ie:inference
|
||||
|
||||
OpenVINO™ 2.0 API:
|
||||
|
||||
@snippet snippets/ov_common.cpp ov_api_2_0:inference
|
||||
|
||||
|
||||
## 7. Process the Inference Results
|
||||
|
||||
### IR v10
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet snippets/ie_common.cpp ie:get_output_tensor
|
||||
|
||||
OpenVINO™ 2.0 API:
|
||||
|
||||
@snippet snippets/ov_common.cpp ov_api_2_0:get_output_tensor_v10
|
||||
|
||||
### IR v11
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet snippets/ie_common.cpp ie:get_output_tensor
|
||||
|
||||
OpenVINO™ 2.0 API:
|
||||
|
||||
@snippet snippets/ov_common.cpp ov_api_2_0:get_output_tensor_aligned
|
||||
|
||||
### ONNX
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet snippets/ie_common.cpp ie:get_output_tensor
|
||||
|
||||
OpenVINO™ 2.0 API:
|
||||
|
||||
@snippet snippets/ov_common.cpp ov_api_2_0:get_output_tensor_aligned
|
||||
|
||||
### From Function
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet snippets/ie_common.cpp ie:get_output_tensor
|
||||
|
||||
OpenVINO™ 2.0 API:
|
||||
|
||||
@snippet snippets/ov_common.cpp ov_api_2_0:get_output_tensor_aligned
|
||||
|
||||
@ -1,12 +0,0 @@
|
||||
# OpenVINO™ graph construction {#openvino_graph_construction}
|
||||
|
||||
OpenVINO™ 2.0 includes nGraph engine in a common part. The `ngraph` namespace was changed to `ov`.
|
||||
Code snippets below show how application code should be changed for migration to OpenVINO™ 2.0.
|
||||
|
||||
nGraph API:
|
||||
|
||||
@snippet snippets/ngraph.cpp ngraph:graph
|
||||
|
||||
OpenVINO™ 2.0 API:
|
||||
|
||||
@snippet snippets/ov_graph.cpp ov:graph
|
||||
@ -1,24 +0,0 @@
|
||||
# OpenVINO™ API 2.0 Transition Guide {#openvino_2_0_transition_guide}
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:hidden:
|
||||
|
||||
openvino_inference_pipeline
|
||||
openvino_graph_construction
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
The OpenVINO™ API 2.0 introduced in order to simplify migration from other frameworks and make the OpenVINO™ API more user-friendly.
|
||||
The list with differences between APIs below:
|
||||
|
||||
- OpenVINO™ API 2.0 uses tensor names or indexes to work with Inputs or Outputs, the old API works with operation names.
|
||||
- Structures for Shapes, element types were changed.
|
||||
- Naming style was changed. The old API uses CamelCaseStyle and OpenVINO™ API 2.0 uses snake_case for function names.
|
||||
- Namespaces were aligned between components.
|
||||
|
||||
Please look at next transition guides to understand how transit own application to OpenVINO™ API 2.0.
|
||||
- [OpenVINO™ Graph Construction](graph_construction.md)
|
||||
- [OpenVINO™ Common Inference pipeline](common_inference_pipeline.md)
|
||||
16
docs/OV_Runtime_UG/migration_ov_2_0/graph_construction.md
Normal file
16
docs/OV_Runtime_UG/migration_ov_2_0/graph_construction.md
Normal file
@ -0,0 +1,16 @@
|
||||
# Model creation in runtime {#openvino_2_0_model_creation}
|
||||
|
||||
OpenVINO™ Runtime API 2.0 includes nGraph engine as a common part. The `ngraph` namespace was changed to `ov`, all other ngraph API is preserved as is.
|
||||
Code snippets below show how application code should be changed for migration to OpenVINO™ Runtime API 2.0.
|
||||
|
||||
### nGraph API
|
||||
|
||||
@snippet snippets/ngraph.cpp ngraph:graph
|
||||
|
||||
### OpenVINO™ Runtime API 2.0:
|
||||
|
||||
@snippet snippets/ov_graph.cpp ov:graph
|
||||
|
||||
**See also:**
|
||||
- [Hello Model Creation C++ Sample](../../../samples/cpp/model_creation_sample/README.md)
|
||||
- [Hello Model Creation Python Sample](../../../samples/python/model_creation_sample/README.md)
|
||||
81
docs/OV_Runtime_UG/migration_ov_2_0/intro.md
Normal file
81
docs/OV_Runtime_UG/migration_ov_2_0/intro.md
Normal file
@ -0,0 +1,81 @@
|
||||
# OpenVINO™ 2.0 Transition Guide {#openvino_2_0_transition_guide}
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:hidden:
|
||||
|
||||
openvino_2_0_inference_pipeline
|
||||
openvino_2_0_configure_devices
|
||||
openvino_2_0_preprocessing
|
||||
openvino_2_0_model_creation
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
### Introduction
|
||||
|
||||
Older versions of OpenVINO (prior to 2022.1) required to change the logic of applications when an user migrates from the frameworks like TensorFlow, ONNX Runtime, PyTorch, PaddlePaddle, etc. The change of application's logic is connected with:
|
||||
|
||||
- Model Optimizer changed input precisions for some inputs. For example, neural langauge processing models with `I64` input are becoming to have `I32` input element type.
|
||||
- Model Optimizer changed layouts for TensorFlow models ((see [Layouts in OpenVINO](../layout_overview.md))). It leads to unexpected user behavior that a user needs to use a different layout for its input data with compare to the framework:
|
||||
![tf_openvino]
|
||||
- Inference Engine API (`InferenceEngine::CNNNetwork`) also applied some conversion rules for input and output precisions because of device plugins limitations.
|
||||
- Users need to specify input shapes during model conversions in Model Optimizer and work with static shapes in the application.
|
||||
|
||||
OpenVINO Runtime API 2.0 is introduced to align logic of working with model as it is done in the frameworks - no layout and precision changes, operates with tensor names and indeces to address inputs and outputs. OpenVINO Runtime is composed of Inference Engine API used for inference and ngraph API targeted to work with models, operations. The OpenVINO API 2.0 has common structure, naming convention styles, namespaces, removes duplicated structures. See [How to migrate to OpenVINO 2.0 API](./common_inference_pipeline.md) for details.
|
||||
|
||||
> **NOTE**: Most important is that your existing application can continue working with OpenVINO Runtime 2.0 as it used to be, but we recommend migration to new API to unlock additional features like [Preprocessing](../preprocessing_overview.md) and [Dynamic shapes support](../DynamicBatching.md).
|
||||
|
||||
### Introduce IR v11
|
||||
|
||||
To support these features, OpenVINO introduced IR v11 which is generated by Model Optimizer by default since 2022.1. The model represented in IR v11 fully matches the original model in a original framework format in terms of inputs and outputs. Also, a user does not have to specify input shapes during the conversion, so the resulting IR v11 contains `-1` to denote undefined dimensions (see [Working with dynamic shapes](../DynamicBatching.md) to fully utilize this feature; or [Changning input shapes](../ShapeInference.md) to reshape to static shapes in the application).
|
||||
|
||||
What is also important to mention - the IR v11 is fully compatible with old applications written with Inference Engine API from older versions of OpenVINO. This is achieved by adding additional runtime information to the IR v11 which is responsible for backwark compatible behavior. So, once the IR v11 is read by the old Inference Engine based application, it's internally converted to IR v10 to provide backward-compatible behavior.
|
||||
|
||||
The IR v11 is supported by all OpenVINO Development tools including Post Training Optimization tool, Benchmark app, etc.
|
||||
|
||||
### IR v10 compatibility
|
||||
|
||||
OpenVINO Runtime API 2.0 also supports model in IR v10 for backward compatibility. So, if a user has an IR v10, such IR v10 can be fed to OpenVINO Runtime as well (see [migration steps](./common_inference_pipeline.md)).
|
||||
|
||||
Some OpenVINO Development Tools also support both IR v10 and IR v11 as an input:
|
||||
- Accuracy checker also supports IR v10, but requires an additional option to denote which API is used underneath.
|
||||
- [Compile tool](../../../tools/compile_tool/README.md) compiles the model to be used in OpenVINO 2.0 API by default. If a user wants to use the resulting compiled blob in Inference Engine API, the additional `ov_api_1_0` option should be passed.
|
||||
|
||||
But the following OpenVINO tools don't support IR v10 as an input, they require to regenerate an IR v11 from the original model with latest Model Optimizer:
|
||||
- Post Training Optimization tool
|
||||
- Deep Learning WorkBench
|
||||
|
||||
### Differences between Inference Engine and OpenVINO Runtime 2.0
|
||||
|
||||
Inference Engine and ngraph APIs are not deprecated, they are fully functional and can be used in applications. But OpenVINO recommends users to migrate to new OpenVINO Runtime API 2.0, because it already has additional features and this list will be extended later. The following list of additional features is supported by new API:
|
||||
- [Working with dynamic shapes](../DynamicBatching.md). The feature is quite usefull for best performance for NLP (Neural Language Processing) models, super resolution models and other which accepts dynamic input shapes.
|
||||
- [Preprocessing of the model](../preprocessing_overview.md) to add preprocessing operations to the inference models and fully ocupay the accelerator and free CPU resources.
|
||||
|
||||
To define a difference on the API level between Inference Engine and OpenVINO RUntime API 2.0, let's define two types of behaviors:
|
||||
- **Old behavior** of OpenVINO supposes:
|
||||
- Model Optimizer can change input element types, order of dimensions (layouts) with compare to the model from the original framework.
|
||||
- Inference Engine can override input and output element types.
|
||||
- Inference Engine API operates with operation names to address inputs and outputs (e.g. InferenceEngine::InferRequest::GetBlob).
|
||||
- Does not support compiling of models with dynamic input shapes.
|
||||
- **New behavior** assumes full model aligment with the framework and is implemented in OpenVINO 2.0:
|
||||
- Model Optimizer preserves the input element types, order of dimensions (layouts) and stores tensor names from the original models.
|
||||
- OpenVINO Runtime 2.0 reads models in any formats (IR v10, IR v11, ONNX, PaddlePaddle, etc) as is.
|
||||
- OpenVINO Runtime API 2.0 operates with tensor names. Note, the difference between tensor names and operations names is that in case if a single operation has several output tensors, such tensors cannot identified in a unique manner, so tensor names are used for addressing as it's usually done in the frameworks.
|
||||
- OpenVINO Runtime API 2.0 can address input and outputs tensors also by its index. Some model formats like ONNX are sensitive to order of inputs, outputs and its preserved by OpenVINO Runtime 2.0.
|
||||
|
||||
The table below demonstrates which behavior **old** or **new** is used depending on a model source, used APIs.
|
||||
|
||||
| API | IR v10 | IR v11 | ONNX file | Model created in code |
|
||||
|-------------------------------|---------|---------|-----------|-----------------------|
|
||||
|Inference Engine / ngraph APIs | Old | Old | Old | Old |
|
||||
|OpenVINO Runtime API 2.0 | Old | New | New | New |
|
||||
|
||||
Please look at next transition guides to understand how migrate Inference Engine-based application to OpenVINO™ Runtime API 2.0:
|
||||
- [OpenVINO™ Common Inference pipeline](common_inference_pipeline.md)
|
||||
- [Preprocess your model](./preprocessing.md)
|
||||
- [Configure device](./configure_devices.md)
|
||||
- [OpenVINO™ Model Creation](graph_construction.md)
|
||||
|
||||
[tf_openvino]: ../../img/tf_openvino.png
|
||||
64
docs/OV_Runtime_UG/migration_ov_2_0/preprocessing.md
Normal file
64
docs/OV_Runtime_UG/migration_ov_2_0/preprocessing.md
Normal file
@ -0,0 +1,64 @@
|
||||
# Preprocessing {#openvino_2_0_preprocessing}
|
||||
|
||||
### Introduction
|
||||
|
||||
Inference Engine API has preprocessing capabilities in `InferenceEngine::CNNNetwork` class. Such preprocessing information is not a part of the main inference graph executed by the [OpenVINO devices](../supported_plugins/Device_Plugins.md), so it is stored and executed separately before an inference stage:
|
||||
- Preprocessing operations are executed on CPU processor for most of the OpenVINO inference plugins. So, instead of occupying of acceleators, CPU processor is also busy with computational tasks.
|
||||
- Preprocessing information stored in `InferenceEngine::CNNNetwork` is lost during saving back to IR file format.
|
||||
|
||||
OpenVINO Runtime API 2.0 introduces [new way of adding preprocessing operations to the model](../preprocessing_overview.md) - each preprocessing or postprocessing operation is integrated directly to the model and compiled together with inference graph:
|
||||
- Add preprocessing operations first using `ov::preprocess::PrePostProcessor`
|
||||
- Compile model on the target then using `ov::Core::compile_model`
|
||||
|
||||
Having preprocessing operations as a part of OpenVINO opset allows to read and serialize preprocessed model as the IR file format.
|
||||
|
||||
It's also important to mention that since OpenVINO 2.0, the Runtime API does not assume any default layouts like Inference Engine did, for example both `{ 1, 224, 224, 3 }` and `{ 1, 3, 224, 224 }` shapes are supposed to have `NCHW` layout while only the last shape has `NCHW`. So, some preprocessing capabilities in OpenVINO Runtime API 2.0 requires explicitly set layouts, see [Layout overview](../layout_overview.md) how to do it. For example, to perform image scaling by partial dimensions `H` and `W`, preprocessing needs to know what dimensions are `H` and `W`.
|
||||
|
||||
> **NOTE**: Use Model Optimizer preprocessing capabilities to insert and optimize preprocessing operations to the model. In this case you don't need to read model in runtime application and set preprocessing, you can use [model caching feature](../Model_caching_overview.md) to improve time to inference stage.
|
||||
|
||||
The steps below demonstrates how to migrate preprocessing scenarios from Inference Engine API to OpenVINO Runtime API 2.0.
|
||||
The snippets suppose we need to preprocess a model input with tensor name `tensor_name`, in Inferenece Engine API using operation names to address the data, it's called `operation_name`.
|
||||
|
||||
### Mean and scale values
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet docs/snippets/ov_preprocessing_migration.cpp mean_scale
|
||||
|
||||
OpenVINO Runtime API 2.0:
|
||||
|
||||
@snippet docs/snippets/ov_preprocessing_migration.cpp ov_mean_scale
|
||||
|
||||
### Precision and layout conversions
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet docs/snippets/ov_preprocessing_migration.cpp conversions
|
||||
|
||||
OpenVINO Runtime API 2.0:
|
||||
|
||||
@snippet docs/snippets/ov_preprocessing_migration.cpp ov_conversions
|
||||
|
||||
### Image scaling
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet docs/snippets/ov_preprocessing_migration.cpp image_scale
|
||||
|
||||
OpenVINO Runtime API 2.0:
|
||||
|
||||
@snippet docs/snippets/ov_preprocessing_migration.cpp ov_image_scale
|
||||
|
||||
### Color space conversions
|
||||
|
||||
Inference Engine API:
|
||||
|
||||
@snippet docs/snippets/ov_preprocessing_migration.cpp color_space
|
||||
|
||||
OpenVINO Runtime API 2.0:
|
||||
|
||||
@snippet docs/snippets/ov_preprocessing_migration.cpp ov_color_space
|
||||
|
||||
**See also:**
|
||||
- [Preprocessing details](../preprocessing_details.md)
|
||||
- [NV12 classification sample](../../../samples/cpp/hello_nv12_input_classification/README.md)
|
||||
3
docs/img/tf_openvino.png
Normal file
3
docs/img/tf_openvino.png
Normal file
@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:c369ce9a1b7f24929aa2f7d954ff577e0f439ea049296dd13741838b91615f38
|
||||
size 47087
|
||||
@ -13,10 +13,8 @@ int main() {
|
||||
InferenceEngine::CNNNetwork network = core.ReadNetwork("model.xml");
|
||||
//! [ie:read_model]
|
||||
|
||||
//! [ie:get_inputs_outputs]
|
||||
InferenceEngine::InputsDataMap inputs = network.getInputsInfo();
|
||||
InferenceEngine::OutputsDataMap outputs = network.getOutputsInfo();
|
||||
//! [ie:get_inputs_outputs]
|
||||
|
||||
//! [ie:compile_model]
|
||||
InferenceEngine::ExecutableNetwork exec_network = core.LoadNetwork(network, "CPU");
|
||||
@ -29,7 +27,6 @@ int main() {
|
||||
//! [ie:get_input_tensor]
|
||||
InferenceEngine::Blob::Ptr input_blob1 = infer_request.GetBlob(inputs.begin()->first);
|
||||
// fill first blob
|
||||
InferenceEngine::SizeVector dims1 = input_blob1->getTensorDesc().getDims();
|
||||
InferenceEngine::MemoryBlob::Ptr minput1 = InferenceEngine::as<InferenceEngine::MemoryBlob>(input_blob1);
|
||||
if (minput1) {
|
||||
// locked memory holder should be alive all time while access to its
|
||||
@ -39,6 +36,7 @@ int main() {
|
||||
auto data = minputHolder.as<InferenceEngine::PrecisionTrait<InferenceEngine::Precision::I32>::value_type*>();
|
||||
// Fill data ...
|
||||
}
|
||||
|
||||
InferenceEngine::Blob::Ptr input_blob2 = infer_request.GetBlob("data2");
|
||||
// fill first blob
|
||||
InferenceEngine::MemoryBlob::Ptr minput2 = InferenceEngine::as<InferenceEngine::MemoryBlob>(input_blob2);
|
||||
|
||||
@ -62,16 +62,11 @@ int main() {
|
||||
//! [ov_api_2_0:create_core]
|
||||
|
||||
//! [ov_api_2_0:read_model]
|
||||
std::shared_ptr<ov::Model> network = core.read_model("model.xml");
|
||||
std::shared_ptr<ov::Model> model = core.read_model("model.xml");
|
||||
//! [ov_api_2_0:read_model]
|
||||
|
||||
//! [ov_api_2_0:get_inputs_outputs]
|
||||
std::vector<ov::Output<ov::Node>> inputs = network->inputs();
|
||||
std::vector<ov::Output<ov::Node>> outputs = network->outputs();
|
||||
//! [ov_api_2_0:get_inputs_outputs]
|
||||
|
||||
//! [ov_api_2_0:compile_model]
|
||||
ov::CompiledModel compiled_model = core.compile_model(network, "CPU");
|
||||
ov::CompiledModel compiled_model = core.compile_model(model, "CPU");
|
||||
//! [ov_api_2_0:compile_model]
|
||||
|
||||
//! [ov_api_2_0:create_infer_request]
|
||||
|
||||
124
docs/snippets/ov_preprocessing_migration.cpp
Normal file
124
docs/snippets/ov_preprocessing_migration.cpp
Normal file
@ -0,0 +1,124 @@
|
||||
// Copyright (C) 2018-2022 Intel Corporation
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
//
|
||||
#include <openvino/runtime/core.hpp>
|
||||
#include <openvino/opsets/opset8.hpp>
|
||||
#include <openvino/core/preprocess/pre_post_process.hpp>
|
||||
|
||||
#include "inference_engine.hpp"
|
||||
|
||||
int main_new() {
|
||||
std::string model_path;
|
||||
std::string tensor_name;
|
||||
|
||||
ov::Core core;
|
||||
std::shared_ptr<ov::Model> model = core.read_model(model_path);
|
||||
ov::preprocess::PrePostProcessor ppp(model);
|
||||
|
||||
{
|
||||
//! [ov_mean_scale]
|
||||
ov::preprocess::PrePostProcessor ppp(model);
|
||||
ov::preprocess::InputInfo& input = ppp.input(tensor_name);
|
||||
// we only need to know where is C dimension
|
||||
input.model().set_layout("...C");
|
||||
// specify scale and mean values, order of operations is important
|
||||
input.preprocess().mean(116.78f).scale({ 57.21f, 57.45f, 57.73f });
|
||||
// insert preprocessing operations to the 'model'
|
||||
model = ppp.build();
|
||||
//! [ov_mean_scale]
|
||||
}
|
||||
|
||||
{
|
||||
//! [ov_conversions]
|
||||
ov::preprocess::PrePostProcessor ppp(model);
|
||||
ov::preprocess::InputInfo& input = ppp.input(tensor_name);
|
||||
input.tensor().set_layout("NHWC").set_element_type(ov::element::u8);
|
||||
input.model().set_layout("NCHW");
|
||||
// layout and precision conversion is inserted automatically,
|
||||
// because tensor format != model input format
|
||||
model = ppp.build();
|
||||
//! [ov_conversions]
|
||||
}
|
||||
|
||||
{
|
||||
//! [ov_color_space]
|
||||
ov::preprocess::PrePostProcessor ppp(model);
|
||||
ov::preprocess::InputInfo& input = ppp.input(tensor_name);
|
||||
input.tensor().set_color_format(ov::preprocess::ColorFormat::NV12_TWO_PLANES);
|
||||
// add NV12 to BGR conversion
|
||||
input.preprocess().convert_color(ov::preprocess::ColorFormat::BGR);
|
||||
// and insert operations to the model
|
||||
model = ppp.build();
|
||||
//! [ov_color_space]
|
||||
}
|
||||
|
||||
{
|
||||
//! [ov_image_scale]
|
||||
ov::preprocess::PrePostProcessor ppp(model);
|
||||
ov::preprocess::InputInfo& input = ppp.input(tensor_name);
|
||||
// scale from the specified tensor size
|
||||
input.tensor().set_spatial_static_shape(448, 448);
|
||||
// need to specify H and W dimensions in model, others are not important
|
||||
input.model().set_layout("??HW");
|
||||
// scale to model shape
|
||||
input.preprocess().resize(ov::preprocess::ResizeAlgorithm::RESIZE_LINEAR);
|
||||
// and insert operations to the model
|
||||
model = ppp.build();
|
||||
//! [ov_image_scale]
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
int main_old() {
|
||||
std::string model_path;
|
||||
std::string operation_name;
|
||||
|
||||
InferenceEngine::Core core;
|
||||
InferenceEngine::CNNNetwork network = core.ReadNetwork(model_path);
|
||||
|
||||
{
|
||||
//! [mean_scale]
|
||||
auto preProcess = network.getInputsInfo()[operation_name]->getPreProcess();
|
||||
preProcess.init(3);
|
||||
preProcess[0]->meanValue = 116.78f;
|
||||
preProcess[1]->meanValue = 116.78f;
|
||||
preProcess[2]->meanValue = 116.78f;
|
||||
preProcess[0]->stdScale = 57.21f;
|
||||
preProcess[1]->stdScale = 57.45f;
|
||||
preProcess[2]->stdScale = 57.73f;
|
||||
preProcess.setVariant(InferenceEngine::MEAN_VALUE);
|
||||
//! [mean_scale]
|
||||
}
|
||||
|
||||
{
|
||||
//! [conversions]
|
||||
auto inputInfo = network.getInputsInfo()[operation_name];
|
||||
inputInfo->setPrecision(InferenceEngine::Precision::U8);
|
||||
inputInfo->setLayout(InferenceEngine::Layout::NHWC);
|
||||
// model input layout is always NCHW in Inference Engine
|
||||
// for shapes with 4 dimensions
|
||||
//! [conversions]
|
||||
}
|
||||
|
||||
{
|
||||
//! [color_space]
|
||||
auto preProcess = network.getInputsInfo()[operation_name]->getPreProcess();
|
||||
// Inference Engine supposes NV12 as two inputs which need to be passed
|
||||
// as InferenceEngine::NV12Blob composed of two Y and UV planes
|
||||
preProcess.setColorFormat(InferenceEngine::NV12);
|
||||
//! [color_space]
|
||||
}
|
||||
|
||||
{
|
||||
//! [image_scale]
|
||||
auto preProcess = network.getInputsInfo()[operation_name]->getPreProcess();
|
||||
// Inference Engine supposes input for resize is always in NCHW layout
|
||||
// while for OpenVINO Runtime API 2.0 `H` and `W` dimensions must be specified
|
||||
// Also, current code snippet supposed resize from dynamic shapes
|
||||
preProcess.setResizeAlgorithm(InferenceEngine::ResizeAlgorithm::RESIZE_BILINEAR);
|
||||
//! [image_scale]
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
95
docs/snippets/ov_properties_migration.cpp
Normal file
95
docs/snippets/ov_properties_migration.cpp
Normal file
@ -0,0 +1,95 @@
|
||||
#include <openvino/runtime/core.hpp>
|
||||
#include <inference_engine.hpp>
|
||||
|
||||
int main_new() {
|
||||
ov::Core core;
|
||||
|
||||
//! [core_get_ro_property]
|
||||
// 'auto' is automatically deduced as std::string
|
||||
// since the type is stored in the property
|
||||
auto full_device_name = core.get_property("CPU", ov::device::full_name);
|
||||
//! [core_get_ro_property]
|
||||
|
||||
//! [core_get_rw_property]
|
||||
// 'auto' is automatically deduced as ov::streams::Num
|
||||
// since the type is stored in the property
|
||||
auto num_streams = core.get_property("CPU", ov::streams::num);
|
||||
//! [core_get_rw_property]
|
||||
|
||||
//! [core_set_property]
|
||||
core.set_property("CPU", ov::enable_profiling(true));
|
||||
//! [core_set_property]
|
||||
|
||||
auto model = core.read_model("sample.xml");
|
||||
//! [core_compile_model]
|
||||
auto compiled_model = core.compile_model(model, "MULTI",
|
||||
ov::device::priorities("GPU", "CPU"),
|
||||
ov::hint::performance_mode(ov::hint::PerformanceMode::THROUGHPUT),
|
||||
ov::hint::inference_precision(ov::element::f32));
|
||||
//! [core_compile_model]
|
||||
|
||||
//! [compiled_model_set_property]
|
||||
// turn CPU off for multi-device execution
|
||||
compiled_model.set_property(ov::device::priorities("GPU"));
|
||||
//! [compiled_model_set_property]
|
||||
|
||||
{
|
||||
//! [compiled_model_get_ro_property]
|
||||
// 'auto' is deduced to 'uint32_t'
|
||||
auto nireq = compiled_model.get_property(ov::optimal_number_of_infer_requests);
|
||||
//! [compiled_model_get_ro_property]
|
||||
}
|
||||
|
||||
{
|
||||
//! [compiled_model_get_rw_property]
|
||||
ov::hint::PerformanceMode perf_model = compiled_model.get_property(ov::hint::performance_mode);
|
||||
//! [compiled_model_get_rw_property]
|
||||
}
|
||||
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
||||
int main_old() {
|
||||
InferenceEngine::Core core;
|
||||
//! [core_get_metric]
|
||||
auto full_device_name = core.GetConfig("CPU", METRIC_KEY(FULL_DEVICE_NAME)).as<std::string>();
|
||||
//! [core_get_metric]
|
||||
|
||||
//! [core_get_config]
|
||||
// a user has to parse std::string after
|
||||
auto num_streams = core.GetMetric("CPU", CONFIG_KEY(CPU_THROUGHPUT_STREAMS)).as<std::string>();
|
||||
//! [core_get_config]
|
||||
|
||||
//! [core_set_config]
|
||||
core.SetConfig({ { CONFIG_KEY(PERF_COUNT), CONFIG_VALUE(YES) } }, "CPU");
|
||||
//! [core_set_config]
|
||||
|
||||
auto model = core.ReadNetwork("sample.xml");
|
||||
//! [core_load_network]
|
||||
auto exec_network = core.LoadNetwork(model, "MULTI", {
|
||||
{ MULTI_CONFIG_KEY(DEVICE_PRIORITIES), "CPU, GPU" },
|
||||
{ CONFIG_KEY(PERFORMANCE_HINT), CONFIG_VALUE(THROUGHPUT) },
|
||||
{ CONFIG_KEY(ENFORCE_BF16), CONFIG_VALUE(NO) } });
|
||||
//! [core_load_network]
|
||||
|
||||
//! [executable_network_set_config]
|
||||
// turn CPU off for multi-device execution
|
||||
exec_network.SetConfig({ { MULTI_CONFIG_KEY(DEVICE_PRIORITIES), "GPU" } });
|
||||
//! [executable_network_set_config]
|
||||
|
||||
{
|
||||
//! [executable_network_get_metric]
|
||||
auto nireq = exec_network.GetMetric(EXEC_NETWORK_METRIC_KEY(OPTIMAL_NUMBER_OF_INFER_REQUESTS)).as<uint32_t>();
|
||||
//! [executable_network_get_metric]
|
||||
}
|
||||
|
||||
{
|
||||
//! [executable_network_get_config]
|
||||
std::string perf_model = exec_network.GetConfig(CONFIG_KEY(PERFORMANCE_HINT)).as<std::string>();
|
||||
//! [executable_network_get_config]
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
Loading…
Reference in New Issue
Block a user