Yuan transition guide restructure (#11778)
* Add Overview page * Revert "Add Overview page" * restructure * update * updates * update * update * Update docs/OV_Runtime_UG/migration_ov_2_0/intro.md Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com> * Update docs/OV_Runtime_UG/migration_ov_2_0/preprocessing.md Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com> * fix formatting * fix formatting Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>
This commit is contained in:
Yuan XuandKarol Blaszczak authored and 
GitHub committed
1 parent
fb09555b6d
commit
fc61b001c0
6 files changed
+134
-117
No files matched your search
@@ -1,21 +1,21 @@
|
||||
# Inference Pipeline {#openvino_2_0_inference_pipeline}
|
||||
|
||||
To infer models with OpenVINO™ Runtime, you usually need to perform the following steps in the application pipeline:
|
||||
- 1. Create a Core object.
|
||||
- 1.1. (Optional) Load extensions.
|
||||
- 2. Read a model from a drive.
|
||||
- 2.1. (Optional) Perform model preprocessing.
|
||||
- 3. Load the model to the device.
|
||||
- 4. Create an inference request.
|
||||
- 5. Fill input tensors with data.
|
||||
- 6. Start inference.
|
||||
- 7. Process the inference results.
|
||||
1. <a href="#create-core">Create a Core object.</a>
|
||||
- 1.1. <a href="#load-extensions">(Optional) Load extensions.</a>
|
||||
2. <a href="#read-model">Read a model from a drive.</a>
|
||||
- 2.1. <a href="#perform-preprocessing">(Optional) Perform model preprocessing.</a>
|
||||
3. <a href="#load-model-to-device">Load the model to the device.</a>
|
||||
4. <a href="#create-inference-request">Create an inference request.</a>
|
||||
5. <a href="#fill-tensor">Fill input tensors with data.</a>
|
||||
6. <a href="#start-inference">Start inference.</a>
|
||||
7. <a href="#process-results">Process the inference results.</a>
|
||||
|
||||
The following code explains how to change the application code for migration to OpenVINO™ Runtime 2.0.
|
||||
Based on the steps, the following code demostrates how to change the application code to migrate to API 2.0.
|
||||
|
||||
## 1. Create Core
|
||||
## <a name="create-core"></a>1. Create a Core Object
|
||||
|
||||
Inference Engine API:
|
||||
**Inference Engine API**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -29,7 +29,7 @@ Inference Engine API:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
OpenVINO™ Runtime API 2.0:
|
||||
**API 2.0**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -43,11 +43,11 @@ OpenVINO™ Runtime API 2.0:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
### 1.1 (Optional) Load Extensions
|
||||
### <a name="load-extensions"></a>1.1 (Optional) Load Extensions
|
||||
|
||||
To load a model with custom operations, you need to add extensions for these operations. It is highly recommended to use [OpenVINO Extensibility API](../../Extensibility_UG/Intro.md) to write extensions. However, you can also load the old extensions to the new OpenVINO™ Runtime:
|
||||
|
||||
Inference Engine API:
|
||||
**Inference Engine API**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -61,7 +61,7 @@ Inference Engine API:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
OpenVINO™ Runtime API 2.0:
|
||||
**API 2.0**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -75,9 +75,9 @@ OpenVINO™ Runtime API 2.0:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
## 2. Read a Model from a Drive
|
||||
## <a name="read-model"></a>2. Read a Model from a Drive
|
||||
|
||||
Inference Engine API:
|
||||
**Inference Engine API**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -91,7 +91,7 @@ Inference Engine API:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
OpenVINO™ Runtime API 2.0:
|
||||
**API 2.0**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -105,18 +105,17 @@ OpenVINO™ Runtime API 2.0:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
Read model has the same structure as the example from [Model Creation](./graph_construction.md) migration guide.
|
||||
Reading a model has the same structure as the example in the [model creation migration guide](./graph_construction.md).
|
||||
|
||||
You can combine read and compile model stages into a single call `ov::Core::compile_model(filename, devicename)`.
|
||||
You can combine reading and compiling a model into a single call `ov::Core::compile_model(filename, devicename)`.
|
||||
|
||||
### 2.1 (Optional) Perform Model Preprocessing
|
||||
### <a name="perform-preprocessing"></a>2.1 (Optional) Perform Model Preprocessing
|
||||
|
||||
When application input data does not perfectly match the model input format, preprocessing may be necessary.
|
||||
See the detailed guide on [how to migrate preprocessing in OpenVINO Runtime API 2.0](./preprocessing.md)
|
||||
When the application input data does not perfectly match the model input format, preprocessing may be necessary. See [preprocessing in API 2.0](./preprocessing.md) for more details.
|
||||
|
||||
## 3. Load the Model to the Device
|
||||
## <a name="load-model-to-device"></a>3. Load the Model to the Device
|
||||
|
||||
Inference Engine API:
|
||||
**Inference Engine API**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -130,7 +129,7 @@ Inference Engine API:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
OpenVINO™ Runtime API 2.0:
|
||||
**API 2.0**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -144,11 +143,11 @@ OpenVINO™ Runtime API 2.0:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
If you need to configure OpenVINO Runtime devices with additional parameters, refer to the [Configure devices](./configure_devices.md) guide.
|
||||
If you need to configure devices with additional parameters for OpenVINO Runtime, refer to [Configuring Devices](./configure_devices.md).
|
||||
|
||||
## 4. Create an Inference Request
|
||||
## <a name="create-inference-request"></a>4. Create an Inference Request
|
||||
|
||||
Inference Engine API:
|
||||
**Inference Engine API**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -162,7 +161,7 @@ Inference Engine API:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
OpenVINO™ Runtime API 2.0:
|
||||
**API 2.0**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -176,9 +175,11 @@ OpenVINO™ Runtime API 2.0:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
## 5. Fill Input Tensors
|
||||
## <a name="fill-tensor"></a>5. Fill Input Tensors with Data
|
||||
|
||||
The Inference Engine API fills inputs as `I32` precision (**not** aligned with the original model):
|
||||
**Inference Engine API**
|
||||
|
||||
The Inference Engine API fills inputs with data of the `I32` precision (**not** aligned with the original model):
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -248,7 +249,9 @@ The Inference Engine API fills inputs as `I32` precision (**not** aligned with t
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
OpenVINO™ Runtime API 2.0 fills inputs as `I64` precision (aligned with the original model):
|
||||
**API 2.0**
|
||||
|
||||
API 2.0 fills inputs with data of the `I64` precision (aligned with the original model):
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -318,9 +321,9 @@ OpenVINO™ Runtime API 2.0 fills inputs as `I64` precision (aligned with the or
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
## 6. Start Inference
|
||||
## <a name="start-inference"></a>6. Start Inference
|
||||
|
||||
Inference Engine API:
|
||||
**Inference Engine API**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -358,7 +361,7 @@ Inference Engine API:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
OpenVINO™ Runtime API 2.0:
|
||||
**API 2.0**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -396,9 +399,11 @@ OpenVINO™ Runtime API 2.0:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
## 7. Process the Inference Results
|
||||
## <a name="process-results"></a>7. Process the Inference Results
|
||||
|
||||
The Inference Engine API processes outputs as `I32` precision (**not** aligned with the original model):
|
||||
**Inference Engine API**
|
||||
|
||||
The Inference Engine API processes outputs as they are of the `I32` precision (**not** aligned with the original model):
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -468,9 +473,12 @@ The Inference Engine API processes outputs as `I32` precision (**not** aligned w
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
OpenVINO™ Runtime API 2.0 processes outputs:
|
||||
- For IR v10 as `I32` precision (**not** aligned with the original model) to match the **old** behavior.
|
||||
- For IR v11, ONNX, ov::Model, Paddle as `I64` precision (aligned with the original model) to match the **new** behavior.
|
||||
**API 2.0**
|
||||
|
||||
API 2.0 processes outputs:
|
||||
|
||||
- as they are of the `I32` precision (**not** aligned with the original model) for OpenVINO IR v10 models, to match the <a href="openvino_2_0_transition_guide#differences-api20-ie">old behavior</a>.
|
||||
- as they are of the `I64` precision (aligned with the original model) for OpenVINO IR v11, ONNX, ov::Model and PaddlePaddle models, to match the <a href="openvino_2_0_transition_guide#differences-api20-ie">new behavior</a>.
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
|
||||
@@ -1,24 +1,22 @@
|
||||
# Configuring Devices {#openvino_2_0_configure_devices}
|
||||
|
||||
### Introduction
|
||||
|
||||
Inference Engine API provides the [ability to configure devices](https://docs.openvino.ai/2021.4/openvino_docs_IE_DG_InferenceEngine_QueryAPI.html) via configuration keys and [get device specific metrics](https://docs.openvino.ai/2021.4/openvino_docs_IE_DG_InferenceEngine_QueryAPI.html#getmetric). The values taken from `InferenceEngine::Core::GetConfig` are requested by the string name, while the return type is `InferenceEngine::Parameter`, making users lost on what the actual type is stored in this parameter.
|
||||
|
||||
OpenVINO Runtime API 2.0 solves these issues by introducing [properties](../supported_plugins/config_properties.md), which unify metrics and configuration key concepts. Their main advantage is that they have the C++ type:
|
||||
API 2.0 solves these issues by introducing [properties](../supported_plugins/config_properties.md), which unify metrics and configuration key concepts. The main advantage is that they have the C++ type:
|
||||
|
||||
```
|
||||
static constexpr Property<std::string> full_name{"FULL_DEVICE_NAME"};
|
||||
```
|
||||
|
||||
The property can be requested from an inference device as:
|
||||
where the property can be requested from an inference device as:
|
||||
|
||||
@snippet ov_properties_migration.cpp core_get_ro_property
|
||||
|
||||
The snippets below explain how to migrate from an Inference Engine device configuration to OpenVINO Runtime API 2.0.
|
||||
The snippets in the following sections demostrate the device configurations for migrating from Inference Engine to API 2.0.
|
||||
|
||||
### Set configuration values
|
||||
## Setting Configuration Values
|
||||
|
||||
Inference Engine API:
|
||||
**Inference Engine API**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -76,7 +74,7 @@ Inference Engine API:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
OpenVINO Runtime API 2.0:
|
||||
**API 2.0**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -134,9 +132,9 @@ OpenVINO Runtime API 2.0:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
### Get Information
|
||||
## Getting Information
|
||||
|
||||
Inference Engine API:
|
||||
**Inference Engine API**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -206,7 +204,7 @@ Inference Engine API:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
OpenVINO Runtime API 2.0:
|
||||
**API 2.0**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Installation & Deployment {#openvino_2_0_deployment}
|
||||
|
||||
One of the main concepts for OpenVINO™ API 2.0 is being *"easy to use"*, which includes:
|
||||
One of the main concepts for OpenVINO™ API 2.0 is being "easy to use", which includes:
|
||||
* Simplification of migration from different frameworks to OpenVINO.
|
||||
* Organization of OpenVINO.
|
||||
* Usage of development tools.
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
# Model Creation in Runtime {#openvino_2_0_model_creation}
|
||||
# Model Creation in OpenVINO™ Runtime {#openvino_2_0_model_creation}
|
||||
|
||||
OpenVINO™ Runtime API 2.0 includes the nGraph engine as a common part. The `ngraph` namespace has been changed to `ov`, but all other parts of the ngraph API have been preserved.
|
||||
The code snippets below show how to change application code for migration to OpenVINO™ Runtime API 2.0.
|
||||
OpenVINO™ Runtime with API 2.0 includes the nGraph engine as a common part. The `ngraph` namespace has been changed to `ov`, but all other parts of the ngraph API have been preserved.
|
||||
|
||||
### nGraph API
|
||||
The code snippets below show how to change the application code for migration to API 2.0.
|
||||
|
||||
## nGraph API
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -17,7 +18,7 @@ The code snippets below show how to change application code for migration to Ope
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
### OpenVINO™ Runtime API 2.0:
|
||||
## API 2.0
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -31,6 +32,7 @@ The code snippets below show how to change application code for migration to Ope
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
**See also:**
|
||||
## Additional Resources
|
||||
|
||||
- [Hello Model Creation C++ Sample](../../../samples/cpp/model_creation_sample/README.md)
|
||||
- [Hello Model Creation Python Sample](../../../samples/python/model_creation_sample/README.md)
|
||||
@@ -1,4 +1,4 @@
|
||||
# Transition to OpenVINO™ API 2.0 {#openvino_2_0_transition_guide}
|
||||
# OpenVINO™ API 2.0 Transition Guide {#openvino_2_0_transition_guide}
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
@@ -14,9 +14,11 @@
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
### Introduction
|
||||
This guide introduces the new OpenVINO™ API: API 2.0, as well as the new OpenVINO IR model format: IR v11. Here, you will find comparisons of their "old" and "new" versions.
|
||||
|
||||
Versions of OpenVINO™ prior to 2022.1 required changes in the application logic when migrating an app from other frameworks, like TensorFlow, ONNX Runtime, PyTorch, PaddlePaddle, etc. The changes were required because:
|
||||
### Introduction of API 2.0
|
||||
|
||||
Versions of OpenVINO prior to 2022.1 required changes in the application logic when migrating an app from other frameworks, such as TensorFlow, ONNX Runtime, PyTorch, PaddlePaddle, etc. The changes were required because:
|
||||
|
||||
- Model Optimizer changed input precisions for some inputs. For example, neural language processing models with `I64` inputs were changed to include `I32` ones.
|
||||
- Model Optimizer changed layouts for TensorFlow models (see the [Layouts in OpenVINO](../layout_overview.md)). It lead to unusual requirement of using the input data with a different layout than that of the framework:
|
||||
@@ -24,56 +26,58 @@ Versions of OpenVINO™ prior to 2022.1 required changes in the application logi
|
||||
- Inference Engine API (`InferenceEngine::CNNNetwork`) applied some conversion rules for input and output precisions due to limitations in device plugins.
|
||||
- Users needed to specify input shapes during model conversions in Model Optimizer, and work with static shapes in the application.
|
||||
|
||||
The new OpenVINO™ introduces API 2.0 (also called OpenVINO API v2) to align the logic of working with models as it is done in their origin frameworks - no layout and precision changes, operating with tensor names and indices to address inputs and outputs. OpenVINO Runtime has combined Inference Engine API used for inference and nGraph API targeted to work with models and operations. API 2.0 has a common structure, naming convention styles, namespaces, and removes duplicated structures. For more details, see the [Changes to Inference Pipeline in OpenVINO API v2](common_inference_pipeline.md).
|
||||
OpenVINO™ 2022.1 has introduced API 2.0 (also called OpenVINO API v2) to align the logic of working with models as it is done in their origin frameworks - no layout and precision changes, operating with tensor names and indices to address inputs and outputs. OpenVINO Runtime has combined Inference Engine API used for inference and nGraph API targeted to work with models and operations. API 2.0 has a common structure, naming convention styles, namespaces, and removes duplicated structures. For more details, see the [Changes to Inference Pipeline in OpenVINO API v2](common_inference_pipeline.md).
|
||||
|
||||
> **NOTE**: Your existing applications will continue to work with OpenVINO Runtime 2022.1, as normal. Although, migration to API 2.0 is strongly recommended. This will allow you to use additional features, like [Preprocessing](../preprocessing_overview.md) and [Dynamic shapes support](../ov_dynamic_shapes.md).
|
||||
> **NOTE**: Your existing applications will continue to work with OpenVINO Runtime 2022.1, as normal. Although, migration to API 2.0 is strongly recommended. This will allow you to use additional features, such as [Preprocessing](../preprocessing_overview.md) and [Dynamic shapes support](../ov_dynamic_shapes.md).
|
||||
|
||||
### Introducing IR v11
|
||||
### The New OpenVINO IR v11
|
||||
|
||||
To support these features, OpenVINO has introduced IR v11, which is now the default version for Model Optimizer. The model represented in IR v11 fully matches the original model in the original framework format in terms of inputs, and outputs. It is also not required to specify input shapes during conversion, which results in IR v11 containing `-1` to denote undefined dimensions. For more details on how to fully utilize this feature, see the [Working with dynamic shapes](../ov_dynamic_shapes.md) guide. For information on how to reshape to static shapes in the application, see the [Changing input shapes](../ShapeInference.md).
|
||||
To support these features, OpenVINO has introduced OpenVINO IR v11, which is now the default version for Model Optimizer. The model represented in OpenVINO IR v11 fully matches the original model in the original framework format in terms of inputs and outputs. It is also not required to specify input shapes during conversion, which results in OpenVINO IR v11 containing `-1` to denote undefined dimensions. For more details on how to fully utilize this feature, see [Working with dynamic shapes](../ov_dynamic_shapes.md). For information on how to reshape to static shapes in application, see [Changing input shapes](../ShapeInference.md).
|
||||
|
||||
IR v11 is fully compatible with old applications written with the Inference Engine API used by older versions of OpenVINO. This backward compatibility is allowed thanks to additional runtime information included in IR v11. This means that if the IR v11 is read by an application based on Inference Engine, it is internally converted to IR v10.
|
||||
OpenVINO IR v11 is fully compatible with applications written with the Inference Engine API used by older versions of OpenVINO. This backward compatibility is allowed thanks to additional runtime information included in OpenVINO IR v11. This means that when OpenVINO IR v11 is read by an application based on Inference Engine, it is internally converted to OpenVINO IR v10.
|
||||
|
||||
IR v11 is supported by all OpenVINO Development tools including Post-Training Optimization tool, Benchmark app, etc.
|
||||
OpenVINO IR v11 is supported by all OpenVINO Development tools including Post-Training Optimization Tool, Benchmark app, etc.
|
||||
|
||||
### IR v10 Compatibility
|
||||
### Backward Compatibility for OpenVINO IR v10
|
||||
|
||||
API 2.0 also supports backward compatibility for models in IR v10. If you have IR v10 files, they can also be fed to OpenVINO Runtime (For more details, see the [migration steps](common_inference_pipeline.md)).
|
||||
API 2.0 also supports backward compatibility for models of OpenVINO IR v10. If you have OpenVINO IR v10 files, they can also be fed to OpenVINO Runtime. For more details, see the [migration steps](common_inference_pipeline.md).
|
||||
|
||||
Some OpenVINO Development Tools also support both IR v10 and IR v11 as an input:
|
||||
- Accuracy checker uses API 2.0 for model accuracy measurement by default. It also supports switching to the old API by using the `--use_new_api False` command line parameter. Both launchers accept IR v10 and v11, but in some cases configuration files should be updated. For more details, see the [Accuracy Checker documentation](https://github.com/openvinotoolkit/open_model_zoo/blob/master/tools/accuracy_checker/openvino/tools/accuracy_checker/launcher/openvino_launcher_readme.md).
|
||||
Some of the OpenVINO Development Tools also support both OpenVINO IR v10 and v11 as an input:
|
||||
- Accuracy checker uses API 2.0 for model accuracy measurement by default. It also supports switching to the old API by using the `--use_new_api False` command-line parameter. Both launchers accept OpenVINO IR v10 and v11, but in some cases configuration files should be updated. For more details, see the [Accuracy Checker documentation](https://github.com/openvinotoolkit/open_model_zoo/blob/master/tools/accuracy_checker/openvino/tools/accuracy_checker/launcher/openvino_launcher_readme.md).
|
||||
- [Compile tool](../../../tools/compile_tool/README.md) compiles the model to be used in API 2.0 by default. To use the resulting compiled blob under the Inference Engine API, the additional `ov_api_1_0` option should be passed.
|
||||
|
||||
Not all tools under the OpenVINO umbrella continue to support IR v10 and will require the latest version of Model Optimizer to generate IR v11 files. They are: Post-Training Optimization Tool and Deep Learning Workbench.
|
||||
However, Post-Training Optimization Tool and Deep Learning Workbench of OpenVINO 2022.1 do not support OpenVINO IR v10. They require the latest version of Model Optimizer to generate OpenVINO IR v11 files.
|
||||
|
||||
> **NOTE**: If you need to quantize your IR v10 models to run with OpenVINO 2022.1, it is recommended to download, and use Post-Training Optimization tool from the OpenVINO 2021.4 release.
|
||||
> **NOTE**: To quantize your OpenVINO IR v10 models to run with OpenVINO 2022.1, download and use Post-Training Optimization Tool of OpenVINO 2021.4.
|
||||
|
||||
### Differences between Inference Engine and OpenVINO Runtime 2022.1
|
||||
### <a name="differences-api20-ie"></a>Differences in API 2.0 and Inference Engine API Behaviors
|
||||
|
||||
Inference Engine and nGraph APIs have not become deprecated with the introduction of the new API, and they can still be used in applications. However, it is highly recommended to migrate to API 2.0, as it offers more features (further extended in future releases), such as:
|
||||
- [Working with dynamic shapes](../ov_dynamic_shapes.md) -- increases performance when working with compatible models, such as NLP (Neural Language Processing) and super-resolution models.
|
||||
- [Preprocessing of the model](../preprocessing_overview.md) -- adds preprocessing operations to inference models, and fully occupies the accelerator, freeing CPU resources.
|
||||
Inference Engine and nGraph APIs do not become deprecated with the introduction of the new API, and they can still be used in applications. However, it is highly recommended to migrate to API 2.0, as it offers more features (further extended in future releases), such as:
|
||||
- [Working with dynamic shapes](../ov_dynamic_shapes.md), which increases performance when working with compatible models such as NLP (Neural Language Processing) and super-resolution models.
|
||||
- [Preprocessing of the model](../preprocessing_overview.md), which adds preprocessing operations to inference models and fully occupies the accelerator, freeing CPU resources.
|
||||
|
||||
The API differences between Inference Engine, and API 2.0, can be defined by two types of behaviors:
|
||||
To understand the differences between Inference Engine API and API 2.0, see the definitions of two types of behaviors first:
|
||||
- **Old behavior** of OpenVINO assumes that:
|
||||
- Model Optimizer can change input element types, order of dimensions (layouts) for the model from the original framework.
|
||||
- Model Optimizer can change input element types and order of dimensions (layouts) for the model from the original framework.
|
||||
- Inference Engine can override input and output element types.
|
||||
- Inference Engine API uses operation names to address inputs and outputs (e.g. InferenceEngine::InferRequest::GetBlob).
|
||||
- Inference Engine API does not support compiling of models with dynamic input shapes.
|
||||
- **New behavior** assumes full model alignment with the framework and is implemented in OpenVINO 2022.1:
|
||||
- Model Optimizer preserves input element types, order of dimensions (layouts), and stores tensor names from the original models.
|
||||
- OpenVINO Runtime 2022.1 reads models in any format (IR v10, IR v11, ONNX, PaddlePaddle, etc.).
|
||||
- **New behavior** implemented in 2022.1 assumes full model alignment with the framework:
|
||||
- Model Optimizer preserves input element types and order of dimensions (layouts), and stores tensor names from the original models.
|
||||
- OpenVINO Runtime 2022.1 reads models in any format (OpenVINO IR v10, OpenVINO IR v11, ONNX, PaddlePaddle, etc.).
|
||||
- API 2.0 uses tensor names for addressing, which is the standard approach among the compatible model frameworks.
|
||||
- API 2.0 can also address input and output tensors by the index. Some model formats like ONNX are sensitive to the input, and output order, which is preserved by OpenVINO 2022.1.
|
||||
- API 2.0 can also address input and output tensors by the index. Some model formats like ONNX are sensitive to the input and output order, which is preserved by OpenVINO 2022.1.
|
||||
|
||||
The table below demonstrates which behavior, **old** or **new**, is used for models based on the two APIs.
|
||||
|
||||
| API | IR v10 | IR v11 | ONNX file | Model created in code |
|
||||
|-------------------------------|---------|---------|-----------|-----------------------|
|
||||
|Inference Engine / nGraph APIs | Old | Old | Old | Old |
|
||||
|API 2.0 | Old | New | New | New |
|
||||
| API | OpenVINO IR v10 | OpenVINO IR v11 | ONNX Files | Models Created in Code |
|
||||
|-------------------------------|------------------|------------------|------------|------------------------|
|
||||
|Inference Engine / nGraph APIs | Old | Old | Old | Old |
|
||||
|API 2.0 | Old | New | New | New |
|
||||
|
||||
Check these transition guides to understand how to migrate Inference Engine-based applications to API 2.0:
|
||||
### More Information
|
||||
|
||||
See the following pages to understand how to migrate Inference Engine-based applications to API 2.0:
|
||||
- [Installation & Deployment](deployment_migration.md)
|
||||
- [OpenVINO™ Common Inference pipeline](common_inference_pipeline.md)
|
||||
- [Preprocess your model](./preprocessing.md)
|
||||
|
||||
@@ -1,41 +1,45 @@
|
||||
# Preprocessing {#openvino_2_0_preprocessing}
|
||||
|
||||
### Introduction
|
||||
This guide introduces how preprocessing works in API 2.0 by a comparison with preprocessing in the previous Inference Engine API. It also demonstrates how to migrate preprocessing scenarios from Inference Engine to API 2.0 via code samples.
|
||||
|
||||
Inference Engine API contains preprocessing capabilities in the `InferenceEngine::CNNNetwork` class. Such preprocessing information is not a part of the main inference graph executed by [OpenVINO devices](../supported_plugins/Device_Plugins.md). Therefore, it is stored and executed separately before the inference stage.
|
||||
## How Preprocessing Works in API 2.0
|
||||
|
||||
Inference Engine API contains preprocessing capabilities in the `InferenceEngine::CNNNetwork` class. Such preprocessing information is not a part of the main inference graph executed by [OpenVINO devices](../supported_plugins/Device_Plugins.md). Therefore, it is stored and executed separately before the inference stage:
|
||||
- Preprocessing operations are executed on the CPU for most OpenVINO inference plugins. Thus, instead of occupying accelerators, they keep the CPU busy with computational tasks.
|
||||
- Preprocessing information stored in `InferenceEngine::CNNNetwork` is lost when saving back to the IR file format.
|
||||
- Preprocessing information stored in `InferenceEngine::CNNNetwork` is lost when saving back to the OpenVINO IR file format.
|
||||
|
||||
OpenVINO Runtime API 2.0 introduces a [new way of adding preprocessing operations to the model](../preprocessing_overview.md) - each preprocessing or postprocessing operation is integrated directly into the model and compiled together with the inference graph.
|
||||
- Add preprocessing operations first using `ov::preprocess::PrePostProcessor`
|
||||
- Then, compile the model on the target, using `ov::Core::compile_model`
|
||||
API 2.0 introduces a [new way of adding preprocessing operations to the model](../preprocessing_overview.md) - each preprocessing or post-processing operation is integrated directly into the model and compiled together with the inference graph:
|
||||
- API 2.0 first adds preprocessing operations by using `ov::preprocess::PrePostProcessor`,
|
||||
- and then compiles the model on the target by using `ov::Core::compile_model`.
|
||||
|
||||
Having preprocessing operations as a part of an OpenVINO opset makes it possible to read and serialize a preprocessed model as the IR file format.
|
||||
Having preprocessing operations as a part of an OpenVINO opset makes it possible to read and serialize a preprocessed model as the OpenVINO™ IR file format.
|
||||
|
||||
More importantly, OpenVINO Runtime API 2.0 does not assume any default layouts, as Inference Engine did. For example, both `{ 1, 224, 224, 3 }` and `{ 1, 3, 224, 224 }` shapes are supposed to be in the `NCHW` layout, while only the latter is. Therefore, some preprocessing capabilities in the API require layouts to be set explicitly. To learn how to do it, refer to the [Layout overview](../layout_overview.md). For example, to perform image scaling by partial dimensions `H` and `W`, preprocessing needs to know what dimensions `H` and `W` are.
|
||||
More importantly, API 2.0 does not assume any default layouts as Inference Engine did. For example, both `{ 1, 224, 224, 3 }` and `{ 1, 3, 224, 224 }` shapes are supposed to be in the `NCHW` layout, while only the latter is. Therefore, some preprocessing capabilities in the API require layouts to be set explicitly. To learn how to do it, refer to the [Layout overview](../layout_overview.md). For example, to perform image scaling by partial dimensions `H` and `W`, preprocessing needs to know what dimensions `H` and `W` are.
|
||||
|
||||
> **NOTE**: Use Model Optimizer preprocessing capabilities to insert preprocessing operations in your model for optimization. Thus, the application does not need to read the model and set preprocessing repeatedly. You can use the [model caching feature](../Model_caching_overview.md) to improve the time-to-inference.
|
||||
|
||||
The steps below demonstrate how to migrate preprocessing scenarios from Inference Engine API to OpenVINO Runtime API 2.0.
|
||||
The snippets assume we need to preprocess a model input with the `tensor_name` in Inference Engine API, using `operation_name` to address the data.
|
||||
The following sections demonstrate how to migrate preprocessing scenarios from Inference Engine API to API 2.0.
|
||||
The snippets assume that you need to preprocess a model input with the `tensor_name` in Inference Engine API, using `operation_name` to address the data.
|
||||
|
||||
#### Importing Preprocessing in Python
|
||||
## Preparation: Import Preprocessing in Python
|
||||
|
||||
In order to utilize preprocessing, the following imports must be added.
|
||||
|
||||
Inference Engine API:
|
||||
**Inference Engine API**
|
||||
|
||||
@snippet docs/snippets/ov_preprocessing_migration.py imports
|
||||
|
||||
OpenVINO Runtime API 2.0:
|
||||
**API 2.0**
|
||||
|
||||
@snippet docs/snippets/ov_preprocessing_migration.py ov_imports
|
||||
|
||||
There are two different namespaces: `runtime`, which contains OpenVINO Runtime API classes; and `preprocess`, which provides Preprocessing API.
|
||||
There are two different namespaces:
|
||||
- `runtime`, which contains API 2.0 classes;
|
||||
- and `preprocess`, which provides Preprocessing API.
|
||||
|
||||
### Mean and Scale Values
|
||||
## Using Mean and Scale Values
|
||||
|
||||
Inference Engine API:
|
||||
**Inference Engine API**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -53,7 +57,7 @@ Inference Engine API:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
OpenVINO Runtime API 2.0:
|
||||
**API 2.0**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -71,9 +75,9 @@ OpenVINO Runtime API 2.0:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
### Precision and Layout Conversions
|
||||
## Converting Precision and Layout
|
||||
|
||||
Inference Engine API:
|
||||
**Inference Engine API**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -91,7 +95,7 @@ Inference Engine API:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
OpenVINO Runtime API 2.0:
|
||||
**API 2.0**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -109,9 +113,9 @@ OpenVINO Runtime API 2.0:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
### Image Scaling
|
||||
## Using Image Scaling
|
||||
|
||||
Inference Engine API:
|
||||
**Inference Engine API**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -129,7 +133,7 @@ Inference Engine API:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
OpenVINO Runtime API 2.0:
|
||||
**API 2.0**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -147,9 +151,9 @@ OpenVINO Runtime API 2.0:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
### Color Space Conversions
|
||||
### Converting Color Space
|
||||
|
||||
Inference Engine API:
|
||||
**Inference Engine API**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -167,7 +171,7 @@ Inference Engine API:
|
||||
|
||||
@endsphinxtabset
|
||||
|
||||
OpenVINO Runtime API 2.0:
|
||||
**API 2.0**
|
||||
|
||||
@sphinxtabset
|
||||
|
||||
@@ -186,6 +190,7 @@ OpenVINO Runtime API 2.0:
|
||||
@endsphinxtabset
|
||||
|
||||
|
||||
**See also:**
|
||||
## Additional Resources
|
||||
|
||||
- [Preprocessing details](../preprocessing_details.md)
|
||||
- [NV12 classification sample](../../../samples/cpp/hello_nv12_input_classification/README.md)
|
||||
Reference in new issue
Block a user