IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

DOCS: Language-agnostic version of 'Changing Input Shapes' - for 22.1 (#13816)

Browse Source 
Removing the 'global' tabs and preparing a language-agnostic version of the article. Replacing png image with a scalable svg file. Proofreading the article.
This commit is contained in:
Sebastian Golebiewski
2022-11-09 15:30:41 +01:00
committed by GitHub
parent 0fca7bb95e
commit 521df07e44
4 changed files with 190 additions and 257 deletions
Download Patch File Download Diff File Expand all files Collapse all files

397
docs/OV_Runtime_UG/ShapeInference.md
View File

@@ -1,287 +1,176 @@
# Changing input shapes {#openvino_docs_OV_UG_ShapeInference}
## Introduction (C++)
@sphinxdirective
.. raw:: html
<div id="switcher-cpp" class="switcher-anchor">C++</div>
.. toctree::
:maxdepth: 1
:hidden:
troubleshooting_reshape_errors
@endsphinxdirective
OpenVINO™ provides capabilities to change model input shape during the runtime.
It may be useful in case you would like to feed model an input that has different size than model input shape.
In case you need to do this only once [prepare a model with updated shapes via Model Optimizer](@ref when_to_specify_input_shapes) for all the other cases follow instructions further.
### Set a new input shape with reshape method
OpenVINO™ enables you to change model input shape during the application runtime. It may be useful when you want to feed the model an input that has different size than the model input shape. The following instructions are for cases where you need to change the model input shape repeatedly.
The `ov::Model::reshape` method updates input shapes and propagates them down to the outputs of the model through all intermediate layers.
Example: Changing the batch size and spatial dimensions of input of a model with an image input:
> **NOTE**: If you need to do this only once, prepare a model with updated shapes via [Model Optimizer](@ref openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide). For more information, refer to the [Specifying --input_shape Command-line Parameter](@ref when_to_specify_input_shapes) article.
![shape_inference_explained](./img/original_vs_reshaped_model.png)
### The reshape method
Please see the code to achieve that:
The reshape method is used as `ov::Model::reshape` in C++ and [Model.reshape](api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.reshape) in Python. The method updates input shapes and propagates them down to the outputs of the model through all intermediate layers.
The code below is an example of how to set a new batch size with the `reshape` method:
@sphinxtabset
@sphinxtab{C++}
@snippet snippets/ShapeInference.cpp picture_snippet
### Set a new batch size with set_batch method
@endsphinxtab
Meaning of the model batch may vary depending on the model design.
In order to change the batch dimension of the model, [set the ov::Layout](@ref declare_model_s_layout) and call the `ov::set_batch` method.
@sphinxtab{Python}
@snippet docs/snippets/ShapeInference.py picture_snippet
@endsphinxtab
@endsphinxtabset
The diagram below presents the results of using the method, where the size of model input is changed with an image input:
![shape_inference_explained](./img/original_vs_reshaped_model.svg)
When using the `reshape` method, you may take one of the approaches:
@anchor usage_of_reshape_method
@sphinxdirective
#. You can pass a new shape to the method in order to change the input shape of the model with a single input. See the example of adjusting spatial dimensions to the input image:
.. tab:: C++
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: spatial_reshape
.. tab:: Python
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: simple_spatials_change
To do the opposite - to resize input image to match the input shapes of the model, use the :ref:`pre-processing API <openvino_docs_OV_UG_Preprocessing_Overview>`.
#. You can express a reshape plan, specifying the input by the port, the index, and the tensor name:
.. tab:: Port
.. tab:: C++
map<ov::Output<ov::Node>, ov::PartialShape specifies input by passing actual input port:
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: [obj_to_shape]
.. tab:: Python
`openvino.runtime.Output` dictionary key specifies input by passing actual input object.
Dictionary values representing new shapes could be `PartialShape`:
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: [obj_to_shape]
.. tab:: Index
.. tab:: C++
map<size_t, ov::PartialShape> specifies input by its index:
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: [idx_to_shape]
.. tab:: Python
`int` dictionary key specifies input by its index.
Dictionary values representing new shapes could be `tuple`:
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: [idx_to_shape]
.. tab:: Tensor Name
.. tab:: C++
map<string, ov::PartialShape> specifies input by its name:
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: [name_to_shape]
.. tab:: Python
`str` dictionary key specifies input by its name.
Dictionary values representing new shapes could be `str`:
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: [name_to_shape]
@endsphinxdirective
You can find the usage scenarios of the `reshape` method in [Hello Reshape SSD Samples](@ref openvino_inference_engine_samples_hello_reshape_ssd_README).
> **NOTE**: In some cases, models may not be ready to be reshaped. Therefore, a new input shape cannot be set neither with [Model Optimizer](@ref openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide) nor the `reshape` method.
### The set_batch method
The meaning of the model batch may vary depending on the model design.
To change the batch dimension of the model, [set the layout](@ref declare_model_s_layout) and call the `set_batch` method.
@sphinxtabset
@sphinxtab{C++}
@snippet snippets/ShapeInference.cpp set_batch
`ov::set_batch` method is a high level API of `ov::Model::reshape` functionality, so all information about `ov::Model::reshape` method implications are applicable for `ov::set_batch` too, including the troubleshooting section.
@endsphinxtab
Once the input shape of `ov::Model` is set, call the `ov::Core::compile_model` method to get an `ov::CompiledModel` object for inference with updated shapes.
@sphinxtab{Python}
There are other approaches to change model input shapes during the stage of [IR generation](@ref when_to_specify_input_shapes) or [ov::Model creation](../OV_Runtime_UG/model_representation.md).
@snippet docs/snippets/ShapeInference.py set_batch
### Dynamic Shape Notice
@endsphinxtab
Shape-changing functionality could be used to turn dynamic model input into a static one and vice versa.
It is recommended to always set static shapes in case if the shape of data is not going to change from one inference to another.
Setting static shapes avoids possible functional limitations, memory and run time overheads for dynamic shapes that vary depending on hardware plugin and model used.
To learn more about dynamic shapes in OpenVINO please see a [dedicated article](../OV_Runtime_UG/ov_dynamic_shapes.md).
@endsphinxtabset
### Usage of Reshape Method <a name="usage_of_reshape_method"></a>
The `set_batch` method is a high-level API of the reshape functionality, so all information about the `reshape` method implications are applicable for `set_batch` too, including the troubleshooting section.
The primary method of the feature is `ov::Model::reshape`. It is overloaded to better serve two main use cases:
Once you set the input shape of the model, call the `compile_model` method to get a `CompiledModel` object for inference with updated shapes.
1) To change input shape of model with single input you may pass new shape into the method. Please see the example of adjusting spatial dimensions to the input image:
@snippet snippets/ShapeInference.cpp spatial_reshape
To do the opposite - resize input image to the input shapes of the model, use the [pre-processing API](../OV_Runtime_UG/preprocessing_overview.md).
2) Otherwise, you can express reshape plan via mapping of input and its new shape:
* `map<ov::Output<ov::Node>, ov::PartialShape` specifies input by passing actual input port
* `map<size_t, ov::PartialShape>` specifies input by its index
* `map<string, ov::PartialShape>` specifies input by its name
There are other approaches to change model input shapes during the stage of [IR generation](@ref when_to_specify_input_shapes) or [model representation](@ref openvino_docs_OV_UG_Model_Representation) in OpenVINO Runtime.
@sphinxdirective
.. tab:: Port
.. important::
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: [obj_to_shape]
.. tab:: Index
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: [idx_to_shape]
.. tab:: Tensor Name
.. doxygensnippet:: docs/snippets/ShapeInference.cpp
:language: cpp
:fragment: [name_to_shape]
Shape-changing functionality could be used to turn dynamic model input into a static one and vice versa. Always set static shapes when the shape of data is NOT going to change from one inference to another. Setting static shapes can avoid memory and runtime overheads for dynamic shapes which may vary depending on hardware plugin and model used. For more information, refer to the :ref:`Dynamic Shapes <openvino_docs_OV_UG_DynamicShapes>`.
@endsphinxdirective
Please find usage scenarios of `reshape` feature in our [samples](Samples_Overview.md) starting with [Hello Reshape Sample](../../samples/cpp/hello_reshape_ssd/README.md).
Practically, some models are not ready to be reshaped. In this case, a new input shape cannot be set with the Model Optimizer or the `ov::Model::reshape` method.
## Additional Resources
@anchor troubleshooting_reshape_errors
### Troubleshooting Reshape Errors
Operation semantics may impose restrictions on input shapes of the operation.
Shape collision during shape propagation may be a sign that a new shape does not satisfy the restrictions.
Changing the model input shape may result in intermediate operations shape collision.
Examples of such operations:
* [Reshape](../ops/shape/Reshape_1.md) operation with a hard-coded output shape value
* [MatMul](../ops/matrix/MatMul_1.md) operation with the `Const` second input cannot be resized by spatial dimensions due to operation semantics
Model structure and logic should not change significantly after model reshaping.
- The Global Pooling operation is commonly used to reduce output feature map of classification models output.
Having the input of the shape [N, C, H, W], Global Pooling returns the output of the shape [N, C, 1, 1].
Model architects usually express Global Pooling with the help of the `Pooling` operation with the fixed kernel size [H, W].
During spatial reshape, having the input of the shape [N, C, H1, W1], Pooling with the fixed kernel size [H, W] returns the output of the shape [N, C, H2, W2], where H2 and W2 are commonly not equal to `1`.
It breaks the classification model structure.
For example, [publicly available Inception family models from TensorFlow*](https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models) have this issue.
- Changing the model input shape may significantly affect its accuracy.
For example, Object Detection models from TensorFlow have resizing restrictions by design.
To keep the model valid after the reshape, choose a new input shape that satisfies conditions listed in the `pipeline.config` file.
For details, refer to the [Tensorflow Object Detection API models resizing techniques](@ref custom-input-shape).
@anchor how-to-fix-non-reshape-able-model
### How To Fix Non-Reshape-able Model
Some operators which prevent normal shape propagation can be fixed. To do so you can:
* see if the issue can be fixed via changing the values of some operators input.
E.g. most common problem of non-reshape-able models is a `Reshape` operator with hardcoded output shape.
You can cut-off hard-coded 2nd input of `Reshape` and fill it in with relaxed values.
For the following example on the picture Model Optimizer CLI should be:
```sh
mo --input_model path/to/model --input data[8,3,224,224],1:reshaped[2]->[0 -1]`
```
With `1:reshaped[2]` we request to cut 2nd input (counting from zero, so `1:` means 2nd inputs) of operation named `reshaped` and replace it with a `Parameter` with shape `[2]`.
With `->[0 -1]` we replace this new `Parameter` by a `Constant` operator which has value `[0, -1]`.
Since `Reshape` operator has `0` and `-1` as a specific values (see the meaning in [the specification](../ops/shape/Reshape_1.md)) it allows to propagate shapes freely without losing the intended meaning of `Reshape`.
![batch_relaxed](./img/batch_relaxation.png)
* transform model during Model Optimizer conversion on the back phase. See [Model Optimizer extension article](../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md)
* transform OpenVINO Model during the runtime. See [OpenVINO Runtime Transformations article](../Extensibility_UG/ov_transformations.md)
* modify the original model with the help of original framework
### Extensibility
OpenVINO provides a special mechanism that allows adding support of shape inference for custom operations. This mechanism is described in the [Extensibility documentation](../Extensibility_UG/Intro.md)
## Introduction (Python)
@sphinxdirective
.. raw:: html
<div id="switcher-python" class="switcher-anchor">Python</div>
@endsphinxdirective
OpenVINO™ provides capabilities to change model input shape during the runtime.
It may be useful in case you would like to feed model an input that has different size than model input shape.
In case you need to do this only once [prepare a model with updated shapes via Model Optimizer](@ref when_to_specify_input_shapes) for all the other cases follow instructions further.
### Set a new input shape with reshape method
The [Model.reshape](api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.reshape) method updates input shapes and propagates them down to the outputs of the model through all intermediate layers.
Example: Changing the batch size and spatial dimensions of input of a model with an image input:
![shape_inference_explained](./img/original_vs_reshaped_model.png)
Please see the code to achieve that:
@sphinxdirective
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: [picture_snippet]
@endsphinxdirective
### Set a new batch size with set_batch method
Meaning of the model batch may vary depending on the model design.
In order to change the batch dimension of the model, [set the layout](@ref declare_model_s_layout) for inputs and call the [set_batch](api/ie_python_api/_autosummary/openvino.runtime.set_batch.html) method.
@sphinxdirective
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: [set_batch]
@endsphinxdirective
[set_batch](api/ie_python_api/_autosummary/openvino.runtime.set_batch.html) method is a high level API of [Model.reshape](api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.reshape) functionality, so all information about [Model.reshape](api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.reshape) method implications are applicable for [set_batch](api/ie_python_api/_autosummary/openvino.runtime.set_batch.html) too, including the troubleshooting section.
Once the input shape of [Model](api/ie_python_api/_autosummary/openvino.runtime.Model.html) is set, call the [compile_model](api/ie_python_api/_autosummary/openvino.runtime.compile_model.html) method to get a [CompiledModel](api/ie_python_api/_autosummary/openvino.runtime.CompiledModel.html) object for inference with updated shapes.
There are other approaches to change model input shapes during the stage of [IR generation](@ref when_to_specify_input_shapes) or [Model creation](../OV_Runtime_UG/model_representation.md).
### Dynamic Shape Notice
Shape-changing functionality could be used to turn dynamic model input into a static one and vice versa.
It is recommended to always set static shapes in case if the shape of data is not going to change from one inference to another.
Setting static shapes avoids possible functional limitations, memory and run time overheads for dynamic shapes that vary depending on hardware plugin and model used.
To learn more about dynamic shapes in OpenVINO please see a [dedicated article](../OV_Runtime_UG/ov_dynamic_shapes.md).
### Usage of Reshape Method <a name="usage_of_reshape_method"></a>
The primary method of the feature is [Model.reshape](api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.reshape). It is overloaded to better serve two main use cases:
1) To change input shape of model with single input you may pass new shape into the method. Please see the example of adjusting spatial dimensions to the input image:
@sphinxdirective
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: [simple_spatials_change]
@endsphinxdirective
To do the opposite - resize input image to the input shapes of the model, use the [pre-processing API](../OV_Runtime_UG/preprocessing_overview.md).
2) Otherwise, you can express reshape plan via dictionary mapping input and its new shape:
Dictionary keys could be
* `str` specifies input by its name
* `int` specifies input by its index
* `openvino.runtime.Output` specifies input by passing actual input object
Dictionary values (representing new shapes) could be
* `list`
* `tuple`
* `PartialShape`
@sphinxdirective
.. tab:: Port
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: [obj_to_shape]
.. tab:: Index
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: [idx_to_shape]
.. tab:: Tensor Name
.. doxygensnippet:: docs/snippets/ShapeInference.py
:language: python
:fragment: [name_to_shape]
@endsphinxdirective
Please find usage scenarios of `reshape` feature in our [samples](Samples_Overview.md), starting with [Hello Reshape Sample](../../samples/python/hello_reshape_ssd/README.md).
Practically, some models are not ready to be reshaped. In this case, a new input shape cannot be set with the Model Optimizer or the `Model.reshape` method.
### Troubleshooting Reshape Errors
Operation semantics may impose restrictions on input shapes of the operation.
Shape collision during shape propagation may be a sign that a new shape does not satisfy the restrictions.
Changing the model input shape may result in intermediate operations shape collision.
Examples of such operations:
* [Reshape](../ops/shape/Reshape_1.md) operation with a hard-coded output shape value
* [MatMul](../ops/matrix/MatMul_1.md) operation with the `Const` second input cannot be resized by spatial dimensions due to operation semantics
Model structure and logic should not change significantly after model reshaping.
- The Global Pooling operation is commonly used to reduce output feature map of classification models output.
Having the input of the shape [N, C, H, W], Global Pooling returns the output of the shape [N, C, 1, 1].
Model architects usually express Global Pooling with the help of the `Pooling` operation with the fixed kernel size [H, W].
During spatial reshape, having the input of the shape [N, C, H1, W1], Pooling with the fixed kernel size [H, W] returns the output of the shape [N, C, H2, W2], where H2 and W2 are commonly not equal to `1`.
It breaks the classification model structure.
For example, [publicly available Inception family models from TensorFlow*](https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models) have this issue.
- Changing the model input shape may significantly affect its accuracy.
For example, Object Detection models from TensorFlow have resizing restrictions by design.
To keep the model valid after the reshape, choose a new input shape that satisfies conditions listed in the `pipeline.config` file.
For details, refer to the [Tensorflow Object Detection API models resizing techniques](@ref custom-input-shape).
### How To Fix Non-Reshape-able Model
Some operators which prevent normal shape propagation can be fixed. To do so you can:
* see if the issue can be fixed via changing the values of some operators input.
E.g. most common problem of non-reshape-able models is a `Reshape` operator with hardcoded output shape.
You can cut-off hard-coded 2nd input of `Reshape` and fill it in with relaxed values.
For the following example on the picture Model Optimizer CLI should be:
```sh
mo --input_model path/to/model --input data[8,3,224,224],1:reshaped[2]->[0 -1]`
```
With `1:reshaped[2]` we request to cut 2nd input (counting from zero, so `1:` means 2nd inputs) of operation named `reshaped` and replace it with a `Parameter` with shape `[2]`.
With `->[0 -1]` we replace this new `Parameter` by a `Constant` operator which has value `[0, -1]`.
Since `Reshape` operator has `0` and `-1` as a specific values (see the meaning in [the specification](../ops/shape/Reshape_1.md)) it allows to propagate shapes freely without losing the intended meaning of `Reshape`.
![batch_relaxed](./img/batch_relaxation.png)
* transform model during Model Optimizer conversion on the back phase. See [Model Optimizer extension article](../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md)
* transform OpenVINO Model during the runtime. See [OpenVINO Runtime Transformations article](../Extensibility_UG/ov_transformations.md)
* modify the original model with the help of original framework
### Extensibility
OpenVINO provides a special mechanism that allows adding support of shape inference for custom operations. This mechanism is described in the [Extensibility documentation](../Extensibility_UG/Intro.md)
* [Extensibility documentation](@ref openvino_docs_Extensibility_UG_Intro) - describes a special mechanism in OpenVINO that allows adding support of shape inference for custom operations.
* `ov::Model::reshape` - in OpenVINO Runtime C++ API
* [Model.reshape](api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.reshape) - in OpenVINO Runtime Python API.
* [Dynamic Shapes](@ref openvino_docs_OV_UG_DynamicShapes)
* [OpenVINO samples](@ref openvino_docs_OV_UG_Samples_Overview)
* [Preprocessing API](@ref openvino_docs_OV_UG_Preprocessing_Overview)

45
docs/OV_Runtime_UG/Troubleshooting_ReshapeMethod.md Normal file
View File

@@ -0,0 +1,45 @@
# Troubleshooting Reshape Errors {#troubleshooting_reshape_errors}
### How To Avoid Shape Collision
Operation semantics may impose restrictions on input shapes of the operation.
Shape collision during shape propagation may be a sign that new shape does not satisfy the restrictions.
Changing the model input shape may result in intermediate operations shape collision. For example, in the following:
* The [Reshape](@ref openvino_docs_ops_shape_Reshape_1) operation with a hard-coded output shape value,
* The [MatMul](@ref openvino_docs_ops_matrix_MatMul_1) operation with the `Const` second input and this input cannot be resized by spatial dimensions due to operation semantics.
Model structure and logic should not change significantly after model reshaping.
- The Global Pooling operation is commonly used to reduce output feature map of classification models output.
Having the input of the shape *[N, C, H, W]*, Global Pooling returns the output of the shape *[N, C, 1, 1]*.
Model architects usually express Global Pooling with the help of the `Pooling` operation with the fixed kernel size *[H, W]*.
During spatial reshape, having the input of the shape *[N, C, H1, W1]*, `Pooling` with the fixed kernel size *[H, W]* returns the output of the shape *[N, C, H2, W2]*, where *H2* and *W2* are commonly not equal to *1*.
It breaks the classification model structure.
For example, the public [Inception family models from TensorFlow](https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models) have this issue.
- Changing the model input shape may significantly affect its accuracy.
For example, Object Detection models from TensorFlow have resizing restrictions by design.
To keep the model valid after the reshape, choose a new input shape that satisfies conditions listed in the `pipeline.config` file.
For details, refer to the [Tensorflow Object Detection API models resizing techniques](@ref custom-input-shape).
@anchor how-to-fix-non-reshape-able-model
### How To Fix Non-Reshape-able Model
To fix some operators which prevent normal shape propagation:
* see if the issue can be fixed via changing the values of some operators' input.
For example, the most common problem of non-reshape-able models is a `Reshape` operator with a hard-coded output shape.
You can cut-off the hard-coded second input of `Reshape` and fill it in with relaxed values.
For the following example in the diagram below, the Model Optimizer CLI should read:
```sh
mo --input_model path/to/model --input data[8,3,224,224],1:reshaped[2]->[0 -1]`
```
With `1:reshaped[2]`, it is required to cut the second input (counting from zero, so `1:` means the second input) of the operation named `reshaped` and replace it with a `Parameter` with shape `[2]`.
With `->[0 -1]`, this new `Parameter` is replaced by a `Constant` operator which has the `[0, -1]` value.
Since the `Reshape` operator has `0` and `-1` as specific values, it allows propagating shapes freely without losing the intended meaning of `Reshape`. For more information, see [the specification](@ref openvino_docs_ops_shape_Reshape_1).
![batch_relaxed](./img/batch_relaxation.png)
* transform the model during Model Optimizer conversion on the back phase. For more information, see the [Model Optimizer extension](@ref openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Customize_Model_Optimizer),
* transform OpenVINO Model during the runtime. For more information, see [OpenVINO Runtime Transformations](@ref openvino_docs_transformations),
* modify the original model with the help of the original framework.

3
docs/OV_Runtime_UG/img/original_vs_reshaped_model.png
View File

@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f22fe23c17bcd2930871af56832ab3df5bbb7df93f1fc7e4a06de3ce2020791d
size 117644

2
docs/OV_Runtime_UG/img/original_vs_reshaped_model.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 84 KiB