openvino/docs/OV_Runtime_UG/ShapeInference.md

# Changing Input Shapes {#openvino_docs_OV_UG_ShapeInference}



@sphinxdirective
.. raw:: html

    <div id="switcher-cpp" class="switcher-anchor">C++</div>
@endsphinxdirective

OpenVINO™ provides capabilities to change model input shape during the runtime.
It may be useful when you want to feed model an input that has different size than model input shape. 
If you need to do this only once, prepare a model with updated shapes via Model Optimizer. See [Specifying --input_shape Command-line Parameter](@ref when_to_specify_input_shapes) for more information. For all the other cases, follow the instructions below.

### Setting a New Input Shape with Reshape Method

The `ov::Model::reshape` method updates input shapes and propagates them down to the outputs of the model through all intermediate layers.
For example, changing the batch size and spatial dimensions of input of a model with an image input:

![shape_inference_explained](./img/original_vs_reshaped_model.png)

Consider the code below to achieve that:

@snippet snippets/ShapeInference.cpp picture_snippet

### Setting a New Batch Size with set_batch Method

The meaning of the model batch may vary depending on the model design.
In order to change the batch dimension of the model, [set the ov::Layout](@ref declare_model_s_layout) and call the `ov::set_batch` method.

@snippet snippets/ShapeInference.cpp set_batch

The `ov::set_batch` method is a high level API of the `ov::Model::reshape` functionality, so all information about the `ov::Model::reshape` method implications are applicable for `ov::set_batch` too, including the troubleshooting section.

Once the input shape of `ov::Model` is set, call the `ov::Core::compile_model` method to get an `ov::CompiledModel` object for inference with updated shapes.

There are other approaches to change model input shapes during the stage of [IR generation](@ref when_to_specify_input_shapes) or [ov::Model creation](../OV_Runtime_UG/model_representation.md).

### Dynamic Shape Notice

Shape-changing functionality could be used to turn dynamic model input into a static one and vice versa.
It is recommended to always set static shapes when the shape of data is not going to change from one inference to another.
Setting static shapes can avoid possible functional limitations, memory, and runtime overheads for dynamic shapes which may vary depending on hardware plugin and model used.
To learn more about dynamic shapes in OpenVINO, see the [Dynamic Shapes](../OV_Runtime_UG/ov_dynamic_shapes.md) page.

### Usage of the Reshape Method <a name="usage_of_reshape_method"></a>

The primary method of the feature is `ov::Model::reshape`. It is overloaded to better serve two main use cases:

1) To change the input shape of the model with a single input, you may pass a new shape to the method. See the example of adjusting spatial dimensions to the input image below:

   @snippet snippets/ShapeInference.cpp spatial_reshape

To do the opposite - resize input image to the input shapes of the model, use the [pre-processing API](../OV_Runtime_UG/preprocessing_overview.md).

2) Otherwise, you can express reshape plan via mapping of input and its new shape:
* `map<ov::Output<ov::Node>, ov::PartialShape` specifies input by passing actual input port
* `map<size_t, ov::PartialShape>` specifies input by its index
* `map<string, ov::PartialShape>` specifies input by its name

@sphinxdirective

.. tab:: Port

    .. doxygensnippet:: docs/snippets/ShapeInference.cpp
       :language: cpp
       :fragment: [obj_to_shape]

.. tab:: Index

    .. doxygensnippet:: docs/snippets/ShapeInference.cpp
       :language: cpp
       :fragment: [idx_to_shape]

.. tab:: Tensor Name

    .. doxygensnippet:: docs/snippets/ShapeInference.cpp
       :language: cpp
       :fragment: [name_to_shape]

@endsphinxdirective

The usage scenarios of the `reshape` feature can be found in [OpenVINO Samples](Samples_Overview.md), starting with the [Hello Reshape Sample](../../samples/cpp/hello_reshape_ssd/README.md).

In practice, some models are not ready to be reshaped. In such cases, a new input shape cannot be set with Model Optimizer or the `ov::Model::reshape` method.

@anchor troubleshooting_reshape_errors
### Troubleshooting Reshape Errors

Operation semantics may impose restrictions on input shapes of the operation. 
Shape collision during shape propagation may be a sign that a new shape does not satisfy the restrictions. 
Changing the model input shape may result in intermediate operations shape collision.

Examples of such operations:
* The [Reshape](../ops/shape/Reshape_1.md) operation with a hard-coded output shape value.
* The [MatMul](../ops/matrix/MatMul_1.md) operation with the `Const` second input and this input cannot be resized by spatial dimensions due to operation semantics.

Model structure and logic should not change significantly after model reshaping.
- The Global Pooling operation is commonly used to reduce output feature map of classification models output.
Having the input of the shape [N, C, H, W], Global Pooling returns the output of the shape [N, C, 1, 1].
Model architects usually express Global Pooling with the help of the `Pooling` operation with the fixed kernel size [H, W].
During spatial reshape, having the input of the shape [N, C, H1, W1], Pooling with the fixed kernel size [H, W] returns the output of the shape [N, C, H2, W2], where H2 and W2 are commonly not equal to `1`.
It breaks the classification model structure.
For example, the publicly available [Inception family models from TensorFlow](https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models) have this issue.

- Changing the model input shape may significantly affect its accuracy.
For example, Object Detection models from TensorFlow have resizing restrictions by design. 
To keep the model valid after the reshape, choose a new input shape that satisfies conditions listed in the `pipeline.config` file. 
For details, refer to the [Tensorflow Object Detection API models resizing techniques](@ref custom-input-shape).

@anchor how-to-fix-non-reshape-able-model
### How To Fix Non-Reshape-able Model

Some operators which prevent normal shape propagation can be fixed. To do so you can:
* see if the issue can be fixed via changing the values of some operators' input. 
For example, the most common problem of non-reshape-able models is a `Reshape` operator with hard-coded output shape.
You can cut-off hard-coded 2nd input of `Reshape` and fill it in with relaxed values.
For the following example on the picture, the Model Optimizer CLI should be:
```sh
mo --input_model path/to/model --input data[8,3,224,224],1:reshaped[2]->[0 -1]`
```
With `1:reshaped[2]`, it's requested to cut the 2nd input (counting from zero, so `1:` means the 2nd input) of the operation named `reshaped` and replace it with a `Parameter` with shape `[2]`.
With `->[0 -1]`, this new `Parameter` is replaced by a `Constant` operator which has the `[0, -1]` value. 
Since the `Reshape` operator has `0` and `-1` as specific values (see the meaning in [this specification](../ops/shape/Reshape_1.md)), it allows propagating shapes freely without losing the intended meaning of `Reshape`.

![batch_relaxed](./img/batch_relaxation.png)

* transform the model during Model Optimizer conversion on the back phase. For more information, see the [Model Optimizer extension](../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md).
* transform OpenVINO Model during the runtime. For more information, see [OpenVINO Runtime Transformations](../Extensibility_UG/ov_transformations.md).
* modify the original model with the help of the original framework.

### Extensibility
OpenVINO provides a special mechanism that allows adding support of shape inference for custom operations. This mechanism is described in the [Extensibility documentation](../Extensibility_UG/Intro.md)

## Introduction (Python)

@sphinxdirective
.. raw:: html

    <div id="switcher-python" class="switcher-anchor">Python</div>
@endsphinxdirective

OpenVINO™ provides capabilities to change model input shape during the runtime.
It may be useful when you want to feed model an input that has different size than model input shape. 
If you need to do this only once, prepare a model with updated shapes via Model Optimizer. See [specifying input shapes](@ref when_to_specify_input_shapes) for more information. For all the other cases, follow the instructions below.

### Setting a New Input Shape with Reshape Method

The [Model.reshape](api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.reshape) method updates input shapes and propagates them down to the outputs of the model through all intermediate layers.
Example: Changing the batch size and spatial dimensions of input of a model with an image input:

![shape_inference_explained](./img/original_vs_reshaped_model.png)

Consider the code below to achieve that:

@sphinxdirective

.. doxygensnippet:: docs/snippets/ShapeInference.py
   :language: python
   :fragment: [picture_snippet]
 
@endsphinxdirective

### Setting a New Batch Size with the set_batch Method

The meaning of the model batch may vary depending on the model design.
In order to change the batch dimension of the model, [set the layout](@ref declare_model_s_layout) for inputs and call the [set_batch](api/ie_python_api/_autosummary/openvino.runtime.set_batch.html) method.

@sphinxdirective

.. doxygensnippet:: docs/snippets/ShapeInference.py
   :language: python
   :fragment: [set_batch]
 
@endsphinxdirective

[set_batch](api/ie_python_api/_autosummary/openvino.runtime.set_batch.html) method is a high level API of [Model.reshape](api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.reshape) functionality, so all information about [Model.reshape](api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.reshape) method implications are applicable for [set_batch](api/ie_python_api/_autosummary/openvino.runtime.set_batch.html) too, including the troubleshooting section.

Once the input shape of [Model](api/ie_python_api/_autosummary/openvino.runtime.Model.html) is set, call the [compile_model](api/ie_python_api/_autosummary/openvino.runtime.compile_model.html) method to get a [CompiledModel](api/ie_python_api/_autosummary/openvino.runtime.CompiledModel.html) object for inference with updated shapes.

There are other approaches to change model input shapes during the stage of [IR generation](@ref when_to_specify_input_shapes) or [Model creation](../OV_Runtime_UG/model_representation.md).

### Dynamic Shape Notice

Shape-changing functionality could be used to turn dynamic model input into a static one and vice versa.
It is recommended to always set static shapes when the shape of data is not going to change from one inference to another.
Setting static shapes can avoid possible functional limitations, memory, and runtime overheads for dynamic shapes which may vary depending on hardware plugin and used model.
To learn more about dynamic shapes in OpenVINO, see the [Dynamic Shapes](../OV_Runtime_UG/ov_dynamic_shapes.md) article.

### Usage of the Reshape Method <a name="usage_of_reshape_method"></a>

The primary method of the feature is [Model.reshape](api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.reshape). It is overloaded to better serve two main use cases:

1) To change the input shape of a model with a single input, you may pass a new shape to the method. See the example of adjusting spatial dimensions to the input image:

@sphinxdirective

.. doxygensnippet:: docs/snippets/ShapeInference.py
   :language: python
   :fragment: [simple_spatials_change]
 
@endsphinxdirective

To do the opposite - resize input image to the input shapes of the model, use the [pre-processing API](../OV_Runtime_UG/preprocessing_overview.md).

2) Otherwise, you can express reshape plan via dictionary mapping input and its new shape:
Dictionary keys could be:
* The `str` key specifies input by its name.
* The `int` key specifies input by its index.
* The `openvino.runtime.Output` key specifies input by passing the actual input object.

Dictionary values (representing new shapes) could be:
* `list`
* `tuple`
* `PartialShape`

@sphinxdirective

.. tab:: Port

    .. doxygensnippet:: docs/snippets/ShapeInference.py
       :language: python
       :fragment: [obj_to_shape]

.. tab:: Index

    .. doxygensnippet:: docs/snippets/ShapeInference.py
       :language: python
       :fragment: [idx_to_shape]

.. tab:: Tensor Name

    .. doxygensnippet:: docs/snippets/ShapeInference.py
       :language: python
       :fragment: [name_to_shape]

@endsphinxdirective

The usage scenarios of the `reshape` feature can be found in [OpenVINO Samples](Samples_Overview.md), starting with the [Hello Reshape Sample](../../samples/python/hello_reshape_ssd/README.md).

In practice, some models are not ready to be reshaped. In such cases, a new input shape cannot be set with Model Optimizer or the `Model.reshape` method.

### Troubleshooting Reshape Errors

Operation semantics may impose restrictions on input shapes of the operation. 
Shape collision during shape propagation may be a sign that a new shape does not satisfy the restrictions. 
Changing the model input shape may result in intermediate operations shape collision.

Examples of such operations:
* [Reshape](../ops/shape/Reshape_1.md) operation with a hard-coded output shape value
* [MatMul](../ops/matrix/MatMul_1.md) operation with the `Const` second input cannot be resized by spatial dimensions due to operation semantics

Model structure and logic should not change significantly after model reshaping.
- The Global Pooling operation is commonly used to reduce output feature map of classification models output.
Having the input of the shape [N, C, H, W], Global Pooling returns the output of the shape [N, C, 1, 1].
Model architects usually express Global Pooling with the help of the `Pooling` operation with the fixed kernel size [H, W].
During spatial reshape, having the input of the shape [N, C, H1, W1], Pooling with the fixed kernel size [H, W] returns the output of the shape [N, C, H2, W2], where H2 and W2 are commonly not equal to `1`.
It breaks the classification model structure.
For example, the publicly available [Inception family models from TensorFlow](https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models) have this issue.

- Changing the model input shape may significantly affect its accuracy.
For example, Object Detection models from TensorFlow have resizing restrictions by design. 
To keep the model valid after the reshape, choose a new input shape that satisfies conditions listed in the `pipeline.config` file. 
For details, refer to the [Tensorflow Object Detection API models resizing techniques](@ref custom-input-shape).

### How To Fix Non-Reshape-able Model

Some operators which prevent normal shape propagation can be fixed. To do so you can:
* see if the issue can be fixed via changing the values of some operators input. 
For example, the most common problem of non-reshape-able models is a `Reshape` operator with hard-coded output shape.
You can cut-off hard-coded 2nd input of `Reshape` and fill it in with relaxed values.
For the following example on the picture Model Optimizer CLI should be:
```sh
mo --input_model path/to/model --input data[8,3,224,224],1:reshaped[2]->[0 -1]`
```
With `1:reshaped[2]`, it's requested to cut the 2nd input (counting from zero, so `1:` means the 2nd input) of the operation named `reshaped` and replace it with a `Parameter` with shape `[2]`.
With `->[0 -1]`, this new `Parameter` is replaced by a `Constant` operator which has value `[0, -1]`. 
Since the `Reshape` operator has `0` and `-1` as specific values (see the meaning in [this specification](../ops/shape/Reshape_1.md)), it allows propagating shapes freely without losing the intended meaning of `Reshape`.

![batch_relaxed](./img/batch_relaxation.png)

* transform the model during Model Optimizer conversion on the back phase. See [Model Optimizer extension](../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md).
* transform OpenVINO Model during the runtime. See [OpenVINO Runtime Transformations](../Extensibility_UG/ov_transformations.md).
* modify the original model with the help of the original framework.

### Extensibility
OpenVINO provides a special mechanism that allows adding support of shape inference for custom operations. This mechanism is described in the [Extensibility documentation](../Extensibility_UG/Intro.md)