[docs] python snippet for dynamic shapes (#10762)

* Create snipp

* link python snipp with doc

* fix docs

* Apply suggestions from code review

Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>

* Fix cpp comments

Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>
This commit is contained in:
Alexey Lebedev 2022-03-11 08:42:33 +03:00 committed by GitHub
parent 4e0a740eb3
commit 97efdb5020
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 234 additions and 11 deletions

View File

@ -42,7 +42,21 @@ To avoid the tricks mentioned in the previous section there is a way to directly
This is achieved with the same reshape method that is used for alternating static shape of inputs.
Dynamic dimensions are specified as `-1` or `ov::Dimension()` instead of a positive number used for static dimensions:
@snippet snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:reshape_undefined
@sphinxdirective
.. tab:: C++
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp
:language: cpp
:fragment: [ov_dynamic_shapes:reshape_undefined]
.. tab:: Python
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.py
:language: python
:fragment: [reshape_undefined]
@endsphinxdirective
To simplify the code, the examples assume that the model has a single input and single output.
However, there are no limitations on the number of inputs and outputs to apply dynamic shapes.
@ -66,7 +80,21 @@ Use this capability to save time on calling `reshape` method in the end applicat
Besides marking a dimension just dynamic, you can also specify lower and/or upper bounds that define a range of allowed values for the dimension.
Bounds are coded as arguments for `ov::Dimension`:
@snippet snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:reshape_bounds
@sphinxdirective
.. tab:: C++
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp
:language: cpp
:fragment: [ov_dynamic_shapes:reshape_bounds]
.. tab:: Python
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.py
:language: python
:fragment: [reshape_bounds]
@endsphinxdirective
Information about bounds gives opportunity for the inference plugin to apply additional optimizations.
Using dynamic shapes assumes the plugins apply more loose optimization technique during model compilation
@ -86,7 +114,21 @@ Preparing model with the reshape method was the first step.
The second step is passing a tensor with an appropriate shape to infer request.
This is similar to [regular steps](integrate_with_your_application.md), but now we can pass tensors with different shapes for the same executable model and even for the same inference request:
@snippet snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:set_input_tensor
@sphinxdirective
.. tab:: C++
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp
:language: cpp
:fragment: [ov_dynamic_shapes:set_input_tensor]
.. tab:: Python
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.py
:language: python
:fragment: [set_input_tensor]
@endsphinxdirective
In the example above `set_input_tensor` is used to specify input tensors.
The real dimensions of the tensor is always static, because it is a concrete tensor and it doesn't have any dimension variations in contrast to model inputs.
@ -97,7 +139,21 @@ Without doing that, the tensor returned by `get_input_tensor` is an empty tensor
Setting shape for input tensor is required when the corresponding input has at least one dynamic dimension regardless of bounds information.
The following example makes the same sequence of two infer request as the previous example but using `get_input_tensor` instead of `set_input_tensor`:
@snippet snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:get_input_tensor
@sphinxdirective
.. tab:: C++
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp
:language: cpp
:fragment: [ov_dynamic_shapes:get_input_tensor]
.. tab:: Python
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.py
:language: python
:fragment: [get_input_tensor]
@endsphinxdirective
### Dynamic Shapes in Outputs
@ -108,13 +164,41 @@ The same is true for other dimensions, like sequence length for NLP models or sp
Whether or not output has dynamic dimensions can be examined by querying output partial shape after model read or reshape.
The same is applicable for inputs. For example:
@snippet snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:print_dynamic
@sphinxdirective
.. tab:: C++
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp
:language: cpp
:fragment: [ov_dynamic_shapes:print_dynamic]
.. tab:: Python
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.py
:language: python
:fragment: [print_dynamic]
@endsphinxdirective
Appearing `?` or ranges like `1..10` means there are dynamic dimensions in corresponding inputs or outputs.
Or more programmatically:
@snippet snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:detect_dynamic
@sphinxdirective
.. tab:: C++
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp
:language: cpp
:fragment: [ov_dynamic_shapes:detect_dynamic]
.. tab:: Python
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.py
:language: python
:fragment: [detect_dynamic]
@endsphinxdirective
If at least one dynamic dimension exists in output of the model, shape of the corresponding output tensor will be set as the result of inference call.
Before the first inference, memory for such a tensor is not allocated and has shape `[0]`.

View File

@ -23,10 +23,10 @@ model->reshape({{ov::Dimension(), ov::Dimension()}}); // {?,?}
model->reshape({{-1, -1}}); // {?,?}
//! [ov_dynamic_shapes:reshape_undefined]
//! [ov_dynamic_shapes:reshape_bounds]
// Both dimensions are dynamic, first may have size within 1..10 and the second is withing 8..512
// Both dimensions are dynamic, first has a size within 1..10 and the second has a size within 8..512
model->reshape({{ov::Dimension(1, 10), ov::Dimension(8, 512)}}); // {1..10,8..512}
// Both dimensions are dynamic, first doesn't have bounds, the second is in 8..512
// Both dimensions are dynamic, first doesn't have bounds, the second is in the range of 8..512
model->reshape({{-1, ov::Dimension(8, 512)}}); // {?,8..512}
//! [ov_dynamic_shapes:reshape_bounds]
}
@ -69,12 +69,12 @@ auto infer_request = executable.create_infer_request();
//! [ov_dynamic_shapes:set_input_tensor]
// The first inference call
// Create tensor compatible to the model input
// Shape {1, 128} is compatible to any reshape statements made in previous examples
// Create tensor compatible with the model input
// Shape {1, 128} is compatible with any reshape statements made in previous examples
auto input_tensor_1 = ov::Tensor(model->input().get_element_type(), {1, 128});
// ... write values to input_tensor_1
// Set the tensor as a model input within infer request
// Set the tensor as an input for the infer request
infer_request.set_input_tensor(input_tensor_1);
// Do the inference

View File

@ -0,0 +1,139 @@
# Copyright (C) 2018-2022 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import numpy as np
#! [import]
import openvino.runtime as ov
#! [import]
#! [reshape_undefined]
core = ov.Core()
model = core.read_model("model.xml")
# Set one static dimension (= 1) and another dynamic dimension (= Dimension())
model.reshape([1, ov.Dimension()])
# The same as above
model.reshape([1, -1])
# The same as above
model.reshape("1, ?")
# Or set both dimensions as dynamic if both are going to be changed dynamically
model.reshape([ov.Dimension(), ov.Dimension()])
# The same as above
model.reshape([-1, -1])
# The same as above
model.reshape("?, ?")
#! [reshape_undefined]
#! [reshape_bounds]
# Both dimensions are dynamic, first has a size within 1..10 and the second has a size within 8..512
model.reshape([ov.Dimension(1, 10), ov.Dimension(8, 512)])
# The same as above
model.reshape([(1, 10), (8, 512)])
# The same as above
model.reshape("1..10, 8..512")
# Both dimensions are dynamic, first doesn't have bounds, the second is in the range of 8..512
model.reshape([-1, (8, 512)])
#! [reshape_bounds]
model = core.read_model("model.xml")
#! [print_dynamic]
# Print output partial shape
print(model.output().partial_shape)
# Print input partial shape
print(model.input().partial_shape)
#! [print_dynamic]
#! [detect_dynamic]
model = core.read_model("model.xml")
if model.input(0).partial_shape.is_dynamic():
# input is dynamic
pass
if model.output(0).partial_shape.is_dynamic():
# output is dynamic
pass
if model.output(0).partial_shape[1].is_dynamic():
# 1-st dimension of output is dynamic
pass
#! [detect_dynamic]
executable = core.compile_model(model)
infer_request = executable.create_infer_request()
#! [set_input_tensor]
# The first inference call
# Create tensor compatible to the model input
# Shape {1, 128} is compatible with any reshape statements made in previous examples
input_tensor1 = ov.Tensor(model.input().element_type, [1, 128])
# ... write values to input_tensor_1
# Set the tensor as an input for the infer request
infer_request.set_input_tensor(input_tensor1)
# Do the inference
infer_request.infer()
# Or pass a tensor in infer to set the tensor as a model input and make the inference
infer_request.infer([input_tensor1])
# Or pass the numpy array to set inputs of the infer request
input_data = np.ones(shape=[1, 128])
infer_request.infer([input_data])
# Retrieve a tensor representing the output data
output_tensor = infer_request.get_output_tensor()
# Copy data from tensor to numpy array
data1 = output_tensor.data[:]
# The second inference call, repeat steps:
# Create another tensor (if the previous one cannot be utilized)
# Notice, the shape is different from input_tensor_1
input_tensor2 = ov.Tensor(model.input().element_type, [1, 200])
# ... write values to input_tensor_2
infer_request.infer([input_tensor2])
# No need to call infer_request.get_output_tensor() again
# output_tensor queried after the first inference call above is valid here.
# But it may not be true for the memory underneath as shape changed, so re-take an output data:
data2 = output_tensor.data[:]
#! [set_input_tensor]
infer_request = executable.create_infer_request()
#! [get_input_tensor]
# Get the tensor, shape is not initialized
input_tensor = infer_request.get_input_tensor()
# Set shape is required
input_tensor.shape = [1, 128]
# ... write values to input_tensor
infer_request.infer()
output_tensor = infer_request.get_output_tensor()
data1 = output_tensor.data[:]
# The second inference call, repeat steps:
# Set a new shape, may reallocate tensor memory
input_tensor.shape = [1, 200]
# ... write values to input_tensor
infer_request.infer()
data2 = output_tensor.data[:]
#! [get_input_tensor]