[docs] python snippet for dynamic shapes (#10762)
* Create snipp * link python snipp with doc * fix docs * Apply suggestions from code review Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com> * Fix cpp comments Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>
This commit is contained in:
parent
4e0a740eb3
commit
97efdb5020
@ -42,7 +42,21 @@ To avoid the tricks mentioned in the previous section there is a way to directly
|
||||
This is achieved with the same reshape method that is used for alternating static shape of inputs.
|
||||
Dynamic dimensions are specified as `-1` or `ov::Dimension()` instead of a positive number used for static dimensions:
|
||||
|
||||
@snippet snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:reshape_undefined
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: C++
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp
|
||||
:language: cpp
|
||||
:fragment: [ov_dynamic_shapes:reshape_undefined]
|
||||
|
||||
.. tab:: Python
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.py
|
||||
:language: python
|
||||
:fragment: [reshape_undefined]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
To simplify the code, the examples assume that the model has a single input and single output.
|
||||
However, there are no limitations on the number of inputs and outputs to apply dynamic shapes.
|
||||
@ -66,7 +80,21 @@ Use this capability to save time on calling `reshape` method in the end applicat
|
||||
Besides marking a dimension just dynamic, you can also specify lower and/or upper bounds that define a range of allowed values for the dimension.
|
||||
Bounds are coded as arguments for `ov::Dimension`:
|
||||
|
||||
@snippet snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:reshape_bounds
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: C++
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp
|
||||
:language: cpp
|
||||
:fragment: [ov_dynamic_shapes:reshape_bounds]
|
||||
|
||||
.. tab:: Python
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.py
|
||||
:language: python
|
||||
:fragment: [reshape_bounds]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
Information about bounds gives opportunity for the inference plugin to apply additional optimizations.
|
||||
Using dynamic shapes assumes the plugins apply more loose optimization technique during model compilation
|
||||
@ -86,7 +114,21 @@ Preparing model with the reshape method was the first step.
|
||||
The second step is passing a tensor with an appropriate shape to infer request.
|
||||
This is similar to [regular steps](integrate_with_your_application.md), but now we can pass tensors with different shapes for the same executable model and even for the same inference request:
|
||||
|
||||
@snippet snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:set_input_tensor
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: C++
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp
|
||||
:language: cpp
|
||||
:fragment: [ov_dynamic_shapes:set_input_tensor]
|
||||
|
||||
.. tab:: Python
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.py
|
||||
:language: python
|
||||
:fragment: [set_input_tensor]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
In the example above `set_input_tensor` is used to specify input tensors.
|
||||
The real dimensions of the tensor is always static, because it is a concrete tensor and it doesn't have any dimension variations in contrast to model inputs.
|
||||
@ -97,7 +139,21 @@ Without doing that, the tensor returned by `get_input_tensor` is an empty tensor
|
||||
Setting shape for input tensor is required when the corresponding input has at least one dynamic dimension regardless of bounds information.
|
||||
The following example makes the same sequence of two infer request as the previous example but using `get_input_tensor` instead of `set_input_tensor`:
|
||||
|
||||
@snippet snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:get_input_tensor
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: C++
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp
|
||||
:language: cpp
|
||||
:fragment: [ov_dynamic_shapes:get_input_tensor]
|
||||
|
||||
.. tab:: Python
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.py
|
||||
:language: python
|
||||
:fragment: [get_input_tensor]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
### Dynamic Shapes in Outputs
|
||||
|
||||
@ -108,13 +164,41 @@ The same is true for other dimensions, like sequence length for NLP models or sp
|
||||
Whether or not output has dynamic dimensions can be examined by querying output partial shape after model read or reshape.
|
||||
The same is applicable for inputs. For example:
|
||||
|
||||
@snippet snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:print_dynamic
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: C++
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp
|
||||
:language: cpp
|
||||
:fragment: [ov_dynamic_shapes:print_dynamic]
|
||||
|
||||
.. tab:: Python
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.py
|
||||
:language: python
|
||||
:fragment: [print_dynamic]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
Appearing `?` or ranges like `1..10` means there are dynamic dimensions in corresponding inputs or outputs.
|
||||
|
||||
Or more programmatically:
|
||||
|
||||
@snippet snippets/ov_dynamic_shapes.cpp ov_dynamic_shapes:detect_dynamic
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: C++
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.cpp
|
||||
:language: cpp
|
||||
:fragment: [ov_dynamic_shapes:detect_dynamic]
|
||||
|
||||
.. tab:: Python
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_dynamic_shapes.py
|
||||
:language: python
|
||||
:fragment: [detect_dynamic]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
If at least one dynamic dimension exists in output of the model, shape of the corresponding output tensor will be set as the result of inference call.
|
||||
Before the first inference, memory for such a tensor is not allocated and has shape `[0]`.
|
||||
|
@ -23,10 +23,10 @@ model->reshape({{ov::Dimension(), ov::Dimension()}}); // {?,?}
|
||||
model->reshape({{-1, -1}}); // {?,?}
|
||||
//! [ov_dynamic_shapes:reshape_undefined]
|
||||
//! [ov_dynamic_shapes:reshape_bounds]
|
||||
// Both dimensions are dynamic, first may have size within 1..10 and the second is withing 8..512
|
||||
// Both dimensions are dynamic, first has a size within 1..10 and the second has a size within 8..512
|
||||
model->reshape({{ov::Dimension(1, 10), ov::Dimension(8, 512)}}); // {1..10,8..512}
|
||||
|
||||
// Both dimensions are dynamic, first doesn't have bounds, the second is in 8..512
|
||||
// Both dimensions are dynamic, first doesn't have bounds, the second is in the range of 8..512
|
||||
model->reshape({{-1, ov::Dimension(8, 512)}}); // {?,8..512}
|
||||
//! [ov_dynamic_shapes:reshape_bounds]
|
||||
}
|
||||
@ -69,12 +69,12 @@ auto infer_request = executable.create_infer_request();
|
||||
//! [ov_dynamic_shapes:set_input_tensor]
|
||||
// The first inference call
|
||||
|
||||
// Create tensor compatible to the model input
|
||||
// Shape {1, 128} is compatible to any reshape statements made in previous examples
|
||||
// Create tensor compatible with the model input
|
||||
// Shape {1, 128} is compatible with any reshape statements made in previous examples
|
||||
auto input_tensor_1 = ov::Tensor(model->input().get_element_type(), {1, 128});
|
||||
// ... write values to input_tensor_1
|
||||
|
||||
// Set the tensor as a model input within infer request
|
||||
// Set the tensor as an input for the infer request
|
||||
infer_request.set_input_tensor(input_tensor_1);
|
||||
|
||||
// Do the inference
|
||||
|
139
docs/snippets/ov_dynamic_shapes.py
Normal file
139
docs/snippets/ov_dynamic_shapes.py
Normal file
@ -0,0 +1,139 @@
|
||||
# Copyright (C) 2018-2022 Intel Corporation
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import numpy as np
|
||||
#! [import]
|
||||
import openvino.runtime as ov
|
||||
#! [import]
|
||||
|
||||
#! [reshape_undefined]
|
||||
core = ov.Core()
|
||||
model = core.read_model("model.xml")
|
||||
|
||||
# Set one static dimension (= 1) and another dynamic dimension (= Dimension())
|
||||
model.reshape([1, ov.Dimension()])
|
||||
|
||||
# The same as above
|
||||
model.reshape([1, -1])
|
||||
|
||||
# The same as above
|
||||
model.reshape("1, ?")
|
||||
|
||||
# Or set both dimensions as dynamic if both are going to be changed dynamically
|
||||
model.reshape([ov.Dimension(), ov.Dimension()])
|
||||
|
||||
# The same as above
|
||||
model.reshape([-1, -1])
|
||||
|
||||
# The same as above
|
||||
model.reshape("?, ?")
|
||||
#! [reshape_undefined]
|
||||
|
||||
#! [reshape_bounds]
|
||||
# Both dimensions are dynamic, first has a size within 1..10 and the second has a size within 8..512
|
||||
model.reshape([ov.Dimension(1, 10), ov.Dimension(8, 512)])
|
||||
|
||||
# The same as above
|
||||
model.reshape([(1, 10), (8, 512)])
|
||||
|
||||
# The same as above
|
||||
model.reshape("1..10, 8..512")
|
||||
|
||||
# Both dimensions are dynamic, first doesn't have bounds, the second is in the range of 8..512
|
||||
model.reshape([-1, (8, 512)])
|
||||
#! [reshape_bounds]
|
||||
|
||||
model = core.read_model("model.xml")
|
||||
|
||||
#! [print_dynamic]
|
||||
# Print output partial shape
|
||||
print(model.output().partial_shape)
|
||||
|
||||
# Print input partial shape
|
||||
print(model.input().partial_shape)
|
||||
#! [print_dynamic]
|
||||
|
||||
#! [detect_dynamic]
|
||||
model = core.read_model("model.xml")
|
||||
|
||||
if model.input(0).partial_shape.is_dynamic():
|
||||
# input is dynamic
|
||||
pass
|
||||
|
||||
if model.output(0).partial_shape.is_dynamic():
|
||||
# output is dynamic
|
||||
pass
|
||||
|
||||
if model.output(0).partial_shape[1].is_dynamic():
|
||||
# 1-st dimension of output is dynamic
|
||||
pass
|
||||
#! [detect_dynamic]
|
||||
|
||||
executable = core.compile_model(model)
|
||||
infer_request = executable.create_infer_request()
|
||||
|
||||
#! [set_input_tensor]
|
||||
# The first inference call
|
||||
|
||||
# Create tensor compatible to the model input
|
||||
# Shape {1, 128} is compatible with any reshape statements made in previous examples
|
||||
input_tensor1 = ov.Tensor(model.input().element_type, [1, 128])
|
||||
# ... write values to input_tensor_1
|
||||
|
||||
# Set the tensor as an input for the infer request
|
||||
infer_request.set_input_tensor(input_tensor1)
|
||||
|
||||
# Do the inference
|
||||
infer_request.infer()
|
||||
|
||||
# Or pass a tensor in infer to set the tensor as a model input and make the inference
|
||||
infer_request.infer([input_tensor1])
|
||||
|
||||
# Or pass the numpy array to set inputs of the infer request
|
||||
input_data = np.ones(shape=[1, 128])
|
||||
infer_request.infer([input_data])
|
||||
|
||||
# Retrieve a tensor representing the output data
|
||||
output_tensor = infer_request.get_output_tensor()
|
||||
|
||||
# Copy data from tensor to numpy array
|
||||
data1 = output_tensor.data[:]
|
||||
|
||||
# The second inference call, repeat steps:
|
||||
|
||||
# Create another tensor (if the previous one cannot be utilized)
|
||||
# Notice, the shape is different from input_tensor_1
|
||||
input_tensor2 = ov.Tensor(model.input().element_type, [1, 200])
|
||||
# ... write values to input_tensor_2
|
||||
|
||||
infer_request.infer([input_tensor2])
|
||||
|
||||
# No need to call infer_request.get_output_tensor() again
|
||||
# output_tensor queried after the first inference call above is valid here.
|
||||
# But it may not be true for the memory underneath as shape changed, so re-take an output data:
|
||||
data2 = output_tensor.data[:]
|
||||
#! [set_input_tensor]
|
||||
|
||||
infer_request = executable.create_infer_request()
|
||||
|
||||
#! [get_input_tensor]
|
||||
# Get the tensor, shape is not initialized
|
||||
input_tensor = infer_request.get_input_tensor()
|
||||
|
||||
# Set shape is required
|
||||
input_tensor.shape = [1, 128]
|
||||
# ... write values to input_tensor
|
||||
|
||||
infer_request.infer()
|
||||
output_tensor = infer_request.get_output_tensor()
|
||||
data1 = output_tensor.data[:]
|
||||
|
||||
# The second inference call, repeat steps:
|
||||
|
||||
# Set a new shape, may reallocate tensor memory
|
||||
input_tensor.shape = [1, 200]
|
||||
# ... write values to input_tensor
|
||||
|
||||
infer_request.infer()
|
||||
data2 = output_tensor.data[:]
|
||||
#! [get_input_tensor]
|
Loading…
Reference in New Issue
Block a user