IntenseWebs/openvino
3
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update Variable State doc (#16358)

Browse Source 
* Update Variable State doc

* Fixed build

* Try to fix build

* Remove error

* Fixed doc

* Fixed links

* Try to fix doc
This commit is contained in:
Ilya Churaev authored and GitHub committed 2023-03-20 10:14:06 +04:00
1 parent bc15596c9e
commit 2739a01d64
9 files changed
+148 -166
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/OV_Runtime_UG/lowlatency2.md
+1 -1
View File
@@ -90,7 +90,7 @@ Steps to Apply LowLatency2
}
4. Use state API. See the :ref:`OpenVINO state API <openvino-state-api>` and the `Example of stateful network inference <example-of-stateful-network-inference>` sections.
4. Use state API. See the :ref:`OpenVINO state API <openvino-state-api>` and the `Example of stateful network inference <example-of-stateful-model-inference>` sections.
Known Limitations
####################
docs/OV_Runtime_UG/lowlatency_deprecated.md
+1 -1
View File
@@ -72,7 +72,7 @@ Steps to Apply LowLatency
}
4. Use state API. See the :ref:`OpenVINO state API <openvino-state-api>` and the :ref:`Example of stateful network inference <example-of-stateful-network-inference>` sections.
4. Use state API. See the :ref:`OpenVINO state API <openvino-state-api>` and the :ref:`Example of stateful network inference <example-of-stateful-model-inference>` sections.
Known Limitations for the LowLatency
####################################
docs/OV_Runtime_UG/network_state_intro.md → docs/OV_Runtime_UG/model_state_intro.md
+24 -45
View File
@@ -1,4 +1,4 @@
# Stateful models {#openvino_docs_OV_UG_network_state_intro}
# Stateful models {#openvino_docs_OV_UG_model_state_intro}
@sphinxdirective
@@ -11,22 +11,22 @@
Several use cases require processing of data sequences. When length of a sequence is known and small enough,
it can be processed with RNN like networks that contain a cycle inside. However, in some cases (e.g., online speech recognition of time series
it can be processed with RNN like models that contain a cycle inside. However, in some cases (e.g., online speech recognition of time series
forecasting) length of data sequence is unknown. Then, data can be divided in small portions and processed step-by-step. The dependency
between data portions should be addressed. For that, networks save some data between inferences - a state. When one dependent sequence is over,
between data portions should be addressed. For that, models save some data between inferences - a state. When one dependent sequence is over,
a state should be reset to initial value and a new sequence can be started.
Several frameworks have special APIs for states in networks. For example, Keras has ``stateful`` - a special option for RNNs, that turns on saving a state between inferences. Kaldi contains special ``Offset`` specifier to define time offset in a network.
Several frameworks have special APIs for states in model. For example, Keras has ``stateful`` - a special option for RNNs, that turns on saving a state between inferences. Kaldi contains special ``Offset`` specifier to define time offset in a model.
OpenVINO also contains a special API to simplify work with networks with states. A state is automatically saved between inferences,
OpenVINO also contains a special API to simplify work with models with states. A state is automatically saved between inferences,
and there is a way to reset a state when needed. A state can also be read or set to some new value between inferences.
OpenVINO State Representation
#############################
OpenVINO contains the ``Variable``, a special abstraction to represent a state in a network. There are two operations: :doc:`Assign <openvino_docs_ops_infrastructure_Assign_3>` - to save a value in a state and :doc:`ReadValue <openvino_docs_ops_infrastructure_ReadValue_3>` - to read a value saved on previous iteration.
OpenVINO contains the ``Variable``, a special abstraction to represent a state in a model. There are two operations: :doc:`Assign <openvino_docs_ops_infrastructure_Assign_3>` - to save a value in a state and :doc:`ReadValue <openvino_docs_ops_infrastructure_ReadValue_3>` - to read a value saved on previous iteration.
To get a model with states ready for inference, convert a model from another framework to OpenVINO IR with Model Optimizer or create an nGraph function.
To get a model with states ready for inference, convert a model from another framework to OpenVINO IR with Model Optimizer or create an OpenVINO model.
(For more information, refer to the :doc:`Build OpenVINO Model section <openvino_docs_OV_UG_Model_Representation>`).
Below is the graph in both forms:
@@ -47,7 +47,7 @@ The ``bin`` file for this graph should contain ``float 0`` in binary form. The c
.. code-block:: xml
<?xml version="1.0" ?>
<net name="summator" version="10">
<net name="summator" version="11">
<layers>
<layer id="0" name="init_value" type="Const" version="opset6">
<data element_type="f32" offset="0" shape="1,1" size="4"/>
@@ -154,65 +154,44 @@ The ``bin`` file for this graph should contain ``float 0`` in binary form. The c
</net>
Example of Creating Model nGraph API
++++++++++++++++++++++++++++++++++++
Example of Creating Model OpenVINO API
++++++++++++++++++++++++++++++++++++++++
In the following example, the ``SinkVector`` is used to create the `ngraph::Function <classngraph.html#doxid-classngraph-1a14d7fe7c605267b52c145579e12d2a5f>`__. For a network with states, except inputs and outputs, the ``Assign`` nodes should also point to the ``Function`` to avoid deleting it during graph transformations. Use the constructor to do it, as shown in the example, or with the special ``add_sinks(const SinkVector& sinks)`` method. After deleting the node from the graph with the ``delete_sink()`` method, a sink can be deleted from ``ngraph::Function``.
.. code-block:: cpp
#include <ngraph/opsets/opset6.hpp>
#include <ngraph/op/util/variable.hpp>
// ...
auto arg = make_shared<ngraph::opset6::Parameter>(element::f32, Shape{1, 1});
auto init_const = ngraph::opset6::Constant::create(element::f32, Shape{1, 1}, {0});
// The ReadValue/Assign operations must be used in pairs in the network.
// For each such a pair, its own variable object must be created.
const std::string variable_name("variable0");
auto variable = std::make_shared<ngraph::Variable>(VariableInfo{PartialShape::dynamic(), element::dynamic, variable_name});
// Creating ngraph::function
auto read = make_shared<ngraph::opset6::ReadValue>(init_const, variable);
std::vector<shared_ptr<ngraph::Node>> args = {arg, read};
auto add = make_shared<ngraph::opset6::Add>(arg, read);
auto assign = make_shared<ngraph::opset6::Assign>(add, variable);
auto add2 = make_shared<ngraph::opset6::Add>(add, read);
auto res = make_shared<ngraph::opset6::Result>(add2);
auto f = make_shared<Function>(ResultVector({res}), ParameterVector({arg}), SinkVector({assign}));
In the following example, the ``SinkVector`` is used to create the ``ov::Model``. For a model with states, except inputs and outputs, the ``Assign`` nodes should also point to the ``Model`` to avoid deleting it during graph transformations. Use the constructor to do it, as shown in the example, or with the special ``add_sinks(const SinkVector& sinks)`` method. After deleting the node from the graph with the ``delete_sink()`` method, a sink can be deleted from ``ov::Model``.
.. doxygensnippet:: docs/snippets/ov_model_with_state_infer.cpp
:language: cpp
:fragment: [model_create]
.. _openvino-state-api:
OpenVINO State API
####################
Inference Engine has the ``InferRequest::QueryState`` method to get the list of states from a network and ``IVariableState`` interface to operate with states. Below is a brief description of methods and the example of how to use this interface.
OpenVINO has the ``InferRequest::query_state`` method to get the list of states from a model and ``ov::IVariableState`` interface to operate with states. Below is a brief description of methods and the example of how to use this interface.
* ``std::string GetName() const`` - returns the name (variable_id) of a corresponding Variable.
* ``void Reset()`` - resets a state to a default value.
* ``void SetState(Blob::Ptr newState)`` - sets a new value for a state.
* ``Blob::CPtr GetState() const`` - returns current value of state.
* ``std::string get_name() const`` - returns the name (variable_id) of a corresponding Variable.
* ``void reset()`` - resets a state to a default value.
* ``void set_state(const ov::Tensor& state)`` - sets a new value for a state.
* ``const ov::Tensor& get_state() const`` - returns current value of state.
.. _example-of-stateful-network-inference:
.. _example-of-stateful-model-inference:
Example of Stateful Network Inference
Example of Stateful Model Inference
#####################################
Based on the IR from the previous section, the example below demonstrates inference of two independent sequences of data. A state should be reset between these sequences.
One infer request and one thread will be used in this example. Using several threads is possible if there are several independent sequences. Then, each sequence can be processed in its own infer request. Inference of one sequence in several infer requests is not recommended. In one infer request, a state will be saved automatically between inferences, but if the first step is done in one infer request and the second in another, a state should be set in a new infer request manually (using the ``IVariableState::SetState`` method).
One infer request and one thread will be used in this example. Using several threads is possible if there are several independent sequences. Then, each sequence can be processed in its own infer request. Inference of one sequence in several infer requests is not recommended. In one infer request, a state will be saved automatically between inferences, but if the first step is done in one infer request and the second in another, a state should be set in a new infer request manually (using the ``ov::IVariableState::set_state`` method).
.. doxygensnippet:: docs/snippets/InferenceEngine_network_with_state_infer.cpp
.. doxygensnippet:: docs/snippets/ov_model_with_state_infer.cpp
:language: cpp
:fragment: [part1]
For more elaborate examples demonstrating how to work with networks with states,
For more elaborate examples demonstrating how to work with models with states,
refer to the speech sample and a demo in the :doc:`Samples Overview <openvino_docs_OV_UG_Samples_Overview>`.
LowLatency Transformations
docs/OV_Runtime_UG/openvino_intro.md
+1 -1
View File
@@ -13,7 +13,7 @@
openvino_docs_OV_UG_Working_with_devices
openvino_docs_OV_UG_ShapeInference
openvino_docs_OV_UG_DynamicShapes
openvino_docs_OV_UG_network_state_intro
openvino_docs_OV_UG_model_state_intro
@endsphinxdirective
docs/OV_Runtime_UG/supported_plugins/CPU.md
+1 -1
View File
@@ -279,7 +279,7 @@ Stateful Models
The CPU plugin supports stateful models without any limitations.
For details, see :doc:`stateful models guide <openvino_docs_OV_UG_network_state_intro>`.
For details, see :doc:`stateful models guide <openvino_docs_OV_UG_model_state_intro>`.
Supported Properties
###########################################################
docs/OV_Runtime_UG/supported_plugins/Device_Plugins.md
+1 -1
View File
@@ -48,7 +48,7 @@ The table below demonstrates support of key features by OpenVINO device plugins.
:doc:`Dynamic shapes <openvino_docs_OV_UG_DynamicShapes>` Yes Partial No No
:doc:`Import/Export <openvino_inference_engine_tools_compile_tool_README>` Yes No Yes No
:doc:`Preprocessing acceleration <openvino_docs_OV_UG_Preprocessing_Overview>` Yes Yes No Partial
:doc:`Stateful models <openvino_docs_OV_UG_network_state_intro>` Yes No Yes No
:doc:`Stateful models <openvino_docs_OV_UG_model_state_intro>` Yes No Yes No
:doc:`Extensibility <openvino_docs_Extensibility_UG_Intro>` Yes Yes No No
========================================================================================= =============== =============== =============== ========================
docs/OV_Runtime_UG/supported_plugins/GNA.md
+1 -1
View File
@@ -209,7 +209,7 @@ To compile a model, use either :doc:`compile Tool <openvino_inference_engine_too
Stateful Models
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
GNA plugin natively supports stateful models. For more details on such models, refer to the :doc:`Stateful models <openvino_docs_OV_UG_network_state_intro>`.
GNA plugin natively supports stateful models. For more details on such models, refer to the :doc:`Stateful models <openvino_docs_OV_UG_model_state_intro>`.
.. note::
docs/snippets/InferenceEngine_network_with_state_infer.cpp
-115
View File
@@ -1,115 +0,0 @@
// Copyright (C) 2018-2020 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include <iostream>
#include <ie_core.hpp>
using namespace InferenceEngine;
int main(int argc, char *argv[]) {
try {
// --------------------------- 1. Load inference engine -------------------------------------
std::cout << "Loading Inference Engine" << std::endl;
Core ie;
// 2. Read a model in OpenVINO Intermediate Representation (.xml and .bin files) or ONNX (.onnx file) format
std::cout << "Loading network files" << std::endl;
CNNNetwork network;
network = ie.ReadNetwork(std::string("c:\\work\\git\\github_dldt3\\openvino\\model-optimizer\\summator.xml"));
network.setBatchSize(1);
// 3. Load network to CPU
ExecutableNetwork executableNet = ie.LoadNetwork(network, "CPU");
// 4. Create Infer Request
InferRequest inferRequest = executableNet.CreateInferRequest();
// 5. Prepare inputs
ConstInputsDataMap cInputInfo = executableNet.GetInputsInfo();
std::vector<Blob::Ptr> ptrInputBlobs;
for (const auto& input : cInputInfo) {
ptrInputBlobs.push_back(inferRequest.GetBlob(input.first));
}
InputsDataMap inputInfo;
inputInfo = network.getInputsInfo();
for (auto &item : inputInfo) {
Precision inputPrecision = Precision::FP32;
item.second->setPrecision(inputPrecision);
}
// 6. Prepare outputs
std::vector<Blob::Ptr> ptrOutputBlobs;
ConstOutputsDataMap cOutputInfo = executableNet.GetOutputsInfo();
for (const auto& output : cOutputInfo) {
ptrOutputBlobs.push_back(inferRequest.GetBlob(output.first));
}
// 7. Initialize memory state before starting
for (auto &&state : inferRequest.QueryState()) {
state.Reset();
}
//! [part1]
// input data
std::vector<float> data = { 1,2,3,4,5,6};
// infer the first utterance
for (size_t next_input = 0; next_input < data.size()/2; next_input++) {
MemoryBlob::Ptr minput = as<MemoryBlob>(ptrInputBlobs[0]);
auto minputHolder = minput->wmap();
std::memcpy(minputHolder.as<void *>(),
&data[next_input],
sizeof(float));
inferRequest.Infer();
// check states
auto states = inferRequest.QueryState();
if (states.empty()) {
throw std::runtime_error("Queried states are empty");
}
auto mstate = as<MemoryBlob>(states[0].GetState());
if (mstate == nullptr) {
throw std::runtime_error("Can't cast state to MemoryBlob");
}
auto state_buf = mstate->rmap();
float * state =state_buf.as<float*>();
std::cout << state[0] << "\n";
}
// resetting state between utterances
std::cout<<"Reset state\n";
for (auto &&state : inferRequest.QueryState()) {
state.Reset();
}
// infer the second utterance
for (size_t next_input = data.size()/2; next_input < data.size(); next_input++) {
MemoryBlob::Ptr minput = as<MemoryBlob>(ptrInputBlobs[0]);
auto minputHolder = minput->wmap();
std::memcpy(minputHolder.as<void *>(),
&data[next_input],
sizeof(float));
inferRequest.Infer();
// check states
auto states = inferRequest.QueryState();
auto mstate = as<MemoryBlob>(states[0].GetState());
auto state_buf = mstate->rmap();
float * state =state_buf.as<float*>();
std::cout << state[0] << "\n";
}
//! [part1]
}
catch (const std::exception &error) {
std::cerr << error.what() << std::endl;
return 1;
}
catch (...) {
std::cerr << "Unknown/internal exception happened" << std::endl;
return 1;
}
std::cerr << "Execution successful" << std::endl;
return 0;
}
docs/snippets/ov_model_with_state_infer.cpp
+118
View File
@@ -0,0 +1,118 @@
// Copyright (C) 2018-2020 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include <iostream>
#include "openvino/op/util/variable.hpp"
#include "openvino/openvino.hpp"
#include "openvino/opsets/opset11.hpp"
int main(int argc, char* argv[]) {
try {
// --------------------------- 1. Load inference engine -------------------------------------
std::cout << "Loading OpenVINO" << std::endl;
ov::Core core;
//! [model_create]
auto arg = std::make_shared<ov::opset11::Parameter>(ov::element::f32, ov::Shape{1, 1});
auto init_const = ov::opset11::Constant::create(ov::element::f32, ov::Shape{1, 1}, {0});
// The ReadValue/Assign operations must be used in pairs in the model.
// For each such a pair, its own variable object must be created.
const std::string variable_name("variable0");
auto variable = std::make_shared<ov::op::util::Variable>(
ov::op::util::VariableInfo{ov::PartialShape::dynamic(), ov::element::dynamic, variable_name});
// Creating ov::Model
auto read = std::make_shared<ov::opset11::ReadValue>(init_const, variable);
std::vector<std::shared_ptr<ov::Node>> args = {arg, read};
auto add = std::make_shared<ov::opset11::Add>(arg, read);
auto assign = std::make_shared<ov::opset11::Assign>(add, variable);
auto add2 = std::make_shared<ov::opset11::Add>(add, read);
auto res = std::make_shared<ov::opset11::Result>(add2);
auto model =
std::make_shared<ov::Model>(ov::ResultVector({res}), ov::SinkVector({assign}), ov::ParameterVector({arg}));
//! [model_create]
// 2. Read a model in OpenVINO Intermediate Representation (.xml and .bin files) or ONNX (.onnx file) format
std::cout << "Loading network files" << std::endl;
// 3. Load network to CPU
ov::CompiledModel compiled_model = core.compile_model(model, "CPU");
// 4. Create Infer Request
ov::InferRequest infer_request = compiled_model.create_infer_request();
// 5. Prepare inputs
std::vector<ov::Tensor> input_tensors;
for (const auto& input : compiled_model.inputs()) {
input_tensors.emplace_back(infer_request.get_tensor(input));
}
// 6. Prepare outputs
std::vector<ov::Tensor> output_tensors;
for (const auto& output : compiled_model.outputs()) {
output_tensors.emplace_back(infer_request.get_tensor(output));
}
// 7. Initialize memory state before starting
for (auto&& state : infer_request.query_state()) {
state.reset();
}
//! [part1]
// input data
std::vector<float> data = {1, 2, 3, 4, 5, 6};
// infer the first utterance
for (size_t next_input = 0; next_input < data.size() / 2; next_input++) {
auto minput = input_tensors[0];
std::memcpy(minput.data(), &data[next_input], sizeof(float));
infer_request.infer();
// check states
auto states = infer_request.query_state();
if (states.empty()) {
throw std::runtime_error("Queried states are empty");
}
auto mstate = states[0].get_state();
if (!mstate) {
throw std::runtime_error("Can't cast state to MemoryBlob");
}
float* state = mstate.data<float>();
std::cout << state[0] << "\n";
}
// resetting state between utterances
std::cout << "Reset state\n";
for (auto&& state : infer_request.query_state()) {
state.reset();
}
// infer the second utterance
for (size_t next_input = data.size() / 2; next_input < data.size(); next_input++) {
auto minput = input_tensors[0];
std::memcpy(minput.data(), &data[next_input], sizeof(float));
infer_request.infer();
// check states
auto states = infer_request.query_state();
auto mstate = states[0].get_state();
float* state = mstate.data<float>();
std::cout << state[0] << "\n";
}
//! [part1]
} catch (const std::exception& error) {
std::cerr << error.what() << std::endl;
return 1;
} catch (...) {
std::cerr << "Unknown/internal exception happened" << std::endl;
return 1;
}
std::cerr << "Execution successful" << std::endl;
return 0;
}