IntenseWebs/openvino
3
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Clarification of Low Latency Transformation and State API documentation (#4771)

Browse Source 
* Assign/ReadValue, LowLatency and StateAPI clarifications

* Apply suggestions from code review: spelling mistakes

Co-authored-by: Anastasiya Ageeva <anastasiya.ageeva@intel.com>

* fixed wording

* Resolve review remarks

Co-authored-by: Anastasiya Ageeva <anastasiya.ageeva@intel.com>
This commit is contained in:
Ivan TikhonovandAnastasiya Ageeva authored and GitHub committed 2021-03-23 12:40:42 +03:00
1 parent 910e83b2a2
commit e877272eca
1 file changed
+113 -52
Download Patch File Download Diff File Expand all files Collapse all files
docs/IE_DG/network_state_intro.md
+113 -52
View File
@@ -23,7 +23,7 @@ The section additionally provides small examples of stateful network and code to
## OpenVINO state representation
OpenVINO contains special abstraction variable to represent state in a network. There are two operations to work with state:
OpenVINO contains a special abstraction `Variable` to represent a state in a network. There are two operations to work with the state:
* `Assign` to save value in state
* `ReadValue` to read value saved on previous iteration
@@ -45,7 +45,7 @@ The `bin` file for this graph should contain float 0 in binary form. Content of
<?xml version="1.0" ?>
<net name="summator" version="10">
<layers>
<layer id="0" name="init_value" type="Const" version="opset5">
<layer id="0" name="init_value" type="Const" version="opset6">
<data element_type="f32" offset="0" shape="1,1" size="4"/>
<output>
<port id="1" precision="FP32">
@@ -54,7 +54,7 @@ The `bin` file for this graph should contain float 0 in binary form. Content of
</port>
</output>
</layer>
<layer id="1" name="read" type="ReadValue" version="opset5">
<layer id="1" name="read" type="ReadValue" version="opset6">
<data variable_id="id"/>
<input>
<port id="0">
@@ -69,7 +69,7 @@ The `bin` file for this graph should contain float 0 in binary form. Content of
</port>
</output>
</layer>
<layer id="2" name="input" type="Parameter" version="opset5">
<layer id="2" name="input" type="Parameter" version="opset6">
<data element_type="f32" shape="1,1"/>
<output>
<port id="0" precision="FP32">
@@ -78,7 +78,7 @@ The `bin` file for this graph should contain float 0 in binary form. Content of
</port>
</output>
</layer>
<layer id="3" name="add_sum" type="Add" version="opset5">
<layer id="3" name="add_sum" type="Add" version="opset6">
<input>
<port id="0">
<dim>1</dim>
@@ -96,7 +96,7 @@ The `bin` file for this graph should contain float 0 in binary form. Content of
</port>
</output>
</layer>
<layer id="4" name="save" type="Assign" version="opset5">
<layer id="4" name="save" type="Assign" version="opset6">
<data variable_id="id"/>
<input>
<port id="0">
@@ -105,7 +105,7 @@ The `bin` file for this graph should contain float 0 in binary form. Content of
</port>
</input>
</layer>
<layer id="10" name="add" type="Add" version="opset5">
<layer id="10" name="add" type="Add" version="opset6">
<data axis="1"/>
<input>
<port id="0">
@@ -124,7 +124,7 @@ The `bin` file for this graph should contain float 0 in binary form. Content of
</port>
</output>
</layer>
<layer id="5" name="output/sink_port_0" type="Result" version="opset5">
<layer id="5" name="output/sink_port_0" type="Result" version="opset6">
<input>
<port id="0">
<dim>1</dim>
@@ -153,14 +153,25 @@ The `bin` file for this graph should contain float 0 in binary form. Content of
### Example of creating model nGraph API
```cpp
auto arg = make_shared<op::Parameter>(element::f32, Shape{1, 1});
auto init_const = op::Constant::create(element::f32, Shape{1, 1}, {0});
auto read = make_shared<op::ReadValue>(init_const, "v0");
std::vector<shared_ptr<Node>> args = {arg, read};
auto add = make_shared<op::Add>(arg, read);
auto assign = make_shared<op::Assign>(add, "v0");
auto add2 = make_shared<op::Add>(add, read);
auto res = make_shared<op::Result>(add2);
#include <ngraph/opsets/opset6.hpp>
#include <ngraph/op/util/variable.hpp>
// ...
auto arg = make_shared<ngraph::opset6::Parameter>(element::f32, Shape{1, 1});
auto init_const = ngraph::opset6::Constant::create(element::f32, Shape{1, 1}, {0});
// The ReadValue/Assign operations must be used in pairs in the network.
// For each such a pair, its own variable object must be created.
const std::string variable_name("variable0");
auto variable = std::make_shared<ngraph::Variable>(VariableInfo{PartialShape::dynamic(), element::dynamic, variable_name});
// Creating ngraph::function
auto read = make_shared<ngraph::opset6::ReadValue>(init_const, variable);
std::vector<shared_ptr<ngraph::Node>> args = {arg, read};
auto add = make_shared<ngraph::opset6::Add>(arg, read);
auto assign = make_shared<ngraph::opset6::Assign>(add, variable);
auto add2 = make_shared<ngraph::opset6::Add>(add, read);
auto res = make_shared<ngraph::opset6::Result>(add2);
auto f = make_shared<Function>(ResultVector({res}), ParameterVector({arg}), SinkVector({assign}));
```
@@ -204,10 +215,12 @@ Decsriptions can be found in [Samples Overview](./Samples_Overview.md)
If the original framework does not have a special API for working with states, after importing the model, OpenVINO representation will not contain Assign/ReadValue layers. For example, if the original ONNX model contains RNN operations, IR will contain TensorIterator operations and the values will be obtained only after the execution of whole TensorIterator primitive, intermediate values from each iteration will not be available. To be able to work with these intermediate values of each iteration and receive them with a low latency after each infer request, a special LowLatency transformation was introduced.
LowLatency transformation changes the structure of the network containing [TensorIterator](../ops/infrastructure/TensorIterator_1.md) by adding the ability to work with state, inserting Assign/ReadValue layers as it is shown in the picture below.
LowLatency transformation changes the structure of the network containing [TensorIterator](../ops/infrastructure/TensorIterator_1.md) and [Loop](../ops/infrastructure/Loop_5.md) by adding the ability to work with the state, inserting the Assign/ReadValue layers as it is shown in the picture below.
![applying_low_latency_example](./img/applying_low_latency.png)
After applying the transformation, ReadValue operations can receive other operations as an input, as shown in the picture above. These inputs should set the initial value for initialization of ReadValue operations. However, such initialization is not supported in the current State API implementation. Input values are ignored and the initial values for the ReadValue operations are set to zeros unless otherwise specified by the user via [State API](#openvino-state-api).
### Steps to apply LowLatency transformation
1. Get CNNNetwork. Any way is acceptable:
@@ -216,7 +229,11 @@ LowLatency transformation changes the structure of the network containing [Tenso
* [from nGraph Function](../nGraph_DG/build_function.md)
2. [Reshape](ShapeInference) CNNNetwork network if necessary
**Necessary case:** the sequence_lengths dimention of input > 1, it means TensorIterator layer will have number_iterations > 1. We should reshape the inputs of the network to set sequence_dimension exactly to 1.
**Necessary case:** the sequence_lengths dimension of input > 1, it means the TensorIterator layer will have number_iterations > 1. We should reshape the inputs of the network to set sequence_dimension exactly to 1.
Usually, the following exception, which occurs after applying a transform when trying to infer the network in a plugin, indicates the need to apply reshape feature: `C++ exception with description "Function is incorrect. Assign and ReadValue operations must be used in pairs in the network."`
This means that there are several pairs of Assign/ReadValue operations with the same variable_id in the network, operations were inserted into each iteration of the TensorIterator.
```cpp
// Network before reshape: Parameter (name: X, shape: [2 (sequence_lengths), 1, 16]) -> TensorIterator (num_iteration = 2, axis = 0) -> ...
@@ -235,41 +252,85 @@ cnnNetwork.reshape({"X" : {1, 1, 16});
InferenceEngine::LowLatency(cnnNetwork);
```
**State naming rule:** a name of state is a concatenation of names: original TensorIterator operation, Parameter of the body, and additional suffix "variable_" + id (0-base indexing, new indexing for each TensorIterator), for example:
```
tensor_iterator_name = "TI_name"
body_parameter_name = "param_name"
state_name = "TI_name/param_name/variable_0"
```
4. [Use state API](#openvino-state-api)
### Known limitations
1. Parameters are directly connected to States (ReadValues).
Removing Parameters from `ngraph::Function` is not possible.
![low_latency_limitation_1](./img/low_latency_limitation_1.png)
**Current solution:** replace Parameter with Constant (freeze) with the value [0, 0, 0 … 0] via [ModelOptimizer CLI](../MO_DG/prepare_model/convert_model/Converting_Model_General.md) `--input` or `--freeze_placeholder_with_value`.
2. Non-reshapable network.
Value of shapes is hard-coded somewhere in the network.
![low_latency_limitation_2](./img/low_latency_limitation_2.png)
**Current solution:** trim non-reshapable layers via [ModelOptimizer CLI](../MO_DG/prepare_model/convert_model/Converting_Model_General.md) `--input`, `--output` or via nGraph.
**State naming rule:** a name of a state is a concatenation of names: original TensorIterator operation, Parameter of the body, and additional suffix "variable_" + id (0-base indexing, new indexing for each TensorIterator). You can use these rules to predict what the name of the inserted State will be after the transformation is applied. For example:
```cpp
// nGraph example:
auto func = cnnNetwork.getFunction();
auto new_const = std::make_shared<ngraph::opset5::Constant>(); // type, shape, value
for (const auto& node : func->get_ops()) {
if (node->get_friendly_name() == "name_of_non_reshapable_const") {
auto bad_const = std::dynamic_pointer_cast<ngraph::opset5::Constant>(node);
ngraph::replace_node(bad_const, new_const); // replace constant
// Precondition in ngraph::function.
// Created TensorIterator and Parameter in body of TensorIterator with names
std::string tensor_iterator_name = "TI_name"
std::string body_parameter_name = "param_name"
std::string idx = "0"; // it's a first variable in the network
// The State will be named "TI_name/param_name/variable_0"
auto state_name = tensor_iterator_name + "//" + body_parameter_name + "//" + "variable_" + idx;
InferenceEngine::CNNNetwork cnnNetwork = InferenceEngine::CNNNetwork{function};
InferenceEngine::LowLatency(cnnNetwork);
InferenceEngine::ExecutableNetwork executableNetwork = core->LoadNetwork(/*cnnNetwork, targetDevice, configuration*/);
// Try to find the Variable by name
auto states = executableNetwork.QueryState();
for (auto& state : states) {
auto name = state.GetName();
if (name == state_name) {
// some actions
}
}
```
4. Use state API. See sections [OpenVINO state API](#openvino-state-api), [Example of stateful network inference](#example-of-stateful-network-inference).
### Known limitations
1. Parameters connected directly to ReadValues (States) after the transformation is applied are not allowed.
Unnecessary parameters may remain on the graph after applying the transformation. The automatic handling of this case inside the transformation is not possible now. Such Parameters should be removed manually from `ngraph::Function` or replaced with a Constant.
![low_latency_limitation_1](./img/low_latency_limitation_1.png)
**Current solutions:**
* Replace Parameter with Constant (freeze) with the value [0, 0, 0 … 0] via [ModelOptimizer CLI](../MO_DG/prepare_model/convert_model/Converting_Model_General.md) `--input` or `--freeze_placeholder_with_value`.
* Use ngraph API to replace Parameter with Constant.
```cpp
// nGraph example. How to replace Parameter with Constant.
auto func = cnnNetwork.getFunction();
// Creating the new Constant with zero values.
auto new_const = std::make_shared<ngraph::opset6::Constant>( /*type, shape, std::vector with zeros*/ );
for (const auto& param : func->get_parameters()) {
// Trying to find the problematic Constant by name.
if (param->get_friendly_name() == "param_name") {
// Replacing the problematic Param with a Constant.
ngraph::replace_node(param, new_const);
// Removing problematic Parameter from ngraph::function
func->remove_parameter(param);
}
}
```
2. Unable to execute reshape precondition to apply the transformation correctly due to hardcoded values of shapes somewhere in the network.
Networks can be non-reshapable, the most common reason is that the value of shapes is hardcoded in the Constant somewhere in the network.
![low_latency_limitation_2](./img/low_latency_limitation_2.png)
**Current solution:** trim non-reshapable layers via [ModelOptimizer CLI](../MO_DG/prepare_model/convert_model/Converting_Model_General.md) `--input`, `--output`. For example, we can trim the Parameter and the problematic Constant in the picture above, using the following command line option:
`--input Reshape_layer_name`. We can also replace the problematic Constant using ngraph, as shown in the example below.
```cpp
// nGraph example. How to replace a Constant with hardcoded values of shapes in the network with another one with the new values.
// Assume we know which Constant (const_with_hardcoded_shape) prevents the reshape from being applied.
// Then we can find this Constant by name on the network and replace it with a new one with the correct shape.
auto func = cnnNetwork.getFunction();
// Creating the new Constant with a correct shape.
// For the example shown in the picture above, the new values of the Constant should be 1, 1, 10 instead of 1, 49, 10
auto new_const = std::make_shared<ngraph::opset6::Constant>( /*type, shape, value_with_correct_shape*/ );
for (const auto& node : func->get_ops()) {
// Trying to find the problematic Constant by name.
if (node->get_friendly_name() == "name_of_non_reshapable_const") {
auto const_with_hardcoded_shape = std::dynamic_pointer_cast<ngraph::opset6::Constant>(node);
// Replacing the problematic Constant with a new one. Do this for all the problematic Constants in the network, then
// you can apply the reshape feature.
ngraph::replace_node(const_with_hardcoded_shape, new_const);
}
}
```