First draft of nGraph documentation (#2271)
* First draft of nGraph documentation * updated according to review comments * Updated * Reviewed the nGraph Transformation section, added missing images * Update nGraph_dg.md * Delete python_api.md Removed since there is already the nGraph_Python_API.md document with a comprehensive overview. Co-authored-by: Andrey Zaytsev <andrey.zaytsev@intel.com> Co-authored-by: CCR\avladimi <anastasiya.ageeva@intel.com>
This commit is contained in:
Ilya Churaev
GitHub
@@ -22,7 +22,7 @@ The OpenVINO™ toolkit includes the following components:
|
||||
TensorFlow*, MXNet*, Kaldi*, ONNX* models.
|
||||
- [Deep Learning Inference Engine](inference_engine_intro.md) — A unified API to allow high performance inference on many hardware types
|
||||
including Intel® CPU, Intel® Processor Graphics, Intel® FPGA, Intel® Neural Compute Stick 2.
|
||||
- [nGraph](nGraph_Flow.md) — graph representation and manipulation engine which is used to represent a model inside Inference Engine and allows the run-time model construction without using Model Optimizer.
|
||||
- [nGraph](../nGraph_DG/nGraph_dg.md) — graph representation and manipulation engine which is used to represent a model inside Inference Engine and allows the run-time model construction without using Model Optimizer.
|
||||
* [OpenCV](https://docs.opencv.org/) — OpenCV* community version compiled for Intel® hardware.
|
||||
Includes PVL libraries for computer vision.
|
||||
* Drivers and runtimes for OpenCL™ version 2.1
|
||||
@@ -48,8 +48,6 @@ inference of a pre-trained and optimized deep learning model and a set of sample
|
||||
|
||||
* [Introduction to Inference Engine](inference_engine_intro.md)
|
||||
|
||||
* [Introduction to nGraph Flow](nGraph_Flow.md)
|
||||
|
||||
* [Understanding Inference Engine Memory Primitives](Memory_primitives.md)
|
||||
|
||||
* [Introduction to Inference Engine Device Query API](InferenceEngine_QueryAPI.md)
|
||||
|
||||
@@ -1,32 +0,0 @@
|
||||
# Graph Debug Capabilities {#openvino_docs_IE_DG_Graph_debug_capabilities}
|
||||
|
||||
Inference Engine supports two different objects for a graph representation: the nGraph function and
|
||||
CNNNetwork. Both representations provide an API to get detailed information about the graph structure.
|
||||
|
||||
## nGraph Function
|
||||
|
||||
To receive additional messages about applied graph modifications, rebuild the nGraph library with
|
||||
the `-DNGRAPH_DEBUG_ENABLE=ON` option.
|
||||
|
||||
To visualize the nGraph function to the xDot format or to an image file, use the
|
||||
`ngraph::pass::VisualizeTree` graph transformation pass:
|
||||
```cpp
|
||||
#include <ngraph/pass/visualize_tree.hpp>
|
||||
|
||||
std::shared_ptr<ngraph::Function> nGraph;
|
||||
...
|
||||
ngraph::pass::VisualizeTree("after.png").run_on_function(nGraph); // Visualize the nGraph function to an image
|
||||
```
|
||||
|
||||
## CNNNetwork
|
||||
|
||||
To serialize the CNNNetwork to the Inference Engine Intermediate Representation (IR) format, use the
|
||||
`CNNNetwork::serialize(...)` method:
|
||||
```cpp
|
||||
std::shared_ptr<ngraph::Function> nGraph;
|
||||
...
|
||||
CNNNetwork network(nGraph);
|
||||
network.serialize("test_ir.xml", "test_ir.bin");
|
||||
```
|
||||
> **NOTE**: CNNNetwork created from the nGraph function might differ from the original nGraph
|
||||
> function because the Inference Engine applies some graph transformation.
|
||||
@@ -269,7 +269,7 @@ build/ - build directory
|
||||
```
|
||||
|
||||
2. **Include Inference Engine, nGraph and OpenCV libraries** in `project/CMakeLists.txt`
|
||||
[OpenCV](https://docs.opencv.org/master/db/df5/tutorial_linux_gcc_cmake.html) integration is needed mostly for pre-processing input data and ngraph for more complex applications using [ngraph API](nGraph_Flow.md).
|
||||
[OpenCV](https://docs.opencv.org/master/db/df5/tutorial_linux_gcc_cmake.html) integration is needed mostly for pre-processing input data and ngraph for more complex applications using [ngraph API](../nGraph_DG/nGraph_dg.md).
|
||||
``` cmake
|
||||
cmake_minimum_required(VERSION 3.0.0)
|
||||
project(project_name)
|
||||
|
||||
@@ -94,7 +94,7 @@ Refer to a dedicated description about [Intermediate Representation and Operatio
|
||||
OpenVINO toolkit is powered by nGraph capabilities for Graph construction API, Graph transformation engine and Reshape.
|
||||
nGraph Function is used as an intermediate representation for a model in the run-time underneath the CNNNetwork API.
|
||||
The conventional representation for CNNNetwork is still available if requested for backward compatibility when some conventional API methods are used.
|
||||
Please refer to the [Overview of nGraph Flow](nGraph_Flow.md) describing the details of nGraph integration into the Inference Engine and co-existence with the conventional representation.
|
||||
Please refer to the [Overview of nGraph](../nGraph_DG/nGraph_dg.md) describing the details of nGraph representation.
|
||||
|
||||
## Inference Engine <a name = "IE"></a>
|
||||
|
||||
|
||||
@@ -61,7 +61,7 @@ Refer to the sections below for details.
|
||||
> ```
|
||||
|
||||
Once you create the `ng_function`, you can use it to run computation on the Inference Engine.
|
||||
As it was shown in [Build a Model with nGraph Library](nGraphTutorial.md), `std::shared_ptr<ngraph::Function>` can be transformed into a `CNNNetwork`.
|
||||
As it was shown in [Build a Model with nGraph Library](../nGraph_DG/build_function.md), `std::shared_ptr<ngraph::Function>` can be transformed into a `CNNNetwork`.
|
||||
|
||||
|
||||
### <a name="stream">Stream as Input</a>
|
||||
@@ -98,4 +98,3 @@ const std::shared_ptr<ngraph::Function> ng_function = ngraph::onnx_import::impor
|
||||
|
||||
[onnx_header]: https://github.com/NervanaSystems/ngraph/blob/master/src/ngraph/frontend/onnx_import/onnx.hpp
|
||||
[onnx_model_zoo]: https://github.com/onnx/models
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ Using Shape Inference {#openvino_docs_IE_DG_ShapeInference}
|
||||
Inference Engine takes three kinds of a model description as an input, which are converted into an `InferenceEngine::CNNNetwork` object:
|
||||
1. [Intermediate Representation (IR)](../MO_DG/IR_and_opsets.md) through `InferenceEngine::Core::ReadNetwork`
|
||||
2. [ONNX model](../IE_DG/OnnxImporterTutorial.md) through `InferenceEngine::Core::ReadNetwork`
|
||||
3. [nGraph::Function](../IE_DG/nGraph_Flow.md) through the constructor of `InferenceEngine::CNNNetwork`
|
||||
3. [nGraph::Function](../nGraph_DG/nGraph_dg.md) through the constructor of `InferenceEngine::CNNNetwork`
|
||||
|
||||
`InferenceEngine::CNNNetwork` keeps an `ngraph::Function` object with the model description internally.
|
||||
The object should have fully defined input shapes to be successfully loaded to the Inference Engine plugins.
|
||||
@@ -51,7 +51,7 @@ Do not use runtime reshaping methods simultaneously, especially do not call the
|
||||
The `InferenceEngine::CNNNetwork::setBatchSize` method causes irreversible conversion of the internal model representation into the legacy model representation.
|
||||
The method does not use nGraph for shape inference which leads to reduced reshape opportunities and may affect the performance of the model.
|
||||
|
||||
There are other approaches to reshape the model during the stage of <a href="_docs_MO_DG_prepare_model_convert_model_Converting_Model_General.html#when_to_specify_input_shapes">IR generation</a> or [nGraph::Function creation](../IE_DG/nGraphTutorial.md).
|
||||
There are other approaches to reshape the model during the stage of <a href="_docs_MO_DG_prepare_model_convert_model_Converting_Model_General.html#when_to_specify_input_shapes">IR generation</a> or [nGraph::Function creation](../nGraph_DG/build_function.md).
|
||||
|
||||
Practically, some models are not ready to be reshaped. In this case, a new input shape cannot be set with the Model Optimizer or the `InferenceEngine::CNNNetwork::reshape` method.
|
||||
|
||||
|
||||
@@ -1,64 +0,0 @@
|
||||
# Build a Model with nGraph Library {#openvino_docs_IE_DG_nGraphTutorial}
|
||||
|
||||
This section illustrates how to construct an nGraph function
|
||||
composed of operations from the `opset3` namespace. Once created,
|
||||
it can wrap into a `CNNNetwork`, creating utility for data scientists
|
||||
or app developers to define a deep-learning model in a neutral way
|
||||
that does not depend on existing Deep Learning (DL) frameworks.
|
||||
|
||||
Operation Set `opsetX` integrates a list of nGraph pre-compiled operations that work
|
||||
for this purpose. In other words, `opsetX` defines a set of operations for building a graph.
|
||||
|
||||
For a complete list of operation sets supported by Inference Engine, see [Available Operations Sets](../ops/opset.md).
|
||||
|
||||
To add custom nGraph operations to an existing `CNNNetwork`, see
|
||||
the [Add Custom nGraph Operations](Extensibility_DG/Intro.md) document.
|
||||
|
||||
Now that you can build graphs with anything from the `opset3` definition, some
|
||||
parameters for shape-relevant (or shape-specific) inputs can be added. The
|
||||
following code prepares a graph for shape-relevant parameters.
|
||||
|
||||
> **NOTE**: `validate_nodes_and_infer_types(ops)` must be included for partial shape inference.
|
||||
|
||||
```cpp
|
||||
#include "ngraph/opsets/opset.hpp"
|
||||
#include "ngraph/opsets/opset3.hpp"
|
||||
|
||||
using namespace std;
|
||||
using namespace ngraph;
|
||||
|
||||
auto arg0 = make_shared<opset3::Parameter>(element::f32, Shape{7});
|
||||
auto arg1 = make_shared<opset3::Parameter>(element::f32, Shape{7});
|
||||
// Create an 'Add' operation with two inputs 'arg0' and 'arg1'
|
||||
auto add0 = make_shared<opset3::Add>(arg0, arg1);
|
||||
auto abs0 = make_shared<opset3::Abs>(add0);
|
||||
// Create a node whose inputs/attributes will be specified later
|
||||
auto acos0 = make_shared<opset3::Acos>();
|
||||
// Create a node using opset factories
|
||||
auto add1 = shared_ptr<Node>(get_opset3().create("Add"));
|
||||
// Set inputs to nodes explicitly
|
||||
acos0->set_argument(0, add0);
|
||||
add1->set_argument(0, acos0);
|
||||
add1->set_argument(1, abs0);
|
||||
|
||||
// Run shape inference on the nodes
|
||||
NodeVector ops{arg0, arg1, add0, abs0, acos0, add1};
|
||||
validate_nodes_and_infer_types(ops);
|
||||
|
||||
// Create a graph with one output (add1) and four inputs (arg0, arg1)
|
||||
auto ng_function = make_shared<Function>(OutputVector{add1}, ParameterVector{arg0, arg1});
|
||||
|
||||
```
|
||||
|
||||
To wrap it into a CNNNetwork, use:
|
||||
```cpp
|
||||
CNNNetwork net (ng_function);
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
* [Available Operation Sets](../ops/opset.md)
|
||||
* [Operation Set `opset1` Specification](../ops/opset1.md)
|
||||
* [Operation Set `opset2` Specification](../ops/opset2.md)
|
||||
* [Operation Set `opset3` Specification](../ops/opset3.md)
|
||||
* [Inference Engine Extensibility Developer Guide](Extensibility_DG/Intro.md)
|
||||
@@ -1,134 +0,0 @@
|
||||
# Introduction to nGraph Flow in Inference Engine {#openvino_docs_IE_DG_nGraph_Flow}
|
||||
|
||||
## New Run-Time Intermediate Representation (IR): nGraph
|
||||
|
||||
Starting from the OpenVINO™ release 2020.1, the Inference Engine integrates the
|
||||
nGraph Core.
|
||||
That implies that the Inference Engine uses a new way to represent a model in run time underneath of
|
||||
the conventional `CNNNetwork` API, which is an instance of `ngraph::Function`.
|
||||
|
||||
Besides the representation update, nGraph integration resulted in the following changes and new features:
|
||||
|
||||
1. New operations sets. When operations from the nGraph Core were combined with conventional layers
|
||||
from `CNNNetwork`, there were created a [new sets of operations called `opset1`, `opset2` and etc.](../ops/opset.md),
|
||||
which covered both interfaces except several not very important cases.
|
||||
Operations from `opset3` are generated by the Model Optimizer and are accepted in the Inference Engine.
|
||||
|
||||
2. New version approach that attaches a version to each operation rather than to the entire IR file format.
|
||||
IR is still versioned but has a different meaning. For details, see [Deep Learning Network Intermediate Representation and Operation Sets in OpenVINO™](../MO_DG/IR_and_opsets.md).
|
||||
|
||||
3. Creating models in run-time without loading IR from an xml/binary file. You can enable it by creating
|
||||
`ngraph::Function` passing it to `CNNNetwork`.
|
||||
|
||||
4. Run-time reshape capability and constant folding are implemented through the nGraph code for more operations compared to previous releases.
|
||||
As a result, more models can be reshaped. For details, see the [dedicated guide about the reshape capability](ShapeInference.md).
|
||||
|
||||
5. Loading model from ONNX format without converting it to the Inference Engine IR.
|
||||
|
||||
The conventional flow that is not based on nGraph is still available.
|
||||
The complete picture of co-existence of legacy and new flows is presented below.
|
||||
The rest of the document describes the coexistence of legacy and new flows showed in the picture below:
|
||||
|
||||
|
||||
|
||||
|
||||
## Read the Intermediate Representation to `CNNNetwork`
|
||||
|
||||
As the new operation set is introduced, the Model Optimizer generates the IR version 10 using the new operations by default.
|
||||
Each layer generated in the IR has a semantics matching to the corresponding operation from the nGraph namespaces `opset1`, `opset2` etc.
|
||||
The IR version 10 automatically triggers the nGraph flow inside the Inference Engine.
|
||||
When such IR is read in an application, the Inference Engine IR reader produces `CNNNetwork` that encapsulates the `ngraph::Function` instance underneath.
|
||||
Thus the OpenVINO IR becomes a new serialization format for the nGraph IR, and it can be deserialized reading the `CNNNetwork`.
|
||||
|
||||
> **IMPORTANT**: Conventional interfaces are used (`CNNNetwork`, the reader), so no changes required in most applications.
|
||||
|
||||
> **NOTE**: While you still can use old APIs, there is an independent process of continuous improvements in the Inference Engine API.
|
||||
> These changes are independent of nGraph integration and do not enable or disable new features.
|
||||
|
||||
Interpretation of the IR version 10 differs from the old IR version.
|
||||
Besides having a different operations set, the IR version 10 ignores the shapes and data types assigned to the ports in an XML file.
|
||||
Both shapes and types are reinferred while loading to the Inference Engine using the nGraph shape and type propagation function that is a part of each nGraph operation.
|
||||
|
||||
### Legacy IR Versions
|
||||
|
||||
Starting from the OpenVINO™ release 2021.1 you cannot read IR version 7 and lower in the Inference Engine.
|
||||
|
||||
## Build a Model in the Application
|
||||
|
||||
Alternative method to feed the Inference Engine with a model is to create the model in the run time.
|
||||
It is achieved by creation of the `ngraph::Function` construction using nGraph operation classes and optionally user-defined operations.
|
||||
For details, see [Add Custom nGraph Operations](Extensibility_DG/AddingNGraphOps.md) and [examples](nGraphTutorial.md).
|
||||
At this stage, the code is completely independent of the rest of the Inference Engine code and can be built separately.
|
||||
After you construct an instance of `ngraph::Function`, you can use it to create `CNNNetwork` by passing it to the new constructor for this class.
|
||||
|
||||
Initializing `CNNNetwork` from the nGraph Function means encapsulating the object and not converting it to a conventional representation.
|
||||
Going to low-level details, technically it is achieved by using another class for the `CNNNetwork` internals.
|
||||
The old representation that is used for former versions of IR before version 10 uses `CNNNetworkImpl`.
|
||||
The new representation that is built around nGraph uses `CNNNetworkNGraphImpl`.
|
||||
|
||||
|
||||
|
||||
## Automatic Conversion to the Old Representation
|
||||
|
||||
The old representation is still required in the cases listed below.
|
||||
When old representation is required, the conversion from the `ngraph::Function` to the old representation is called automatically.
|
||||
The following methods lead to the automatic conversion:
|
||||
|
||||
1. Using the old API, which is expected to produce an old representation. Guaranteed to be read-only. Once you call such a method, the original nGraph representation is preserved and continues to be used in the successive calls.
|
||||
|
||||
1.1. `CNNNetwork::serialize`. Dumps the old representation after automatically called conversion. Cannot be used to dump IR V10. For details, see [Graph Debug Capabilities](Graph_debug_capabilities.md).
|
||||
|
||||
2. Calling `CNNNetwork` methods that modify the model. After that nGraph representation is lost and cannot be used afterwards.
|
||||
|
||||
1.1. `CNNNetwork::addLayer`
|
||||
|
||||
1.2. CNNNetwork::setBatchSize. Still implemented through old logic for backward compatibility without using nGraph capabilities.
|
||||
For details, see [Using Shape Inference](ShapeInference.md).
|
||||
|
||||
3. Using methods that return objects inside an old representation.
|
||||
Using these methods does not mean modification of the model, but you are not limited by the API to make read-only changes.
|
||||
These methods should be used in the read-only mode with respect to a model representation.
|
||||
If the model is changed, for example attribute of some layer is changed or layers are reconnected, the modification is lost whenever any method that uses nGraph is called, including methods inside plugins like CNNNetwork::reshape.
|
||||
It is hard to predict whether the nGraph function is used in a plugin or other methods of CNNNetworks, so modifying a network using the following methods is *strongly not recommended*.
|
||||
This is an important limitation that is introduced for the old API calls listed below:
|
||||
|
||||
1.1. `Data::getInputTo`
|
||||
|
||||
1.2. `Data::getCreatorLayer`
|
||||
|
||||
1.3. `CNNNetwork::getLayerByName`
|
||||
|
||||
1.4. Iterating over `CNNLayer` objects in `CNNNetwork`: `CNNNetwork::begin`, `details::CNNNetworkIterator` class.
|
||||
|
||||
4. Using a conventional plugin that accepts the old representation only.
|
||||
|
||||
Though the conversion is always a one-way process, which means there is no method to convert back, there are important caveats.
|
||||
|
||||
In the cases [1] and [3], both representations are held underneath and you should use the old representation in the read-only mode only from the caller side.
|
||||
It is hard to track from the Inference Engine side whether the API is used in the read-only mode or for modification of the model.
|
||||
|
||||
That is why when using potentially modifying methods listed in section [3] above, you should not modify the model via those methods.
|
||||
Use a direct manipulation of the nGraph function instead.
|
||||
|
||||
## Conversion Function
|
||||
|
||||
Inference Engine implements the conversion function that is used when the nGraph function is transformed to the old `CNNNetworkImpl` representation.
|
||||
This conversion function is hidden and you cannot call it directly from the application.
|
||||
Nevertheless, it is an important component of the model transformation pipeline in the Inference Engine.
|
||||
Some issues of models may be caught during the conversion process in this function.
|
||||
Exceptions are thrown in this function, and you should know what this function does to find a root cause.
|
||||
|
||||
The conversion function performs the following steps:
|
||||
|
||||
1. Convert and decompose some operations as the first step of the nGraph function preparation for optimization.
|
||||
Reduce operation set to easily optimize it at the next stages.
|
||||
For example, decomposing of BatchNormInference happens at this stage.
|
||||
|
||||
2. Optimizing transformations that usually happen in the Model Optimizer are called here, because the nGraph function is not always read from an already optimized IR.
|
||||
|
||||
3. Changing operation set from `opsetX` to legacy layer semantics described in the [Legacy Layers Catalog](../MO_DG/prepare_model/convert_model/Legacy_IR_Layers_Catalog_Spec.md).
|
||||
The model is still represented as the nGraph function at this stage, but the operation set is completely different.
|
||||
|
||||
4. One-to-one conversion of nGraph representation to the corresponding `CNNNetworkImpl` without changing its semantics.
|
||||
You can see the result of the conversion by calling the `CNNNetwork::serialize` method, which produces legacy IR semantics, which is not nGraph-based even if it is applied to `CNNNetwork` constructed from the nGraph Function.
|
||||
It may help in debugging, see [Graph Debug Capabilities](Graph_debug_capabilities.md) to view all options for dumping new and old IR representations.
|
||||
@@ -38,7 +38,7 @@ Detailed guides
|
||||
* [Build](@ref plugin_build) a plugin library using CMake\*
|
||||
* Plugin and its components [testing](@ref plugin_testing)
|
||||
* [Quantized networks](@ref quantized_networks)
|
||||
* [Writing ngraph transformations](@ref new_ngraph_transformation) guide
|
||||
* [Writing ngraph transformations](@ref ngraph_transformation) guide
|
||||
|
||||
API References
|
||||
-----------------------
|
||||
|
||||
@@ -1,455 +0,0 @@
|
||||
# Writing ngraph transformations {#new_ngraph_transformation}
|
||||
|
||||
This guide contains all necessary information that could help you to start writing nGraph transformations.
|
||||
|
||||
First of all before writing transformation make sure that there is no transformation with the same functionality
|
||||
in [Transformation Library](group__ie__transformation__api.html). To start writing transformation it's good to know
|
||||
how [Transformation Library](group__ie__transformation__api.html) is structured, how transformations are organized
|
||||
and where to put your transformation code.
|
||||
|
||||
Let's start from reviewing transformations library structure.
|
||||
Transformations library is independent from InferenceEngine target library named as `inference_engine_transformations`
|
||||
and located in `inference-engine/src/transformations` directory.
|
||||
|
||||
Transformations root directory contains two folders:
|
||||
1. ngraph_ops - legacy opset operations needed for nGraph to CNNNetwork conversion.
|
||||
> **Note**: this operation are prohibited to use inside new plugins until they are not moved to separate directory with allowed operations.
|
||||
2. transformations - includes all transformations, utils, runtime info attributes and pass managers.
|
||||
> **Note**: do not use transformation that belongs to `ngraph::pass::ConvertOpSet1ToLegacy` transformations until they are not moved to separate directory with allowed transformations.
|
||||
|
||||
Transformation flow in transformation library has several layers:
|
||||
1. Pass managers - executes any type of transformations and provides additional debug capabilities.
|
||||
2. Transformations - performs particular transformation algorithm on `ngraph::Function`.
|
||||
3. Low level functions that takes set of nodes and performs some transformation action.
|
||||
They are not mandatory and all transformation code can be located inside transformation.
|
||||
But if some transformation parts can potentially be reused in other transformations we suggest to keep them as a separate functions.
|
||||
|
||||
To decide where to store your transformation code please follow these rules:
|
||||
1. If it's plugin specific transformation and can't be reused by other plugins keep source code inside plugin.
|
||||
2. If this transformation relates to OpSetXToOpSetY conversion or it's common optimization then keep sources inside transformation library.
|
||||
|
||||
After you decided where to store your transformation code you can start develop your own nGraph transformation.
|
||||
|
||||
## Table of Contents:
|
||||
|
||||
### 1. [`ngraph::Function` and graph representation](#ngraph_function)
|
||||
### 2. [Transformations types](#transformations_types)
|
||||
### 2.1 [Function pass](#function_pass)
|
||||
### 2.2 [Matcher pass](#matcher_pass)
|
||||
### 2.3 [GraphRewrite pass](#graph_rewrite_pass)
|
||||
### 3. [Pattern matching](#pattern_matching)
|
||||
### 4. [Working with ngraph::Function](#working_with_ngraph_function)
|
||||
### 5. [Transformation writing essentials](#transformation_writing_essentials)
|
||||
### 6. [Common mistakes in transformations](#common_mistakes)
|
||||
### 7. [Using pass manager](#using_pass_manager)
|
||||
### 8. [How to debug transformations](#how_to_debug_transformations)
|
||||
### 9. [Disabling/Enabling specific transformations for plugin X](#disabling_transformation)
|
||||
### 10. [Transformations testing](#transformations_testing)
|
||||
|
||||
## ngraph::Function and graph representation <a name="ngraph_function"></a>
|
||||
|
||||
nGraph function is a very simple thing: it stores shared pointers to `ngraph::op::Result` and `ngraph::op::Parameter` operations that are inputs and outputs of the graph.
|
||||
All other operations hold each other via shared pointers: child operation holds its parent (hard link). If operation has no consumers and it's not Result operation
|
||||
(shared pointer counter is zero) then it will be destructed and won't be accessible anymore. Each operation in `ngraph::Function` has a `std::shared_ptr<ngraph::Node>` type.
|
||||
|
||||
Below you can find examples how `ngraph::Function` can be created:
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph_utils:simple_function
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph_utils:advanced_function
|
||||
|
||||
## Transformations types <a name="transformations_types"></a>
|
||||
|
||||
nGraph has tree main transformation types: `ngraph::pass::FunctionPass` - strait forward way to work with `ngraph::Function` directly;
|
||||
`ngraph::pass::MatcherPass` - pattern based transformation approach; `ngraph::pass::GraphRewrite` - container for matcher passes.
|
||||
|
||||
![transformations_structure]
|
||||
|
||||
###1. ngraph::pass::FunctionPass <a name="function_pass"></a>
|
||||
|
||||
`ngraph::pass::FunctionPass` is used for transformations that take entire `ngraph::Function` as input and process it.
|
||||
|
||||
Template for FunctionPass transformation class
|
||||
|
||||
@snippet src/template_function_transformation.hpp function_pass:template_transformation_hpp
|
||||
|
||||
@snippet src/template_function_transformation.cpp function_pass:template_transformation_cpp
|
||||
|
||||
Using `ngraph::FunctionPass` you need to override `run_on_function` method where you will write transformation code.
|
||||
Return value must be `true` if original function has changed during transformation (new operation were added or operations replacement was made or node attributes were changed) otherwise it must be `false`.
|
||||
For transformation API please follow [working with ngraph::Function](#working_with_ngraph_function) section.
|
||||
Also `ngraph::FunctionPass` based transformations can be executed via `pass::Manager`. See examples in [Using pass manager](#using_pass_manager) section.
|
||||
|
||||
###2. ngraph::pass::MatcherPass <a name="matcher_pass"></a>
|
||||
|
||||
`ngraph::pass::MatcherPass` is used for pattern based transformations.
|
||||
|
||||
Template for MatcherPass transformation class
|
||||
@snippet src/template_pattern_transformation.hpp graph_rewrite:template_transformation_hpp
|
||||
|
||||
@snippet src/template_pattern_transformation.cpp graph_rewrite:template_transformation_cpp
|
||||
|
||||
Using `ngraph::pass::MatcherPass` you need to complete these steps:
|
||||
1. Create pattern
|
||||
2. Implement callback
|
||||
3. Register pattern and Matcher
|
||||
4. MatcherPass execution
|
||||
|
||||
So let's go though each of this steps.
|
||||
|
||||
### Create pattern
|
||||
Pattern is a single root `ngraph::Function`. But the only difference is that you don't need to create function object, you just create and connect nGraph or special pattern operations.
|
||||
And then take the last created operation and put it as a root of the pattern. This root node will be used as a root node in pattern matching.
|
||||
> **Note**: any nodes in pattern that have no consumers and not registered as root won't be used in pattern matching.
|
||||
|
||||
@snippet example_ngraph_utils.cpp pattern:simple_example
|
||||
|
||||
You may have noticed that `Parameter` operation in example has type and shape specified. These attributes are needed only to create Parameter operation class and won't be used in pattern matching.
|
||||
|
||||
But what if we want to match pattern where `ShapeOf` takes any operation as input? To find an answer please follow [pattern matching](#pattern_matching) section.
|
||||
|
||||
### Implement callback
|
||||
Callback is an action applied to every pattern entrance. In general callback is lambda function that takes Matcher object with detected sub-graph.
|
||||
|
||||
@snippet example_ngraph_utils.cpp pattern:callback_example
|
||||
|
||||
Example above shows callback structure and how Matcher can be used for accessing nodes detected by pattern.
|
||||
Callback return value must be `true` if root node was replaced and another pattern can't be applied to the same root node otherwise it must be `false`.
|
||||
> **Note**: it's not recommended to manipulate with nodes that are under root node. This may affect GraphRewrite execution as it's expected that all nodes that comes after root node in topological order are valid and can be used in pattern matching.
|
||||
|
||||
MatcherPass also provides functionality that allows to report which newly created nodes can be used in additional pattern matching.
|
||||
If MatcherPass was registered in `pass::Manager` or `pass::GraphRewrite` then this registered nodes will be added for additional pattern matching.
|
||||
That means that matcher passes registered in `pass::GraphRewrite` will be applied to this nodes.
|
||||
|
||||
Example below shows how single MatcherPass can fuse sequence of operations using `register_new_node` method.
|
||||
|
||||
@snippet src/template_pattern_transformation.cpp matcher_pass:relu_fusion
|
||||
|
||||
> **Note**: if you register multiple nodes please add them in topological order. We do not topologically sort this nodes as it's time consuming operation.
|
||||
|
||||
### Register pattern and Matcher
|
||||
The last step is to register Matcher and callback inside MatcherPass pass. And to do this you need to call `register_matcher` method.
|
||||
> **Note**: Only one matcher can be registered for single MatcherPass class.
|
||||
|
||||
```cpp
|
||||
// Register matcher and callback
|
||||
register_matcher(m, callback);
|
||||
```
|
||||
### Matcher pass execution
|
||||
MatcherPass has multiple ways to be executed:
|
||||
1. Run on a single node - it can be useful if you want to run MatcherPass inside another transformation.
|
||||
@snippet src/template_pattern_transformation.cpp matcher_pass:run_on_node
|
||||
2. Run on `ngraph::Function` using GraphRewrite - this approach gives ability to run MatcherPass on whole `ngraph::Functoin`. Moreover multiple MatcherPass transformation can be registered in a single GraphRewite to be executed in a single graph traversal.
|
||||
@snippet src/template_pattern_transformation.cpp matcher_pass:graph_rewrite
|
||||
3. Run on `ngraph::Function` using `pass::Manager` - this approach helps you to register MatcherPass for execution on `ngraph::Function` as another transformation types.
|
||||
@snippet src/template_pattern_transformation.cpp matcher_pass:manager
|
||||
|
||||
|
||||
###3. ngraph::pass::GraphRewrite <a name="graph_rewrite_pass"></a>
|
||||
|
||||
GraphRewrite pass serves for running multiple matcher passes on `ngraph::Function` in a single graph traversal.
|
||||
Example:
|
||||
|
||||
@snippet src/template_pattern_transformation.cpp matcher_pass:graph_rewrite
|
||||
|
||||
In addition GraphRewrite handles nodes that were registered by MatcherPasses during their execution. This nodes will be added to the beginning of sequence with nodes for pattern matching.
|
||||
|
||||
> **Note**: when using `pass::Manager` temporary GraphRewrite is used to execute single MatcherPass.
|
||||
|
||||
GraphRewrite has two algorithms for MatcherPasses execution. First algorithm is a straight-forward. It applies each MatcherPass in registraion order to current node.
|
||||
|
||||
![graph_rewrite_execution]
|
||||
|
||||
But it is nor really efficient when you have a lot of registered passes. So first of all GraphRewrite check that all MatcherPass patterns has type based root node (it means that type of this node is not hidden into predicate).
|
||||
And then creates map from registered MatcherPases. That helps to avoid additional cost of applying each MatcherPass for each node.
|
||||
|
||||
![graph_rewrite_efficient_search]
|
||||
|
||||
## Pattern matching <a name="pattern_matching"></a>
|
||||
|
||||
Sometimes patterns can't be expressed via regular nGraph operations or it is too complicated.
|
||||
For example if you want to detect Convolution->Add sub-graph without specifying particular input type for Convolution operation or you want to create pattern where some of operations can have different types.
|
||||
And for these cases nGraph provides additional helpers to construct patterns for GraphRewrite transformations.
|
||||
|
||||
There are two main helpers:
|
||||
1. `ngraph::pattern::any_input` - helps to express inputs if their types are undefined.
|
||||
2. `ngraph::pattern::wrap_type<T>` - helps to express nodes of pattern without specifying node attributes.
|
||||
|
||||
Let's go through example to have better understanding how it works:
|
||||
|
||||
> **Note**: node attributes do not participate in pattern matching and needed only for operations creation. Only operation types participate in pattern matching.
|
||||
|
||||
Example below shows basic usage of `pattern::any_input`.
|
||||
Here we construct Multiply pattern with arbitrary first input and Constant as a second input.
|
||||
Also as Multiply is commutative operation it does not matter in which order we set inputs (any_input/Constant or Constant/any_input) because both cases will be matched.
|
||||
|
||||
@snippet example_ngraph_utils.cpp pattern:label_example
|
||||
|
||||
This example show how we can construct pattern when operation has arbitrary number of inputs.
|
||||
|
||||
@snippet example_ngraph_utils.cpp pattern:concat_example
|
||||
|
||||
This example shows how to use predicate to construct pattern. Also it shows how to match pattern manually on given node.
|
||||
|
||||
@snippet example_ngraph_utils.cpp pattern:predicate_example
|
||||
|
||||
> **Note**: be careful with manual matching because Matcher object holds matched nodes. To clear match use m->clear_state() method.
|
||||
|
||||
## Working with ngraph::Function <a name="working_with_ngraph_function"></a>
|
||||
|
||||
In this chapter we will review nGraph API that allows us to manipulate with `ngraph::Function`.
|
||||
|
||||
###1. ngraph::Node input and output ports
|
||||
|
||||
First of all let's talk about `ngraph::Node` input/output ports. Each nGraph operation has input and output ports except cases when operation has `Result`, `Parameter` or `Constant` type.
|
||||
|
||||
Every port belongs to its node so using port we can access parent node, get shape and type for particular input/output, get all consumers in case of output port and get producer node in case of input port.
|
||||
With output port we can set inputs for newly created operations.
|
||||
|
||||
Lets look at code example.
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:ports_example
|
||||
|
||||
You may notice that we usually construct operations in this way:
|
||||
```cpp
|
||||
std::shared_ptr<Node> neg_const = opset1::Constant::create(sub->get_input_element_type(1), Shape{1}, {-1}));
|
||||
Output<Node> data = node->input_value(0);
|
||||
auto neg = std::make_shared<ngraph::opset1::Multiply>(data, neg_const);
|
||||
```
|
||||
In this example `opset3::Multiply` operation takes `Output<Node>` and `std::shared_ptr<Node>` as inputs. But constructor takes both as `Output<Node>`.
|
||||
In this case `std::shared_ptr<Node>` will be automatically converted to `Output<Node>` if node has exactly one output port otherwise conversion will raise an exception.
|
||||
|
||||
###2. ngraph::Node replacement
|
||||
|
||||
nGraph provides two ways for node replacement: via nGraph helper function and directly via port methods. We are going to review both of them.
|
||||
|
||||
Let's start with nGraph helper functions. The most popular function is `ngraph::replace_node(old_node, new_node)`.
|
||||
|
||||
We will review real replacement case where Negative operation replaces with Multiply.
|
||||
|
||||
![ngraph_replace_node]
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:replace_node
|
||||
|
||||
`ngraph::replace_node` has a constraint that number of output ports for both of ops must be the same otherwise it will raise an exception.
|
||||
|
||||
|
||||
The alternative way to do the same replacement is next:
|
||||
```cpp
|
||||
// All neg->output(0) consumers will be moved to mul->output(0) port
|
||||
neg->output(0).replace(mul->output(0));
|
||||
```
|
||||
|
||||
Another transformation example is insertion.
|
||||
|
||||
![ngraph_insert_node]
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:insert_node
|
||||
|
||||
The alternative way to insert operation is to make a node copy and use `replace_node`:
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:insert_node_with_copy
|
||||
|
||||
###3. ngraph::Node elimination
|
||||
|
||||
Another type of node replacement is its elimination.
|
||||
|
||||
To eliminate operation nGraph has special method that consider all limitations related to InferenceEngine.
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:eliminate_node
|
||||
|
||||
`replace_output_update_name` in case of successful replacement it automatically preserves friendly name and runtime info.
|
||||
|
||||
|
||||
## Transformation writing essentials <a name="transformation_writing_essentials"></a>
|
||||
|
||||
When developing transformation we need to follow next transformation rules:
|
||||
|
||||
###1. Operation Set (OpSet)
|
||||
|
||||
Which OpSet to use in your transformation? The right answer is latest that exists at the moment. An exception is ConvertOpSetXToOpSetY transformations where operations from OpSetX and OpSetY are required to use.
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:include
|
||||
|
||||
###2. Dynamic Shape and Rank
|
||||
|
||||
nGraph has two types for shape representation:
|
||||
`ngraph::Shape` - represents static shape.
|
||||
`ngraph::PartialShape` - represents dynamic shape. That means that rank or some of dimensions are dynamic (undefined).
|
||||
`ngraph::PartialShape` can be converted to `ngraph::Shape` using `get_shape()` method if all dimensions are static otherwise conversion will raise an exception.
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:shape
|
||||
|
||||
But in most cases before getting static shape using `get_shape()` method you need to check that shape is static.
|
||||
|
||||
Also if your transformation requires only input shape rank or particular dimension value for some reason please do not use `get_shape()` method. See example below how not to use `get_shape()`
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:shape_check
|
||||
|
||||
Not using `get_shape()` method makes your transformation more flexible and applicable for more cases.
|
||||
|
||||
###3. Friendly Names
|
||||
|
||||
Each `ngraph::Node` has unique name (is used for nGraph internals) and friendly name. In transformations we care only about friendly name because it represents name from IR.
|
||||
Also friendly name is used as output tensor name (until we do not have other way to represent output tensor name) and user code that requests intermediate outputs based on this names.
|
||||
So not to loose friendly name when replacing node with other node or sub-graph we need to set original friendly name to the latest node in replacing sub-garph. See example below.
|
||||
|
||||
```cpp
|
||||
// Replace Div operation with Power and Multiply sub-graph and set original friendly name to Multiply operation
|
||||
auto pow = std::make_shared<ngraph::opset1::Power>(div->input(1).get_source_output(),
|
||||
op::Constant::create(div->get_input_element_type(1), Shape{1}, {-1}));
|
||||
auto mul = std::make_shared<ngraph::opset1::Multiply>(div->input(0).get_source_output(), pow);
|
||||
mul->set_friendly_name(div->get_friendly_name());
|
||||
ngraph::replace_node(div, mul);
|
||||
```
|
||||
|
||||
In more advanced cases when replaced operation has several outputs and we add additional consumers to its outputs we make decision how to set friendly name by arrangement.
|
||||
|
||||
###4. Runtime Info
|
||||
|
||||
Runtime info is a map `std::map<std::string, std::shared_ptr<Variant>>` located inside `ngraph::Node` class. It represents additional attributes in `ngraph::Node`.
|
||||
These attributes can be set by users or by plugins and when executing transformation that changes `ngraph::Function` we need to preserve this attributes as they won't be automatically propagated.
|
||||
In most cases transformations has next types: 1:1 (replace node with another node), 1:N (replace node with a sub-graph), N:1 (fuse sub-graph into a single node), N:M (any other transformation).
|
||||
Currently there is no mechanism that automatically detects transformation types so we need to propagate this runtime information manually. See examples below.
|
||||
|
||||
```cpp
|
||||
// Replace Transpose with Reshape operation (1:1)
|
||||
ngraph::copy_runtime_info(transpose, reshape);
|
||||
```
|
||||
|
||||
```cpp
|
||||
// Replace Div operation with Power and Multiply sub-graph (1:N)
|
||||
ngraph::copy_runtime_info(div, {pow, mul});
|
||||
```
|
||||
|
||||
```cpp
|
||||
// Fuse Convolution with Add operation (N:1)
|
||||
ngraph::copy_runtime_info({conv, bias}, {conv_ie});
|
||||
```
|
||||
|
||||
```cpp
|
||||
// Any other transformation that replaces one sub-graph with another sub-graph (N:M)
|
||||
ngraph::copy_runtime_info({a, b, c}, {e, f});
|
||||
```
|
||||
|
||||
When transformation has multiple fusions or decompositions `ngraph::copy_runtime_info` must be called multiple times for each case.
|
||||
|
||||
> **Note**: copy_runtime_info removes rt_info from destination nodes. If you want to keep it you need to specify them in source nodes like this: copy_runtime_info({a, b, c}, {a, b})
|
||||
|
||||
###5. Constant Folding
|
||||
|
||||
If your transformation inserts constant sub-graphs that needs to be folded do not forget to use `ngraph::pass::ConstantFolding()` after your transformation or call constant folding directly for operation.
|
||||
Example below shows how constant sub-graph can be constructed.
|
||||
|
||||
```cpp
|
||||
// After ConstantFolding pass Power will be replaced with Constant
|
||||
auto pow = std::make_shared<ngraph::opset3::Power>(
|
||||
opset3::Constant::create(element::f32, Shape{1}, {2})
|
||||
opset3::Constant::create(element::f32, Shape{1}, {3}));
|
||||
auto mul = std::make_shared<ngraph::opset3::Multiply>(input /* not constant input */, pow);
|
||||
```
|
||||
|
||||
Manual constant folding is more preferable than `ngraph::pass::ConstantFolding()` because it is much faster.
|
||||
|
||||
Below you can find an example of manual constant folding:
|
||||
|
||||
@snippet src/template_pattern_transformation.cpp manual_constant_folding
|
||||
|
||||
## Common mistakes in transformations <a name="common_mistakes"></a>
|
||||
|
||||
In transformation development process
|
||||
|
||||
* Do not use deprecated nGraph API. Deprecated methods has `NGRAPH_DEPRECATED` macros in its definition.
|
||||
* Do not pass `shared_ptr<Node>` as input for other node if type of node is unknown or it has multiple outputs. Use explicit output port.
|
||||
* If you replace node with another node that produce different shape you need to remember that new shape won't be propagated until first `validate_nodes_and_infer_types` call for `ngraph::Function`. If you are using `pass::Manager` it will automatically call this method after each transformation execution.
|
||||
* Do not forget to call `ngraph::ConstantFolding` pass if your transformation creates constant sub-graphs.
|
||||
* Use latest OpSet if you are not developing downgrade transformation pass.
|
||||
* When developing callback for `ngraph::pass::MatcherPass` do not change nodes that comes after root node in topological order.
|
||||
|
||||
## Using pass manager <a name="using_pass_manager"></a>
|
||||
|
||||
`ngraph::pass::Manager` is a container class that can store list of transformations and execute them. The main idea of this class is to have high-level representation for grouped list of transformations.
|
||||
It can register and apply any [transformation types](#transformations_types) on function.
|
||||
In addition `ngraph::pass::Manager` has extended debug capabilities (find more information in [how to debug transformations](#how_to_debug_transformations) section).
|
||||
|
||||
Example below shows basic usage of `ngraph::pass::Manager`
|
||||
|
||||
@snippet src/template_pattern_transformation.cpp matcher_pass:manager3
|
||||
|
||||
Another example how multiple matcher passes can be united into single GraphRewrite.
|
||||
|
||||
@snippet src/template_pattern_transformation.cpp matcher_pass:manager2
|
||||
|
||||
## How to debug transformations <a name="how_to_debug_transformations"></a>
|
||||
|
||||
The most popular tool for transformations debugging is `ngraph::pass::VisualizeTree` transformation that visualize ngraph::Function.
|
||||
|
||||
Usage example:
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:visualize
|
||||
|
||||
`ngraph::pass::VisualizeTree` can be parametrized via environment variables:
|
||||
|
||||
```
|
||||
NGRAPH_VISUALIZE_TREE_OUTPUT_SHAPES=1 - visualize shapes
|
||||
NGRAPH_VISUALIZE_TREE_OUTPUT_TYPES=1 - visualize types
|
||||
```
|
||||
|
||||
> **Note**: current VisualTree has not user friendly interface and it will be changed in nearest future. The intention is to move visualize abilities inside transformations.
|
||||
|
||||
If you are using `ngraph::pass::Manager` to run sequence of transformations you can get additional debug capabilities by using next environment variables:
|
||||
|
||||
```
|
||||
NGRAPH_PROFILE_PASS_ENABLE=1 - enables performance measurement for each transformation and prints execution status
|
||||
NGRAPH_ENABLE_VISUALIZE_TRACING=1 - enables visualization after each transformation. By default it saves dot and svg files.
|
||||
```
|
||||
|
||||
> **Note**: make sure that you have dot installed on your machine otherwise it will silently save only dot file without svg file.
|
||||
|
||||
## Disabling/Enabling specific transformations for plugin X <a name="disabling_transformation"></a>
|
||||
|
||||
This topic mostly related to conversion to legacy opset and plugins that based on CNNNetwork but still this mechanism can be applied for other cases.
|
||||
Let's suppose that plugin X enabled `opset3::StridedSlice` operation support and you want to disable `ngraph::pass::ConvertStridedSliceToCrop` transformation for plugin X.
|
||||
To do this you need to create callback on plugin side and pass it to transformation. And also you need to update particular transformation to use this callback.
|
||||
|
||||
```cpp
|
||||
// Update callback to be able to use m_transformation_callback if this transformation based on GraphRewrite.
|
||||
ngraph::graph_rewrite_callback callback = [this](pattern::Matcher &m) {
|
||||
...
|
||||
}
|
||||
|
||||
// Use transformation_callback not to execute transformation if callback returns true for given node
|
||||
if (m_transformation_callback(node)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Implement transformation callback and pass it directly to transformation or pass::Manager
|
||||
const auto transformations_callback = [](const std::shared_ptr<const ::ngraph::Node> &node) -> bool {
|
||||
return std::dynamic_pointer_cast<const ::ngraph::opset3::StridedSlice>(node) != nullptr;
|
||||
};
|
||||
|
||||
// Register transformation and pass callback to pass::Manager
|
||||
ngraph::pass::Manager manager;
|
||||
manager.register_pass<ngraph::pass::ConvertStridedSliceToCrop>();
|
||||
// pass::Manager will set callback to all reistered transformations automatically
|
||||
manager.set_callback(transformations_callback);
|
||||
manager.run_passes(f);
|
||||
```
|
||||
|
||||
## Transformations testing <a name="transformations_testing"></a>
|
||||
|
||||
If you are developing new transformation inside plugin you need to add test into `template_plugin/tests/functional/transformations` folder.
|
||||
We have two types of tests: nGraph reader tests located in `inference-engine/tests/functional/inference_engine/ngraph_reader` and transformation tests located in `inference-engine/tests/functional/inference_engine/transformations`
|
||||
Reader tests are IR based and test end to end conversion from IR to CNNNetwork. Transformation tests test single ngraph transformations or low level functiont that are used inside transformations.
|
||||
|
||||
The basic transformation test looks like this:
|
||||
|
||||
@snippet tests/functional/transformations/template_transformations_test.cpp transformation:test
|
||||
|
||||
|
||||
[ngraph_replace_node]: ../images/ngraph_replace_node.png
|
||||
[ngraph_insert_node]: ../images/ngraph_insert_node.png
|
||||
[transformations_structure]: ../images/transformations_structure.png
|
||||
[register_new_node]: ../images/register_new_node.png
|
||||
[graph_rewrite_execution]: ../images/graph_rewrite_execution.png
|
||||
[graph_rewrite_efficient_search]: ../images/graph_rewrite_efficient_search.png
|
||||
@@ -80,7 +80,7 @@ Actual graph compilation is done in the `ExecutableNetwork` constructor. Refer t
|
||||
The function accepts a const shared pointer to `ngraph::Function` object and performs the following steps:
|
||||
|
||||
1. Deep copies a const object to a local object, which can later be modified.
|
||||
2. Applies common and plugin-specific transformations on a copied graph to make the graph more friendly to hardware operations. For details how to write custom plugin-specific transformation, please, refer to [Writing ngraph transformations](@ref new_ngraph_transformation) guide. See detailed topics about network representation:
|
||||
2. Applies common and plugin-specific transformations on a copied graph to make the graph more friendly to hardware operations. For details how to write custom plugin-specific transformation, please, refer to [Writing ngraph transformations](@ref ngraph_transformation) guide. See detailed topics about network representation:
|
||||
* [Intermediate Representation and Operation Sets](../_docs_MO_DG_IR_and_opsets.html)
|
||||
* [Quantized networks](@ref quantized_networks).
|
||||
|
||||
|
||||
@@ -43,7 +43,7 @@ To build test binaries together with other build artifacts, use the `make all` c
|
||||
|
||||
### Tests for plugin-specific ngraph transformations
|
||||
|
||||
Please, refer to [Transformation testing](@ref new_ngraph_transformation) guide.
|
||||
Please, refer to [Transformation testing](@ref ngraph_transformation) guide.
|
||||
|
||||
### How to Extend Inference Engine Plugin Tests
|
||||
|
||||
@@ -55,4 +55,4 @@ as input graphs used by tests. In this case, to test a new layer with layer test
|
||||
the `IE::ngraphFunctions` library, which is also included in the Inference Engine Developer package, with a new nGraph function
|
||||
including the corresponding operation.
|
||||
|
||||
> **NOTE**: When implementing a new subgraph test, add new single-layer tests for each operation of the subgraph if such test does not exist.
|
||||
> **NOTE**: When implementing a new subgraph test, add new single-layer tests for each operation of the subgraph if such test does not exist.
|
||||
|
||||
@@ -14,7 +14,6 @@
|
||||
<tab type="user" url="@ref plugin_build" visibile="yes" title="Build Your Plugin with CMake*"/>
|
||||
<tab type="user" url="@ref plugin_testing" visibile="yes" title="Test Your Plugin"/>
|
||||
<tab type="user" url="@ref quantized_networks" visibile="yes" title="Quantized networks guide"/>
|
||||
<tab type="user" url="@ref new_ngraph_transformation" visibile="yes" title="Writing ngraph transformations"/>
|
||||
</tab>
|
||||
<!-- API References -->
|
||||
<tab type="usergroup" title="API REFERENCE">
|
||||
@@ -23,4 +22,4 @@
|
||||
</tab>
|
||||
<tab type="usergroup" title="MAIN OPENVINO™ DOCS" url="../index.html"/>
|
||||
</navindex>
|
||||
</doxygenlayout>
|
||||
</doxygenlayout>
|
||||
|
||||
@@ -268,7 +268,6 @@
|
||||
<tab type="user" title="Inference Engine API Changes History" url="@ref openvino_docs_IE_DG_API_Changes"/>
|
||||
<tab type="user" title="Inference Engine Memory primitives" url="@ref openvino_docs_IE_DG_Memory_primitives"/>
|
||||
<tab type="user" title="Inference Engine Device Query API" url="@ref openvino_docs_IE_DG_InferenceEngine_QueryAPI"/>
|
||||
<tab type="user" title="Place of nGraph in the Inference Engine" url="@ref openvino_docs_IE_DG_nGraph_Flow"/>
|
||||
<tab type="usergroup" title="Inference Engine Extensibility Mechanism" url="@ref openvino_docs_IE_DG_Extensibility_DG_Intro">
|
||||
<tab type="user" title="Extension Library" url="@ref openvino_docs_IE_DG_Extensibility_DG_Extension"/>
|
||||
<tab type="user" title="Custom Operations" url="@ref openvino_docs_IE_DG_Extensibility_DG_AddingNGraphOps"/>
|
||||
@@ -281,7 +280,6 @@
|
||||
<tab type="user" title="[DEPRECATED] Migration from Inference Engine Plugin API to Core API" url="@ref openvino_docs_IE_DG_Migration_CoreAPI"/>
|
||||
<tab type="user" title="Introduction to Performance Topics" url="@ref openvino_docs_IE_DG_Intro_to_Performance"/>
|
||||
<tab type="user" title="Inference Engine Python* API Overview" url="@ref openvino_inference_engine_ie_bridges_python_docs_api_overview"/>
|
||||
<tab type="user" title="Build a Model with nGraph" url="@ref openvino_docs_IE_DG_nGraphTutorial"/>
|
||||
<tab type="user" title="Read an ONNX model" url="@ref openvino_docs_IE_DG_ONNX_Support"/>
|
||||
<tab type="user" title="[DEPRECATED] Import an ONNX model" url="@ref openvino_docs_IE_DG_OnnxImporterTutorial"/>
|
||||
<tab type="user" title="Graph Debug Capabilities" url="@ref openvino_docs_IE_DG_Graph_debug_capabilities"/>
|
||||
@@ -314,6 +312,17 @@
|
||||
<tab type="user" title="Glossary" url="@ref openvino_docs_IE_DG_Glossary"/>
|
||||
</tab>
|
||||
|
||||
<!-- nGraph -->
|
||||
<tab type="usergroup" title="nGraph Developer Guide" url="@ref openvino_docs_nGraph_DG_DevGuide">
|
||||
<tab type="user" title="Introduction" url="@ref openvino_docs_nGraph_DG_Introduction"/>
|
||||
<tab type="user" title="Basic Concepts" url="@ref openvino_docs_nGraph_DG_basic_concepts"/>
|
||||
<tab type="user" title="Operation sets" url="@ref openvino_docs_MO_DG_IR_and_opsets"/>
|
||||
<tab type="user" title="Graph building" url="@ref openvino_docs_nGraph_DG_build_function"/>
|
||||
<tab type="user" title="Transfomations" url="@ref ngraph_transformation"/>
|
||||
<tab type="user" title="Debug capabilities" url="@ref openvino_docs_nGraph_DG_Debug_capabilities"/>
|
||||
<tab type="user" title="Python API" url="@ref openvino_docs_nGraph_DG_PythonAPI"/>
|
||||
</tab>
|
||||
|
||||
<!-- Deployment Manager Guide -->
|
||||
<tab type="user" title="Deployment Manager Guide" url="@ref openvino_docs_install_guides_deployment_manager_tool"/>
|
||||
|
||||
|
||||
@@ -19,6 +19,7 @@ OpenVINO™ toolkit includes the following components:
|
||||
- [Deep Learning Model Optimizer](MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) - A cross-platform command-line tool for importing models and
|
||||
preparing them for optimal execution with the Inference Engine. The Model Optimizer imports, converts, and optimizes models, which were trained in popular frameworks, such as Caffe*,
|
||||
TensorFlow*, MXNet*, Kaldi*, and ONNX*.
|
||||
- [nGraph](nGraph_DG/nGraph_dg.md) - A unified API for graph representation and manipulation engine which is used to represent a model inside Inference Engine and allows the run-time model construction without using Model Optimizer.
|
||||
- [Deep Learning Inference Engine](IE_DG/inference_engine_intro.md) - A unified API to allow high performance inference on many hardware types
|
||||
including the following:
|
||||
- Intel® CPU
|
||||
|
||||
36
docs/nGraph_DG/build_function.md
Normal file
36
docs/nGraph_DG/build_function.md
Normal file
@@ -0,0 +1,36 @@
|
||||
# Build nGraph Function {#openvino_docs_nGraph_DG_build_function}
|
||||
|
||||
This section illustrates how to construct an nGraph function
|
||||
composed of operations from an available opset. Once created,
|
||||
it can wrap into a `CNNNetwork`, creating utility for data scientists
|
||||
or app developers to define a deep-learning model in a neutral way
|
||||
that does not depend on existing Deep Learning (DL) frameworks.
|
||||
|
||||
Operation Set `opsetX` integrates a list of nGraph pre-compiled operations that work
|
||||
for this purpose. In other words, `opsetX` defines a set of operations for building a graph.
|
||||
|
||||
For a complete list of operation sets supported by Inference Engine, see [Available Operations Sets](../ops/opset.md).
|
||||
|
||||
To add custom nGraph operations to an existing `CNNNetwork`, see
|
||||
the [Add Custom nGraph Operations](../IE_DG/Extensibility_DG/Intro.md) document.
|
||||
|
||||
Below you can find examples on to how build `ngraph::Function` from the `opset3` operations:
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:include
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph_utils:simple_function
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph_utils:advanced_function
|
||||
|
||||
To wrap it into a CNNNetwork, use:
|
||||
```cpp
|
||||
CNNNetwork net (ng_function);
|
||||
```
|
||||
## See Also
|
||||
|
||||
* [Available Operation Sets](../ops/opset.md)
|
||||
* [Operation Set `opset1` Specification](../ops/opset1.md)
|
||||
* [Operation Set `opset2` Specification](../ops/opset2.md)
|
||||
* [Operation Set `opset3` Specification](../ops/opset3.md)
|
||||
* [Operation Set `opset4` Specification](../ops/opset4.md)
|
||||
* [Inference Engine Extensibility Developer Guide](../IE_DG/Extensibility_DG/Intro.md)
|
||||
3
docs/nGraph_DG/img/graph_rewrite_efficient_search.png
Normal file
3
docs/nGraph_DG/img/graph_rewrite_efficient_search.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:05eb8600d2c905975674f3a0a5dc676107d22f65f2a1f78ee1cfabc1771721ea
|
||||
size 41307
|
||||
3
docs/nGraph_DG/img/graph_rewrite_execution.png
Normal file
3
docs/nGraph_DG/img/graph_rewrite_execution.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:17cd470c6d04d7aabbdb4a08e31f9c97eab960cf7ef5bbd3a541df92db38f26b
|
||||
size 40458
|
||||
3
docs/nGraph_DG/img/register_new_node.png
Normal file
3
docs/nGraph_DG/img/register_new_node.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:80297287c81a2f27b7e74895738afd90844354a8dd745757e8321e2fb6ed547e
|
||||
size 31246
|
||||
3
docs/nGraph_DG/img/transformations_structure.png
Normal file
3
docs/nGraph_DG/img/transformations_structure.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:0b206c602626f17ba5787810b9a28f9cde511448c3e63a5c7ba976cee7868bdb
|
||||
size 14907
|
||||
46
docs/nGraph_DG/intro.md
Normal file
46
docs/nGraph_DG/intro.md
Normal file
@@ -0,0 +1,46 @@
|
||||
# Introduction to the nGraph {#openvino_docs_nGraph_DG_Introduction}
|
||||
|
||||
The Inference Engine integrates the nGraph to represent a model in run time underneath of the conventional
|
||||
`CNNNetwork` API, which is an instance of `ngraph::Function`.
|
||||
|
||||
Besides the representation update, nGraph supports new features:
|
||||
|
||||
1. nGraph contains several [sets of operations which are called `opset1`, `opset2` and etc.](../ops/opset.md).
|
||||
Operations from these operation sets are generated by the Model Optimizer and are accepted in the Inference Engine.
|
||||
|
||||
2. Operation version is attached to each operation rather than to the entire IR file format.
|
||||
IR is still versioned but has a different meaning. For details, see [Deep Learning Network Intermediate Representation and Operation Sets in OpenVINO™](../MO_DG/IR_and_opsets.md).
|
||||
|
||||
3. Creating models in run-time without loading IR from an xml/binary file. You can enable it by creating
|
||||
`ngraph::Function` passing it to `CNNNetwork`.
|
||||
|
||||
4. Run-time reshape capability and constant folding are implemented through the nGraph code for more operations compared to previous releases.
|
||||
As a result, more models can be reshaped. For details, see the [dedicated guide about the reshape capability](../IE_DG/ShapeInference.md).
|
||||
|
||||
5. Loading [model from ONNX format](../IE_DG/ONNX_Support.md) without converting it to the Inference Engine IR.
|
||||
|
||||
6. nGraph representation supports dynamic shapes. You can use `CNNNetwork::reshape()` method in order to specialize input shapes.
|
||||
|
||||
The complete picture of existed flow is presented below.
|
||||
|
||||
|
||||
|
||||
## Read the Intermediate Representation to `CNNNetwork`
|
||||
|
||||
The IR version 10 automatically triggers the nGraph flow inside the Inference Engine.
|
||||
When such IR is read in an application, the Inference Engine IR reader produces `CNNNetwork` that encapsulates the `ngraph::Function` instance underneath.
|
||||
|
||||
Interpretation of the IR version 10 differs from the old IR version.
|
||||
Besides having a different operations set, the IR version 10 ignores the shapes and data types assigned to the ports in an XML file.
|
||||
Both shapes and types are reinferred while loading to the Inference Engine using the nGraph shape and type propagation function that is a part of each nGraph operation.
|
||||
|
||||
## Build a Model in the Application
|
||||
|
||||
Alternative method to feed the Inference Engine with a model is to create the model in the run time.
|
||||
It is achieved by creation of the `ngraph::Function` construction using nGraph operation classes and optionally user-defined operations.
|
||||
For details, see [Add Custom nGraph Operations](../IE_DG/Extensibility_DG/AddingNGraphOps.md) and [examples](build_function.md).
|
||||
At this stage, the code is completely independent of the rest of the Inference Engine code and can be built separately.
|
||||
After you construct an instance of `ngraph::Function`, you can use it to create `CNNNetwork` by passing it to the new constructor for this class.
|
||||
|
||||
## See Also
|
||||
- [Available Operations Sets](../ops/opset.md)
|
||||
443
docs/nGraph_DG/nGraphTransformation.md
Normal file
443
docs/nGraph_DG/nGraphTransformation.md
Normal file
@@ -0,0 +1,443 @@
|
||||
# Overview of Transformations API {#ngraph_transformation}
|
||||
|
||||
This guide contains all necessary information that you need to start implementing nGraph transformations.
|
||||
|
||||
## Prerequisites
|
||||
Before creating a transformation, do the following:
|
||||
|
||||
* Make sure that there is no transformation with the same functionality in the [Transformation Library](group__ie__transformation__api.html)
|
||||
* Learn how the [Transformation Library](group__ie__transformation__api.html) is structured and how transformations are organized
|
||||
* Understand where to put your transformation code
|
||||
|
||||
### Transformation Library Structure
|
||||
Transformation library is independent from Inference Engine target library named as `inference_engine_transformations`
|
||||
and is located in the `inference-engine/src/transformations` directory.
|
||||
|
||||
Transformations root directory contains two folders:
|
||||
* `ngraph_ops` - Contains legacy opset operations needed for nGraph to CNNNetwork conversion.
|
||||
> **NOTE**: This operation is prohibited inside new plugins until they are not moved to a separate directory with allowed operations.
|
||||
* `transformations` - Includes all transformations, utils, runtime info attributes, and pass managers.
|
||||
> **NOTE**: Do not use transformation that belongs to `ngraph::pass::ConvertOpSet1ToLegacy` transformations until they are not moved to a separate directory with allowed transformations.
|
||||
|
||||
### Transformation Flow Layers
|
||||
Transformation flow in the transformation library has several layers:
|
||||
|
||||
1. Pass managers - Execute any type of transformations and provide additional debug capabilities.
|
||||
2. Transformations - Perform a particular transformation algorithm on `ngraph::Function`.
|
||||
3. Low-level functions - Take a set of nodes and perform some transformation action.
|
||||
They are not mandatory and all transformation code can be located inside the transformation.
|
||||
But if some transformation parts can potentially be reused in other transformations, we suggest keeping them as a separate functions.
|
||||
|
||||
### Location for Your Transformation Code
|
||||
To decide where to store your transformation code, please follow these rules:
|
||||
|
||||
1. If it is a plugin-specific transformation and cannot be reused by other plugins, keep source code inside plugin.
|
||||
2. If this transformation relates to the OpSetXToOpSetY conversion or it is common optimization, keep sources inside the transformation library.
|
||||
|
||||
After you decide where to store your transformation code, you can start developing your own nGraph transformation.
|
||||
|
||||
## ngraph::Function and graph representation <a name="ngraph_function"></a>
|
||||
|
||||
An nGraph function is a simple thing: it stores shared pointers to `ngraph::op::Result` and `ngraph::op::Parameter` operations that are inputs and outputs of the graph.
|
||||
All other operations hold each other via shared pointers: child operation holds its parent (hard link). If the operation has no consumers and it is not a Result operation
|
||||
(shared pointer counter is zero), it is destructed and is not accessible anymore. Each operation in `ngraph::Function` has a `std::shared_ptr<ngraph::Node>` type.
|
||||
|
||||
For examples of how to build an nGraph function, see the [Build nGraph Function](./build_function.md) page.
|
||||
|
||||
## Transformations types <a name="transformations_types"></a>
|
||||
|
||||
nGraph has three main transformation types:
|
||||
|
||||
* `ngraph::pass::FunctionPass` - straightforward way to work with `ngraph::Function` directly
|
||||
* `ngraph::pass::MatcherPass` - pattern-based transformation approach
|
||||
* `ngraph::pass::GraphRewrite` - container for matcher passes
|
||||
|
||||
![transformations_structure]
|
||||
|
||||
### ngraph::pass::FunctionPass <a name="function_pass"></a>
|
||||
|
||||
`ngraph::pass::FunctionPass` is used for transformations that take entire `ngraph::Function` as an input and process it.
|
||||
|
||||
Template for FunctionPass transformation class
|
||||
|
||||
@snippet src/template_function_transformation.hpp function_pass:template_transformation_hpp
|
||||
|
||||
@snippet src/template_function_transformation.cpp function_pass:template_transformation_cpp
|
||||
|
||||
Using `ngraph::FunctionPass`, you need to override the `run_on_function` method where you will write the transformation code.
|
||||
Return value is `true` if the original function has changed during transformation (new operation was added, or operations replacement was made, or node attributes were changed); otherwise, it is `false`.
|
||||
For transformation API, please follow the [working with ngraph::Function](#working_with_ngraph_function) section.
|
||||
Also `ngraph::FunctionPass` based transformations can be executed via `pass::Manager`. See the examples in the [Using pass manager](#using_pass_manager) section.
|
||||
|
||||
### ngraph::pass::MatcherPass <a name="matcher_pass"></a>
|
||||
|
||||
`ngraph::pass::MatcherPass` is used for pattern-based transformations.
|
||||
|
||||
Template for MatcherPass transformation class
|
||||
@snippet src/template_pattern_transformation.hpp graph_rewrite:template_transformation_hpp
|
||||
|
||||
@snippet src/template_pattern_transformation.cpp graph_rewrite:template_transformation_cpp
|
||||
|
||||
To use `ngraph::pass::MatcherPass`, you need to complete these steps:
|
||||
1. Create a pattern
|
||||
2. Implement a callback
|
||||
3. Register the pattern and Matcher
|
||||
4. Execute MatcherPass
|
||||
|
||||
So let's go through each of these steps.
|
||||
|
||||
### Create a pattern
|
||||
Pattern is a single root `ngraph::Function`. But the only difference is that you do not need to create a function object, you just need to create and connect nGraph or special pattern operations. Then you need to take the last created operation and put it as a root of the pattern. This root node will be used as a root node in pattern matching.
|
||||
> **NOTE**: Any nodes in a pattern that have no consumers and are not registered as root will not be used in pattern matching.
|
||||
|
||||
@snippet example_ngraph_utils.cpp pattern:simple_example
|
||||
|
||||
The `Parameter` operation in the example above has type and shape specified. These attributes are needed only to create Parameter operation class and will not be used in pattern matching.
|
||||
|
||||
For instructions on how to match a pattern where `ShapeOf` takes any operation as an input, follow the [pattern matching](#pattern_matching) section.
|
||||
|
||||
### Implement callback
|
||||
Callback is an action applied to every pattern entrance. In general, callback is the lambda function that takes Matcher object with detected subgraph.
|
||||
|
||||
@snippet example_ngraph_utils.cpp pattern:callback_example
|
||||
|
||||
The example above shows the callback structure and how Matcher can be used for accessing nodes detected by pattern.
|
||||
Callback return value is `true` if root node was replaced and another pattern cannot be applied to the same root node; otherwise, it is `false`.
|
||||
> **NOTE**: It is not recommended to manipulate with nodes that are under root node. This may affect GraphRewrite execution as it is expected that all nodes that come after root node in topological order are valid and can be used in pattern matching.
|
||||
|
||||
MatcherPass also provides functionality that allows reporting of the newly created nodes that can be used in additional pattern matching.
|
||||
If MatcherPass was registered in `pass::Manager` or `pass::GraphRewrite`, these registered nodes will be added for additional pattern matching.
|
||||
That means that matcher passes registered in `pass::GraphRewrite` will be applied to these nodes.
|
||||
|
||||
The example below shows how single MatcherPass can fuse sequence of operations using the `register_new_node` method.
|
||||
|
||||
@snippet src/template_pattern_transformation.cpp matcher_pass:relu_fusion
|
||||
|
||||
> **NOTE**: If you register multiple nodes, please add them in topological order. We do not topologically sort these nodes as it is a time-consuming operation.
|
||||
|
||||
### Register pattern and Matcher
|
||||
The last step is to register Matcher and callback inside the MatcherPass pass. To do this, call the `register_matcher` method.
|
||||
> **NOTE**: Only one matcher can be registered for a single MatcherPass class.
|
||||
|
||||
```cpp
|
||||
// Register matcher and callback
|
||||
register_matcher(m, callback);
|
||||
```
|
||||
### Execute MatcherPass
|
||||
MatcherPass has multiple ways to be executed:
|
||||
* Run on a single node - it can be useful if you want to run MatcherPass inside another transformation.
|
||||
@snippet src/template_pattern_transformation.cpp matcher_pass:run_on_node
|
||||
* Run on `ngraph::Function` using GraphRewrite - this approach gives ability to run MatcherPass on whole `ngraph::Function`. Moreover, multiple MatcherPass transformation can be registered in a single GraphRewite to be executed in a single graph traversal.
|
||||
@snippet src/template_pattern_transformation.cpp matcher_pass:graph_rewrite
|
||||
* Run on `ngraph::Function` using `pass::Manager` - this approach helps you to register MatcherPass for execution on `ngraph::Function` as another transformation types.
|
||||
@snippet src/template_pattern_transformation.cpp matcher_pass:manager
|
||||
|
||||
|
||||
### ngraph::pass::GraphRewrite <a name="graph_rewrite_pass"></a>
|
||||
|
||||
GraphRewrite pass serves for running multiple matcher passes on `ngraph::Function` in a single graph traversal.
|
||||
Example:
|
||||
|
||||
@snippet src/template_pattern_transformation.cpp matcher_pass:graph_rewrite
|
||||
|
||||
In addition, GraphRewrite handles nodes that were registered by MatcherPasses during their execution. This nodes will be added to the beginning of the sequence with nodes for pattern matching.
|
||||
|
||||
> **NOTE**: when using `pass::Manager` temporary GraphRewrite is used to execute single MatcherPass.
|
||||
|
||||
GraphRewrite has two algorithms for MatcherPasses execution. First algorithm is straightforward. It applies each MatcherPass in registration order to current node.
|
||||
|
||||
![graph_rewrite_execution]
|
||||
|
||||
But it is not really efficient when you have a lot of registered passes. So first of all GraphRewrite checks that all MatcherPass patterns has type-based root node (it means that type of this node is not hidden into predicate).
|
||||
And then creates map from registered MatcherPasses. That helps to avoid additional cost of applying each MatcherPass for each node.
|
||||
|
||||
![graph_rewrite_efficient_search]
|
||||
|
||||
## Pattern Matching <a name="pattern_matching"></a>
|
||||
|
||||
Sometimes patterns cannot be expressed via regular nGraph operations or it is too complicated.
|
||||
For example, if you want to detect Convolution->Add sub-graph without specifying particular input type for Convolution operation or you want to create a pattern where some of operations can have different types.
|
||||
And for these cases nGraph provides additional helpers to construct patterns for GraphRewrite transformations.
|
||||
|
||||
There are two main helpers:
|
||||
1. `ngraph::pattern::any_input` - helps to express inputs if their types are undefined.
|
||||
2. `ngraph::pattern::wrap_type<T>` - helps to express nodes of pattern without specifying node attributes.
|
||||
|
||||
Let's go through the example to have better understanding of how it works:
|
||||
|
||||
> **NOTE**: Node attributes do not participate in pattern matching and are needed only for operations creation. Only operation types participate in pattern matching.
|
||||
|
||||
The example below shows basic usage of `pattern::any_input`.
|
||||
Here we construct Multiply pattern with arbitrary first input and Constant as a second input.
|
||||
Also as Multiply is commutative operation, it does not matter in which order we set inputs (any_input/Constant or Constant/any_input) because both cases will be matched.
|
||||
|
||||
@snippet example_ngraph_utils.cpp pattern:label_example
|
||||
|
||||
This example shows how we can construct a pattern when operation has arbitrary number of inputs.
|
||||
|
||||
@snippet example_ngraph_utils.cpp pattern:concat_example
|
||||
|
||||
This example shows how to use predicate to construct a pattern. Also it shows how to match pattern manually on given node.
|
||||
|
||||
@snippet example_ngraph_utils.cpp pattern:predicate_example
|
||||
|
||||
> **NOTE**: Be careful with manual matching because Matcher object holds matched nodes. To clear a match, use the m->clear_state() method.
|
||||
|
||||
## Working with ngraph::Function <a name="working_with_ngraph_function"></a>
|
||||
|
||||
In this chapter we will review nGraph API that allows us to manipulate with `ngraph::Function`.
|
||||
|
||||
### ngraph::Node input and output ports
|
||||
|
||||
First of all let's talk about `ngraph::Node` input/output ports. Each nGraph operation has input and output ports except cases when operation has `Result`, `Parameter`, or `Constant` type.
|
||||
|
||||
Every port belongs to its node, so using a port we can access parent node, get shape and type for particular input/output, get all consumers in case of output port, and get producer node in case of input port.
|
||||
With output port we can set inputs for newly created operations.
|
||||
|
||||
Lets look at the code example.
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:ports_example
|
||||
|
||||
You may notice that we usually construct operations in this way:
|
||||
```cpp
|
||||
std::shared_ptr<Node> neg_const = opset1::Constant::create(sub->get_input_element_type(1), Shape{1}, {-1}));
|
||||
Output<Node> data = node->input_value(0);
|
||||
auto neg = std::make_shared<ngraph::opset1::Multiply>(data, neg_const);
|
||||
```
|
||||
In this example, the `opset3::Multiply` operation takes `Output<Node>` and `std::shared_ptr<Node>` as inputs. But the constructor takes both as `Output<Node>`.
|
||||
In this case, `std::shared_ptr<Node>` will be automatically converted to `Output<Node>` if node has exactly one output port; otherwise, conversion raises an exception.
|
||||
|
||||
### ngraph::Node replacement
|
||||
|
||||
nGraph provides two ways for node replacement: via nGraph helper function and directly via port methods. We are going to review both of them.
|
||||
|
||||
Let's start with nGraph helper functions. The most popular function is `ngraph::replace_node(old_node, new_node)`.
|
||||
|
||||
We will review real replacement case where Negative operation is replaced with Multiply.
|
||||
|
||||
![ngraph_replace_node]
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:replace_node
|
||||
|
||||
`ngraph::replace_node` has a constraint that number of output ports for both of ops must be the same; otherwise, it raises an exception.
|
||||
|
||||
|
||||
The alternative way to do the same replacement is the following:
|
||||
```cpp
|
||||
// All neg->output(0) consumers will be moved to mul->output(0) port
|
||||
neg->output(0).replace(mul->output(0));
|
||||
```
|
||||
|
||||
Another transformation example is insertion.
|
||||
|
||||
![ngraph_insert_node]
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:insert_node
|
||||
|
||||
The alternative way to the insert operation is to make a node copy and use `replace_node`:
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:insert_node_with_copy
|
||||
|
||||
### ngraph::Node elimination
|
||||
|
||||
Another type of node replacement is its elimination.
|
||||
|
||||
To eliminate operation, nGraph has special method that considers all limitations related to InferenceEngine.
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:eliminate_node
|
||||
|
||||
`replace_output_update_name` in case of successful replacement it automatically preserves friendly name and runtime info.
|
||||
|
||||
|
||||
## Transformation writing essentials <a name="transformation_writing_essentials"></a>
|
||||
|
||||
When developing a transformation, you need to follow these transformation rules:
|
||||
|
||||
### Operation Set (OpSet)
|
||||
|
||||
Use the latest version of OpSet in your transformation. An exception is ConvertOpSetXToOpSetY transformations, where you must use operations from OpSetX and OpSetY.
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:include
|
||||
|
||||
### Dynamic Shape and Rank
|
||||
|
||||
nGraph has two types for shape representation:
|
||||
`ngraph::Shape` - represents static shape.
|
||||
`ngraph::PartialShape` - represents dynamic shape. It means that rank or some of dimensions are dynamic (undefined).
|
||||
`ngraph::PartialShape` can be converted to `ngraph::Shape` using the `get_shape()` method if all dimensions are static; otherwise, conversion raises an exception.
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:shape
|
||||
|
||||
But in most cases before getting static shape using `get_shape()` method, you need to check that shape is static.
|
||||
|
||||
Also if your transformation requires only input shape rank or particular dimension value, please do not use the `get_shape()` method. See the example below demonstrating how to avoid using `get_shape()`
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:shape_check
|
||||
|
||||
Not using `get_shape()` method makes your transformation more flexible and applicable for more cases.
|
||||
|
||||
###3. Friendly Names
|
||||
|
||||
Each `ngraph::Node` has a unique name (used for nGraph internals) and a friendly name. In transformations we care only about friendly name because it represents the name from intermediate representation (IR).
|
||||
Also friendly name is used as output tensor name (until we do not have other way to represent output tensor name) and user code that requests intermediate outputs based on these names.
|
||||
To avoid loosing friendly name when replacing node with other node or subgraph, set the original friendly name to the latest node in replacing subgraph. See the example below.
|
||||
|
||||
```cpp
|
||||
// Replace Div operation with Power and Multiply sub-graph and set original friendly name to Multiply operation
|
||||
auto pow = std::make_shared<ngraph::opset1::Power>(div->input(1).get_source_output(),
|
||||
op::Constant::create(div->get_input_element_type(1), Shape{1}, {-1}));
|
||||
auto mul = std::make_shared<ngraph::opset1::Multiply>(div->input(0).get_source_output(), pow);
|
||||
mul->set_friendly_name(div->get_friendly_name());
|
||||
ngraph::replace_node(div, mul);
|
||||
```
|
||||
|
||||
In more advanced cases, when replaced operation has several outputs and we add additional consumers to its outputs, we make a decision how to set friendly name by arrangement.
|
||||
|
||||
###4. Runtime Info
|
||||
|
||||
Runtime info is a map `std::map<std::string, std::shared_ptr<Variant>>` located inside `ngraph::Node` class. It represents additional attributes in `ngraph::Node`.
|
||||
These attributes can be set by users or by plugins and when executing transformation that changes `ngraph::Function` we need to preserve these attributes as they will not be automatically propagated.
|
||||
In most cases, transformations have the following types: 1:1 (replace node with another node), 1:N (replace node with a sub-graph), N:1 (fuse sub-graph into a single node), N:M (any other transformation).
|
||||
Currently, there is no mechanism that automatically detects transformation types, so we need to propagate this runtime information manually. See the examples below.
|
||||
|
||||
```cpp
|
||||
// Replace Transpose with Reshape operation (1:1)
|
||||
ngraph::copy_runtime_info(transpose, reshape);
|
||||
```
|
||||
|
||||
```cpp
|
||||
// Replace Div operation with Power and Multiply sub-graph (1:N)
|
||||
ngraph::copy_runtime_info(div, {pow, mul});
|
||||
```
|
||||
|
||||
```cpp
|
||||
// Fuse Convolution with Add operation (N:1)
|
||||
ngraph::copy_runtime_info({conv, bias}, {conv_ie});
|
||||
```
|
||||
|
||||
```cpp
|
||||
// Any other transformation that replaces one sub-graph with another sub-graph (N:M)
|
||||
ngraph::copy_runtime_info({a, b, c}, {e, f});
|
||||
```
|
||||
|
||||
When transformation has multiple fusions or decompositions, `ngraph::copy_runtime_info` must be called multiple times for each case.
|
||||
|
||||
> **Note**: copy_runtime_info removes rt_info from destination nodes. If you want to keep it, you need to specify them in source nodes like this: copy_runtime_info({a, b, c}, {a, b})
|
||||
|
||||
###5. Constant Folding
|
||||
|
||||
If your transformation inserts constant sub-graphs that need to be folded, do not forget to use `ngraph::pass::ConstantFolding()` after your transformation or call constant folding directly for operation.
|
||||
The example below shows how constant subgraph can be constructed.
|
||||
|
||||
```cpp
|
||||
// After ConstantFolding pass Power will be replaced with Constant
|
||||
auto pow = std::make_shared<ngraph::opset3::Power>(
|
||||
opset3::Constant::create(element::f32, Shape{1}, {2})
|
||||
opset3::Constant::create(element::f32, Shape{1}, {3}));
|
||||
auto mul = std::make_shared<ngraph::opset3::Multiply>(input /* not constant input */, pow);
|
||||
```
|
||||
|
||||
Manual constant folding is more preferable than `ngraph::pass::ConstantFolding()` because it is much faster.
|
||||
|
||||
Below you can find an example of manual constant folding:
|
||||
|
||||
@snippet src/template_pattern_transformation.cpp manual_constant_folding
|
||||
|
||||
## Common mistakes in transformations <a name="common_mistakes"></a>
|
||||
|
||||
In transformation development process:
|
||||
|
||||
* Do not use deprecated nGraph API. Deprecated methods has the `NGRAPH_DEPRECATED` macros in its definition.
|
||||
* Do not pass `shared_ptr<Node>` as an input for other node if type of node is unknown or it has multiple outputs. Use explicit output port.
|
||||
* If you replace node with another node that produces different shape, remember that new shape will not be propagated until the first `validate_nodes_and_infer_types` call for `ngraph::Function`. If you are using `pass::Manager`, it will automatically call this method after each transformation execution.
|
||||
* Do not forget to call the `ngraph::ConstantFolding` pass if your transformation creates constant subgraphs.
|
||||
* Use latest OpSet if you are not developing downgrade transformation pass.
|
||||
* When developing a callback for `ngraph::pass::MatcherPass`, do not change nodes that come after the root node in topological order.
|
||||
|
||||
## Using pass manager <a name="using_pass_manager"></a>
|
||||
|
||||
`ngraph::pass::Manager` is a container class that can store the list of transformations and execute them. The main idea of this class is to have high-level representation for grouped list of transformations.
|
||||
It can register and apply any [transformation types](#transformations_types) on function.
|
||||
In addition, `ngraph::pass::Manager` has extended debug capabilities (find more information in the [how to debug transformations](#how_to_debug_transformations) section).
|
||||
|
||||
The example below shows basic usage of `ngraph::pass::Manager`
|
||||
|
||||
@snippet src/template_pattern_transformation.cpp matcher_pass:manager3
|
||||
|
||||
Another example shows how multiple matcher passes can be united into single GraphRewrite.
|
||||
|
||||
@snippet src/template_pattern_transformation.cpp matcher_pass:manager2
|
||||
|
||||
## How to debug transformations <a name="how_to_debug_transformations"></a>
|
||||
|
||||
The most popular tool for transformations debugging is the `ngraph::pass::VisualizeTree` transformation, which visualizes ngraph::Function.
|
||||
|
||||
Usage example:
|
||||
|
||||
@snippet example_ngraph_utils.cpp ngraph:visualize
|
||||
|
||||
`ngraph::pass::VisualizeTree` can be parametrized via environment variables:
|
||||
|
||||
```
|
||||
NGRAPH_VISUALIZE_TREE_OUTPUT_SHAPES=1 - visualize shapes
|
||||
NGRAPH_VISUALIZE_TREE_OUTPUT_TYPES=1 - visualize types
|
||||
```
|
||||
|
||||
> **Note**: current VisualTree does not have user-friendly interface and it will be changed in the nearest future. The intention is to move visualization abilities inside transformations.
|
||||
|
||||
If you are using `ngraph::pass::Manager` to run sequence of transformations, you can get additional debug capabilities by using the following environment variables:
|
||||
|
||||
```
|
||||
NGRAPH_PROFILE_PASS_ENABLE=1 - enables performance measurement for each transformation and prints execution status
|
||||
NGRAPH_ENABLE_VISUALIZE_TRACING=1 - enables visualization after each transformation. By default, it saves dot and svg files.
|
||||
```
|
||||
|
||||
> **Note**: Make sure that you have dot installed on your machine; otherwise, it will silently save only dot file without svg file.
|
||||
|
||||
## Disabling/Enabling specific transformations for plugin X <a name="disabling_transformation"></a>
|
||||
|
||||
This topic is mostly related to conversion to legacy opset and plugins that are based on CNNNetwork. But this mechanism still can be applied for other cases.
|
||||
Let's suppose that plugin X enabled the `opset3::StridedSlice` operation support and you want to disable the `ngraph::pass::ConvertStridedSliceToCrop` transformation for plugin X.
|
||||
To do this, you need to create a callback on plugin side and pass it to transformation. And also you need to update particular transformation to use this callback.
|
||||
|
||||
```cpp
|
||||
// Update callback to be able to use m_transformation_callback if this transformation based on GraphRewrite.
|
||||
ngraph::graph_rewrite_callback callback = [this](pattern::Matcher &m) {
|
||||
...
|
||||
}
|
||||
|
||||
// Use transformation_callback not to execute transformation if callback returns true for given node
|
||||
if (m_transformation_callback(node)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Implement transformation callback and pass it directly to transformation or pass::Manager
|
||||
const auto transformations_callback = [](const std::shared_ptr<const ::ngraph::Node> &node) -> bool {
|
||||
return std::dynamic_pointer_cast<const ::ngraph::opset3::StridedSlice>(node) != nullptr;
|
||||
};
|
||||
|
||||
// Register transformation and pass callback to pass::Manager
|
||||
ngraph::pass::Manager manager;
|
||||
manager.register_pass<ngraph::pass::ConvertStridedSliceToCrop>();
|
||||
// pass::Manager will set callback to all reistered transformations automatically
|
||||
manager.set_callback(transformations_callback);
|
||||
manager.run_passes(f);
|
||||
```
|
||||
|
||||
## Transformations testing <a name="transformations_testing"></a>
|
||||
|
||||
If you are developing new transformation inside plugin, you need to add test into the `template_plugin/tests/functional/transformations` folder.
|
||||
We have two types of tests: nGraph reader tests located in `inference-engine/tests/functional/inference_engine/ngraph_reader` and transformation tests located in `inference-engine/tests/functional/inference_engine/transformations`
|
||||
Reader tests are IR based and test end-to-end conversion from IR to CNNNetwork. Transformation tests test single ngraph transformations or low-level functions that are used inside transformations.
|
||||
|
||||
The basic transformation test looks like this:
|
||||
|
||||
@snippet tests/functional/transformations/template_transformations_test.cpp transformation:test
|
||||
|
||||
|
||||
[ngraph_replace_node]: ../images/ngraph_replace_node.png
|
||||
[ngraph_insert_node]: ../images/ngraph_insert_node.png
|
||||
[transformations_structure]: ../images/transformations_structure.png
|
||||
[register_new_node]: ../images/register_new_node.png
|
||||
[graph_rewrite_execution]: ../images/graph_rewrite_execution.png
|
||||
[graph_rewrite_efficient_search]: ../images/graph_rewrite_efficient_search.png
|
||||
39
docs/nGraph_DG/nGraph_basic_concepts.md
Normal file
39
docs/nGraph_DG/nGraph_basic_concepts.md
Normal file
@@ -0,0 +1,39 @@
|
||||
# nGraph Basic Concepts {#openvino_docs_nGraph_DG_basic_concepts}
|
||||
|
||||
The nGraph represents neural networks in uniform format. User can create different operations and combined their to one `ngraph::Function`.
|
||||
|
||||
## nGraph Function and Graph Representation <a name="ngraph_function"></a>
|
||||
|
||||
nGraph function is a very simple thing: it stores shared pointers to `ngraph::op::Result` and `ngraph::op::Parameter` operations that are inputs and outputs of the graph.
|
||||
All other operations hold each other via shared pointers: child operation holds its parent (hard link). If operation has no consumers and it's not Result operation
|
||||
(shared pointer counter is zero) then it will be destructed and won't be accessible anymore. Each operation in `ngraph::Function` has a `std::shared_ptr<ngraph::Node>` type.
|
||||
|
||||
For details on how to build an nGraph Function, see the [Build nGraph Function](./build_function.md) page.
|
||||
|
||||
## Operations
|
||||
|
||||
`ngraph::Op` represents any abstract operations in the nGraph representation. You need to use this class to create [custom operations](../IE_DG/Extensibility_DG/AddingNGraphOps.md).
|
||||
|
||||
## Operation Sets
|
||||
|
||||
Operation set represents the set of some nGraph operations. `nGraph::Opset` is a class which provide a functionality to work with operation sets.
|
||||
Custom operation set should be created to support custom operation. Please read [Extensibility DevGuide](../IE_DG/Extensibility_DG/Intro.md) for more details.
|
||||
|
||||
## Static and Partial Shapes
|
||||
|
||||
nGraph has two types for shape representation:
|
||||
|
||||
* `ngraph::Shape` - Represents static (fully defined) shapes.
|
||||
|
||||
* `ngraph::PartialShape` - Represents dynamic shapes. That means that the rank or some of dimensions are dynamic (undefined). `ngraph::PartialShape` can be converted to `ngraph::Shape` using the `get_shape()` method if all dimensions are static; otherwise the conversion raises an exception.
|
||||
|
||||
For examples, see the Dynamic Shape and Rank section in the [Overview of Transformations API](./nGraphTransformation.md).
|
||||
|
||||
|
||||
## Transformation API
|
||||
|
||||
nGraph transformation API allows you to manipulate the graph represented by `nGraph::Function`. For more details, see the [Overview of Transformations API](./nGraphTransformation.md) section.
|
||||
|
||||
## Pattern Matcher
|
||||
|
||||
For more details, see the Pattern Matching section in the [Overview of Transformations API](./nGraphTransformation.md).
|
||||
16
docs/nGraph_DG/nGraph_debug_capabilities.md
Normal file
16
docs/nGraph_DG/nGraph_debug_capabilities.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# nGraph Debug Capabilities {#openvino_docs_nGraph_DG_Debug_capabilities}
|
||||
|
||||
nGraph representation provides an API to get detailed information about the graph structure.
|
||||
|
||||
To receive additional messages about applied graph modifications, rebuild the nGraph library with
|
||||
the `-DNGRAPH_DEBUG_ENABLE=ON` option.
|
||||
|
||||
To visualize the nGraph function to the xDot format or to an image file, use the
|
||||
`ngraph::pass::VisualizeTree` graph transformation pass:
|
||||
```cpp
|
||||
#include <ngraph/pass/visualize_tree.hpp>
|
||||
|
||||
std::shared_ptr<ngraph::Function> nGraph;
|
||||
...
|
||||
ngraph::pass::VisualizeTree("after.png").run_on_function(nGraph); // Visualize the nGraph function to an image
|
||||
```
|
||||
11
docs/nGraph_DG/nGraph_dg.md
Normal file
11
docs/nGraph_DG/nGraph_dg.md
Normal file
@@ -0,0 +1,11 @@
|
||||
# nGraph Developer Guide {#openvino_docs_nGraph_DG_DevGuide}
|
||||
|
||||
## Table of Contents
|
||||
|
||||
* [Introduction to the nGraph](intro.md)
|
||||
* [Basic Concepts](./nGraph_basic_concepts.md)
|
||||
* [Operation Sets](../ops/opset.md)
|
||||
* [Graph Constructing](./build_function.md)
|
||||
* [Transformations](./nGraphTransformation.md)
|
||||
* [Debug Capabilities](./nGraph_debug_capabilities.md)
|
||||
* [Python API](./nGraph_Python_API.md)
|
||||
Reference in New Issue
Block a user