IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

MO dev guide refactoring (#3266) (#3595)

Browse Source 
* Release mo dev guide refactoring (#3266)

* Updated MO extension guide

* Minor change and adding svg images

* Added additional information about operation extractors. Fixed links and markdown issues

* Added missing file with information about Caffe Python layers and image for MO transformations dependencies graph

* Added section with common graph transformations attributes and diagram with anchor transformations. Added list of available front phase transformations

* Added description of front-phase transformations except the scope-defined and points defined. Removed legacy document and examples for such transformations.

* Added sections about node name pattern defined front phase transformations. Copy-pasted the old one for the points defined front transformation

* Added description of the rest of front transformations and and all middle and back phase transformations

* Refactored Legacy_Mode_for_Caffe_Custom_Layers and updated the Customize_Model_Optimizer with information about extractors order

* Added TOC for the MO Dev guide document and updated SVG images with PNG ones

* Fixed broken link. Removed redundant image

* Fixed broken links

* Added information about attributes 'run_not_recursively', 'force_clean_up' and 'force_shape_inference' of the transformation

* Code review comments

* Added a section about `Port`s

* Extended Ports description with examples

* Added information about Connections

* Updated MO README.md and removed a lot of redundant and misleading information

* Updates to the Customize_Model_Optimizer.md

* More updates to the Customize_Model_Optimizer.md

* Final updates for the Customize_Model_Optimizer.md

* Fixed some broken links

* More fixed links

* Refactored Custom Layers Guide: removed legacy and incorrect text, added up-to-date.

* Draft implementation of the Custom layer guide example for the MO part

* Fixed broken links using #. Change layer->operation in extensibility documents

* Updated Custom operation guide with IE part

* Fixed broken links and minor updates to the Custom Operations Guide

* Updating links

* Layer->Operation

* Moved FFTOp implementation to the template extension

* Update the CMake for template_extension to build the FFT op conditionally

* Fixed template extension compilation

* Fixed CMake for template extension

* Fixed broken snippet

* Added mri_demo script and updated documentation

* One more compilation error fix

* Added missing header for a demo file

* Added reference to OpenCV

* Fixed unit test for the template extension

* Fixed typos in the template extension

* Fixed compilation of template extension for case when ONNX importer is disabled

Co-authored-by: Alexander Zhogov <alexander.zhogov@intel.com>
This commit is contained in:
Evgeny Lazarev 2021-01-14 16:28:53 +03:00 committed by GitHub
parent 08afa4fd97
commit dbad8809bf
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
46 changed files with 2391 additions and 2081 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

463
docs/HOWTO/Custom_Layers_Guide.md
View File

@ -1,200 +1,371 @@
# Custom Layers Guide {#openvino_docs_HOWTO_Custom_Layers_Guide}
# Custom Operations Guide {#openvino_docs_HOWTO_Custom_Layers_Guide}
The Intel® Distribution of OpenVINO™ toolkit supports neural network model layers in multiple frameworks including TensorFlow*, Caffe*, MXNet*, Kaldi* and ONNX*. The list of known layers is different for each of the supported frameworks. To see the layers supported by your framework, refer to [supported frameworks](../MO_DG/prepare_model/Supported_Frameworks_Layers.md).
The Intel® Distribution of OpenVINO™ toolkit supports neural network models trained with multiple frameworks including
TensorFlow*, Caffe*, MXNet*, Kaldi* and ONNX* file format. The list of supported operations (layers) is different for
each of the supported frameworks. To see the operations supported by your framework, refer to
[Supported Framework Layers](../MO_DG/prepare_model/Supported_Frameworks_Layers.md).
Custom layers are layers that are not included in the list of known layers. If your topology contains any layers that are not in the list of known layers, the Model Optimizer classifies them as custom.
Custom operations are operations that are not included in the list of known operations. If your model contains any
operation that is not in the list of known operations, the Model Optimizer is not able to generate an Intermediate
Representation (IR) for this model.
This guide illustrates the workflow for running inference on topologies featuring custom layers, allowing you to plug in your own implementation for existing or completely new layers.
For a step-by-step example of creating and executing a custom layer, see the [Custom Layer Implementation Tutorials for Linux and Windows.](https://github.com/david-drew/OpenVINO-Custom-Layers/tree/master/2019.r2.0)
This guide illustrates the workflow for running inference on topologies featuring custom operations, allowing you to
plug in your own implementation for existing or completely new operation.
## Terms used in this guide
> **NOTE:** *Layer* — The legacy term for an *operation* which came from Caffe\* framework. Currently it is not used.
> Refer to the [Deep Learning Network Intermediate Representation and Operation Sets in OpenVINO™](../MO_DG/IR_and_opsets.md)
> for more information on the topic.
- *Layer* — The abstract concept of a math function that is selected for a specific purpose (relu, sigmoid, tanh, convolutional). This is one of a sequential series of building blocks within the neural network.
- *Kernel* — The implementation of a layer function, in this case, the math programmed (in C++ and Python) to perform the layer operation for target hardware (CPU or GPU).
- *Intermediate Representation (IR)* — Neural Network used only by the Inference Engine in OpenVINO abstracting the different frameworks and describing topology, layer parameters and weights.
The original format will be a supported framework such as TensorFlow, Caffe, or MXNet.
## Terms Used in This Guide
- *Model Extension Generator* — Generates template source code files for each of the extensions needed by the Model Optimizer and the Inference Engine.
- *Intermediate Representation (IR)* — Neural Network used only by the Inference Engine in OpenVINO abstracting the
different frameworks and describing the model topology, operations parameters and weights.
- *Operation* — The abstract concept of a math function that is selected for a specific purpose. Operations supported by
OpenVINO™ are listed in the supported operation set provided in the [Available Operations Sets](../ops/opset.md).
Examples of the operations are: [ReLU](../ops/activation/ReLU_1.md), [Convolution](../ops/convolution/Convolution_1.md),
[Add](../ops/arithmetic/Add_1.md), etc.
- *Kernel* — The implementation of a operation function in the OpenVINO™ plugin, in this case, the math programmed (in
C++ and OpenCL) to perform the operation for a target hardware (CPU or GPU).
- *Inference Engine Extension* — Device-specific module implementing custom operations (a set of kernels).
## Custom Operation Support Overview
There are three steps to support inference of a model with custom operation(s):
1. Add support for a custom operation in the [Model Optimizer](../MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) so
the Model Optimizer can generate the IR with the operation.
2. Create an operation set and implement a custom nGraph operation in it as described in the
[Custom nGraph Operation](../IE_DG/Extensibility_DG/AddingNGraphOps.md).
3. Implement a customer operation in one of the [Inference Engine](../IE_DG/Deep_Learning_Inference_Engine_DevGuide.md)
plugins to support inference of this operation using a particular target hardware (CPU, GPU or VPU).
To see the operations that are supported by each device plugin for the Inference Engine, refer to the
[Supported Devices](../IE_DG/supported_plugins/Supported_Devices.md).
> **NOTE:** If a device doesn't support a particular operation, an alternative to creating a new operation is to target
> an additional device using the HETERO plugin. The [Heterogeneous Plugin](../IE_DG/supported_plugins/HETERO.md) may be
> used to run an inference model on multiple devices allowing the unsupported operations on one device to "fallback" to
> run on another device (e.g., CPU) that does support those operations.
### Custom Operation Support for the Model Optimizer
Model Optimizer model conversion pipeline is described in details in "Model Conversion Pipeline" section on the
[Model Optimizer Extensibility](../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md).
It is recommended to read that article first for a better understanding of the following material.
Model Optimizer provides extensions mechanism to support new operations and implement custom model transformations to
generate optimized IR. This mechanism is described in the "Model Optimizer Extensions" section on the
[Model Optimizer Extensibility](../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md).
Two types of the Model Optimizer extensions should be implemented to support custom operation at minimum:
1. Operation class for a new operation. This class stores information about the operation, its attributes, shape
inference function, attributes to be saved to an IR and some others internally used attributes. Refer to the
"Model Optimizer Operation" section on the
[Model Optimizer Extensibility](../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md) for the
detailed instruction on how to implement it.
2. Operation attributes extractor. The extractor is responsible for parsing framework-specific representation of the
operation and uses corresponding operation class to update graph node attributes with necessary attributes of the
operation. Refer to the "Operation Extractor" section on the
[Model Optimizer Extensibility](../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md) for the
detailed instruction on how to implement it.
> **NOTE:** In some cases you may need to implement some transformation to support the operation. This topic is covered
> in the "Graph Transformation Extensions" section on the
> [Model Optimizer Extensibility](../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md).
## Custom Operations Extensions for the Inference Engine
Inference Engine provides extensions mechanism to support new operations. This mechanism is described in the
[Inference Engine Extensibility Mechanism](../IE_DG/Extensibility_DG/Intro.md).
Each device plugin includes a library of optimized implementations to execute known operations which must be extended to
execute a custom operation. The custom operation extension is implemented according to the target device:
- Custom Operation CPU Extension
- A compiled shared library (`.so`, `.dylib` or `.dll`) needed by the CPU Plugin for executing the custom operation
on a CPU. Refer to the [How to Implement Custom CPU Operations](../IE_DG/Extensibility_DG/CPU_Kernel.md) for more
details.
- Custom Operation GPU Extension
- OpenCL source code (.cl) for the custom operation kernel that will be compiled to execute on the GPU along with a
operation description file (.xml) needed by the GPU Plugin for the custom operation kernel. Refer to the
[How to Implement Custom GPU Operations](../IE_DG/Extensibility_DG/GPU_Kernel.md) for more details.
- Custom Operation VPU Extension
- OpenCL source code (.cl) for the custom operation kernel that will be compiled to execute on the VPU along with a
operation description file (.xml) needed by the VPU Plugin for the custom operation kernel. Refer to the
[How to Implement Custom Operations for VPU](../IE_DG/Extensibility_DG/VPU_Kernel.md) for more details.
Also, it is necessary to implement nGraph custom operation according to the
[Custom nGraph Operation](../IE_DG/Extensibility_DG/AddingNGraphOps.md) so the Inference Engine can read an IR with this
operation and correctly infer output tensors shape and type.
## Enabling Magnetic Resonance Image Reconstruction Model
This chapter provides a step-by-step instruction on how to enable the magnetic resonance image reconstruction model
implemented in the [repository](https://github.com/rmsouza01/Hybrid-CS-Model-MRI/) using a custom operation on CPU. The
example is prepared for a model generated from the repository with hash `2ede2f96161ce70dcdc922371fe6b6b254aafcc8`.
### Download and Convert the Model to a Frozen TensorFlow\* Model Format
The original pre-trained model is provided in the hdf5 format which is not supported by OpenVINO directly and needs to
be converted to TensorFlow\* frozen model format first.
1. Download repository `https://github.com/rmsouza01/Hybrid-CS-Model-MRI`:<br
```bash
git clone https://github.com/rmsouza01/Hybrid-CS-Model-MRI
git checkout 2ede2f96161ce70dcdc922371fe6b6b254aafcc8
```
2. Convert pre-trained `.hdf5` to a frozen `.pb` graph using the following script (tested with TensorFlow==1.15.0 and
Keras==2.2.4) which should be executed from the root of the cloned repository:<br>
```py
import keras as K
import numpy as np
import Modules.frequency_spatial_network as fsnet
import tensorflow as tf
under_rate = '20'
stats = np.load("Data/stats_fs_unet_norm_" + under_rate + ".npy")
var_sampling_mask = np.load("Data/sampling_mask_" + under_rate + "perc.npy")
model = fsnet.wnet(stats[0], stats[1], stats[2], stats[3], kshape = (5,5), kshape2=(3,3))
model_name = "Models/wnet_" + under_rate + ".hdf5"
model.load_weights(model_name)
inp = np.random.standard_normal([1, 256, 256, 2]).astype(np.float32)
np.save('inp', inp)
sess = K.backend.get_session()
sess.as_default()
graph_def = sess.graph.as_graph_def()
graph_def = tf.graph_util.convert_variables_to_constants(sess, graph_def, ['conv2d_44/BiasAdd'])
with tf.gfile.FastGFile('wnet_20.pb', 'wb') as f:
f.write(graph_def.SerializeToString())
```
- *Inference Engine Extension* — Device-specific module implementing custom layers (a set of kernels).
As a result the TensorFlow\* frozen model file "wnet_20.pb" is generated.
### Convert the Frozen TensorFlow\* Model to Intermediate Representation
## Custom Layer Overview
The [Model Optimizer](../MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) searches the list of known layers for each layer contained in the input model topology before building the model's internal representation, optimizing the model, and producing the Intermediate Representation files.
The [Inference Engine](../IE_DG/Deep_Learning_Inference_Engine_DevGuide.md) loads the layers from the input model IR files into the specified device plugin, which will search a list of known layer implementations for the device. If your topology contains layers that are not in the list of known layers for the device, the Inference Engine considers the layer to be unsupported and reports an error. To see the layers that are supported by each device plugin for the Inference Engine, refer to the [Supported Devices](../IE_DG/supported_plugins/Supported_Devices.md) documentation.
<br>
> **NOTE:** If a device doesn't support a particular layer, an alternative to creating a new custom layer is to target an additional device using the HETERO plugin. The [Heterogeneous Plugin](../IE_DG/supported_plugins/HETERO.md) may be used to run an inference model on multiple devices allowing the unsupported layers on one device to "fallback" to run on another device (e.g., CPU) that does support those layers.
## Custom Layer Implementation Workflow
When implementing a custom layer for your pre-trained model in the Intel® Distribution of OpenVINO™ toolkit, you will need to add extensions to both the Model Optimizer and the Inference Engine.
## Custom Layer Extensions for the Model Optimizer
The following figure shows the basic processing steps for the Model Optimizer highlighting the two necessary custom layer extensions, the Custom Layer Extractor and the Custom Layer Operation.
![](img/MO_extensions_flow.png)
The Model Optimizer first extracts information from the input model which includes the topology of the model layers along with parameters, input and output format, etc., for each layer. The model is then optimized from the various known characteristics of the layers, interconnects, and data flow which partly comes from the layer operation providing details including the shape of the output for each layer. Finally, the optimized model is output to the model IR files needed by the Inference Engine to run the model.
The Model Optimizer starts with a library of known extractors and operations for each [supported model framework](../MO_DG/prepare_model/Supported_Frameworks_Layers.md) which must be extended to use each unknown custom layer. The custom layer extensions needed by the Model Optimizer are:
- Custom Layer Extractor
- Responsible for identifying the custom layer operation and extracting the parameters for each instance of the custom layer. The layer parameters are stored per instance and used by the layer operation before finally appearing in the output IR. Typically the input layer parameters are unchanged, which is the case covered by this tutorial.
- Custom Layer Operation
- Responsible for specifying the attributes that are supported by the custom layer and computing the output shape for each instance of the custom layer from its parameters. <br> The `--mo-op` command-line argument shown in the examples below generates a custom layer operation for the Model Optimizer.
## Custom Layer Extensions for the Inference Engine
The following figure shows the basic flow for the Inference Engine highlighting two custom layer extensions for the CPU and GPU Plugins, the Custom Layer CPU extension and the Custom Layer GPU Extension.
![](img/IE_extensions_flow.png)
Each device plugin includes a library of optimized implementations to execute known layer operations which must be extended to execute a custom layer. The custom layer extension is implemented according to the target device:
- Custom Layer CPU Extension
- A compiled shared library (.so or .dll binary) needed by the CPU Plugin for executing the custom layer on the CPU.
- Custom Layer GPU Extension
- OpenCL source code (.cl) for the custom layer kernel that will be compiled to execute on the GPU along with a layer description file (.xml) needed by the GPU Plugin for the custom layer kernel.
## Model Extension Generator
Using answers to interactive questions or a *.json* configuration file, the Model Extension Generator tool generates template source code files for each of the extensions needed by the Model Optimizer and the Inference Engine. To complete the implementation of each extension, the template functions may need to be edited to fill-in details specific to the custom layer or the actual custom layer functionality itself.
### Command-line
The Model Extension Generator is included in the Intel® Distribution of OpenVINO™ toolkit installation and is run using the command (here with the "--help" option):
Firstly, open the model in the TensorBoard or other TensorFlow* model visualization tool. The model supports dynamic
batch dimension because the value for the batch dimension is not hardcoded in the model. Model Optimizer need to set all
dynamic dimensions to some specific value to create the IR, therefore specify the command line parameter `-b 1` to set
the batch dimension equal to 1. The actual batch size dimension can be changed at runtime using the Inference Engine API
described in the [Using Shape Inference](../IE_DG/ShapeInference.md). Also refer to
[Converting a Model Using General Conversion Parameters](../MO_DG/prepare_model/convert_model/Converting_Model_General.md)
and [Convert Your TensorFlow* Model](../MO_DG/prepare_model/convert_model/Convert_Model_From_TensorFlow.md)
for more details and command line parameters used for the model conversion.
```bash
python3 /opt/intel/openvino/deployment_tools/tools/extension_generator/extgen.py new --help
./<MO_INSTALL_DIR>/mo.py --input_model <PATH_TO_MODEL>/wnet_20.pb -b 1
```
where the output will appear similar to:
```
usage: You can use any combination of the following arguments:
Arguments to configure extension generation in the interactive mode:
optional arguments:
-h, --help show this help message and exit
--mo-caffe-ext generate a Model Optimizer Caffe* extractor
--mo-mxnet-ext generate a Model Optimizer MXNet* extractor
--mo-tf-ext generate a Model Optimizer TensorFlow* extractor
--mo-op generate a Model Optimizer operation
--ie-cpu-ext generate an Inference Engine CPU extension
--ie-gpu-ext generate an Inference Engine GPU extension
--output_dir OUTPUT_DIR
set an output directory. If not specified, the current
directory is used by default.
Model Optimizer produces the following error:
```bash
[ ERROR ] List of operations that cannot be converted to Inference Engine IR:
[ ERROR ] Complex (1)
[ ERROR ] lambda_2/Complex
[ ERROR ] IFFT2D (1)
[ ERROR ] lambda_2/IFFT2D
[ ERROR ] ComplexAbs (1)
[ ERROR ] lambda_2/Abs
[ ERROR ] Part of the nodes was not converted to IR. Stopped.
```
The available command-line arguments are used to specify which extension(s) to generate templates for the Model Optimizer or Inference Engine. The generated extension files for each argument will appear starting from the top of the output directory as follows:
The error means that the Model Optimizer doesn't know how to handle 3 types of TensorFlow\* operations: "Complex",
"IFFT2D" and "ComplexAbs". In order to see more details about the conversion process run the model conversion with
additional parameter `--log_level DEBUG`. It is worth to mention the following lines from the detailed output:
Command-line Argument | Output Directory Location |
--------------------- | ------------------------------ |
`--mo-caffe-ext` | user_mo_extensions/front/caffe |
`--mo-mxnet-ext` | user_mo_extensions/front/mxnet |
`--mo-tf-ext` | user_mo_extensions/front/tf |
`--mo-op` | user_mo_extensions/ops |
`--ie-cpu-ext` | user_ie_extensions/cpu |
`--ie-gpu-ext` | user_ie_extensions/gpu |
```bash
[ INFO ] Called "tf_native_tf_node_infer" for node "lambda_2/Complex"
[ <TIMESTAMP> ] [ DEBUG ] [ tf:228 ] Added placeholder with name 'lambda_2/lambda_3/strided_slice_port_0_ie_placeholder'
[ <TIMESTAMP> ] [ DEBUG ] [ tf:228 ] Added placeholder with name 'lambda_2/lambda_4/strided_slice_port_0_ie_placeholder'
[ <TIMESTAMP> ] [ DEBUG ] [ tf:241 ] update_input_in_pbs: replace input 'lambda_2/lambda_3/strided_slice' with input 'lambda_2/lambda_3/strided_slice_port_0_ie_placeholder'
[ <TIMESTAMP> ] [ DEBUG ] [ tf:249 ] Replacing input '0' of the node 'lambda_2/Complex' with placeholder 'lambda_2/lambda_3/strided_slice_port_0_ie_placeholder'
[ <TIMESTAMP> ] [ DEBUG ] [ tf:241 ] update_input_in_pbs: replace input 'lambda_2/lambda_4/strided_slice' with input 'lambda_2/lambda_4/strided_slice_port_0_ie_placeholder'
[ <TIMESTAMP> ] [ DEBUG ] [ tf:249 ] Replacing input '1' of the node 'lambda_2/Complex' with placeholder 'lambda_2/lambda_4/strided_slice_port_0_ie_placeholder'
[ <TIMESTAMP> ] [ DEBUG ] [ tf:148 ] Inferred shape of the output tensor with index '0' of the node 'lambda_2/Complex': '[ 1 256 256]'
[ <TIMESTAMP> ] [ DEBUG ] [ infer:145 ] Outputs:
[ <TIMESTAMP> ] [ DEBUG ] [ infer:32 ] output[0]: shape = [ 1 256 256], value = <UNKNOWN>
[ <TIMESTAMP> ] [ DEBUG ] [ infer:129 ] --------------------
[ <TIMESTAMP> ] [ DEBUG ] [ infer:130 ] Partial infer for lambda_2/IFFT2D
[ <TIMESTAMP> ] [ DEBUG ] [ infer:131 ] Op: IFFT2D
[ <TIMESTAMP> ] [ DEBUG ] [ infer:132 ] Inputs:
[ <TIMESTAMP> ] [ DEBUG ] [ infer:32 ] input[0]: shape = [ 1 256 256], value = <UNKNOWN>
```
### Extension Workflow
This is a part of the log of the partial inference phase of the model conversion. See the "Partial Inference" section on
the [Model Optimizer Extensibility](../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md) for
more information about this phase. Model Optimizer inferred output shape for the unknown operation of type "Complex"
using a "fallback" to TensorFlow\*. However, it is not enough to generate the IR because Model Optimizer doesn't know
which attributes of the operation should be saved to IR. So it is necessary to implement Model Optimizer extensions to
support these operations.
The workflow for each generated extension follows the same basic steps:
Before going into the extension development it is necessary to understand what these unsupported operations do according
to the TensorFlow\* framework specification.
![](img/MEG_generic_flow.png)
* "Complex" - returns a tensor of complex type constructed from two real input tensors specifying real and imaginary
part of a complex number.
* "IFFT2D" - returns a tensor with inverse 2-dimensional discrete Fourier transform over the inner-most 2 dimensions of
an input.
* "ComplexAbs" - returns a tensor with absolute values of input tensor with complex numbers.
**Step 1: Generate:** Use the Model Extension Generator to generate the Custom Layer Template Files.
The part of the model with all three unsupported operations is depicted below:
**Step 2: Edit:** Edit the Custom Layer Template Files as necessary to create the specialized Custom Layer Extension Source Code.
![Unsupported sub-graph](img/unsupported_subgraph.png)
**Step 3: Specify:** Specify the custom layer extension locations to be used by the Model Optimizer or Inference Engine.
This model uses complex numbers during the inference but Inference Engine does not support tensors of this data type. So
it is necessary to find a way how to avoid using tensors of such a type in the model. Fortunately, the complex tensor
appear as a result of "Complex" operation, is used as input in the "IFFT2D" operation then is passed to "ComplexAbs"
which produces real value tensor as output. So there are just 3 operations consuming/producing complex tensors in the
model.
## Caffe\* Models with Custom Layers <a name="caffe-models-with-custom-layers"></a>
Let's design an OpenVINO operation "FFT" which get a single real number tensor describing the complex number and
produces a single real number tensor describing output complex tensor. This way the fact that the model uses complex
numbers is hidden inside the "FFT" operation implementation. The operation gets a tensor of shape `[N, H, W, 2]` and
produces the output tensor with the same shape, where the innermost dimension contains pairs of real numbers describing
the complex number (its real and imaginary part). As we will see further this operation will allow us to support the
model. The implementation of the Model Optimizer operation should be saved to `mo_extensions/ops/FFT.py` file:
If your Caffe\* model has custom layers:
@snippet FFT.py fft:operation
**Register the custom layers as extensions to the Model Optimizer**. For instructions, see [Extending Model Optimizer with New Primitives](../MO_DG/prepare_model/customize_model_optimizer/Extending_Model_Optimizer_with_New_Primitives.md). When your custom layers are registered as extensions, the Model Optimizer generates a valid and optimized Intermediate Representation. You will need a bit of Python\* code that lets the Model Optimizer;
The attribute `inverse` is a flag specifying type of the FFT to apply: forward or inverse.
- Generate a valid Intermediate Representation according to the rules you specified.
- Be independent from the availability of Caffe on your computer.
If your model contains Custom Layers, it is important to understand the internal workflow of the Model Optimizer. Consider the following example.
See the "Model Optimizer Operation" section on the
[Model Optimizer Extensibility](../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md) for the
detailed instruction on how to implement the operation.
**Example**:
Now it is necessary to implement extractor for the "IFFT2D" operation according to the
"Operation Extractor" section on the
[Model Optimizer Extensibility](../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md). The
following snippet provides two extractors: one for "IFFT2D", another one for "FFT2D", however only on of them is used
in this example. The implementation should be saved to the file `mo_extensions/front/tf/FFT_ext.py`.
The network has:
@snippet FFT_ext.py fft_ext:extractor
* One input layer (#1)
* One output Layer (#5)
* Three internal layers (#2, 3, 4)
> **NOTE:** The graph is in inconsistent state after extracting node attributes because according to original operation
> "IFFT2D" semantic it should have an input consuming a tensor of complex numbers, but the extractor instantiated an
> operation "FFT" which expects a real tensor with specific layout. But the inconsistency will be resolved during
> applying front phase transformations discussed below.
The custom and standard layer types are:
The output shape of the operation "AddV2" from the picture above is `[N, H, W, 2]`. Where the innermost dimension
contains pairs of real numbers describing the complex number (its real and imaginary part). The following "StridedSlice"
operations split the input tensor into 2 parts to get a tensor of real and a tensor of imaginary parts which are then
consumed with the "Complex" operation to produce a tensor of complex numbers. These "StridedSlice" and "Complex"
operations can be removed so the "FFT" operation will get a real value tensor encoding complex numbers. To achieve this
we implement the front phase transformation which searches for a pattern of two "StridedSlice" operations with specific
attributes producing data to "Complex" operation and removes it from the graph. Refer to the
"Pattern-Defined Front Phase Transformations" section on the
[Model Optimizer Extensibility](../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md) for more
information on how this type of transformation works. The code snippet should be saved to the file
`mo_extensions/front/tf/Complex.py`.
* Layers #2 and #5 are implemented as Model Optimizer extensions.
* Layers #1 and #4 are supported in Model Optimizer out-of-the box.
* Layer #3 is neither in the list of supported layers nor in extensions, but is specified in CustomLayersMapping.xml.
@snippet Complex.py complex:transformation
> **NOTE**: If any of the layers are not in one of three categories described above, the Model Optimizer fails with an appropriate message and a link to the corresponding question in [Model Optimizer FAQ](../MO_DG/prepare_model/Model_Optimizer_FAQ.md).
> **NOTE:** The graph is in inconsistent state because the "ComplexAbs" operation consumes complex value tensor but
> "FFT" produces real value tensor.
The general process is as shown:
Now lets implement a transformation which replace a "ComplexAbs" operation with a sub-graph of primitive operations
which calculate the result using the following formulae: \f$module(z) = \sqrt{real(z) \cdot real(z) + imag(z) \cdot imag(z)}\f$.
Original "IFFT2D" operation produces tensor of complex values, but the "FFT" operation produces a real value tensor with
the same format and shape as the input for the operation. So the input shape for the "ComplexAbs" will be `[N, H, W, 2]`
with the innermost dimension containing tuple with real and imaginary part of a complex number. In order to calculate
absolute values for the complex tensor we do the following:
1. Raise all elements in the power of 2.
2. Calculate a reduced sum over the innermost dimension.
3. Calculate a square root.
![Example custom layer network](img/mo_caffe_priorities.png)
<br>
The implementation should be saved to the file `mo_extensions/front/tf/ComplexAbs.py` and provided below:
**Step 1:** The example model is fed to the Model Optimizer that **loads the model** with the special parser built on top of the `caffe.proto` file. In case of failure, the Model Optimizer asks you to prepare the parser that can read the model. For more information, refer to the Model Optimizer, <a href="MO_FAQ.html#FAQ1">FAQ #1</a>.
@snippet ComplexAbs.py complex_abs:transformation
**Step 2:** The Model Optimizer **extracts the attributes of all layers** by going through the list of layers and attempting to find the appropriate extractor. In order of priority, the Model Optimizer checks if the layer is:
* A. Registered as a Model Optimizer extension
* B. Registered as a standard Model Optimizer layer
When the Model Optimizer finds a satisfying condition from the list above, it extracts the attributes according to the following rules:
* For A. - takes only the parameters specified in the extension
* For B. - takes only the parameters specified in the standard extractor
<br>
Now it is possible to convert the model using the following command line:
```bash
./<MO_INSTALL_DIR>/mo.py --input_model <PATH_TO_MODEL>/wnet_20.pb -b 1 --extensions mo_extensions/
```
**Step 3:** The Model Optimizer **calculates the output shape of all layers**. The logic is the same as it is for the priorities. **Important:** the Model Optimizer always takes the first available option.
The sub-graph corresponding to the originally non-supported one is depicted on the image below:
**Step 4:** The Model Optimizer **optimizes the original model and produces the two Intermediate Representation (IR) files in .xml and .bin**.
<br>
![Converted sub-graph](img/converted_subgraph.png)
## TensorFlow\* Models with Custom Layers <a name="Tensorflow-models-with-custom-layers"></a>
> **NOTE:** Model Optimizer performed conversion of the model from NHWC to NCHW layout that is why the dimension with
> the value 2 moved to another position.
You have two options for TensorFlow\* models with custom layers:
<br>
### Inference Engine Extension Implementation
Now it is necessary to implement the extension for the CPU plugin with operation "FFT" introduced previously. The code
below is based on the template extension described on the
[Inference Engine Extensibility Mechanism](../IE_DG/Extensibility_DG/Intro.md).
* **Register those layers as extensions to the Model Optimizer.** In this case, the Model Optimizer generates a valid and optimized Intermediate Representation.
* **If you have sub-graphs that should not be expressed with the analogous sub-graph in the Intermediate Representation, but another sub-graph should appear in the model, the Model Optimizer provides such an option.** This feature is helpful for many TensorFlow models. To read more, see [Sub-graph Replacement in the Model Optimizer](../MO_DG/prepare_model/customize_model_optimizer/Subgraph_Replacement_Model_Optimizer.md).
## MXNet\* Models with Custom Layers <a name="mxnet-models-with-custom-layers"></a>
#### CMake Build File
The first step is to create a CMake configuration file which builds the extension. The content of the "CMakeLists.txt"
file is the following:
There are two options to convert your MXNet* model that contains custom layers:
@snippet ../template_extension/CMakeLists.txt cmake:extension
1. Register the custom layers as extensions to the Model Optimizer. For instructions, see [Extending MXNet Model Optimizer with New Primitives](../MO_DG/prepare_model/customize_model_optimizer/Extending_MXNet_Model_Optimizer_with_New_Primitives.md). When your custom layers are registered as extensions, the Model Optimizer generates a valid and optimized Intermediate Representation. You can create Model Optimizer extensions for both MXNet layers with op `Custom` and layers which are not standard MXNet layers.
The CPU FFT kernel implementation uses OpenCV to perform the FFT that is why the extension library is linked with
"opencv_core" which comes with the OpenVINO.
2. If you have sub-graphs that should not be expressed with the analogous sub-graph in the Intermediate Representation, but another sub-graph should appear in the model, the Model Optimizer provides such an option. In MXNet the function is actively used for ssd models provides an opportunity to for the necessary subgraph sequences and replace them. To read more, see [Sub-graph Replacement in the Model Optimizer](../MO_DG/prepare_model/customize_model_optimizer/Subgraph_Replacement_Model_Optimizer.md).
#### Custom nGraph Operation "FFT" Implementation
The next step is to create the nGraph operation FFT. The header file "fft_op.hpp" has the following content:
## Kaldi\* Models with Custom Layers <a name="Kaldi-models-with-custom-layers"></a>
For information on converting your Kaldi* model containing custom layers see [Converting a Kaldi Model in the Model Optimizer Developer Guide](../MO_DG/prepare_model/convert_model/Convert_Model_From_Kaldi.md).
@snippet ../template_extension/fft_op.hpp fft_op:header
## ONNX\* Models with Custom Layers <a name="ONNX-models-with-custom-layers"></a>
For information on converting your ONNX* model containing custom layers see [Converting an ONNX Model in the Model Optimizer Developer Guide](../MO_DG/prepare_model/convert_model/Convert_Model_From_ONNX.md).
The operation has just one boolean attribute `inverse`. Implementation of the necessary nGraph operation functions are
in the "fft_op.cpp" file with the following content:
## Step-by-Step Custom Layers Tutorial
For a step-by-step walk-through creating and executing a custom layer, see [Custom Layer Implementation Tutorial for Linux and Windows.](https://github.com/david-drew/OpenVINO-Custom-Layers/tree/master/2019.r2.0)
@snippet ../template_extension/fft_op.cpp fft_op:implementation
Refer to the [Custom nGraph Operation](../IE_DG/Extensibility_DG/AddingNGraphOps.md) for more details.
#### CPU FFT Kernel Implementation
The operation implementation for CPU plugin uses OpenCV to perform the FFT. The header file "fft_kernel.hpp" has the
following content:
@snippet ../template_extension/fft_kernel.hpp fft_kernel:header
The "fft_kernel.cpp" with the implementation of the CPU has the following content:
@snippet ../template_extension/fft_kernel.cpp fft_kernel:implementation
Refer to the [How to Implement Custom CPU Operations](../IE_DG/Extensibility_DG/CPU_Kernel.md) for more details.
#### Extension Library Implementation
The last step is to create an extension library "extension.cpp" and "extension.hpp" which will include the FFT
operation for the CPU plugin. The code of the library is described in the [Extension Library](../IE_DG/Extensibility_DG/Extension.md).
### Building and Running the Custom Extension
In order to build the extension run the following:<br>
```bash
mkdir build && cd build
source /opt/intel/openvino/bin/setupvars.sh
cmake .. -DCMAKE_BUILD_TYPE=Release
make --jobs=$(nproc)
```
The result of this command is a compiled shared library (`.so`, `.dylib` or `.dll`). It should be loaded in the
application using `Core` class instance method `AddExtension` like this
`core.AddExtension(make_so_pointer<IExtension>(compiled_library_file_name), "CPU");`.
To test that the extension is implemented correctly we can run the "mri_reconstruction_demo.py" with the following content:
@snippet mri_reconstruction_demo.py mri_demo:demo
The script can be executed using the following command line:
```bash
python3 mri_reconstruction_demo.py \
-m <PATH_TO_IR>/wnet_20.xml \
-i <PATH_TO_SAMPLE_MRI_IMAGE>.npy \
-p <Hybrid-CS-Model-MRI_repo>/Data/sampling_mask_20perc.npy \
-l <PATH_TO_BUILD_DIR>/libtemplate_extension.so \
-d CPU
```
## Additional Resources
- Intel® Distribution of OpenVINO™ toolkit home page: [https://software.intel.com/en-us/openvino-toolkit](https://software.intel.com/en-us/openvino-toolkit)
- OpenVINO™ toolkit online documentation: [https://docs.openvinotoolkit.org](https://docs.openvinotoolkit.org)
- [Model Optimizer Developer Guide](../MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md)
- [Model Optimizer Extensibility](../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md)
- [Inference Engine Extensibility Mechanism](../IE_DG/Extensibility_DG/Intro.md)
- [Inference Engine Samples Overview](../IE_DG/Samples_Overview.md)
- [Overview of OpenVINO™ Toolkit Pre-Trained Models](@ref omz_models_intel_index)
@ -204,9 +375,7 @@ For a step-by-step walk-through creating and executing a custom layer, see [Cust
## Converting Models:
- [Convert Your Caffe* Model](../MO_DG/prepare_model/convert_model/Convert_Model_From_Caffe.md)
- [Convert Your Kaldi* Model](../MO_DG/prepare_model/convert_model/Convert_Model_From_Kaldi.md)
- [Convert Your TensorFlow* Model](../MO_DG/prepare_model/convert_model/Convert_Model_From_TensorFlow.md)
- [Convert Your MXNet* Model](../MO_DG/prepare_model/convert_model/Convert_Model_From_MxNet.md)
- [Convert Your ONNX* Model](../MO_DG/prepare_model/convert_model/Convert_Model_From_ONNX.md)

3
docs/HOWTO/img/IE_extensions_flow.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c2f362a39ae6c2af080e4f055b6fdba4954f918f85731545d1df3d687d9213d5
size 421056

3
docs/HOWTO/img/MEG_generic_flow.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cb5c700d003936779455353bfa4ed9432410c0975c46e2dfd30c6a1abccd1727
size 23320

3
docs/HOWTO/img/MO_extensions_flow.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:99d6b5146be85fa408dc5432883c3e2745cffe890133854a97dcf22f5c5962d4
size 47564

3
docs/HOWTO/img/converted_subgraph.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f7c8ab4f15874d235968471bcf876c89c795d601e69891208107b8b72aa58eb1
size 70014

3
docs/HOWTO/img/mo_caffe_priorities.png
View File

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0a4de6e502cae7542f1f311bcdbea6bb145f960f0d27d86a03160d1a60133778
size 301310

3
docs/HOWTO/img/unsupported_subgraph.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3d5ccf51fe1babb93d96d042494695a6a6e055d1f8ebf7eef5083d54d8987a23
size 58789

57
docs/HOWTO/mo_extensions/front/tf/Complex.py Normal file
View File

@ -0,0 +1,57 @@
"""
Copyright (C) 2018-2020 Intel Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
#! [complex:transformation]
import logging as log
import numpy as np
from mo.front.common.replacement import FrontReplacementSubgraph
from mo.graph.graph import Graph
class Complex(FrontReplacementSubgraph):
enabled = True
def pattern(self):
return dict(
nodes=[
('strided_slice_real', dict(op='StridedSlice')),
('strided_slice_imag', dict(op='StridedSlice')),
('complex', dict(op='Complex')),
],
edges=[
('strided_slice_real', 'complex', {'in': 0}),
('strided_slice_imag', 'complex', {'in': 1}),
])
@staticmethod
def replace_sub_graph(graph: Graph, match: dict):
strided_slice_real = match['strided_slice_real']
strided_slice_imag = match['strided_slice_imag']
complex_node = match['complex']
# make sure that both strided slice operations get the same data as input
assert strided_slice_real.in_port(0).get_source() == strided_slice_imag.in_port(0).get_source()
# identify the output port of the operation producing datat for strided slice nodes
input_node_output_port = strided_slice_real.in_port(0).get_source()
input_node_output_port.disconnect()
# change the connection so now all consumers of "complex_node" get data from input node of strided slice nodes
complex_node.out_port(0).get_connection().set_source(input_node_output_port)
#! [complex:transformation]

40
docs/HOWTO/mo_extensions/front/tf/ComplexAbs.py Normal file
View File

@ -0,0 +1,40 @@
"""
Copyright (C) 2018-2020 Intel Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
#! [complex_abs:transformation]
import numpy as np
from extensions.ops.elementwise import Pow
from extensions.ops.ReduceOps import ReduceSum
from mo.front.common.replacement import FrontReplacementOp
from mo.graph.graph import Graph, Node
from mo.ops.const import Const
class ComplexAbs(FrontReplacementOp):
op = "ComplexAbs"
enabled = True
def replace_op(self, graph: Graph, node: Node):
pow_2 = Const(graph, {'value': np.float32(2.0)}).create_node()
reduce_axis = Const(graph, {'value': np.int32(-1)}).create_node()
pow_0_5 = Const(graph, {'value': np.float32(0.5)}).create_node()
sq = Pow(graph, dict(name=node.in_node(0).name + '/sq', power=2.0)).create_node([node.in_node(0), pow_2])
sum = ReduceSum(graph, dict(name=sq.name + '/sum')).create_node([sq, reduce_axis])
sqrt = Pow(graph, dict(name=sum.name + '/sqrt', power=0.5)).create_node([sum, pow_0_5])
return [sqrt.id]
#! [complex_abs:transformation]

47
docs/HOWTO/mo_extensions/front/tf/FFT_ext.py Normal file
View File

@ -0,0 +1,47 @@
"""
Copyright (C) 2018-2020 Intel Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
# ! [fft_ext:extractor]
from ...ops.FFT import FFT
from mo.front.extractor import FrontExtractorOp
from mo.utils.error import Error
class FFT2DFrontExtractor(FrontExtractorOp):
op = 'FFT2D'
enabled = True
@classmethod
def extract(cls, node):
attrs = {
'inverse': 0
}
FFT.update_node_stat(node, attrs)
return cls.enabled
class IFFT2DFrontExtractor(FrontExtractorOp):
op = 'IFFT2D'
enabled = True
@classmethod
def extract(cls, node):
attrs = {
'inverse': 1
}
FFT.update_node_stat(node, attrs)
return cls.enabled
# ! [fft_ext:extractor]

40
docs/HOWTO/mo_extensions/ops/FFT.py Normal file
View File

@ -0,0 +1,40 @@
"""
Copyright (C) 2018-2020 Intel Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
#! [fft:operation]
from mo.front.common.partial_infer.elemental import copy_shape_infer
from mo.graph.graph import Node, Graph
from mo.ops.op import Op
class FFT(Op):
op = 'FFT'
enabled = False
def __init__(self, graph: Graph, attrs: dict):
super().__init__(graph, {
'type': self.op,
'op': self.op,
'version': 'custom_opset',
'inverse': None,
'in_ports_count': 1,
'out_ports_count': 1,
'infer': copy_shape_infer
}, attrs)
def backend_attrs(self):
return ['inverse']
#! [fft:operation]

119
docs/HOWTO/mri_reconstruction_demo.py Normal file
View File

@ -0,0 +1,119 @@
"""
Copyright (C) 2018-2020 Intel Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
#! [mri_demo:demo]
import numpy as np
import cv2 as cv
import argparse
import time
from openvino.inference_engine import IECore
def kspace_to_image(kspace):
assert(len(kspace.shape) == 3 and kspace.shape[-1] == 2)
fft = cv.idft(kspace, flags=cv.DFT_SCALE)
img = cv.magnitude(fft[:,:,0], fft[:,:,1])
return cv.normalize(img, dst=None, alpha=255, beta=0, norm_type=cv.NORM_MINMAX, dtype=cv.CV_8U)
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='MRI reconstrution demo for network from https://github.com/rmsouza01/Hybrid-CS-Model-MRI (https://arxiv.org/abs/1810.12473)')
parser.add_argument('-i', '--input', dest='input', help='Path to input .npy file with MRI scan data.')
parser.add_argument('-p', '--pattern', dest='pattern', help='Path to sampling mask in .npy format.')
parser.add_argument('-m', '--model', dest='model', help='Path to .xml file of OpenVINO IR.')
parser.add_argument('-l', '--cpu_extension', dest='cpu_extension', help='Path to extensions library with FFT implementation.')
parser.add_argument('-d', '--device', dest='device', default='CPU',
help='Optional. Specify the target device to infer on; CPU, '
'GPU, HDDL or MYRIAD is acceptable. For non-CPU targets, '
'HETERO plugin is used with CPU fallbacks to FFT implementation. '
'Default value is CPU')
args = parser.parse_args()
xml_path = args.model
assert(xml_path.endswith('.xml'))
bin_path = xml_path[:xml_path.rfind('.xml')] + '.bin'
ie = IECore()
ie.add_extension(args.cpu_extension, "CPU")
net = ie.read_network(xml_path, bin_path)
device = 'CPU' if args.device == 'CPU' else ('HETERO:' + args.device + ',CPU')
exec_net = ie.load_network(net, device)
# Hybrid-CS-Model-MRI/Data/stats_fs_unet_norm_20.npy
stats = np.array([2.20295299e-01, 1.11048916e+03, 4.16997984e+00, 4.71741395e+00], dtype=np.float32)
# Hybrid-CS-Model-MRI/Data/sampling_mask_20perc.npy
var_sampling_mask = np.load(args.pattern) # TODO: can we generate it in runtime?
print('Sampling ratio:', 1.0 - var_sampling_mask.sum() / var_sampling_mask.size)
data = np.load(args.input)
num_slices, height, width = data.shape[0], data.shape[1], data.shape[2]
pred = np.zeros((num_slices, height, width), dtype=np.uint8)
data /= np.sqrt(height * width)
print('Compute...')
start = time.time()
for slice_id, kspace in enumerate(data):
kspace = kspace.copy()
# Apply sampling
kspace[var_sampling_mask] = 0
kspace = (kspace - stats[0]) / stats[1]
# Forward through network
input = np.expand_dims(kspace.transpose(2, 0, 1), axis=0)
outputs = exec_net.infer(inputs={'input_1': input})
output = next(iter(outputs.values()))
output = output.reshape(height, width)
# Save predictions
pred[slice_id] = cv.normalize(output, dst=None, alpha=255, beta=0, norm_type=cv.NORM_MINMAX, dtype=cv.CV_8U)
print('Elapsed time: %.1f seconds' % (time.time() - start))
WIN_NAME = 'MRI reconstruction with OpenVINO'
slice_id = 0
def callback(pos):
global slice_id
slice_id = pos
kspace = data[slice_id]
img = kspace_to_image(kspace)
kspace[var_sampling_mask] = 0
masked = kspace_to_image(kspace)
rec = pred[slice_id]
# Add a header
border_size = 20
render = cv.hconcat((img, masked, rec))
render = cv.copyMakeBorder(render, border_size, 0, 0, 0, cv.BORDER_CONSTANT, value=255)
cv.putText(render, 'Original', (0, 15), cv.FONT_HERSHEY_SIMPLEX, 0.5, color=0)
cv.putText(render, 'Sampled (PSNR %.1f)' % cv.PSNR(img, masked), (width, 15), cv.FONT_HERSHEY_SIMPLEX, 0.5, color=0)
cv.putText(render, 'Reconstructed (PSNR %.1f)' % cv.PSNR(img, rec), (width*2, 15), cv.FONT_HERSHEY_SIMPLEX, 0.5, color=0)
cv.imshow(WIN_NAME, render)
cv.waitKey(1)
cv.namedWindow(WIN_NAME, cv.WINDOW_NORMAL)
print(num_slices)
cv.createTrackbar('Slice', WIN_NAME, num_slices // 2, num_slices - 1, callback)
callback(num_slices // 2) # Trigger initial visualization
cv.waitKey()
#! [mri_demo:demo]

7
docs/IE_DG/Extensibility_DG/AddingNGraphOps.md
View File

@ -1,4 +1,4 @@
# Add Custom nGraph Operations {#openvino_docs_IE_DG_Extensibility_DG_AddingNGraphOps}
# Custom nGraph Operation {#openvino_docs_IE_DG_Extensibility_DG_AddingNGraphOps}
Inference Engine Extension API allows to register operation sets (opsets) with custom nGraph operations, it allows to support Networks with unknown operations.
@ -71,10 +71,9 @@ nGraph provides opsets mechanism for operation versioning. Different opsets dist
When specifying opset names, follow the rules below:
* Use unique opset names.
* Do not use the following built-in opset names: `extension`, `experimental`, `opset1`, `opest2`.
* Do not use the following built-in opset names: `extension`, `experimental`, `opset1`, `opset2`, `opset3`, ... , `opsetN`.
* Make sure that the Model Optimizer and your extension use the same opset names.
* IR v10 layers have the mandatory `version` attribute specifying the opset.
* `opset1` is the name of default operations set.
* IR v10 operations have the mandatory `version` attribute specifying the opset.
Operations from the default opset cannot be redefined.
Use a custom opset to create a new operation or extend functionality of an existing operation from another opset.

2
docs/IE_DG/Extensibility_DG/CPU_Kernel.md
View File

@ -1,4 +1,4 @@
# How to Implement Custom CPU Layers {#openvino_docs_IE_DG_Extensibility_DG_CPU_Kernel}
# How to Implement Custom CPU Operations {#openvino_docs_IE_DG_Extensibility_DG_CPU_Kernel}
The primary vehicle for the performance of the CPU codepath in the Inference Engine is the Intel® Math Kernel Library for Deep Neural Networks (Intel® MKL-DNN), and new CPU kernels extend the Inference Engine plugin for the Intel MKL-DNN. Implementing the InferenceEngine::ILayerExecImpl defines a general CPU-side extension. There are no Intel MKL-DNN specifics in the way you need to implement a kernel.

5
docs/IE_DG/Extensibility_DG/Extension.md
View File

@ -1,7 +1,10 @@
# Extension Library {#openvino_docs_IE_DG_Extensibility_DG_Extension}
Inference Engine provides an InferenceEngine::IExtension interface, which defines the interface for Inference Engine Extension libraries.
All extension libraries should be inherited from this interface.
All extension libraries should be inherited from this interface. The example below contains implementation of two operations: `Template`
used as an example in this document and `FFT` used as a more complex example from the [Custom Operations Guide](../../HOWTO/Custom_Layers_Guide.md).
> **NOTE**: `FFT` operation is implemented using OpenCV library functions `cv::dft` and `cv::idft`.
Based on that, declaration of an extension class can look as follows:

31
docs/IE_DG/Extensibility_DG/GPU_Kernel.md
View File

@ -1,16 +1,16 @@
# How to Implement Custom GPU Layers {#openvino_docs_IE_DG_Extensibility_DG_GPU_Kernel}
# How to Implement Custom GPU Operations {#openvino_docs_IE_DG_Extensibility_DG_GPU_Kernel}
The GPU codepath abstracts many details about OpenCL&trade;. You need to provide the kernel code in OpenCL C and the configuration file that connects the kernel and its parameters to the parameters of the layer.
The GPU codepath abstracts many details about OpenCL&trade;. You need to provide the kernel code in OpenCL C and the configuration file that connects the kernel and its parameters to the parameters of the operation.
There are two options of using custom layer configuration file:
There are two options of using custom operation configuration file:
* Include a section with your kernels into the global automatically-loaded `cldnn_global_custom_kernels/cldnn_global_custom_kernels.xml` file, which is hosted in the `<INSTALL_DIR>/deployment_tools/inference_engine/bin/intel64/{Debug/Release}` folder
* Call the `InferenceEngine::Core::SetConfig()` method from your application with the `InferenceEngine::PluginConfigParams::KEY_CONFIG_FILE` key and the configuration file name as a value before loading the network that uses custom layers to the plugin:
* Call the `InferenceEngine::Core::SetConfig()` method from your application with the `InferenceEngine::PluginConfigParams::KEY_CONFIG_FILE` key and the configuration file name as a value before loading the network that uses custom operations to the plugin:
@snippet snippets/GPU_Kernel.cpp part0
All Inference Engine samples, except trivial `hello_classification`,
feature a dedicated command-line option `-c` to load custom kernels. For example, to load custom layers for the classification sample, run the command below:
feature a dedicated command-line option `-c` to load custom kernels. For example, to load custom operations for the classification sample, run the command below:
```sh
$ ./classification_sample -m <path_to_model>/bvlc_alexnet_fp16.xml -i ./validation_set/daily/227x227/apron.bmp -d GPU
-c <absolute_path_to_config>/custom_layer_example.xml
@ -19,7 +19,7 @@ $ ./classification_sample -m <path_to_model>/bvlc_alexnet_fp16.xml -i ./validati
## Configuration File Format <a name="config-file-format"></a>
The configuration file is expected to follow the `.xml` file structure
with a node of the type `CustomLayer` for every custom layer you provide.
with a node of the type `CustomLayer` for every custom operation you provide.
The definitions described in the sections below use the following notations:
@ -32,14 +32,13 @@ Notation | Description
### CustomLayer Node and Sub-node Structure
`CustomLayer` node contains the entire configuration for a single custom
layer.
`CustomLayer` node contains the entire configuration for a single custom operation.
| Attribute Name |\# | Description |
|-----|-----|-----|
| `name` | (1) | The name of the layer type to be used. This name should be identical to the type used in the IR.|
| `type` | (1) | Must be `SimpleGPU`. |
| `version` | (1) | Must be `1`. |
| `name` | (1) | The name of the operation type to be used. This name should be identical to the type used in the IR.|
| `type` | (1) | Must be `SimpleGPU`. |
| `version` | (1) | Must be `1`. |
**Sub-nodes**: `Kernel` (1), `Buffers` (1), `CompilerOptions` (0+),
`WorkSizes` (0/1)
@ -69,9 +68,9 @@ the sources during compilation (JIT).
| Attribute Name | \# | Description |
|------|-------|------|
| `name` | (1) | The name of the defined JIT. For static constants, this can include the value as well (taken as a string). |
| `param` | (0/1) | This parameter value is used as the value of this JIT definition. |
| `param` | (0/1) | This parameter value is used as the value of this JIT definition. |
| `type` | (0/1) | The parameter type. Accepted values: `int`, `float`, and `int[]`, `float[]` for arrays. |
| `default` | (0/1) | The default value to be used if the specified parameters is missing from the layer in the IR. |
| `default` | (0/1) | The default value to be used if the specified parameters is missing from the operation in the IR. |
**Sub-nodes:** None
@ -92,7 +91,7 @@ weights or biases).
| Attribute Name | \# | Description |
|----|-----|------|
| `name` | (1) | Name of a blob attached to a layer in the IR |
| `name` | (1) | Name of a blob attached to a operation in the IR |
| `arg-index` | (1) | 0-based index in the entry function arguments to be bound to |
**Sub-nodes**: None
@ -105,7 +104,7 @@ weights or biases).
|------|-------|-------|
| `arg-index` | (1) | 0-based index in the entry function arguments to be bound to. |
| `type` | (1) | `input` or `output` |
| `port-index` | (1) | 0-based index in the layer’s input/output ports in the IR |
| `port-index` | (1) | 0-based index in the operation input/output ports in the IR |
| `format` | (0/1) | Data layout declaration for the tensor. Accepted values: `BFYX`, `BYXF`, `YXFB`, `FYXB` (also in all lowercase). Default value: `BFYX` |
### CompilerOptions Node and Sub-node Structure
@ -178,7 +177,7 @@ For an example, see [Example Kernel](#example-kernel).
| `<TENSOR>_PITCHES_SIZE`| The size of the `<TENSOR>_PITCHES` array |
| `<TENSOR>_OFFSET`| The number of elements from the start of the tensor to the first valid element (bypassing the lower padding) |
All `<TENSOR>` values are automatically defined for every tensor
bound to this layer (`INPUT0`, `INPUT1`, `OUTPUT0`, and so on), as shown
bound to this operation (`INPUT0`, `INPUT1`, `OUTPUT0`, and so on), as shown
in the following for example:
```sh

23
docs/IE_DG/Extensibility_DG/Intro.md
View File

@ -2,19 +2,22 @@
Inference Engine Extensibility API allows to add support of custom operations to the Inference Engine.
Extension should contain operation sets with custom operations and execution kernels for custom operations.
Physically, an extension library can be represented as a dynamic library exporting the single `CreateExtension` function that allows to create a new extension instance.
Physically, an extension library can be represented as a dynamic library exporting the single `CreateExtension` function
that allows to create a new extension instance.
Extensibility library can be loaded to the InferenceEngine::Core object using the InferenceEngine::Core::AddExtension method.
Extensibility library can be loaded to the `InferenceEngine::Core` object using the
`InferenceEngine::Core::AddExtension` method.
## Inference Engine Extension Library
Inference Engine Extension dynamic library contains several main components:
Inference Engine Extension dynamic library contains several components:
* [Extension class](Extension.md):
* [Extension Library](Extension.md):
- Contains custom operation sets
- Provides CPU implementations for custom operations
* [Custom operations](Intro.md):
- Allows to use InferenceEngine::Core::ReadNetwork to read Intermediate Representation (IR) with unsupported operations
* [Custom nGraph Operation](AddingNGraphOps.md):
- Allows to use `InferenceEngine::Core::ReadNetwork` to read Intermediate Representation (IR) with unsupported
operations
- Allows to create `ngraph::Function` with unsupported operations
- Provides shape inference mechanism for custom operations
@ -26,13 +29,13 @@ at `<dldt source tree>/docs/template_extension`.
The Inference Engine workflow involves the creation of custom kernels and either custom or existing operations.
An _Operation_ is a Network building block implemented in the training framework, for example, `Convolution` in Caffe*.
An _Operation_ is a network building block implemented in the training framework, for example, `Convolution` in Caffe*.
A _Kernel_ is defined as the corresponding implementation in the Inference Engine.
Refer to the [Custom Layers in the Model Optimizer](../../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md) section for details on how
mapping between framework layers and Inference Engine kernels is registered.
Refer to the [Model Optimizer Extensibility](../../MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md)
for details on how a mapping between framework operations and Inference Engine kernels is registered.
In short, you can plug your own kernel implementations into the Inference Engine and map them to the layers in the original framework.
In short, you can plug your own kernel implementations into the Inference Engine and map them to the operations in the original framework.
The following pages describe how to integrate custom _kernels_ into the Inference Engine:

9
docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md
View File

@ -77,7 +77,6 @@ Model Optimizer produces an Intermediate Representation (IR) of the network, whi
* [Converting DeepSpeech from TensorFlow](prepare_model/convert_model/tf_specific/Convert_DeepSpeech_From_Tensorflow.md)
* [Converting Language Model on One Billion Word Benchmark from TensorFlow](prepare_model/convert_model/tf_specific/Convert_lm_1b_From_Tensorflow.md)
* [Converting Neural Collaborative Filtering Model from TensorFlow*](prepare_model/convert_model/tf_specific/Convert_NCF_From_Tensorflow.md)
* [Converting TensorFlow* Object Detection API Models](prepare_model/convert_model/tf_specific/Convert_Object_Detection_API_Models.md)
* [Converting TensorFlow*-Slim Image Classification Model Library Models](prepare_model/convert_model/tf_specific/Convert_Slim_Library_Models.md)
* [Converting CRNN Model from TensorFlow*](prepare_model/convert_model/tf_specific/Convert_CRNN_From_Tensorflow.md)
@ -91,17 +90,15 @@ Model Optimizer produces an Intermediate Representation (IR) of the network, whi
* [Model Optimizations Techniques](prepare_model/Model_Optimization_Techniques.md)
* [Cutting parts of the model](prepare_model/convert_model/Cutting_Model.md)
* [Sub-graph Replacement in Model Optimizer](prepare_model/customize_model_optimizer/Subgraph_Replacement_Model_Optimizer.md)
* [(Deprecated) Case-Study: Converting SSD models created with the TensorFlow* Object Detection API](prepare_model/customize_model_optimizer/TensorFlow_SSD_ObjectDetection_API.md)
* [(Deprecated) Case-Study: Converting Faster R-CNN models created with the TensorFlow* Object Detection API](prepare_model/customize_model_optimizer/TensorFlow_Faster_RCNN_ObjectDetection_API.md)
* [Supported Framework Layers](prepare_model/Supported_Frameworks_Layers.md)
* [Intermediate Representation and Operation Sets](IR_and_opsets.md)
* [Operations Specification](../ops/opset.md)
* [Intermediate Representation suitable for INT8 inference](prepare_model/convert_model/IR_suitable_for_INT8_inference.md)
* [Custom Layers in Model Optimizer](prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md)
* [Model Optimizer Extensibility](prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md)
* [Extending Model Optimizer with New Primitives](prepare_model/customize_model_optimizer/Extending_Model_Optimizer_with_New_Primitives.md)
* [Extending Model Optimizer with Caffe Python Layers](prepare_model/customize_model_optimizer/Extending_Model_Optimizer_with_Caffe_Python_Layers.md)
* [Extending Model Optimizer with Custom MXNet* Operations](prepare_model/customize_model_optimizer/Extending_MXNet_Model_Optimizer_with_New_Primitives.md)
* [Legacy Mode for Caffe* Custom Layers](prepare_model/customize_model_optimizer/Legacy_Mode_for_Caffe_Custom_Layers.md)
* [Model Optimizer Frequently Asked Questions](prepare_model/Model_Optimizer_FAQ.md)
* [Known Issues](Known_Issues_Limitations.md)

4
docs/MO_DG/prepare_model/convert_model/Converting_Model.md
View File

@ -38,5 +38,5 @@ Framework-specific parameters for:
## See Also
* [Configuring the Model Optimizer](../Config_Model_Optimizer.md)
* [IR Notation Reference](../../IR_and_opsets.md)
* [Custom Layers in Model Optimizer](../customize_model_optimizer/Customize_Model_Optimizer.md)
* [Model Cutting](Cutting_Model.md)
* [Model Optimizer Extensibility](../customize_model_optimizer/Customize_Model_Optimizer.md)
* [Model Cutting](Cutting_Model.md)

1
docs/MO_DG/prepare_model/convert_model/Cutting_Model.md
View File

@ -9,7 +9,6 @@ The following examples are the situations when model cutting is useful or even r
* model has pre- or post-processing parts that cannot be translated to existing Inference Engine layers.
* model has a training part that is convenient to be kept in the model, but not used during inference.
* model is too complex (contains lots of unsupported operations that cannot be easily implemented as custom layers), so the complete model cannot be converted in one shot.
* model is one of the supported [SSD models](../customize_model_optimizer/TensorFlow_SSD_ObjectDetection_API.md). In this case, you need to cut a post-processing part off.
* problem with model conversion in the Model Optimizer or inference in the Inference Engine occurred. To localize the issue, limit the scope for conversion by iteratively searching for problematic places in the model.
* single custom layer or a combination of custom layers is isolated for debugging purposes.

1327
docs/MO_DG/prepare_model/customize_model_optimizer/Customize_Model_Optimizer.md
View File

File diff suppressed because it is too large Load Diff

64
docs/MO_DG/prepare_model/customize_model_optimizer/Extending_MXNet_Model_Optimizer_with_New_Primitives.md
View File

@ -1,45 +1,41 @@
# Extending the MXNet Model Optimizer with New Primitives {#openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Extending_MXNet_Model_Optimizer_with_New_Primitives}
# Extending Model Optimizer for Custom MXNet* Operations {#openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Extending_MXNet_Model_Optimizer_with_New_Primitives}
This section describes how you can create a Model Optimizer extension for a custom layer from your MXNet* model. It supplements the main document [Extending Model Optimizer with New Primitives](Extending_Model_Optimizer_with_New_Primitives.md) and provides a step-by-step procedure. To create an extension for a particular layer, perform the following steps:
This section provides instruction on how to support a custom MXNet operation (or as it called in the MXNet documentation
"operator" or "layer") which is not a part of the MXNet operation set. For example, if the operator is implemented using
the following [guide](https://mxnet.apache.org/versions/1.7.0/api/faq/new_op.html).
This section describes a procedure on how to extract operator attributes in the Model Optimizer. The rest of the
operation enabling pipeline and documentation on how to support MXNet operations from standard MXNet operation set is
described in the main document [Customize_Model_Optimizer](Customize_Model_Optimizer.md).
## Writing Extractor for Custom MXNet Operation
Custom MXNet operations have an attribute `op` (defining the type of the operation) equal to `Custom` and attribute
`op_type` which is an operation type defined by an user. Implement extractor class inherited from the
`MXNetCustomFrontExtractorOp` class instead of `FrontExtractorOp` class used for standard framework operations in order
to extract attributes for such kind of operations. The `op` class attribute value should be set to the `op_type` value
so the extractor is triggered for this kind of operation.
There is the example of the extractor for the custom operation registered with type (`op_type` value) equal to
`MyCustomOp` having attribute `my_attribute` of the floating point type with default value `5.6`. In this sample we
assume that we have already created the `CustomOp` class (inherited from `Op` class) for the Model Optimizer operation
for this MXNet custom operation as described in the [Customize_Model_Optimizer](Customize_Model_Optimizer.md).
1. Create the file `custom_proposal_ext.py` in the folder `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/mxnet`
If your MXNet layer has op `Custom`, create the `CustomProposalFrontExtractor` class inherited from `MXNetCustomFrontExtractorOp`:
```py
from mo.front.extractor import MXNetCustomFrontExtractorOp
class CustomProposalFrontExtractor(MXNetCustomFrontExtractorOp):
pass
```
Otherwise, for layers that are not standard MXNet layers, create the `ProposalFrontExtractor` class inherited from `FrontExtractorOp`:
```py
from mo.front.extractor import FrontExtractorOp
class ProposalFrontExtractor(FrontExtractorOp):
pass
```
2. Specify the operation that the extractor refers to and a specific flag. The flag represents whether the operation should be used by the Model Optimizer or should be excluded from processing:
```py
from mo.front.extractor import MXNetCustomFrontExtractorOp
class CustomProposalFrontExtractor(MXNetCustomFrontExtractorOp):
op = '_contrib_Proposal'
enabled = True
```
3. Register a mapping rule between the original model and the `PythonProposalOp` attributes by overriding the following function:
```py
from extension.ops.custom_op import CustomOp # implementation of the MO operation class
from mo.front.mxnet.extractors.utils import get_mxnet_layer_attrs
from mo.front.extractor import MXNetCustomFrontExtractorOp
from mo.ops.op import Op
class CustomProposalFrontExtractor(MXNetCustomFrontExtractorOp):
op = '_contrib_Proposal'
enabled = True
class CustomProposalFrontExtractor(MXNetCustomFrontExtractorOp): # inherit from specific base class
op = 'MyCustomOp' # the value corresponding to the `op_type` value of the MXNet operation
enabled = True # the extractor is enabled
@staticmethod
def extract(node):
attrs = get_mxnet_layer_attrs(node.symbol_dict)
attrs = get_mxnet_layer_attrs(node.symbol_dict) # parse the attributes to a dictionary with string values
node_attrs = {
'feat_stride': attrs.float('feat_stride', 16)
'my_attribute': attrs.float('my_attribute', 5.6)
}
# update the attributes of the node
Op.get_op_class_by_name('Proposal').update_node_stat(node, node_attrs) # <------ here goes the name ('Proposal') of the Operation that was implemented before
return __class__.enabled
```
CustomOp.update_node_stat(node, node_attrs) # update the attributes of the node
return self.enabled
```

89
docs/MO_DG/prepare_model/customize_model_optimizer/Extending_Model_Optimizer_with_Caffe_Python_Layers.md Normal file
View File

@ -0,0 +1,89 @@
# Extending Model Optimizer with Caffe* Python Layers {#openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Extending_Model_Optimizer_With_Caffe_Python_Layers}
This section provides instruction on how to support a custom Caffe operation written only in Python. For example, the
[Faster-R-CNN model]((http://dl.dropboxusercontent.com/s/o6ii098bu51d139/faster_rcnn_models.tgz?dl=0)) implemented in
Caffe contains a custom layer Proposal written in Python. The layer is described in the
[Faster-R-CNN protoxt](https://raw.githubusercontent.com/rbgirshick/py-faster-rcnn/master/models/pascal_voc/VGG16/faster_rcnn_end2end/test.prototxt)
the following way:
```sh
layer {
name: 'proposal'
type: 'Python'
bottom: 'rpn_cls_prob_reshape'
bottom: 'rpn_bbox_pred'
bottom: 'im_info'
top: 'rois'
python_param {
module: 'rpn.proposal_layer'
layer: 'ProposalLayer'
param_str: "'feat_stride': 16"
}
}
```
This section describes only a procedure on how to extract operator attributes in the Model Optimizer. The rest of the
operation enabling pipeline and documentation on how to support other Caffe operations (written in C++) is described in
the main document [Customize_Model_Optimizer](Customize_Model_Optimizer.md).
## Writing Extractor for Caffe Python Layer
Custom Caffe Python layers have an attribute `type` (defining the type of the operation) equal to `Python` and two
mandatory attributes `module` and `layer` in the `python_param` dictionary. The `module` defines the Python module name
with the layer implementation, while `layer` value is an operation type defined by an user. In order to extract
attributes for such an operation it is necessary to implement extractor class inherited from the
`CaffePythonFrontExtractorOp` class instead of `FrontExtractorOp` class used for standard framework layers. The `op`
class attribute value should be set to the `module + "." + layer` value so the extractor is triggered for this kind of
operation.
Here is a simplified example of the extractor for the custom operation Proposal from Faster-R-CNN model mentioned above.
The full code with additional checks is provided in the
`<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/caffe/proposal_python_ext.py`. The sample code uses
operation `ProposalOp` which corresponds to `Proposal` operation described in the [Available Operations Sets](../../../ops/opset.md)
document. Refer to the source code below for a detailed explanation of the extractor.
```py
from extensions.ops.proposal import ProposalOp
from mo.front.extractor import CaffePythonFrontExtractorOp
class ProposalPythonFrontExtractor(CaffePythonFrontExtractorOp):
op = 'rpn.proposal_layer.ProposalLayer' # module + "." + layer
enabled = True # extractor is enabled
@staticmethod
def extract_proposal_params(node, defaults):
param = node.pb.python_param # get the protobuf message representation of the layer attributes
# parse attributes from the layer protobuf message to a Python dictionary
attrs = CaffePythonFrontExtractorOp.parse_param_str(param.param_str)
update_attrs = defaults
# the operation expects ratio and scale values to be called "ratio" and "scale" while Caffe uses different names
if 'ratios' in attrs:
attrs['ratio'] = attrs['ratios']
del attrs['ratios']
if 'scales' in attrs:
attrs['scale'] = attrs['scales']
del attrs['scales']
update_attrs.update(attrs)
ProposalOp.update_node_stat(node, update_attrs) # update the node attributes
@classmethod
def extract(cls, node):
# define default values for the Proposal layer attributes
defaults = {
'feat_stride': 16,
'base_size': 16,
'min_size': 16,
'ratio': [0.5, 1, 2],
'scale': [8, 16, 32],
'pre_nms_topn': 6000,
'post_nms_topn': 300,
'nms_thresh': 0.7
}
cls.extract_proposal_params(node, defaults)
return cls.enabled
```
## See Also
* [Customize_Model_Optimizer](Customize_Model_Optimizer.md)
* [Legacy Mode for Caffe* Custom Layers](Legacy_Mode_for_Caffe_Custom_Layers.md)

477
docs/MO_DG/prepare_model/customize_model_optimizer/Extending_Model_Optimizer_with_New_Primitives.md
View File

@ -1,476 +1,3 @@
# Extending the Model Optimizer with New Primitives {#openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Extending_Model_Optimizer_with_New_Primitives}
# Extending Model Optimizer with New Primitives {#openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Extending_Model_Optimizer_with_New_Primitives}
This section explains how to register a custom layer in the Model Optimizer, including how to register Proposal as a custom layer. This section also demonstrates how `Proposal` works as a custom layer.
Model Optimizer loads the model, goes through the topology, and tries to find each layer type in the list of known layers. If the Model Optimizer does not find a layer in that list, it looks for the layer in the list of custom layers. If the Model Optimizer fails to find the layer among the defined custom layers, it registers a Caffe\* fallback for for the output shape inference. If the Model Optimizer does not find Caffe and cannot infer shapes, the Model Optimizer fails with an appropriate message.
You must know two things about custom layers with the Model Optimizer:
* How to map a subgraph in a FW model to a subgraph consisting of Inference Engine layers. For Caffe, the subgraph is a 1-to-1 mapping of a Caffe layer to an Inference Engine layer.
* How to infer shapes for unknown subgraphs. This can be either for a step in which the internal representation consists of framework-specific layers, or for a step in which the internal representation consists of Inference Engine layers.
You also have the option of a framework fallback for unknown subgraphs, for when the original framework is used for inference of output shapes of operations. The example below demonstrates the case in which the framework is not available or should not be used.
## Preparing an Example Topology
> **NOTE**: Skip this section if you have a topology with a layer that is not known to the Model Optimizer.
The information in this section prepares a Caffe\* model with the provided, deployment-ready `prototxt` for a
well-known topology called
[Faster-R-CNN protoxt](https://raw.githubusercontent.com/rbgirshick/py-faster-rcnn/master/models/pascal_voc/VGG16/faster_rcnn_end2end/test.prototxt)
to demonstrate the workflow. To use this example, you must have
[weights and biases](http://dl.dropboxusercontent.com/s/o6ii098bu51d139/faster_rcnn_models.tgz?dl=0) for inference,
because `prototxt` just describes the structure of the topology.
1. Download the `.caffemodel` and `.prototxt` files
2. Run the Model Optimizer on the `.caffemodel` and `.prototxt` files:
```shell
python mo.py --input_model VGG16_faster_rcnn_final.caffemodel --input_proto test.prototxt
```
You will likely see the error message:
```shell
Error parsing text-format caffe.NetParameter: 196:16: Message type "caffe.DropoutParameter" has no field named "scale_train".
```
Whether you see the error depends on your Caffe version. For example, BVLC Caffe does not support the boolean parameter `scale_train` for the `dropout` layer. The error message does not matter, because the dropout layer is needed only for training, and the Model Optimizer removes it.
3. To proceed, comment out these lines in `test.prototxt`:
```sh
...
layer {
name: "drop6"
type: "Dropout"
bottom: "fc6"
top: "fc6"
dropout_param {
dropout_ratio: 0.5
# scale_train: false # <-------------- comment out this line
}
}
...
layer {
name: "drop7"
type: "Dropout"
bottom: "fc7"
top: "fc7"
dropout_param {
dropout_ratio: 0.5
# scale_train: false # <-------------- comment out this line
}
}
...
```
4. Run the Model Optimizer on this model again:
```shell
python mo.py --input_model VGG16_faster_rcnn_final.caffemodel --input_proto test.prototxt
```
You get the model successfully converted to Intermediate Representation, and you can infer it with the Inference Engine.
However, the aim of this tutorial is to demonstrate the way of supporting custom layers not yet supported by the Model Optimizer.
If you want to understand better how Model Optimizer works, remove the extension for layer `Proposal` and follow all steps of this tutorial.
5. Remove the extension for layer `Proposal`:
```sh
mkdir extensions/old
mv extensions/front/caffe/proposal_python_ext.py extensions/old/proposal_python_ext_old.py
mv extensions/ops/proposal_python_example.py extensions/old/proposal_python__example_old.py
```
6. Now you can run the Model Optimizer on this model once again:
```sh
python mo.py --input_model VGG16_faster_rcnn_final.caffemodel --input_proto test.prototxt
```
You will see the message:
```shell
[ ERROR ] Found custom layer proposal. Model Optimizer does not support this layer.
Please, register it in CustomLayersMapping.xml or implement extension.
For more information please refer to Model Optimizer FAQ, question #FAQ45.
```
This message means the Model Optimizer can load the model, but is unable to infer the shape and handle the custom layer properties.
## Registering a Custom Layer as a Model Optimizer Extension
In the following sections, you will learn how to make the Model Optimizer independent from Caffe\* when processing a
model that has a custom layer. In this example, the custom layer is referred to as the Proposal layer.
Use this section to implement the mapping rules for the `Proposal` layer attributes and the output shape calculation. As part of these steps, you must first create a class for the `Proposal` layer and inherit it from general-purpose Op that defines the interface of every new custom layer.
In this section, it is important to understand the `Op` class and its function. The implementation of this class shows that it expects a graph and attributes to be passed when initializing. The graph and attributes are in `<INSTALL_DIR>/deployment_tools/model_optimizer/mo/ops/op.py`
`Op` keeps the attributes for each operation and contains logic for handling node creation for internal model representation. `Op` is responsible for dumping each particular operation to the `.xml` format for the Intermediate Representation. By inheriting from it, the technical items are complete and you concentrate on the specificity of this layer: the attributes it supports and the rules on computing its output shape.
Follow these steps:
1. Create the file `python_proposal.py` in the directory `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/ops`:
```python
from mo.ops.op import Op
class PythonProposalOp(Op):
pass
```
2. Define the name of the operation and make a stub constructor:
```python
from mo.ops.op import Op
class PythonProposalOp(Op):
op = 'Proposal'
def __init__(self, graph, attrs):
super().__init__(graph)
```
3. Every `Op` must have three specific fields defined: `type`, `op`, and `infer`. In most cases, the `type` and `op` names are the same, and `infer` is defined as a function to compute the output shape. Reflect these fields in your constructor:
```python
from mo.ops.op import Op
class PythonProposalOp(Op):
op = 'Proposal'
def __init__(self, graph, attrs):
mandatory_props = {
'type': __class__.op,
'op': __class__.op,
'infer': None
}
super().__init__(graph, mandatory_props, attrs)
```
According to the Intermediate Representation catalog, Proposal layer has the following attributes:
* `pre_nms_topn`
* `post_nms_topn`
* `nms_thresh`
* `feat_stride`
* `min_size`
* `base_size`
* `ratio`
* `scale`
4. In defining supported attribute names, it is best to use the same names as in the original models. The names are similar to parameters and have no connection with the model layer properties. For clarity, you can use the name `my_ratio` for `ratio`. Other than defining the list of supported parameters, you can define only the parameters that appear in the Intermediate Representation in the `backend_attrs` method.
Define your attributes:
```python
class PythonProposalOp(Op):
# ... constructor
def supported_attrs(self):
return [
'pre_nms_topn',
'post_nms_topn',
'nms_thresh',
'feat_stride',
'min_size',
'base_size',
'ratio',
'scale'
]
```
5. Model Optimizer now knows how to create the layer called Proposal when it is in the topology and what attributes this layer has. However, the Model Optimizer does not know how to calculate the output shape of this operation. Define a rule to calculate the output shape:
```python
import numpy as np
from mo.graph.graph import Node
from mo.ops.op import Op
class PythonProposalOp(Op):
def __init__(self, graph, attrs):
mandatory_props = {
'type': __class__.op,
'op': __class__.op,
'infer': PythonProposalOp.calculate_output_shape
}
super().__init__(graph, mandatory_props, attrs)
# ... supported attrs
@staticmethod
def calculate_output_shape(node: Node):
node.out_node().shape = (1, 1, 1, 1) # any Proposal now has always the same output
```
6. According to the Intermediate Representation catalog, Proposal layer has the following output calculation formula, where shape dynamically depends on the `post_nms_topn` parameter.
Implement the output calculation formula in Python\*:
```python
import numpy as np
class PythonProposalOp(Op):
# ... static fields
# ... constructor
# ... supported attrs
@staticmethod
def calculate_output_shape(node: Node):
input_shape = node.in_node(0).shape
out_shape = np.array([0, 0], dtype=np.int64)
# rois blob: holds R regions of interest, each is a 5 - tuple
# (n, x1, y1, x2, y2) specifying an image batch index n and a
# rectangle(x1, y1, x2, y2)
out_shape[0] = input_shape[0] * node.post_nms_topn
out_shape[1] = 5
node.out_node(0).shape = out_shape
```
The node does not contain this parameter because it should be initialized in the constructor and in other parameters. The Inference Engine contains the implementation of a Caffe\*-like Proposal layer and works well with the default values from `caffe.proto`:
```
// Message that stores parameters used by ProposalLayer message ProposalParameter { optional uint32 feat_stride = 1 [default = 16]; optional uint32 base_size = 2 [default = 16]; optional uint32 min_size = 3 [default = 16]; repeated float ratio = 4; repeated float scale = 5; optional uint32 pre_nms_topn = 6 [default = 6000]; optional uint32 post_nms_topn = 7 [default = 300]; optional float nms_thresh = 8 [default = 0.7]; }
```
7. Change the constructor as follows:
```python
class PythonProposalOp(Op):
# ... static fields
def __init__(self, graph, attrs):
mandatory_props = {
'type': __class__.op,
'op': __class__.op,
'feat_stride': 16,
'base_size': 16,
'min_size': 16,
'ratio': [0.5, 1, 2],
'scale': [8, 16, 32],
'pre_nms_topn': 6000,
'post_nms_topn': 300,
'nms_thresh': 0.7,
'infer': PythonProposalOp.calculate_output_shape
}
super().__init__(graph, mandatory_props, attrs)
# ... supported attrs
# ... calculate output shape
```
It is mandatory to call two functions right after the implementation of that class:
```
class ProposalPythonOp(Op):
...
register_caffe_python_extractor(ProposalPythonOp, 'rpn.proposal_layer.ProposalLayer')
Op.excluded_classes.append(ProposalPythonOp)
```
Note that the first call <code>register_caffe_python_extractor(ProposalPythonOp, 'rpn.proposal_layer.ProposalLayer')</code> registers the extension of the layer in the Model Optimizer that will be found by a specific name (it is mandatory to join module name and layer name): <code>'rpn.proposal_layer.ProposalLayer'</code>.
The second call prevents the Model Optimizer from using this extension as if it is an extension for a layer with type `Proposal`. Otherwise, this layer can be chosen as an implementation of extension that can lead to potential issues.
**Summary**
In this section you implemented support for a custom layer with type `Python` that is `Proposal` layer in the topology. You learned how to calculate output shape of this layer.
The values of attributes are hardcoded, and in the next section you will learn how to extract these values from original framework model (Caffe model in this case).
## Registering Rules to Pass Extension Layer Properties from a Caffe\* Model to the Intermediate Representation
Model Optimizer now knows how to set the shape of the `PythonProposalOp` operation, but it is incorrect to initialize attributes with same values for every operation. Instead, the values should be extracted from the original topology. Model Optimizer does not know how to map the custom layer properties to the `PythonProposalOp`. For this, you must register the `FrontExtractorOp` instance.
> **NOTE**: This step is required only if the layer requires parameters from the original model.
1. Remove call functions `register_caffe_python_extractor` and `Op.excluded_classes.append` from the file with `op`, because you will implement extracted attributes from prototxt by yourself.
There are multiple types of layers in Caffe: for example, `Convolution` and `Pooling`. Also, there is a specific type for custom Python\* layers called `Python`. Therefore, it is necessary to distinguish between those 'usual' types of layers and custom ones. If you want to implement extensions for a layer with type different to `Python`, you need to inherit your class of operation (for example, `ProposalFrontExtractor`) from `FrontExtractorOp`. Otherwise, inherit your class of operation from `CaffePythonFrontExtractorOp`.
2. Create a file `python_proposal_ext.py` in the folder `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/caffe`
```py
from mo.front.extractor import CaffePythonFrontExtractorOp
class PythonProposalFrontExtractor(CaffePythonFrontExtractorOp):
pass
```
For other layers types, inherit from `FrontExtractorOp`:
```py
from mo.front.extractor import FrontExtractorOp
class ProposalFrontExtractor(FrontExtractorOp):
pass
```
You will implement extractor for layer with type `Python`, however, the steps are generally the same for layers with other types.
3. Specify the operation that the extractor refers to and a specific flag. The flag represents whether the operation should be used by the Model Optimizer or should be excluded from processing:
```py
from mo.front.extractor import CaffePythonFrontExtractorOp
class PythonProposalFrontExtractor(CaffePythonFrontExtractorOp):
op = 'rpn.proposal_layer.ProposalLayer'
enabled = True
```
4. Register a mapping rule between the original model and the `PythonProposalOp` attributes by overriding the following function:
```py
from mo.front.extractor import CaffePythonFrontExtractorOp
from mo.ops.op import Op
class ProposalPythonFrontExtractor(CaffePythonFrontExtractorOp):
op = 'rpn.proposal_layer.ProposalLayer'
enabled = True
@staticmethod
def extract(node):
proto_layer = node.pb
param = proto_layer.python_param # each layer has a specific parameter, take a look at caffe.proto
python_params = str(param.param_str) # for Python layers, all params are in param_str
attrs = {
'feat_stride': int(python_params.split(':')[-1])
}
# update the attributes of the node
Op.get_op_class_by_name('Proposal').update_node_stat(node, attrs) # <------ here goes the name ('Proposal') of the Operation that was implemented before
return __class__.enabled
```
> **NOTE:** if you implement extension for layer with type different to `Python`, change the following line: <code>Op.get_op_class_by_name('Proposal').update_node_stat(node, attrs)</code> to this line: <code>Op.get_op_class_by_name(__class__.op).update_node_stat(node, mapping_rule)</code>.
You have successfully extracted the parameter `feat_stride` from `prototxt`, assuming it is the only parameter in this layer.
5. To increase the implementation flexibility:
```py
from mo.front.extractor import CaffePythonFrontExtractorOp
from mo.ops.op import Op
class PythonProposalFrontExtractor(CaffePythonFrontExtractorOp):
op = 'rpn.proposal_layer.ProposalLayer'
enabled = True
@staticmethod
def extract(node):
param = node.pb.python_param
attrs = CaffePythonFrontExtractorOp.parse_param_str(param.param_str)
Op.get_op_class_by_name('Proposal').update_node_stat(node, attrs)
return ProposalPythonFrontExtractor.enabled
```
You can successfully convert the model. Open the `.xml` file and view your code:
```xml
...
<layer id="42" name="proposal" precision="FP32" type="Python">
<data base_size="16" feat_stride="16" min_size="16" nms_thresh="0.7" post_nms_topn="300" pre_nms_topn="6000" ratio="[0.5, 1, 2]" scale="[8, 16, 32]"/>
<input>
<port id="0">
<dim>1</dim>
<dim>18</dim>
<dim>15</dim>
<dim>15</dim>
</port>
<port id="1">
<dim>1</dim>
<dim>36</dim>
<dim>15</dim>
<dim>15</dim>
</port>
<port id="2">
<dim>1</dim>
<dim>3</dim>
</port>
</input>
<output>
<port id="3">
<dim>300</dim>
<dim>5</dim>
</port>
</output>
</layer>
...
```
Look at the output shape of the custom layer you implemented. The shape was calculated according to the rules specified in `PythonProposalOp`. The `ratio` and `scale` properties have the value `[0.5, 1, 2]` and `[8, 16, 32]`. They have square brackets because they are originally a repeated parameter. You converted the parameter to a list in `PythonProposalOp`. Model Optimizer cast the value to a string. According to Python\* rules, a list has a string representation of opening and closing square brackets and values joined by commas.
This is not a valid notation for the Intermediate Representation specification, because repeated parameters must be separated by a comma but without the brackets. Therefore, you must override the Model Optimizer default behavior regarding how it handles those parameters during the Intermediate Representation emitting stage, after the optimizations are complete. To do so, implement `backend_attrs()` in the `PythonProposalOp` class:
```python
class PythonProposalOp(Op):
... other methods
def backend_attrs(self) -> list:
"""
Gets list of attributes that should appear in resulting IR
Returns:
list of attributes names or list of tuples (name of attribute, pre-processing rule)
"""
return [
( # a tuple per attribute
'ratio', # name of attribute
# pre-processing rule in a form of lambda
# lambda takes a PythonProposalOp node with all defined properties
# it translates [1,2,3] -> "1,2,3"
lambda node: ','.join(map(str, node['ratio']))
),
(
'scale',
lambda node: ','.join(map(str, node['scale']))
),
'feat_stride',
'base_size',
'min_size',
'pre_nms_topn',
'post_nms_topn',
'nms_thresh'
]
```
The model can now be successfully converted.
Open the `.xml` file. `ratio` and `scale` have the expected correct values `0.5,1,2` and `8,16,32`:
```xml
...
<layer id="33" name="proposal" precision="FP32" type="Python">
<data base_size="16" feat_stride="16" min_size="16" nms_thresh="0.7" post_nms_topn="300" pre_nms_topn="6000" ratio="0.5,1,2" scale="8,16,32"/>
<input>
...
</input>
<output>
...
</output>
</layer>
...
```
> **NOTE**: Model Optimizer supports the Faster-R-CNN topology. Run the following command for the same Intermediate Representation:
```sh
python mo.py --input_model VGG16_faster_rcnn_final.caffemodel --input_proto test.prototxt --extensions <INSTALL_DIR>/deployment_tools/inference-engine/samples/object_detection_sample/fasterrcnn_extensions
```
**Summary**
In this section you learned how to:
1. Create a framework-independent extension implementation of the Intermediate Representation custom layer with unified logic for calculating output shapes, specified set of attributes
2. Use the Framework-Specific property extractor to map original model custom layer properties to the expected properties of the Framework-Independent extension
3. Manipulate the custom layer properties representation in the resulting Intermediate Representation
Files used in this section:
* `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/ops/python_proposal.py`:
```py
import networkx as nx
import numpy as np
from mo.front.extractor import attr_getter
from mo.graph.graph import Node
from mo.ops.op import Op
class ProposalOp(Op):
op = 'Proposal'
def __init__(self, graph: nx.MultiDiGraph, attrs: dict):
mandatory_props = {
'type': __class__.op,
'op': __class__.op,
'post_nms_topn': 300, # default in caffe-shared
'infer': ProposalOp.proposal_infer
}
super().__init__(graph, mandatory_props, attrs)
def supported_attrs(self):
return [
'feat_stride',
'base_size',
'min_size',
'ratio',
'scale',
'pre_nms_topn',
'post_nms_topn',
'nms_thresh'
]
def backend_attrs(self):
return [
'feat_stride',
'base_size',
'min_size',
('ratio', lambda node: attr_getter(node, 'ratio')),
('scale', lambda node: attr_getter(node, 'scale')),
'pre_nms_topn',
'post_nms_topn',
'nms_thresh',
]
@staticmethod
def proposal_infer(node: Node):
input_shape = node.in_node(0).shape
out_shape = np.array([0, 0], dtype=np.int64)
# rois blob: holds R regions of interest, each is a 5 - tuple
# (n, x1, y1, x2, y2) specifying an image batch index n and a
# rectangle(x1, y1, x2, y2)
out_shape[0] = input_shape[0] * node.post_nms_topn
out_shape[1] = 5
node.out_node(0).shape = out_shape
```
* `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/caffe/python_proposal_ext.py`:
```py
from mo.front.extractor import CaffePythonFrontExtractorOp
from mo.ops.op import Op
class ProposalPythonFrontExtractor(CaffePythonFrontExtractorOp):
op = 'rpn.proposal_layer.ProposalLayer'
enabled = True
@staticmethod
def extract(node):
param = node.pb.python_param
attrs = CaffePythonFrontExtractorOp.parse_param_str(param.param_str)
Op.get_op_class_by_name('Proposal').update_node_stat(node, attrs)
return ProposalPythonFrontExtractor.enabled
```
This page is deprecated. Please, refer to [Model Optimizer Extensibility](Customize_Model_Optimizer.md) page for more information.

34
docs/MO_DG/prepare_model/customize_model_optimizer/Legacy_Mode_for_Caffe_Custom_Layers.md
View File

@ -1,10 +1,23 @@
# Legacy Mode for Caffe* Custom Layers {#openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Legacy_Mode_for_Caffe_Custom_Layers}
> **NOTE**: This functionality is deprecated and will be removed in future releases.
> **NOTE**: This functionality is deprecated and will be removed in the future releases.
Model Optimizer can register custom layers in a way that the output shape is calculated by the Caffe\* framework installed on your system. This chapter covers this option.
Model Optimizer can register custom layers in a way that the output shape is calculated by the Caffe\* framework
installed on your system. This approach has several limitations:
> **NOTE**: Caffe Python\* API has an issue when layer name does not correspond to the name of its top. The fix was implemented on [BVLC Caffe\*](https://github.com/BVLC/caffe/commit/35a7b87ad87457291dfc79bf8a7e7cf7ef278cbb). The Caffe framework on your computer must contain this fix. Otherwise, Caffe framework can unexpectedly fail during the fallback procedure.
* If your layer output shape depends on dynamic parameters, input data or previous layers parameters, calculation of
output shape of the layer via Caffe can be incorrect. For example, `SimplerNMS` is filtering out bounding boxes that do
not satisfy the condition. Internally, Caffe fallback forwards the whole net without any meaningful data - just some
noise. It is natural to get only one bounding box (0,0,0,0) instead of expected number (for example, 15). There is an
option to patch Caffe accordingly, however, it makes success of Intermediate Representation generation on the patched
Caffe on the particular machine. To keep the solution independent from Caffe, we recommend to use extensions mechanism
for such layers described in the [Model Optimizer Extensibility](Customize_Model_Optimizer.md).
* It is not possible to produce Intermediate Representation on a machine that does not have Caffe installed.
> **NOTE**: Caffe Python\* API has an issue when layer name does not correspond to the name of its top. The fix was
> implemented on [BVLC Caffe\*](https://github.com/BVLC/caffe/commit/35a7b87ad87457291dfc79bf8a7e7cf7ef278cbb). The
> Caffe framework on your computer must contain this fix. Otherwise, Caffe framework can unexpectedly fail during the
> fallback procedure.
> **NOTE**: The Caffe fallback feature was validated against [this GitHub revision](https://github.com/BVLC/caffe/tree/99466224dac86ddb86296b1e727794fb836bd80f). You may have issues with forks or later Caffe framework versions.
@ -25,7 +38,8 @@ Where:
**Example**:
1. `Proposal` layer has parameters, and they appear in the Intermediate Representation. The parameters are stored in the `proposal_param` property of the layer:
1. `Proposal` layer has parameters, and they appear in the Intermediate Representation. The parameters are stored in
the `proposal_param` property of the layer:
```shell
\<CustomLayer NativeType="Proposal" hasParam ="true" protoParamName = "proposal_param"/\>
```
@ -34,16 +48,6 @@ Where:
\<CustomLayer NativeType="CustomLayer" hasParam ="false"/\>
```
For this feature, you need an appropriate version of Caffe installed on the computer on which you run the Model Optimizer.
## Constraints of Using the Caffe Fallback
Several layers in the Caffe\* framework can have shapes that dynamically depend on the input data, not only the layers that proceed the layer and its parameters. For example, `SimplerNMS` is filtering out bounding boxes that do not satisfy the condition. Internally, Caffe fallback forwards the whole net without any meaningful data - just some noise. It is natural to get only one bounding box (0,0,0,0) instead of expected number (for example, 15). There is an option to patch Caffe accordingly, however, it makes success of Intermediate Representation generation on the patched Caffe on the particular machine. To keep the solution independent from Caffe, we recommend to use extensions mechanism for such layers.
Known cases like `Proposal`, `DetectionOutput`, `SimplerNMS` are implemented as extensions and can be used out of the box.
A detailed description of supported layers is in the [Operations Specification](../../../ops/opset.md) document.
## Building Caffe\*
1. Build Caffe\* with Python\* 3.5:
@ -68,4 +72,4 @@ python3
import caffe
```
If Caffe was installed correctly, the `caffe` module is imported without errors.
If Caffe was installed correctly, the `caffe` module is imported without errors.

363
docs/MO_DG/prepare_model/customize_model_optimizer/Subgraph_Replacement_Model_Optimizer.md
View File

@ -1,363 +1,4 @@
# Sub-Graph Replacement in the Model Optimizer {#openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Subgraph_Replacement_Model_Optimizer}
Several reasons exist for why the Model Optimizer could not generate an Intermediate Representation for a model. However, in some cases, the Intermediate Representation could be generated after providing certain hints to the tool. The examples of hints below are mostly related to TensorFlow\*, but potentially could be actual for models created in any framework:
* Topology contains an operation (or a sub-graph of operations) not known for Model Optimizer, but this operation (sub-graph) could be expressed as a combination of known operations. A hint would be a description of this combination to the tool).
* Sub-graph of operations in the topology expresses a single layer known to Inference Engine.
* TensorFlow and Inference Engine use different layouts of tensors, NHWC and NCHW respectively. If some tensor in NHWC layout is flattened (for example, all the dimensions are squashed into single dim), it is not possible to convert it to NCHW layout required for Inference Engine, so Model Optimizer cannot produce correct Intermediate Representation.
The detailed solutions for the examples above are given later, the next subsection shows what is common in all three examples.
## Sub-graph Replacement
In these cases, the sub-graph (or a single node) of initial graph is replaced with a new sub-graph (single node). The sub-graph replacement consists of the following steps:
1. Identify an existing sub-graph for replacement
2. Generate a new sub-graph
3. Connect a new sub-graph to the graph (create input/output edges to the new sub-graph)
4. Create output edges out of a new sub-graph to the graph
5. Do something with the original sub-graph (for example, remove it)
Model Optimizer provides several ways to perform most of the sub-graph replacement steps. The next subsections describe these methods.
## Replace a Single Operation with a Sub-graph of Operations
For example, there is an operation `SquaredDifference` in TensorFlow which calculates \f$(a - b)^2\f$, where \f$a\f$ and \f$b\f$ are input tensors. Inference Engine does not support such operation. However, `SquaredDifference` could be expressed using two `Power` operations and one `Eltwise Add`. The `Power` operation calculates \f$scale * (a ^ {power}) + shift\f$, where \f$a\f$ is a tensor and \f$scale\f$, \f$power\f$ and \f$shift\f$ are float values. The first `Power` operation negates the value of tensor \f$b\f$. The second one is used to square the result of \f$a + (- b)\f$ which is calculated using the `Eltwise Add` operation applied to tensor \f$a\f$ and tensor \f$-b\f$.
Given that, we can replace all `SquaredDifference` operations in the initial model with two `Power` and one `Eltwise` operations. The replacer is implemented in the following file `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/SquaredDifference.py`.
```python
import networkx as nx
from mo.front.common.replacement import FrontReplacementOp
from mo.graph.graph import Node
from mo.ops.eltwise import Eltwise
from mo.ops.power import Power
class SquaredDifference(FrontReplacementOp):
"""
Example class illustrating how to implement replacement of a single op in the front-end of the MO pipeline.
This class replaces a single op SquaredDifference by a sub-graph consisting of 3 lower-level ops.
"""
op = "SquaredDifference"
enabled = True
def replace_op(self, graph: nx.MultiDiGraph, node: Node):
negate = Power(graph, dict(scale=-1, name=node.name + '/negate_'))
add = Eltwise(graph, dict(operation='sum', name=node.name + '/add_'))
squared = Power(graph, dict(power=2, name=node.name + '/squared_'))
out_node = squared.create_node([add.create_node([node.in_node(0), negate.create_node([node.in_node(1)])])])
# Replace edge from out port 0 of the matched node with a edge from node out_node.id with port 0.
# The "explicit" version of the return value is: [(out_node.id, 0)])
return [out_node.id]
```
Model Optimizer internal representation of the graph uses the networkx module.
**Key lines**:
* Line 1: Imports this module.
* Line 3: Imports class `FrontReplacementOp` that is used to replace operation of particular type with a new sub-graph. This class performs the first step of the sub-graph replacement (identifies an existing sub-graph for replacement). It is important to mention that the replacement happens before shape inference and creation of data nodes representing tensors with values. At this stage of model conversion pipeline, all nodes in the graph are operation nodes or nodes of type `Const` that produce tensor with fixed value embedded into the node.
* Line 4: Imports class `Node` representing a single node in the computation graph.
* Lines 5 - 6: Import classes representing operations `Power` and `Eltwise`. These classes are inherited from base class `mo.ops.Op` that represents operation and stores its attributes.
* Line 9: Defines class `SquaredDifference` inherited from `FrontReplacementOp`. This is a replacer class that is automatically registered and executed by Model Optimizer. Since the class is located in the common (not framework) specific directory `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front`, it is used for replacement for all supported frameworks.
* Line 15: Defines the class variable `op` that stores the name of the operation to be replaced. In this case, it is `SquaredDifference`.
* Line 16: Defines class variable `enabled` that controls whether the replacer is enabled or not. The only function that should be implemented in the class is `replace_op`. It gets graph to operate on and an instance of node of desired operation (`SquaredDifference` in this case). This function performs step two and three of the sub-graph replacement (generates a new sub-graph to replace with and connects a new sub-graph to the graph).
* Lines 19 - 21: Create instances of operations classes with required attributes.
* Line 23: Creates a sub-graph from the operations defined above. The `create_node` method of the `Op` class generates `Node` from the `Op` and uses single mandatory argument - the list of input nodes (represented as instances of `Node` class) to create input edges to the node being generated. Inputs of the `SquaredDifference` node are retrieved using `node.in_node(0)` and `node.in_node(1)` method calls. The `Eltwise Add` node gets first input as initial first input of `SquaredDifference` node, the second input of `add` is the result of negation of the second input of `SquaredDifference` node: `[add.create_node([node.in_node(0), negate.create_node([node.in_node(1)])])]`. Then the result of `Add` node is squared. `out_node` node performs this calculation.
The `replace_op` function returns a list of node names used to create output edges of the sub-graph to connect it with the rest of the graph. Each element of the list describes mapping between old output edge of the matched node and new sub-graph node and output edge index. The i-th element of the list corresponds to the i-th output tensor of the matched node. In this case, `SquaredDifference` produces single tensor through output port 0, so the returned list contains single element. In general, each element is a tuple, where the first element is the name of a new node producing required tensor and the second is the output port for that tensor. If the output port is 0, it is possible to use shortcut - just the name of the node instead of a tuple. Line 26 uses this shortcut. The returned value is used to create the new sub-graph output edges (step 4 of the sub-graph replacement).
Default implementation of the `FrontReplacementOp` class removes matched node and all its input/output edges (step 5 of the sub-graph replacement).
Another example of such kind of replacement is in the `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/Sub.py` class where all instances of `Sub` operations are replaced with two operations: `Power` to negate the second argument and the `Eltwise` to perform elementwise add.
## Replace Sub-graph of Operations with a New Sub-graph of Operations
The previous example considered situation when one single node of a specific type is replaced. When it is necessary to replace a sub-graph of operations it is necessary to tell Model Optimizer how to identify this sub-graph. There are three ways to achieve that:
* Use graph isomorphism pattern of the networkx module
* Use nodes name pattern to identify `scope` (according to TensorFlow terminology) to be replaced
* Use sets of `start` and `end` node names to match all nodes "between" them
The next sections explain each option using real examples.
### Replace Sub-graph of Operations Using Graph Isomorphism Pattern <a name="replace-using-isomorphism-pattern"></a>
networkx Python\* module provides methods to find graph isomorphic to the given one using nodes and edges match: for example, `networkx.algorithms.isomorphism.categorical_node_match`, `networkx.algorithms.isomorphism.categorical_multiedge_match`. Model Optimizer uses these methods and provides simple API to use that feature.
For example, the Caffe\* has layer called [Mean-Variance Normalization (MVN)](http://caffe.berkeleyvision.org/tutorial/layers/mvn.html), which is also supported by the Inference Engine. This layer is implemented with low-level operations in TensorFlow: `Mean`, `StopGradient`, `SquaredDifference`, `Squeeze` and `FusedBatchNorm`. Model Optimizer should replace sub-graph with these operations with a single Inference Engine layer of type `MVN`.
The file `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/tf/mvn.py` performs such a replacement. The first part of the file is:
```python
class MVN(FrontReplacementSubgraph):
enabled = True
def pattern(self):
log.debug('Enabled MVN replacement')
return dict(
nodes=[
('mean', dict(op='Mean')),
('stop_grad', dict(op='StopGradient')),
('sqdiff', dict(op='SquaredDifference')),
('variance', dict(op='Mean')),
('squeeze_mean', dict(op='Squeeze')),
('squeeze_variance', dict(op='Squeeze')),
('fbn', dict(op='FusedBatchNorm')),
],
edges=[
('mean', 'stop_grad', {'in': 0}),
('stop_grad', 'sqdiff', {'in': 1}),
('sqdiff', 'variance', {'in': 0}),
('mean', 'squeeze_mean', {'in': 0}),
('variance', 'squeeze_variance', {'in': 0}),
('squeeze_mean', 'fbn', {'in': 3}),
('squeeze_variance', 'fbn', {'in': 4}),
],
node_attrs=['op'],
edge_attrs=['in'])
```
**Key lines**:
* Line 1: Defines class `MVN` inherited from class `FrontReplacementSubgraph` that performs sub-graph replacement using sub-graph isomorphism pattern.
* Line 3: Sets class variable `enabled` to value True meaning that this replacer is enabled.
* The function `pattern` defines the sub-graph constraints to be matched. It returns a dictionary with four keys:
* the `nodes` defines a list of nodes to be matched. Each element in the list is a tuple. The first element is the alias name assigned for the matched node, the second element is a dictionary with desired attributes of the node.
* the `edges` defines a list of edges to be matched. Each element in the list is a tuple. The first and the second elements are the start and end edge nodes alias names respectively. The third element is a dictionary with desired edge attributes.
* the `node_attrs` contains the names of nodes attributes to use during sub-graph isomorphism search.
* the `edge_attrs` contains the names of edges attributes to use during sub-graph isomorphism search.
The sub-graph is matched if all provided constraints are satisfied. If at least one node with desired attributes is missing or at least one defined edge is absent, the sub-graph is not matched.
* Line 9: Adds constraint that sub-graph should contain node with attribute `op` with value `Mean`. The matched node gets an alias name `mean`. The same way the line 10 add constrain for node `StopGradient`, the matched node gets an alias name `stop_grad`.
* Line 18: Defines edge from node with alias name `mean` to node with alias name `stop_grad` having attribute `in` equal to 0. This means that the output of node `mean` is connected to the node `stop_grad` as a first input (Model Optimizer uses zero-based indexing that is why `in` is 0). Another example of defining the edges constraints is in line 25 where the edge from `squeeze_mean` is connected to the `fbn` node as fourth input.
* Lines 26 - 27: Specify a list of attributes to be checked. In fact, these lists are just list of all keys in the dictionaries for node and edge attributes.
Now when the Model Optimizer knows how to find sub-graph (step 1 of the sub-graph replacement), it is necessary to implement function that will perform actual sub-graph replacement (step 2 and 3). The code for this function is:
```python
def replace_sub_graph(self, graph: nx.MultiDiGraph, match: dict):
fbn = match['fbn']
input = fbn.in_node(0)
log.debug('Found potential MVN pattern after {} with name {}'.format(input.op, input.name))
if input.id != match['mean'].in_node(0).id or input.id != match['sqdiff'].in_node(0).id:
return
log.debug('Confirmed MVN pattern after {} with name {}'.format(input.op, input.name))
MVN = Op.get_op_class_by_name('MVN')
mvn = MVN(graph, dict(
name=fbn.name + '/MVN_',
eps=fbn.eps,
required_reduction_indices=[1,2] if fbn.data_format == b'NHWC' else [2,3]
))
mvn.attrs['old_infer'] = mvn.attrs['infer']
mvn.attrs['infer'] = __class__.infer
mul = Eltwise(graph, dict(operation='mul', name=fbn.name + '/Mul_'))
add = Eltwise(graph, dict(operation='sum', name=fbn.name + '/Add_'))
input_gamma = fbn.in_node(1)
input_beta = fbn.in_node(2)
mean_reduction = match['mean'].in_node(1)
variance_reduction = match['mean'].in_node(1)
new_subgraph = add.create_node([
mul.create_node([
mvn.create_node([input, mean_reduction, variance_reduction]),
input_gamma
]),
input_beta
])
replace_node(fbn, new_subgraph)
```
The function accepts two arguments - the graph and the dictionary `match`. The keys in the dictionary are the alias names of matched nodes (defined in the `nodes` list in the function `pattern`) and the values are the matched node of the graph (the instance of Node object).
The function generates new sub-graph with node of type `MVN` and two nodes of the type `Eltwise` calculating sum and product. There is nothing interesting in how the graph is generated and mathematics behind that, so attention will be put to two aspects of this function.
The first one is the call to function `replace_node` in line 36. `FusedBatchNorm` node is replaced with the output node of the generated sub-graph: all input edges of the `FusedBatchNorm` node are re-connected to the `new_subgraph` node, all consumers of the `FusedBatchNorm` node are updated to get inputs from the `new_subgraph` node. This action connects newly generated sub-graph with an existing graph (step 4 of the sub-graph replacement).
The second one is that the default implementation of the inference function for `MVN` operation is overwritten. In line 16, the default implementation of the inference function for `MVN` is saved to attribute `old_infer`. In line 17, the new inference function is saved to the instance of the `MVN` operation class. The new inference function code looks the following way:
```python
@staticmethod
def infer(node: Node):
if not(node.in_node(1).has_valid('value') and node.in_node(2).has_valid('value')):
log.warning('Reduction indices for mean and variance for MVN node {} are not constants'.format(node.name))
return
if not(all(node.in_node(1).value == node.required_reduction_indices) and
all(node.in_node(2).value == node.required_reduction_indices)):
log.warning('Reduction indices for mean {} and variance {} do not match required ones {}'.format(
node.in_node(1).value,
node.in_node(2).value,
node.required_reduction_indices
))
return
node.graph.remove_edge(node.in_node(1).id, node.id)
node.graph.remove_edge(node.in_node(2).id, node.id)
node.old_infer(node)
```
The `infer` function is needed to infer value of the node (if it is possible) and to infer shapes of the output tensors of the node (mandatory). The custom `infer` function performs additional checks that describe limitations of the `MVN` layer implementation in the Inference Engine. For example, reduction indices for mean and variance must be constants (line 10), while in TensorFlow they could be computed during model inference. In addition, the function removes two edges from the graph (lines 17 and 18) because all required information is already stored in the `MVN` node attributes. This is due to different `MVN` layer implementation in Inference Engine and TensorFlow\*: `mean` and `variance` are attributes of the node in Inference Engine while in TensorFlow they are input tensors. Edges are not removed in the `replace_sub_graph` function, because these edges are used in the `infer` function (lines 7-12).
The last action in the `infer` method (line 19) is to call default infer function for the `MVN`, which is saved in the attribute `old_infer` of the node to infer output tensors shapes.
On the step 5 of the sub-graph replacement, six matching nodes are automatically removed during the dead code elimination pass that is performed after applying of custom sub-graph replacements defined. Six matching nodes are no more connected to the inputs of the network after replacing node `fbn` with a newly created sub-graph node. Since they are not marked as output nodes (using `--output` command line parameter), they could be removed.
The replacement works for all sub-graph isomorphism instances found in the network.
### Replace Sub-graph of Operations Using Nodes Name Pattern
TensorFlow uses a mechanism of scope to group related operation nodes. It is a good practice to put nodes performing particular task into the scope. This approach divides a graph into logical blocks that are easier to review in TensorBoard\*. The `scope`, in fact, just defines a common prefix for the node names in the scope.
For example, Inception topologies contain several types of so-called "Inception blocks". Some of them are exactly equal to each other, but located in different places of the network. For example, Inception V4 from `tensorflow.contrib.slim` module has inception blocks `Mixed_5b`, `Mixed_5c` and `Mixed_5d` with exactly the same nodes with the same attributes.
Now consider situation when someone implemented these Inception blocks extremely efficiently using single Inference Engine custom layer called `InceptionBlock` and would like to replace these blocks with instances of the layer to decrease inference time. Model Optimizer provides mechanism to replace sub-graph of operations defined by the regular expressions for the node names prefixes (scope). In this particular case, some of the patterns are: `.*InceptionV4/Mixed_5b`, `.*InceptionV4/Mixed_5c` and `.*InceptionV4/Mixed_5d`. Each pattern starts with `.*`, because a prefix `InceptionV4` is added to all nodes names during a model freeze.
The sub-graph replacement using nodes name pattern is a bit trickier than replacements of single operation and networkx isomorphism pattern described above. You should do the following additional steps in comparison with previously described replacements:
1. Prepare configuration file template defining node names patterns and information about custom layer attributes.
2. Run Model Optimizer with command line parameter to add information about input and output nodes of the specified sub-graphs.
Consider the following possible configuration file for the Inception Block replacer:
```json
[
{
"custom_attributes": {
"attr1_key": "attr1_value",
"attr2_key": 123456
},
"id": "InceptionBlockReplacer",
"op": "InceptionBlock",
"instances": [
".*InceptionV4/Mixed_5b",
".*InceptionV4/Mixed_5c",
".*InceptionV4/Mixed_5d"
],
"match_kind": "scope"
}
]
```
The `.json` file contains list of dictionaries. Each dictionary defines one replacement. Each replacement is defined with several keys:
* `id` (mandatory) is a unique identifier of the replacer. It is used in the Python\* code that implements sub-graph replacement to link the class and the replacement description from the configuration file.
* `match_kind` (mandatory) is a string that specifies what matching algorithm is used. Currently supported `scope` and `points`. In this example, the first one is considered. The `points` match kind is described below.
* `instances` (mandatory) specifies instances of the sub-graph to be matched. It contains a list of node names prefixes patterns for the match kind `scope`.
* `custom_attributes` (optional) is a dictionary with static attributes of the layer to be dumped to Inference Engine Intermediate Representation `.xml` file.
* `op` (optional) is used only if the sub-graph replacement Python code is not needed, because the sub-graph should be replaced with a single node of type `op`. If this attribute is not set, it is necessary to implement Python code with sub-graph generation code. Both options are considered in this example.
When the configuration file is ready, run the Model Optimizer with regular command line parameters pointing to the file with model and input shapes (if necessary) and additional parameter `--tensorflow_custom_operations_config_update` pointing to the generated configuration file. If the file is correct, Model Optimizer adds two keys to the `InceptionBlockReplacer` dictionary: `inputs` and `outputs` with the following content:
```json
[
{
"id": "InceptionBlockReplacer",
...
"inputs": [
[
{
"node": "Branch_2/Conv2d_0a_1x1/Conv2D$",
"port": 0
},
{
"node": "Branch_3/AvgPool_0a_3x3/AvgPool$",
"port": 0
},
{
"node": "Branch_1/Conv2d_0a_1x1/Conv2D$",
"port": 0
},
{
"node": "Branch_0/Conv2d_0a_1x1/Conv2D$",
"port": 0
}
]
],
"outputs": [
{
"node": "concat$",
"port": 0
}
]
}
]
```
The value for key `inputs` is a list of lists describing input tensors of the sub-graph. Each element of the top-level list corresponds to one unique input tensor of the sub-graph. Each internal list describes a list of nodes consuming this tensor and port numbers where the tensor is consumed. Model Optimizer generates regular expressions for the input nodes names to uniquely identify them in each instance of the sub-graph defined by the `instances`. Denote these nodes as input nodes of the sub-graph.
In the InceptionV4 topology, the `InceptionV4/Mixed_5b` block has four input tensors from outside of the sub-graph, but all of them are produced by the node `InceptionV4/Mixed_5a/concat`. Therefore, the top-level list of the `inputs` contains one list corresponding to this tensor. Four input nodes of the sub-graph consume the tensor produced by `InceptionV4/Mixed_5a/concat` node. In this case, all four input nodes consume input tensor into port 0.
The order of items in the internal list describing nodes does not matter, but the order of elements in the top-level list is important. This order defines the order in which the Model Optimizer attaches input tensors to a new generated node if the sub-graph is replaced with a single node. The i-th input node of the sub-graph is obtained using call `match.single_input_node(i)` in the sub-graph replacer code. More information about API is given below. If you need to change the order of input tensors, you can edit the configuration file in the text-editor.
The value for the key `outputs` is a list describing nodes of the sub-graph producing tensor that goes outside of the sub-graph or does not have child nodes. Denote these nodes as output nodes of the sub-graph. The order of elements in the list is important. The i-th element of the list describes the i-th output tensor of the sub-graph, which could be obtained using call `match.output_node(i)`. The order of elements can be manually changed in the configuration file. Model Optimizer uses this order to connect output edges if the sub-graph is replaced with a single node.
Now, when meaning of `inputs` and `outputs` attributes is clean, return back to the replacer implementation. The replacer `InceptionBlockReplacer` contains attribute `op` with the value `InceptionBlock`, which means that the identified sub-graph should be replaced with a single layer of type `InceptionBlock`. This layer is not known for the Model Optimizer, so it is necessary to define it. See [Extending the Model Optimizer with New Primitives](Extending_Model_Optimizer_with_New_Primitives.md). You must create file `extension/ops/InceptionBlock.py` with the following content:
```python
import numpy as np
from mo.graph.graph import Node
from mo.ops.op import Op
class InceptionBlock(Op):
op = "InceptionBlock"
enabled = True
def __init__(self, graph, attrs):
super().__init__(graph, attrs, {
'type': __class__.op,
'op': __class__.op,
})
```
The shape inference function is not defined. In this case, Model Optimizer uses TensorFlow fallback to calculate shapes of the sub-graph output tensors.
Run the Model Optimizer with the regular command line parameters, path to the model file and input shape (if necessary), and the parameter `--tensorflow_use_custom_operations_config` and point to the created configuration file. Model Optimizer generates Intermediate Representation `.xml` file with three sequential layers of type `InceptionBlock` like in the following example:
```xml
<layer id="1658" name="InceptionBlock1877" precision="FP32" type="InceptionBlock">
<input>
<port id="0">
<dim>1</dim>
<dim>384</dim>
<dim>35</dim>
<dim>35</dim>
</port>
</input>
<output>
<port id="1">
<dim>1</dim>
<dim>384</dim>
<dim>35</dim>
<dim>35</dim>
</port>
</output>
</layer>
```
The implementation of the sub-graph replacement by scope with a single layer is complete. The next subsection explains
how Model Optimizer replaces sub-graph identified by start/end nodes (`points`) with another sub-graph.
### <a name="sub_graph_replacement_using_points"></a> Replace Sub-graph of Operations Using Points
In this scenario, for the matching algorithm user defines the sub-graph via a set of "start" and "end" nodes.
Given the set, the Model Optimizer performs the following steps:
1. Starts graph traversal from every _start_ nodes following the direction of the graph edges.
The search stops in _end_ nodes or in case of nodes without further children. All visited nodes are added to the matched sub-graph.
2. Starts another graph traversal from each non-start node of the sub-graph, i.e. every node except nodes from "start" set.
In this step the edges are traversed in the opposite edge direction. All newly visited nodes are added to the
matched sub-graph. This step is needed to add nodes required for calculation values of internal nodes of the
matched sub-graph.
3. Checks that all "end" nodes were reached from "input" nodes. If no then exit with error.
4. Check that there are no "Placeholder" operations among added nodes. If it is not true then some side branch of
the sub-graph (added in step 2) depends on inputs of the network. Such configuration is not correct so exit with error.
This algorithm finds all nodes "between" start and end nodes. Also nodes needed for calculation of non-input nodes of the
matched sub-graph produce _constant_ values because they do not depend on input of the network.
**This sub-graph match has a limitation that each start node must have only one input**. Therefore, it is not possible
to specify, for example, convolution node as input because it has two inputs: data tensor and tensor with weights.
For example of replacement with points, please refer to the case-study of the
[conversion for the SSD models, created with TensorFlow Object Detection API](TensorFlow_SSD_ObjectDetection_API.md).
The document has been deprecated. Refer to the [Model Optimizer Extensibility](Subgraph_Replacement_Model_Optimizer.md)
for the up-to-date documentation.

449
docs/MO_DG/prepare_model/customize_model_optimizer/TensorFlow_Faster_RCNN_ObjectDetection_API.md
View File

@ -1,449 +0,0 @@
# Converting Faster R-CNN models, created with TensorFlow Object Detection API {#openvino_docs_MO_DG_prepare_model_customize_model_optimizer_TensorFlow_Faster_RCNN_ObjectDetection_API}
This is a deprecated page. Please, consider reading [this](../convert_model/tf_specific/Convert_Object_Detection_API_Models.md) page describing new approach to convert Object Detection API models giving closer to TensorFlow inference results.
## Converting models created with TensorFlow Object Detection API version equal or higher than 1.6.0
This chapter describes how to convert selected Faster R-CNN models from the TensorFlow Object Detection API zoo version equal or higher than 1.6.0. The full list of supported models is provided in the table below. Note that currently batch size 1 is supported only. The only Inference Engine plugin supporting these topologies inference is CPU.
The Faster R-CNN models contain several building blocks similar to building blocks from SSD models so it is highly recommended to read chapter about [enabling TensorFlow Object Detection API SSD models](TensorFlow_SSD_ObjectDetection_API.md) first. Detailed information about Faster R-CNN topologies is provided [here](https://arxiv.org/abs/1506.01497).
The TensorFlow network consists of a number of big blocks grouped by scope:
* `Preprocessor` performs scaling/resizing of the image and converts input data to [0, 1] interval. Has two outputs: the first one is modified input image and the second one is a constant tensor with shape (batch_size, 3) and values (resized_image_height, resized_image_width, 3).
* `FirstStageFeatureExtractor` is a backbone feature extractor.
* `FirstStageBoxPredictor` calculates boxes and classes predictions.
* `GridAnchorGenerator` generates anchors coordinates.
* `ClipToWindow` crops anchors to the resized image size.
* `Decode` decodes coordinates of boxes using anchors and data from the `FirstStageBoxPredictor`.
* `BatchMultiClassNonMaxSuppression` performs non maximum suppression.
* `map` scales coordinates of boxes to [0, 1] interval by dividing coordinates by (resized_image_height, resized_image_width).
* `map_1` scales coordinates from [0, 1] interval to resized image sizes.
* `SecondStageFeatureExtractor` is a feature extractor for predicted Regions of interest (ROIs).
* `SecondStageBoxPredictor` refines box coordinates according `SecondStageFeatureExtractor`.
* `SecondStagePostprocessor` is Detection Output layer performing final boxes predictions.
### Sub-graph replacements
There are three sub-graph replacements defined in the `extensions/front/tf/legacy_faster_rcnn_support.json` used to convert these models:
* the first one replaces the `Preprocessor` block. The implementation of this replacer is in the `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/tf/Preprocessor.py`
* the second one replaces a number of blocks in the the graph including `GridAnchorGenerator`, `ClipToWindow`, `Decode`, `BatchMultiClassNonMaxSuppression`, `Tile`, `Tile_1` and `map` with Proposal and ROIRooling layers and some additional layers to pre-process input data
* the third one replaces `SecondStagePostprocessor` with a DetectionOutput layer.
The second replacer is defined using the following configuration that matches sub-graph by points:
```json
{
"custom_attributes": {
"nms_threshold": 0.7,
"feat_stride": 16,
"max_proposals": 100,
"anchor_base_size": 256,
"anchor_scales": [0.25, 0.5, 1.0, 2.0],
"anchor_aspect_ratios": [0.5, 1.0, 2.0],
"roi_spatial_scale": 0.0625
},
"id": "TFObjectDetectionAPIFasterRCNNProposalAndROIPooling",
"include_inputs_to_sub_graph": true,
"include_outputs_to_sub_graph": true,
"instances": {
"end_points": [
"CropAndResize",
"map_1/TensorArrayStack/TensorArrayGatherV3",
"map_1/while/strided_slice/Enter",
"BatchMultiClassNonMaxSuppression/map/TensorArrayStack_4/TensorArrayGatherV3"
],
"start_points": [
"FirstStageBoxPredictor/concat",
"FirstStageBoxPredictor/concat_1",
"GridAnchorGenerator/Identity",
"Shape",
"CropAndResize"
]
},
"match_kind": "points"
}
```
The `start_points` list contains the following nodes:
* `FirstStageBoxPredictor/concat` node produces box coordinates predictions.
* `FirstStageBoxPredictor/concat_1` node produces classes predictions which will be used for the ROIs
* `GridAnchorGenerator/Identity` node produces anchors coordinates.
* `Shape` and `CropAndResize` nodes are specified as inputs to correctly isolate the required sub-graph. Refer to the [chapter](Subgraph_Replacement_Model_Optimizer.md) for more information about replacements by points.
The `end_points` list contains the following nodes:
* `CropAndResize` is the node that performs ROI pooling operation.
* `map_1/TensorArrayStack/TensorArrayGatherV3`, `map_1/while/strided_slice/Enter` and `BatchMultiClassNonMaxSuppression/map/TensorArrayStack_4/TensorArrayGatherV3` are specified to correctly isolate the sub-graph.
The `custom_attributes` dictionary contains attributes where most values are taken from the topology-specific configuration file `samples/configs/faster_rcnn_*.config` of the [TensorFlow Object Detection API repository](https://github.com/tensorflow/models/tree/master/research/object_detection):
* `nms_threshold` is the value of the `first_stage_nms_iou_threshold` parameter.
* `feat_stride` is the value of the `height_stride` and `width_stride` parameters. Inference Engine supports case when these two values are equal that is why the replacement configuration file contains just one parameter.
* `max_proposals` is the value of the `max_total_detections` parameter which is a maximum number of proposal boxes from the Proposal layer and detected boxes.
* `anchor_base_size` is the base size of the generated anchor. The 256 is the default value for this parameter and it is not specified in the configuration file.
* `anchor_scales" is the value of the `scales` attrbite.
* `anchor_aspect_ratios` is the value of the `aspect_ratios` attribute.
* `roi_spatial_scale` is needed for the Inference Engine ROIPooling layer. It is the default value that is not actually used.
The identifier for this replacer is `TFObjectDetectionAPIFasterRCNNProposalAndROIPooling`. The Python implementation of this replacer is in the file `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/tf/FasterRCNNs.py`.
The first four functions of the replacer class are the following:
```python
class TFObjectDetectionAPIFasterRCNNProposalAndROIPooling(FrontReplacementFromConfigFileSubGraph):
"""
This class replaces sub-graph of operations with Proposal and ROIPooling layers and additional layers transforming
tensors from layout of TensorFlow to layout required by Inference Engine.
Refer to comments inside the function for more information about performed actions.
"""
replacement_id = 'TFObjectDetectionAPIFasterRCNNProposalAndROIPooling'
def run_after(self):
return [PreprocessorReplacement]
def run_before(self):
return [SecondStagePostprocessorReplacement]
def output_edges_match(self, graph: nx.DiGraph, match: SubgraphMatch, new_sub_graph: dict):
return {match.output_node(0)[0].id: new_sub_graph['roi_pooling_node'].id}
def nodes_to_remove(self, graph: nx.MultiDiGraph, match: SubgraphMatch):
new_list = match.matched_nodes_names().copy()
# do not remove nodes that produce box predictions and class predictions
new_list.remove(match.single_input_node(0)[0].id)
new_list.remove(match.single_input_node(1)[0].id)
return new_list
```
The function `run_after` returns list of Python classes inherited from one of the replacer classes (`FrontReplacementOp`, `FrontReplacementPattern`, `FrontReplacementFromConfigFileSubGraph` etc) those current sub-graph replacement class must be run after. In this case the replacer must be run after the `Preprocessor` is removed by the `PreprocessorReplacement` replacer. Similar way the `run_before` function is used to tell Model Optimizer to execute `SecondStagePostprocessorReplacement` before this replacer.
The `output_edges_match` function describes matching between the output nodes of the sub-graph before replacement and after. In this case the only needed output node of the sub-graph is the `CropAndResize` node which is identified with `match.output_node(0)[0]`. The new output node which is created in the `generate_sub_graph` function is identified with `new_sub_graph['roi_pooling_node']`.
The `nodes_to_remove` function takes the default list of nodes to be removed which contains all matched nodes and remove from them two input nodes which are identified with `match.single_input_node(0)[0]` and `match.single_input_node(1)[0]`. These nodes will be connected as inputs to new nodes being generated in the `generate_sub_graph` function so they should node be removed.
The code generating new sub-graph is the following:
```python
def generate_sub_graph(self, graph: nx.MultiDiGraph, match: SubgraphMatch):
log.debug('TFObjectDetectionAPIFasterRCNNProposal: matched_nodes = {}'.format(match.matched_nodes_names()))
config_attrs = match.custom_replacement_desc.custom_attributes
nms_threshold = config_attrs['nms_threshold']
feat_stride = config_attrs['feat_stride']
max_proposals = config_attrs['max_proposals']
anchor_base_size = config_attrs['anchor_base_size']
roi_spatial_scale = config_attrs['roi_spatial_scale']
proposal_ratios = config_attrs['anchor_aspect_ratios']
proposal_scales = config_attrs['anchor_scales']
anchors_count = len(proposal_ratios) * len(proposal_scales)
```
These lines get parameters defined in the sub-graph replacement configuration file and calculate initial anchors count.
```python
# get the ROIPool size from the CropAndResize which performs the same action
if 'CropAndResize' not in graph.nodes():
raise Error('Failed to find node with name "CropAndResize" in the topology. Probably this is not Faster'
' RCNN topology or it is not supported')
roi_pool_size = Node(graph, 'CropAndResize').in_node(3).value[0]
```
The code above gets the ROI Pooling spatial output dimension size as a value from the fourth argument of the node with name `CropAndResize`.
```python
# Convolution/matmul node that produces classes predictions
# Permute result of the tensor with classes permissions so it will be in a correct layout for Softmax
predictions_node = match.single_input_node(1)[0].in_node(0).in_node(0)
permute_predictions_op = Permute(graph, {'order': np.array([0, 2, 3, 1])})
permute_predictions_node = permute_predictions_op.create_node([], dict(name=predictions_node.name + '/Permute_'))
insert_node_after(predictions_node, permute_predictions_node, 0)
reshape_classes_op = Reshape(graph, {'dim': np.array([0, -1, 2])})
reshape_classes_node = reshape_classes_op.create_node([permute_predictions_node],
dict(name='Reshape_FirstStageBoxPredictor_Class_'))
update_attrs(reshape_classes_node, 'shape_attrs', 'dim')
softmax_conf_op = Softmax(graph, {'axis': 1})
softmax_conf_node = softmax_conf_op.create_node([reshape_classes_node],
dict(name='FirstStageBoxPredictor_SoftMax_Class_'))
```
The output with class predictions from the `FirstStageBoxPredictor` is generated with a convolution operation. The convolution output data layout in TensorFlow is NHWC while Inference Engine uses NCHW layout. Model Optimizer by default converts the weights of TensorFlow convolutions to produce output tensor in NCHW layout required by Inference Engine. The issue arises because the class predictions tensor is passed through the Softmax operation to produce class probabilities. The Inference Engine Softmax is performed over the fastest-changing dimension which is 'W' in Inference Engine. Thus, the softmax operation will be performed over a wrong dimension after conversion of the convolution node producing classes predicitions. The solution is to add Permute and Reshape operations to prepare the input data for Softmax. The Reshape operation is required to make the size of the fastest-changing dimension equal to 2, because there are 2 classes being predicted: background and foreground.
Another issue is that layout of elements in the predicted classes tensor is different between TensorFlow and Inference Engine Proposal layer requirements. In TensorFlow the tensor has the following virtual layout [N, H, W, num_anchors, num_classes] while the Inference Engine Proposal layer requires in the following virtual layout [N, num_classes, num_anchors, H, W]. Thus, it is necessary to reshape, permute and then reshape again output from the Softmax to the required shape for the Proposal layer:
```python
reshape_softmax_op = Reshape(graph, {'dim': np.array([1, anchors_count, 2, -1])})
reshape_softmax_node = reshape_softmax_op.create_node([softmax_conf_node], dict(name='Reshape_Softmax_Class_'))
update_attrs(reshape_softmax_node, 'shape_attrs', 'dim')
permute_reshape_softmax_op = Permute(graph, {'order': np.array([0, 1, 3, 2])})
permute_reshape_softmax_node = permute_reshape_softmax_op.create_node([reshape_softmax_node],
dict(name='Permute_'))
# implement custom reshape infer function because we need to know the input convolution node output dimension
# sizes but we can know it only after partial infer
reshape_permute_op = Reshape(graph, {'dim': np.ones([4]), 'anchors_count': anchors_count,
'conv_node': predictions_node})
reshape_permute_op.attrs['old_infer'] = reshape_permute_op.attrs['infer']
reshape_permute_op.attrs['infer'] = __class__.classes_probabilities_reshape_shape_infer
reshape_permute_node = reshape_permute_op.create_node([permute_reshape_softmax_node],
dict(name='Reshape_Permute_Class_'))
update_attrs(reshape_permute_node, 'shape_attrs', 'dim')
```
The Proposal layer has 3 inputs: classes probabilities, boxes predictions and a input shape of the image. The first two tensors are ready so it is necessary to create the Const operation that produces the desired third input tensor.
```python
# create constant input with the image height, width and scale H and scale W (if present) required for Proposal
const_value = np.array([[input_height, input_width, 1]], dtype=np.float32)
const_op = Const(graph, dict(value=const_value, shape=const_value.shape))
const_node = const_op.create_node([], dict(name='Proposal_const_image_size_'))
```
Now add the Proposal layer:
```python
proposal_op = ProposalOp(graph, dict(min_size=10, framework='tensorflow', box_coordinate_scale=10,
box_size_scale=5, post_nms_topn=max_proposals, feat_stride=feat_stride,
ratio=proposal_ratios, scale=proposal_scales, base_size=anchor_base_size,
pre_nms_topn=2**31 - 1,
nms_thresh=nms_threshold))
proposal_node = proposal_op.create_node([reshape_permute_node,
match.single_input_node(0)[0].in_node(0).in_node(0),
const_node],
dict(name=proposal_op.attrs['type'] + '_'))
```
The box coordinates in the TensorFlow are in the following layout "YXYX" while Inference Engine uses "XYXY" layout so it is necessary to swap coordinates produced by Proposal layer. It is implemented with help of a convolution node with a special filter of a size [5, 5]:
```python
proposal_reshape_4d_op = Reshape(graph, {'dim': np.array([max_proposals, 1, 1, 5])})
proposal_reshape_4d_node = proposal_reshape_4d_op.create_node([proposal_node], dict(name="reshape_4d_"))
update_attrs(proposal_reshape_4d_node, 'shape_attrs', 'dim')
# create convolution node to swap X and Y coordinates in the proposals
conv_filter_const_data = np.array(np.array([[1, 0, 0, 0, 0],
[0, 0, 1, 0, 0],
[0, 1, 0, 0, 0],
[0, 0, 0, 0, 1],
[0, 0, 0, 1, 0]],
dtype=np.float32).reshape([1, 1, 5, 5]), dtype=np.float32)
conv_filter_const_op = Const(graph, dict(value=conv_filter_const_data, spatial_dims=np.array([2, 3])))
conv_filter_const_node = conv_filter_const_op.create_node([], dict(name="conv_weights"))
conv_op = Op(graph, {
'op': 'Conv2D',
'bias_addable': False,
'spatial_dims': np.array([1, 2]),
'channel_dims': np.array([3]),
'batch_dims': np.array([0]),
'pad': None,
'pad_spatial_shape': None,
'input_feature_channel': 2,
'output_feature_channel': 2,
'output_shape': [max_proposals, 1, 1, 5],
'dilation': np.array([1, 1, 1, 1], dtype=np.int64),
'stride': np.array([1, 1, 1, 1]),
'type': 'Convolution',
'group': None,
'layout': 'NHWC',
'infer': __class__.fake_conv_shape_infer})
predictions_node = conv_op.create_node([proposal_reshape_4d_node, conv_filter_const_node], dict(name="conv_"))
update_ie_fields(graph.node[predictions_node.id])
proposal_reshape_2d_op = Reshape(graph, {'dim': np.array([max_proposals, 5])})
proposal_reshape_2d_node = proposal_reshape_2d_op.create_node([predictions_node], dict(name="reshape_2d_"))
# set specific name for this Reshape operation so we can use it in the DetectionOutput replacer
proposal_reshape_2d_node['name'] = 'swapped_proposals'
```
The ROIPooling layer in TensorFlow is implemented with operation called `CropAndResize` with bi-linear filtration. Inference Engine implementation of the ROIPooling layer with bi-linear filtration requires input boxes coordinates be scaled to [0, 1] interval. Adding elementwise multiplication of box coordinates solves this issue:
```python
# the TF implementation of Proposal with bi-linear filtration need proposals scaled by image size
proposal_scale_const = np.array([1.0, 1 / input_height, 1 / input_width, 1 / input_height, 1 / input_width],
dtype=np.float32)
proposal_scale_const_op = Const(graph, dict(value=proposal_scale_const, shape=proposal_scale_const.shape))
proposal_scale_const_node = proposal_scale_const_op.create_node([], dict(name='Proposal_scale_const_'))
scale_proposals_op = Eltwise(graph, {'operation': 'mul'})
scale_proposals_node = scale_proposals_op.create_node([proposal_reshape_2d_node, proposal_scale_const_node],
dict(name='scale_proposals_'))
```
The last step is to create the ROIPooling node with 2 inputs: the identified feature maps from the `FirstStageFeatureExtractor` and the scaled output of the Proposal layer:
```python
feature_extractor_output_nodes = scope_output_nodes(graph, 'FirstStageFeatureExtractor')
if len(feature_extractor_output_nodes) != 1:
raise Error("Failed to determine FirstStageFeatureExtractor output node to connect it to the ROIPooling."
"Found the following nodes: {}".format([node.name for node in feature_extractor_output_nodes]))
roi_pooling_op = ROIPooling(graph, dict(method="bilinear", framework="tensorflow",
pooled_h=roi_pool_size, pooled_w=roi_pool_size,
spatial_scale=roi_spatial_scale))
roi_pooling_node = roi_pooling_op.create_node([feature_extractor_output_nodes[0], scale_proposals_node],
dict(name='ROI_Pooling_'))
return {'roi_pooling_node': roi_pooling_node}
```
The are two additional methods implemented in the replacer class:
* The `fake_conv_shape_infer` is a silly infer function for the convolution that permutes X and Y coordinates of the Proposal output which avoids setting a lot of internal attributes required for propoper shape inference.
* The "classes_probabilities_reshape_shape_infer" function is used to update the output dimension of the reshape operation. The output spatial dimensions depends on the convolution output spatial dimensions thus they are not known until the shape inference pass which is performed after this sub-graph replacement class. So this custom infer function is called instead of default Reshape shape inference function, updates the required attribute "dim" of the node with the convolution output spatial dimensions which are known at the time of calling this inference function and then call the default Reshape inference function.
```python
@staticmethod
def fake_conv_shape_infer(node: Node):
node.out_node(0).shape = node.in_node(0).shape
# call functions to update internal attributes required for correct IR generation
mark_input_bins(node)
assign_dims_to_weights(node.in_node(1), [0, 1], node.input_feature_channel, node.output_feature_channel, 4)
@staticmethod
def classes_probabilities_reshape_shape_infer(node: Node):
# now we can determine the reshape dimensions from Convolution node
conv_node = node.conv_node
conv_output_shape = conv_node.out_node().shape
# update desired shape of the Reshape node
node.dim = np.array([0, conv_output_shape[1], conv_output_shape[2], node.anchors_count * 2])
node.old_infer(node)
```
The second replacer defined in the sub-graph replacement configuration file replaces the `SecondStagePostprocessor` block and is defined using scope:
```json
{
"custom_attributes": {
"code_type": "caffe.PriorBoxParameter.CENTER_SIZE",
"confidence_threshold": 0.01,
"keep_top_k": 300,
"nms_threshold": 0.6,
"pad_mode": "caffe.ResizeParameter.CONSTANT",
"resize_mode": "caffe.ResizeParameter.WARP",
"max_detections_per_class": 100,
"num_classes": 90
},
"id": "SecondStagePostprocessorReplacement",
"inputs": [
[
{
"node": "Reshape$",
"port": 0
}
],
[
{
"node": "Reshape_1$",
"port": 0
}
],
[
{
"node": "ExpandDims$",
"port": 0
}
]
],
"instances": [
".*SecondStagePostprocessor/"
],
"match_kind": "scope",
"outputs": [
{
"node": "BatchMultiClassNonMaxSuppression/map/TensorArrayStack/TensorArrayGatherV3$",
"port": 0
}
]
}
```
The replacement code is similar to the `SecondStagePostprocessor` replacement for the SSDs topologies. The are two major difference:
* The tensor with bounding boxes doesn't contain locations for class 0 (background class) but Inference Engine Detection Output layer requires it. The Const node with some dummy values are created and concatenated with the tensor.
* The priors tensor is not constant like in SSDs so the bounding boxes tensor must be scaled with variances [0.1, 0.1, 0.2, 0.2].
The described above difference are resolved with the following code:
```python
# TF produces locations tensor without boxes for background.
# Inference Engine DetectionOutput layer requires background boxes so we generate them with some values
# and concatenate with locations tensor
fake_background_locs_blob = np.tile([[[1, 1, 2, 2]]], [max_detections_per_class, 1, 1])
fake_background_locs_const_op = Const(graph, dict(value=fake_background_locs_blob,
shape=fake_background_locs_blob.shape))
fake_background_locs_const_node = fake_background_locs_const_op.create_node([])
reshape_loc_op = Reshape(graph, {'dim': np.array([max_detections_per_class, num_classes, 4])})
reshape_loc_node = reshape_loc_op.create_node([match.single_input_node(0)[0].in_node(0)],
dict(name='Reshape_loc_'))
concat_loc_op = Concat(graph, {'axis': 1})
concat_loc_node = concat_loc_op.create_node([fake_background_locs_const_node, reshape_loc_node],
dict(name='Concat_fake_loc_'))
# blob with variances
variances_blob = np.array([0.1, 0.1, 0.2, 0.2])
variances_const_op = Const(graph, dict(value=variances_blob, shape=variances_blob.shape))
variances_const_node = variances_const_op.create_node([])
# reshape locations tensor to 2D so it could be passed to Eltwise which will be converted to ScaleShift
reshape_loc_2d_op = Reshape(graph, {'dim': np.array([-1, 4])})
reshape_loc_2d_node = reshape_loc_2d_op.create_node([concat_loc_node], dict(name='reshape_locs_2d_'))
# element-wise multiply locations with variances
eltwise_locs_op = Eltwise(graph, {'operation': 'mul'})
eltwise_locs_node = eltwise_locs_op.create_node([reshape_loc_2d_node, variances_const_node],
dict(name='scale_locs_'))
```
### Example of Model Optimizer Command-Line for TensorFlow's Faster R-CNNs
The final command line to convert Faster R-CNNs from the TensorFlow* Object Detection Zoo is the following:
```sh
./mo.py --input_model=<path_to_frozen.pb> --output=detection_boxes,detection_scores,num_detections --tensorflow_use_custom_operations_config extensions/front/tf/legacy_faster_rcnn_support.json
```
Note that there are minor changes that should be made to the and sub-graph replacement configuration file `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/tf/legacy_faster_rcnn_support.json` before converting particular Faster R-CNN topology. Refer to the table below.
### Sub-Graph Replacement Configuration File Parameters to Convert Different Faster R-CNN Models
|Model Name | Configuration File Changes|
|:----|:----:|
| faster_rcnn_inception_v2_coco | None
| faster_rcnn_resnet50_coco | None
| faster_rcnn_resnet50_lowproposals_coco | None
| faster_rcnn_resnet101_coco | None
| faster_rcnn_resnet101_lowproposals_coco | None
| faster_rcnn_inception_resnet_v2_atrous_coco | "feat_stride: 8"
| faster_rcnn_inception_resnet_v2_atrous_lowproposals_coco| "feat_stride: 8"

339
docs/MO_DG/prepare_model/customize_model_optimizer/TensorFlow_SSD_ObjectDetection_API.md
View File

@ -1,339 +0,0 @@
# (Deprecated) Case Study: Converting SSD Models Created with TensorFlow* Object Detection API {#openvino_docs_MO_DG_prepare_model_customize_model_optimizer_TensorFlow_SSD_ObjectDetection_API}
This is a deprecated page. Please, consider reading [this](../convert_model/tf_specific/Convert_Object_Detection_API_Models.md) page describing new approach to convert Object Detection API models giving closer to TensorFlow inference results.
## Converting Models Created with TensorFlow Object Detection API Version prior 1.6.0
As explained in the [Sub-graph Replacement in Model Optimizer](Subgraph_Replacement_Model_Optimizer.md) section, there are multiple
ways to setup the sub-graph matching. In this example we are focusing on the defining the sub-graph via a set of
"start" and "end" nodes.
The result of matching is two buckets of nodes:
* Nodes "between" start and end nodes.
* Nodes connected to the first list, but just on the constant path (e.g. these nodes are not connected to the inputs of the entire graph).
Let's look closer to the SSD models from the TensorFlow* detection model
<a href="https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md">zoo</a>:
[SSD MobileNet](http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2017_11_17.tar.gz) and
[SSD InceptionV2](http://download.tensorflow.org/models/object_detection/ssd_inception_v2_coco_2017_11_17.tar.gz).
* Nodes "between" start and end nodes
* Nodes connected to the first list, but just on the constant path (for example, these nodes are not connected to the inputs of the entire graph). Let's look closer to the SSD models from the TensorFlow\* detection model <a href="https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md">zoo</a> : [SSD MobileNet](http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2017_11_17.tar.gz) and [SSD InceptionV2](http://download.tensorflow.org/models/object_detection/ssd_inception_v2_coco_2017_11_17.tar.gz).
A distinct layer of any SSD topology is the `DetectionOutput` layer. This layer is implemented with a dozens of primitive operations in TensorFlow, while in Inference Engine, it is one [layer](../../../ops/opset.md). Thus, to convert a SSD model from the TensorFlow, the Model Optimizer should replace the entire sub-graph of operations that implement the `DetectionOutput` layer with a single well-known `DetectionOutput` node.
The Inference Engine `DetectionOutput` layer consumes three tensors in the following order:
1. Tensor with locations of bounding boxes
2. Tensor with confidences for each bounding box
3. Tensor with prior boxes (anchors in TensorFlow terminology)
`DetectionOutput` layer produces one tensor with seven numbers for each actual detection. There are more output tensors in the TensorFlow Object Detection API, but the values in them are consistent with the Inference Engine ones.
The difference with [other examples](Subgraph_Replacement_Model_Optimizer.md) is that here the `DetectionOutput` sub-graph is replaced with a new sub-graph (not a single layer).
Look at sub-graph replacement configuration file `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/tf/legacy_ssd_support.json` that is used to enable two models listed above:
```json
[
{
"custom_attributes": {
"code_type": "caffe.PriorBoxParameter.CENTER_SIZE",
"confidence_threshold": 0.01,
"keep_top_k": 200,
"nms_threshold": 0.45,
"pad_mode": "caffe.ResizeParameter.CONSTANT",
"resize_mode": "caffe.ResizeParameter.WARP"
},
"id": "TFObjectDetectionAPIDetectionOutput",
"include_inputs_to_sub_graph": true,
"include_outputs_to_sub_graph": true,
"instances": {
"end_points": [
"detection_boxes",
"detection_scores",
"num_detections"
],
"start_points": [
"Postprocessor/Shape",
"Postprocessor/Slice",
"Postprocessor/ExpandDims",
"Postprocessor/Reshape_1"
]
},
"match_kind": "points"
},
{
"custom_attributes": {
},
"id": "PreprocessorReplacement",
"inputs": [
[
{
"node": "map/Shape$",
"port": 0
},
{
"node": "map/TensorArrayUnstack/Shape$",
"port": 0
},
{
"node": "map/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3$",
"port": 2
}
]
],
"instances": [
".*Preprocessor/"
],
"match_kind": "scope",
"outputs": [
{
"node": "sub$",
"port": 0
},
{
"node": "map/TensorArrayStack_1/TensorArrayGatherV3$",
"port": 0
}
]
}
]
```
**Key lines**:
* Lines 3-10 define static attributes that will be saved to the Intermediate Representation `.xml` file for `DetectionOutput` layer.
* Lines 12 and 13 define values for attributes that should be always set to "true" for this release of the Model Optimizer. These two attributes are specific for sub-graph match by points only.
* Lines 14-26 define one instance of the sub-graph to be match. It is an important difference between sub-graph matching by scope and points. Several instances could be specified for matching by scope, but matching with points allows specifying just one instance. So the full node names (not regular expressions like in case of match with scope) are specified in `instances` dictionary.
The second sub-graph replacer with identifier `PreprocessorReplacement` is used to remove the `Preprocessing` block from the graph. The replacer removes all nodes from this scope except nodes performing mean value subtraction and scaling (if applicable). Implementation of the replacer is in the `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/tf/Preprocessor.py` file.
Now let's analyze the structure of the topologies generated with the Object Detection API. There are several blocks in the graph performing particular task:
* `Preprocessor` block resizes, scales and subtracts mean values from the input image.
* `FeatureExtractor` block is a [MobileNet](https://arxiv.org/abs/1704.04861) or other backbone to extract features.
* `MultipleGridAnchorGenerator` block creates initial bounding boxes locations (anchors).
* `Postprocessor` block acts as a `DetectionOutput` layer. So we need to replace `Postprocessor` block with `DetectionOutput` layer. It is necessary to add all input nodes of the `Postprocessor` scope to the list `start_points`. Consider inputs of each of these nodes:
* `Postprocessor/Shape` consumes tensor with locations.
* `Postprocessor/Slice` consumes tensor with confidences.
* `Postprocessor/ExpandDims` consumes tensor with prior boxes.
* `Postprocessor/Reshape_1` consumes tensor with locations similarly to the `Postprocessor/Shape` node. Despite the fact that the last node `Postprocessor/Reshape_1` gets the same tensor as node `Postprocessor/Shape`, it must be explicitly put to the list.
Object Detection API `Postprocessor` block generates output nodes: `detection_boxes`, `detection_scores`, `num_detections`, `detection_classes`.
Now consider the implementation of the sub-graph replacer, available in the `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/tf/SSDs.py`. The file is rather big, so only some code snippets are used:
```python
class PostprocessorReplacement(FrontReplacementFromConfigFileSubGraph):
replacement_id = 'TFObjectDetectionAPIDetectionOutput'
```
These lines define the new `PostprocessorReplacement` class inherited from `FrontReplacementFromConfigFileSubGraph`. `FrontReplacementFromConfigFileSubGraph` is designed to replace sub-graph of operations described in the configuration file. There are methods to override for implementing custom replacement logic that we need:
* `generate_sub_graph` performs new sub-graph generation and returns dictionary where key is an alias name for the node and value is a Node objects. The dictionary has the same format as parameter `match` in the `replace_sub_graph` method in the example with <a href="Subgraph_Replacement_Model_Optimizer.html#replace-using-isomorphism-pattern">networkx sub-graph isomorphism pattern</a>. This dictionary is passed as argument to the next three methods, so it should contain entries the for nodes that the functions need.
* `input_edges_match` specifies mapping between input edges to sub-graph before replacement and after replacement. The key of the dictionary is a tuple specifying input tensor of the sub-graph before replacement: sub-graph input node name and input port number for this node. The value for this key is also a tuple specifying the node where this tensor should be attached during replacement: the node name (or alias name of the node) and the input port for this node. If the port number is zero, the parameter could be omitted so the key or value is just a node name (alias). Default implementation of the method returns an empty dictionary, so Model Optimizer does not create new edges.
* `output_edges_match` returns mapping between old output edges of the matched nodes and new sub-graph node and output edge index. The format is similar to the dictionary returned in the `input_edges_match` method. The only difference is that instead of specifying input port numbers for the nodes it is necessary to specify output port number. Of course, this mapping is needed for the output nodes only. Default implementation of the method returns an empty dictionary, so the Model Optimizer does not create new edges.
* `nodes_to_remove` specifies list of nodes that Model Optimizer should remove after sub-graph replacement. Default implementation of the method removes all sub-graph nodes.
Review of the replacer code, considering details of the `DetectionOutput` layer implementation in the Inference Engine. There are several constraints to the input tensors of the `DetectionOutput` layer:
* The tensor with locations must be of shape `[#&zwj;batch, #&zwj;prior_boxes * 4]` or `[#&zwj;batch, #&zwj;prior_boxes * 5]` depending on shared locations between different batches or not.
* The tensor with confidences must be of shape `[#&zwj;batch, #&zwj;prior_boxes * #&zwj;classes]` and confidences values are in range [0, 1], that is passed through `softmax` layer.
* The tensor with prior boxes must be of shape `[#&zwj;batch, 2, #&zwj;prior_boxes * 4]`. Inference Engine expects that it contains variance values which TensorFlow Object Detection API does not add.
To enable these models, add `Reshape` operations for locations and confidences tensors and update the values for the prior boxes to include the variance constants (they are not there in TensorFlow Object Detection API).
Look at the `generate_sub_graph` method:
```python
def generate_sub_graph(self, graph: nx.MultiDiGraph, match: SubgraphMatch):
log.debug('PostprocessorReplacement.generate_sub_graph')
log.debug('matched_nodes = {}'.format(match.matched_nodes_names()))
# softmax to be applied to the confidence
softmax_conf_op = Softmax(graph, {'axis': 2, 'nchw_layout': True})
softmax_conf_node = softmax_conf_op.add_node(dict(name='DetectionOutput_SoftMax_conf_'))
# Inference Engine DetectionOutput layer consumes flattened tensors
# reshape operation to flatten locations tensor
reshape_loc_op = Reshape(graph, {'dim': np.array([0, -1])})
reshape_loc_node = reshape_loc_op.add_node(dict(name='DetectionOutput_Reshape_loc_'))
# Inference Engine DetectionOutput layer consumes flattened tensors
# reshape operation to flatten confidence tensor
reshape_conf_op = Reshape(graph, {'dim': np.array([0, -1])})
reshape_conf_node = reshape_conf_op.add_node(dict(name='DetectionOutput_Reshape_conf_'))
# create Node object from Op class
detection_output_op = DetectionOutput(graph, match.custom_replacement_desc.custom_attributes)
detection_output_op.attrs['old_infer'] = detection_output_op.attrs['infer']
detection_output_op.attrs['infer'] = __class__.do_infer
detection_output_node = detection_output_op.add_node(dict(name=detection_output_op.attrs['type'] + '_'))
# create internal edges of the sub-graph. In this case we add edges to connect input port 0 and 1 of the
# detection output with output of reshape of locations and reshape of confidence
create_edge(softmax_conf_node, reshape_conf_node, 0, 0)
create_edge(reshape_loc_node, detection_output_node, 0, 0)
create_edge(reshape_conf_node, detection_output_node, 0, 1)
return {'detection_output_node': detection_output_node, 'reshape_conf_node': softmax_conf_node,
'reshape_loc_node': reshape_loc_node}
```
The method has two inputs: the graph to operate on and the instance of `SubgraphMatch` object, which describes matched sub-graph. The latter class has several useful methods to get particular input/output node of the sub-graph by input/output index or by node name pattern. Examples of these methods usage are given below.
**Key lines**:
* Lines 6 and 7 create new instance of operation of type `Softmax` and graph Node object corresponding to that operation.
* Lines 11-12 and 16-17 create new instance of operation of type `Reshape` to reshape locations and confidences tensors correspondingly.
* Lines 20-23 create new instance of operation `DetectionOutput` and graph Node object corresponding to that operation.
* Lines 27-29 connect `softmax` node with `reshape` node and connect two reshaped locations and confidences tensors with `DetectionOutput` node.
* Lines 30-31 define dictionary with aliases for detection output node, reshape locations and confidences nodes. These aliases are used in the `input_edges_match` and `output_edges_match` methods.
The `input_edges_match` method is the following:
```python
def input_edges_match(self, graph: nx.DiGraph, match: SubgraphMatch, new_sub_graph: dict):
locs_consumer_node, locs_consumer_node_port = match.input_nodes(0)[0]
conf_consumer_node, conf_consumer_node_port = match.input_nodes(1)[0]
priors_consumer_node, priors_consumer_node_port = match.input_nodes(2)[0]
# create matching nodes for locations and confidence tensors using simple scheme "old_node_name: new_node_name"
# which in fact means "(old_node_name, 0): (new_node_name, 0)", while first '0' means old_port and the second
# zero defines 'new_port'.
return {locs_consumer_node.id: new_sub_graph['reshape_loc_node'].id,
conf_consumer_node.id: new_sub_graph['reshape_conf_node'].id,
priors_consumer_node.id: (new_sub_graph['detection_output_node'].id, 2),
}
```
The method has three parameters: input `graph`, `match` object describing matched sub-graph and `new_sub_graph` dictionary with alias names returned from the `generate_sub_graph` method.
**Key lines**:
* Lines 2-4 initialize Node objects and input ports for the nodes where the input tensors for the sub-graph are consumed. The method `match.input_nodes(ind)` returns list of tuples where the first element is a Node object and the second is the input port for this node which consumes the ind-th input tensor of the sub-graph. `input_points` list in the configuration file defines the order of input tensors to the sub-graph. For example, the `locs_consumer_node` object of type Node is a node that consumes tensor with locations in the port with number `locs_consumer_node_port`.
* Lines 8-11 define dictionary with the mapping of tensors as described above. Note that the attribute `id` of the Node object contains the name of the node in the graph.
The `output_edges_match` method is the following:
```python
def output_edges_match(self, graph: nx.DiGraph, match: SubgraphMatch, new_sub_graph: dict):
# the DetectionOutput in IE produces single tensor, but in TF it produces two tensors, so we need to create only
# one output edge match
return {match.output_node(0)[0].id: new_sub_graph['detection_output_node'].id}
```
The method has the same three parameters as `input_edges_match` method. The returned dictionary contains mapping just for one tensor initially produces by the first output node of the sub-graph (which is `detection_boxes` according to the configuration file) to a single output tensor of the created `DetectionOutput` node. In fact, it is possible to use any output node of the initial sub-graph in mapping, because the sub-graph output nodes are the output nodes of the whole graph (their output is not consumed by any other nodes).
Now, the Model Optimizer knows how to replace the sub-graph. The last step to enable the model is to cut-off some parts of the graph not needed during inference.
It is necessary to remove the `Preprocessor` block where image is resized. Inference Engine does not support dynamic input shapes, so the Model Optimizer must froze the input image size, and thus, resizing of the image is not necessary. This is achieved by replacer `<INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/tf/Preprocessor.py` which is executed automatically.
There are several `Switch` operations in the `Postprocessor` block without output edges. For example:
```sh
Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/cond/cond/switch_t
```
```sh
Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/cond/cond/switch_f
```
```sh
Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/cond_1/cond/switch_t
```
```sh
Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/cond_1/cond/switch_f
```
Model Optimizer marks these nodes as output nodes of the topology. Some parts of the `Posprocessor` blocks are not removed during sub-graph replacement because of that. In order to fix this issue, it is necessary to specify output nodes of the graph manually using the `--output` command line parameter.
###Example Model Optimizer Command-Line for TensorFlow\* SSD
The final command line to convert SSDs from the TensorFlow Object Detection API Zoo is:
```shell
./mo_tf.py --input_model=<path_to_frozen.pb> --tensorflow_use_custom_operations_config extensions/front/tf/legacy_ssd_support.json --output="detection_boxes,detection_scores,num_detections"
```
## Converting MobileNet V2 model created with TensorFlow Object Detection API <a name="convert_mobilenet_v2"></a>
The [MobileNet V2 model](http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_coco_2018_03_29.tar.gz) differs from the previous version, so converting the model requires a new sub-graph replacement configuration file and new command line parameters. The major differences are:
* The `Preprocessor` block has two outputs: the pre-processed image and the pre-processed image size.
* The `Postprocessor` block has one more input (in comparison with models created with TensorFlow Object Detection API
version 1.6 or lower): the pre-processed image size.
* Some node names have been changed in the `Postprocessor` block.
The updated sub-graph replacement configuration file `extensions/front/tf/ssd_v2_support.json` reflecting these changes
is the following:
```json
[
{
"custom_attributes": {
"code_type": "caffe.PriorBoxParameter.CENTER_SIZE",
"confidence_threshold": 0.01,
"keep_top_k": 200,
"nms_threshold": 0.6,
"pad_mode": "caffe.ResizeParameter.CONSTANT",
"resize_mode": "caffe.ResizeParameter.WARP"
},
"id": "TFObjectDetectionAPIDetectionOutput",
"include_inputs_to_sub_graph": true,
"include_outputs_to_sub_graph": true,
"instances": {
"end_points": [
"detection_boxes",
"detection_scores",
"num_detections"
],
"start_points": [
"Postprocessor/Shape",
"Postprocessor/scale_logits",
"Postprocessor/ExpandDims",
"Postprocessor/Reshape_1",
"Postprocessor/ToFloat"
]
},
"match_kind": "points"
},
{
"custom_attributes": {
},
"id": "PreprocessorReplacement",
"inputs": [
[
{
"node": "map/Shape$",
"port": 0
},
{
"node": "map/TensorArrayUnstack/Shape$",
"port": 0
},
{
"node": "map/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3$",
"port": 2
}
]
],
"instances": [
".*Preprocessor/"
],
"match_kind": "scope",
"outputs": [
{
"node": "sub$",
"port": 0
},
{
"node": "map/TensorArrayStack_1/TensorArrayGatherV3$",
"port": 0
}
]
}
]
```
### Example of Model Optimizer Command-Line for TensorFlow SSD MobileNet V2
The final command line to convert MobileNet SSD V2 from the TensorFlow Object Detection Zoo is the following:
```sh
./mo_tf.py --input_model=<path_to_frozen.pb> --tensorflow_use_custom_operations_config extensions/front/tf/ssd_v2_support.json --output="detection_boxes,detection_scores,num_detections"
```

20
docs/doxygen/ie_docs.xml
View File

@ -38,17 +38,15 @@
</tab>
<tab type="user" title="Model Optimizations Techniques" url="@ref openvino_docs_MO_DG_prepare_model_Model_Optimization_Techniques"/>
<tab type="user" title="Cutting off Parts of a Model" url="@ref openvino_docs_MO_DG_prepare_model_convert_model_Cutting_Model"/>
<tab type="usergroup" title="Sub-graph Replacement in Model Optimizer" url="@ref openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Subgraph_Replacement_Model_Optimizer">
<tab type="user" title="[DEPRECATED] Case-Study: Converting SSD models created with the TensorFlow* Object Detection API" url="@ref openvino_docs_MO_DG_prepare_model_customize_model_optimizer_TensorFlow_SSD_ObjectDetection_API"/>
<tab type="user" title="[DEPRECATED] Case-Study: Converting Faster R-CNN models created with the TensorFlow* Object Detection API" url="@ref openvino_docs_MO_DG_prepare_model_customize_model_optimizer_TensorFlow_Faster_RCNN_ObjectDetection_API"/>
</tab>
<tab type="user" title="Sub-graph Replacement in Model Optimizer" url="@ref openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Subgraph_Replacement_Model_Optimizer"/>
<tab type="user" title="Supported Framework Layers" url="@ref openvino_docs_MO_DG_prepare_model_Supported_Frameworks_Layers"/>
<tab type="user" title="[DEPRECATED] IR Notation Reference" url="@ref openvino_docs_MO_DG_prepare_model_convert_model_Legacy_IR_Layers_Catalog_Spec"/>
<tab type="user" title="IR suitable for INT8 inference" url="@ref openvino_docs_MO_DG_prepare_model_convert_model_IR_suitable_for_INT8_inference"/>
</tab>
<tab type="usergroup" title="Custom Layers in Model Optimizer" url="@ref openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Customize_Model_Optimizer">
<tab type="usergroup" title="Model Optimizer Extensibility" url="@ref openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Customize_Model_Optimizer">
<tab type="user" title="Extending Model Optimizer with New Primitives" url="@ref openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Extending_Model_Optimizer_with_New_Primitives"/>
<tab type="user" title="Extending MXNet Model Optimizer with New Primitives" url="@ref openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Extending_MXNet_Model_Optimizer_with_New_Primitives"/>
<tab type="user" title="Extending Model Optimizer with Caffe* Python Layers" url="@ref openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Extending_Model_Optimizer_With_Caffe_Python_Layers"/>
<tab type="user" title="Extending Model Optimizer for Custom MXNet* Operations" url="@ref openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Extending_MXNet_Model_Optimizer_with_New_Primitives"/>
<tab type="user" title="Legacy Mode for Caffe* Custom Layers" url="@ref openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Legacy_Mode_for_Caffe_Custom_Layers"/>
<tab type="user" title="[DEPRECATED] Offloading Sub-Graph Inference" url="https://docs.openvinotoolkit.org/2020.1/_docs_MO_DG_prepare_model_customize_model_optimizer_Offloading_Sub_Graph_Inference.html"/>
</tab>
@ -58,10 +56,10 @@
</tab>
<!-- Model Downloader -->
<tab id="model_downloader" type="user" title="Model Downloader" url="@ref omz_tools_downloader_README"/>
<!-- Custom Layers Guide -->
<tab type="usergroup" title="Custom Layers Guide" url="@ref openvino_docs_HOWTO_Custom_Layers_Guide"></tab>
</tab>
<!-- Custom Operations Guide -->
<tab type="usergroup" title="Custom Operations Guide" url="@ref openvino_docs_HOWTO_Custom_Layers_Guide"></tab>
</tab>
<!-- Intermediate Representation and Operations Sets -->
<tab id="intermediate_representaton_and_operations_sets" type="usergroup" title="Intermediate Representation and Operations Sets" url="@ref openvino_docs_MO_DG_IR_and_opsets">
<tab type="usergroup" title="Available Operations Sets" url="@ref openvino_docs_ops_opset">
@ -300,7 +298,7 @@
<!-- Compile Tool -->
<tab type="user" title="Compile Tool" url="@ref openvino_inference_engine_tools_compile_tool_README"/>
<!-- API References -->
<tab id="api_references" type="usergroup" title="API References">
<!-- IE C -->

6
docs/how_tos/how-to-links.md
View File

@ -12,8 +12,8 @@
* [Introducing int8 quantization for fast CPU inference using OpenVINO](https://www.intel.ai/introducing-int8-quantization-for-fast-cpu-inference-using-openvino/)
* [Accelerate Vision-based AI with Intel® Distribution of OpenVINO™ Toolkit](https://www.intel.ai/accelerate-vision-based-ai-with-intel-distribution-of-openvino-toolkit/)
## Custom Layers Guide
To learn about what is *custom layers* and how to work with them in the Deep Learning Deployment Toolkit, see the [Custom Layers Guide](../HOWTO/Custom_Layers_Guide.md).
## Custom Operations Guide
To learn about what is *custom operation* and how to work with them in the Deep Learning Deployment Toolkit, see the [Custom Operations Guide](../HOWTO/Custom_Layers_Guide.md).
## Introducing OpenVINO™ and Computer Vision | IoT Developer Show Season 2 | Intel Software
@ -65,4 +65,4 @@ To learn about what is *custom layers* and how to work with them in the Deep Lea
[performance-boost-dl]: ../img/performance-boost-DL-algorithm.jpg
[digital-security-surveillance]: ../img/digital-security-surveillance.jpg
[robotics-with-AI]: ../img/robotics-with-AI.jpg
[people-counter-syestem]: ../img/people-counter-syestem.jpg
[people-counter-syestem]: ../img/people-counter-syestem.jpg

3
docs/img/MO_connection_example_1.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1e432a8beb8290adec0d498f5c4aff63eed36c7fdf4bade3db0e0b0bdc5ff70f
size 26737

3
docs/img/MO_conversion_pipeline.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bb7beec6a00edbde13eca331d513f72b35b442f41f3a0c0e84f5185a1dcbf9ec
size 50220

3
docs/img/MO_graph_after_extractors.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bae3a4992d6d4f157674496b6230ae68f0cc33a58e4f305eeb935c09b5278409
size 28518

3
docs/img/MO_graph_after_loader.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c9bc3815efa5c7a2e2b3db4bfdc229b991c86346b086a74e5b02fa512cb3811f
size 26772

3
docs/img/MO_graph_before_partial_inference.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:22f39675a2dab15ff466094d623444ae2bf3d9447113bc4528e19ef275932260
size 51382

3
docs/img/MO_ports_example_1.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:86866c2c0a192eaaa78875ef324334d2d50eb488aa5a6ed980cb3b0d21f35953
size 26142

3
docs/img/MO_ports_example_2.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0540bb419eacf351d9e311c8b25c057c0c02f070b8615ae351e7cd21e3a9e020
size 25934

3
docs/img/MO_transformations_graph.png Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1fd333909d651648152c881122cdc6261e40737a96b2ef4eb291cce9da84e713
size 133129

15
docs/template_extension/CMakeLists.txt
View File

@ -9,13 +9,24 @@ set(TARGET_NAME "template_extension")
find_package(ngraph REQUIRED OPTIONAL_COMPONENTS onnx_importer)
find_package(InferenceEngine REQUIRED)
find_package(OpenCV QUIET COMPONENTS core)
file(GLOB_RECURSE SRC *.cpp)
set(SRC cpu_kernel.cpp extension.cpp op.cpp)
if (OpenCV_FOUND)
set(SRC ${SRC} fft_kernel.cpp fft_op.cpp)
endif()
add_library(${TARGET_NAME} MODULE ${SRC})
if (OpenCV_FOUND)
target_compile_definitions(${TARGET_NAME} PRIVATE OPENCV_IMPORT_ENABLED)
target_link_libraries(${TARGET_NAME} PRIVATE opencv_core)
endif()
target_compile_definitions(${TARGET_NAME} PRIVATE IMPLEMENT_INFERENCE_EXTENSION_API)
target_link_libraries(${TARGET_NAME} PRIVATE IE::inference_engine ${NGRAPH_LIBRARIES})
target_link_libraries(${TARGET_NAME} PRIVATE IE::inference_engine
${NGRAPH_LIBRARIES})
if (ngraph_onnx_importer_FOUND)
target_link_libraries(${TARGET_NAME} PRIVATE ${ONNX_IMPORTER_LIBRARIES})

37
docs/template_extension/extension.cpp
View File

@ -4,6 +4,10 @@
#include "extension.hpp"
#include "cpu_kernel.hpp"
#include "op.hpp"
#ifdef OPENCV_IMPORT_ENABLED
#include "fft_op.hpp"
#include "fft_kernel.hpp"
#endif
#include <ngraph/ngraph.hpp>
#ifdef NGRAPH_ONNX_IMPORT_ENABLED
#include <onnx_import/onnx_utils.hpp>
@ -12,7 +16,6 @@
#include <map>
#include <memory>
#include <string>
#include <unordered_map>
#include <vector>
using namespace TemplateExtension;
@ -27,6 +30,14 @@ Extension::Extension() {
int64_t add = node.get_attribute_value<int64_t>("add");
return {std::make_shared<Operation>(ng_inputs.at(0), add)};
});
#ifdef OPENCV_IMPORT_ENABLED
ngraph::onnx_import::register_operator(
FFTOp::type_info.name, 1, "custom_domain", [](const ngraph::onnx_import::Node& node) -> ngraph::OutputVector {
ngraph::OutputVector ng_inputs{node.get_ng_inputs()};
bool inverse = node.get_attribute_value<int64_t>("inverse");
return {std::make_shared<FFTOp>(ng_inputs.at(0), inverse)};
});
#endif
#endif
}
//! [extension:ctor]
@ -35,7 +46,10 @@ Extension::Extension() {
Extension::~Extension() {
#ifdef NGRAPH_ONNX_IMPORT_ENABLED
ngraph::onnx_import::unregister_operator(Operation::type_info.name, 1, "custom_domain");
#endif
#ifdef OPENCV_IMPORT_ENABLED
ngraph::onnx_import::unregister_operator(FFTOp::type_info.name, 1, "custom_domain");
#endif // OPENCV_IMPORT_ENABLED
#endif // NGRAPH_ONNX_IMPORT_ENABLED
}
//! [extension:dtor]
@ -56,6 +70,9 @@ std::map<std::string, ngraph::OpSet> Extension::getOpSets() {
std::map<std::string, ngraph::OpSet> opsets;
ngraph::OpSet opset;
opset.insert<Operation>();
#ifdef OPENCV_IMPORT_ENABLED
opset.insert<FFTOp>();
#endif
opsets["custom_opset"] = opset;
return opsets;
}
@ -66,14 +83,26 @@ std::vector<std::string> Extension::getImplTypes(const std::shared_ptr<ngraph::N
if (std::dynamic_pointer_cast<Operation>(node)) {
return {"CPU"};
}
#ifdef OPENCV_IMPORT_ENABLED
if (std::dynamic_pointer_cast<FFTOp>(node)) {
return {"CPU"};
}
#endif
return {};
}
//! [extension:getImplTypes]
//! [extension:getImplementation]
InferenceEngine::ILayerImpl::Ptr Extension::getImplementation(const std::shared_ptr<ngraph::Node> &node, const std::string &implType) {
if (std::dynamic_pointer_cast<Operation>(node) && implType == "CPU") {
return std::make_shared<OpImplementation>(node);
if (implType == "CPU") {
if (std::dynamic_pointer_cast<Operation>(node)) {
return std::make_shared<OpImplementation>(node);
}
#ifdef OPENCV_IMPORT_ENABLED
if (std::dynamic_pointer_cast<FFTOp>(node) && implType == "CPU") {
return std::make_shared<FFTImpl>(node);
}
#endif
}
return nullptr;
}

119
docs/template_extension/fft_kernel.cpp Normal file
View File

@ -0,0 +1,119 @@
// Copyright (C) 2020 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
//! [fft_kernel:implementation]
#include "fft_kernel.hpp"
#include "fft_op.hpp"
#include <details/ie_exception.hpp>
#include <ie_layouts.h>
#include <opencv2/opencv.hpp>
using namespace TemplateExtension;
FFTImpl::FFTImpl(const std::shared_ptr<ngraph::Node> &node) {
auto castedNode = std::dynamic_pointer_cast<FFTOp>(node);
if (!castedNode)
THROW_IE_EXCEPTION << "Cannot create implementation for unknown operation!";
if (castedNode->inputs().size() != 1 || castedNode->outputs().size() != 1)
THROW_IE_EXCEPTION << "Cannot create implementation for operation with incorrect number of inputs or outputs!";
if (castedNode->get_input_partial_shape(0).is_dynamic() || castedNode->get_output_partial_shape(0).is_dynamic())
THROW_IE_EXCEPTION << "Cannot create implementation for op with dynamic shapes!";
if (castedNode->get_input_element_type(0) != ngraph::element::f32 || castedNode->get_output_element_type(0) != ngraph::element::f32)
THROW_IE_EXCEPTION << "Operation supports only FP32 tensors.";
inpShape = castedNode->get_input_shape(0);
outShape = castedNode->get_output_shape(0);
inverse = castedNode->inverse;
}
InferenceEngine::StatusCode FFTImpl::getSupportedConfigurations(std::vector<InferenceEngine::LayerConfig> &conf,
InferenceEngine::ResponseDesc *resp) noexcept {
std::vector<InferenceEngine::DataConfig> inDataConfig;
std::vector<InferenceEngine::DataConfig> outDataConfig;
InferenceEngine::SizeVector order(inpShape.size());
std::iota(order.begin(), order.end(), 0);
// Allow any offset before data
size_t offset((std::numeric_limits<size_t>::max)());
// Input shape
InferenceEngine::DataConfig inpConf;
inpConf.desc = InferenceEngine::TensorDesc(InferenceEngine::Precision::FP32, inpShape, {inpShape, order, offset});
inDataConfig.push_back(inpConf);
// Output shape
InferenceEngine::DataConfig outConf;
outConf.desc = InferenceEngine::TensorDesc(InferenceEngine::Precision::FP32, outShape, {outShape, order, offset});
outDataConfig.push_back(outConf);
InferenceEngine::LayerConfig layerConfig;
layerConfig.inConfs = inDataConfig;
layerConfig.outConfs = outDataConfig;
conf.push_back(layerConfig);
return InferenceEngine::StatusCode::OK;
}
InferenceEngine::StatusCode FFTImpl::init(InferenceEngine::LayerConfig &config, InferenceEngine::ResponseDesc *resp) noexcept {
try {
if (config.inConfs.size() != 1 || config.outConfs.size() != 1) {
THROW_IE_EXCEPTION << "Operation cannot be initialized with incorrect number of inputs/outputs!";
}
if (config.outConfs[0].desc.getPrecision() != InferenceEngine::Precision::FP32 ||
config.inConfs[0].desc.getPrecision() != InferenceEngine::Precision::FP32) {
THROW_IE_EXCEPTION << "Operation supports only FP32 precisions!";
}
} catch (InferenceEngine::details::InferenceEngineException& ex) {
if (resp) {
strncpy(resp->msg, error.c_str(), sizeof(resp->msg) - 1);
resp->msg[sizeof(resp->msg)-1] = 0;
}
return InferenceEngine::GENERAL_ERROR;
}
return InferenceEngine::OK;
}
static cv::Mat infEngineBlobToMat(const InferenceEngine::Blob::Ptr& blob)
{
// NOTE: Inference Engine sizes are reversed.
std::vector<size_t> dims = blob->getTensorDesc().getDims();
std::vector<int> size(dims.begin(), dims.end());
auto precision = blob->getTensorDesc().getPrecision();
CV_Assert(precision == InferenceEngine::Precision::FP32);
return cv::Mat(size, CV_32F, (void*)blob->buffer());
}
InferenceEngine::StatusCode FFTImpl::execute(std::vector<InferenceEngine::Blob::Ptr> &inputs,
std::vector<InferenceEngine::Blob::Ptr> &outputs,
InferenceEngine::ResponseDesc *resp) noexcept {
cv::Mat inp = infEngineBlobToMat(inputs[0]);
cv::Mat out = infEngineBlobToMat(outputs[0]);
const int n = inp.size[0];
const int h = inp.size[2];
const int w = inp.size[3];
cv::Mat complex(h, w, CV_32FC2), interleavedOut(h, w, CV_32FC2);
for (int i = 0; i < n; ++i) {
std::vector<cv::Mat> components = {
cv::Mat(h, w, CV_32F, inp.ptr<float>(i, 0)),
cv::Mat(h, w, CV_32F, inp.ptr<float>(i, 1))
};
cv::merge(components, complex);
if (!inverse)
cv::dft(complex, interleavedOut);
else
cv::idft(complex, interleavedOut, cv::DFT_SCALE);
components = {
cv::Mat(h, w, CV_32F, out.ptr<float>(i, 0)),
cv::Mat(h, w, CV_32F, out.ptr<float>(i, 1))
};
cv::split(interleavedOut, components);
}
return InferenceEngine::OK;
}
//! [fft_kernel:implementation]

32
docs/template_extension/fft_kernel.hpp Normal file
View File

@ -0,0 +1,32 @@
// Copyright (C) 2020 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
// source: https://github.com/openvinotoolkit/openvino/tree/master/docs/template_extension
//! [fft_kernel:header]
#pragma once
#include <ie_iextension.h>
#include <ngraph/ngraph.hpp>
namespace TemplateExtension {
class FFTImpl : public InferenceEngine::ILayerExecImpl {
public:
explicit FFTImpl(const std::shared_ptr<ngraph::Node>& node);
InferenceEngine::StatusCode getSupportedConfigurations(std::vector<InferenceEngine::LayerConfig> &conf,
InferenceEngine::ResponseDesc *resp) noexcept override;
InferenceEngine::StatusCode init(InferenceEngine::LayerConfig &config,
InferenceEngine::ResponseDesc *resp) noexcept override;
InferenceEngine::StatusCode execute(std::vector<InferenceEngine::Blob::Ptr> &inputs,
std::vector<InferenceEngine::Blob::Ptr> &outputs,
InferenceEngine::ResponseDesc *resp) noexcept override;
private:
ngraph::Shape inpShape;
ngraph::Shape outShape;
bool inverse;
std::string error;
};
}
//! [fft_kernel:header]

34
docs/template_extension/fft_op.cpp Normal file
View File

@ -0,0 +1,34 @@
// Copyright (C) 2020 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
//! [fft_op:implementation]
#include "fft_op.hpp"
using namespace TemplateExtension;
constexpr ngraph::NodeTypeInfo FFTOp::type_info;
FFTOp::FFTOp(const ngraph::Output<ngraph::Node>& inp, bool _inverse) : Op({inp}) {
constructor_validate_and_infer_types();
inverse = _inverse;
}
void FFTOp::validate_and_infer_types() {
auto outShape = get_input_partial_shape(0);
set_output_type(0, get_input_element_type(0), outShape);
}
std::shared_ptr<ngraph::Node> FFTOp::clone_with_new_inputs(const ngraph::OutputVector &new_args) const {
if (new_args.size() != 1) {
throw ngraph::ngraph_error("Incorrect number of new arguments");
}
return std::make_shared<FFTOp>(new_args.at(0), inverse);
}
bool FFTOp::visit_attributes(ngraph::AttributeVisitor &visitor) {
visitor.on_attribute("inverse", inverse);
return true;
}
//! [fft_op:implementation]

28
docs/template_extension/fft_op.hpp Normal file
View File

@ -0,0 +1,28 @@
// Copyright (C) 2020 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
//! [fft_op:header]
#pragma once
#include <ngraph/ngraph.hpp>
namespace TemplateExtension {
class FFTOp : public ngraph::op::Op {
public:
static constexpr ngraph::NodeTypeInfo type_info{"FFT", 0};
const ngraph::NodeTypeInfo& get_type_info() const override { return type_info; }
FFTOp() = default;
FFTOp(const ngraph::Output<ngraph::Node>& inp, bool inverse);
void validate_and_infer_types() override;
std::shared_ptr<ngraph::Node> clone_with_new_inputs(const ngraph::OutputVector& new_args) const override;
bool visit_attributes(ngraph::AttributeVisitor& visitor) override;
bool inverse;
};
}
//! [fft_op:header]

2
inference-engine/tests/unit/inference_engine/ie_extension_test.cpp
View File

@ -47,7 +47,7 @@ TEST(ExtensionTests, testGetImplTypesThrowsIfNgraphNodeIsNullPtr) {
TEST(ExtensionTests, testGetImplementation) {
IExtensionPtr extension = make_so_pointer<IExtension>(getExtensionPath());
auto opset = extension->getOpSets().begin()->second;
std::shared_ptr<ngraph::Node> op(opset.create(opset.get_types_info().begin()->name));
std::shared_ptr<ngraph::Node> op(opset.create("Template"));
ASSERT_NE(nullptr, extension->getImplementation(op, extension->getImplTypes(op)[0]));
}

147
model-optimizer/README.md
View File

@ -1,48 +1,21 @@
## Project structure
Project structure:
<pre>
|-- root
|-- extensions
|-- front/caffe
|-- CustomLayersMapping.xml.example - example of file for registering custom Caffe layers in 2017R3 public
manner
|-- mo
|-- back - Back-End logic: contains IR emitting logic
|-- front - Front-End logic: contains matching between Framework-specific layers and IR specific, calculation
of output shapes for each registered layer
|-- graph - Graph utilities to work with internal IR representation
|-- middle - Graph transformations - optimizations of the model
|-- pipeline - Sequence of steps required to create IR for each framework
|-- utils - Utility functions
|-- tf_call_ie_layer - Sources for TensorFlow fallback in Inference Engine during model inference
|-- mo.py - Centralized entry point that can be used for any supported framework
|-- mo_caffe.py - Entry point particularly for Caffe
|-- mo_mxnet.py - Entry point particularly for MXNet
|-- mo_tf.py - Entry point particularly for TensorFlow
|-- ModelOptimizer - Entry point particularly for Caffe that contains same CLI as 2017R3 publicly released
Model Optimizer
</pre>
## Prerequisites
Model Optimizer requires:
1. Python 3 or newer
2. [Optional] Please read about use cases that require Caffe available on the machine (:doc:`caffe_dependency`).
Please follow the steps described (:doc:`caffe_build`).
2. [Optional] Please read about use cases that require Caffe\* to be available on the machine in the documentation.
## Installation instructions
1. Go to the Model Optimizer folder:
<pre>
cd PATH_TO_INSTALL_DIR/deployment_tools/model_optimizer/model_optimizer_tensorflow
cd PATH_TO_INSTALL_DIR/deployment_tools/model_optimizer
</pre>
2. Create virtual environment and activate it. This option is strongly recommended as it creates a Python sandbox and
dependencies for Model Optimizer do not influence global Python configuration, installed libraries etc. At the same
time, special flag ensures that system-wide Python libraries are also available in this sandbox. Skip this
dependencies for the Model Optimizer do not influence global Python configuration, installed libraries etc. At the
same time, special flag ensures that system-wide Python libraries are also available in this sandbox. Skip this
step only if you do want to install all Model Optimizer dependencies globally:
* Create environment:
@ -54,110 +27,28 @@ Model Optimizer requires:
. .env3/bin/activate
</pre>
3. Install dependencies. If you want to convert models only from particular framework, you should use one of
available <code>requirements_*.txt</code> files corresponding to the framework of choice. For example, for Caffe use
<code>requirements_caffe.txt</code> and so on. When you decide to switch later to other frameworks, please install dependencies
for them using the same mechanism:
available <code>requirements_\*.txt</code> files corresponding to the framework of choice. For example, for Caffe
use <code>requirements_caffe.txt</code> and so on. When you decide to switch later to other frameworks, please
install dependencies for them using the same mechanism:
<pre>
pip3 install -r requirements.txt
</pre>
pip3 install -r requirements.txt
</pre>
Or you can use the installation scripts from the "install_prerequisites" directory.
4. [OPTIONAL] If you use Windows OS, most probably you get python version of `protobuf` library. It is known to be rather slow,
and you can use a boosted version of library by building the .egg file (Python package format) yourself,
using instructions below (section 'How to boost Caffe model loading') for the target OS and Python, or install it
with the pre-built .egg (it is built for Python 3.4, 3.5, 3.6, 3.7):
<pre>
<pre>
python3 -m easy_install protobuf-3.6.1-py3.6-win-amd64.egg
</pre>
</pre>
It overrides the protobuf python package installed by the previous command.
Set environment variable to enable boost in protobuf performance:
<pre>
<pre>
set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp
</pre>
## Command-Line Interface (CLI)
The following short examples are framework-dependent. Please read the complete help
with --help option for details across all frameworks:
<pre>
python3 mo.py --help
</pre>
There are several scripts that convert a model:
1. <code>mo.py</code> -- universal entry point that can convert a model from any supported framework
2. <code>mo_caffe.py</code> -- dedicated script for Caffe models conversion
3. <code>mo_mxnet.py</code> -- dedicated script for MXNet models conversion
4. <code>mo_tf.py</code> -- dedicated script for TensorFlow models conversion
5. <code>mo_onnx.py</code> -- dedicated script for ONNX models conversion
6. <code>mo_kaldi.py</code> -- dedicated script for Kaldi models conversion
<code>mo.py</code> can deduce original framework where input model was trained by an extension of
the model file. Or <code>--framework</code> option can be used for this purpose if model files
don't have standard extensions (<code>.pb</code> - for TensorFlow models, <code>.params</code> - for MXNet models,
<code>.caffemodel</code> - for Caffe models). So, the following commands are equivalent::
<pre>
python3 mo.py --input_model /user/models/model.pb
python3 mo.py --framework tf --input_model /user/models/model.pb
</pre>
The following examples illustrate the shortest command lines to convert a model per
framework.
### Convert TensorFlow model
To convert a frozen TensorFlow model contained in binary file <code>model-file.pb</code>, run
dedicated entry point <code>mo_tf.py</code>:
python3 mo_tf.py --input_model model-file.pb
### Convert Caffe model
To convert a Caffe model contained in <code>model-file.prototxt</code> and <code>model-file.caffemodel</code> run
dedicated entry point <code>mo_caffe.py</code>:
<pre>
python3 mo_caffe.py --input_model model-file.caffemodel
</pre>
### Convert MXNet model
To Convert an MXNet model in <code>model-file-symbol.json</code> and <code>model-file-0000.params</code> run
dedicated entry point <code>mo_mxnet.py</code>:
<pre>
python3 mo_mxnet.py --input_model model-file
</pre>
> **NOTE**: for TensorFlow* all Placeholder ops are represented as Input layers in the final IR.
### Convert ONNX* model
The Model Optimizer assumes that you have an ONNX model that was directly downloaded from a public repository or converted from any framework that supports exporting to the ONNX format.
Use the mo_onnx.py script to simply convert a model with the path to the input model .onnx file:
<pre>
python3 mo_onnx.py --input_model model-file.onnx
</pre>
Input channels re-ordering, scaling, subtraction of mean values and other preprocessing features
are not applied by default. To pass necessary values to Model Optimizer, please run <code>mo.py</code>
(or <code>mo_tf.py</code>, <code>mo_caffe.py</code>, <code>mo_mxnet.py</code>) with <code>--help</code> and
examine all available options.
## Working with Inference Engine
To the moment, Inference Engine is the only consumer of IR models that Model Optimizer produces.
The whole workflow and more documentation on the structure of IR are documented in the Developer Guide
of Inference Engine. Note that sections about running Model Optimizer refer to the old version
of the tool and can not be applied to the current version of Model Optimizer.
</pre>
## Setup development environment
@ -185,14 +76,6 @@ of the tool and can not be applied to the current version of Model Optimizer.
1. Run the following command:
<pre>
pylint mo/ mo.py
pylint mo/ extensions/ mo.py
</pre>
### How to check requirements dependencies
1. Run the following command:
<pre>
cat requirements_file | docker run -i --rm pyupio/safety safety check --stdin
</pre>
> **NOTE**: here <code>requirements_file</code> is one of the following: <code>requirements.txt</code>, <code>requirements_caffe.txt</code>, <code>requirements_tf.txt</code>, <code>requirements_tf2.txt</code>, <code>requirements_mxnet.txt</code>, <code>requirements_dev.txt</code>.