openvino/docs/Extensibility_UG/frontend_extensions.md

Frontend Extensions

The goal of this chapter is to explain how to use Frontend extension classes to facilitate mapping of custom operations from framework model representation to OpenVINO representation. Refer to Introduction to OpenVINO Extension to understand entire flow.

This API is applicable for new frontends only, which exist for ONNX and PaddlePaddle. If a different model format is used, follow legacy Model Optimizer Extensions guide.

Note

: This documentation is written based on the Template extension, which demonstrates extension development details based on minimalistic Identity operation that is a placeholder for your real custom operation. You can review the complete code, which is fully compliable, to see how it works.

Single Operation Mapping with OpExtension

This section covers the case when a single operation in framework representation is mapped to a single operation in OpenVINO representation. This is called one-to-one mapping. There is OpExtension class that works well if all the following conditions are satisfied:

1. Number of inputs to operation in the Framework representation is the same as in the OpenVINO representation.

2. Number of outputs is also the same in both representations.

3. Inputs can be indexed and are mapped in order correspondingly, e.g. input with index 0 in framework representation maps to input with index 0 in OpenVINO representation and so on.

4. The same for outputs.

5. Each attribute in OpenVINO operation can be initialized from one of the attributes of original operation or by some predefined constant value. Value of copied attributes cannot contain expressions, value is accepted as-is, so type of a value should be compatible.

Note

: OpExtension class is currently available for ONNX frontend only. PaddlePaddle frontend has named inputs and outputs for operation (not indexed) therefore OpExtension mapping is not applicable for this case.

The next example maps ONNX operation with type “Identity” to OpenVINO template extension Identity class.

@snippet ov_extensions.cpp frontend_extension_Identity_header @snippet ov_extensions.cpp frontend_extension_Identity

The mapping doesn’t involve any attributes, as operation Identity doesn’t have them.

Extension objects, like just constructed extension can be used to add to the OpenVINO runtime just before the loading a model that contains custom operations:

@snippet ov_extensions.cpp frontend_extension_read_model

Or extensions can be constructed in a separately compiled shared library. Separately compiled library can be used in Model Optimizer or benchmark_app. Read about how to build and load such library in chapter “Create library with extensions” in Introduction to OpenVINO Extension.

If operation have multiple inputs and/or outputs they will be mapped in order. The type of elements in input/output tensors should match expected types in the surrounding operations. For example, if custom operation produces f32 data type then operation that consumes this output should also support f32. Otherwise, model conversion fails with an error, there are no automatic type conversion happens.

Converting to Standard OpenVINO Operation

OpExtension class can be used when mapping to one of the operations from standard OpenVINO operation set is what you need and there is no class like TemplateExtension::Identity implemented.

Here is an example for a custom framework operation “MyRelu”. Suppose it is mathematically equivalent to standard Relu that exists in OpenVINO operation set, but for some reason has type name “MyRelu”. In this case you can directly say that “MyRelu” -> Relu mapping should be used:

@sphinxtabset

@sphinxtab{C++} @snippet ov_extensions.cpp frontend_extension_MyRelu @endsphinxtab @sphinxtab{Python} @snippet ov_extensions.py py_frontend_extension_MyRelu @endsphinxtab

@endsphinxtabset

In the resulting converted OpenVINO model, “MyRelu” operation will be replaced by the standard operation Relu from the latest available OpenVINO operation set. Notice that when standard operation is used, it can be specified using just a type string (“Relu”) instead of using a ov::opset8::Relu class name as a template parameter for OpExtension. This method is available for operations from the standard operation set only. For a user custom OpenVINO operation the corresponding class should be always specified as a template parameter as it was demonstrated with TemplateExtension::Identity.

Attributes Mapping

As described above, OpExtension is useful when attributes can be mapped one by one or initialized by a constant. If the set of attributes in framework representation and OpenVINO representation completely match by their names and types, nothing should be specified in OpExtension constructor parameters. The attributes are discovered and mapped automatically based on visit_attributes method that should be defined for any OpenVINO operation.

Imagine you have CustomOperation class implementation that has two attributes with names attr1 and attr2:

@snippet ov_extensions.cpp frontend_extension_CustomOperation

And original model in framework representation also has operation with name “CustomOperatoin” with the same attr1 and attr2 attributes. Then with the following code:

@snippet ov_extensions.cpp frontend_extension_CustomOperation_as_is

both attr1 and attr2 are copied from framework representation to OpenVINO representation automatically. If for some reason names of attributes are different but values still can be copied “as-is” you can pass attribute names mapping in OpExtension constructor:

@snippet ov_extensions.cpp frontend_extension_CustomOperation_rename

Where fw_attr1 and fw_attr2 are names for corresponding attributes in framework operation representation.

If copying of an attribute is not what you need, OpExtension also can set attribute to predefined constant value. For the same CustomOperation, imagine you want to set attr2 to value 5 instead of copying from fw_attr2, to achieve that do the following:

@snippet ov_extensions.cpp frontend_extension_CustomOperation_rename_set

So the conclusion is that each attribute of target OpenVINO operation should be initialized either by

1. Setting automatically due to name matching

2. Mapped by attribute name

3. Set to a constant value

This is achieved by specifying maps as arguments for OpExtension constructor.

Mapping to Multiple Operations with ConversionExtension

Previous sections cover the case when a single operation is mapped to a single operation with optional adjustment in names and attribute values. That is likely enough for your own custom operation with existing C++ kernel implementation. In this case your framework representation and OpenVINO representation for the operation are under your control and inputs/outpus/attributes can be aligned to make OpExtension usable.

In case if one-to-one mapping is not possible, decomposition to multiple operations should be considered. It is achieved by using more verbose and less automated ConversionExtension class. It enables writing arbitrary code to replace a single framework operation by multiple connected OpenVINO operations constructing dependency graph of any complexity.

ConversionExtension maps a single operation to a function which builds a graph using OpenVINO operation classes. Follow chapter [Build a Model in OpenVINO Runtime](@ref ov_ug_build_model) to learn how to use OpenVINO operation classes to build a fragment of model for replacement.

The next example illustrates using ConversionExtension for conversion of “ThresholdedRelu” from ONNX according to the formula: ThresholdedRelu(x, alpha) -> Multiply(x, Convert(Greater(x, alpha), type=float)).

Note

: ThresholdedRelu is one of the standard ONNX operators which is supported by ONNX frontend natively out-of-the-box. Here we are re-implementing it to illustrate how you can add a similar support for your custom operation instead of ThresholdedRelu.

@sphinxtabset

@sphinxtab{C++} @snippet ov_extensions.cpp frontend_extension_ThresholdedReLU_header @endsphinxtab @sphinxtab{Python} @snippet ov_extensions.py py_frontend_extension_ThresholdedReLU_header @endsphinxtab

@endsphinxtabset

@sphinxtabset

@sphinxtab{C++} @snippet ov_extensions.cpp frontend_extension_ThresholdedReLU @endsphinxtab @sphinxtab{Python} @snippet ov_extensions.py py_frontend_extension_ThresholdedReLU @endsphinxtab

@endsphinxtabset

To access original framework operation attribute value and connect to inputs, node object of type NodeContext is used. It has two main methods:

• NodeContext::get_input to get input with a given index,

• NodeContext::get_attribute to get attribute value with a given name.

The conversion function should return a vector of node outputs that are mapped to corresponding outputs of the original framework operation in the same order.