From 6f7e9cd7864c2ffb1715333f41e6cabf6b762799 Mon Sep 17 00:00:00 2001 From: Mateusz Tabaka Date: Tue, 30 May 2023 10:53:59 +0200 Subject: [PATCH] Update docs for frontend extensions (#16752) * Update docs for frontend extensions * Apply suggestions from code review Co-authored-by: Roman Kazantsev * fix order in openvino_framework_map * no discard return value * add note of openvino_contrib repo * update example for PT * note * add paragraph of named inputs and outputs Signed-off-by: Mateusz Tabaka * title underline too short * review comments * remove m_ prefix from CustomOp attr names --------- Signed-off-by: Mateusz Tabaka Co-authored-by: Roman Kazantsev Co-authored-by: Michal Lukaszewski --- docs/Extensibility_UG/Intro.md | 2 + docs/Extensibility_UG/frontend_extensions.md | 173 ++++++++++++++---- docs/snippets/CMakeLists.txt | 3 +- docs/snippets/ov_extensions.cpp | 77 +++++++- docs/snippets/ov_extensions.py | 29 +++ src/frontends/pytorch/src/node_context.cpp | 2 + .../py_frontend_tests/test_torch_frontend.py | 25 ++- 7 files changed, 258 insertions(+), 53 deletions(-) diff --git a/docs/Extensibility_UG/Intro.md b/docs/Extensibility_UG/Intro.md index 582cc259965..4e8ab54b83f 100644 --- a/docs/Extensibility_UG/Intro.md +++ b/docs/Extensibility_UG/Intro.md @@ -126,6 +126,8 @@ The ``Identity`` is a custom operation class defined in :doc:`Custom Operation G When Python API is used, there is no way to implement a custom OpenVINO operation. Even if custom OpenVINO operation is implemented in C++ and loaded into the runtime by a shared library, there is still no way to add a frontend mapping extension that refers to this custom operation. In this case, use C++ shared library approach to implement both operations semantics and framework mapping. Python can still be used to map and decompose operations when only operations from the standard OpenVINO operation set are used. + +.. _create_a_library_with_extensions: Create a Library with Extensions ++++++++++++++++++++++++++++++++ diff --git a/docs/Extensibility_UG/frontend_extensions.md b/docs/Extensibility_UG/frontend_extensions.md index 95d33cb5d3a..63021cd1720 100644 --- a/docs/Extensibility_UG/frontend_extensions.md +++ b/docs/Extensibility_UG/frontend_extensions.md @@ -19,6 +19,11 @@ guide. operation that is a placeholder for your real custom operation. You can review the complete code, which is fully compilable, to see how it works. + +.. note:: + You can find more examples of extensions in `openvino_contrib repository `_. + + Single Operation Mapping with OpExtension ######################################### @@ -108,10 +113,18 @@ as it was demonstrated with ``TemplateExtension::Identity``. Attribute Mapping ++++++++++++++++++ -As described above, ``OpExtension`` is useful when attributes can be mapped one by one or initialized by a constant. +As described above, ``OpExtension`` is useful when attributes can be mapped one by one or initialized by a constant. +Attributes in OpenVINO operators are identified by their names, so for frameworks that also have named attributes (like TensorFlow, PaddlePaddle, ONNX), +you can specify name to name mapping. For frameworks where OpenVINO operator's attributes can be mapped to one of the framework +operator inputs (like PyTorch), there's a name to input index mapping. + + +Named attributes mapping +^^^^^^^^^^^^^^^^^^^^^^^^ + If the set of attributes in framework representation and OpenVINO representation completely match by their names and types, -nothing should be specified in OpExtension constructor parameters. The attributes are discovered and mapped -automatically based on ``visit_attributes`` method that should be defined for any OpenVINO operation. +no attribute mapping has to be specified in OpExtension constructor parameters. The attributes are discovered and mapped automatically +based on ``visit_attributes`` method that should be defined for any OpenVINO operation. Imagine you have CustomOperation class implementation that has two attributes with names: ``attr1`` and ``attr2``. @@ -119,14 +132,15 @@ Imagine you have CustomOperation class implementation that has two attributes wi :language: cpp :fragment: [frontend_extension_CustomOperation] -And the original model in the framework representation also has operation named “CustomOperation” with the same +And original model in framework representation also has operation with name ``CustomOperation`` with the same ``attr1`` and ``attr2`` attributes. Then with the following code: .. doxygensnippet:: docs/snippets/ov_extensions.cpp :language: cpp :fragment: [frontend_extension_CustomOperation_as_is] -Both ``attr1`` and ``attr2`` are copied from framework representation to OpenVINO representation automatically. +Both ``attr1`` and ``attr2`` are copied from framework representation to OpenVINO representation automatically. + If for some reason names of attributes are different but values still can be copied “as-is” you can pass attribute names mapping in ``OpExtension`` constructor: @@ -144,48 +158,123 @@ to achieve that do the following: :language: cpp :fragment: [frontend_extension_CustomOperation_rename_set] + So the conclusion is that each attribute of target OpenVINO operation should be initialized either by 1. Setting automatically due to name matching 2. Mapped by attribute name 3. Set to a constant value -This is achieved by specifying maps as arguments for `OpExtension` constructor. +This is achieved by specifying maps as arguments for ``OpExtension`` constructor. + + +Attribute mapping with named inputs and outputs +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Mappings in previous examples assume that inputs and outputs of an operator in framework model representation come +with a particular order so you can directly map framework operation input ``0`` to OpenVINO operation input ``0`` and so on. +That's not always the case, for frameworks like PaddlePaddle, operation inputs and outputs are identified by their names +and may be defined in any order. So to map it to OpenVINO operation inputs and outputs, you have to specify that order yourself. +This can be done by creating two vector of strings, one for input and one for output, where framework operation +input name at position ``i`` maps to OpenVINO operation input at position ``i`` (and similarly for outputs). + + +Let's see the following example. Like previously, we'd like to map ``CustomOperation`` in the original model, +to OpenVINO ``CustomOperation`` as is (so their name and attributes names match). This time, that framework operation +inputs and outputs are not stricly ordered and can be identified by their names ``A``, ``B``, ``C`` for inputs +and ``X``, ``Y`` for outputs. Those inputs and outputs can be mapped to OpenVINO operation, such that inputs +``A``, ``B``, ``C`` map to OpenVINO ``CustomOperation`` first, second and third input and ``X`` and ``Y`` +outputs map to OpenVINO ``CustomOperation`` first and second output respectively. + +Given that, such custom operation can be registered by the following: + +.. doxygensnippet:: docs/snippets/ov_extensions.cpp + :language: cpp + :fragment: [frontend_extension_CustomOperation_as_is_paddle] + + +Second example shows how to map the operation with named inputs and outputs, but when names of attributes are different: + +.. doxygensnippet:: docs/snippets/ov_extensions.cpp + :language: cpp + :fragment: [frontend_extension_CustomOperation_rename_paddle] + + +and the last one shows how to map the operation with named inputs and outputs, but when (in order to correctly map framework +operation to OpenVINO operation) one of the attributes has to be set to predefined value: + +.. doxygensnippet:: docs/snippets/ov_extensions.cpp + :language: cpp + :fragment: [frontend_extension_CustomOperation_rename_set_paddle] + + +Mapping attributes from operation inputs +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For models (like PyTorch models), where operations have attributes on the input list, you can specify name to input index mapping. +For example, imagine you have created a custom OpenVINO operation that implements a variant of ELU activation function +with two attributes ``alpha`` and ``beta``: + +.. math:: + + CustomElu=\left\lbrace + \begin{array}{ll} + beta * x & \textrm{if x > 0} \newline + alpha * (exp(x) - 1) & \textrm{otherwise} + \end{array} + \right. + +Below is a snippet of ``CustomElu`` class showing how to define its attributes: + +.. doxygensnippet:: docs/snippets/ov_extensions.cpp + :language: cpp + :fragment: [frontend_extension_framework_map_CustomElu] + +Let's see an example of how you can map ``CustomElu`` to PyTorch `aten::elu `_ +(note that if ``beta`` is equal to ``1``, ``CustomElu`` works the same as ``aten::elu``). +``aten::elu`` has ``alpha`` attribute second on the input list, but it doesn't have ``beta``. +So in order to map it to ``CustomElu`` you can use the following: + +.. doxygensnippet:: docs/snippets/ov_extensions.cpp + :language: cpp + :fragment: [frontend_extension_framework_map_CustomElu_mapping] + +This will map ``alpha`` to the second input and map ``beta`` attribute to constant value ``1.0f``. + +Such created extension can be used, e.g. in dynamic library, please refer to :ref:`Create a library with extensions `. Mapping custom operations to frontends with OPENVINO_FRAMEWORK_MAP macro ---------------------------------------------------------------------------- +######################################################################## -.. note:: - - Below solution works only for ONNX and Tensorflow frontends. +``OPENVINO_FRAMEWORK_MAP`` is a macro that should be used inside OpenVINO operation's class definition and that lets you specify +the mapping between this operation to a frontend operation. -``OPENVINO_FRAMEWORK_MAP`` is a macro that should be used inside OpenVINO operation's class definition -and that lets you specify the mapping between this operation to a frontend operation. +Let's consider the following example. Imagine you have an ONNX model with ``CustomOp`` operation (and this operation has ``mode`` attribute), +a TensorFlow model with ``CustomOpV3`` operation (this operation has ``axis`` attribute) and a PaddlePaddle model with ``CustomOp`` (with ``mode`` attribute) +that has input named "X" and output named "Out" and all of them can be implemented with a single OpenVINO operation ``CustomOp`` like follows: -Let's consider the following example. Imagine you have an ONNX model with ``CustomOp`` operation -(and this operation has ``mode`` attribute) and a Tensorflow model with ``CustomOpV3`` operation -(this operation has ``axis`` attribute) and both of them can be implemented with a single OpenVINO -operation ``CustomOp`` like follows: - -.. doxygensnippet:: ov_extensions.cpp +.. doxygensnippet:: docs/snippets/ov_extensions.cpp :language: cpp :fragment: [frontend_extension_framework_map_macro_headers] -.. doxygensnippet:: ov_extensions.cpp +.. doxygensnippet:: docs/snippets/ov_extensions.cpp :language: cpp :fragment: [frontend_extension_framework_map_macro_CustomOp] +Let's take a closer look at the parameters this macro takes (note that there are two flavors - the second one is to map +for PaddlePaddle operations where input and output names have to be specified). -Let's take a closer look at the parameters this macro takes: - -.. code-block::cpp +.. code-block:: cpp OPENVINO_FRAMEWORK_MAP(framework, name, attributes_map, attributes_values) + OPENVINO_FRAMEWORK_MAP(framework, input_names, output_names, name, attributes_map, attributes_values) - ``framework`` - framework name. - ``name`` - the framework operation name. It's optional if the OpenVINO custom operation name - (that is the name that is passed as the first parameter to `OPENVINO_OP` macro) is the - same as the framework operation name and both ``attributes_map`` and ``attributes_values`` are not provided. + (that is the name that is passed as the first parameter to ``OPENVINO_OP`` macro) is the same + as the framework operation name and both ``attributes_map`` and ``attributes_values`` are not provided. +- ``input_names`` - vector of strings that specify the names of inputs (needed to map PaddlePaddle to OpenVINO operations), +- ``output_names`` - vector of strings that specify the names of outputs (needed to map PaddlePaddle to OpenVINO operations), - ``attributes_map`` - used to provide a mapping between OpenVINO operation attribute and framework operation attribute. Contains key-value pairs, where key is an OpenVINO operation attribute name and value is its corresponding framework operation attribute name. @@ -196,14 +285,21 @@ Let's take a closer look at the parameters this macro takes: operation attribute name and the value is this attribute value. This parameter cannot be provided if ``attributes_map`` contains all of OpenVINO operation attributes or if ``attributes_map`` is not provided. -In the example above, ``OPENVINO_FRAMEWORK_MAP`` is used twice. +In the example above, ``OPENVINO_FRAMEWORK_MAP`` is used three times. First, OpenVINO ``CustomOp`` is mapped to ONNX ``CustomOp`` operation, ``m_mode`` attribute is mapped to ``mode`` -attribute, while ``m_axis`` attribute gets the default value ``-1``. Secondly, OpenVINO `CustomOp` is mapped -to Tensorflow ``CustomOpV3`` operation, ``m_axis`` attribute is mapped to ``axis`` attribute, while ``m_mode`` -attribute gets the default value ``linear``. +attribute, while ``m_axis`` attribute gets the default value ``-1``. Secondly, OpenVINO ``CustomOp`` is mapped +to TensorFlow ``CustomOpV3`` operation, ``m_axis`` attribute is mapped to ``axis`` attribute, while ``m_mode`` +attribute gets the default value ``"linear"``. Thirdly, OpenVINO ``CustomOp`` is mapped to PaddlePaddle ``CustomOp`` operation, +``m_mode`` attribute is mapped to ``mode`` attribute, while ``m_axis`` attribute gets the default value ``-1``. +This mapping also specifies the input name "X" and output name "Out". The last step is to register this custom operation by following: -@snippet ov_extensions.cpp frontend_extension_framework_map_macro_add_extension + + +.. doxygensnippet:: docs/snippets/ov_extensions.cpp + :language: cpp + :fragment: [frontend_extension_framework_map_macro_add_extension] + Mapping to Multiple Operations with ConversionExtension ####################################################### @@ -222,7 +318,7 @@ operations constructing dependency graph of any complexity. operation classes. Follow chapter :ref:`Build a Model in OpenVINO Runtime ` to learn how to use OpenVINO operation classes to build a fragment of model for replacement. -The next example illustrates using ``ConversionExtension`` for conversion of “ThresholdedRelu” +Below example illustrates using ``ConversionExtension`` for conversion of “ThresholdedRelu” from ONNX according to the formula: ``ThresholdedRelu(x, alpha) -> Multiply(x, Convert(Greater(x, alpha), type=float))``. .. note:: @@ -264,12 +360,23 @@ from ONNX according to the formula: ``ThresholdedRelu(x, alpha) -> Multiply(x, C :fragment: [py_frontend_extension_ThresholdedReLU] -To access original framework operation attribute value and connect to inputs, -``node`` object of type ``NodeContext`` is used. It has two main methods: +The next example shows how to use ``ConversionExtension`` to convert PyTorch +`aten::hardtanh `_ +to demonstrate how to use ``get_values_from_const_input`` function to fetch an attribute value from input: + + +.. doxygensnippet:: docs/snippets/ov_extensions.py + :language: python + :fragment: [py_frontend_extension_aten_hardtanh] + + +To access original framework operation attribute value and connect to inputs, ``node`` object of type ``NodeContext`` is used. It has three main methods: * ``NodeContext::get_input`` to get input with a given index, -* ``NodeContext::get_attribute`` to get attribute value with a given name. +* ``NodeContext::get_attribute`` to get attribute value with a given name, + +* ``NodeContext::get_values_from_const_input`` to get an attribute with a given input index. The conversion function should return a vector of node outputs that are mapped to corresponding outputs of the original framework operation in the same order. diff --git a/docs/snippets/CMakeLists.txt b/docs/snippets/CMakeLists.txt index 4d824e7c2bc..c41deb340e1 100644 --- a/docs/snippets/CMakeLists.txt +++ b/docs/snippets/CMakeLists.txt @@ -62,7 +62,8 @@ target_include_directories(${TARGET_NAME} PRIVATE "${OpenVINO_SOURCE_DIR}/src/in "${OpenVINO_SOURCE_DIR}/src/frontends/common/include" "${OpenVINO_SOURCE_DIR}/src/core/template_extension/new" "${OpenVINO_SOURCE_DIR}/src/frontends/onnx/frontend/include" - "${OpenVINO_SOURCE_DIR}/src/frontends/tensorflow/include") + "${OpenVINO_SOURCE_DIR}/src/frontends/tensorflow/include" + "${OpenVINO_SOURCE_DIR}/src/frontends/paddle/include") ov_mark_target_as_cc(${TARGET_NAME}) diff --git a/docs/snippets/ov_extensions.cpp b/docs/snippets/ov_extensions.cpp index a9c11e421f5..afe6533830a 100644 --- a/docs/snippets/ov_extensions.cpp +++ b/docs/snippets/ov_extensions.cpp @@ -1,4 +1,4 @@ -// Copyright (C) 2018-2021 Intel Corporation +// Copyright (C) 2018-2023 Intel Corporation // SPDX-License-Identifier: Apache-2.0 // #include @@ -21,6 +21,7 @@ #include #include #include +#include //! [frontend_extension_framework_map_macro_headers] #include @@ -53,12 +54,13 @@ class CustomOp : public ov::op::Op { public: OPENVINO_OP("CustomOp"); - OPENVINO_FRAMEWORK_MAP(onnx, "CustomOp", { {"m_mode", "mode"} }, { {"m_axis", -1} }); - OPENVINO_FRAMEWORK_MAP(tensorflow, "CustomOpV3", { {"m_axis", "axis"} }, { {"m_mode", "linear"} }); + OPENVINO_FRAMEWORK_MAP(onnx, "CustomOp", { {"mode", "mode"} }, { {"axis", -1} }); + OPENVINO_FRAMEWORK_MAP(tensorflow, "CustomOpV3", { {"axis", "axis"} }, { {"mode", "linear"} }); + OPENVINO_FRAMEWORK_MAP(paddle, {"X"}, {"Out"}, "CustomOp", { {"mode", "mode"} }, { {"axis", -1} }); bool visit_attributes(ov::AttributeVisitor& visitor) override { - visitor.on_attribute("m_mode", m_mode); - visitor.on_attribute("m_axis", m_axis); + visitor.on_attribute("mode", m_mode); + visitor.on_attribute("axis", m_axis); return true; } @@ -67,6 +69,39 @@ public: std::shared_ptr clone_with_new_inputs(const ov::OutputVector&) const override { return nullptr; } }; +//! [frontend_extension_framework_map_CustomElu] +class CustomElu : public ov::op::Op { +private: + float m_alpha; + float m_beta; + +public: + OPENVINO_OP("CustomElu"); + + CustomElu() = default; + + CustomElu(const ov::Output& input, float alpha, float beta) : Op({input}), m_alpha(alpha), m_beta(beta) { + constructor_validate_and_infer_types(); + } + + void validate_and_infer_types() override { + set_output_size(1); + set_output_type(0, get_input_element_type(0), get_input_partial_shape(0)); + } + + bool visit_attributes(ov::AttributeVisitor& visitor) override { + visitor.on_attribute("alpha", m_alpha); + visitor.on_attribute("beta", m_beta); + return true; + } + + std::shared_ptr clone_with_new_inputs(const ov::OutputVector& inputs) const override { + return std::make_shared(inputs[0], m_alpha, m_beta); + } +}; +//! [frontend_extension_framework_map_CustomElu] + + int main() { { //! [add_extension] @@ -119,6 +154,10 @@ core.add_extension(ov::frontend::OpExtension<>("Relu", "MyRelu")); core.add_extension(ov::frontend::OpExtension()); //! [frontend_extension_CustomOperation_as_is] +//! [frontend_extension_CustomOperation_as_is_paddle] +core.add_extension(ov::frontend::OpExtension({"A", "B", "C"}, {"X", "Y"})); +//! [frontend_extension_CustomOperation_as_is_paddle] + //! [frontend_extension_CustomOperation_rename] core.add_extension(ov::frontend::OpExtension( std::map{ {"attr1", "fw_attr1"}, {"attr2", "fw_attr2"} }, @@ -126,6 +165,16 @@ core.add_extension(ov::frontend::OpExtension( )); //! [frontend_extension_CustomOperation_rename] +//! [frontend_extension_CustomOperation_rename_paddle] +core.add_extension(ov::frontend::OpExtension( + {"A", "B", "C"}, + {"X", "Y"}, + std::map{ {"attr1", "fw_attr1"}, {"attr2", "fw_attr2"} }, + {} +)); +//! [frontend_extension_CustomOperation_rename_paddle] + + //! [frontend_extension_CustomOperation_rename_set] core.add_extension(ov::frontend::OpExtension( std::map{ {"attr1", "fw_attr1"} }, @@ -133,6 +182,24 @@ core.add_extension(ov::frontend::OpExtension( )); //! [frontend_extension_CustomOperation_rename_set] +//! [frontend_extension_CustomOperation_rename_set_paddle] +core.add_extension(ov::frontend::OpExtension( + {"A", "B", "C"}, + {"X", "Y"}, + std::map{ {"attr1", "fw_attr1"} }, + { {"attr2", 5} } +)); +//! [frontend_extension_CustomOperation_rename_set_paddle] + +{ +//! [frontend_extension_framework_map_CustomElu_mapping] +auto extension = std::make_shared>("aten::elu", + std::map{{"alpha", 1}}, + std::map{{"beta", 1.0f}}); +//! [frontend_extension_framework_map_CustomElu_mapping] +} + + //! [frontend_extension_ThresholdedReLU] core.add_extension(ov::frontend::ConversionExtension( "ThresholdedRelu", diff --git a/docs/snippets/ov_extensions.py b/docs/snippets/ov_extensions.py index 10219e6f29d..3b0d320eba3 100644 --- a/docs/snippets/ov_extensions.py +++ b/docs/snippets/ov_extensions.py @@ -38,3 +38,32 @@ def conversion(node): core.add_extension(ConversionExtension("ThresholdedRelu", conversion)) #! [py_frontend_extension_ThresholdedReLU] + + +#! [py_frontend_extension_aten_hardtanh] +import torch +from openvino.frontend import ConversionExtension, NodeContext +from openvino.tools.mo import convert_model + + +class HardTanh(torch.nn.Module): + def __init__(self, min_val, max_val): + super(HardTanh, self).__init__() + self.min_val = min_val + self.max_val = max_val + + def forward(self, inp): + return torch.nn.functional.hardtanh(inp, self.min_val, self.max_val) + + +def convert_hardtanh(node: NodeContext): + inp = node.get_input(0) + min_value = node.get_values_from_const_input(1) + max_value = node.get_values_from_const_input(2) + return ops.clamp(inp, min_value, max_value).outputs() + + +model = HardTanh(min_val=0.1, max_val=2.0) +hardtanh_ext = ConversionExtension("aten::hardtanh", convert_hardtanh) +ov_model = convert_model(input_model=model, extensions=[hardtanh_ext]) +#! [py_frontend_extension_aten_hardtanh] diff --git a/src/frontends/pytorch/src/node_context.cpp b/src/frontends/pytorch/src/node_context.cpp index 1eec75c3f25..c3ae7f9008e 100644 --- a/src/frontends/pytorch/src/node_context.cpp +++ b/src/frontends/pytorch/src/node_context.cpp @@ -228,6 +228,8 @@ Any NodeContext::get_values_from_const_input(int index) const { return get_constant_data(constant); case element::u16: return get_constant_data(constant); + case element::boolean: + return get_constant_data(constant); default: FRONT_END_GENERAL_CHECK(false, "Input with index: ", index, " has unsupported type."); } diff --git a/tests/layer_tests/py_frontend_tests/test_torch_frontend.py b/tests/layer_tests/py_frontend_tests/test_torch_frontend.py index 1c7a29ea7af..d348f933843 100644 --- a/tests/layer_tests/py_frontend_tests/test_torch_frontend.py +++ b/tests/layer_tests/py_frontend_tests/test_torch_frontend.py @@ -130,20 +130,17 @@ def test_conversion_extension(): def convert_vector_norm(node: NodeContext): - try: - inp = node.get_input(0) - ord = node.get_values_from_const_input(1) - assert ord == math.inf - dim = node.get_values_from_const_input(2) - if dim is None: - inp = ops.reshape(inp, ops.constant(np.array([-1])), False) - reduce_axes = np.array([0]) - else: - reduce_axes = np.array(dim) - rm = ops.reduce_max(ops.abs(inp), reduce_axes, False) - return rm.outputs() - except Exception as e: - print(e) + inp = node.get_input(0) + ord = node.get_values_from_const_input(1) + assert ord == math.inf + dim = node.get_values_from_const_input(2) + if dim is None: + inp = ops.reshape(inp, ops.constant(np.array([-1])), False) + reduce_axes = np.array([0]) + else: + reduce_axes = np.array(dim) + rm = ops.reduce_max(ops.abs(inp), reduce_axes, False) + return rm.outputs() fem = FrontEndManager()