e3fdfc4e09
* shift to rst * test snippets * test build fixes * change code block * test new path * change path * add cancel * change note format * add docs * change path to snippet * change path to snippet * change list format * fix list * fix snippets path * fix format * fix lists * fix snippet * compiled model doc fix * change indentation * small fixes to format
231 lines
11 KiB
Markdown
231 lines
11 KiB
Markdown
# Plugin {#openvino_docs_ov_plugin_dg_plugin}
|
|
|
|
@sphinxdirective
|
|
|
|
OpenVINO Plugin usually represents a wrapper around a backend. Backends can be:
|
|
|
|
* OpenCL-like backend (e.g. clDNN library) for GPU devices.
|
|
* oneDNN backend for Intel CPU devices.
|
|
* NVIDIA cuDNN for NVIDIA GPUs.
|
|
|
|
The responsibility of OpenVINO Plugin:
|
|
|
|
* Initializes a backend and throw exception in ``Engine`` constructor if backend cannot be initialized.
|
|
* Provides information about devices enabled by a particular backend, e.g. how many devices, their properties and so on.
|
|
* Loads or imports :doc:`compiled model <openvino_docs_ov_plugin_dg_compiled_model>` objects.
|
|
|
|
In addition to the OpenVINO Public API, the OpenVINO provides the Plugin API, which is a set of functions and helper classes that simplify new plugin development:
|
|
|
|
* header files in the ``src/inference/dev_api/openvino`` directory
|
|
* implementations in the ``src/inference/src/dev/`` directory
|
|
* symbols in the OpenVINO shared library
|
|
|
|
To build an OpenVINO plugin with the Plugin API, see the :doc:`OpenVINO Plugin Building <openvino_docs_ov_plugin_dg_plugin_build>` guide.
|
|
|
|
Plugin Class
|
|
############
|
|
|
|
OpenVINO Plugin API provides the helper ov::IPlugin class recommended to use as a base class for a plugin.
|
|
Based on that, declaration of a plugin class can look as follows:
|
|
|
|
.. doxygensnippet:: src/plugins/template/src/plugin.hpp
|
|
:language: cpp
|
|
:fragment: [plugin:header]
|
|
|
|
|
|
Class Fields
|
|
++++++++++++
|
|
|
|
The provided plugin class also has several fields:
|
|
|
|
* ``m_backend`` - a backend engine that is used to perform actual computations for model inference. For ``Template`` plugin ``ov::runtime::Backend`` is used which performs computations using OpenVINO™ reference implementations.
|
|
* ``m_waitExecutor`` - a task executor that waits for a response from a device about device tasks completion.
|
|
* ``m_cfg`` of type ``Configuration``:
|
|
|
|
.. doxygensnippet:: src/plugins/template/src/config.hpp
|
|
:language: cpp
|
|
:fragment: [configuration:header]
|
|
|
|
As an example, a plugin configuration has three value parameters:
|
|
|
|
* ``device_id`` - particular device ID to work with. Applicable if a plugin supports more than one ``Template`` device. In this case, some plugin methods, like ``set_property``, ``query_model``, and ``compile_model``, must support the ov::device::id property.
|
|
* ``perf_counts`` - boolean value to identify whether to collect performance counters during :doc:`Inference Request <openvino_docs_ov_plugin_dg_infer_request>` execution.
|
|
* ``streams_executor_config`` - configuration of ``ov::threading::IStreamsExecutor`` to handle settings of multi-threaded context.
|
|
* ``performance_mode`` - configuration of ``ov::hint::PerformanceMode`` to set the performance mode.
|
|
* ``disable_transformations`` - allows to disable transformations which are applied in the process of model compilation.
|
|
* ``exclusive_async_requests`` - allows to use exclusive task executor for asynchronous infer requests.
|
|
|
|
Plugin Constructor
|
|
++++++++++++++++++
|
|
|
|
A plugin constructor must contain code that checks the ability to work with a device of the ``Template``
|
|
type. For example, if some drivers are required, the code must check
|
|
driver availability. If a driver is not available (for example, OpenCL runtime is not installed in
|
|
case of a GPU device or there is an improper version of a driver is on a host machine), an exception
|
|
must be thrown from a plugin constructor.
|
|
|
|
A plugin must define a device name enabled via the ``set_device_name()`` method of a base class:
|
|
|
|
.. doxygensnippet:: src/plugins/template/src/plugin.cpp
|
|
:language: cpp
|
|
:fragment: [plugin:ctor]
|
|
|
|
Plugin Destructor
|
|
+++++++++++++++++
|
|
|
|
A plugin destructor must stop all plugins activities, and clean all allocated resources.
|
|
|
|
|
|
.. doxygensnippet:: src/plugins/template/src/plugin.cpp
|
|
:language: cpp
|
|
:fragment: [plugin:dtor]
|
|
|
|
compile_model()
|
|
+++++++++++++++
|
|
|
|
The plugin should implement two ``compile_model()`` methods: the first one compiles model without remote context, the second one with remote context if plugin supports.
|
|
|
|
This is the most important function of the ``Plugin`` class is to create an instance of compiled ``CompiledModel``,
|
|
which holds a backend-dependent compiled model in an internal representation:
|
|
|
|
.. doxygensnippet:: src/plugins/template/src/plugin.cpp
|
|
:language: cpp
|
|
:fragment: [plugin:compile_model]
|
|
|
|
.. doxygensnippet:: src/plugins/template/src/plugin.cpp
|
|
:language: cpp
|
|
:fragment: [plugin:compile_model_with_remote]
|
|
|
|
Before a creation of an ``CompiledModel`` instance via a constructor, a plugin may check if a provided
|
|
ov::Model object is supported by a device if it is needed.
|
|
|
|
Actual model compilation is done in the ``CompiledModel`` constructor. Refer to the :doc:`CompiledModel Implementation Guide <openvino_docs_ov_plugin_dg_compiled_model>` for details.
|
|
|
|
.. note::
|
|
|
|
Actual configuration map used in ``CompiledModel`` is constructed as a base plugin configuration set via ``Plugin::set_property``, where some values are overwritten with ``config`` passed to ``Plugin::compile_model``. Therefore, the config of ``Plugin::compile_model`` has a higher priority.
|
|
|
|
transform_model()
|
|
+++++++++++++++++
|
|
|
|
The function accepts a const shared pointer to `ov::Model` object and applies common and device-specific transformations on a copied model to make it more friendly to hardware operations. For details how to write custom device-specific transformation, refer to :doc:`Writing OpenVINO™ transformations <openvino_docs_transformations>` guide. See detailed topics about model representation:
|
|
|
|
* :doc:`Intermediate Representation and Operation Sets <openvino_docs_MO_DG_IR_and_opsets>`
|
|
* :doc:`Quantized models <openvino_docs_ov_plugin_dg_quantized_models>`.
|
|
|
|
|
|
.. doxygensnippet:: src/plugins/template/src/plugin.cpp
|
|
:language: cpp
|
|
:fragment: [plugin:transform_model]
|
|
|
|
.. note::
|
|
|
|
After all these transformations, an ``ov::Model`` object contains operations which can be perfectly mapped to backend kernels. E.g. if backend has kernel computing ``A + B`` operations at once, the ``transform_model`` function should contain a pass which fuses operations ``A`` and ``B`` into a single custom operation `A + B` which fits backend kernels set.
|
|
|
|
query_model()
|
|
+++++++++++++
|
|
|
|
Use the method with the ``HETERO`` mode, which allows to distribute model execution between different
|
|
devices based on the ``ov::Node::get_rt_info()`` map, which can contain the ``affinity`` key.
|
|
The ``query_model`` method analyzes operations of provided ``model`` and returns a list of supported
|
|
operations via the ov::SupportedOpsMap structure. The ``query_model`` firstly applies ``transform_model`` passes to input ``ov::Model`` argument. After this, the transformed model in ideal case contains only operations are 1:1 mapped to kernels in computational backend. In this case, it's very easy to analyze which operations is supposed (``m_backend`` has a kernel for such operation or extensions for the operation is provided) and not supported (kernel is missed in ``m_backend``):
|
|
|
|
1. Store original names of all operations in input ``ov::Model``.
|
|
2. Apply ``transform_model`` passes. Note, the names of operations in a transformed model can be different and we need to restore the mapping in the steps below.
|
|
3. Construct ``supported`` map which contains names of original operations. Note that since the inference is performed using OpenVINO™ reference backend, the decision whether the operation is supported or not depends on whether the latest OpenVINO opset contains such operation.
|
|
4. ``ov.SupportedOpsMap`` contains only operations which are fully supported by ``m_backend``.
|
|
|
|
.. doxygensnippet:: src/plugins/template/src/plugin.cpp
|
|
:language: cpp
|
|
:fragment: [plugin:query_model]
|
|
|
|
set_property()
|
|
++++++++++++++
|
|
|
|
Sets new values for plugin property keys:
|
|
|
|
.. doxygensnippet:: src/plugins/template/src/plugin.cpp
|
|
:language: cpp
|
|
:fragment: [plugin:set_property]
|
|
|
|
In the snippet above, the ``Configuration`` class overrides previous configuration values with the new
|
|
ones. All these values are used during backend specific model compilation and execution of inference requests.
|
|
|
|
.. note::
|
|
|
|
The function must throw an exception if it receives an unsupported configuration key.
|
|
|
|
get_property()
|
|
++++++++++++++
|
|
|
|
Returns a current value for a specified property key:
|
|
|
|
.. doxygensnippet:: src/plugins/template/src/plugin.cpp
|
|
:language: cpp
|
|
:fragment: [plugin:get_property]
|
|
|
|
The function is implemented with the ``Configuration::Get`` method, which wraps an actual configuration
|
|
key value to the ov::Any and returns it.
|
|
|
|
.. note::
|
|
|
|
The function must throw an exception if it receives an unsupported configuration key.
|
|
|
|
import_model()
|
|
++++++++++++++
|
|
|
|
The importing of compiled model mechanism allows to import a previously exported backend specific model and wrap it
|
|
using an :doc:`CompiledModel <openvino_docs_ov_plugin_dg_compiled_model>` object. This functionality is useful if
|
|
backend specific model compilation takes significant time and/or cannot be done on a target host
|
|
device due to other reasons.
|
|
|
|
During export of backend specific model using ``CompiledModel::export_model``, a plugin may export any
|
|
type of information it needs to import a compiled model properly and check its correctness.
|
|
For example, the export information may include:
|
|
|
|
* Compilation options (state of ``Plugin::m_cfg`` structure).
|
|
* Information about a plugin and a device type to check this information later during the import and throw an exception if the ``model`` stream contains wrong data. For example, if devices have different capabilities and a model compiled for a particular device cannot be used for another, such type of information must be stored and checked during the import.
|
|
* Compiled backend specific model itself.
|
|
|
|
.. doxygensnippet:: src/plugins/template/src/plugin.cpp
|
|
:language: cpp
|
|
:fragment: [plugin:import_model]
|
|
|
|
.. doxygensnippet:: src/plugins/template/src/plugin.cpp
|
|
:language: cpp
|
|
:fragment: [plugin:import_model_with_remote]
|
|
|
|
|
|
create_context()
|
|
++++++++++++++++
|
|
|
|
The Plugin should implement ``Plugin::create_context()`` method which returns ``ov::RemoteContext`` in case if plugin supports remote context, in other case the plugin can throw an exception that this method is not implemented.
|
|
|
|
.. doxygensnippet:: src/plugins/template/src/plugin.cpp
|
|
:language: cpp
|
|
:fragment: [plugin:create_context]
|
|
|
|
|
|
get_default_context()
|
|
+++++++++++++++++++++
|
|
|
|
``Plugin::get_default_context()`` also needed in case if plugin supports remote context, if the plugin doesn't support it, this method can throw an exception that functionality is not implemented.
|
|
|
|
.. doxygensnippet:: src/plugins/template/src/plugin.cpp
|
|
:language: cpp
|
|
:fragment: [plugin:get_default_context]
|
|
|
|
Create Instance of Plugin Class
|
|
###############################
|
|
|
|
OpenVINO plugin library must export only one function creating a plugin instance using OV_DEFINE_PLUGIN_CREATE_FUNCTION macro:
|
|
|
|
.. doxygensnippet:: src/plugins/template/src/plugin.cpp
|
|
:language: cpp
|
|
:fragment: [plugin:create_plugin_engine]
|
|
|
|
|
|
Next step in a plugin library implementation is the :doc:`CompiledModel <openvino_docs_ov_plugin_dg_compiled_model>` class.
|
|
|
|
@endsphinxdirective
|