Add documentation for compiled model (#16332)
* Files renaming * Updated CompiledModel documentation * Fixed typo * Fixed comments * Fix comments and try to fix the documentation * Fixed execution devices
This commit is contained in:
Ilya Churaev
GitHub
parent
6889101415
commit
e6ceed0bb9
@ -1,4 +1,4 @@
|
||||
# Build Plugin Using CMake {#openvino_docs_ie_plugin_dg_plugin_build}
|
||||
# Build Plugin Using CMake {#openvino_docs_ov_plugin_dg_plugin_build}
|
||||
|
||||
OpenVINO build infrastructure provides the OpenVINO Developer Package for plugin development.
|
||||
|
||||
|
||||
89
docs/IE_PLUGIN_DG/CompiledModel.md
Normal file
89
docs/IE_PLUGIN_DG/CompiledModel.md
Normal file
@ -0,0 +1,89 @@
|
||||
# Compiled Model {#openvino_docs_ov_plugin_dg_compiled_model}
|
||||
|
||||
ov::CompiledModel class functionality:
|
||||
- Compile an ov::Model instance to a backend specific graph representation
|
||||
- Create an arbitrary number of ov::InferRequest objects
|
||||
- Hold some common resources shared between different instances of ov::InferRequest. For example:
|
||||
- ov::ICompiledModel::m_task_executor task executor to implement asynchronous execution
|
||||
- ov::ICompiledModel::m_callback_executor task executor to run an asynchronous inference request callback in a separate thread
|
||||
|
||||
CompiledModel Class
|
||||
------------------------
|
||||
|
||||
OpenVINO Plugin API provides the interface ov::ICompiledModel which should be used as a base class for a compiled model. Based on that, a declaration of an compiled model class can look as follows:
|
||||
|
||||
@snippet src/compiled_model.hpp compiled_model:header
|
||||
|
||||
### Class Fields
|
||||
|
||||
The example class has several fields:
|
||||
|
||||
- `m_request_id` - Tracks a number of created inference requests, which is used to distinguish different inference requests during profiling via the Intel® Instrumentation and Tracing Technology (ITT) library.
|
||||
- `m_cfg` - Defines a configuration a compiled model was compiled with.
|
||||
- `m_model` - Keeps a reference to transformed `ov::Model` which is used in OpenVINO reference backend computations. Note, in case of other backends with backend specific graph representation `m_model` has different type and represents backend specific graph or just a set of computational kernels to perform an inference.
|
||||
- `m_loaded_from_cache` - Allows to understand that model was loaded from cache.
|
||||
|
||||
### CompiledModel Constructor
|
||||
|
||||
This constructor accepts a generic representation of a model as an ov::Model and is compiled into a backend specific device graph:
|
||||
|
||||
@snippet src/compiled_model.cpp compiled_model:ctor
|
||||
|
||||
The implementation `compile_model()` is fully device-specific.
|
||||
|
||||
### compile_model()
|
||||
|
||||
The function accepts a const shared pointer to `ov::Model` object and applies OpenVINO passes using `transform_model()` function, which defines plugin-specific conversion pipeline. To support low precision inference, the pipeline can include Low Precision Transformations. These transformations are usually hardware specific. You can find how to use and configure Low Precisions Transformations in [Low Precision Transformations](@ref openvino_docs_OV_UG_lpt) guide.
|
||||
|
||||
@snippet src/compiled_model.cpp compiled_model:compile_model
|
||||
|
||||
> **NOTE**: After all these steps, the backend specific graph is ready to create inference requests and perform inference.
|
||||
|
||||
### export_model()
|
||||
|
||||
The implementation of the method should write all data to the `model_stream`, which is required to import a backend specific graph later in the `Plugin::import_model` method:
|
||||
|
||||
@snippet src/compiled_model.cpp compiled_model:export_model
|
||||
|
||||
### create_sync_infer_request()
|
||||
|
||||
The method creates an synchronous inference request and returns it.
|
||||
|
||||
@snippet src/compiled_model.cpp compiled_model:create_sync_infer_request
|
||||
|
||||
While the public OpenVINO API has a single interface for inference request, which can be executed in synchronous and asynchronous modes, a plugin library implementation has two separate classes:
|
||||
|
||||
- [Synchronous inference request](@ref openvino_docs_ie_plugin_dg_infer_request), which defines pipeline stages and runs them synchronously in the `infer` method.
|
||||
- [Asynchronous inference request](@ref openvino_docs_ie_plugin_dg_async_infer_request), which is a wrapper for a synchronous inference request and can run a pipeline asynchronously. Depending on a device pipeline structure, it can has one or several stages:
|
||||
- For single-stage pipelines, there is no need to define this method and create a class derived from ov::IAsyncInferRequest. For single stage pipelines, a default implementation of this method creates ov::IAsyncInferRequest wrapping a synchronous inference request and runs it asynchronously in the `m_request_executor` executor.
|
||||
- For pipelines with multiple stages, such as performing some preprocessing on host, uploading input data to a device, running inference on a device, or downloading and postprocessing output data, schedule stages on several task executors to achieve better device use and performance. You can do it by creating a sufficient number of inference requests running in parallel. In this case, device stages of different inference requests are overlapped with preprocessing and postprocessing stage giving better performance.
|
||||
> **IMPORTANT**: It is up to you to decide how many task executors you need to optimally execute a device pipeline.
|
||||
|
||||
|
||||
### create_infer_request()
|
||||
|
||||
The method creates an asynchronous inference request and returns it.
|
||||
|
||||
@snippet src/compiled_model.cpp compiled_model:create_infer_request
|
||||
|
||||
### get_property()
|
||||
|
||||
Returns a current value for a property with the name `name`. The method extracts configuration values a compiled model is compiled with.
|
||||
|
||||
@snippet src/compiled_model.cpp compiled_model:get_property
|
||||
|
||||
This function is the only way to get configuration values when a model is imported and compiled by other developers and tools.
|
||||
|
||||
### set_property()
|
||||
|
||||
The methods allows to set compiled model specific properties.
|
||||
|
||||
@snippet src/compiled_model.cpp compiled_model:set_property
|
||||
|
||||
### get_runtime_model()
|
||||
|
||||
The methods returns the runtime model with backend specific information.
|
||||
|
||||
@snippet src/compiled_model.cpp compiled_model:get_runtime_model
|
||||
|
||||
The next step in plugin library implementation is the [Synchronous Inference Request](@ref openvino_docs_ie_plugin_dg_infer_request) class.
|
||||
@ -1,90 +0,0 @@
|
||||
# Executable Network {#openvino_docs_ie_plugin_dg_executable_network}
|
||||
|
||||
`ExecutableNetwork` class functionality:
|
||||
- Compile an InferenceEngine::ICNNNetwork instance to a backend specific graph representation
|
||||
- Create an arbitrary number of `InferRequest` objects
|
||||
- Hold some common resources shared between different instances of `InferRequest`. For example:
|
||||
- InferenceEngine::IExecutableNetworkInternal::_taskExecutor task executor to implement asynchronous execution
|
||||
- InferenceEngine::IExecutableNetworkInternal::_callbackExecutor task executor to run an asynchronous inference request callback in a separate thread
|
||||
|
||||
`ExecutableNetwork` Class
|
||||
------------------------
|
||||
|
||||
Inference Engine Plugin API provides the helper InferenceEngine::ExecutableNetworkThreadSafeDefault class recommended to use as a base class for an executable network. Based on that, a declaration of an executable network class can look as follows:
|
||||
|
||||
@snippet src/compiled_model.hpp executable_network:header
|
||||
|
||||
#### Class Fields
|
||||
|
||||
The example class has several fields:
|
||||
|
||||
- `_requestId` - Tracks a number of created inference requests, which is used to distinguish different inference requests during profiling via the Intel® Instrumentation and Tracing Technology (ITT) library.
|
||||
- `_cfg` - Defines a configuration an executable network was compiled with.
|
||||
- `_plugin` - Refers to a plugin instance.
|
||||
- `_function` - Keeps a reference to transformed `ngraph::Function` which is used in ngraph reference backend computations. Note, in case of other backends with backend specific graph representation `_function` has different type and represents backend specific graph or just a set of computational kernels to perform an inference.
|
||||
- `_inputIndex` - maps a name of input with its index among all network inputs.
|
||||
- `_outputIndex` - maps a name of output with its index among all network outputs.
|
||||
|
||||
### `ExecutableNetwork` Constructor with `ICNNNetwork`
|
||||
|
||||
This constructor accepts a generic representation of a neural network as an InferenceEngine::ICNNNetwork reference and is compiled into a backend specific device graph:
|
||||
|
||||
@snippet src/compiled_model.cpp executable_network:ctor_cnnnetwork
|
||||
|
||||
The implementation `CompileNetwork` is fully device-specific.
|
||||
|
||||
### `CompileNetwork()`
|
||||
|
||||
The function accepts a const shared pointer to `ngraph::Function` object and performs the following steps:
|
||||
|
||||
1. Applies nGraph passes using `TransformNetwork` function, which defines plugin-specific conversion pipeline. To support low precision inference, the pipeline can include Low Precision Transformations. These transformations are usually hardware specific. You can find how to use and configure Low Precisions Transformations in [Low Precision Transformations](@ref openvino_docs_OV_UG_lpt) guide.
|
||||
2. Maps the transformed graph to a backend specific graph representation (for example, to CPU plugin internal graph representation).
|
||||
3. Allocates and fills memory for graph weights, backend specific memory handles and so on.
|
||||
|
||||
@snippet src/compiled_model.cpp executable_network:map_graph
|
||||
|
||||
> **NOTE**: After all these steps, the backend specific graph is ready to create inference requests and perform inference.
|
||||
|
||||
### `ExecutableNetwork` Constructor Importing from Stream
|
||||
|
||||
This constructor creates a backend specific graph by importing from a stream object:
|
||||
|
||||
> **NOTE**: The export of backend specific graph is done in the `Export` method, and data formats must be the same for both import and export.
|
||||
|
||||
### `Export()`
|
||||
|
||||
The implementation of the method should write all data to the `model` stream, which is required to import a backend specific graph later in the `Plugin::Import` method:
|
||||
|
||||
@snippet src/compiled_model.cpp executable_network:export
|
||||
|
||||
### `CreateInferRequest()`
|
||||
|
||||
The method creates an asynchronous inference request and returns it. While the public Inference Engine API has a single interface for inference request, which can be executed in synchronous and asynchronous modes, a plugin library implementation has two separate classes:
|
||||
|
||||
- [Synchronous inference request](@ref openvino_docs_ie_plugin_dg_infer_request), which defines pipeline stages and runs them synchronously in the `Infer` method.
|
||||
- [Asynchronous inference request](@ref openvino_docs_ie_plugin_dg_async_infer_request), which is a wrapper for a synchronous inference request and can run a pipeline asynchronously. Depending on a device pipeline structure, it can has one or several stages:
|
||||
- For single-stage pipelines, there is no need to define this method and create a class derived from InferenceEngine::AsyncInferRequestThreadSafeDefault. For single stage pipelines, a default implementation of this method creates InferenceEngine::AsyncInferRequestThreadSafeDefault wrapping a synchronous inference request and runs it asynchronously in the `_taskExecutor` executor.
|
||||
- For pipelines with multiple stages, such as performing some preprocessing on host, uploading input data to a device, running inference on a device, or downloading and postprocessing output data, schedule stages on several task executors to achieve better device use and performance. You can do it by creating a sufficient number of inference requests running in parallel. In this case, device stages of different inference requests are overlapped with preprocessing and postprocessing stage giving better performance.
|
||||
> **IMPORTANT**: It is up to you to decide how many task executors you need to optimally execute a device pipeline.
|
||||
|
||||
@snippet src/compiled_model.cpp executable_network:create_infer_request
|
||||
|
||||
### `GetMetric()`
|
||||
|
||||
Returns a metric value for a metric with the name `name`. A metric is a static type of information about an executable network. Examples of metrics:
|
||||
|
||||
- EXEC_NETWORK_METRIC_KEY(NETWORK_NAME) - name of an executable network
|
||||
- EXEC_NETWORK_METRIC_KEY(OPTIMAL_NUMBER_OF_INFER_REQUESTS) - heuristic to denote an optimal (or at least sub-optimal) number of inference requests needed to run asynchronously to use the current device fully
|
||||
- Any other executable network metric specific for a particular device. Such metrics and possible values must be declared in a plugin configuration public header, for example, `template/config.hpp`
|
||||
|
||||
The IE_SET_METRIC_RETURN helper macro sets metric value and checks that the actual metric type matches a type of the specified value.
|
||||
|
||||
### `GetConfig()`
|
||||
|
||||
Returns a current value for a configuration key with the name `name`. The method extracts configuration values an executable network is compiled with.
|
||||
|
||||
@snippet src/compiled_model.cpp executable_network:get_config
|
||||
|
||||
This function is the only way to get configuration values when a network is imported and compiled by other developers and tools (for example, the [Compile tool](@ref openvino_inference_engine_tools_compile_tool_README).
|
||||
|
||||
The next step in plugin library implementation is the [Synchronous Inference Request](@ref openvino_docs_ie_plugin_dg_infer_request) class.
|
||||
@ -7,12 +7,12 @@
|
||||
:caption: Converting and Preparing Models
|
||||
:hidden:
|
||||
|
||||
Implement Plugin Functionality <openvino_docs_ie_plugin_dg_plugin>
|
||||
Implement Executable Network Functionality <openvino_docs_ie_plugin_dg_executable_network>
|
||||
Implement Plugin Functionality <openvino_docs_ov_plugin_dg_plugin>
|
||||
Implement Compiled Model Functionality <openvino_docs_ov_plugin_dg_compiled_model>
|
||||
Implement Synchronous Inference Request <openvino_docs_ie_plugin_dg_infer_request>
|
||||
Implement Asynchronous Inference Request <openvino_docs_ie_plugin_dg_async_infer_request>
|
||||
openvino_docs_ie_plugin_dg_plugin_build
|
||||
openvino_docs_ie_plugin_dg_plugin_testing
|
||||
openvino_docs_ov_plugin_dg_plugin_build
|
||||
openvino_docs_ov_plugin_dg_plugin_testing
|
||||
openvino_docs_ie_plugin_detailed_guides
|
||||
openvino_docs_ie_plugin_api_references
|
||||
|
||||
@ -27,14 +27,14 @@ OpenVINO Plugin Library
|
||||
|
||||
OpenVINO plugin dynamic library consists of several main components:
|
||||
|
||||
1. [Plugin class](@ref openvino_docs_ie_plugin_dg_plugin):
|
||||
1. [Plugin class](@ref openvino_docs_ov_plugin_dg_plugin):
|
||||
- Provides information about devices of a specific type.
|
||||
- Can create an [compiled model](@ref openvino_docs_ie_plugin_dg_executable_network) instance which represents a Neural
|
||||
- Can create an [compiled model](@ref openvino_docs_ov_plugin_dg_compiled_model) instance which represents a Neural
|
||||
Network backend specific graph structure for a particular device in opposite to the ov::Model
|
||||
which is backend-independent.
|
||||
- Can import an already compiled graph structure from an input stream to an
|
||||
[compiled model](@ref openvino_docs_ie_plugin_dg_executable_network) object.
|
||||
2. [Compiled Modek class](@ref openvino_docs_ie_plugin_dg_executable_network):
|
||||
[compiled model](@ref openvino_docs_ov_plugin_dg_compiled_model) object.
|
||||
2. [Compiled Modek class](@ref openvino_docs_ov_plugin_dg_compiled_model):
|
||||
- Is an execution configuration compiled for a particular device and takes into account its capabilities.
|
||||
- Holds a reference to a particular device and a task executor for this device.
|
||||
- Can create several instances of [Inference Request](@ref openvino_docs_ie_plugin_dg_infer_request).
|
||||
@ -55,8 +55,8 @@ at `<openvino source dir>/src/plugins/template`.
|
||||
Detailed guides
|
||||
-----------------------
|
||||
|
||||
* [Build](@ref openvino_docs_ie_plugin_dg_plugin_build) a plugin library using CMake
|
||||
* Plugin and its components [testing](@ref openvino_docs_ie_plugin_dg_plugin_testing)
|
||||
* [Build](@ref openvino_docs_ov_plugin_dg_plugin_build) a plugin library using CMake
|
||||
* Plugin and its components [testing](@ref openvino_docs_ov_plugin_dg_plugin_testing)
|
||||
* [Quantized networks](@ref openvino_docs_ie_plugin_dg_quantized_networks)
|
||||
* [Low precision transformations](@ref openvino_docs_OV_UG_lpt) guide
|
||||
* [Writing OpenVINO™ transformations](@ref openvino_docs_transformations) guide
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
# Plugin {#openvino_docs_ie_plugin_dg_plugin}
|
||||
# Plugin {#openvino_docs_ov_plugin_dg_plugin}
|
||||
|
||||
OpenVINO Plugin usually represents a wrapper around a backend. Backends can be:
|
||||
- OpenCL-like backend (e.g. clDNN library) for GPU devices.
|
||||
@ -8,7 +8,7 @@ OpenVINO Plugin usually represents a wrapper around a backend. Backends can be:
|
||||
The responsibility of OpenVINO Plugin:
|
||||
- Initializes a backend and throw exception in `Engine` constructor if backend cannot be initialized.
|
||||
- Provides information about devices enabled by a particular backend, e.g. how many devices, their properties and so on.
|
||||
- Loads or imports [compiled model](@ref openvino_docs_ie_plugin_dg_executable_network) objects.
|
||||
- Loads or imports [compiled model](@ref openvino_docs_ov_plugin_dg_compiled_model) objects.
|
||||
|
||||
In addition to the OpenVINO Public API, the OpenVINO provides the Plugin API, which is a set of functions and helper classes that simplify new plugin development:
|
||||
|
||||
@ -16,7 +16,7 @@ In addition to the OpenVINO Public API, the OpenVINO provides the Plugin API, wh
|
||||
- implementations in the `src/inference/src/dev/` directory
|
||||
- symbols in the OpenVINO shared library
|
||||
|
||||
To build an OpenVINO plugin with the Plugin API, see the [OpenVINO Plugin Building](@ref openvino_docs_ie_plugin_dg_plugin_build) guide.
|
||||
To build an OpenVINO plugin with the Plugin API, see the [OpenVINO Plugin Building](@ref openvino_docs_ov_plugin_dg_plugin_build) guide.
|
||||
|
||||
Plugin Class
|
||||
------------------------
|
||||
@ -75,7 +75,7 @@ which holds a backend-dependent compiled model in an internal representation:
|
||||
Before a creation of an `CompiledModel` instance via a constructor, a plugin may check if a provided
|
||||
ov::Model object is supported by a device if it is needed.
|
||||
|
||||
Actual model compilation is done in the `CompiledModel` constructor. Refer to the [CompiledModel Implementation Guide](@ref openvino_docs_ie_plugin_dg_executable_network) for details.
|
||||
Actual model compilation is done in the `CompiledModel` constructor. Refer to the [CompiledModel Implementation Guide](@ref openvino_docs_ov_plugin_dg_compiled_model) for details.
|
||||
|
||||
> **NOTE**: Actual configuration map used in `CompiledModel` is constructed as a base plugin
|
||||
> configuration set via `Plugin::set_property`, where some values are overwritten with `config` passed to `Plugin::compile_model`.
|
||||
@ -130,7 +130,7 @@ key value to the ov::Any and returns it.
|
||||
### import_model()
|
||||
|
||||
The importing of compiled model mechanism allows to import a previously exported backend specific model and wrap it
|
||||
using an [CompiledModel](@ref openvino_docs_ie_plugin_dg_executable_network) object. This functionality is useful if
|
||||
using an [CompiledModel](@ref openvino_docs_ov_plugin_dg_compiled_model) object. This functionality is useful if
|
||||
backend specific model compilation takes significant time and/or cannot be done on a target host
|
||||
device due to other reasons.
|
||||
|
||||
@ -167,4 +167,4 @@ OpenVINO plugin library must export only one function creating a plugin instance
|
||||
|
||||
@snippet template/src/plugin.cpp plugin:create_plugin_engine
|
||||
|
||||
Next step in a plugin library implementation is the [CompiledModel](@ref openvino_docs_ie_plugin_dg_executable_network) class.
|
||||
Next step in a plugin library implementation is the [CompiledModel](@ref openvino_docs_ov_plugin_dg_compiled_model) class.
|
||||
|
||||
@ -1,10 +1,10 @@
|
||||
# Plugin Testing {#openvino_docs_ie_plugin_dg_plugin_testing}
|
||||
# Plugin Testing {#openvino_docs_ov_plugin_dg_plugin_testing}
|
||||
|
||||
OpenVINO tests infrastructure provides a predefined set of functional tests and utilities. They are used to verify a plugin using the OpenVINO public API.
|
||||
All the tests are written in the [Google Test C++ framework](https://github.com/google/googletest).
|
||||
|
||||
OpenVINO Plugin tests are included in the `openvino::funcSharedTests` CMake target which is built within the OpenVINO repository
|
||||
(see [Build Plugin Using CMake](@ref openvino_docs_ie_plugin_dg_plugin_build) guide). This library contains tests definitions (the tests bodies) which can be parametrized and instantiated in plugins depending on whether a plugin supports a particular feature, specific sets of parameters for test on supported operation set and so on.
|
||||
(see [Build Plugin Using CMake](@ref openvino_docs_ov_plugin_dg_plugin_build) guide). This library contains tests definitions (the tests bodies) which can be parametrized and instantiated in plugins depending on whether a plugin supports a particular feature, specific sets of parameters for test on supported operation set and so on.
|
||||
|
||||
Test definitions are split into tests class declaration (see `src/tests/functional/plugin/shared/include`) and tests class implementation (see `src/tests/functional/plugin/shared/src`) and include the following scopes of plugin conformance tests:
|
||||
|
||||
@ -35,7 +35,7 @@ To use these tests for your own plugin development, link the `openvino::funcShar
|
||||
> **NOTE**: A plugin may contain its own tests for use cases that are specific to hardware or need to be extensively tested.
|
||||
|
||||
To build test binaries together with other build artifacts, use the `make all` command. For details, see
|
||||
[Build Plugin Using CMake*](@ref openvino_docs_ie_plugin_dg_plugin_build).
|
||||
[Build Plugin Using CMake*](@ref openvino_docs_ov_plugin_dg_plugin_build).
|
||||
|
||||
### How to Extend OpenVINO Plugin Tests
|
||||
|
||||
|
||||
@ -15,19 +15,19 @@
|
||||
#include "template/config.hpp"
|
||||
#include "transformations/utils/utils.hpp"
|
||||
|
||||
// ! [executable_network:ctor_cnnnetwork]
|
||||
// ! [compiled_model:ctor]
|
||||
ov::template_plugin::CompiledModel::CompiledModel(const std::shared_ptr<ov::Model>& model,
|
||||
const std::shared_ptr<const ov::IPlugin>& plugin,
|
||||
const std::shared_ptr<ov::threading::ITaskExecutor>& task_executor,
|
||||
const Configuration& cfg,
|
||||
bool loaded_from_cache)
|
||||
: ov::ICompiledModel(model, plugin, task_executor), // Disable default threads creation
|
||||
_cfg(cfg),
|
||||
m_cfg(cfg),
|
||||
m_model(model),
|
||||
m_loaded_from_cache(loaded_from_cache) {
|
||||
// TODO: if your plugin supports device ID (more that single instance of device can be on host machine)
|
||||
// you should select proper device based on KEY_DEVICE_ID or automatic behavior
|
||||
// In this case, _waitExecutor should also be created per device.
|
||||
// In this case, m_wait_executor should also be created per device.
|
||||
try {
|
||||
compile_model(m_model);
|
||||
} catch (const InferenceEngine::Exception& e) {
|
||||
@ -39,9 +39,9 @@ ov::template_plugin::CompiledModel::CompiledModel(const std::shared_ptr<ov::Mode
|
||||
OPENVINO_THROW("Generic exception is thrown");
|
||||
}
|
||||
}
|
||||
// ! [executable_network:ctor_cnnnetwork]
|
||||
// ! [compiled_model:ctor]
|
||||
|
||||
// ! [executable_network:map_graph]
|
||||
// ! [compiled_model:compile_model]
|
||||
// forward declaration
|
||||
void transform_model(const std::shared_ptr<ov::Model>& model);
|
||||
|
||||
@ -50,9 +50,16 @@ void ov::template_plugin::CompiledModel::compile_model(const std::shared_ptr<ov:
|
||||
transform_model(model);
|
||||
// Perform any other steps like allocation and filling backend specific memory handles and so on
|
||||
}
|
||||
// ! [executable_network:map_graph]
|
||||
// ! [compiled_model:compile_model]
|
||||
|
||||
// ! [executable_network:create_infer_request]
|
||||
// ! [compiled_model:create_sync_infer_request]
|
||||
std::shared_ptr<ov::ISyncInferRequest> ov::template_plugin::CompiledModel::create_sync_infer_request() const {
|
||||
return std::make_shared<InferRequest>(
|
||||
std::static_pointer_cast<const ov::template_plugin::CompiledModel>(shared_from_this()));
|
||||
}
|
||||
// ! [compiled_model:create_sync_infer_request]
|
||||
|
||||
// ! [compiled_model:create_infer_request]
|
||||
std::shared_ptr<ov::IAsyncInferRequest> ov::template_plugin::CompiledModel::create_infer_request() const {
|
||||
auto internal_request = create_sync_infer_request();
|
||||
auto async_infer_request = std::make_shared<AsyncInferRequest>(
|
||||
@ -63,23 +70,23 @@ std::shared_ptr<ov::IAsyncInferRequest> ov::template_plugin::CompiledModel::crea
|
||||
|
||||
return async_infer_request;
|
||||
}
|
||||
// ! [compiled_model:create_infer_request]
|
||||
|
||||
std::shared_ptr<ov::ISyncInferRequest> ov::template_plugin::CompiledModel::create_sync_infer_request() const {
|
||||
return std::make_shared<InferRequest>(
|
||||
std::static_pointer_cast<const ov::template_plugin::CompiledModel>(shared_from_this()));
|
||||
}
|
||||
// ! [executable_network:create_infer_request]
|
||||
|
||||
// ! [compiled_model:set_property]
|
||||
void ov::template_plugin::CompiledModel::set_property(const ov::AnyMap& properties) {
|
||||
OPENVINO_NOT_IMPLEMENTED;
|
||||
}
|
||||
// ! [compiled_model:set_property]
|
||||
|
||||
ov::RemoteContext ov::template_plugin::CompiledModel::get_context() const {
|
||||
OPENVINO_NOT_IMPLEMENTED;
|
||||
}
|
||||
|
||||
// ! [compiled_model:get_runtime_model]
|
||||
std::shared_ptr<const ov::Model> ov::template_plugin::CompiledModel::get_runtime_model() const {
|
||||
return m_model;
|
||||
}
|
||||
// ! [compiled_model:get_runtime_model]
|
||||
|
||||
std::shared_ptr<const ov::template_plugin::Plugin> ov::template_plugin::CompiledModel::get_template_plugin() const {
|
||||
auto plugin = get_plugin();
|
||||
@ -89,7 +96,7 @@ std::shared_ptr<const ov::template_plugin::Plugin> ov::template_plugin::Compiled
|
||||
return template_plugin;
|
||||
}
|
||||
|
||||
// ! [executable_network:get_config]
|
||||
// ! [compiled_model:get_property]
|
||||
ov::Any ov::template_plugin::CompiledModel::get_property(const std::string& name) const {
|
||||
const auto& add_ro_properties = [](const std::string& name, std::vector<ov::PropertyName>& properties) {
|
||||
properties.emplace_back(ov::PropertyName{name, ov::PropertyMutability::RO});
|
||||
@ -97,6 +104,7 @@ ov::Any ov::template_plugin::CompiledModel::get_property(const std::string& name
|
||||
const auto& default_ro_properties = []() {
|
||||
std::vector<ov::PropertyName> ro_properties{ov::model_name,
|
||||
ov::supported_properties,
|
||||
ov::execution_devices,
|
||||
ov::loaded_from_cache,
|
||||
ov::optimal_number_of_infer_requests};
|
||||
return ro_properties;
|
||||
@ -134,8 +142,11 @@ ov::Any ov::template_plugin::CompiledModel::get_property(const std::string& name
|
||||
return decltype(ov::model_name)::value_type(model_name);
|
||||
} else if (ov::loaded_from_cache == name) {
|
||||
return m_loaded_from_cache;
|
||||
} else if (ov::execution_devices == name) {
|
||||
return decltype(ov::execution_devices)::value_type{get_plugin()->get_device_name() + "." +
|
||||
std::to_string(m_cfg.device_id)};
|
||||
} else if (ov::optimal_number_of_infer_requests == name) {
|
||||
unsigned int value = _cfg.streams_executor_config._streams;
|
||||
unsigned int value = m_cfg.streams_executor_config._streams;
|
||||
return decltype(ov::optimal_number_of_infer_requests)::value_type(value);
|
||||
} else if (ov::supported_properties == name) {
|
||||
auto ro_properties = default_ro_properties();
|
||||
@ -148,13 +159,13 @@ ov::Any ov::template_plugin::CompiledModel::get_property(const std::string& name
|
||||
return decltype(ov::supported_properties)::value_type(supported_properties);
|
||||
}
|
||||
|
||||
return _cfg.Get(name);
|
||||
return m_cfg.Get(name);
|
||||
}
|
||||
// ! [executable_network:get_config]
|
||||
// ! [compiled_model:get_property]
|
||||
|
||||
// ! [executable_network:export]
|
||||
void ov::template_plugin::CompiledModel::export_model(std::ostream& modelStream) const {
|
||||
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, "ExecutableNetwork::Export");
|
||||
// ! [compiled_model:export_model]
|
||||
void ov::template_plugin::CompiledModel::export_model(std::ostream& model_stream) const {
|
||||
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, "CompiledModel::export_model");
|
||||
|
||||
std::stringstream xmlFile, binFile;
|
||||
ov::pass::Serialize serializer(xmlFile, binFile);
|
||||
@ -164,11 +175,11 @@ void ov::template_plugin::CompiledModel::export_model(std::ostream& modelStream)
|
||||
auto m_model = xmlFile.str();
|
||||
|
||||
auto dataSize = static_cast<std::uint64_t>(m_model.size());
|
||||
modelStream.write(reinterpret_cast<char*>(&dataSize), sizeof(dataSize));
|
||||
modelStream.write(m_model.c_str(), dataSize);
|
||||
model_stream.write(reinterpret_cast<char*>(&dataSize), sizeof(dataSize));
|
||||
model_stream.write(m_model.c_str(), dataSize);
|
||||
|
||||
dataSize = static_cast<std::uint64_t>(m_constants.size());
|
||||
modelStream.write(reinterpret_cast<char*>(&dataSize), sizeof(dataSize));
|
||||
modelStream.write(reinterpret_cast<char*>(&m_constants[0]), dataSize);
|
||||
model_stream.write(reinterpret_cast<char*>(&dataSize), sizeof(dataSize));
|
||||
model_stream.write(reinterpret_cast<char*>(&m_constants[0]), dataSize);
|
||||
}
|
||||
// ! [executable_network:export]
|
||||
// ! [compiled_model:export_model]
|
||||
|
||||
@ -17,10 +17,10 @@ class Plugin;
|
||||
class InferRequest;
|
||||
|
||||
/**
|
||||
* @class ExecutableNetwork
|
||||
* @brief Interface of executable network
|
||||
* @class CompiledModel
|
||||
* @brief Implementation of compiled model
|
||||
*/
|
||||
// ! [executable_network:header]
|
||||
// ! [compiled_model:header]
|
||||
class CompiledModel : public ov::ICompiledModel {
|
||||
public:
|
||||
CompiledModel(const std::shared_ptr<ov::Model>& model,
|
||||
@ -51,12 +51,12 @@ private:
|
||||
void compile_model(const std::shared_ptr<ov::Model>& model);
|
||||
std::shared_ptr<const Plugin> get_template_plugin() const;
|
||||
|
||||
mutable std::atomic<std::size_t> _requestId = {0};
|
||||
Configuration _cfg;
|
||||
mutable std::atomic<std::size_t> m_request_id = {0};
|
||||
Configuration m_cfg;
|
||||
std::shared_ptr<ov::Model> m_model;
|
||||
const bool m_loaded_from_cache;
|
||||
};
|
||||
// ! [executable_network:header]
|
||||
// ! [compiled_model:header]
|
||||
|
||||
} // namespace template_plugin
|
||||
} // namespace ov
|
||||
|
||||
@ -35,17 +35,17 @@ ov::template_plugin::InferRequest::InferRequest(const std::shared_ptr<const ov::
|
||||
: ov::ISyncInferRequest(model) {
|
||||
// TODO: allocate infer request device and host buffers if needed, fill actual list of profiling tasks
|
||||
|
||||
auto requestID = std::to_string(get_template_model()->_requestId.fetch_add(1));
|
||||
auto requestID = std::to_string(get_template_model()->m_request_id.fetch_add(1));
|
||||
|
||||
std::string name = get_template_model()->m_model->get_friendly_name() + "_Req" + requestID;
|
||||
m_profiling_task = {
|
||||
openvino::itt::handle("Template" + std::to_string(get_template_model()->_cfg.device_id) + "_" + name +
|
||||
openvino::itt::handle("Template" + std::to_string(get_template_model()->m_cfg.device_id) + "_" + name +
|
||||
"_Preprocess"),
|
||||
openvino::itt::handle("Template" + std::to_string(get_template_model()->_cfg.device_id) + "_" + name +
|
||||
openvino::itt::handle("Template" + std::to_string(get_template_model()->m_cfg.device_id) + "_" + name +
|
||||
"_Postprocess"),
|
||||
openvino::itt::handle("Template" + std::to_string(get_template_model()->_cfg.device_id) + "_" + name +
|
||||
openvino::itt::handle("Template" + std::to_string(get_template_model()->m_cfg.device_id) + "_" + name +
|
||||
"_StartPipeline"),
|
||||
openvino::itt::handle("Template" + std::to_string(get_template_model()->_cfg.device_id) + "_" + name +
|
||||
openvino::itt::handle("Template" + std::to_string(get_template_model()->m_cfg.device_id) + "_" + name +
|
||||
"_WaitPipline"),
|
||||
};
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user