Move template infer request (#15696)

* Move Template Infer Requests to new API

* Removed const_pointer_cast from plugin

* Fixed tests

* Fixed async tests

* Fixed some comments

* Added print ov::Tensor

* Fixed ONNX Frontend tests with multiple outputs to the same tensor

* Revert "Added print ov::Tensor"

This reverts commit b752f506bb.

* Fixed ov_core tests

* Fixed some tests

* Fixed batched tensors tests

* Fixed some tests

* Fixed more tests

* Fixed template plugin tests

* Fixed LP tests

* Fixed some comments

* Fixed some documentation issues

* Fixed comments

* Increase timeout because build termenated in case of common changes
This commit is contained in:
Ilya Churaev
2023-02-21 07:03:07 +04:00
committed by GitHub
parent ce3ac296ae
commit a5ec5f5476
28 changed files with 581 additions and 681 deletions
+3 -3
View File
@@ -12,7 +12,7 @@ OpenVINO Runtime Plugin API provides the base InferenceEngine::AsyncInferRequest
OpenVINO Runtime Plugin API provides the base InferenceEngine::AsyncInferRequestThreadSafeDefault class for a custom asynchronous inference request implementation:
@snippet src/template_async_infer_request.hpp async_infer_request:header
@snippet src/async_infer_request.hpp async_infer_request:header
#### Class Fields
@@ -30,7 +30,7 @@ The main goal of the `AsyncInferRequest` constructor is to define a device pipel
- `waitPipeline` is a CPU non-compute task that waits for a response from a remote device.
- `inferPostprocess` is a CPU compute task.
@snippet src/template_async_infer_request.cpp async_infer_request:ctor
@snippet src/async_infer_request.cpp async_infer_request:ctor
The stages are distributed among two task executors in the following way:
@@ -46,4 +46,4 @@ Inference request stages are also profiled using IE_PROFILING_AUTO_SCOPE, which
In the asynchronous request destructor, it is necessary to wait for a pipeline to finish. It can be done using the InferenceEngine::AsyncInferRequestThreadSafeDefault::StopAndWait method of the base class.
@snippet src/template_async_infer_request.cpp async_infer_request:dtor
@snippet src/async_infer_request.cpp async_infer_request:dtor
+8 -8
View File
@@ -12,7 +12,7 @@ Inference Engine Plugin API provides the helper InferenceEngine::IInferRequestIn
to use as a base class for a synchronous inference request implementation. Based of that, a declaration
of a synchronous request class can look as follows:
@snippet src/template_infer_request.hpp infer_request:header
@snippet src/infer_request.hpp infer_request:header
#### Class Fields
@@ -34,7 +34,7 @@ The example class has several fields:
The constructor initializes helper fields and calls methods which allocate blobs:
@snippet src/template_infer_request.cpp infer_request:ctor
@snippet src/infer_request.cpp infer_request:ctor
> **NOTE**: Call InferenceEngine::CNNNetwork::getInputsInfo and InferenceEngine::CNNNetwork::getOutputsInfo to specify both layout and precision of blobs, which you can set with InferenceEngine::InferRequest::SetBlob and get with InferenceEngine::InferRequest::GetBlob. A plugin uses these hints to determine its internal layouts and precisions for input and output blobs if needed.
@@ -42,7 +42,7 @@ The constructor initializes helper fields and calls methods which allocate blobs
Decrements a number of created inference requests:
@snippet src/template_infer_request.cpp infer_request:dtor
@snippet src/infer_request.cpp infer_request:dtor
### `InferImpl()`
@@ -50,13 +50,13 @@ Decrements a number of created inference requests:
- Checks blobs set by users
- Calls the `InferImpl` method defined in a derived class to call actual pipeline stages synchronously
@snippet src/template_infer_request.cpp infer_request:infer_impl
@snippet src/infer_request.cpp infer_request:infer_impl
#### 1. `inferPreprocess`
Below is the code of the `inferPreprocess` method to demonstrate Inference Engine common preprocessing step handling:
@snippet src/template_infer_request.cpp infer_request:infer_preprocess
@snippet src/infer_request.cpp infer_request:infer_preprocess
**Details:**
* `InferImpl` must call the InferenceEngine::IInferRequestInternal::execDataPreprocessing function, which executes common Inference Engine preprocessing step (for example, applies resize or color conversion operations) if it is set by the user. The output dimensions, layout and precision matches the input information set via InferenceEngine::CNNNetwork::getInputsInfo.
@@ -66,18 +66,18 @@ Below is the code of the `inferPreprocess` method to demonstrate Inference Engin
Executes a pipeline synchronously using `_executable` object:
@snippet src/template_infer_request.cpp infer_request:start_pipeline
@snippet src/infer_request.cpp infer_request:start_pipeline
#### 3. `inferPostprocess`
Converts output blobs if precisions of backend output blobs and blobs passed by user are different:
@snippet src/template_infer_request.cpp infer_request:infer_postprocess
@snippet src/infer_request.cpp infer_request:infer_postprocess
### `GetPerformanceCounts()`
The method sets performance counters which were measured during pipeline stages execution:
@snippet src/template_infer_request.cpp infer_request:get_performance_counts
@snippet src/infer_request.cpp infer_request:get_performance_counts
The next step in the plugin library implementation is the [Asynchronous Inference Request](@ref openvino_docs_ie_plugin_dg_async_infer_request) class.