IntenseWebs/openvino
3
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Move template infer request (#15696)

Browse Source 
* Move Template Infer Requests to new API

* Removed const_pointer_cast from plugin

* Fixed tests

* Fixed async tests

* Fixed some comments

* Added print ov::Tensor

* Fixed ONNX Frontend tests with multiple outputs to the same tensor

* Revert "Added print ov::Tensor"

This reverts commit b752f506bb.

* Fixed ov_core tests

* Fixed some tests

* Fixed batched tensors tests

* Fixed some tests

* Fixed more tests

* Fixed template plugin tests

* Fixed LP tests

* Fixed some comments

* Fixed some documentation issues

* Fixed comments

* Increase timeout because build termenated in case of common changes
This commit is contained in:
Ilya Churaev authored and GitHub committed 2023-02-21 07:03:07 +04:00
1 parent ce3ac296ae
commit a5ec5f5476
28 files changed
+581 -681
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.ci/azure/linux.yml
+1 -1
View File
@@ -71,7 +71,7 @@ jobs:
maxParallel: '2'
# About 150% of total time
timeoutInMinutes: '120'
timeoutInMinutes: '180'
pool:
name: LIN_VMSS_VENV_F16S_U20_WU2
docs/IE_PLUGIN_DG/AsyncInferRequest.md
+3 -3
View File
@@ -12,7 +12,7 @@ OpenVINO Runtime Plugin API provides the base InferenceEngine::AsyncInferRequest
OpenVINO Runtime Plugin API provides the base InferenceEngine::AsyncInferRequestThreadSafeDefault class for a custom asynchronous inference request implementation:
@snippet src/template_async_infer_request.hpp async_infer_request:header
@snippet src/async_infer_request.hpp async_infer_request:header
#### Class Fields
@@ -30,7 +30,7 @@ The main goal of the `AsyncInferRequest` constructor is to define a device pipel
- `waitPipeline` is a CPU non-compute task that waits for a response from a remote device.
- `inferPostprocess` is a CPU compute task.
@snippet src/template_async_infer_request.cpp async_infer_request:ctor
@snippet src/async_infer_request.cpp async_infer_request:ctor
The stages are distributed among two task executors in the following way:
@@ -46,4 +46,4 @@ Inference request stages are also profiled using IE_PROFILING_AUTO_SCOPE, which
In the asynchronous request destructor, it is necessary to wait for a pipeline to finish. It can be done using the InferenceEngine::AsyncInferRequestThreadSafeDefault::StopAndWait method of the base class.
@snippet src/template_async_infer_request.cpp async_infer_request:dtor
@snippet src/async_infer_request.cpp async_infer_request:dtor
docs/IE_PLUGIN_DG/InferRequest.md
+8 -8
View File
@@ -12,7 +12,7 @@ Inference Engine Plugin API provides the helper InferenceEngine::IInferRequestIn
to use as a base class for a synchronous inference request implementation. Based of that, a declaration
of a synchronous request class can look as follows:
@snippet src/template_infer_request.hpp infer_request:header
@snippet src/infer_request.hpp infer_request:header
#### Class Fields
@@ -34,7 +34,7 @@ The example class has several fields:
The constructor initializes helper fields and calls methods which allocate blobs:
@snippet src/template_infer_request.cpp infer_request:ctor
@snippet src/infer_request.cpp infer_request:ctor
> **NOTE**: Call InferenceEngine::CNNNetwork::getInputsInfo and InferenceEngine::CNNNetwork::getOutputsInfo to specify both layout and precision of blobs, which you can set with InferenceEngine::InferRequest::SetBlob and get with InferenceEngine::InferRequest::GetBlob. A plugin uses these hints to determine its internal layouts and precisions for input and output blobs if needed.
@@ -42,7 +42,7 @@ The constructor initializes helper fields and calls methods which allocate blobs
Decrements a number of created inference requests:
@snippet src/template_infer_request.cpp infer_request:dtor
@snippet src/infer_request.cpp infer_request:dtor
### `InferImpl()`
@@ -50,13 +50,13 @@ Decrements a number of created inference requests:
- Checks blobs set by users
- Calls the `InferImpl` method defined in a derived class to call actual pipeline stages synchronously
@snippet src/template_infer_request.cpp infer_request:infer_impl
@snippet src/infer_request.cpp infer_request:infer_impl
#### 1. `inferPreprocess`
Below is the code of the `inferPreprocess` method to demonstrate Inference Engine common preprocessing step handling:
@snippet src/template_infer_request.cpp infer_request:infer_preprocess
@snippet src/infer_request.cpp infer_request:infer_preprocess
**Details:**
* `InferImpl` must call the InferenceEngine::IInferRequestInternal::execDataPreprocessing function, which executes common Inference Engine preprocessing step (for example, applies resize or color conversion operations) if it is set by the user. The output dimensions, layout and precision matches the input information set via InferenceEngine::CNNNetwork::getInputsInfo.
@@ -66,18 +66,18 @@ Below is the code of the `inferPreprocess` method to demonstrate Inference Engin
Executes a pipeline synchronously using `_executable` object:
@snippet src/template_infer_request.cpp infer_request:start_pipeline
@snippet src/infer_request.cpp infer_request:start_pipeline
#### 3. `inferPostprocess`
Converts output blobs if precisions of backend output blobs and blobs passed by user are different:
@snippet src/template_infer_request.cpp infer_request:infer_postprocess
@snippet src/infer_request.cpp infer_request:infer_postprocess
### `GetPerformanceCounts()`
The method sets performance counters which were measured during pipeline stages execution:
@snippet src/template_infer_request.cpp infer_request:get_performance_counts
@snippet src/infer_request.cpp infer_request:get_performance_counts
The next step in the plugin library implementation is the [Asynchronous Inference Request](@ref openvino_docs_ie_plugin_dg_async_infer_request) class.
src/core/include/openvino/runtime/tensor.hpp
+9
View File
@@ -29,6 +29,7 @@ class CoreImpl;
class InferRequest;
class RemoteContext;
class VariableState;
class ISyncInferRequest;
class IInferRequestInternalWrapper;
/**
@@ -54,6 +55,7 @@ protected:
friend class ov::InferRequest;
friend class ov::RemoteContext;
friend class ov::VariableState;
friend class ov::ISyncInferRequest;
friend class ov::IInferRequestInternalWrapper;
friend class InferenceEngine::IAsyncInferRequestWrapper;
@@ -137,6 +139,13 @@ public:
*/
Shape get_shape() const;
/**
* @brief Reports whether the tensor is continuous or not
*
* @return true if blob is continuous
*/
bool is_continuous() const;
/**
* @brief Returns the total number of elements (a product of all the dims or 1 for scalar)
* @return The total number of elements
src/core/src/runtime/ov_tensor.cpp
+23 -2
View File
@@ -41,7 +41,7 @@ Tensor::Tensor(const element::Type element_type, const Shape& shape, const Alloc
auto blob_allocator =
(allocator_impl != nullptr) ? allocator_impl->_impl : std::make_shared<ie::BlobAllocator>(allocator._impl);
_impl = make_blob_with_precision(
{ie::details::convertPrecision(element_type), shape, ie::TensorDesc::getLayoutByRank(shape.size())},
{ie::details::convertPrecision(element_type), shape, ie::TensorDesc::getLayoutByDims(shape)},
blob_allocator);
_impl->allocate();
}
@@ -110,7 +110,7 @@ void Tensor::set_shape(const ov::Shape& shape) {
}
Shape Tensor::get_shape() const {
OV_TENSOR_STATEMENT({ return _impl->getTensorDesc().getDims(); });
OV_TENSOR_STATEMENT({ return _impl->getTensorDesc().getBlockingDesc().getBlockDims(); });
}
Strides Tensor::get_strides() const {
@@ -173,4 +173,25 @@ Tensor::operator bool() const noexcept {
return (!!_impl);
}
bool Tensor::is_continuous() const {
if (get_element_type().bitwidth() < 8)
// OpenVINO doesn't support strides for lp types
return true;
const auto& shape = get_shape();
const auto& type = get_element_type();
std::vector<size_t> strides(shape.size());
if (!shape.empty()) {
strides[shape.size() - 1] = 1;
}
auto size = shape.size();
for (size_t i = 1; i < size; i++) {
strides[size - i - 1] = strides[size - i] * shape[size - i];
}
ov::Strides byte_strides(strides.size());
for (size_t i = 0; i < strides.size(); ++i)
byte_strides[i] = strides[i] * type.size();
return byte_strides == get_strides();
}
} // namespace ov
src/inference/dev_api/openvino/runtime/iasync_infer_request.hpp
+1 -1
View File
@@ -136,7 +136,7 @@ public:
*
* @return Pointer to the compiled model
*/
const std::shared_ptr<ov::ICompiledModel>& get_compiled_model() const override;
const std::shared_ptr<const ov::ICompiledModel>& get_compiled_model() const override;
/**
* @brief Gets inputs for infer request
src/inference/dev_api/openvino/runtime/iinfer_request.hpp
+1 -1
View File
@@ -94,7 +94,7 @@ public:
*
* @return Pointer to the compiled model
*/
virtual const std::shared_ptr<ov::ICompiledModel>& get_compiled_model() const = 0;
virtual const std::shared_ptr<const ov::ICompiledModel>& get_compiled_model() const = 0;
/**
* @brief Gets inputs for infer request
src/inference/dev_api/openvino/runtime/isync_infer_request.hpp
+24 -12
View File
@@ -11,13 +11,14 @@
#include <exception>
#include <memory>
#include <openvino/runtime/tensor.hpp>
#include <unordered_map>
#include <vector>
#include "openvino/core/descriptor/tensor.hpp"
#include "openvino/runtime/common.hpp"
#include "openvino/runtime/iinfer_request.hpp"
#include "openvino/runtime/profiling_info.hpp"
#include "openvino/runtime/tensor.hpp"
namespace ov {
@@ -31,7 +32,7 @@ public:
*
* @param compiled_model pointer to compiled model
*/
ISyncInferRequest(const std::shared_ptr<ov::ICompiledModel>& compiled_model);
ISyncInferRequest(const std::shared_ptr<const ov::ICompiledModel>& compiled_model);
/**
* @brief Gets an input/output tensor for inference.
@@ -102,7 +103,7 @@ public:
*
* @return Pointer to the compiled model
*/
const std::shared_ptr<ov::ICompiledModel>& get_compiled_model() const override;
const std::shared_ptr<const ov::ICompiledModel>& get_compiled_model() const override;
protected:
struct FoundPort {
@@ -120,11 +121,6 @@ protected:
}
};
/**
* @brief Finds input or output port
* @return structure which contains index of Input/Output or report that port wasn't found
*/
FoundPort find_port(const ov::Output<const ov::Node>& port) const;
/**
* @brief Converts batched tensors to tensor
*/
@@ -142,12 +138,28 @@ protected:
*/
void check_tensors() const override;
std::vector<ov::Tensor> m_input_tensors;
std::vector<ov::Tensor> m_output_tensors;
std::unordered_map<size_t, std::vector<ov::Tensor>> m_batched_tensors;
/**
* @brief Allocate tensor with using custom allocator
*
* @param port input/output port for tensor
* @param allocate_callback function which allocates the tensor
*/
void allocate_tensor(const ov::Output<const ov::Node>& port,
const std::function<void(ov::Tensor& tensor)>& allocate_callback);
std::unordered_map<std::shared_ptr<ov::descriptor::Tensor>, std::vector<ov::Tensor>> m_batched_tensors;
private:
std::shared_ptr<ov::ICompiledModel> m_compiled_model;
std::shared_ptr<const ov::ICompiledModel> m_compiled_model;
// Mutable to return reference to ov::Tensor
mutable std::unordered_map<std::shared_ptr<ov::descriptor::Tensor>, ov::Tensor> m_tensors;
ov::Tensor& get_ref_tensor(const ov::Output<const ov::Node>& port) const;
/**
* @brief Finds input or output port
* @return structure which contains index of Input/Output or report that port wasn't found
*/
FoundPort find_port(const ov::Output<const ov::Node>& port) const;
};
}; // namespace ov
src/inference/src/dev/converter_utils.cpp
+19 -7
View File
@@ -467,15 +467,27 @@ public:
}
void SetBlob(const std::string& name, const InferenceEngine::Blob::Ptr& data) override {
m_request->set_tensor(find_port(name), ov::Tensor{data, {}});
try {
m_request->set_tensor(find_port(name), ov::Tensor{data, {}});
} catch (const ov::Exception& ex) {
const std::string what = ex.what();
if (what.find("Failed to set tensor") != std::string::npos) {
IE_THROW(ParameterMismatch) << what;
}
IE_THROW(GeneralError) << what;
}
}
void SetBlobs(const std::string& name, const std::vector<InferenceEngine::Blob::Ptr>& blobs) override {
std::vector<ov::Tensor> tensors;
for (const auto& blob : blobs) {
tensors.emplace_back(ov::Tensor{blob, {}});
try {
std::vector<ov::Tensor> tensors;
for (const auto& blob : blobs) {
tensors.emplace_back(ov::Tensor{blob, {}});
}
m_request->set_tensors(find_port(name), tensors);
} catch (const ov::Exception& ex) {
IE_THROW(GeneralError) << ex.what();
}
m_request->set_tensors(find_port(name), tensors);
}
InferenceEngine::Blob::Ptr GetBlob(const std::string& name) override {
@@ -678,7 +690,7 @@ public:
m_request->SetCallback(std::move(callback));
}
const std::shared_ptr<ov::ICompiledModel>& get_compiled_model() const override {
const std::shared_ptr<const ov::ICompiledModel>& get_compiled_model() const override {
if (!m_compiled_model) {
std::lock_guard<std::mutex> lock(m_mutex);
if (!m_compiled_model) {
@@ -700,7 +712,7 @@ public:
private:
std::shared_ptr<InferenceEngine::IInferRequestInternal> m_request;
mutable std::shared_ptr<ov::ICompiledModel> m_compiled_model;
mutable std::shared_ptr<const ov::ICompiledModel> m_compiled_model;
mutable std::mutex m_mutex;
};
src/inference/src/dev/converter_utils.hpp
+3 -5
View File
@@ -24,16 +24,14 @@ std::shared_ptr<const ov::Model> convert_model(const InferenceEngine::CNNNetwork
std::shared_ptr<::InferenceEngine::IInferencePlugin> convert_plugin(const std::shared_ptr<::ov::IPlugin>& plugin);
std::shared_ptr<::ov::IPlugin> convert_plugin(const std::shared_ptr<::InferenceEngine::IInferencePlugin>& plugin);
// TODO: remove export after changes in template plugin
OPENVINO_RUNTIME_API std::shared_ptr<::InferenceEngine::IExecutableNetworkInternal> convert_compiled_model(
std::shared_ptr<::InferenceEngine::IExecutableNetworkInternal> convert_compiled_model(
const std::shared_ptr<::ov::ICompiledModel>& model);
std::shared_ptr<::ov::ICompiledModel> convert_compiled_model(
const std::shared_ptr<::InferenceEngine::IExecutableNetworkInternal>& model);
// TODO: remove export after changes in template plugin
OPENVINO_RUNTIME_API std::shared_ptr<::InferenceEngine::IInferRequestInternal> convert_infer_request(
std::shared_ptr<::InferenceEngine::IInferRequestInternal> convert_infer_request(
const std::shared_ptr<::ov::IAsyncInferRequest>& request);
OPENVINO_RUNTIME_API std::shared_ptr<::ov::IAsyncInferRequest> convert_infer_request(
std::shared_ptr<::ov::IAsyncInferRequest> convert_infer_request(
const std::shared_ptr<::InferenceEngine::IInferRequestInternal>& request);
} // namespace legacy_convert
src/inference/src/dev/iasync_infer_request.cpp
+7 -2
View File
@@ -62,6 +62,7 @@ void ov::IAsyncInferRequest::wait() {
}
future.wait();
future.get();
}
bool ov::IAsyncInferRequest::wait_for(const std::chrono::milliseconds& timeout) {
@@ -247,14 +248,18 @@ void ov::IAsyncInferRequest::stop_and_wait() {
}
void ov::IAsyncInferRequest::infer() {
m_sync_request->infer();
DisableCallbackGuard disableCallbackGuard{this};
infer_impl([&] {
infer_thread_unsafe();
});
wait();
}
void ov::IAsyncInferRequest::check_tensors() const {
m_sync_request->check_tensors();
}
const std::shared_ptr<ov::ICompiledModel>& ov::IAsyncInferRequest::get_compiled_model() const {
const std::shared_ptr<const ov::ICompiledModel>& ov::IAsyncInferRequest::get_compiled_model() const {
return m_sync_request->get_compiled_model();
}
src/inference/src/dev/isync_infer_request.cpp
+80 -41
View File
@@ -4,6 +4,8 @@
#include "openvino/runtime/isync_infer_request.hpp"
#include <unordered_map>
#include "cpp_interfaces/plugin_itt.hpp"
#include "openvino/core/except.hpp"
#include "openvino/core/layout.hpp"
@@ -15,7 +17,6 @@
#include "openvino/runtime/tensor.hpp"
namespace {
void check_batched_tensors(const ov::Output<const ov::Node>& input, const std::vector<ov::Tensor>& tensors) {
OPENVINO_ASSERT(!tensors.empty(), "set_input_tensors/set_tensors can't be called with empty tensors");
OPENVINO_ASSERT(
@@ -76,6 +77,7 @@ void check_batched_tensors(const ov::Output<const ov::Node>& input, const std::v
element_type,
" and shape ",
batched_shape);
OPENVINO_ASSERT(item.is_continuous(), "Strides for batched tensors should be default.");
}
}
@@ -83,8 +85,19 @@ void check_batched_tensors(const ov::Output<const ov::Node>& input, const std::v
ov::IInferRequest::~IInferRequest() = default;
ov::ISyncInferRequest::ISyncInferRequest(const std::shared_ptr<ov::ICompiledModel>& compiled_model)
: m_compiled_model(compiled_model) {}
ov::ISyncInferRequest::ISyncInferRequest(const std::shared_ptr<const ov::ICompiledModel>& compiled_model)
: m_compiled_model(compiled_model) {
OPENVINO_ASSERT(m_compiled_model);
// Create map of empty tensors
for (const auto& input : get_inputs()) {
if (m_tensors.find(input.get_tensor_ptr()) == m_tensors.end())
m_tensors[input.get_tensor_ptr()] = ov::Tensor();
}
for (const auto& output : get_outputs()) {
if (m_tensors.find(output.get_tensor_ptr()) == m_tensors.end())
m_tensors[output.get_tensor_ptr()] = ov::Tensor();
}
}
const std::vector<ov::Output<const ov::Node>>& ov::ISyncInferRequest::get_inputs() const {
return m_compiled_model->inputs();
@@ -92,15 +105,24 @@ const std::vector<ov::Output<const ov::Node>>& ov::ISyncInferRequest::get_inputs
const std::vector<ov::Output<const ov::Node>>& ov::ISyncInferRequest::get_outputs() const {
return m_compiled_model->outputs();
}
const std::shared_ptr<ov::ICompiledModel>& ov::ISyncInferRequest::get_compiled_model() const {
const std::shared_ptr<const ov::ICompiledModel>& ov::ISyncInferRequest::get_compiled_model() const {
return m_compiled_model;
}
ov::ISyncInferRequest::FoundPort ov::ISyncInferRequest::find_port(const ov::Output<const ov::Node>& port) const {
auto check_nodes = [](const ov::Node* node1, const ov::Node* node2) {
return node1 == node2 ||
(node1->get_friendly_name() == node2->get_friendly_name() &&
node1->get_type_info() == node2->get_type_info() &&
node1->outputs().size() == node2->outputs().size() && node1->inputs().size() == node2->inputs().size());
};
ov::ISyncInferRequest::FoundPort::Type type = ov::ISyncInferRequest::FoundPort::Type::INPUT;
for (const auto& ports : {get_inputs(), get_outputs()}) {
for (size_t i = 0; i < ports.size(); i++) {
if (ports[i] == port) {
// TODO: Fix port comparison
// if (ports[i] == port) {
if (ports[i].get_index() == port.get_index() && ports[i].get_names() == port.get_names() &&
check_nodes(ports[i].get_node(), port.get_node())) {
return {i, type};
}
}
@@ -110,6 +132,7 @@ ov::ISyncInferRequest::FoundPort ov::ISyncInferRequest::find_port(const ov::Outp
}
void ov::ISyncInferRequest::convert_batched_tensors() {
std::unordered_map<std::shared_ptr<ov::descriptor::Tensor>, ov::Tensor> prepared_tensors;
for (const auto& item : m_batched_tensors) {
auto tmp_shape = item.second.at(0).get_shape();
auto tmp_et = item.second.at(0).get_element_type();
@@ -128,61 +151,60 @@ void ov::ISyncInferRequest::convert_batched_tensors() {
} else {
input_tensor = ov::Tensor(tmp_et, tmp_shape);
}
auto ptr = input_tensor.data<uint8_t>();
auto ptr = static_cast<uint8_t*>(input_tensor.data());
// Perform memory copy
ov::parallel_for(input_tensor.get_size(), [&](size_t i) {
ov::parallel_for(item.second.size(), [&](size_t i) {
const auto& tensor = item.second.at(i);
memcpy(ptr + i * tensor.get_byte_size(), tensor.data<uint8_t>(), tensor.get_byte_size());
memcpy(ptr + i * tensor.get_byte_size(), static_cast<uint8_t*>(tensor.data()), tensor.get_byte_size());
});
set_tensor(get_inputs()[item.first], input_tensor);
prepared_tensors[item.first] = input_tensor;
}
for (const auto& item : prepared_tensors) {
if (m_tensors.count(item.first))
m_tensors[item.first] = item.second;
}
}
ov::Tensor& ov::ISyncInferRequest::get_ref_tensor(const ov::Output<const ov::Node>& port) const {
auto found_port = find_port(port);
OPENVINO_ASSERT(found_port.found(), "Cannot find tensor for port ", port);
auto ports = found_port.is_input() ? get_inputs() : get_outputs();
auto it = m_tensors.find(ports.at(found_port.idx).get_tensor_ptr());
OPENVINO_ASSERT(it != m_tensors.end(), "Cannot find tensor for port: ", port);
return it->second;
}
ov::Tensor ov::ISyncInferRequest::get_tensor(const ov::Output<const ov::Node>& port) const {
OV_ITT_SCOPED_TASK(InferenceEngine::itt::domains::Plugin, "get_tensor");
auto found_port = find_port(port);
OPENVINO_ASSERT(!found_port.found(), "Cannot find tensor for port ", port);
if (found_port.is_input()) {
auto input = m_compiled_model->inputs().at(found_port.idx);
// TODO: Support dynamic inputs
// if (input.get_partial_shape().is_dynamic())
return m_input_tensors.at(found_port.idx);
}
auto output = m_compiled_model->outputs().at(found_port.idx);
// TODO: Support dynamic inputs
// if (output.get_partial_shape().is_dynamic())
return m_output_tensors.at(found_port.idx);
return get_ref_tensor(port);
}
void ov::ISyncInferRequest::set_tensor(const ov::Output<const ov::Node>& port, const ov::Tensor& tensor) {
OV_ITT_SCOPED_TASK(InferenceEngine::itt::domains::Plugin, "set_tensor");
auto found_port = find_port(port);
OPENVINO_ASSERT(!found_port.found(), "Cannot find tensor for port ", port);
OPENVINO_ASSERT(
port.get_element_type() == tensor.get_element_type(),
"Failed to set output tensor, the tensor element type is not corresponding with output element type");
OPENVINO_ASSERT(port.get_partial_shape().is_dynamic() || tensor.get_shape() == port.get_shape(),
"Input tensor size is not equal with model input size (",
tensor.get_shape(),
" != ",
port.get_shape(),
").");
OPENVINO_ASSERT(found_port.found(), "Cannot find tensor for port ", port);
try {
check_tensor(port, tensor);
} catch (const ov::Exception& ex) {
OPENVINO_UNREACHABLE("Failed to set tensor. ", ex.what());
}
if (found_port.is_input()) {
m_input_tensors.at(found_port.idx) = tensor;
m_batched_tensors.erase(found_port.idx);
m_tensors.at(get_inputs().at(found_port.idx).get_tensor_ptr()) = tensor;
m_batched_tensors.erase(get_inputs().at(found_port.idx).get_tensor_ptr());
} else {
m_output_tensors.at(found_port.idx) = tensor;
m_tensors.at(get_outputs().at(found_port.idx).get_tensor_ptr()) = tensor;
}
}
std::vector<ov::Tensor> ov::ISyncInferRequest::get_tensors(const ov::Output<const ov::Node>& port) const {
OV_ITT_SCOPED_TASK(InferenceEngine::itt::domains::Plugin, "get_tensors");
auto found_port = find_port(port);
OPENVINO_ASSERT(!found_port.found() && found_port.is_input(), "Cannot find input tensors for port ", port);
if (m_batched_tensors.count(found_port.idx))
return m_batched_tensors.at(found_port.idx);
OPENVINO_ASSERT(found_port.found(), "Cannot find input tensors for port ", port);
if (found_port.is_input() && m_batched_tensors.count(get_inputs().at(found_port.idx).get_tensor_ptr()))
return m_batched_tensors.at(get_inputs().at(found_port.idx).get_tensor_ptr());
return {};
}
@@ -190,7 +212,7 @@ void ov::ISyncInferRequest::set_tensors(const ov::Output<const ov::Node>& port,
const std::vector<ov::Tensor>& tensors) {
OV_ITT_SCOPED_TASK(InferenceEngine::itt::domains::Plugin, "set_tensors");
auto found_port = find_port(port);
OPENVINO_ASSERT(!found_port.found() && found_port.is_input(), "Cannot find input tensors for port ", port);
OPENVINO_ASSERT(found_port.found() && found_port.is_input(), "Cannot find input tensors for port ", port);
if (tensors.size() == 1) {
set_tensor(port, tensors[0]);
return;
@@ -213,6 +235,11 @@ void ov::ISyncInferRequest::check_tensor(const ov::Output<const ov::Node>& port,
bool is_input = ov::op::util::is_parameter(port.get_node());
std::string tensor_type = is_input ? "input" : "output";
OPENVINO_ASSERT(port.get_element_type() == tensor.get_element_type(),
"The tensor element type is not corresponding with output element type (",
tensor.get_element_type(),
" != ",
port.get_element_type());
bool is_dynamic = port.get_partial_shape().is_dynamic();
OPENVINO_ASSERT(is_dynamic || port.get_shape() == tensor.get_shape(),
"The ",
@@ -224,15 +251,27 @@ void ov::ISyncInferRequest::check_tensor(const ov::Output<const ov::Node>& port,
" expecting ",
port.get_shape(),
".");
OPENVINO_ASSERT(tensor.data() != nullptr, "Tensor data equal nullptr!");
// FIXME: SyncInferRequest is a friend only to check that blob is correct
OPENVINO_ASSERT(ov::shape_size(tensor._impl->getTensorDesc().getDims()) ==
ov::shape_size(tensor._impl->getTensorDesc().getBlockingDesc().getBlockDims()),
"Tensor is corrupted!");
}
void ov::ISyncInferRequest::allocate_tensor(const ov::Output<const ov::Node>& port,
const std::function<void(ov::Tensor& tensor)>& allocate_callback) {
auto& tensor = get_ref_tensor(port);
allocate_callback(tensor);
}
void ov::ISyncInferRequest::check_tensors() const {
const auto& inputs = m_compiled_model->inputs();
for (size_t i = 0; i < inputs.size(); i++) {
check_tensor(inputs[i], m_input_tensors[i]);
check_tensor(inputs[i], m_tensors.at(inputs[i].get_tensor_ptr()));
}
const auto& outputs = m_compiled_model->outputs();
for (size_t i = 0; i < outputs.size(); i++) {
check_tensor(outputs[i], m_output_tensors[i]);
check_tensor(outputs[i], m_tensors.at(outputs[i].get_tensor_ptr()));
}
}
src/inference/src/infer_request.cpp
+1 -1
View File
@@ -273,7 +273,7 @@ std::vector<VariableState> InferRequest::query_state() {
}
CompiledModel InferRequest::get_compiled_model() {
OV_INFER_REQ_CALL_STATEMENT(return {_impl->get_compiled_model(), _so});
OV_INFER_REQ_CALL_STATEMENT(return {std::const_pointer_cast<ICompiledModel>(_impl->get_compiled_model()), _so});
}
bool InferRequest::operator!() const noexcept {
src/plugins/template/backend/int_executable.cpp
+2 -1
View File
@@ -86,7 +86,8 @@ bool runtime::interpreter::INTExecutable::call(const vector<shared_ptr<runtime::
// map function outputs -> HostTensor
for (size_t output_count = 0; output_count < get_results().size(); ++output_count) {
auto output = get_results()[output_count]->output(0).get_tensor_ptr();
results_map.emplace(output, results_map.size());
if (!results_map.count(output))
results_map.emplace(output, output_count);
}
EvaluationContext eval_context;
src/plugins/template/src/CMakeLists.txt
-1
View File
@@ -26,7 +26,6 @@ ov_mark_target_as_cc(${TARGET_NAME})
target_include_directories(${TARGET_NAME} PRIVATE
"${CMAKE_CURRENT_SOURCE_DIR}"
"${OpenVINO_SOURCE_DIR}/src/inference/src/dev" # TODO: remove after migration to new infer request
"${TEMPLATE_PLUGIN_SOURCE_DIR}/include")
# link common Inference Engine libraries
src/plugins/template/src/async_infer_request.cpp
+52
View File
@@ -0,0 +1,52 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include "async_infer_request.hpp"
#include "infer_request.hpp"
#include "openvino/runtime/iinfer_request.hpp"
#include "template_itt.hpp"
// ! [async_infer_request:ctor]
TemplatePlugin::AsyncInferRequest::AsyncInferRequest(const std::shared_ptr<TemplatePlugin::InferRequest>& request,
const InferenceEngine::ITaskExecutor::Ptr& task_executor,
const InferenceEngine::ITaskExecutor::Ptr& wait_executor,
const InferenceEngine::ITaskExecutor::Ptr& callback_executor)
: ov::IAsyncInferRequest(request, task_executor, callback_executor),
m_wait_executor(wait_executor) {
// In current implementation we have CPU only tasks and no needs in 2 executors
// So, by default single stage pipeline is created.
// This stage executes InferRequest::infer() using cpuTaskExecutor.
// But if remote asynchronous device is used the pipeline can by splitted tasks that are executed by cpuTaskExecutor
// and waiting tasks. Waiting tasks can lock execution thread so they use separate threads from other executor.
constexpr const auto remoteDevice = false;
if (remoteDevice) {
m_pipeline = {{task_executor,
[this, request] {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin,
"TemplatePlugin::AsyncInferRequest::infer_preprocess_and_start_pipeline");
request->infer_preprocess();
request->start_pipeline();
}},
{m_wait_executor,
[this, request] {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin,
"TemplatePlugin::AsyncInferRequest::wait_pipeline");
request->wait_pipeline();
}},
{task_executor, [this, request] {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin,
"TemplatePlugin::AsyncInferRequest::infer_postprocess");
request->infer_postprocess();
}}};
}
}
// ! [async_infer_request:ctor]
// ! [async_infer_request:dtor]
TemplatePlugin::AsyncInferRequest::~AsyncInferRequest() {
ov::IAsyncInferRequest::stop_and_wait();
}
// ! [async_infer_request:dtor]
src/plugins/template/src/async_infer_request.hpp
+30
View File
@@ -0,0 +1,30 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#pragma once
#include <memory>
#include "infer_request.hpp"
#include "openvino/runtime/iasync_infer_request.hpp"
#include "openvino/runtime/iinfer_request.hpp"
namespace TemplatePlugin {
// ! [async_infer_request:header]
class AsyncInferRequest : public ov::IAsyncInferRequest {
public:
AsyncInferRequest(const std::shared_ptr<InferRequest>& request,
const InferenceEngine::ITaskExecutor::Ptr& task_executor,
const InferenceEngine::ITaskExecutor::Ptr& wait_executor,
const InferenceEngine::ITaskExecutor::Ptr& callback_executor);
~AsyncInferRequest();
private:
InferenceEngine::ITaskExecutor::Ptr m_wait_executor;
};
// ! [async_infer_request:header]
} // namespace TemplatePlugin
src/plugins/template/src/compiled_model.cpp
+11 -54
View File
@@ -6,15 +6,12 @@
#include <memory>
#include "converter_utils.hpp"
#include "async_infer_request.hpp"
#include "ie_ngraph_utils.hpp"
#include "ie_plugin_config.hpp"
#include "openvino/core/except.hpp"
#include "openvino/runtime/iinfer_request.hpp"
#include "openvino/runtime/isync_infer_request.hpp"
#include "plugin.hpp"
#include "template/config.hpp"
#include "template_async_infer_request.hpp"
#include "template_itt.hpp"
#include "transformations/utils/utils.hpp"
using namespace TemplatePlugin;
@@ -109,65 +106,25 @@ void transform_model(const std::shared_ptr<ov::Model>& model);
void TemplatePlugin::CompiledModel::compile_model(const std::shared_ptr<ov::Model>& model) {
// apply plugins transformations
transform_model(model);
// Generate backend specific blob mappings. For example Inference Engine uses not ngraph::Result nodes friendly name
// as inference request output names but the name of the layer before.
// TODO: Remove it
size_t idx = 0;
for (auto&& result : model->get_results()) {
const auto& input = result->input_value(0);
auto name = ov::op::util::get_ie_output_name(input);
if (_outputIndex.emplace(name, idx).second)
idx++;
}
for (auto&& parameter : model->get_parameters()) {
_inputIndex.emplace(parameter->get_friendly_name(), model->get_parameter_index(parameter));
}
// Perform any other steps like allocation and filling backend specific memory handles and so on
}
// ! [executable_network:map_graph]
// ! [executable_network:create_infer_request]
std::shared_ptr<ov::IAsyncInferRequest> TemplatePlugin::CompiledModel::create_infer_request() const {
// auto internal_request = create_sync_infer_request();
std::vector<std::shared_ptr<const ov::Node>> _inputs, _outputs;
for (const auto& output : m_model->inputs()) {
_inputs.emplace_back(output.get_node_shared_ptr());
}
for (const auto& output : outputs()) {
_outputs.emplace_back(output.get_node_shared_ptr());
}
auto internal_request = create_sync_infer_request();
auto async_infer_request =
std::make_shared<AsyncInferRequest>(std::static_pointer_cast<TemplatePlugin::InferRequest>(internal_request),
get_task_executor(),
get_template_plugin()->_waitExecutor,
get_callback_executor());
auto internal_request = std::make_shared<TemplateInferRequest>(
_inputs,
_outputs,
std::static_pointer_cast<const TemplatePlugin::CompiledModel>(shared_from_this()));
auto async_infer_request = std::make_shared<TemplateAsyncInferRequest>(
std::static_pointer_cast<TemplatePlugin::TemplateInferRequest>(internal_request),
get_task_executor(),
get_template_plugin()->_waitExecutor,
get_callback_executor());
async_infer_request->setPointerToExecutableNetworkInternal(
ov::legacy_convert::convert_compiled_model(std::const_pointer_cast<ov::ICompiledModel>(shared_from_this())));
return ov::legacy_convert::convert_infer_request(async_infer_request);
return async_infer_request;
}
std::shared_ptr<ov::ISyncInferRequest> TemplatePlugin::CompiledModel::create_sync_infer_request() const {
OPENVINO_NOT_IMPLEMENTED;
// std::vector<std::shared_ptr<const ov::Node>> _inputs, _outputs;
// for (const auto& output : m_model->inputs()) {
// _inputs.emplace_back(output.get_node_shared_ptr());
// }
// for (const auto& output : outputs()) {
// _outputs.emplace_back(output.get_node_shared_ptr());
// }
//
// return std::make_shared<TemplateInferRequest>(
// _inputs,
// _outputs,
// std::static_pointer_cast<const TemplatePlugin::CompiledModel>(shared_from_this()));
return std::make_shared<InferRequest>(
std::static_pointer_cast<const TemplatePlugin::CompiledModel>(shared_from_this()));
}
// ! [executable_network:create_infer_request]
src/plugins/template/src/compiled_model.hpp
+3 -5
View File
@@ -9,11 +9,11 @@
#include "openvino/runtime/isync_infer_request.hpp"
#include "openvino/runtime/tensor.hpp"
#include "template_config.hpp"
#include "template_infer_request.hpp"
namespace TemplatePlugin {
class Plugin;
class InferRequest;
/**
* @class ExecutableNetwork
@@ -43,17 +43,15 @@ protected:
std::shared_ptr<ov::ISyncInferRequest> create_sync_infer_request() const override;
private:
friend class TemplateInferRequest;
friend class InferRequest;
friend class Plugin;
void compile_model(const std::shared_ptr<ov::Model>& model);
std::shared_ptr<const Plugin> get_template_plugin() const;
std::atomic<std::size_t> _requestId = {0};
mutable std::atomic<std::size_t> _requestId = {0};
Configuration _cfg;
std::shared_ptr<ov::Model> m_model;
std::map<std::string, std::size_t> _inputIndex;
std::map<std::string, std::size_t> _outputIndex;
};
// ! [executable_network:header]
src/plugins/template/src/infer_request.cpp
+237
View File
@@ -0,0 +1,237 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include "infer_request.hpp"
#include <algorithm>
#include <map>
#include <memory>
#include <string>
#include <utility>
#include "ngraph/runtime/host_tensor.hpp"
#include "openvino/core/except.hpp"
#include "openvino/runtime/profiling_info.hpp"
#include "plugin.hpp"
#include "template_itt.hpp"
using Time = std::chrono::high_resolution_clock;
namespace {
void allocate_tensor_impl(ov::Tensor& tensor, const ov::element::Type& element_type, const ov::Shape& shape) {
if (!tensor || tensor.get_element_type() != element_type) {
tensor = ov::Tensor(element_type, shape);
} else {
tensor.set_shape(shape);
}
}
} // namespace
// ! [infer_request:ctor]
TemplatePlugin::InferRequest::InferRequest(const std::shared_ptr<const TemplatePlugin::CompiledModel>& model)
: ov::ISyncInferRequest(model) {
// TODO: allocate infer request device and host buffers if needed, fill actual list of profiling tasks
auto requestID = std::to_string(get_template_model()->_requestId.fetch_add(1));
std::string name = get_template_model()->m_model->get_friendly_name() + "_Req" + requestID;
m_profiling_task = {
openvino::itt::handle("Template" + std::to_string(get_template_model()->_cfg.deviceId) + "_" + name +
"_Preprocess"),
openvino::itt::handle("Template" + std::to_string(get_template_model()->_cfg.deviceId) + "_" + name +
"_Postprocess"),
openvino::itt::handle("Template" + std::to_string(get_template_model()->_cfg.deviceId) + "_" + name +
"_StartPipeline"),
openvino::itt::handle("Template" + std::to_string(get_template_model()->_cfg.deviceId) + "_" + name +
"_WaitPipline"),
};
m_executable = get_template_model()->get_template_plugin()->_backend->compile(get_template_model()->m_model);
// Allocate plugin backend specific memory handles
m_backend_input_tensors.resize(get_inputs().size());
m_backend_output_tensors.resize(get_outputs().size());
// Allocate input/output tensors
for (const auto& input : get_inputs()) {
allocate_tensor(input, [input](ov::Tensor& tensor) {
// Can add a check to avoid double work in case of shared tensors
allocate_tensor_impl(tensor,
input.get_element_type(),
input.get_partial_shape().is_dynamic() ? ov::Shape{0} : input.get_shape());
});
}
for (const auto& output : get_outputs()) {
allocate_tensor(output, [output](ov::Tensor& tensor) {
// Can add a check to avoid double work in case of shared tensors
allocate_tensor_impl(tensor,
output.get_element_type(),
output.get_partial_shape().is_dynamic() ? ov::Shape{0} : output.get_shape());
});
}
}
// ! [infer_request:ctor]
std::vector<ov::VariableState> TemplatePlugin::InferRequest::query_state() const {
OPENVINO_NOT_IMPLEMENTED;
}
std::shared_ptr<const TemplatePlugin::CompiledModel> TemplatePlugin::InferRequest::get_template_model() const {
auto& compiled_model = get_compiled_model();
auto template_model = std::dynamic_pointer_cast<const TemplatePlugin::CompiledModel>(compiled_model);
OPENVINO_ASSERT(template_model);
return template_model;
}
// ! [infer_request:dtor]
TemplatePlugin::InferRequest::~InferRequest() = default;
// ! [infer_request:dtor]
// ! [infer_request:infer_impl]
void TemplatePlugin::InferRequest::infer() {
// TODO: fill with actual list of pipeline stages, which are executed synchronously for sync infer requests
infer_preprocess();
start_pipeline();
wait_pipeline(); // does nothing in current implementation
infer_postprocess();
}
// ! [infer_request:infer_impl]
// ! [infer_request:infer_preprocess]
void TemplatePlugin::InferRequest::infer_preprocess() {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, m_profiling_task[Preprocess]);
auto start = Time::now();
convert_batched_tensors();
check_tensors();
// Allocate backend tensors
OPENVINO_ASSERT(get_inputs().size() == m_backend_input_tensors.size());
for (size_t i = 0; i < get_inputs().size(); i++) {
auto tensor = get_tensor(get_inputs()[i]);
if (tensor.is_continuous()) {
// No ROI extraction is needed
m_backend_input_tensors[i] =
get_template_model()->get_template_plugin()->_backend->create_tensor(tensor.get_element_type(),
tensor.get_shape(),
tensor.data());
} else {
OPENVINO_ASSERT(tensor.get_element_type().bitwidth() % 8 == 0,
"Template plugin: Unsupported ROI tensor with element type having ",
std::to_string(tensor.get_element_type().bitwidth()),
" bits size");
ov::Shape shape = tensor.get_shape();
// Perform manual extraction of ROI tensor
// Basic implementation doesn't take axis order into account `desc.getBlockingDesc().getOrder()`
// Performance of manual extraction is not optimal, but it is ok for template implementation
m_backend_input_tensors[i] =
get_template_model()->get_template_plugin()->_backend->create_tensor(tensor.get_element_type(),
tensor.get_shape());
auto* src_data = static_cast<uint8_t*>(tensor.data());
auto dst_tensor = std::dynamic_pointer_cast<ngraph::runtime::HostTensor>(m_backend_input_tensors[i]);
OPENVINO_ASSERT(dst_tensor, "Template plugin error: Can't cast created tensor to HostTensor");
auto* dst_data = dst_tensor->get_data_ptr<uint8_t>();
std::vector<size_t> indexes(shape.size());
for (size_t dst_idx = 0; dst_idx < ov::shape_size(shape); dst_idx++) {
size_t val = dst_idx;
size_t src_idx = 0;
for (size_t j1 = 0; j1 < indexes.size(); j1++) {
size_t j = indexes.size() - j1 - 1;
indexes[j] = val % shape[j];
val /= shape[j];
src_idx += indexes[j] * tensor.get_strides()[j];
}
memcpy(dst_data + dst_idx * tensor.get_element_type().size(),
src_data + src_idx,
tensor.get_element_type().size());
}
}
}
// Tensors can be dynamic, so in this case we need to allocate tensors with right shape
OPENVINO_ASSERT(get_outputs().size() == m_backend_output_tensors.size());
for (size_t i = 0; i < get_outputs().size(); i++) {
const auto& result = get_template_model()->m_model->get_results()[i];
if (result->get_output_partial_shape(0).is_dynamic()) {
m_backend_output_tensors[i] = get_template_model()->get_template_plugin()->_backend->create_tensor();
continue;
}
auto tensor = get_tensor(get_outputs()[i]);
m_backend_output_tensors[i] =
get_template_model()->get_template_plugin()->_backend->create_tensor(tensor.get_element_type(),
tensor.get_shape(),
tensor.data());
}
m_durations[Preprocess] = Time::now() - start;
}
// ! [infer_request:infer_preprocess]
// ! [infer_request:start_pipeline]
void TemplatePlugin::InferRequest::start_pipeline() {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, m_profiling_task[StartPipeline])
auto start = Time::now();
m_executable->call(m_backend_output_tensors, m_backend_input_tensors);
m_durations[StartPipeline] = Time::now() - start;
}
// ! [infer_request:start_pipeline]
void TemplatePlugin::InferRequest::wait_pipeline() {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, m_profiling_task[WaitPipeline])
auto start = Time::now();
// TODO: Wait pipeline using driver API or other synchronizations methods
// NOTE: not used in current implementation since `startPipeline` executes pipiline synchronously
m_durations[WaitPipeline] = Time::now() - start;
}
// ! [infer_request:infer_postprocess]
void TemplatePlugin::InferRequest::infer_postprocess() {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, m_profiling_task[Postprocess]);
auto start = Time::now();
OPENVINO_ASSERT(get_outputs().size() == m_backend_output_tensors.size());
for (size_t i = 0; i < get_outputs().size(); i++) {
const auto& result = get_template_model()->m_model->get_results()[i];
if (result->get_output_partial_shape(0).is_dynamic()) {
auto host_tensor = m_backend_output_tensors[i];
ov::Output<const ov::Node> output{result->output(0).get_node(), result->output(0).get_index()};
allocate_tensor(output, [host_tensor](ov::Tensor& tensor) {
allocate_tensor_impl(tensor, host_tensor->get_element_type(), host_tensor->get_shape());
// tensor.set_shape(host_tensor->get_shape());
host_tensor->read(static_cast<char*>(tensor.data()), host_tensor->get_size_in_bytes());
});
}
}
m_durations[Postprocess] = Time::now() - start;
}
// ! [infer_request:infer_postprocess]
// ! [infer_request:set_blobs_impl]
void TemplatePlugin::InferRequest::set_tensors_impl(const ov::Output<const ov::Node> port,
const std::vector<ov::Tensor>& tensors) {
for (const auto& input : get_inputs()) {
if (input == port) {
m_batched_tensors[input.get_tensor_ptr()] = tensors;
return;
}
}
OPENVINO_UNREACHABLE("Cannot find input tensors for port ", port);
}
// ! [infer_request:set_blobs_impl]
// ! [infer_request:get_performance_counts]
std::vector<ov::ProfilingInfo> TemplatePlugin::InferRequest::get_profiling_info() const {
std::vector<ov::ProfilingInfo> info;
const auto fill_profiling_info = [](const std::string& name,
const std::chrono::duration<float, std::micro>& time) -> ov::ProfilingInfo {
ov::ProfilingInfo p_info;
p_info.status = ov::ProfilingInfo::Status::EXECUTED;
p_info.node_name = name;
p_info.cpu_time = p_info.real_time = std::chrono::duration_cast<std::chrono::milliseconds>(time);
return p_info;
};
info.emplace_back(fill_profiling_info("input preprocessing", m_durations[Preprocess]));
info.emplace_back(fill_profiling_info("execution time", m_durations[StartPipeline]));
info.emplace_back(fill_profiling_info("output postprocessing", m_durations[Postprocess]));
return info;
}
// ! [infer_request:get_performance_counts]
src/plugins/template/src/infer_request.hpp
+57
View File
@@ -0,0 +1,57 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#pragma once
#include <array>
#include <chrono>
#include <map>
#include <memory>
#include <string>
#include <vector>
#include "executable.hpp"
#include "ngraph/runtime/tensor.hpp"
#include "openvino/itt.hpp"
#include "openvino/runtime/isync_infer_request.hpp"
namespace TemplatePlugin {
// forward declaration
class CompiledModel;
// ! [infer_request:header]
class InferRequest : public ov::ISyncInferRequest {
public:
explicit InferRequest(const std::shared_ptr<const TemplatePlugin::CompiledModel>& compiled_model);
~InferRequest();
void infer() override;
std::vector<ov::VariableState> query_state() const override;
std::vector<ov::ProfilingInfo> get_profiling_info() const override;
// pipeline methods-stages which are used in async infer request implementation and assigned to particular executor
void infer_preprocess();
void start_pipeline();
void wait_pipeline();
void infer_postprocess();
void set_tensors_impl(const ov::Output<const ov::Node> port, const std::vector<ov::Tensor>& tensors) override;
private:
std::shared_ptr<const CompiledModel> get_template_model() const;
enum { Preprocess, Postprocess, StartPipeline, WaitPipeline, numOfStages };
std::array<openvino::itt::handle_t, numOfStages> m_profiling_task;
// for performance counters
std::array<std::chrono::duration<float, std::micro>, numOfStages> m_durations;
std::vector<std::shared_ptr<ngraph::runtime::Tensor>> m_backend_input_tensors;
std::vector<std::shared_ptr<ngraph::runtime::Tensor>> m_backend_output_tensors;
std::shared_ptr<ngraph::runtime::Executable> m_executable;
};
// ! [infer_request:header]
} // namespace TemplatePlugin
src/plugins/template/src/plugin.cpp
+1 -1
View File
@@ -151,7 +151,7 @@ std::shared_ptr<ov::ICompiledModel> TemplatePlugin::Plugin::import_model(std::is
// ! [plugin:query_network]
ov::SupportedOpsMap TemplatePlugin::Plugin::query_model(const std::shared_ptr<const ov::Model>& model,
const ov::AnyMap& properties) const {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, "Plugin::query_model");
OV_ITT_SCOPED_TASK(TemplatePlugin::itt::domains::TemplatePlugin, "Plugin::query_model");
Configuration fullConfig{properties, _cfg, false};
src/plugins/template/src/plugin.hpp
+1 -1
View File
@@ -46,7 +46,7 @@ public:
private:
friend class TemplatePlugin::CompiledModel;
friend class TemplateInferRequest;
friend class InferRequest;
std::shared_ptr<ngraph::runtime::Backend> _backend;
Configuration _cfg;
src/plugins/template/src/template_async_infer_request.cpp
-51
View File
@@ -1,51 +0,0 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include "template_async_infer_request.hpp"
#include "template_itt.hpp"
using namespace TemplatePlugin;
// ! [async_infer_request:ctor]
TemplateAsyncInferRequest::TemplateAsyncInferRequest(const TemplateInferRequest::Ptr& inferRequest,
const InferenceEngine::ITaskExecutor::Ptr& cpuTaskExecutor,
const InferenceEngine::ITaskExecutor::Ptr& waitExecutor,
const InferenceEngine::ITaskExecutor::Ptr& callbackExecutor)
: AsyncInferRequestThreadSafeDefault(inferRequest, cpuTaskExecutor, callbackExecutor),
_inferRequest(inferRequest),
_waitExecutor(waitExecutor) {
// In current implementation we have CPU only tasks and no needs in 2 executors
// So, by default single stage pipeline is created.
// This stage executes InferRequest::Infer() using cpuTaskExecutor.
// But if remote asynchronous device is used the pipeline can by splitted tasks that are executed by cpuTaskExecutor
// and waiting tasks. Waiting tasks can lock execution thread so they use separate threads from other executor.
constexpr const auto remoteDevice = false;
if (remoteDevice) {
_pipeline = {{cpuTaskExecutor,
[this] {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin,
"TemplateAsyncInferRequest::PreprocessingAndStartPipeline");
_inferRequest->inferPreprocess();
_inferRequest->startPipeline();
}},
{_waitExecutor,
[this] {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, "TemplateAsyncInferRequest::WaitPipeline");
_inferRequest->waitPipeline();
}},
{cpuTaskExecutor, [this] {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, "TemplateAsyncInferRequest::Postprocessing");
_inferRequest->inferPostprocess();
}}};
}
}
// ! [async_infer_request:ctor]
// ! [async_infer_request:dtor]
TemplateAsyncInferRequest::~TemplateAsyncInferRequest() {
InferenceEngine::AsyncInferRequestThreadSafeDefault::StopAndWait();
}
// ! [async_infer_request:dtor]
src/plugins/template/src/template_async_infer_request.hpp
-29
View File
@@ -1,29 +0,0 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#pragma once
#include <cpp_interfaces/impl/ie_infer_async_request_thread_safe_default.hpp>
#include "template_infer_request.hpp"
namespace TemplatePlugin {
// ! [async_infer_request:header]
class TemplateAsyncInferRequest : public InferenceEngine::AsyncInferRequestThreadSafeDefault {
public:
TemplateAsyncInferRequest(const TemplateInferRequest::Ptr& inferRequest,
const InferenceEngine::ITaskExecutor::Ptr& taskExecutor,
const InferenceEngine::ITaskExecutor::Ptr& waitExecutor,
const InferenceEngine::ITaskExecutor::Ptr& callbackExecutor);
~TemplateAsyncInferRequest();
private:
TemplateInferRequest::Ptr _inferRequest;
InferenceEngine::ITaskExecutor::Ptr _waitExecutor;
};
// ! [async_infer_request:header]
} // namespace TemplatePlugin
src/plugins/template/src/template_infer_request.cpp
-380
View File
@@ -1,380 +0,0 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include "template_infer_request.hpp"
#include <debug.h>
#include <ie_compound_blob.h>
#include <algorithm>
#include <map>
#include <memory>
#include <ngraph/runtime/host_tensor.hpp>
#include <ngraph/runtime/reference/convert.hpp>
#include <string>
#include <utility>
#include "blob_factory.hpp"
#include "compiled_model.hpp"
#include "ie_api.h"
#include "ie_common.h"
#include "ie_ngraph_utils.hpp"
#include "ie_remote_blob.hpp"
#include "openvino/core/except.hpp"
#include "openvino/core/partial_shape.hpp"
#include "plugin.hpp"
#include "template_itt.hpp"
using namespace TemplatePlugin;
using namespace InferenceEngine;
using Time = std::chrono::high_resolution_clock;
// ! [infer_request:ctor]
TemplateInferRequest::TemplateInferRequest(const std::vector<std::shared_ptr<const ov::Node>>& inputs,
const std::vector<std::shared_ptr<const ov::Node>>& outputs,
const std::shared_ptr<const TemplatePlugin::CompiledModel>& compiled_model)
: IInferRequestInternal(inputs, outputs),
m_compiled_model(compiled_model) {
createInferRequest();
}
void TemplateInferRequest::createInferRequest() {
// TODO: allocate infer request device and host buffers if needed, fill actual list of profiling tasks
auto compiled_model = std::const_pointer_cast<TemplatePlugin::CompiledModel>(m_compiled_model);
auto requestID = std::to_string(compiled_model->_requestId.fetch_add(1));
std::string name = m_compiled_model->m_model->get_friendly_name() + "_Req" + requestID;
_profilingTask = {
openvino::itt::handle("Template" + std::to_string(m_compiled_model->_cfg.deviceId) + "_" + name +
"_Preprocess"),
openvino::itt::handle("Template" + std::to_string(m_compiled_model->_cfg.deviceId) + "_" + name +
"_Postprocess"),
openvino::itt::handle("Template" + std::to_string(m_compiled_model->_cfg.deviceId) + "_" + name +
"_StartPipline"),
openvino::itt::handle("Template" + std::to_string(m_compiled_model->_cfg.deviceId) + "_" + name +
"_WaitPipline"),
};
_executable = m_compiled_model->get_template_plugin()->_backend->compile(m_compiled_model->m_model);
allocateDeviceBuffers();
allocateBlobs();
}
// ! [infer_request:ctor]
// ! [infer_request:dtor]
TemplateInferRequest::~TemplateInferRequest() {
auto compiled_model = std::const_pointer_cast<TemplatePlugin::CompiledModel>(m_compiled_model);
compiled_model->_requestId--;
}
// ! [infer_request:dtor]
void TemplateInferRequest::allocateDeviceBuffers() {
// Allocate plugin backend specific memory handles
_inputTensors.resize(_networkInputs.size());
_outputTensors.resize(_networkOutputs.size());
}
template <typename BlobData>
static void AllocateImplSingle(const BlobData& blobData, BlobMap& blobMap, const SizeVector& dims) {
const auto& precision = blobData.second->getTensorDesc().getPrecision();
auto layout = blobData.second->getTensorDesc().getLayout();
Blob::Ptr& blob = blobMap[blobData.first];
if (!blob) {
blob = make_blob_with_precision({precision, dims, layout});
blob->allocate();
} else {
blob->setShape(dims);
}
}
template <typename BlobDataMap>
static void AllocateImpl(const BlobDataMap& userDataMap, BlobMap& userBlobMap, bool isInputBlob = true) {
for (const auto& userData : userDataMap) {
auto tensorDesc = userData.second->getTensorDesc();
AllocateImplSingle(userData, userBlobMap, tensorDesc.getDims());
}
}
void TemplateInferRequest::allocateBlobs() {
AllocateImpl(_networkInputs, _inputs);
AllocateImpl(_networkOutputs, _outputs, false);
}
// ! [infer_request:infer_impl]
void TemplateInferRequest::InferImpl() {
// TODO: fill with actual list of pipeline stages, which are executed synchronously for sync infer requests
inferPreprocess();
startPipeline();
waitPipeline(); // does nothing in current implementation
inferPostprocess();
}
// ! [infer_request:infer_impl]
// ! [infer_request:infer_preprocess]
void TemplateInferRequest::inferPreprocess() {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, _profilingTask[Preprocess]);
auto start = Time::now();
convertBatchedInputBlobs();
// NOTE: After IInferRequestInternal::execDataPreprocessing call
// input can points to other memory region than it was allocated in constructor.
for (auto&& input : _inputs) {
auto index = m_compiled_model->_inputIndex.at(input.first);
const auto& parameter_type = m_compiled_model->m_model->get_parameters()[index]->get_element_type();
auto mem_blob = InferenceEngine::as<InferenceEngine::MemoryBlob>(input.second);
auto isNonRoiDesc = [](const BlockingDesc& desc) {
size_t exp_stride = 1;
for (size_t i = 0; i < desc.getBlockDims().size(); i++) {
size_t rev_idx = desc.getBlockDims().size() - i - 1;
if (desc.getStrides()[rev_idx] != exp_stride || desc.getOffsetPaddingToData()[rev_idx] != 0) {
return false;
}
exp_stride *= desc.getBlockDims()[rev_idx];
}
return true;
};
if (isNonRoiDesc(input.second->getTensorDesc().getBlockingDesc())) {
// No ROI extraction is needed
_inputTensors[index] = m_compiled_model->get_template_plugin()->_backend->create_tensor(
parameter_type,
mem_blob->getTensorDesc().getBlockingDesc().getBlockDims(),
mem_blob->rmap().as<void*>());
} else {
OPENVINO_ASSERT(parameter_type.bitwidth() % 8 == 0,
"Template plugin: Unsupported ROI tensor with element type having ",
std::to_string(parameter_type.bitwidth()),
" bits size");
ov::Shape shape = input.second->getTensorDesc().getDims();
// Perform manual extraction of ROI tensor
// Basic implementation doesn't take axis order into account `desc.getBlockingDesc().getOrder()`
// Performance of manual extraction is not optimal, but it is ok for template implementation
_inputTensors[index] =
m_compiled_model->get_template_plugin()->_backend->create_tensor(parameter_type, shape);
auto desc = mem_blob->getTensorDesc();
auto* src_data = mem_blob->rmap().as<uint8_t*>();
auto dst_tensor = std::dynamic_pointer_cast<ngraph::runtime::HostTensor>(_inputTensors[index]);
OPENVINO_ASSERT(dst_tensor, "Template plugin error: Can't cast created tensor to HostTensor");
auto* dst_data = dst_tensor->get_data_ptr<uint8_t>();
std::vector<size_t> indexes(shape.size());
for (size_t dst_idx = 0; dst_idx < ov::shape_size(shape); dst_idx++) {
size_t val = dst_idx;
size_t src_idx = 0;
for (size_t j1 = 0; j1 < indexes.size(); j1++) {
size_t j = indexes.size() - j1 - 1;
indexes[j] = val % shape[j] + desc.getBlockingDesc().getOffsetPaddingToData()[j];
val /= shape[j];
src_idx += indexes[j] * desc.getBlockingDesc().getStrides()[j];
}
memcpy(dst_data + dst_idx * parameter_type.size(),
src_data + src_idx * parameter_type.size(),
parameter_type.size());
}
}
}
for (auto&& output : _outputs) {
auto mem_blob = InferenceEngine::as<InferenceEngine::MemoryBlob>(output.second);
auto index = m_compiled_model->_outputIndex.at(output.first);
const auto& result = m_compiled_model->m_model->get_results()[index];
if (result->get_output_partial_shape(0).is_dynamic()) {
_outputTensors[index] = m_compiled_model->get_template_plugin()->_backend->create_tensor();
continue;
}
const auto& resultShape = result->get_shape();
const auto& resultType = result->get_element_type();
_outputTensors[index] =
m_compiled_model->get_template_plugin()->_backend->create_tensor(resultType,
resultShape,
mem_blob->wmap().as<void*>());
}
_durations[Preprocess] = Time::now() - start;
}
// ! [infer_request:infer_preprocess]
// ! [infer_request:start_pipeline]
void TemplateInferRequest::startPipeline() {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, _profilingTask[StartPipeline])
auto start = Time::now();
_executable->call(_outputTensors, _inputTensors);
_durations[StartPipeline] = Time::now() - start;
}
// ! [infer_request:start_pipeline]
void TemplateInferRequest::waitPipeline() {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, _profilingTask[WaitPipeline])
auto start = Time::now();
// TODO: Wait pipeline using driver API or other synchronizations methods
// NOTE: not used in current implementation since `startPipeline` executes pipiline synchronously
_durations[WaitPipeline] = Time::now() - start;
}
// ! [infer_request:infer_postprocess]
void TemplateInferRequest::inferPostprocess() {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, _profilingTask[Postprocess]);
auto start = Time::now();
for (auto&& output : _outputs) {
auto index = m_compiled_model->_outputIndex.at(output.first);
const auto& result = m_compiled_model->m_model->get_results()[index];
if (result->get_output_partial_shape(0).is_dynamic()) {
// Touch blob to allocate it
GetBlob(output.first);
}
auto mem_blob = InferenceEngine::as<InferenceEngine::MemoryBlob>(output.second);
if (result->get_output_partial_shape(0).is_dynamic()) {
auto tensor = _outputTensors[m_compiled_model->_outputIndex.at(output.first)];
tensor->read(mem_blob->wmap().as<char*>(), tensor->get_size_in_bytes());
}
}
_durations[Postprocess] = Time::now() - start;
}
// ! [infer_request:infer_postprocess]
// ! [infer_request:get_blob]
InferenceEngine::Blob::Ptr TemplateInferRequest::GetBlob(const std::string& name) {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, "GetBlob");
InputInfo::Ptr foundInput;
DataPtr foundOutput;
Blob::Ptr data;
const SizeVector oneVector = {1};
if (findInputAndOutputBlobByName(name, foundInput, foundOutput)) {
data = _inputs[name];
SizeVector dims;
if (!data) {
auto&& parameters = m_compiled_model->m_model->get_parameters();
const auto& pshape = parameters.at(m_compiled_model->_inputIndex.at(name))->get_partial_shape();
dims = pshape.is_dynamic() ? SizeVector({0}) : pshape.get_shape();
AllocateImplSingle(*_networkInputs.find(name), _inputs, dims);
data = _inputs[name];
} else {
dims = data->getTensorDesc().getDims();
}
checkBlob(data, name, true, foundInput->getTensorDesc().getLayout() != SCALAR ? dims : oneVector);
} else {
data = _outputs[name];
SizeVector dims;
auto has_zeros = [](const SizeVector& vec) {
return std::any_of(vec.cbegin(), vec.cend(), [](size_t e) {
return e == 0;
});
};
if (!has_zeros(foundOutput->getTensorDesc().getDims())) {
dims = foundOutput->getTensorDesc().getDims();
} else if (_outputTensors[m_compiled_model->_outputIndex.at(name)] &&
_outputTensors[m_compiled_model->_outputIndex.at(name)]->get_partial_shape().is_static()) {
dims = _outputTensors[m_compiled_model->_outputIndex.at(name)]->get_shape();
} else {
auto rank = foundOutput->getTensorDesc().getDims().size();
dims = SizeVector(rank == 0 ? 1 : rank, 0);
}
if (data->getTensorDesc().getDims() != dims) {
AllocateImplSingle(*_networkOutputs.find(name), _outputs, dims);
data = _outputs[name];
}
checkBlob(data, name, false, foundOutput->getTensorDesc().getLayout() != SCALAR ? dims : oneVector);
}
return data;
}
// ! [infer_request:get_blob]
// ! [infer_request:set_blob]
void TemplateInferRequest::SetBlob(const std::string& name, const InferenceEngine::Blob::Ptr& userBlob) {
OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, "SetBlob");
if (name.empty()) {
IE_THROW(NotFound) << "Failed to set blob with empty name";
}
if (!userBlob)
IE_THROW(NotAllocated) << "Failed to set empty blob with name: \'" << name << "\'";
InputInfo::Ptr foundInput;
DataPtr foundOutput;
auto has_zeros = [](const SizeVector& vec) {
return std::any_of(vec.cbegin(), vec.cend(), [](size_t e) {
return e == 0;
});
};
const bool isInput = findInputAndOutputBlobByName(name, foundInput, foundOutput);
const bool compoundBlobPassed = userBlob->is<CompoundBlob>();
const bool remoteBlobPassed = userBlob->is<InferenceEngine::RemoteBlob>();
if (!compoundBlobPassed && !remoteBlobPassed && userBlob->buffer() == nullptr)
IE_THROW(NotAllocated) << "Input data was not allocated. Input name: \'" << name << "\'";
bool input_dynamic = foundInput && has_zeros(foundInput->getInputData()->getDims());
bool output_dynamic = foundOutput && has_zeros(foundOutput->getDims());
if (userBlob->size() == 0 && !(input_dynamic || output_dynamic)) {
IE_THROW() << "Input data is empty. Input name: \'" << name << "\'";
}
size_t dataSize = userBlob->size();
if (isInput) {
// ilavreno: the condition below is obsolete, but we need an exact list of precisions
// which are supports by G-API preprocessing
if (foundInput->getPrecision() != userBlob->getTensorDesc().getPrecision()) {
IE_THROW(ParameterMismatch)
<< "Failed to set Blob with precision not corresponding to user input precision";
}
// if (foundInput->getLayout() != userBlob->getTensorDesc().getLayout() &&
// foundInput->getTensorDesc().getBlockingDesc().getBlockDims() !=
// userBlob->getTensorDesc().getBlockingDesc().getBlockDims()) {
// IE_THROW(ParameterMismatch) << "Failed to set Blob with layout not corresponding to user input layout";
// }
size_t inputSize = _inputs[name]->getTensorDesc().getLayout() != InferenceEngine::Layout::SCALAR
? InferenceEngine::details::product(_inputs[name]->getTensorDesc().getDims())
: 1;
if (!input_dynamic && dataSize != inputSize) {
IE_THROW() << "Input blob size is not equal network input size (" << dataSize << "!=" << inputSize << ").";
}
_inputs[name] = userBlob;
_batched_inputs.erase(name);
} else {
if (compoundBlobPassed) {
IE_THROW(NotImplemented) << "cannot set compound blob: supported only for input pre-processing";
}
size_t outputSize = _outputs[name]->getTensorDesc().getLayout() != InferenceEngine::Layout::SCALAR
? details::product(_outputs[name]->getTensorDesc().getDims())
: 1;
if (!output_dynamic && dataSize != outputSize) {
IE_THROW() << "Output blob size is not equal network output size (" << dataSize << "!=" << outputSize
<< ").";
}
if (foundOutput->getPrecision() != userBlob->getTensorDesc().getPrecision()) {
IE_THROW(ParameterMismatch)
<< "Failed to set Blob with precision not corresponding to user output precision";
}
// if (foundOutput->getLayout() != userBlob->getTensorDesc().getLayout() &&
// foundOutput->getTensorDesc().getBlockingDesc().getBlockDims() !=
// userBlob->getTensorDesc().getBlockingDesc().getBlockDims()) {
// IE_THROW(ParameterMismatch) << "Failed to set Blob with layout not corresponding to user output layout";
// }
_outputs[name] = userBlob;
}
}
// ! [infer_request:set_blob]
// ! [infer_request:set_blobs_impl]
void TemplateInferRequest::SetBlobsImpl(const std::string& name, const InferenceEngine::BatchedBlob::Ptr& batchedBlob) {
_batched_inputs[name] = batchedBlob;
}
// ! [infer_request:set_blobs_impl]
// ! [infer_request:get_performance_counts]
std::map<std::string, InferenceEngineProfileInfo> TemplateInferRequest::GetPerformanceCounts() const {
std::map<std::string, InferenceEngineProfileInfo> perfMap;
InferenceEngineProfileInfo info;
info.execution_index = 0;
info.status = InferenceEngineProfileInfo::EXECUTED;
info.cpu_uSec = info.realTime_uSec = static_cast<long long>(_durations[Preprocess].count());
perfMap["1. input preprocessing"] = info;
info.cpu_uSec = info.realTime_uSec = 0;
perfMap["2. input transfer to a device"] = info;
info.cpu_uSec = info.realTime_uSec = static_cast<long long>(_durations[StartPipeline].count());
perfMap["3. execution time"] = info;
info.cpu_uSec = info.realTime_uSec = 0;
perfMap["4. output transfer from a device"] = info;
info.cpu_uSec = info.realTime_uSec = static_cast<long long>(_durations[Postprocess].count());
perfMap["5. output postprocessing"] = info;
return perfMap;
}
// ! [infer_request:get_performance_counts]
src/plugins/template/src/template_infer_request.hpp
-68
View File
@@ -1,68 +0,0 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#pragma once
#include <array>
#include <chrono>
#include <cpp_interfaces/interface/ie_iinfer_request_internal.hpp>
#include <executable.hpp>
#include <ie_input_info.hpp>
#include <map>
#include <memory>
#include <ngraph/runtime/tensor.hpp>
#include <openvino/itt.hpp>
#include <string>
#include <vector>
namespace TemplatePlugin {
// forward declaration
class CompiledModel;
// ! [infer_request:header]
class TemplateInferRequest : public InferenceEngine::IInferRequestInternal {
public:
typedef std::shared_ptr<TemplateInferRequest> Ptr;
TemplateInferRequest(const std::vector<std::shared_ptr<const ov::Node>>& inputs,
const std::vector<std::shared_ptr<const ov::Node>>& outputs,
const std::shared_ptr<const TemplatePlugin::CompiledModel>& compiled_model);
~TemplateInferRequest();
void InferImpl() override;
std::map<std::string, InferenceEngine::InferenceEngineProfileInfo> GetPerformanceCounts() const override;
// pipeline methods-stages which are used in async infer request implementation and assigned to particular executor
void inferPreprocess();
void startPipeline();
void waitPipeline();
void inferPostprocess();
InferenceEngine::Blob::Ptr GetBlob(const std::string& name) override;
void SetBlob(const std::string& name, const InferenceEngine::Blob::Ptr& userBlob) override;
void SetBlobsImpl(const std::string& name, const InferenceEngine::BatchedBlob::Ptr& batchedBlob) override;
private:
void createInferRequest();
void allocateDeviceBuffers();
void allocateBlobs();
enum { Preprocess, Postprocess, StartPipeline, WaitPipeline, numOfStages };
std::shared_ptr<const CompiledModel> m_compiled_model;
std::array<openvino::itt::handle_t, numOfStages> _profilingTask;
// for performance counters
std::array<std::chrono::duration<float, std::micro>, numOfStages> _durations;
InferenceEngine::BlobMap _networkOutputBlobs;
std::vector<std::shared_ptr<ngraph::runtime::Tensor>> _inputTensors;
std::vector<std::shared_ptr<ngraph::runtime::Tensor>> _outputTensors;
std::shared_ptr<ngraph::runtime::Executable> _executable;
};
// ! [infer_request:header]
} // namespace TemplatePlugin
src/tests/functional/plugin/shared/include/behavior/infer_request/wait.hpp
+7 -6
View File
@@ -13,7 +13,7 @@ TEST_P(InferRequestWaitTests, CorrectOneAsyncInferWithGetInOutWithInfWait) {
// Create InferRequest
InferenceEngine::InferRequest req;
InferenceEngine::Blob::Ptr blob =
FuncTestUtils::createAndFillBlob(cnnNet.getOutputsInfo().begin()->second->getTensorDesc());
FuncTestUtils::createAndFillBlob(cnnNet.getOutputsInfo().begin()->second->getTensorDesc());
ASSERT_NO_THROW(req = execNet.CreateInferRequest());
ASSERT_NO_THROW(blob = req.GetBlob(cnnNet.getInputsInfo().begin()->first));
req.Infer();
@@ -29,7 +29,7 @@ TEST_P(InferRequestWaitTests, canStartAsyncInferWithGetInOutWithStatusOnlyWait)
// Create InferRequest
InferenceEngine::InferRequest req;
InferenceEngine::Blob::Ptr blob =
FuncTestUtils::createAndFillBlob(cnnNet.getOutputsInfo().begin()->second->getTensorDesc());
FuncTestUtils::createAndFillBlob(cnnNet.getOutputsInfo().begin()->second->getTensorDesc());
ASSERT_NO_THROW(req = execNet.CreateInferRequest());
ASSERT_NO_THROW(blob = req.GetBlob(cnnNet.getInputsInfo().begin()->first));
req.Infer();
@@ -44,7 +44,7 @@ TEST_P(InferRequestWaitTests, FailedAsyncInferWithNegativeTimeForWait) {
// Create InferRequest
InferenceEngine::InferRequest req;
InferenceEngine::Blob::Ptr blob =
FuncTestUtils::createAndFillBlob(cnnNet.getOutputsInfo().begin()->second->getTensorDesc());
FuncTestUtils::createAndFillBlob(cnnNet.getOutputsInfo().begin()->second->getTensorDesc());
ASSERT_NO_THROW(req = execNet.CreateInferRequest());
ASSERT_NO_THROW(blob = req.GetBlob(cnnNet.getInputsInfo().begin()->first));
req.Infer();
@@ -76,13 +76,14 @@ TEST_P(InferRequestWaitTests, returnDeviceBusyOnSetBlobAfterAsyncInfer) {
auto outputBlob = req.GetBlob(cnnNet.getInputsInfo().begin()->first);
InferenceEngine::StatusCode sts;
sts = req.Wait(InferenceEngine::InferRequest::WaitMode::STATUS_ONLY);
ASSERT_EQ(InferenceEngine::StatusCode::INFER_NOT_STARTED, sts);
ASSERT_TRUE(InferenceEngine::StatusCode::INFER_NOT_STARTED == sts ||
sts == InferenceEngine::StatusCode::RESULT_NOT_READY);
req.StartAsync();
sts = req.Wait(InferenceEngine::InferRequest::WaitMode::RESULT_READY);
ASSERT_EQ(InferenceEngine::StatusCode::OK, sts);
try {
req.SetBlob(cnnNet.getInputsInfo().begin()->first, outputBlob);
} catch (const std::exception &e) {
} catch (const std::exception& e) {
std::cout << "Exception: " << e.what() << std::endl;
}
sts = req.Wait(InferenceEngine::InferRequest::WaitMode::STATUS_ONLY);
@@ -100,4 +101,4 @@ TEST_P(InferRequestWaitTests, returnDeviceBusyOnGetBlobAfterAsyncInfer) {
ASSERT_NO_THROW(req.SetBlob(cnnNet.getInputsInfo().begin()->first, outputBlob));
}
} // namespace BehaviorTestsDefinitions
} // namespace BehaviorTestsDefinitions