CVS-44774: Fixed preprocessing for template plugin (#4118)

* Fixed preprocessing for template plugin

* Added more tests instances

* Split common transformation to smaller ones which can be used by plugins

* Moved preprocessing transformation to Plugin API

* Added PreprocessConversionTest tests

* Disabled tests on GPU: CVS-51764

* Disabled some tests on VPU and TEMPLATE

* Support for input layout conversions in TEMPLATE plugin

* Improvements in Template Plugin

* Fixed compilation

* Fixes

* Disables some tests

* Fixed compilation on Windows

* Fixed docs

docs/nGraph_DG/nGraphTransformation.md

@@ -61,9 +61,9 @@ nGraph has three main transformation types:
 
 Template for FunctionPass transformation class
 
-@snippet src/template_function_transformation.hpp function_pass:template_transformation_hpp
+@snippet src/transformations/template_function_transformation.hpp function_pass:template_transformation_hpp
 
-@snippet src/template_function_transformation.cpp function_pass:template_transformation_cpp
+@snippet src/transformations/template_function_transformation.cpp function_pass:template_transformation_cpp
 
 Using `ngraph::FunctionPass`, you need to override the `run_on_function` method where you will write the transformation code.
 Return value is `true` if the original function has changed during transformation (new operation was added, or operations replacement was made, or node attributes were changed); otherwise, it is `false`.
@@ -75,9 +75,9 @@ Also `ngraph::FunctionPass` based transformations can be executed via `pass::Man
 `ngraph::pass::MatcherPass` is used for pattern-based transformations.
 
 Template for MatcherPass transformation class
-@snippet src/template_pattern_transformation.hpp graph_rewrite:template_transformation_hpp
+@snippet src/transformations/template_pattern_transformation.hpp graph_rewrite:template_transformation_hpp
 
-@snippet src/template_pattern_transformation.cpp graph_rewrite:template_transformation_cpp
+@snippet src/transformations/template_pattern_transformation.cpp graph_rewrite:template_transformation_cpp
 
 To use `ngraph::pass::MatcherPass`, you need to complete these steps:
 1. Create a pattern
@@ -113,7 +113,7 @@ That means that matcher passes registered in `pass::GraphRewrite` will be applie
 
 The example below shows how single MatcherPass can fuse sequence of operations using the `register_new_node` method.
 
-@snippet src/template_pattern_transformation.cpp matcher_pass:relu_fusion
+@snippet src/transformations/template_pattern_transformation.cpp matcher_pass:relu_fusion
 
 > **NOTE**: If you register multiple nodes, please add them in topological order. We do not topologically sort these nodes as it is a time-consuming operation.
 
@@ -128,11 +128,11 @@ register_matcher(m, callback);
 ### Execute MatcherPass
 MatcherPass has multiple ways to be executed:
 * Run on a single node - it can be useful if you want to run MatcherPass inside another transformation.
-@snippet src/template_pattern_transformation.cpp matcher_pass:run_on_node
+@snippet src/transformations/template_pattern_transformation.cpp matcher_pass:run_on_node
 * Run on `ngraph::Function` using GraphRewrite - this approach gives ability to run MatcherPass on whole `ngraph::Function`. Moreover, multiple MatcherPass transformation can be registered in a single GraphRewite to be executed in a single graph traversal.
-@snippet src/template_pattern_transformation.cpp matcher_pass:graph_rewrite
+@snippet src/transformations/template_pattern_transformation.cpp matcher_pass:graph_rewrite
 * Run on `ngraph::Function` using `pass::Manager` - this approach helps you to register MatcherPass for execution on `ngraph::Function` as another transformation types.
-@snippet src/template_pattern_transformation.cpp matcher_pass:manager
+@snippet src/transformations/template_pattern_transformation.cpp matcher_pass:manager
 
 
 ### ngraph::pass::GraphRewrite <a name="graph_rewrite_pass"></a>
@@ -140,7 +140,7 @@ MatcherPass has multiple ways to be executed:
 GraphRewrite pass serves for running multiple matcher passes on `ngraph::Function` in a single graph traversal.
 Example:
 
-@snippet src/template_pattern_transformation.cpp matcher_pass:graph_rewrite
+@snippet src/transformations/template_pattern_transformation.cpp matcher_pass:graph_rewrite
 
 In addition, GraphRewrite handles nodes that were registered by MatcherPasses during their execution. This nodes will be added to the beginning of the sequence with nodes for pattern matching.
 
@@ -352,7 +352,7 @@ Manual constant folding is more preferable than `ngraph::pass::ConstantFolding()
 
 Below you can find an example of manual constant folding:
 
-@snippet src/template_pattern_transformation.cpp manual_constant_folding
+@snippet src/transformations/template_pattern_transformation.cpp manual_constant_folding
 
 ## Common mistakes in transformations <a name="common_mistakes"></a>
 
@@ -373,11 +373,11 @@ In addition, `ngraph::pass::Manager` has extended debug capabilities (find more
 
 The example below shows basic usage of `ngraph::pass::Manager`
 
-@snippet src/template_pattern_transformation.cpp matcher_pass:manager3
+@snippet src/transformations/template_pattern_transformation.cpp matcher_pass:manager3
 
 Another example shows how multiple matcher passes can be united into single GraphRewrite.
 
-@snippet src/template_pattern_transformation.cpp matcher_pass:manager2
+@snippet src/transformations/template_pattern_transformation.cpp matcher_pass:manager2
 
 > **Note:** nGraph used to have the `pass::PassConfig` class for transformation pipeline manipulation.
 This mechanism is now obsolete and the `pass::PassConfig` class will be removed in future release.

docs/template_plugin/src/CMakeLists.txt

@@ -5,7 +5,7 @@
 # [cmake:plugin]
 set(TARGET_NAME "templatePlugin")
 
-file(GLOB SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/*.cpp)
+file(GLOB_RECURSE SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/*.cpp)
 file(GLOB_RECURSE HEADERS ${CMAKE_CURRENT_SOURCE_DIR}/*.hpp)
 
 # adds a shared library with plugin

docs/template_plugin/src/template_config.cpp

@@ -28,7 +28,7 @@ Configuration::Configuration(const ConfigMap& config, const Configuration & defa
         } else if (CONFIG_KEY(DEVICE_ID) == key) {
             deviceId = std::stoi(value);
             if (deviceId > 0) {
-                IE_THROW() << "Device ID " << deviceId << " is not supported";
+                IE_THROW(NotImplemented) << "Device ID " << deviceId << " is not supported";
             }
         } else if (CONFIG_KEY(PERF_COUNT) == key) {
             perfCount = (CONFIG_VALUE(YES) == value);

docs/template_plugin/src/template_executable_network.cpp

@@ -16,6 +16,8 @@ using namespace TemplatePlugin;
 
 // ! [executable_network:ctor_cnnnetwork]
 TemplatePlugin::ExecutableNetwork::ExecutableNetwork(const std::shared_ptr<const ngraph::Function>& function,
+                                                     const InferenceEngine::InputsDataMap&          inputInfoMap,
+                                                     const InferenceEngine::OutputsDataMap&         outputsInfoMap,
                                                      const Configuration&                           cfg,
                                                      const Plugin::Ptr&                             plugin) :
     InferenceEngine::ExecutableNetworkThreadSafeDefault(nullptr, nullptr), // Disable default threads creation
@@ -25,14 +27,14 @@ TemplatePlugin::ExecutableNetwork::ExecutableNetwork(const std::shared_ptr<const
     // you should select proper device based on KEY_DEVICE_ID or automatic behavior
     // In this case, _waitExecutor should also be created per device.
     try {
-        CompileNetwork(function);
+        CompileNetwork(function, inputInfoMap, outputsInfoMap);
         InitExecutor(); // creates thread-based executor using for async requests
     } catch (const InferenceEngine::Exception&) {
         throw;
     } catch (const std::exception & e) {
-        IE_THROW() << "Standard exception from compilation library: " << e.what();
+        IE_THROW(Unexpected) << "Standard exception from compilation library: " << e.what();
     } catch (...) {
-        IE_THROW() << "Generic exception is thrown";
+        IE_THROW(Unexpected) << "Generic exception is thrown";
     }
 }
 // ! [executable_network:ctor_cnnnetwork]
@@ -64,6 +66,8 @@ TemplatePlugin::ExecutableNetwork::ExecutableNetwork(std::istream &       model,
 
     // TODO: implement Import / Export of configuration options and merge with `cfg`
     // TODO: implement Import / Export of network precisions, layouts, preprocessing info
+    InferenceEngine::InputsDataMap inputInfoMap;
+    InferenceEngine::OutputsDataMap outputInfoMap;
 
     auto cnnnetwork = _plugin->GetCore()->ReadNetwork(xmlString, std::move(dataBlob));
 
@@ -72,27 +76,31 @@ TemplatePlugin::ExecutableNetwork::ExecutableNetwork(std::istream &       model,
     SetPointerToPlugin(_plugin->shared_from_this());
 
     try {
-        CompileNetwork(cnnnetwork.getFunction());
+        CompileNetwork(cnnnetwork.getFunction(), inputInfoMap, outputInfoMap);
         InitExecutor(); // creates thread-based executor using for async requests
     } catch (const InferenceEngine::Exception&) {
         throw;
     } catch (const std::exception & e) {
-        IE_THROW() << "Standard exception from compilation library: " << e.what();
+        IE_THROW(Unexpected) << "Standard exception from compilation library: " << e.what();
     } catch (...) {
-        IE_THROW() << "Generic exception is thrown";
+        IE_THROW(Unexpected) << "Generic exception is thrown";
     }
 }
 // ! [executable_network:ctor_import_stream]
 
 // ! [executable_network:map_graph]
 // forward declaration
-std::shared_ptr<ngraph::Function> TransformNetwork(const std::shared_ptr<const ngraph::Function>& function);
+std::shared_ptr<ngraph::Function> TransformNetwork(const std::shared_ptr<const ngraph::Function>& function,
+                                                   const InferenceEngine::InputsDataMap & inputInfoMap,
+                                                   const InferenceEngine::OutputsDataMap& outputsInfoMap);
 
-void TemplatePlugin::ExecutableNetwork::CompileNetwork(const std::shared_ptr<const ngraph::Function>& function) {
+void TemplatePlugin::ExecutableNetwork::CompileNetwork(const std::shared_ptr<const ngraph::Function>& function,
+                                                       const InferenceEngine::InputsDataMap & inputInfoMap,
+                                                       const InferenceEngine::OutputsDataMap& outputsInfoMap) {
     // TODO: perform actual graph compilation / mapping to backend graph representation / kernels
 
     // apply plugins transformations
-    _function = TransformNetwork(function);
+    _function = TransformNetwork(function, inputInfoMap, outputsInfoMap);
 
     // Generate backend specific blob mappings. For example Inference Engine uses not ngraph::Result nodes friendly name
     // as inference request output names but the name of the layer before.

docs/template_plugin/src/template_executable_network.hpp

@@ -25,6 +25,8 @@ class Plugin;
 class ExecutableNetwork : public InferenceEngine::ExecutableNetworkThreadSafeDefault {
 public:
     ExecutableNetwork(const std::shared_ptr<const ngraph::Function>& function,
+                      const InferenceEngine::InputsDataMap&          inputInfoMap,
+                      const InferenceEngine::OutputsDataMap&         outputsInfoMap,
                       const Configuration&                           cfg,
                       const std::shared_ptr<Plugin>&                 plugin);
 
@@ -46,7 +48,9 @@ public:
 private:
     friend class TemplateInferRequest;
 
-    void CompileNetwork(const std::shared_ptr<const ngraph::Function>& function);
+    void CompileNetwork(const std::shared_ptr<const ngraph::Function>& function,
+                        const InferenceEngine::InputsDataMap&          inputInfoMap,
+                        const InferenceEngine::OutputsDataMap&         outputsInfoMap);
     void InitExecutor();
 
     std::atomic<std::size_t>                    _requestId = {0};

docs/template_plugin/src/template_infer_request.cpp

@@ -61,7 +61,8 @@ template<typename BlobDataMap, typename GetNetworkPrecisionF>
 static void AllocateImpl(const BlobDataMap& userDataMap,
                          BlobMap& userBlobMap,
                          BlobMap& deviceBlobMap,
-                         GetNetworkPrecisionF&& GetNetworkPrecision) {
+                         GetNetworkPrecisionF&& GetNetworkPrecision,
+                         bool isInputBlob = true) {
     for (auto&& userData : userDataMap) {
         auto& dims = userData.second->getTensorDesc().getDims();
         const auto devicePrecision = Precision::FP32;
@@ -77,7 +78,7 @@ static void AllocateImpl(const BlobDataMap& userDataMap,
             case Precision::FP32 : {
                 userBlob = InferenceEngine::make_shared_blob<float>({userPrecision, dims, userLayout});
             } break;
-            default: IE_THROW() << "Template Plugin: Unsupported Input/Output Precision";
+            default: IE_THROW(NotImplemented) << "Template Plugin: Unsupported Input/Output Precision";
         }
         userBlob->allocate();
         userBlobMap[userData.first] = userBlob;
@@ -92,12 +93,16 @@ static void AllocateImpl(const BlobDataMap& userDataMap,
                     deviceBlob = InferenceEngine::make_shared_blob<float>({devicePrecision, dims, deviceLayout});
                 }
             } break;
-            default: IE_THROW() << "Template Plugin: Unsupported network Input/Output Presision";
+            default: IE_THROW(NotImplemented) << "Template Plugin: Unsupported network Input/Output Presision";
         }
-        // preprocessing converts user input blob to desired device input blob automatically
-        // NOTE: this is not supported for output user blobs yet
         if (userBlob != deviceBlob) {
+            if (isInputBlob) {
+                // preprocessing converts user input blob to desired device input blob automatically
                 deviceBlob->allocate();
+            } else {
+                // NOTE: this is not supported for output user blobs yet
+                IE_THROW(NotImplemented) << "Template Plugin: does not support setPrecision, setLayout for outputs";
+            }
         }
         deviceBlobMap[userData.first] = deviceBlob;
     }
@@ -111,7 +116,7 @@ void TemplateInferRequest::allocateBlobs() {
     auto&& results = _executableNetwork->_function->get_results();
     AllocateImpl(_networkOutputs, _outputs, _networkOutputBlobs, [&] (const std::string& blobName) {
         return results.at(_executableNetwork->_outputIndex.at(blobName))->get_element_type();
-    });
+    }, false);
 }
 
 // ! [infer_request:infer_impl]
@@ -140,7 +145,7 @@ static void blobCopy(const Blob::Ptr& src, const Blob::Ptr& dst) {
                     blobCopy<std::uint8_t, float>(src, dst);
                 } break;
                 default : {
-                    IE_THROW() << "Unsupported precision conversion from "
+                    IE_THROW(NotImplemented) << "Unsupported precision conversion from "
                         << src->getTensorDesc().getPrecision() <<" to " << dst->getTensorDesc().getPrecision();
                 }
             }
@@ -152,13 +157,13 @@ static void blobCopy(const Blob::Ptr& src, const Blob::Ptr& dst) {
                     blobCopy<float, std::uint8_t>(src, dst);
                 } break;
                 default : {
-                    IE_THROW() << "Unsupported precision conversion from "
+                    IE_THROW(NotImplemented) << "Unsupported precision conversion from "
                         << src->getTensorDesc().getPrecision() <<" to " << dst->getTensorDesc().getPrecision();
                 }
             }
         } break;
         default : {
-            IE_THROW() << "Unsupported precision conversion from " << src->getTensorDesc().getPrecision();
+            IE_THROW(NotImplemented) << "Unsupported precision conversion from " << src->getTensorDesc().getPrecision();
         }
     }
 }

docs/template_plugin/src/template_plugin.cpp

@@ -22,7 +22,8 @@
 #include "template_plugin.hpp"
 #include "template_executable_network.hpp"
 #include "template_infer_request.hpp"
-#include "template_pattern_transformation.hpp"
+#include "transformations/template_pattern_transformation.hpp"
+#include "transformations/preprocessing/preprocessing.hpp"
 
 using namespace TemplatePlugin;
 
@@ -52,12 +53,17 @@ Plugin::~Plugin() {
 
 // ! [plugin:transform_network]
 
-std::shared_ptr<ngraph::Function> TransformNetwork(const std::shared_ptr<const ngraph::Function>& function) {
+std::shared_ptr<ngraph::Function> TransformNetwork(const std::shared_ptr<const ngraph::Function>& function,
+                                                   const InferenceEngine::InputsDataMap & inputInfoMap,
+                                                   const InferenceEngine::OutputsDataMap& outputsInfoMap) {
     // 1. Copy ngraph::Function first to apply some transformations which modify original ngraph::Function
     auto transformedNetwork = ngraph::clone_function(*function);
 
     // 2. Perform common optimizations and device-specific transformations
     ngraph::pass::Manager passManager;
+    // Example: register transformation to convert preprocessing information to graph nodes
+    passManager.register_pass<ngraph::pass::AddPreprocessing>(inputInfoMap);
+    // TODO: add post-processing based on outputsInfoMap
     // Example: register CommonOptimizations transformation from transformations library
     passManager.register_pass<ngraph::pass::CommonOptimizations>();
     // Template plugin handles only FP32 networks
@@ -81,8 +87,12 @@ InferenceEngine::ExecutableNetworkInternal::Ptr Plugin::LoadExeNetworkImpl(const
                                                                            const ConfigMap &config) {
     OV_ITT_SCOPED_TASK(itt::domains::TemplatePlugin, "Plugin::LoadExeNetworkImpl");
 
+    InferenceEngine::InputsDataMap networkInputs = network.getInputsInfo();
+    InferenceEngine::OutputsDataMap networkOutputs = network.getOutputsInfo();
+
     auto fullConfig = Configuration{ config, _cfg };
-    return std::make_shared<ExecutableNetwork>(network.getFunction(), fullConfig,
+    return std::make_shared<ExecutableNetwork>(network.getFunction(),
+        networkInputs, networkOutputs, fullConfig,
         std::static_pointer_cast<Plugin>(shared_from_this()));
 }
 // ! [plugin:load_exe_network_impl]
@@ -114,7 +124,7 @@ InferenceEngine::QueryNetworkResult Plugin::QueryNetwork(const InferenceEngine::
     }
 
     // 2. It is needed to apply all transformations as it is done in LoadExeNetworkImpl
-    auto transformedFunction = TransformNetwork(function);
+    auto transformedFunction = TransformNetwork(function, network.getInputsInfo(), network.getOutputsInfo());
 
     // 3. The same input node can be transformed into supported and unsupported backend node
     // So we need store as supported either unsupported node sets
@@ -246,7 +256,7 @@ InferenceEngine::Parameter Plugin::GetMetric(const std::string& name, const std:
         using uint = unsigned int;
         IE_SET_METRIC_RETURN(RANGE_FOR_ASYNC_INFER_REQUESTS, std::make_tuple(uint{1}, uint{1}, uint{1}));
     } else  {
-        IE_THROW() << "Unsupported device metric: " << name;
+        IE_THROW(NotFound) << "Unsupported device metric: " << name;
     }
 }
 // ! [plugin:get_metric]

docs/template_plugin/src/transformations/preprocessing/mean_image_or_value.cpp

@@ -0,0 +1,48 @@
+// Copyright (C) 2021 Intel Corporation
+// SPDX-License-Identifier: Apache-2.0
+//
+
+#include <ngraph/opsets/opset3.hpp>
+#include <ngraph/pass/manager.hpp>
+#include <ngraph/pattern/op/wrap_type.hpp>
+
+#include "transformations/preprocessing/mean_image_or_value.hpp"
+
+using namespace ngraph;
+
+NGRAPH_RTTI_DEFINITION(ngraph::pass::AddMeanSubtract, "AddMeanSubtract", 0);
+
+ngraph::pass::AddMeanSubtract::AddMeanSubtract(const MeanMap & inputInfoMap) {
+    // RUN_ON_FUNCTION_SCOPE(AddMeanSubtract);
+    auto param = ngraph::pattern::wrap_type<ngraph::opset3::Parameter>();
+
+    ngraph::matcher_pass_callback callback = [=] (pattern::Matcher& m) {
+        auto param = std::dynamic_pointer_cast<ngraph::opset3::Parameter>(m.get_match_root());
+        if (!param) {
+            return false;
+        }
+
+        auto it = inputInfoMap.find(param->get_friendly_name());
+        if (it == inputInfoMap.end()) {
+            return false;
+        }
+
+        auto mean_const = it->second;
+        NGRAPH_CHECK(mean_const->get_element_type() == ngraph::element::f32,
+            "Mean for ", param->get_friendly_name(), " must have f32 type");
+
+        auto copy_param = param->clone_with_new_inputs({});
+        auto sub = std::make_shared<ngraph::opset3::Subtract>(copy_param, mean_const);
+
+        ngraph::replace_node(param, sub);
+        sub->set_argument(0, param);
+
+        // Return true as the root node was changed
+        return true;
+    };
+
+    // Register pattern with Parameter operation as a pattern root node
+    auto m = std::make_shared<ngraph::pattern::Matcher>(param, "AddMeanSubtract");
+    // Register Matcher
+    register_matcher(m, callback);
+}

docs/template_plugin/src/transformations/preprocessing/mean_image_or_value.hpp

@@ -0,0 +1,33 @@
+// Copyright (C) 2021 Intel Corporation
+// SPDX-License-Identifier: Apache-2.0
+//
+
+#pragma once
+
+#include <map>
+#include <string>
+
+#include <ngraph/op/constant.hpp>
+#include <ngraph/pass/graph_rewrite.hpp>
+
+#include "transformations_visibility.hpp"
+
+namespace ngraph {
+namespace pass {
+
+class AddMeanSubtract;
+
+}  // namespace pass
+}  // namespace ngraph
+
+/**
+ * @ingroup ie_transformation_common_api
+ * @brief Add `meanValue` or `meanImage` preprocessing to input nodes
+ */
+class ngraph::pass::AddMeanSubtract : public ngraph::pass::MatcherPass {
+public:
+    using MeanMap = std::map<std::string, std::shared_ptr<ngraph::op::v0::Constant>>;
+
+    NGRAPH_RTTI_DECLARATION;
+    explicit AddMeanSubtract(const MeanMap & inputInfoMap);
+};

docs/template_plugin/src/transformations/preprocessing/preprocessing.cpp

@@ -0,0 +1,101 @@
+// Copyright (C) 2021 Intel Corporation
+// SPDX-License-Identifier: Apache-2.0
+//
+
+#include <ngraph/pass/manager.hpp>
+#include <ngraph/opsets/opset3.hpp>
+
+#include "transformations/preprocessing/mean_image_or_value.hpp"
+#include "transformations/preprocessing/std_scale.hpp"
+#include "transformations/preprocessing/preprocessing.hpp"
+
+NGRAPH_RTTI_DEFINITION(ngraph::pass::AddPreprocessing, "AddPreprocessing", 0);
+
+ngraph::pass::AddPreprocessing::AddPreprocessing(const InferenceEngine::InputsDataMap & inputInfoMap)
+    : m_inputInfoMap(inputInfoMap) { }
+
+bool ngraph::pass::AddPreprocessing::run_on_function(std::shared_ptr<ngraph::Function> f) {
+    ngraph::pass::AddMeanSubtract::MeanMap meanMap;
+    ngraph::pass::AddStdScale::ScaleMap scaleMap;
+
+    for (const auto & it : m_inputInfoMap) {
+        bool has_scales = false, has_mean_values = false, has_mean_image = false;
+        const InferenceEngine::PreProcessInfo & pInfo = it.second->getPreProcess();
+        const auto & inputDims = it.second->getTensorDesc().getDims();
+        const size_t cn = pInfo.getNumberOfChannels();
+        std::vector<float> meanValues(cn), stdScales(cn);
+        InferenceEngine::Blob::Ptr meanImage = nullptr;
+
+        for (size_t c = 0; c < cn; ++c) {
+            if ((stdScales[c] = pInfo[c]->stdScale) != 1.0f) {
+                has_scales = true;
+            }
+
+            if ((meanValues[c] = pInfo[c]->meanValue) != 0.0f) {
+                has_mean_values = true;
+            }
+
+            if (pInfo[c]->meanData != nullptr) {
+                has_mean_image = true;
+                if (c == 0) {
+                    meanImage = pInfo[c]->meanData;
+                    NGRAPH_CHECK(meanImage->getTensorDesc().getPrecision() == InferenceEngine::Precision::FP32,
+                        "Only InferenceEngine::Precision::FP32 precision is supported for PreProcessChannel::meanData");
+                } else {
+                    NGRAPH_CHECK(meanImage->getTensorDesc() == pInfo[c]->meanData->getTensorDesc(),
+                        "TensorDesc for PreProcessChannel::meanData must be equal");
+                }
+            }
+        }
+
+        // no preprocessing for current input
+        if (!has_mean_values && !has_scales && !has_mean_image) {
+            continue;
+        }
+
+        NGRAPH_CHECK(!(has_mean_image && has_scales),
+            "Only PreProcessChannel::meanData or PreProcessChannel::meanValue can be set.");
+
+        if (has_scales) {
+            ngraph::Shape shape(inputDims.size(), 1);
+            shape[1] = stdScales.size(); // C
+            scaleMap[it.first] = ngraph::opset3::Constant::create(ngraph::element::f32, shape, stdScales);
+        }
+
+        if (has_mean_values) {
+            ngraph::Shape shape(inputDims.size(), 1);
+            shape[1] = meanValues.size(); // C
+            meanMap[it.first] = ngraph::opset3::Constant::create(ngraph::element::f32, shape, meanValues);
+        } else if (has_mean_image) {
+            ngraph::Shape shape = { cn };
+            auto dims = meanImage->getTensorDesc().getDims();
+            std::copy(dims.begin(), dims.end(), std::back_inserter(shape));
+
+            std::vector<float> meanImageData(ngraph::shape_size(shape));
+            for (size_t c = 0, i = 0; c < cn; ++c) {
+                auto lm = pInfo[c]->meanData->buffer();
+                const float *data = lm.as<const float *>();
+
+                std::memcpy(&meanImageData[i], data, meanImage->byteSize());
+                i += meanImage->size();
+            }
+
+            meanMap[it.first] = ngraph::opset3::Constant::create(ngraph::element::f32,
+                shape, meanImageData);
+        }
+    }
+
+    ngraph::pass::Manager manager(get_pass_config());
+    auto preproc = manager.register_pass<ngraph::pass::GraphRewrite>();
+
+    if (!scaleMap.empty()) {
+        preproc->add_matcher<ngraph::pass::AddStdScale>(scaleMap);
+    }
+    if (!meanMap.empty()) {
+        preproc->add_matcher<ngraph::pass::AddMeanSubtract>(meanMap);
+    }
+
+    manager.run_passes(f);
+
+    return false;
+}

docs/template_plugin/src/transformations/preprocessing/preprocessing.hpp

@@ -0,0 +1,35 @@
+// Copyright (C) 2021 Intel Corporation
+// SPDX-License-Identifier: Apache-2.0
+//
+
+#pragma once
+
+#include <ngraph/pass/pass.hpp>
+
+#include "ie_input_info.hpp"
+
+namespace ngraph {
+namespace pass {
+
+class AddPreprocessing;
+
+}  // namespace pass
+}  // namespace ngraph
+
+/**
+ * @brief Converts the following preprocessing information to ngraph operations:
+ *  - InferenceEngine::PreProcessInfo->PreProcessChannel::meanData -> Subtract
+ *  - InferenceEngine::PreProcessInfo->PreProcessChannel::meanValue -> Subtract
+ *  - InferenceEngine::PreProcessInfo->PreProcessChannel::stdScale -> Multiply
+ *
+ * The order of operations is the following:
+ *      (x - mean) * stdScale
+ */
+class ngraph::pass::AddPreprocessing : public ngraph::pass::FunctionPass {
+    const InferenceEngine::InputsDataMap & m_inputInfoMap;
+public:
+    NGRAPH_RTTI_DECLARATION;
+    explicit AddPreprocessing(const InferenceEngine::InputsDataMap & inputInfoMap);
+
+    bool run_on_function(std::shared_ptr<ngraph::Function> f) override;
+};

docs/template_plugin/src/transformations/preprocessing/std_scale.cpp

@@ -0,0 +1,48 @@
+// Copyright (C) 2021 Intel Corporation
+// SPDX-License-Identifier: Apache-2.0
+//
+
+#include <ngraph/opsets/opset3.hpp>
+#include <ngraph/pass/manager.hpp>
+#include <ngraph/pattern/op/wrap_type.hpp>
+
+#include "transformations/preprocessing/std_scale.hpp"
+
+using namespace ngraph;
+
+NGRAPH_RTTI_DEFINITION(ngraph::pass::AddStdScale, "AddStdScale", 0);
+
+ngraph::pass::AddStdScale::AddStdScale(const ScaleMap& inputInfoMap) {
+    // RUN_ON_FUNCTION_SCOPE(AddStdScale);
+    auto param = ngraph::pattern::wrap_type<ngraph::opset3::Parameter>();
+
+    ngraph::matcher_pass_callback callback = [=] (pattern::Matcher& m) {
+        auto param = std::dynamic_pointer_cast<ngraph::opset3::Parameter>(m.get_match_root());
+        if (!param) {
+            return false;
+        }
+
+        auto it = inputInfoMap.find(param->get_friendly_name());
+        if (it == inputInfoMap.end()) {
+            return false;
+        }
+
+        auto scale_const = it->second;
+        NGRAPH_CHECK(scale_const->get_element_type() == ngraph::element::f32,
+            "Scale for ", param->get_friendly_name(), " must have f32 type");
+
+        auto copy_param = param->clone_with_new_inputs({});
+        auto mul = std::make_shared<ngraph::opset3::Multiply>(copy_param, it->second);
+
+        ngraph::replace_node(param, mul);
+        mul->set_argument(0, param);
+
+        // Return true as the root node was changed
+        return true;
+    };
+
+    // Register pattern with Parameter operation as a pattern root node
+    auto m = std::make_shared<ngraph::pattern::Matcher>(param, "AddStdScale");
+    // Register Matcher
+    register_matcher(m, callback);
+}

docs/template_plugin/src/transformations/preprocessing/std_scale.hpp

@@ -0,0 +1,33 @@
+// Copyright (C) 2021 Intel Corporation
+// SPDX-License-Identifier: Apache-2.0
+//
+
+#pragma once
+
+#include <map>
+#include <string>
+
+#include <ngraph/op/constant.hpp>
+#include <ngraph/pass/graph_rewrite.hpp>
+
+#include "transformations_visibility.hpp"
+
+namespace ngraph {
+namespace pass {
+
+class AddStdScale;
+
+}  // namespace pass
+}  // namespace ngraph
+
+/**
+ * @ingroup ie_transformation_common_api
+ * @brief Add `stdScale` preprocessing to input nodes
+ */
+class ngraph::pass::AddStdScale : public ngraph::pass::MatcherPass {
+public:
+    using ScaleMap = std::map<std::string, std::shared_ptr<ngraph::op::v0::Constant>>;
+
+    NGRAPH_RTTI_DECLARATION;
+    explicit AddStdScale(const ScaleMap& inputInfoMap);
+};

docs/template_plugin/src/transformations/template_pattern_transformation.cpp

@@ -2,8 +2,8 @@
 // SPDX-License-Identifier: Apache-2.0
 //
 
-#include "template_pattern_transformation.hpp"
+#include "transformations/template_pattern_transformation.hpp"
-#include "template_function_transformation.hpp"
+#include "transformations/template_function_transformation.hpp"
 
 #include <ngraph/opsets/opset3.hpp>
 #include <ngraph/pass/manager.hpp>

docs/template_plugin/src/transformations/template_pattern_transformation.hpp

@@ -16,7 +16,7 @@ class ReluReluFusionMatcher;
 }  // namespace ngraph
 
 // ! [graph_rewrite:template_transformation_hpp]
-// template_pattern_transformation.hpp
+// transformations/template_pattern_transformation.hpp
 /**
  * @ingroup ie_transformation_common_api
  * @brief Add transformation description.

docs/template_plugin/tests/functional/shared_tests_instances/behavior/preprocessing.cpp

@@ -19,7 +19,7 @@ const std::vector<std::map<std::string, std::string>> configs = {
     {}
 };
 
-INSTANTIATE_TEST_CASE_P(PreprocessingPrecisionConvertTestsViaSetInput, PreprocessingPrecisionConvertTest,
+INSTANTIATE_TEST_CASE_P(smoke_PreprocessingPrecisionConvertTestsViaSetInput, PreprocessingPrecisionConvertTest,
                         ::testing::Combine(
                                 ::testing::ValuesIn(inputPrecisions),
                                 ::testing::Values(4),   // Number of input tensor channels
@@ -28,7 +28,7 @@ INSTANTIATE_TEST_CASE_P(PreprocessingPrecisionConvertTestsViaSetInput, Preproces
                                 ::testing::ValuesIn(configs)),
                         PreprocessingPrecisionConvertTest::getTestCaseName);
 
-INSTANTIATE_TEST_CASE_P(PreprocessingPrecisionConvertTestsViaGetBlob, PreprocessingPrecisionConvertTest,
+INSTANTIATE_TEST_CASE_P(smoke_PreprocessingPrecisionConvertTestsViaGetBlob, PreprocessingPrecisionConvertTest,
                         ::testing::Combine(
                                 ::testing::ValuesIn(inputPrecisions),
                                 ::testing::Values(4),       // Number of input tensor channels (blob_copy only supports 4d and 5d tensors)

docs/template_plugin/tests/functional/shared_tests_instances/behavior/set_preprocess.cpp

@@ -19,6 +19,15 @@ const std::vector<std::map<std::string, std::string>> configs = {
     {}
 };
 
+const std::vector<std::map<std::string, std::string>> multiConfigs = {
+        {{ InferenceEngine::MultiDeviceConfigParams::KEY_MULTI_DEVICE_PRIORITIES,
+           CommonTestUtils::DEVICE_TEMPLATE }}
+};
+
+const std::vector<std::map<std::string, std::string>> heteroConfigs = {
+        {{ "TARGET_FALLBACK", CommonTestUtils::DEVICE_TEMPLATE }}
+};
+
 INSTANTIATE_TEST_CASE_P(smoke_BehaviorTests, PreprocessTest,
                         ::testing::Combine(
                                 ::testing::ValuesIn(netPrecisions),
@@ -26,6 +35,20 @@ INSTANTIATE_TEST_CASE_P(smoke_BehaviorTests, PreprocessTest,
                                 ::testing::ValuesIn(configs)),
                         PreprocessTest::getTestCaseName);
 
+INSTANTIATE_TEST_CASE_P(smoke_Multi_BehaviorTests, PreprocessTest,
+                        ::testing::Combine(
+                                ::testing::ValuesIn(netPrecisions),
+                                ::testing::Values(CommonTestUtils::DEVICE_MULTI),
+                                ::testing::ValuesIn(multiConfigs)),
+                        PreprocessTest::getTestCaseName);
+
+INSTANTIATE_TEST_CASE_P(smoke_Hetero_BehaviorTests, PreprocessTest,
+                        ::testing::Combine(
+                                ::testing::ValuesIn(netPrecisions),
+                                ::testing::Values(CommonTestUtils::DEVICE_HETERO),
+                                ::testing::ValuesIn(heteroConfigs)),
+                        PreprocessTest::getTestCaseName);
+
 const std::vector<InferenceEngine::Precision> ioPrecisions = {
     InferenceEngine::Precision::FP32,
     InferenceEngine::Precision::U8

docs/template_plugin/tests/functional/skip_tests_config.cpp

@@ -12,10 +12,10 @@ std::vector<std::string> disabledTestPatterns() {
         ".*ExclusiveAsyncRequests.*",
         ".*reusableCPUStreamsExecutor.*",
         R"(.*SplitLayerTest.*numSplits\=30.*)",
-        // CVS-44774
-        ".*PreprocessTest.*",
         // CVS-51758
         ".*PreprocessConversionTest.*oPRC=U8.*",
-        ".*PreprocessConversionTest.*oLT=NHWC.*"
+        ".*PreprocessConversionTest.*oLT=NHWC.*",
+        ".*PreprocessingPrecisionConvertTestsViaSetInput.*SetInput.*",
+        ".*PreprocessingPrecisionConvertTestsViaGetBlob.*GetBlob.*",
     };
 }

docs/template_plugin/tests/functional/transformations/preprocessing.cpp

@@ -0,0 +1,183 @@
+// // Copyright (C) 2021 Intel Corporation
+// // SPDX-License-Identifier: Apache-2.0
+// //
+
+// #include <gtest/gtest.h>
+
+// #include <string>
+// #include <memory>
+// #include <map>
+
+// #include <ngraph/function.hpp>
+// #include <ngraph/opsets/opset5.hpp>
+// #include <ngraph/pass/manager.hpp>
+
+// #include <transformations/init_node_info.hpp>
+// #include <transformations/preprocessing/std_scale.hpp>
+// #include <transformations/preprocessing/mean_image_or_value.hpp>
+
+// #include "common_test_utils/ngraph_test_utils.hpp"
+
+
+// using namespace testing;
+// using namespace ngraph;
+
+
+// TEST(TransformationTests, Preprocessing_AddStdScale) {
+//     std::shared_ptr<Function> f(nullptr), f_ref(nullptr);
+
+//     const Shape data_shape{1, 3, 14, 14};
+//     const Shape scale_shape{3, 1, 1};
+//     {
+//         auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
+//         auto relu = std::make_shared<opset5::Relu>(data);
+//         f = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
+//         auto scales = opset5::Constant::create(element::f32, scale_shape,
+//             std::vector<float>(shape_size(scale_shape), 2.0f));
+//         pass::Manager m;
+//         m.register_pass<pass::InitNodeInfo>();
+//         m.register_pass<pass::AddStdScale>(pass::AddStdScale::ScaleMap{ { data->get_friendly_name(), scales } });
+//         m.run_passes(f);
+//     }
+//     {
+//         auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
+//         auto scales = opset5::Constant::create(element::f32, scale_shape,
+//             std::vector<float>(shape_size(scale_shape), 2.0f));
+//         auto mul = std::make_shared<opset5::Multiply>(data, scales);
+//         auto relu = std::make_shared<opset5::Relu>(mul);
+//         f_ref = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
+//     }
+
+//     auto res = compare_functions(f, f_ref);
+//     ASSERT_TRUE(res.first) << res.second;
+// }
+
+// TEST(TransformationTests, Preprocessing_AddMeanValue) {
+//     std::shared_ptr<Function> f(nullptr), f_ref(nullptr);
+
+//     const Shape data_shape{1, 3, 14, 14};
+//     const Shape mean_shape{3, 1, 1};
+//     {
+//         auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
+//         auto relu = std::make_shared<opset5::Relu>(data);
+//         f = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
+//         auto meanValues = opset5::Constant::create(element::f32, mean_shape,
+//             std::vector<float>(shape_size(mean_shape), 2.0f));
+//         pass::Manager m;
+//         m.register_pass<pass::InitNodeInfo>();
+//         m.register_pass<pass::AddMeanSubtract>(pass::AddMeanSubtract::MeanMap{ { data->get_friendly_name(), meanValues } });
+//         m.run_passes(f);
+//     }
+//     {
+//         auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
+//         auto meanValues = opset5::Constant::create(element::f32, mean_shape,
+//             std::vector<float>(shape_size(mean_shape), 2.0f));
+//         auto sub = std::make_shared<opset5::Subtract>(data, meanValues);
+//         auto relu = std::make_shared<opset5::Relu>(sub);
+//         f_ref = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
+//     }
+
+//     auto res = compare_functions(f, f_ref);
+//     ASSERT_TRUE(res.first) << res.second;
+// }
+
+// TEST(TransformationTests, Preprocessing_AddMeanImage) {
+//     std::shared_ptr<Function> f(nullptr), f_ref(nullptr);
+
+//     const Shape data_shape{1, 3, 14, 14};
+//     const Shape mean_shape{3, 14, 14};
+//     {
+//         auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
+//         auto relu = std::make_shared<opset5::Relu>(data);
+//         f = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
+//         auto meanValues = opset5::Constant::create(element::f32, mean_shape,
+//             std::vector<float>(shape_size(mean_shape), 2.0f));
+//         pass::Manager m;
+//         m.register_pass<pass::InitNodeInfo>();
+//         m.register_pass<pass::AddMeanSubtract>(pass::AddMeanSubtract::MeanMap{ { data->get_friendly_name(), meanValues } });
+//         m.run_passes(f);
+//     }
+//     {
+//         auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
+//         auto meanValues = opset5::Constant::create(element::f32, mean_shape,
+//             std::vector<float>(shape_size(mean_shape), 2.0f));
+//         auto sub = std::make_shared<opset5::Subtract>(data, meanValues);
+//         auto relu = std::make_shared<opset5::Relu>(sub);
+//         f_ref = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
+//     }
+
+//     auto res = compare_functions(f, f_ref);
+//     ASSERT_TRUE(res.first) << res.second;
+// }
+
+// TEST(TransformationTests, Preprocessing_AddMeanImageAndScale) {
+//     std::shared_ptr<Function> f(nullptr), f_ref(nullptr);
+
+//     const Shape data_shape{1, 3, 14, 14};
+//     const Shape mean_shape{3, 14, 14};
+//     const Shape scale_shape{3, 1, 1};
+//     {
+//         auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
+//         auto relu = std::make_shared<opset5::Relu>(data);
+//         f = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
+//         auto meanValues = opset5::Constant::create(element::f32, mean_shape,
+//             std::vector<float>(shape_size(mean_shape), 2.0f));
+//         auto scaleValues = opset5::Constant::create(element::f32, scale_shape,
+//             std::vector<float>(shape_size(scale_shape), 2.0f));
+//         pass::Manager m;
+//         m.register_pass<pass::InitNodeInfo>();
+//         m.register_pass<pass::AddStdScale>(pass::AddStdScale::ScaleMap{ { data->get_friendly_name(), scaleValues } });
+//         m.register_pass<pass::AddMeanSubtract>(pass::AddMeanSubtract::MeanMap{ { data->get_friendly_name(), meanValues } });
+//         m.run_passes(f);
+//     }
+//     {
+//         auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
+//         auto meanValues = opset5::Constant::create(element::f32, mean_shape,
+//             std::vector<float>(shape_size(mean_shape), 2.0f));
+//         auto scaleValues = opset5::Constant::create(element::f32, scale_shape,
+//             std::vector<float>(shape_size(scale_shape), 2.0f));
+//         auto sub = std::make_shared<opset5::Subtract>(data, meanValues);
+//         auto mul = std::make_shared<opset5::Multiply>(sub, scaleValues);
+//         auto relu = std::make_shared<opset5::Relu>(mul);
+//         f_ref = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
+//     }
+
+//     auto res = compare_functions(f, f_ref);
+//     ASSERT_TRUE(res.first) << res.second;
+// }
+
+// TEST(TransformationTests, Preprocessing_AddMeanValueAndScale) {
+//     std::shared_ptr<Function> f(nullptr), f_ref(nullptr);
+
+//     const Shape data_shape{1, 3, 14, 14};
+//     const Shape mean_shape{3, 1, 1};
+//     const Shape scale_shape{3, 1, 1};
+//     {
+//         auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
+//         auto relu = std::make_shared<opset5::Relu>(data);
+//         f = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
+//         auto meanValues = opset5::Constant::create(element::f32, mean_shape,
+//             std::vector<float>(shape_size(mean_shape), 2.0f));
+//         auto scaleValues = opset5::Constant::create(element::f32, scale_shape,
+//             std::vector<float>(shape_size(scale_shape), 2.0f));
+//         pass::Manager m;
+//         m.register_pass<pass::InitNodeInfo>();
+//         m.register_pass<pass::AddStdScale>(pass::AddStdScale::ScaleMap{ { data->get_friendly_name(), scaleValues } });
+//         m.register_pass<pass::AddMeanSubtract>(pass::AddMeanSubtract::MeanMap{ { data->get_friendly_name(), meanValues } });
+//         m.run_passes(f);
+//     }
+//     {
+//         auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
+//         auto meanValues = opset5::Constant::create(element::f32, mean_shape,
+//             std::vector<float>(shape_size(mean_shape), 2.0f));
+//         auto scaleValues = opset5::Constant::create(element::f32, scale_shape,
+//             std::vector<float>(shape_size(scale_shape), 2.0f));
+//         auto sub = std::make_shared<opset5::Subtract>(data, meanValues);
+//         auto mul = std::make_shared<opset5::Multiply>(sub, meanValues);
+//         auto relu = std::make_shared<opset5::Relu>(mul);
+//         f_ref = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
+//     }
+
+//     auto res = compare_functions(f, f_ref);
+//     ASSERT_TRUE(res.first) << res.second;
+// }

inference-engine/src/plugin_api/precision_utils.h

@@ -29,6 +29,9 @@ namespace InferenceEngine {
  * @defgroup ie_dev_api_plugin_api Plugin base classes
  * @brief A set of base and helper classes to implement a plugin class
  *
+ * @defgroup ie_dev_api_preproc_api Preprocessing API
+ * @brief A set transformations to convert InferenceEngine::PreProcessInfo to ngraph operations
+ * 
  * @defgroup ie_dev_api_exec_network_api Executable Network base classes
  * @brief A set of base and helper classes to implement an executable network class
  * 

inference-engine/tests/functional/plugin/shared/include/behavior/set_preprocess.hpp

@@ -124,10 +124,11 @@ TEST_P(PreprocessTest, SetMeanImagePreProcessGetBlob) {
         auto outMem = outBlob->cbuffer();
         const auto* outData = outMem.as<const float*>();
         ASSERT_EQ(inBlob->size(), outBlob->size());
-        for (size_t i = 0; i < inBlob->size(); i++)
+        for (size_t i = 0; i < inBlob->size(); i++) {
             ASSERT_EQ(inData[i] + inData[i], outData[i]);
         }
     }
+}
 
 TEST_P(PreprocessTest, SetMeanImagePreProcessSetBlob) {
     // Skip test according to plugin specific disabledTestPatterns() (if any)
@@ -255,10 +256,11 @@ TEST_P(PreprocessTest, SetMeanValuePreProcessGetBlob) {
         auto outMem = outBlob->cbuffer();
         const auto* outData = outMem.as<const float*>();
         ASSERT_EQ(inBlob->size(), outBlob->size());
-        for (size_t i = 0; i < inBlob->size(); i++)
+        for (size_t i = 0; i < inBlob->size(); i++) {
             ASSERT_EQ(inData[i] + 5, outData[i]);
         }
     }
+}
 
 TEST_P(PreprocessTest, SetMeanValuePreProcessSetBlob) {
     // Skip test according to plugin specific disabledTestPatterns() (if any)
@@ -511,10 +513,11 @@ TEST_P(PreprocessTest, SetScalePreProcessGetBlob) {
         auto outMem = outBlob->cbuffer();
         const auto* outData = outMem.as<const float*>();
         ASSERT_EQ(inBlob->size(), outBlob->size());
-        for (size_t i = 0; i < inBlob->size(); i++)
+        for (size_t i = 0; i < inBlob->size(); i++) {
             ASSERT_EQ(inData[i]*2, outData[i]);
         }
     }
+}
 
 
 TEST_P(PreprocessTest, SetScalePreProcessSetBlob) {