IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Changed Template plugin public property (#16496)

Browse Source 
* Changed template plugin public property

* Add property documentation

* Fixed comments

* Fixed typo
This commit is contained in:
Ilya Churaev 2023-03-23 19:34:49 +04:00 committed by GitHub
parent de0a4e16fb
commit 2755b32fb9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
13 changed files with 95 additions and 264 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
docs/IE_PLUGIN_DG/Intro.md
View File

@ -11,6 +11,7 @@
Implement Compiled Model Functionality <openvino_docs_ov_plugin_dg_compiled_model>
Implement Synchronous Inference Request <openvino_docs_ov_plugin_dg_infer_request>
Implement Asynchronous Inference Request <openvino_docs_ov_plugin_dg_async_infer_request>
Provide Plugin Specific Properties <openvino_docs_ov_plugin_dg_properties>
Implement Remote Context <openvino_docs_ov_plugin_dg_remote_context>
Implement Remote Tensor <openvino_docs_ov_plugin_dg_remote_tensor>
openvino_docs_ov_plugin_dg_plugin_build
@ -45,9 +46,11 @@ OpenVINO plugin dynamic library consists of several main components:
- Can extract performance counters for an inference pipeline execution profiling.
4. [Asynchronous Inference Request class](@ref openvino_docs_ov_plugin_dg_async_infer_request):
- Wraps the [Inference Request](@ref openvino_docs_ov_plugin_dg_infer_request) class and runs pipeline stages in parallel on several task executors based on a device-specific pipeline structure.
5. [Remote Context](@ref openvino_docs_ov_plugin_dg_remote_context):
5. [Plugin specific properties](@ref openvino_docs_ov_plugin_dg_properties):
- Provides the plugin specific properties.
6. [Remote Context](@ref openvino_docs_ov_plugin_dg_remote_context):
- Provides the device specific remote context. Context allows to create remote tensors.
6. [Remote Tensor](@ref openvino_docs_ov_plugin_dg_remote_tensor)
7. [Remote Tensor](@ref openvino_docs_ov_plugin_dg_remote_tensor)
- Provides the device specific remote tensor API and implementation.
> **NOTE**: This documentation is written based on the `Template` plugin, which demonstrates plugin

1
docs/IE_PLUGIN_DG/Plugin.md
View File

@ -42,6 +42,7 @@ As an example, a plugin configuration has three value parameters:
- `perf_counts` - boolean value to identify whether to collect performance counters during [Inference Request](@ref openvino_docs_ov_plugin_dg_infer_request) execution.
- `streams_executor_config` - configuration of `ov::threading::IStreamsExecutor` to handle settings of multi-threaded context.
- `performance_mode` - configuration of `ov::hint::PerformanceMode` to set the performance mode.
- `disable_transformations` - allows to disable transformations which are applied in the process of model compilation.
### Plugin Constructor

10
docs/IE_PLUGIN_DG/Properties.md Normal file
View File

@ -0,0 +1,10 @@
# Plugin Properties {#openvino_docs_ov_plugin_dg_properties}
Plugin can provide own device specific properties.
Property Class
------------------------
OpenVINO API provides the interface ov::Property which allows to define the property and access rights. Based on that, a declaration of plugin specific properties can look as follows:
@snippet include/template/properties.hpp properties:public_header

1
docs/IE_PLUGIN_DG/layout.xml
View File

@ -79,6 +79,7 @@
</tab>
<tab type="user" url="@ref infer_request" visibile="yes" title="Implement Synchronous Inference Request"/>
<tab type="user" url="@ref async_infer_request" visibile="yes" title="Implement Asynchronous Inference Request"/>
<tab type="user" url="@ref properties" visibile="yes" title="Provide Plugin Specific Properties"/>
<tab type="user" url="@ref remote_context" visibile="yes" title="Implement Remote Context"/>
<tab type="user" url="@ref remote_tensor" visibile="yes" title="Implement Remote Tensor"/>
</tab>

12
src/plugins/template/include/template/config.hpp → src/plugins/template/include/template/properties.hpp
View File

@ -3,10 +3,10 @@
//
/**
* @brief A header that defines advanced related properties for DLIA plugins.
* @brief A header that defines advanced related properties for Template plugin.
* These properties should be used in set_property() and compile_model() methods of plugins
*
* @file template/config.hpp
* @file template/properties.hpp
*/
#pragma once
@ -18,14 +18,14 @@
namespace ov {
namespace template_plugin {
// ! [public_header:properties]
// ! [properties:public_header]
/**
* @brief Defines the number of throutput streams used by TEMPLATE plugin.
* @brief Allows to disable all transformations for execution inside the TEMPLATE plugin.
*/
static constexpr Property<uint32_t, PropertyMutability::RW> throughput_streams{"THROUGHPUT_STREAMS"};
static constexpr Property<bool, PropertyMutability::RW> disable_transformations{"DISABLE_TRANSFORMATIONS"};
// ! [public_header:properties]
// ! [properties:public_header]
} // namespace template_plugin
} // namespace ov

7
src/plugins/template/src/compiled_model.cpp
View File

@ -12,7 +12,6 @@
#include "itt.hpp"
#include "openvino/runtime/properties.hpp"
#include "plugin.hpp"
#include "template/config.hpp"
#include "transformations/utils/utils.hpp"
// ! [compiled_model:ctor]
@ -47,6 +46,8 @@ ov::template_plugin::CompiledModel::CompiledModel(const std::shared_ptr<ov::Mode
void transform_model(const std::shared_ptr<ov::Model>& model);
void ov::template_plugin::CompiledModel::compile_model(const std::shared_ptr<ov::Model>& model) {
if (m_cfg.disable_transformations)
return;
// apply plugins transformations
transform_model(model);
// Perform any other steps like allocation and filling backend specific memory handles and so on
@ -107,9 +108,7 @@ ov::Any ov::template_plugin::CompiledModel::get_property(const std::string& name
return ro_properties;
};
const auto& default_rw_properties = []() {
std::vector<ov::PropertyName> rw_properties{ov::device::id,
ov::enable_profiling,
ov::template_plugin::throughput_streams};
std::vector<ov::PropertyName> rw_properties{ov::device::id, ov::enable_profiling};
return rw_properties;
};
const auto& to_string_vector = [](const std::vector<ov::PropertyName>& properties) {

11
src/plugins/template/src/config.cpp
View File

@ -7,7 +7,8 @@
#include <cpp_interfaces/interface/ie_internal_plugin_config.hpp>
#include <ie_plugin_config.hpp>
#include "template/config.hpp"
#include "openvino/runtime/properties.hpp"
#include "template/properties.hpp"
using namespace ov::template_plugin;
@ -22,8 +23,8 @@ Configuration::Configuration(const ov::AnyMap& config, const Configuration& defa
const auto& key = c.first;
const auto& value = c.second;
if (ov::template_plugin::throughput_streams == key) {
streams_executor_config.set_property(CONFIG_KEY(CPU_THROUGHPUT_STREAMS), value);
if (ov::template_plugin::disable_transformations == key) {
disable_transformations = value.as<bool>();
} else if (streamExecutorConfigKeys.end() !=
std::find(std::begin(streamExecutorConfigKeys), std::end(streamExecutorConfigKeys), key)) {
streams_executor_config.set_property(key, value);
@ -51,7 +52,9 @@ ov::Any Configuration::Get(const std::string& name) const {
return {std::to_string(device_id)};
} else if (name == CONFIG_KEY(PERF_COUNT)) {
return {perf_count};
} else if (name == ov::template_plugin::throughput_streams || name == CONFIG_KEY(CPU_THROUGHPUT_STREAMS)) {
} else if (name == ov::template_plugin::disable_transformations) {
return {disable_transformations};
} else if (name == ov::num_streams) {
return {std::to_string(streams_executor_config._streams)};
} else if (name == CONFIG_KEY(CPU_BIND_THREAD)) {
return streams_executor_config.get_property(name);

1
src/plugins/template/src/config.hpp
View File

@ -34,6 +34,7 @@ struct Configuration {
bool perf_count = true;
ov::threading::IStreamsExecutor::Config streams_executor_config;
ov::hint::PerformanceMode performance_mode = ov::hint::PerformanceMode::UNDEFINED;
bool disable_transformations = false;
};
// ! [configuration:header]

7
src/plugins/template/src/plugin.cpp
View File

@ -12,7 +12,7 @@
#include "openvino/pass/manager.hpp"
#include "openvino/runtime/properties.hpp"
#include "remote_context.hpp"
#include "template/config.hpp"
#include "template/properties.hpp"
#include "transformations/common_optimizations/common_optimizations.hpp"
#include "transformations/common_optimizations/convert_compression_only_to_legacy.hpp"
#include "transformations/control_flow/unroll_if.hpp"
@ -171,6 +171,9 @@ ov::SupportedOpsMap ov::template_plugin::Plugin::query_model(const std::shared_p
auto supported = ov::get_supported_nodes(
model,
[&](std::shared_ptr<ov::Model>& model) {
// skip transformations in case of user config
if (fullConfig.disable_transformations)
return;
// 1. It is needed to apply all transformations as it is done in compile_model
transform_model(model);
},
@ -228,7 +231,7 @@ ov::Any ov::template_plugin::Plugin::get_property(const std::string& name, const
std::vector<ov::PropertyName> rw_properties{ov::device::id,
ov::enable_profiling,
ov::hint::performance_mode,
ov::template_plugin::throughput_streams};
ov::template_plugin::disable_transformations};
return rw_properties;
};
const auto& to_string_vector = [](const std::vector<ov::PropertyName>& properties) {

10
src/plugins/template/tests/functional/shared_tests_instances/behavior/plugin/configuration_tests.cpp
View File

@ -4,7 +4,7 @@
#include "behavior/plugin/configuration_tests.hpp"
#include <template/config.hpp>
#include "openvino/runtime/properties.hpp"
using namespace BehaviorTestsDefinitions;
@ -14,13 +14,13 @@ const std::vector<InferenceEngine::Precision> netPrecisions = {InferenceEngine::
InferenceEngine::Precision::FP16};
const std::vector<std::map<std::string, std::string>> configs = {
{{ov::template_plugin::throughput_streams.name(), InferenceEngine::PluginConfigParams::CPU_THROUGHPUT_AUTO}},
{{ov::template_plugin::throughput_streams.name(), InferenceEngine::PluginConfigParams::CPU_THROUGHPUT_NUMA}},
{{ov::template_plugin::throughput_streams.name(), "8"}},
{{ov::num_streams.name(), InferenceEngine::PluginConfigParams::CPU_THROUGHPUT_AUTO}},
{{ov::num_streams.name(), InferenceEngine::PluginConfigParams::CPU_THROUGHPUT_NUMA}},
{{ov::num_streams.name(), "8"}},
};
const std::vector<std::map<std::string, std::string>> inconfigs = {
{{ov::template_plugin::throughput_streams.name(), CONFIG_VALUE(NO)}},
{{ov::num_streams.name(), CONFIG_VALUE(NO)}},
};
INSTANTIATE_TEST_SUITE_P(smoke_BehaviorTests,

51
src/plugins/template/tests/functional/transformations/disable_transformations_test.cpp Normal file
View File

@ -0,0 +1,51 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include <gtest/gtest.h>
#include <memory>
#include <queue>
#include <string>
#include "common_test_utils/graph_comparator.hpp"
#include "common_test_utils/ngraph_test_utils.hpp"
#include "functional_test_utils/ov_plugin_cache.hpp"
#include "openvino/opsets/opset11.hpp"
#include "template/properties.hpp"
TEST(DisableTransformationsTests, TestTemplatePluginProperty) {
std::shared_ptr<ov::Model> m(nullptr), m_ref(nullptr);
{
auto data = std::make_shared<ov::opset11::Parameter>(ov::element::f32, ov::Shape{3, 1, 2});
auto like = ov::opset11::Constant::create(ov::element::i32, ov::Shape{1}, {1});
auto cvtlike = std::make_shared<ov::opset11::ConvertLike>(data, like);
m = std::make_shared<ov::Model>(ov::NodeVector{cvtlike}, ov::ParameterVector{data});
}
{
auto data = std::make_shared<ov::opset11::Parameter>(ov::element::f32, ov::Shape{3, 1, 2});
auto cvt = std::make_shared<ov::opset11::Convert>(data, ov::element::i32);
m_ref = std::make_shared<ov::Model>(ov::NodeVector{cvt}, ov::ParameterVector{data});
}
auto core = ov::test::utils::PluginCache::get().core("TEMPLATE");
auto transformed_comp_model = core->compile_model(m, "TEMPLATE");
auto no_transformed_comp_model =
core->compile_model(m, "TEMPLATE", ov::template_plugin::disable_transformations(true));
// Clone is needed only for comparison
auto transformed_model = transformed_comp_model.get_runtime_model()->clone();
auto no_transformed_model = no_transformed_comp_model.get_runtime_model()->clone();
auto res = compare_functions(m, m_ref);
ASSERT_FALSE(res.first);
res = compare_functions(transformed_model, no_transformed_model);
ASSERT_FALSE(res.first);
res = compare_functions(transformed_model, m_ref);
ASSERT_TRUE(res.first) << res.second;
res = compare_functions(no_transformed_model, m);
ASSERT_TRUE(res.first) << res.second;
}

183
src/plugins/template/tests/functional/transformations/preprocessing.cpp
View File

@ -1,183 +0,0 @@
// // Copyright (C) 2018-2023 Intel Corporation
// // SPDX-License-Identifier: Apache-2.0
// //
// #include <gtest/gtest.h>
// #include <string>
// #include <memory>
// #include <map>
// #include <ngraph/function.hpp>
// #include <ngraph/opsets/opset5.hpp>
// #include <ngraph/pass/manager.hpp>
// #include <transformations/init_node_info.hpp>
// #include <transformations/preprocessing/std_scale.hpp>
// #include <transformations/preprocessing/mean_image_or_value.hpp>
// #include "common_test_utils/ngraph_test_utils.hpp"
// using namespace testing;
// using namespace ngraph;
// TEST(TransformationTests, Preprocessing_AddStdScale) {
// std::shared_ptr<Function> f(nullptr), f_ref(nullptr);
// const Shape data_shape{1, 3, 14, 14};
// const Shape scale_shape{3, 1, 1};
// {
// auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
// auto relu = std::make_shared<opset5::Relu>(data);
// f = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
// auto scales = opset5::Constant::create(element::f32, scale_shape,
// std::vector<float>(shape_size(scale_shape), 2.0f));
// pass::Manager m;
// m.register_pass<pass::InitNodeInfo>();
// m.register_pass<pass::AddStdScale>(pass::AddStdScale::ScaleMap{ { data->get_friendly_name(), scales } });
// m.run_passes(f);
// }
// {
// auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
// auto scales = opset5::Constant::create(element::f32, scale_shape,
// std::vector<float>(shape_size(scale_shape), 2.0f));
// auto div = std::make_shared<opset5::Divide>(data, scales);
// auto relu = std::make_shared<opset5::Relu>(div);
// f_ref = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
// }
// auto res = compare_functions(f, f_ref);
// ASSERT_TRUE(res.first) << res.second;
// }
// TEST(TransformationTests, Preprocessing_AddMeanValue) {
// std::shared_ptr<Function> f(nullptr), f_ref(nullptr);
// const Shape data_shape{1, 3, 14, 14};
// const Shape mean_shape{3, 1, 1};
// {
// auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
// auto relu = std::make_shared<opset5::Relu>(data);
// f = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
// auto meanValues = opset5::Constant::create(element::f32, mean_shape,
// std::vector<float>(shape_size(mean_shape), 2.0f));
// pass::Manager m;
// m.register_pass<pass::InitNodeInfo>();
// m.register_pass<pass::AddMeanSubtract>(pass::AddMeanSubtract::MeanMap{ { data->get_friendly_name(), meanValues } });
// m.run_passes(f);
// }
// {
// auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
// auto meanValues = opset5::Constant::create(element::f32, mean_shape,
// std::vector<float>(shape_size(mean_shape), 2.0f));
// auto sub = std::make_shared<opset5::Subtract>(data, meanValues);
// auto relu = std::make_shared<opset5::Relu>(sub);
// f_ref = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
// }
// auto res = compare_functions(f, f_ref);
// ASSERT_TRUE(res.first) << res.second;
// }
// TEST(TransformationTests, Preprocessing_AddMeanImage) {
// std::shared_ptr<Function> f(nullptr), f_ref(nullptr);
// const Shape data_shape{1, 3, 14, 14};
// const Shape mean_shape{3, 14, 14};
// {
// auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
// auto relu = std::make_shared<opset5::Relu>(data);
// f = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
// auto meanValues = opset5::Constant::create(element::f32, mean_shape,
// std::vector<float>(shape_size(mean_shape), 2.0f));
// pass::Manager m;
// m.register_pass<pass::InitNodeInfo>();
// m.register_pass<pass::AddMeanSubtract>(pass::AddMeanSubtract::MeanMap{ { data->get_friendly_name(), meanValues } });
// m.run_passes(f);
// }
// {
// auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
// auto meanValues = opset5::Constant::create(element::f32, mean_shape,
// std::vector<float>(shape_size(mean_shape), 2.0f));
// auto sub = std::make_shared<opset5::Subtract>(data, meanValues);
// auto relu = std::make_shared<opset5::Relu>(sub);
// f_ref = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
// }
// auto res = compare_functions(f, f_ref);
// ASSERT_TRUE(res.first) << res.second;
// }
// TEST(TransformationTests, Preprocessing_AddMeanImageAndScale) {
// std::shared_ptr<Function> f(nullptr), f_ref(nullptr);
// const Shape data_shape{1, 3, 14, 14};
// const Shape mean_shape{3, 14, 14};
// const Shape scale_shape{3, 1, 1};
// {
// auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
// auto relu = std::make_shared<opset5::Relu>(data);
// f = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
// auto meanValues = opset5::Constant::create(element::f32, mean_shape,
// std::vector<float>(shape_size(mean_shape), 2.0f));
// auto scaleValues = opset5::Constant::create(element::f32, scale_shape,
// std::vector<float>(shape_size(scale_shape), 2.0f));
// pass::Manager m;
// m.register_pass<pass::InitNodeInfo>();
// m.register_pass<pass::AddStdScale>(pass::AddStdScale::ScaleMap{ { data->get_friendly_name(), scaleValues } });
// m.register_pass<pass::AddMeanSubtract>(pass::AddMeanSubtract::MeanMap{ { data->get_friendly_name(), meanValues } });
// m.run_passes(f);
// }
// {
// auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
// auto meanValues = opset5::Constant::create(element::f32, mean_shape,
// std::vector<float>(shape_size(mean_shape), 2.0f));
// auto scaleValues = opset5::Constant::create(element::f32, scale_shape,
// std::vector<float>(shape_size(scale_shape), 2.0f));
// auto sub = std::make_shared<opset5::Subtract>(data, meanValues);
// auto div = std::make_shared<opset5::Divide>(sub, scaleValues);
// auto relu = std::make_shared<opset5::Relu>(div);
// f_ref = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
// }
// auto res = compare_functions(f, f_ref);
// ASSERT_TRUE(res.first) << res.second;
// }
// TEST(TransformationTests, Preprocessing_AddMeanValueAndScale) {
// std::shared_ptr<Function> f(nullptr), f_ref(nullptr);
// const Shape data_shape{1, 3, 14, 14};
// const Shape mean_shape{3, 1, 1};
// const Shape scale_shape{3, 1, 1};
// {
// auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
// auto relu = std::make_shared<opset5::Relu>(data);
// f = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
// auto meanValues = opset5::Constant::create(element::f32, mean_shape,
// std::vector<float>(shape_size(mean_shape), 2.0f));
// auto scaleValues = opset5::Constant::create(element::f32, scale_shape,
// std::vector<float>(shape_size(scale_shape), 2.0f));
// pass::Manager m;
// m.register_pass<pass::InitNodeInfo>();
// m.register_pass<pass::AddStdScale>(pass::AddStdScale::ScaleMap{ { data->get_friendly_name(), scaleValues } });
// m.register_pass<pass::AddMeanSubtract>(pass::AddMeanSubtract::MeanMap{ { data->get_friendly_name(), meanValues } });
// m.run_passes(f);
// }
// {
// auto data = std::make_shared<opset5::Parameter>(element::f32, data_shape);
// auto meanValues = opset5::Constant::create(element::f32, mean_shape,
// std::vector<float>(shape_size(mean_shape), 2.0f));
// auto scaleValues = opset5::Constant::create(element::f32, scale_shape,
// std::vector<float>(shape_size(scale_shape), 2.0f));
// auto sub = std::make_shared<opset5::Subtract>(data, meanValues);
// auto div = std::make_shared<opset5::Divide>(sub, meanValues);
// auto relu = std::make_shared<opset5::Relu>(div);
// f_ref = std::make_shared<Function>(NodeVector{relu}, ParameterVector{data});
// }
// auto res = compare_functions(f, f_ref);
// ASSERT_TRUE(res.first) << res.second;
// }

58
src/plugins/template/tests/functional/transformations/template_transformations_test.cpp
View File

@ -1,58 +0,0 @@
// Copyright (C) 2018-2023 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include <gtest/gtest.h>
#include <string>
#include <memory>
#include <queue>
#include <ngraph/ngraph.hpp>
#include <ngraph/opsets/opset3.hpp>
#include <transformations/init_node_info.hpp>
#include <transformations/utils/utils.hpp>
#include "common_test_utils/ngraph_test_utils.hpp"
using namespace testing;
// ! [transformation:test]
TEST(TransformationTests, DISABLED_TemplateTest) {
std::shared_ptr<ngraph::Function> f, f_ref;
// f - ngraph::Function for applying transformation
// f_ref - ngraph::Function that is expected after applying transformation
{
// Example function
auto data = std::make_shared<ngraph::opset3::Parameter>(ngraph::element::f32, ngraph::Shape{3, 1, 2});
auto divide_constant = ngraph::opset3::Constant::create(ngraph::element::f32, ngraph::Shape{1}, {1.5});
auto divide = std::make_shared<ngraph::opset3::Divide>(data, divide_constant);
f = std::make_shared<ngraph::Function>(ngraph::NodeVector{divide}, ngraph::ParameterVector{data});
// This transformation init runtime info attributes
ov::pass::InitNodeInfo().run_on_model(f);
// Run transformation
// ngraph::pass::MyTransformation().run_on_function(f);
// Check that after applying transformation all runtime info attributes was correctly propagated
ASSERT_NO_THROW(check_rt_info(f));
}
{
// Example reference function
auto data = std::make_shared<ngraph::opset3::Parameter>(ngraph::element::f32, ngraph::Shape{3, 1, 2});
auto divide_constant = ngraph::opset3::Constant::create(ngraph::element::f32, ngraph::Shape{1}, {1.5});
auto pow = std::make_shared<ngraph::opset3::Power>(divide_constant,
ngraph::opset3::Constant::create(ngraph::element::f32, ngraph::Shape{1}, {-1}));
auto mul = std::make_shared<ngraph::opset3::Multiply>(data, pow);
f_ref = std::make_shared<ngraph::Function>(ngraph::NodeVector{mul}, ngraph::ParameterVector{data});
}
// Compare that processed function and expected function are the same
auto res = compare_functions(f, f_ref);
ASSERT_TRUE(res.first) << res.second;
}
// ! [transformation:test]