IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[IE] Convert to unsigned NMS:0 ->Gather path (#6474)

Browse Source 
* inserted Convert to unsigned

* moved declarations from hpp into cpp, specification corrected

* added static const modifier

* updated convert specification

* minor corrections

* split into 3 passes(Init, Propogate, Update), renamed final pass to ConvertNmsGatherPathToUnsigned

* added description why transformation is needed

* added matcher for several NMS versions, removed TRANSFORMATIONS_API macros from cpp

* applied comments:
- used GraphRewrite instead of FunctionPass
- simplified some expressions
- corrected case when Converts output goes to multiple nodes
- added to MOC transformations
- other minor corrections

* removed redundant namespace prefixes

* fixed #include <ngraph/pass/graph_rewrite.hpp>

* removed matcher_scope, debug code, and redundant dynamic_cast
This commit is contained in:
Pavel Esir 2021-08-23 18:35:37 +03:00 committed by GitHub
parent b18d0105a1
commit 29193982d4
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
7 changed files with 328 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
docs/ops/movement/Gather_8.md
View File

@ -6,7 +6,8 @@
**Short description**: *Gather* operation takes slices of data of the first input tensor according to the indices
specified with the second input tensor and axis from the third input. Semantics of this operation is identical to
TensorFlow\* [Gather](https://www.tensorflow.org/api_docs/python/tf/gather) operation.
TensorFlow\* [Gather](https://www.tensorflow.org/api_docs/python/tf/gather) operation but also includes
support of negative indices.
**Detailed description**
@ -15,6 +16,8 @@ TensorFlow\* [Gather](https://www.tensorflow.org/api_docs/python/tf/gather) oper
Where `data`, `indices` and `axis` are tensors from first, second and third inputs correspondingly, `b` is
the number of batch dimensions. `N` and `M` are numbers of dimensions of `data` and `indices` tensors, respectively.
Allowed values for indices are in the range `[-data.shape[axis], data.shape[axis] - 1]`. If index value exceed allowed
range output data for corresponding index will be filled with zeros (Example 7).
**Attributes**:
* *batch_dims*
@ -142,13 +145,23 @@ data = [1, 2, 3, 4, 5]
output = [1, 4, 5]
```
Example 7 with indices out of the range:
```
batch_dims = 0
axis = 0
indices = [3, 10, -20]
data = [1, 2, 3, 4, 5]
output = [4, 0, 0]
```
**Inputs**
* **1**: `data` tensor of type *T* with arbitrary data. **Required.**
* **2**: `indices` tensor of type *T_IND* with indices to gather. 0D tensor (scalar) for indices is also allowed.
The values for indices are in the range `[-data[axis], data[axis] - 1]`.
Negative values of indices indicate reverse indexing from `data[axis]`.
The values for indices are in the range `[-data.shape[axis], data.shape[axis] - 1]`.
Negative values of indices indicate reverse indexing from `data.shape[axis]`.
**Required.**
* **3**: Scalar or 1D tensor `axis` of *T_AXIS* type is a dimension index to gather data from. For example,

11
docs/ops/type/Convert_1.md
View File

@ -8,9 +8,16 @@
**Detailed description**
Conversion from one supported type to another supported type is always allowed. User must be aware of precision loss and value change caused by range difference between two types. For example, a 32-bit float `3.141592` may be round to a 32-bit int `3`. The result of unsupported conversions is undefined, e.g. conversion of negative signed integer value to any unsigned integer type.
Conversion from one supported type to another supported type is always allowed. User must be aware of precision loss
and value change caused by range difference between two types. For example, a 32-bit float `3.141592` may be round
to a 32-bit int `3`.
Output elements are represented as follows:
Conversion of negative signed integer to unsigned integer value happens in accordance with c++ standard. Notably,
result is the unique value of the destination unsigned type that is congruent to the source integer modulo 2^N (where
N is the bit width of the destination type). For example, when an int32 value `-1` is converted to uint32 the result
will be `uint32 max` which is `4,294,967,295`.
The result of unsupported conversions is undefined. Output elements are represented as follows:
\f[
o_{i} = Convert(a_{i})

3
inference-engine/src/offline_transformations/src/moc_transformations.cpp
View File

@ -26,6 +26,7 @@
#include <transformations/common_optimizations/dilated_convolution_converter.hpp>
#include <transformations/common_optimizations/binarize_weights.hpp>
#include <transformations/common_optimizations/conv_to_binary_conv.hpp>
#include <transformations/common_optimizations/convert_nms_gather_path_to_unsigned.hpp>
#include <transformations/common_optimizations/eliminate_unsqueeze_gather.hpp>
#include <transformations/common_optimizations/split_squeeze_concat_fusion.hpp>
#include <transformations/common_optimizations/transpose_sinking.hpp>
@ -60,6 +61,8 @@ bool ngraph::pass::MOCTransformations::run_on_function(std::shared_ptr<ngraph::F
manager.register_pass<ngraph::pass::RemoveFilteringBoxesBySize>();
manager.register_pass<ngraph::pass::ConvertQuantizeDequantize>();
manager.register_pass<ngraph::pass::SimplifyShapeOfSubGraph>();
// workaround until dynamism in NMS is not supported
manager.register_pass<ngraph::pass::ConvertNmsGatherPathToUnsigned>();
auto transpose_sinking = manager.register_pass<ngraph::pass::GraphRewrite>();
transpose_sinking->add_matcher<ngraph::pass::TransposeSinking>();

32
inference-engine/src/transformations/include/transformations/common_optimizations/convert_nms_gather_path_to_unsigned.hpp Normal file
View File

@ -0,0 +1,32 @@
// Copyright (C) 2018-2021 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#pragma once
#include <transformations_visibility.hpp>
#include <ngraph/pass/graph_rewrite.hpp>
namespace ngraph {
namespace pass {
class TRANSFORMATIONS_API ConvertNmsGatherPathToUnsigned;
} // namespace pass
} // namespace ngraph
/**
* @ingroup ie_transformation_common_api
* @brief Converts Gather indices to unsigned if indices are from NMS selected indices output.
* NMS returns -1 for not selected boxes, old version of Gather fill corresponding
* output for such indices with zero.
* But new Gather-8 has support of negative indices indicating counting from the end.
* In order to keep such behaviour (until dynamism is not supported) instead of -1 new
* Gather-8 will accept UINT32_MAX which is always outside of the bounds
* and corresponding output for such indices in gather always will be filled with zeros.
*/
class ngraph::pass::ConvertNmsGatherPathToUnsigned: public ngraph::pass::GraphRewrite {
public:
NGRAPH_RTTI_DECLARATION;
ConvertNmsGatherPathToUnsigned();
};

3
inference-engine/src/transformations/src/transformations/common_optimizations/common_optimizations.cpp
View File

@ -44,6 +44,7 @@
#include "transformations/common_optimizations/split_squeeze_concat_fusion.hpp"
#include "transformations/common_optimizations/transpose_to_reshape.hpp"
#include "transformations/common_optimizations/strides_optimization.hpp"
#include "transformations/common_optimizations/convert_nms_gather_path_to_unsigned.hpp"
#include "transformations/op_conversions/bidirectional_sequences_decomposition.hpp"
#include "transformations/op_conversions/convert_pad_to_group_conv.hpp"
#include "transformations/op_conversions/convert_divide.hpp"
@ -66,7 +67,6 @@
#include "transformations/op_conversions/reduce_l1_decomposition.hpp"
#include "transformations/op_conversions/reduce_l2_decomposition.hpp"
#include "transformations/op_conversions/hswish_decomposition.hpp"
#include "transformations/op_conversions/convert_previous_nms_to_nms_5.hpp"
#include "transformations/op_conversions/hsigmoid_decomposition.hpp"
#include "transformations/op_conversions/log_softmax_decomposition.hpp"
#include "transformations/op_conversions/mvn6_decomposition.hpp"
@ -91,6 +91,7 @@ bool ngraph::pass::CommonOptimizations::run_on_function(std::shared_ptr<ngraph::
manager.register_pass<ngraph::pass::SimplifyShapeOfSubGraph>();
manager.register_pass<ngraph::pass::ConstantFolding>();
manager.register_pass<ngraph::pass::RemoveFilteringBoxesBySize>(); // Resolves dynamism (replaces NonZero), CF needed
manager.register_pass<ngraph::pass::ConvertNmsGatherPathToUnsigned>(); // workaround until dynamism in NMS is not supported
// TODO: move to KMB
manager.register_pass<ngraph::pass::ConvertQuantizeDequantize>();

128
inference-engine/src/transformations/src/transformations/common_optimizations/convert_nms_gather_path_to_unsigned.cpp Normal file
View File

@ -0,0 +1,128 @@
// Copyright (C) 2018-2021 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include "transformations/common_optimizations/convert_nms_gather_path_to_unsigned.hpp"
#include <ngraph/pattern/op/wrap_type.hpp>
#include <ngraph/opsets/opset1.hpp>
#include <ngraph/opsets/opset3.hpp>
#include <ngraph/opsets/opset5.hpp>
#include <ngraph/opsets/opset8.hpp>
#include <ngraph/op/util/broadcast_base.hpp>
#include <ngraph/op/util/gather_base.hpp>
#include <ngraph/rt_info.hpp>
#include <memory>
#include "itt.hpp"
#include "ngraph/node.hpp"
using namespace ngraph;
using namespace std;
class InitNMSPath: public pass::MatcherPass {
public:
NGRAPH_RTTI_DECLARATION;
InitNMSPath() {
MATCHER_SCOPE(InitNMSPath);
auto nms_pattern = pattern::wrap_type<opset1::NonMaxSuppression,
opset3::NonMaxSuppression,
opset5::NonMaxSuppression>();
matcher_pass_callback callback = [=](pattern::Matcher &m) {
const auto& out_nodes = m.get_match_root()->output(0).get_target_inputs();
for (const auto& out_node : out_nodes) {
auto& out_rt_info = out_node.get_node()->get_rt_info();
out_rt_info["NMS_SELECTED_INDICES"] = make_shared<VariantWrapper<string>>("");
}
return true;
};
auto m = make_shared<pattern::Matcher>(nms_pattern, matcher_name);
register_matcher(m, callback);
}
};
NGRAPH_RTTI_DEFINITION(InitNMSPath, "InitNMSPath", 0);
class PropagateNMSPath: public pass::MatcherPass {
public:
NGRAPH_RTTI_DECLARATION;
PropagateNMSPath(){
MATCHER_SCOPE(PropagateNMSPath);
auto node_pattern = pattern::wrap_type<
opset8::Squeeze,
opset8::Unsqueeze,
opset8::Reshape,
op::util::BroadcastBase,
opset8::StridedSlice,
opset8::VariadicSplit,
opset8::Concat,
opset8::Convert>();
matcher_pass_callback callback = [=](pattern::Matcher &m) {
auto node = m.get_match_root();
const auto & inputs = node->input_values();
if (any_of(inputs.begin(), inputs.end(), [](const Output<Node> & output) {
return output.get_node()->get_rt_info().count("NMS_SELECTED_INDICES");
})) {
auto & rt_info = node->get_rt_info();
rt_info["NMS_SELECTED_INDICES"] = make_shared<VariantWrapper<string>>("");
}
return true;
};
auto m = make_shared<pattern::Matcher>(node_pattern, matcher_name);
register_matcher(m, callback);
}
};
NGRAPH_RTTI_DEFINITION(PropagateNMSPath, "PropagateNMSPath", 0);
class UpdateConvertGather: public pass::MatcherPass {
public:
NGRAPH_RTTI_DECLARATION;
UpdateConvertGather(){
MATCHER_SCOPE(UpdateConvertGather);
auto node_pattern = pattern::wrap_type<op::util::GatherBase>();
matcher_pass_callback callback = [=](pattern::Matcher &m) {
auto gather = m.get_match_root();
auto indices = gather->input_value(1);
const auto& rt_info = indices.get_node()->get_rt_info();
if (!rt_info.count("NMS_SELECTED_INDICES"))
return false;
auto out_type = (indices.get_element_type() == element::i64 ? element::u64 : element::u32);
auto existing_convert = dynamic_pointer_cast<opset8::Convert>(indices.get_node_shared_ptr());
if (existing_convert && indices.get_target_inputs().size() == 1) {
existing_convert->set_convert_element_type(out_type);
existing_convert->validate_and_infer_types();
} else {
auto new_convert_to_unsigned = make_shared<opset8::Convert>(indices, out_type);
gather->input(1).replace_source_output(new_convert_to_unsigned);
copy_runtime_info(gather, new_convert_to_unsigned);
}
return true;
};
auto m = make_shared<pattern::Matcher>(node_pattern, matcher_name);
register_matcher(m, callback);
}
};
NGRAPH_RTTI_DEFINITION(UpdateConvertGather, "UpdateConvertGather", 0);
pass::ConvertNmsGatherPathToUnsigned::ConvertNmsGatherPathToUnsigned() {
add_matcher<InitNMSPath>();
add_matcher<PropagateNMSPath>();
add_matcher<UpdateConvertGather>();
}
NGRAPH_RTTI_DEFINITION(pass::ConvertNmsGatherPathToUnsigned, "ConvertNmsGatherPathToUnsigned", 0);

138
inference-engine/tests/functional/inference_engine/transformations/convert_nms_gather_path_to_unsigned_test.cpp Normal file
View File

@ -0,0 +1,138 @@
// Copyright (C) 2018-2021 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include <gtest/gtest.h>
#include "common_test_utils/ngraph_test_utils.hpp"
#include "ngraph/pass/visualize_tree.hpp"
#include <ngraph/function.hpp>
#include <ngraph/opsets/opset8.hpp>
#include <ngraph/pass/manager.hpp>
#include <transformations/common_optimizations/convert_nms_gather_path_to_unsigned.hpp>
#include <transformations/init_node_info.hpp>
using namespace testing;
using namespace ngraph;
using namespace std;
TEST(TransformationTests, test_convert_to_unsigned_nms_gather_1) {
// if Convert doesn't exist
shared_ptr<Function> f(nullptr), f_ref(nullptr);
{
auto boxes = make_shared<opset8::Parameter>(element::f32, Shape{1, 1000, 4});
auto scores = make_shared<opset8::Parameter>(element::f32, Shape{1, 1, 1000});
auto nms = make_shared<opset8::NonMaxSuppression>(boxes, scores);
auto begin = opset8::Constant::create(element::i32, Shape{1}, {3});
auto end = opset8::Constant::create(element::i32, Shape{1}, {4});
auto strides = opset8::Constant::create(element::i32, Shape{1}, {1});
auto ss_node = make_shared<opset8::StridedSlice>(nms->output(0), begin, end, strides, vector<int64_t>{1, 0}, vector<int64_t>{1, 0});
// squeeze can be represented as reshape
auto squeeze_node = make_shared<opset8::Reshape>(ss_node, opset8::Constant::create(element::i32, Shape{1}, {-1}), true);
// usually input to gather data goes after reshape NMS scores
auto reshape_node = make_shared<opset8::Reshape>(scores, opset8::Constant::create(element::i32, Shape{1}, {-1}), true);
auto gather = make_shared<opset8::Gather>(reshape_node, squeeze_node, opset8::Constant::create(element::i32, Shape{1}, {0}));
f = make_shared<Function>(NodeVector{gather}, ParameterVector{boxes, scores});
pass::Manager manager;
manager.register_pass<pass::InitNodeInfo>();
manager.register_pass<pass::ConvertNmsGatherPathToUnsigned>();
manager.run_passes(f);
ASSERT_NO_THROW(check_rt_info(f));
}
{
auto boxes = make_shared<opset8::Parameter>(element::f32, Shape{1, 1000, 4});
auto scores = make_shared<opset8::Parameter>(element::f32, Shape{1, 1, 1000});
auto nms = make_shared<opset8::NonMaxSuppression>(boxes, scores);
auto begin = opset8::Constant::create(element::i32, Shape{1}, {3});
auto end = opset8::Constant::create(element::i32, Shape{1}, {4});
auto strides = opset8::Constant::create(element::i32, Shape{1}, {1});
auto ss_node = make_shared<opset8::StridedSlice>(nms->output(0), begin, end, strides, vector<int64_t>{1, 0}, vector<int64_t>{1, 0});
// squeeze can be represented as reshape
auto squeeze_node = make_shared<opset8::Reshape>(ss_node, opset8::Constant::create(element::i32, Shape{1}, {-1}), true);
auto convert = make_shared<opset8::Convert>(squeeze_node, element::Type_t::u64);
auto reshape_node = make_shared<opset8::Reshape>(scores, opset8::Constant::create(element::i32, Shape{1}, {-1}), true);
auto gather = make_shared<opset8::Gather>(reshape_node, convert, opset8::Constant::create(element::i32, Shape{1}, {0}));
f_ref = make_shared<Function>(NodeVector{gather}, ParameterVector{boxes, scores});
}
auto res = compare_functions(f, f_ref);
ASSERT_TRUE(res.first) << res.second;
}
TEST(TransformationTests, test_convert_to_unsigned_nms_gather_2) {
// if Convert already exists
shared_ptr<Function> f(nullptr), f_ref(nullptr);
{
auto boxes = make_shared<opset8::Parameter>(element::f32, Shape{1, 1000, 4});
auto scores = make_shared<opset8::Parameter>(element::f32, Shape{1, 1, 1000});
auto nms = make_shared<opset8::NonMaxSuppression>(boxes, scores);
auto begin = opset8::Constant::create(element::i32, Shape{1}, {3});
auto end = opset8::Constant::create(element::i32, Shape{1}, {4});
auto strides = opset8::Constant::create(element::i32, Shape{1}, {1});
auto ss_node = make_shared<opset8::StridedSlice>(nms->output(0), begin, end, strides, vector<int64_t>{1, 0}, vector<int64_t>{1, 0});
// squeeze can be represented as reshape
auto squeeze_node = make_shared<opset8::Reshape>(ss_node, opset8::Constant::create(element::i32, Shape{1}, {-1}), true);
auto convert = make_shared<opset8::Convert>(squeeze_node, element::Type_t::i32);
// usually input to gather data goes after reshape NMS scores
auto reshape_node = make_shared<opset8::Reshape>(scores, opset8::Constant::create(element::i32, Shape{1}, {-1}), true);
auto gather = make_shared<opset8::Gather>(reshape_node, convert, opset8::Constant::create(element::i32, Shape{1}, {0}));
f = make_shared<Function>(NodeVector{gather}, ParameterVector{boxes, scores});
pass::Manager manager;
manager.register_pass<pass::InitNodeInfo>();
manager.register_pass<pass::ConvertNmsGatherPathToUnsigned>();
manager.run_passes(f);
ASSERT_NO_THROW(check_rt_info(f));
}
{
auto boxes = make_shared<opset8::Parameter>(element::f32, Shape{1, 1000, 4});
auto scores = make_shared<opset8::Parameter>(element::f32, Shape{1, 1, 1000});
auto nms = make_shared<opset8::NonMaxSuppression>(boxes, scores);
auto begin = opset8::Constant::create(element::i32, Shape{1}, {3});
auto end = opset8::Constant::create(element::i32, Shape{1}, {4});
auto strides = opset8::Constant::create(element::i32, Shape{1}, {1});
auto ss_node = make_shared<opset8::StridedSlice>(nms->output(0), begin, end, strides, vector<int64_t>{1, 0}, vector<int64_t>{1, 0});
// squeeze can be represented as reshape
auto squeeze_node = make_shared<opset8::Reshape>(ss_node, opset8::Constant::create(element::i32, Shape{1}, {-1}), true);
auto convert = make_shared<opset8::Convert>(squeeze_node, element::Type_t::u32);
auto reshape_node = make_shared<opset8::Reshape>(scores, opset8::Constant::create(element::i32, Shape{1}, {-1}), true);
auto gather = make_shared<opset8::Gather>(reshape_node, convert, opset8::Constant::create(element::i32, Shape{1}, {0}));
f_ref = make_shared<Function>(NodeVector{gather}, ParameterVector{boxes, scores});
}
auto res = compare_functions(f, f_ref);
ASSERT_TRUE(res.first) << res.second;
}
TEST(TransformationTests, test_convert_to_unsigned_nms_gather_3) {
// if NMS output goes not into Gather indices no converts should be inserted
auto boxes = make_shared<opset8::Parameter>(element::f32, Shape{1, 1000, 4});
auto scores = make_shared<opset8::Parameter>(element::f32, Shape{1, 1, 1000});
auto nms = make_shared<opset8::NonMaxSuppression>(boxes, scores);
auto gather = make_shared<opset8::Gather>(nms->output(0), opset8::Constant::create(element::i32, Shape{1}, {2}),
opset8::Constant::create(element::i32, Shape{1}, {0}));
shared_ptr<Function> f = make_shared<Function>(NodeVector{gather}, ParameterVector{boxes, scores});
pass::Manager manager;
manager.register_pass<pass::InitNodeInfo>();
manager.register_pass<pass::ConvertNmsGatherPathToUnsigned>();
manager.run_passes(f);
ASSERT_NO_THROW(check_rt_info(f));
ASSERT_EQ(count_ops_of_type<opset1::Convert>(f), 0);
}