IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Deformable convolution (#4312)

Browse Source 
* Add deformable_convolution unit tests

* add 2D unit test

* Add unit-tests for 3D deformable_convolution

* create shared test class for deformable_convolution

* create single layer test for cpu plugin

* create single layer test for serialization

* add deformable_convolution to opset1

* create empty evaluate method

* add group and deformable_group to parameters

* Create impl for generateInput method

* add deformable_val to deformable_conv paramset

* create additional unit-tests

* Remove 3D single layer tests

* Update GenerateInput and SetUp method

* Update parameters values for SLT

* Update GenerateInput method and create Validate method

* Create additional parameters set for cpu plugin SLT

* Create unit-test with padding and add it to disabled

* add interpreter unit tests to manifest

* style-apply

* create file for reference impl

* Update year of copyright and make Validate an override method

* Update parameters names for serialization single layer tests

* Update parameters names for functional single layer tests

* add failing unit tests for interpreter to manifest

* make tests parameters more readable

* style-apply

* Include deformable_convolution inside evaluates_map

* add support for groups parameter for reference impl

* style-apply

* remove DeformableConvolutionParams struct

* fix bug with filter

* fix bug with offset type

* Update interpreter manifest

* impl evalute method

* style-apply

* Update year

* Update test names

* add utils func for deformable convolution operation

* fix filter group count

* add calculation of index of offset

* Update offsets in unit test

* add support for multiple input channels

* add padding tests

* add padding support

* style apply

* update copyright year

* create validation check helper

* Update convolve function name

* update copyright year

* style-apply

* remove integer type from serialize layer tests

* add tests for deformable_groups

* fix bug with group > 1

* style-apply

* add group unit test

* create additional group tests

* fix bug with groups attribute

* Enhance dynamic shape inference of validate and infer types method

* Add type_prop unit tests

* Fix broken op create test in python api

* fix bug with shapes in group tests

* update deformable_convolution method

* add unit test with 2 groups and 2 defromable groups

* Fix code style.

* Update UT manifests with  current test status.

* Refactored backend test: names, removed duplication, add TOODs

* Add missing test cases in 'integral offsets' group.

* Fixed group attribute.

* Update interprer manifest to disable tests with integer offsets.

* Fix style.

* Remove changes in operator class.

* Revert "Enhance dynamic shape inference of validate and infer types method"

This reverts commit 2f9ce2ccd4.

* Revert "Add type_prop unit tests"

This reverts commit 944af98b8c.

* Revert "Fix broken op create test in python api"

This reverts commit 72fbfc2967.

* Fix op class.

* Convert implementation to 2D.

* Simplify implementation.

* Fix centos build.

* Reimplemented offsets handling.

* Fixed integral offsets test cases.

* Fixed deformable group attribute.

* Add bilinear interpolation.

* Refactoring regarding tolerance_bits.

* Fix groups & def_groups test case.

* Add more unit tests for group & defgroup attribute.

* Remove debug code.

* Minor refactoring.

* Add integer types to SLT.

* Revert "Add integer types to SLT."

This reverts commit 2fefe8926d.

* Add tests with real number offsets.

* Refactored bilinear_interpolation().

* Turned on SLT.

Additionally refactored and offset input set to range <0,2>.

* Update headers with short version.

* Fix SLT offests generation with int offsets.

* Add integer types to SLT.

* Fix grup + def_group test case.

* Add ticket to address IE_CPU backend test failures

* Enable real resolution for deformable values

* Add op to list of trusted operations in python script

* Fix comparison of integer expressions of different signedness compilation error

* Add comment with closing namespace and empty lines

Co-authored-by: pszmel <piotr.szmelczynski@intel.com>
Co-authored-by: ggalieroc <gabriele.galiero.casay@intel.com>
This commit is contained in:
Jozef Daniecki 2021-04-19 13:12:09 +02:00 committed by GitHub
parent b92fa8f303
commit ef70e5187c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
15 changed files with 3305 additions and 86 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

68
inference-engine/tests/functional/inference_engine/serialization/single_layer/deformable_convolution.cpp Normal file
View File

@ -0,0 +1,68 @@
// Copyright (C) 2021 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include <vector>
#include "shared_test_classes/single_layer/deformable_convolution.hpp"
using namespace LayerTestsDefinitions;
namespace {
TEST_P(DeformableConvolutionLayerTest, Serialize) {
Serialize();
}
const std::vector<InferenceEngine::Precision> netPrecisions = {
InferenceEngine::Precision::FP32, InferenceEngine::Precision::FP16,
InferenceEngine::Precision::I32, InferenceEngine::Precision::I16};
const std::vector<std::vector<size_t>> offsets = {{1, 18, 28, 28}};
const std::vector<std::vector<size_t>> filters = {{1, 1, 3, 3}};
const std::vector<std::vector<size_t>> strides = {{1, 1}};
const std::vector<std::vector<ptrdiff_t>> padBegins = {{0, 0}};
const std::vector<std::vector<ptrdiff_t>> padEnds ={{0, 0}};
const std::vector<std::vector<size_t>> dilations = {{1, 1}};
const std::vector<size_t> groups = {1};
const std::vector<size_t> defor_groups = {1};
const std::vector<size_t> numOutChannels = {1};
const auto conv2DParams_ExplicitPadding = ::testing::Combine(
::testing::ValuesIn(offsets), ::testing::ValuesIn(filters),
::testing::ValuesIn(strides), ::testing::ValuesIn(padBegins),
::testing::ValuesIn(padEnds), ::testing::ValuesIn(dilations),
::testing::ValuesIn(groups), ::testing::ValuesIn(defor_groups),
::testing::ValuesIn(numOutChannels),
::testing::Values(ngraph::op::PadType::EXPLICIT));
const auto conv2DParams_AutoPadValid = ::testing::Combine(
::testing::ValuesIn(offsets), ::testing::ValuesIn(filters),
::testing::ValuesIn(strides),
::testing::Values(std::vector<ptrdiff_t>({0, 0})),
::testing::Values(std::vector<ptrdiff_t>({0, 0})),
::testing::ValuesIn(dilations), ::testing::ValuesIn(groups),
::testing::ValuesIn(defor_groups), ::testing::ValuesIn(numOutChannels),
::testing::Values(ngraph::op::PadType::VALID));
INSTANTIATE_TEST_CASE_P(
smoke_DeformableConvolution2D_Serialization_ExplicitPadding, DeformableConvolutionLayerTest,
::testing::Combine(
conv2DParams_ExplicitPadding, ::testing::ValuesIn(netPrecisions),
::testing::Values(InferenceEngine::Precision::UNSPECIFIED),
::testing::Values(InferenceEngine::Precision::UNSPECIFIED),
::testing::Values(InferenceEngine::Layout::ANY),
::testing::Values(InferenceEngine::Layout::ANY),
::testing::Values(std::vector<size_t>({1, 1, 28, 28})),
::testing::Values(CommonTestUtils::DEVICE_CPU)),
DeformableConvolutionLayerTest::getTestCaseName);
INSTANTIATE_TEST_CASE_P(
smoke_DeformableConvolution2D__Serialization_AutoPadValid, DeformableConvolutionLayerTest,
::testing::Combine(
conv2DParams_AutoPadValid, ::testing::ValuesIn(netPrecisions),
::testing::Values(InferenceEngine::Precision::UNSPECIFIED),
::testing::Values(InferenceEngine::Precision::UNSPECIFIED),
::testing::Values(InferenceEngine::Layout::ANY),
::testing::Values(InferenceEngine::Layout::ANY),
::testing::Values(std::vector<size_t>({1, 1, 28, 28})),
::testing::Values(CommonTestUtils::DEVICE_CPU)),
DeformableConvolutionLayerTest::getTestCaseName);
} // namespace

92
inference-engine/tests/functional/plugin/cpu/shared_tests_instances/single_layer_tests/deformable_convolution.cpp Normal file
View File

@ -0,0 +1,92 @@
// Copyright (C) 2021 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include <vector>
#include "common_test_utils/test_constants.hpp"
#include "single_layer_tests/deformable_convolution.hpp"
using namespace LayerTestsDefinitions;
namespace {
const std::vector<InferenceEngine::Precision> netPrecisions = {
InferenceEngine::Precision::FP32, InferenceEngine::Precision::FP16,
InferenceEngine::Precision::I32, InferenceEngine::Precision::I16};
/* ============= 2D DeformableConvolution ============= */
const std::vector<std::vector<size_t>> deformable_vals = {{1, 18, 28, 28}};
const std::vector<std::vector<size_t>> kernels = {{1, 1, 3, 3}};
const std::vector<std::vector<size_t>> strides = {{1, 1}};
const std::vector<std::vector<ptrdiff_t>> padBegins = {{0, 0}};
const std::vector<std::vector<ptrdiff_t>> padEnds ={{0, 0}};
const std::vector<std::vector<size_t>> dilations = {{1, 1}};
const std::vector<size_t> groups = {1};
const std::vector<size_t> defor_groups = {1};
const std::vector<size_t> numOutChannels = {1, 5};
const std::vector<size_t> multiple_defor_groups = {4};
const std::vector<std::vector<size_t>> deform_vals = {{1, 200, 220, 220}};
const std::vector<std::vector<size_t>> kernel = {{64, 4, 5, 5}};
const auto deformableConv2DParams_ExplicitPadding = ::testing::Combine(
::testing::ValuesIn(deformable_vals),
::testing::ValuesIn(kernels), ::testing::ValuesIn(strides),
::testing::ValuesIn(padBegins), ::testing::ValuesIn(padEnds),
::testing::ValuesIn(dilations), ::testing::ValuesIn(groups),
::testing::ValuesIn(defor_groups), ::testing::ValuesIn(numOutChannels),
::testing::Values(ngraph::op::PadType::EXPLICIT));
const auto deformableConv2DParams_AutoPadValid = ::testing::Combine(
::testing::ValuesIn(deformable_vals),
::testing::ValuesIn(kernels), ::testing::ValuesIn(strides),
::testing::Values(std::vector<ptrdiff_t>({0, 0})),
::testing::Values(std::vector<ptrdiff_t>({0, 0})),
::testing::ValuesIn(dilations), ::testing::ValuesIn(groups),
::testing::ValuesIn(defor_groups), ::testing::ValuesIn(numOutChannels),
::testing::Values(ngraph::op::PadType::VALID));
const auto deformableConv2DParams_DeformableGroups_AutoPadExplicit = ::testing::Combine(
::testing::ValuesIn(deform_vals),
::testing::ValuesIn(kernel), ::testing::ValuesIn(strides),
::testing::Values(std::vector<ptrdiff_t>({0, 0})),
::testing::Values(std::vector<ptrdiff_t>({0, 0})),
::testing::ValuesIn(dilations), ::testing::ValuesIn(groups),
::testing::ValuesIn(multiple_defor_groups), ::testing::ValuesIn(numOutChannels),
::testing::Values(ngraph::op::PadType::EXPLICIT));
INSTANTIATE_TEST_CASE_P(
smoke_DeformableConvolution2D_ExplicitPadding, DeformableConvolutionLayerTest,
::testing::Combine(
deformableConv2DParams_ExplicitPadding, ::testing::ValuesIn(netPrecisions),
::testing::Values(InferenceEngine::Precision::UNSPECIFIED),
::testing::Values(InferenceEngine::Precision::UNSPECIFIED),
::testing::Values(InferenceEngine::Layout::ANY),
::testing::Values(InferenceEngine::Layout::ANY),
::testing::Values(std::vector<size_t>({1, 1, 30, 30})),
::testing::Values(CommonTestUtils::DEVICE_CPU)),
DeformableConvolutionLayerTest::getTestCaseName);
INSTANTIATE_TEST_CASE_P(
smoke_DeformableConvolution2D_AutoPadValid, DeformableConvolutionLayerTest,
::testing::Combine(
deformableConv2DParams_AutoPadValid, ::testing::ValuesIn(netPrecisions),
::testing::Values(InferenceEngine::Precision::UNSPECIFIED),
::testing::Values(InferenceEngine::Precision::UNSPECIFIED),
::testing::Values(InferenceEngine::Layout::ANY),
::testing::Values(InferenceEngine::Layout::ANY),
::testing::Values(std::vector<size_t>({1, 1, 30, 30})),
::testing::Values(CommonTestUtils::DEVICE_CPU)),
DeformableConvolutionLayerTest::getTestCaseName);
INSTANTIATE_TEST_CASE_P(
smoke_DeformableConvolution2D_DeformableGroups_ExplicitPadding, DeformableConvolutionLayerTest,
::testing::Combine(
deformableConv2DParams_DeformableGroups_AutoPadExplicit, ::testing::ValuesIn(netPrecisions),
::testing::Values(InferenceEngine::Precision::UNSPECIFIED),
::testing::Values(InferenceEngine::Precision::UNSPECIFIED),
::testing::Values(InferenceEngine::Layout::ANY),
::testing::Values(InferenceEngine::Layout::ANY),
::testing::Values(std::vector<size_t>({1, 4, 224, 224})),
::testing::Values(CommonTestUtils::DEVICE_CPU)),
DeformableConvolutionLayerTest::getTestCaseName);
} // namespace

15
inference-engine/tests/functional/plugin/shared/include/single_layer_tests/deformable_convolution.hpp Normal file
View File

@ -0,0 +1,15 @@
// Copyright (C) 2021 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#pragma once
#include "shared_test_classes/single_layer/deformable_convolution.hpp"
namespace LayerTestsDefinitions {
TEST_P(DeformableConvolutionLayerTest, CompareWithRefs) {
Run();
}
} // namespace LayerTestsDefinitions

53
inference-engine/tests/functional/shared_test_classes/include/shared_test_classes/single_layer/deformable_convolution.hpp Normal file
View File

@ -0,0 +1,53 @@
// Copyright (C) 2021 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#pragma once
#include <tuple>
#include <vector>
#include <string>
#include <memory>
#include "shared_test_classes/base/layer_test_utils.hpp"
#include "ngraph_functions/builders.hpp"
#include "ngraph_functions/utils/ngraph_helpers.hpp"
namespace LayerTestsDefinitions {
// ! [test_convolution:definition]
typedef std::tuple<
InferenceEngine::SizeVector, // Deformable values size
InferenceEngine::SizeVector, // Kernel size
InferenceEngine::SizeVector, // Strides
std::vector<ptrdiff_t>, // Pad begin
std::vector<ptrdiff_t>, // Pad end
InferenceEngine::SizeVector, // Dilation
size_t, // Groups
size_t, // Deformable groups
size_t, // Num out channels
ngraph::op::PadType // Padding type
> deformableConvSpecificParams;
typedef std::tuple<
deformableConvSpecificParams,
InferenceEngine::Precision, // Net precision
InferenceEngine::Precision, // Input precision
InferenceEngine::Precision, // Output precision
InferenceEngine::Layout, // Input layout
InferenceEngine::Layout, // Output layout
InferenceEngine::SizeVector, // Input shapes
LayerTestsUtils::TargetDevice // Device name
> deformableConvLayerTestParamsSet;
class DeformableConvolutionLayerTest : public testing::WithParamInterface<deformableConvLayerTestParamsSet>,
virtual public LayerTestsUtils::LayerTestsCommon {
public:
static std::string getTestCaseName(testing::TestParamInfo<deformableConvLayerTestParamsSet> obj);
InferenceEngine::Blob::Ptr GenerateInput(const InferenceEngine::InputInfo &info) const override;
protected:
void SetUp() override;
};
// ! [test_convolution:definition]
} // namespace LayerTestsDefinitions

84
inference-engine/tests/functional/shared_test_classes/src/single_layer/deformable_convolution.cpp Normal file
View File

@ -0,0 +1,84 @@
// Copyright (C) 2021 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#include "shared_test_classes/single_layer/deformable_convolution.hpp"
namespace LayerTestsDefinitions {
std::string DeformableConvolutionLayerTest::getTestCaseName(testing::TestParamInfo<deformableConvLayerTestParamsSet> obj) {
deformableConvSpecificParams convParams;
InferenceEngine::Precision netPrecision;
InferenceEngine::Precision inPrc, outPrc;
InferenceEngine::Layout inLayout, outLayout;
InferenceEngine::SizeVector inputShapes;
std::string targetDevice;
std::tie(convParams, netPrecision, inPrc, outPrc, inLayout, outLayout, inputShapes, targetDevice) =
obj.param;
ngraph::op::PadType padType;
InferenceEngine::SizeVector offsets, filter, stride, dilation;
std::vector<ptrdiff_t> padBegin, padEnd;
size_t groups, deformable_groups, convOutChannels;
std::tie(offsets, filter, stride, padBegin, padEnd, dilation, groups, deformable_groups, convOutChannels, padType) = convParams;
std::ostringstream result;
result << "IS=" << CommonTestUtils::vec2str(inputShapes) << "_";
result << "DV" << CommonTestUtils::vec2str(offsets) << "_";
result << "K" << CommonTestUtils::vec2str(filter) << "_";
result << "S" << CommonTestUtils::vec2str(stride) << "_";
result << "PB" << CommonTestUtils::vec2str(padBegin) << "_";
result << "PE" << CommonTestUtils::vec2str(padEnd) << "_";
result << "D=" << CommonTestUtils::vec2str(dilation) << "_";
result << "G=" << groups << "_";
result << "DG=" << deformable_groups << "_";
result << "O=" << convOutChannels << "_";
result << "AP=" << padType << "_";
result << "netPRC=" << netPrecision.name() << "_";
result << "inPRC=" << inPrc.name() << "_";
result << "outPRC=" << outPrc.name() << "_";
result << "inL=" << inLayout << "_";
result << "outL=" << outLayout << "_";
result << "trgDev=" << targetDevice;
return result.str();
}
InferenceEngine::Blob::Ptr DeformableConvolutionLayerTest::GenerateInput(const InferenceEngine::InputInfo &info) const {
InferenceEngine::Blob::Ptr blobPtr;
const std::string& name = info.name();
if (name == "a_data") {
blobPtr = LayerTestsUtils::LayerTestsCommon::GenerateInput(info);
} else if (name == "b_offset_vals") {
blobPtr = FuncTestUtils::createAndFillBlobFloat(info.getTensorDesc(), 2, 0, 10);
} else if (name == "c_filter_vals") {
blobPtr = LayerTestsUtils::LayerTestsCommon::GenerateInput(info);
}
return blobPtr;
}
void DeformableConvolutionLayerTest::SetUp() {
deformableConvSpecificParams convParams;
std::vector<size_t> inputShape;
InferenceEngine::Precision netPrecision;
std::tie(convParams, netPrecision, inPrc, outPrc, inLayout, outLayout, inputShape, targetDevice) =
this->GetParam();
ngraph::op::PadType padType;
InferenceEngine::SizeVector offsets, filter, stride, dilation;
std::vector<ptrdiff_t> padBegin, padEnd;
size_t groups, deformable_groups, convOutChannels;
std::tie(offsets, filter, stride, padBegin, padEnd, dilation, groups, deformable_groups, convOutChannels, padType) = convParams;
auto ngPrc = FuncTestUtils::PrecisionUtils::convertIE2nGraphPrc(netPrecision);
auto params = ngraph::builder::makeParams(ngPrc, {inputShape, offsets, filter});
auto paramOuts = ngraph::helpers::convert2OutputVector(
ngraph::helpers::castOps2Nodes<ngraph::op::Parameter>(params));
auto data = std::make_shared<ngraph::op::Parameter>(ngPrc, ngraph::Shape(inputShape));
data->set_friendly_name("a_data");
auto offset_vals = std::make_shared<ngraph::op::Parameter>(ngPrc, ngraph::Shape(offsets));
offset_vals->set_friendly_name("b_offset_vals");
auto filter_vals = std::make_shared<ngraph::op::Parameter>(ngPrc, ngraph::Shape(filter));
filter_vals->set_friendly_name("c_filter_vals");
auto deformable_conv = std::make_shared<ngraph::opset1::DeformableConvolution>(data, offset_vals, filter_vals,
stride, padBegin, padEnd, dilation, padType, groups, deformable_groups);
ngraph::ResultVector results{std::make_shared<ngraph::opset1::Result>(deformable_conv)};
function = std::make_shared<ngraph::Function>(results, ngraph::ParameterVector{data, offset_vals, filter_vals}, "deformable_convolution");
}
} // namespace LayerTestsDefinitions

1
inference-engine/tests/ie_test_utils/functional_test_utils/layer_tests_summary/summarize.py
View File

@ -38,6 +38,7 @@ verified_operations = [
'Concat-0',
'ConvertLike-1',
'Convolution-1',
'DeformableConvolution-1',
'DetectionOutput-0',
'Divide-1',
'ExperimentalDetectronDetectionOutput-6',

234
ngraph/core/reference/include/ngraph/runtime/reference/deformable_convolution.hpp Normal file
View File

@ -0,0 +1,234 @@
// Copyright (C) 2021 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
#pragma once
#include "ngraph/runtime/reference/convolution.hpp"
namespace ngraph
{
namespace runtime
{
namespace reference
{
namespace def_conv_impl
{
inline void validate_params(const Shape& in_shape,
const Shape& o_shape,
const Shape& f_shape,
const Strides& strides,
const Strides& dilations,
const CoordinateDiff& pads_begin,
const CoordinateDiff& pads_end)
{
// this implementation supports 2D deformable convolutions
NGRAPH_CHECK(in_shape.size() == 4, "Unsupported input rank: ", in_shape);
NGRAPH_CHECK(o_shape.size() == 4, "Unsupported offset rank: ", o_shape);
NGRAPH_CHECK(f_shape.size() == 4, "Unsupported kernel rank: ", f_shape);
const auto spatial_dims = in_shape.size() - 2;
NGRAPH_CHECK(strides.size() == spatial_dims,
"Strides not definied for all and only spatial dimensions");
NGRAPH_CHECK(dilations.size() == spatial_dims,
"Dilations not defined for all and only spatial dimensions");
NGRAPH_CHECK((pads_begin.size() == pads_end.size()) &&
(pads_begin.size() == spatial_dims),
"Pads not defined for all and only spatial dimensions");
}
inline Shape shape_reduce(const Shape& s) { return Shape(++s.begin(), s.end()); }
inline Shape shape_scale(Shape s, size_t groups)
{
s[0] /= groups;
return s;
}
template <typename inputType>
inline float bilinear_interpolation(const inputType* data,
const float x_idx,
const float y_idx,
const int x_size,
const int y_size)
{
const int x1 = std::max(static_cast<int>(std::floor(x_idx)), 0);
const int x2 = std::min(static_cast<int>(std::ceil(x_idx)), x_size - 1);
const int y1 = std::max(static_cast<int>(std::floor(y_idx)), 0);
const int y2 = std::min(static_cast<int>(std::ceil(y_idx)), y_size - 1);
const float distX = x_idx - x1;
const float distY = y_idx - y1;
const float value11 = data[y1 * x_size + x1];
const float value12 = data[y2 * x_size + x1];
const float value21 = data[y1 * x_size + x2];
const float value22 = data[y2 * x_size + x2];
const float value = (1 - distX) * (1 - distY) * value11 +
(1 - distX) * distY * value12 +
distX * (1 - distY) * value21 + distX * distY * value22;
return value;
}
template <typename T>
void convolve_2D_channels(const ConvolutionParams& p,
const int64_t deformable_groups,
const T* batch,
const Shape& batch_shape,
const T* offsets,
const Shape& offset_shape,
const T* filter,
const Shape& filter_shape,
T* out)
{
const int input_size_y = batch_shape[1];
const int input_size_x = batch_shape[2];
const int filter_size_y = filter_shape[1];
const int filter_size_x = filter_shape[2];
const int dilated_filter_size_y =
filter_size_y + (filter_size_y - 1) * (p.dilation[0] - 1);
const int dilated_filter_size_x =
filter_size_x + (filter_size_x - 1) * (p.dilation[1] - 1);
const int input_channel_size = shape_size(shape_reduce(batch_shape));
const int filter_channel_size = shape_size(shape_reduce(filter_shape));
const int offsets_size = shape_size(offset_shape);
const int offsets_spatial_size = shape_size(shape_reduce(offset_shape));
const int offsets_channel_size = 2 * offsets_spatial_size;
const int filter_channels_count = filter_shape[0];
int out_idx = 0;
for (int i_y = -p.pads_begin[0];
i_y <= (p.pads_end[0] + input_size_y - dilated_filter_size_y);
i_y += p.strides[0])
{
for (int i_x = -p.pads_begin[1];
i_x <= (p.pads_end[1] + input_size_x - dilated_filter_size_x);
i_x += p.strides[1])
{
auto input_channel = batch;
auto filter_channel = filter;
T sum = 0;
auto group_offsets_channel = offsets;
for (int dg = 0; dg < deformable_groups; dg++)
{
for (int fc = 0; fc < filter_channels_count / deformable_groups;
fc++)
{
auto offsets_channel = group_offsets_channel;
for (int f_y = 0; f_y < filter_size_y; ++f_y)
{
for (int f_x = 0; f_x < filter_size_x; ++f_x)
{
T y_offset = offsets_channel[out_idx];
T x_offset =
offsets_channel[offsets_spatial_size + out_idx];
T rel_i_y = i_y + (f_y * p.dilation[0]) + y_offset;
T rel_i_x = i_x + (f_x * p.dilation[1]) + x_offset;
offsets_channel += offsets_channel_size;
bool padding = !(in_range(rel_i_x, {0, input_size_x}) &&
in_range(rel_i_y, {0, input_size_y}));
if (padding)
continue;
int f_buf_idx = (f_y * filter_size_x) + f_x;
sum += bilinear_interpolation(input_channel,
rel_i_x,
rel_i_y,
input_size_x,
input_size_y) *
filter_channel[f_buf_idx];
}
}
input_channel += input_channel_size;
filter_channel += filter_channel_size;
}
group_offsets_channel += offsets_size / deformable_groups;
}
out[out_idx++] = sum;
}
}
}
} // namespace def_conv_impl
template <typename T>
void deformable_convolution(const T* in,
const T* offsets,
const T* filters,
T* out,
const Shape& in_shape,
const Shape& o_shape,
const Shape& f_shape,
const Shape& out_shape,
const Strides& strides,
const Strides& dilation,
const CoordinateDiff& pads_begin,
const CoordinateDiff& pads_end,
const int64_t groups,
const int64_t deformable_groups)
{
using namespace def_conv_impl;
validate_params(
in_shape, o_shape, f_shape, strides, dilation, pads_begin, pads_end);
// here we are converting all param types to int's to avoid arithmetic issues
// (e.g signed + unsigned) in indexes calculation later
ConvolutionParams params{strides, dilation, pads_begin, pads_end};
const size_t groups_count = static_cast<size_t>(groups);
const size_t batches_count = in_shape[in_batch_axis];
const Shape group_in_shape = shape_scale(shape_reduce(in_shape), groups);
const size_t group_in_size = shape_size(group_in_shape);
const Shape group_offset_shape = shape_scale(shape_reduce(o_shape), groups);
const size_t group_offset_size = shape_size(group_offset_shape);
const size_t group_offset_batch_size = shape_size(shape_reduce(o_shape));
const size_t deformable_groups_per_group =
std::ceil(static_cast<float>(deformable_groups) / static_cast<float>(groups));
const size_t group_filters_count = f_shape[filter_out_ch_axis] / groups;
const Shape group_filter_shape = shape_reduce(f_shape);
const size_t group_filter_size = shape_size(group_filter_shape);
const size_t out_ch_size = shape_size(shape_reduce(shape_reduce(out_shape)));
for (size_t batch_idx = 0; batch_idx < batches_count; ++batch_idx)
{
const T* group_filters = filters;
const T* group_offsets = offsets;
for (size_t group_idx = 0; group_idx < groups_count; ++group_idx)
{
for (size_t f_idx = 0; f_idx < group_filters_count; ++f_idx)
{
convolve_2D_channels(params,
deformable_groups_per_group,
in,
group_in_shape,
group_offsets,
group_offset_shape,
group_filters,
group_filter_shape,
out);
group_filters += group_filter_size;
out += out_ch_size;
}
in += group_in_size;
if (deformable_groups > 1)
{
group_offsets += (deformable_groups_per_group * group_offset_size);
}
}
offsets += group_offset_batch_size;
}
}
} // namespace reference
} // namespace runtime
} // namespace ngraph

9
ngraph/core/src/op/deformable_convolution.cpp
View File

@ -171,13 +171,18 @@ void op::v1::DeformableConvolution::validate_and_infer_types()
return;
}
}
// adjust filter shape to reuse regular infer_convolution_forward()
const auto new_filters_pshape = [&](int groups) {
auto new_shape(filters_shape);
new_shape[1] *= groups;
return new_shape;
}(m_group);
result_shape = infer_convolution_forward(this,
data_batch_shape,
Strides(m_strides.size(), 1), // dummy data dilations
m_pads_begin,
m_pads_end,
filters_shape,
new_filters_pshape,
m_strides,
m_dilations);

1
ngraph/test/CMakeLists.txt
View File

@ -325,6 +325,7 @@ set(MULTI_TEST_SRC
backend/detection_output.in.cpp
backend/dft.in.cpp
backend/divide.in.cpp
backend/deformable_convolution.in.cpp
backend/dyn_reshape.in.cpp
backend/strided_slice.in.cpp
backend/dynamic.in.cpp

2622
ngraph/test/backend/deformable_convolution.in.cpp Normal file
View File

File diff suppressed because it is too large Load Diff

11
ngraph/test/runtime/ie/unit_test.manifest
View File

@ -1669,3 +1669,14 @@ onnx_upsample6_dynamic
# random values returned from the plugin: ticket 51762
onnx_model_deformable_conv_2d
# DeformableConvolution groups attribute: ticket 53312
IE_CPU.deformable_convolution_2D_zeroed_offsets_groups_basic
IE_CPU.deformable_convolution_2D_zeroed_offsets_groups_complex
IE_CPU.deformable_convolution_2D_zeroed_offsets_groups_and_deforgroups
IE_CPU.deformable_convolution_2D_integral_offsets_groups_basic
IE_CPU.deformable_convolution_2D_integral_offsets_groups_complex
IE_CPU.deformable_convolution_2D_integral_offsets_groups_and_deforgroups
IE_CPU.deformable_convolution_2D_real_offsets_groups_basic
IE_CPU.deformable_convolution_2D_real_offsets_groups_complex
IE_CPU.deformable_convolution_2D_real_offsets_groups_and_deforgroups

32
ngraph/test/runtime/interpreter/evaluates_map.cpp
View File

@ -20,6 +20,7 @@
#include <ngraph/runtime/reference/ctc_greedy_decoder_seq_len.hpp>
#include <ngraph/runtime/reference/ctc_loss.hpp>
#include <ngraph/runtime/reference/cum_sum.hpp>
#include <ngraph/runtime/reference/deformable_convolution.hpp>
#include <ngraph/runtime/reference/detection_output.hpp>
#include <ngraph/runtime/reference/elu.hpp>
#include <ngraph/runtime/reference/embedding_bag_offsets_sum.hpp>
@ -331,6 +332,37 @@ namespace
return true;
}
template <element::Type_t ET>
bool evaluate(const shared_ptr<op::v1::DeformableConvolution>& op,
const HostTensorVector& outputs,
const HostTensorVector& inputs)
{
const auto in_data_ptr = inputs[0]->get_data_ptr<ET>();
const auto offset_data_ptr = inputs[1]->get_data_ptr<ET>();
const auto filter_data_ptr = inputs[2]->get_data_ptr<ET>();
auto out_data_ptr = outputs[0]->get_data_ptr<ET>();
const auto& out_shape = outputs[0]->get_shape();
const auto& in_shape = inputs[0]->get_shape();
const auto& offset_shape = inputs[1]->get_shape();
const auto& filter_shape = inputs[2]->get_shape();
runtime::reference::deformable_convolution<typename element_type_traits<ET>::value_type>(
in_data_ptr,
offset_data_ptr,
filter_data_ptr,
out_data_ptr,
in_shape,
offset_shape,
filter_shape,
out_shape,
op->get_strides(),
op->get_dilations(),
op->get_pads_begin(),
op->get_pads_end(),
op->get_group(),
op->get_deformable_group());
return true;
}
namespace cum_sum_v0
{
template <element::Type_t t1, element::Type_t t2>

1
ngraph/test/runtime/interpreter/opset_int_tbl.hpp
View File

@ -43,6 +43,7 @@ NGRAPH_OP(Convolution, ngraph::op::v1)
NGRAPH_OP(ConvolutionBackpropData, ngraph::op::v1)
NGRAPH_OP(GroupConvolution, ngraph::op::v1)
NGRAPH_OP(GroupConvolutionBackpropData, ngraph::op::v1)
NGRAPH_OP(DeformableConvolution, ngraph::op::v1)
NGRAPH_OP(LessEqual, op::v1)
NGRAPH_OP(LogicalAnd, op::v1)
NGRAPH_OP(LogicalOr, op::v1)

80
ngraph/test/type_prop/convolution.cpp
View File

@ -2702,83 +2702,3 @@ TEST(type_prop, conv_bprop_v1_partial_auto_padding_lower)
ASSERT_EQ(conv->get_pads_begin(), (CoordinateDiff{0, 0}));
ASSERT_EQ(conv->get_pads_end(), (CoordinateDiff{0, 0}));
}
TEST(type_prop, deformable_conv_incorrect_group)
{
const PartialShape data_batch_shape{1, 3, 96, 96};
const PartialShape deformable_values_shape{1, 50, 5, 5};
const PartialShape filters_shape{4, 3, 5, 5};
auto param0 = make_shared<op::Parameter>(element::f32, data_batch_shape);
auto param1 = make_shared<op::Parameter>(element::f32, deformable_values_shape);
auto param2 = make_shared<op::Parameter>(element::f32, filters_shape);
try
{
make_shared<op::v1::DeformableConvolution>(param0,
param1,
param2,
Strides{},
CoordinateDiff{},
CoordinateDiff{},
Strides{},
op::PadType::EXPLICIT,
2);
FAIL() << "DeformableConvolution created with incorrect 'group' value";
}
catch (const NodeValidationFailure& error)
{
EXPECT_HAS_SUBSTRING(error.what(), "input data shape must be evenly divisible");
}
try
{
make_shared<op::v1::DeformableConvolution>(param0,
param1,
param2,
Strides{},
CoordinateDiff{},
CoordinateDiff{},
Strides{},
op::PadType::EXPLICIT,
3);
FAIL() << "DeformableConvolution created with incorrect 'group' value";
}
catch (const NodeValidationFailure& error)
{
EXPECT_HAS_SUBSTRING(error.what(), "weights shape must be evenly divisible");
}
}
TEST(type_prop, deformable_conv_incorrect_deformable_group)
{
const PartialShape data_batch_shape{1, 3, 96, 96};
const PartialShape deformable_values_shape{1, 50, 5, 5};
const PartialShape filters_shape{3, 3, 5, 5};
auto param0 = make_shared<op::Parameter>(element::f32, data_batch_shape);
auto param1 = make_shared<op::Parameter>(element::f32, deformable_values_shape);
auto param2 = make_shared<op::Parameter>(element::f32, filters_shape);
try
{
make_shared<op::v1::DeformableConvolution>(param0,
param1,
param2,
Strides{},
CoordinateDiff{},
CoordinateDiff{},
Strides{},
op::PadType::EXPLICIT,
1,
7);
FAIL() << "DeformableConvolution created with incorrect 'deformable group' value";
}
catch (const NodeValidationFailure& error)
{
EXPECT_HAS_SUBSTRING(error.what(), "deformable values input must be evenly divisible");
}
}

88
ngraph/test/type_prop/deformable_convolution.cpp
View File

@ -11,9 +11,9 @@ using namespace ngraph;
TEST(type_prop, deformable_conv_v1_partial_auto_padding_same)
{
const PartialShape data_batch_shape{1, 4, 5, 5};
const PartialShape data_batch_shape{1, 8, 5, 5};
const PartialShape deformable_shape{1, 4, 3, 3};
const PartialShape filters_shape{4, 4, 3, 3};
const PartialShape filters_shape{4, 2, 3, 3};
Strides strides{1, 1};
CoordinateDiff pads_begin{0, 0};
CoordinateDiff pads_end{0, 0};
@ -112,9 +112,9 @@ TEST(type_prop, deformable_conv_v1_partial_auto_padding_same_nc_dims_dynamic_sam
TEST(type_prop, deformable_conv_v1_partial_auto_padding_same_spatial_dims_dynamic)
{
const PartialShape data_batch_shape{1, 4, Dimension::dynamic(), 5};
const PartialShape data_batch_shape{1, 8, Dimension::dynamic(), 5};
const PartialShape deformable_shape{1, 4, 3, 3};
const PartialShape filters_shape{4, 4, 3, 3};
const PartialShape filters_shape{4, 2, 3, 3};
Strides strides{1, 1};
CoordinateDiff pads_begin{0, 0};
CoordinateDiff pads_end{0, 0};
@ -143,3 +143,83 @@ TEST(type_prop, deformable_conv_v1_partial_auto_padding_same_spatial_dims_dynami
ASSERT_EQ(deformable_conv->get_pads_begin(), (CoordinateDiff{0, 1}));
ASSERT_EQ(deformable_conv->get_pads_end(), (CoordinateDiff{0, 1}));
}
TEST(type_prop, deformable_conv_incorrect_group)
{
const PartialShape data_batch_shape{1, 3, 96, 96};
const PartialShape deformable_values_shape{1, 50, 5, 5};
const PartialShape filters_shape{4, 3, 5, 5};
auto param0 = make_shared<op::Parameter>(element::f32, data_batch_shape);
auto param1 = make_shared<op::Parameter>(element::f32, deformable_values_shape);
auto param2 = make_shared<op::Parameter>(element::f32, filters_shape);
try
{
make_shared<op::v1::DeformableConvolution>(param0,
param1,
param2,
Strides{},
CoordinateDiff{},
CoordinateDiff{},
Strides{},
op::PadType::EXPLICIT,
2);
FAIL() << "DeformableConvolution created with incorrect 'group' value";
}
catch (const NodeValidationFailure& error)
{
EXPECT_HAS_SUBSTRING(error.what(), "input data shape must be evenly divisible");
}
try
{
make_shared<op::v1::DeformableConvolution>(param0,
param1,
param2,
Strides{},
CoordinateDiff{},
CoordinateDiff{},
Strides{},
op::PadType::EXPLICIT,
3);
FAIL() << "DeformableConvolution created with incorrect 'group' value";
}
catch (const NodeValidationFailure& error)
{
EXPECT_HAS_SUBSTRING(error.what(), "weights shape must be evenly divisible");
}
}
TEST(type_prop, deformable_conv_incorrect_deformable_group)
{
const PartialShape data_batch_shape{1, 3, 96, 96};
const PartialShape deformable_values_shape{1, 50, 5, 5};
const PartialShape filters_shape{3, 3, 5, 5};
auto param0 = make_shared<op::Parameter>(element::f32, data_batch_shape);
auto param1 = make_shared<op::Parameter>(element::f32, deformable_values_shape);
auto param2 = make_shared<op::Parameter>(element::f32, filters_shape);
try
{
make_shared<op::v1::DeformableConvolution>(param0,
param1,
param2,
Strides{},
CoordinateDiff{},
CoordinateDiff{},
Strides{},
op::PadType::EXPLICIT,
1,
7);
FAIL() << "DeformableConvolution created with incorrect 'deformable group' value";
}
catch (const NodeValidationFailure& error)
{
EXPECT_HAS_SUBSTRING(error.what(), "deformable values input must be evenly divisible");
}
}