IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Dynamic reshapes (#7788)

Browse Source 
* Merged and compiling

* Fix for dynamic shape type

* review fixes

* renamed blob shape to tensor shape, small improvements

* fix code style

* added parsing of multiple shapes

* store latency per group, add isIdleRequestAvailable() to Infer Queue

* added cached random inputs

* redesign pipeline, added new metrics(avg, max, min), added metrics per groups

* fixed code style

* small improvements

* modified tensor parameters parsing

* modified -i parameter parsing: added possibility to specify input names

* implemented image cashing

* added cashed blobs creating

* added -pcseq flag, modified batch filling, changes fps formula

* improvements

* code formatting

* code formatting2

* apply suggestions from review

* replaced Buffer class with InferenceEngine Blobs

* use batch size in blobs filling

* added shared blob allocator to handle blob's data

* fixed warnings & code style

* allocate blobs

* fix for networks with image info input

* added comments & fixed codestyle

* clear data in free() in SharedBlobAllocator

* remove unnecessary check

* Delimeter is changed to ::

* stylefix

* added layout from string function, small improvements

* modified parsing to enable : in input parameters

* small fixes

* small fixes

* added missed blob allocation, fixes

* [TEST]added support for remote blobs

* fix remote blobs

* new inputs/files output format

* removed vectors resize which caused bugs

* made cl::Buffer type under ifdef, fix inputs filling

* changed batch() function to not throwing exceptions

* removed unused var

* fix code style

* replace empty name in input files with name from net input

* restored old behaviour for static models

* fix code style

* fix warning - made const iterator

* fix warning - remove reference in loop variable

* added random and image_info input types to -i, fix problem with layout

* replaced batch() with getBatchSize() in main

* fix layout, shape, tensor shape parameters parsing

* upd help messages for input, tensor shape and pcseq command

* added buffer for cl output blobs, small fixes

Signed-off-by: ivikhrev <ivan.vikhrev@intel.com>

* added legacy mode

* restore setBlob

* code style formatting

* move collecting latency for groups under flag

* removed not applicable layouts

* added hint to error message when wrong input name in -tensor_shape was specified

* added new metrics to statistics report

* Apply suggestions from code review

* fix binary blobs filling when layout is CN

* apply suggestions

* moved file in the right place after rebase

* improved -pcseq output

* updated args and readme

* removed TEMPLATE plugin registration

* fix -shape arg  decsription

* enable providing several -i args as input

* renamed legacy_mode to inference_only and made it default for static models, renamed tensor_shape to data_shape

* upd readme

* use getBlob() in inference only mode

* fix old input type for static case

* fix typo

* upd readme

* move log about benchmark mode to the measuring perfomance step

* added class for latency metrics

* upd readme, fix typos, renamed funcs

* fix warning and upd parsing to avoid error with : in file paths

* fix error on centos : error: use of deleted function ‘std::basic_stringstream<char>::basic_stringstream(const std::basic_stringstream<char>&)

* added check for key in inputs

* renamed input to inputs

* adjust batch size for binary blobs

* replaced warning with exception in bench mode defining

* align measurement cycle with master

Co-authored-by: ivikhrev <ivan.vikhrev@intel.com>
This commit is contained in:
Fedor Zharinov
2021-12-17 12:20:43 +03:00
committed by GitHub
parent 4c4d006c5a
commit e9874ec1d4
13 changed files with 1533 additions and 562 deletions
Download Patch File Download Diff File Expand all files Collapse all files

268
samples/cpp/benchmark_app/utils.hpp
View File

@@ -4,15 +4,43 @@
#pragma once
#include <chrono>
#include <iomanip>
#include <map>
#include <samples/slog.hpp>
#include <string>
#include <vector>
#include "ngraph/partial_shape.hpp"
typedef std::chrono::high_resolution_clock Time;
typedef std::chrono::nanoseconds ns;
inline uint64_t getDurationInMilliseconds(uint32_t duration) {
return duration * 1000LL;
}
inline uint64_t getDurationInNanoseconds(uint32_t duration) {
return duration * 1000000000LL;
}
inline double get_duration_ms_till_now(Time::time_point& startTime) {
return std::chrono::duration_cast<ns>(Time::now() - startTime).count() * 0.000001;
};
inline std::string double_to_string(const double number) {
std::stringstream ss;
ss << std::fixed << std::setprecision(2) << number;
return ss.str();
};
namespace benchmark_app {
struct InputInfo {
InferenceEngine::Precision precision;
InferenceEngine::SizeVector shape;
ngraph::PartialShape partialShape;
InferenceEngine::SizeVector dataShape;
std::string layout;
InferenceEngine::Layout originalLayout;
std::vector<float> scale;
std::vector<float> mean;
bool isImage() const;
@@ -25,43 +53,56 @@ struct InputInfo {
size_t depth() const;
};
using InputsInfo = std::map<std::string, InputInfo>;
using PartialShapes = std::map<std::string, ngraph::PartialShape>;
} // namespace benchmark_app
std::vector<std::string> parseDevices(const std::string& device_string);
uint32_t deviceDefaultDeviceDurationInSeconds(const std::string& device);
std::map<std::string, std::string> parseNStreamsValuePerDevice(const std::vector<std::string>& devices,
const std::string& values_string);
InferenceEngine::Layout getLayoutFromString(const std::string& string_layout);
std::string getShapeString(const InferenceEngine::SizeVector& shape);
std::string getShapesString(const benchmark_app::PartialShapes& shapes);
std::string getShapesString(const InferenceEngine::ICNNNetwork::InputShapes& shapes);
size_t getBatchSize(const benchmark_app::InputsInfo& inputs_info);
std::vector<std::string> split(const std::string& s, char delim);
std::map<std::string, std::vector<float>> parseScaleOrMean(const std::string& scale_mean,
const benchmark_app::InputsInfo& inputs_info);
std::vector<ngraph::Dimension> parsePartialShape(const std::string& partial_shape);
InferenceEngine::SizeVector parseTensorShape(const std::string& data_shape);
std::pair<std::string, std::vector<std::string>> parseInputFiles(const std::string& file_paths_string);
std::map<std::string, std::vector<std::string>> parseInputArguments(const std::vector<std::string>& args);
template <typename T>
std::map<std::string, std::string> parseInputParameters(const std::string parameter_string,
const std::map<std::string, T>& input_info) {
// Parse parameter string like "input0[value0],input1[value1]" or "[value]" (applied to all
// inputs)
std::map<std::string, std::string> return_value;
std::map<std::string, std::vector<std::string>> parseInputParameters(const std::string parameter_string,
const std::map<std::string, T>& input_info) {
// Parse parameter string like "[value0]", "[value0][value1]" or "input0[value0][value1],input1[value2][value3]"
// (applied to all inputs)
std::map<std::string, std::vector<std::string>> return_value;
std::string search_string = parameter_string;
auto start_pos = search_string.find_first_of('[');
auto input_name = search_string.substr(0, start_pos);
while (start_pos != std::string::npos) {
auto end_pos = search_string.find_first_of(']');
if (end_pos == std::string::npos)
break;
auto input_name = search_string.substr(0, start_pos);
if (start_pos)
input_name = search_string.substr(0, start_pos);
auto input_value = search_string.substr(start_pos + 1, end_pos - start_pos - 1);
if (!input_name.empty()) {
return_value[input_name] = input_value;
return_value[input_name].push_back(input_value);
} else {
for (auto& item : input_info) {
return_value[item.first] = input_value;
return_value[item.first].push_back(input_value);
}
}
search_string = search_string.substr(end_pos + 1);
if (search_string.empty() || search_string.front() != ',')
if (search_string.empty() || (search_string.front() != ',' && search_string.front() != '['))
break;
search_string = search_string.substr(1);
if (search_string.front() == ',')
search_string = search_string.substr(1);
start_pos = search_string.find_first_of('[');
}
if (!search_string.empty())
@@ -70,87 +111,156 @@ std::map<std::string, std::string> parseInputParameters(const std::string parame
}
template <typename T>
benchmark_app::InputsInfo getInputsInfo(const std::string& shape_string,
const std::string& layout_string,
const size_t batch_size,
const std::string& scale_string,
const std::string& mean_string,
const std::map<std::string, T>& input_info,
bool& reshape_required) {
std::map<std::string, std::string> shape_map = parseInputParameters(shape_string, input_info);
std::map<std::string, std::string> layout_map = parseInputParameters(layout_string, input_info);
std::vector<benchmark_app::InputsInfo> getInputsInfo(const std::string& shape_string,
const std::string& layout_string,
const size_t batch_size,
const std::string& data_shapes_string,
const std::string& scale_string,
const std::string& mean_string,
const std::map<std::string, T>& input_info,
bool& reshape_required) {
std::map<std::string, std::vector<std::string>> shape_map = parseInputParameters(shape_string, input_info);
std::map<std::string, std::vector<std::string>> data_shapes_map =
parseInputParameters(data_shapes_string, input_info);
std::map<std::string, std::vector<std::string>> layout_map = parseInputParameters(layout_string, input_info);
size_t min_size = 1, max_size = 1;
if (!data_shapes_map.empty()) {
min_size = std::min_element(data_shapes_map.begin(),
data_shapes_map.end(),
[](std::pair<std::string, std::vector<std::string>> a,
std::pair<std::string, std::vector<std::string>> b) {
return a.second.size() < b.second.size() && a.second.size() != 1;
})
->second.size();
max_size = std::max_element(data_shapes_map.begin(),
data_shapes_map.end(),
[](std::pair<std::string, std::vector<std::string>> a,
std::pair<std::string, std::vector<std::string>> b) {
return a.second.size() < b.second.size();
})
->second.size();
if (min_size != max_size) {
throw std::logic_error(
"Shapes number for every input should be either 1 or should be equal to shapes number of other inputs");
}
}
reshape_required = false;
benchmark_app::InputsInfo info_map;
for (auto& item : input_info) {
benchmark_app::InputInfo info;
auto name = item.first;
auto descriptor = item.second->getTensorDesc();
// Precision
info.precision = descriptor.getPrecision();
// Shape
if (shape_map.count(name)) {
std::vector<size_t> parsed_shape;
for (auto& dim : split(shape_map.at(name), ',')) {
parsed_shape.push_back(std::stoi(dim));
}
info.shape = parsed_shape;
reshape_required = true;
} else {
info.shape = descriptor.getDims();
}
// Layout
if (layout_map.count(name)) {
info.layout = layout_map.at(name);
std::transform(info.layout.begin(), info.layout.end(), info.layout.begin(), ::toupper);
} else {
std::stringstream ss;
ss << descriptor.getLayout();
info.layout = ss.str();
}
// Update shape with batch if needed
if (batch_size != 0) {
std::size_t batch_index = info.layout.find("N");
if ((batch_index != std::string::npos) && (info.shape.at(batch_index) != batch_size)) {
info.shape[batch_index] = batch_size;
std::vector<benchmark_app::InputsInfo> info_maps;
for (size_t i = 0; i < min_size; ++i) {
benchmark_app::InputsInfo info_map;
for (auto& item : input_info) {
benchmark_app::InputInfo info;
auto name = item.first;
auto descriptor = item.second->getTensorDesc();
// Precision
info.precision = descriptor.getPrecision();
// Partial Shape
if (shape_map.count(name)) {
std::vector<ngraph::Dimension> parsed_shape;
if (shape_map.at(name).size() > 1) {
throw std::logic_error(
"shape command line parameter doesn't support multiple shapes for one input.");
}
info.partialShape = parsePartialShape(shape_map.at(name)[0]);
reshape_required = true;
} else {
info.partialShape = item.second->getPartialShape();
}
if (info.partialShape.is_dynamic() && info.isImage()) {
throw std::logic_error(
"benchmark_app supports only binary and random data as input for dynamic models at this moment.");
}
// Tensor Shape
if (info.partialShape.is_dynamic() && data_shapes_map.count(name)) {
info.dataShape = parseTensorShape(data_shapes_map.at(name)[i % data_shapes_map.at(name).size()]);
} else if (info.partialShape.is_static()) {
info.dataShape = info.partialShape.get_shape();
if (data_shapes_map.find(name) != data_shapes_map.end()) {
throw std::logic_error(
"Network's input \"" + name +
"\" is static. Use -shape argument for static inputs instead of -data_shape.");
}
} else if (!data_shapes_map.empty()) {
throw std::logic_error("Can't find network input name \"" + name + "\" in \"-data_shape " +
data_shapes_string + "\" command line parameter");
} else {
throw std::logic_error(
"data_shape command line parameter should be set in case of network with dynamic shapes.");
}
// Layout
info.originalLayout = descriptor.getLayout();
if (layout_map.count(name)) {
if (layout_map.at(name).size() > 1) {
throw std::logic_error(
"layout command line parameter doesn't support multiple layouts for one input.");
}
info.layout = layout_map.at(name)[0];
std::transform(info.layout.begin(), info.layout.end(), info.layout.begin(), ::toupper);
} else {
std::stringstream ss;
ss << descriptor.getLayout();
info.layout = ss.str();
}
// Update shape with batch if needed (only in static shape case)
// Update blob shape only not affecting network shape to trigger dynamic batch size case
if (batch_size != 0) {
std::size_t batch_index = info.layout.find("N");
if ((batch_index != std::string::npos) && (info.dataShape.at(batch_index) != batch_size)) {
if (info.partialShape.is_static()) {
info.partialShape[batch_index] = batch_size;
}
info.dataShape[batch_index] = batch_size;
reshape_required = true;
}
}
info_map[name] = info;
}
// Update scale and mean
std::map<std::string, std::vector<float>> scale_map = parseScaleOrMean(scale_string, info_map);
std::map<std::string, std::vector<float>> mean_map = parseScaleOrMean(mean_string, info_map);
for (auto& item : info_map) {
if (item.second.isImage()) {
item.second.scale.assign({1, 1, 1});
item.second.mean.assign({0, 0, 0});
if (scale_map.count(item.first)) {
item.second.scale = scale_map.at(item.first);
}
if (mean_map.count(item.first)) {
item.second.mean = mean_map.at(item.first);
}
}
}
info_map[name] = info;
info_maps.push_back(info_map);
}
// Update scale and mean
std::map<std::string, std::vector<float>> scale_map = parseScaleOrMean(scale_string, info_map);
std::map<std::string, std::vector<float>> mean_map = parseScaleOrMean(mean_string, info_map);
for (auto& item : info_map) {
if (item.second.isImage()) {
item.second.scale.assign({1, 1, 1});
item.second.mean.assign({0, 0, 0});
if (scale_map.count(item.first)) {
item.second.scale = scale_map.at(item.first);
}
if (mean_map.count(item.first)) {
item.second.mean = mean_map.at(item.first);
}
}
}
return info_map;
return info_maps;
}
template <typename T>
benchmark_app::InputsInfo getInputsInfo(const std::string& shape_string,
const std::string& layout_string,
const size_t batch_size,
const std::string& scale_string,
const std::string& mean_string,
const std::map<std::string, T>& input_info) {
std::vector<benchmark_app::InputsInfo> getInputsInfo(const std::string& shape_string,
const std::string& layout_string,
const size_t batch_size,
const std::string& data_shapes_string,
const std::string& scale_string,
const std::string& mean_string,
const std::map<std::string, T>& input_info) {
bool reshape_required = false;
return getInputsInfo<T>(shape_string,
layout_string,
batch_size,
data_shapes_string,
scale_string,
mean_string,
input_info,
@@ -160,4 +270,4 @@ benchmark_app::InputsInfo getInputsInfo(const std::string& shape_string,
#ifdef USE_OPENCV
void dump_config(const std::string& filename, const std::map<std::string, std::map<std::string, std::string>>& config);
void load_config(const std::string& filename, std::map<std::string, std::map<std::string, std::string>>& config);
#endif
#endif