[AUTO] Enable Benchmark APP to parse the ov::device::properties through dump_config()/load_config() (#14057)
* [AUTO] update property setting logic. * Update core::set_property() logic to only support primary property for AUTO and MULTI. * Separate AUTO and MULTI supported properties. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Add test case. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * 1. revert the changes for ie core. 2. Enable AUTO/MULTI only accepting its own properties. 3. Enable AUTO/MULTI accepting device properties passed from loadnetwork(). Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Revert "[AUTO]Update the incorrect config test for Myriad (#13271)" This reverts commit0552d98802
. * MULTI only accepts its own properties that is same as AUTO currently. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Add test cases for AUTO/MULTI property test. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update to enable MULTI supporting hw primary property setting throw the compile_model(). Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Remove the added test case for setting secondary property through set_property(). Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * 1. For AUTO/MULTI, property setting will be passed via core::compile_model() instead of core::set_property(). 2. update the logic to infer precision setting that will transform into secondary property setting to each hw device. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * set default value for nstreams when -d AUTO/MULTI and no nstreams setting from command line. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update code format. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Enable BA to support if -d AUTO:MULTI,xxx/MULTI:AUTO,xxx. while AUTO Plugin need to update the logic of generating supported config list to virtual device. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * update. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * move device configuration handling outside form function main. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Revert "move device configuration handling outside form function main." This reverts commitef77bfc602
. * Update. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * update. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Setting property performance hint to 'THROUGHPUT' and passing to executable network if no setting for of AUTO/MULTI. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update. * 1. Fix the issue that the dumped configuration file retrieves empty config list when using OpenCV to dump configuration. 2. Enable load_config() to load secondary property. 3. ov:device:properties from command line will replace the ov::device::properties loaded from configuration file. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update the performance statistics showing format. * Update format. * Update. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update samples/cpp/benchmark_app/main.cpp Co-authored-by: Nadezhda Ageeva <nkogteva@gmail.com> * Update samples/cpp/benchmark_app/main.cpp Co-authored-by: Nadezhda Ageeva <nkogteva@gmail.com> * Update. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update README to add some simple configuration files loaded into benchmark app. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * 1. Enable dump/load function for AUTO to support hw device property setting thorugh configuration JSON. 2. Update README.md to example a simple JSON file containing the hw device properties setting for AUTO/MULTI. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update. Signed-off-by: Wang, Yang <yang4.wang@intel.com> * Update. * Update. * Update. * Drop the changes within dump/load of OpenCV config parsing. * Update. * Correct property name. Update ov::num_stream to ov::inference_precision. Signed-off-by: Wang, Yang <yang4.wang@intel.com> Co-authored-by: Nadezhda Ageeva <nkogteva@gmail.com>
This commit is contained in:
parent
fe52337840
commit
a37723e8cf
@ -154,6 +154,27 @@ Options:
|
|||||||
-pcseq Optional. Report latencies for each shape in -data_shape sequence.
|
-pcseq Optional. Report latencies for each shape in -data_shape sequence.
|
||||||
-dump_config Optional. Path to JSON file to dump IE parameters, which were set by application.
|
-dump_config Optional. Path to JSON file to dump IE parameters, which were set by application.
|
||||||
-load_config Optional. Path to JSON file to load custom IE parameters. Please note, command line parameters have higher priority then parameters from configuration file.
|
-load_config Optional. Path to JSON file to load custom IE parameters. Please note, command line parameters have higher priority then parameters from configuration file.
|
||||||
|
Example 1: a simple JSON file for HW device with primary properties.
|
||||||
|
{
|
||||||
|
"CPU": {"NUM_STREAMS": "3", "PERF_COUNT": "NO"}
|
||||||
|
}
|
||||||
|
Example 2: a simple JSON file for meta device(AUTO/MULTI) with HW device properties.
|
||||||
|
{
|
||||||
|
"AUTO": {
|
||||||
|
"PERFORMANCE_HINT": "",
|
||||||
|
"PERF_COUNT": "NO",
|
||||||
|
"DEVICE_PROPERTIES": {
|
||||||
|
"CPU": {
|
||||||
|
"INFERENCE_PRECISION_HINT": "f32",
|
||||||
|
"NUM_STREAMS": "3"
|
||||||
|
},
|
||||||
|
"GPU": {
|
||||||
|
"INFERENCE_PRECISION_HINT": "f32",
|
||||||
|
"NUM_STREAMS": "5"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
-infer_precision "<element type>"Optional. Inference precision
|
-infer_precision "<element type>"Optional. Inference precision
|
||||||
-ip <value> Optional. Specifies precision for all input layers of the model.
|
-ip <value> Optional. Specifies precision for all input layers of the model.
|
||||||
-op <value> Optional. Specifies precision for all output layers of the model.
|
-op <value> Optional. Specifies precision for all output layers of the model.
|
||||||
|
@ -170,8 +170,29 @@ static const char use_device_mem_message[] =
|
|||||||
// @brief message for load config option
|
// @brief message for load config option
|
||||||
static const char load_config_message[] =
|
static const char load_config_message[] =
|
||||||
"Optional. Path to JSON file to load custom IE parameters."
|
"Optional. Path to JSON file to load custom IE parameters."
|
||||||
" Please note, command line parameters have higher priority then parameters from configuration "
|
" Please note, command line parameters have higher priority then parameters from configuration file.\n"
|
||||||
"file.";
|
" Example 1: a simple JSON file for HW device with primary properties.\n"
|
||||||
|
" {\n"
|
||||||
|
" \"CPU\": {\"NUM_STREAMS\": \"3\", \"PERF_COUNT\": \"NO\"}\n"
|
||||||
|
" }\n"
|
||||||
|
" Example 2: a simple JSON file for meta device(AUTO/MULTI) with HW device "
|
||||||
|
"properties.\n"
|
||||||
|
" {\n"
|
||||||
|
" \"AUTO\": {\n"
|
||||||
|
" \"PERFORMANCE_HINT\": \"\",\n"
|
||||||
|
" \"PERF_COUNT\": \"NO\",\n"
|
||||||
|
" \"DEVICE_PROPERTIES\": {\n"
|
||||||
|
" \"CPU\": {\n"
|
||||||
|
" \"INFERENCE_PRECISION_HINT\": \"f32\",\n"
|
||||||
|
" \"NUM_STREAMS\": \"3\"\n"
|
||||||
|
" },\n"
|
||||||
|
" \"GPU\": {\n"
|
||||||
|
" \"INFERENCE_PRECISION_HINT\": \"f32\",\n"
|
||||||
|
" \"NUM_STREAMS\": \"5\"\n"
|
||||||
|
" }\n"
|
||||||
|
" }\n"
|
||||||
|
" }\n"
|
||||||
|
" }\n";
|
||||||
|
|
||||||
// @brief message for dump config option
|
// @brief message for dump config option
|
||||||
static const char dump_config_message[] =
|
static const char dump_config_message[] =
|
||||||
|
@ -157,6 +157,7 @@ void setDeviceProperty(ov::Core& core,
|
|||||||
std::string& device,
|
std::string& device,
|
||||||
ov::AnyMap& device_config,
|
ov::AnyMap& device_config,
|
||||||
const std::pair<std::string, ov::Any>& property,
|
const std::pair<std::string, ov::Any>& property,
|
||||||
|
std::map<std::string, bool>& is_dev_set_property,
|
||||||
const std::pair<std::string, ov::Any>& config = {}) {
|
const std::pair<std::string, ov::Any>& config = {}) {
|
||||||
auto supported_properties = core.get_property(device, ov::supported_properties);
|
auto supported_properties = core.get_property(device, ov::supported_properties);
|
||||||
auto supported = [&](const std::string& key) {
|
auto supported = [&](const std::string& key) {
|
||||||
@ -172,10 +173,17 @@ void setDeviceProperty(ov::Core& core,
|
|||||||
|
|
||||||
if (device_property.first.empty())
|
if (device_property.first.empty())
|
||||||
return;
|
return;
|
||||||
if (device_config.find(device) == device_config.end()) {
|
|
||||||
|
if (device_config.find(device) == device_config.end() || // device properties not existed
|
||||||
|
config.first.empty() && // not setting default value to property
|
||||||
|
(!FLAGS_load_config.empty() &&
|
||||||
|
is_dev_set_property[device])) { // device properties loaded from file and overwrite is not happened
|
||||||
|
is_dev_set_property[device] = false;
|
||||||
|
device_config.erase(device);
|
||||||
device_config.insert(ov::device::properties(device, device_property));
|
device_config.insert(ov::device::properties(device, device_property));
|
||||||
} else {
|
} else {
|
||||||
auto& properties = device_config[device].as<ov::AnyMap>();
|
auto& properties = device_config[device].as<ov::AnyMap>();
|
||||||
|
// property present in device properties has higher priority than property set with default value
|
||||||
properties.emplace(device_property);
|
properties.emplace(device_property);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@ -230,6 +238,10 @@ int main(int argc, char* argv[]) {
|
|||||||
// Parse devices
|
// Parse devices
|
||||||
auto devices = parse_devices(device_name);
|
auto devices = parse_devices(device_name);
|
||||||
|
|
||||||
|
std::map<std::string, bool> is_dev_set_property = {};
|
||||||
|
// initialize flags to ensure ov::device::properties should only be set once per device.
|
||||||
|
for (auto& dev : devices)
|
||||||
|
is_dev_set_property[dev] = true;
|
||||||
// Parse nstreams per device
|
// Parse nstreams per device
|
||||||
std::map<std::string, std::string> device_nstreams = parse_value_per_device(devices, FLAGS_nstreams);
|
std::map<std::string, std::string> device_nstreams = parse_value_per_device(devices, FLAGS_nstreams);
|
||||||
std::map<std::string, std::string> device_infer_precision =
|
std::map<std::string, std::string> device_infer_precision =
|
||||||
@ -237,8 +249,10 @@ int main(int argc, char* argv[]) {
|
|||||||
|
|
||||||
// Load device config file if specified
|
// Load device config file if specified
|
||||||
std::map<std::string, ov::AnyMap> config;
|
std::map<std::string, ov::AnyMap> config;
|
||||||
|
bool is_load_config = false;
|
||||||
if (!FLAGS_load_config.empty()) {
|
if (!FLAGS_load_config.empty()) {
|
||||||
load_config(FLAGS_load_config, config);
|
load_config(FLAGS_load_config, config);
|
||||||
|
is_load_config = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** This vector stores paths to the processed images with input names**/
|
/** This vector stores paths to the processed images with input names**/
|
||||||
@ -381,10 +395,17 @@ int main(int argc, char* argv[]) {
|
|||||||
std::map<std::string, std::string> devices_property;
|
std::map<std::string, std::string> devices_property;
|
||||||
ov::util::Read<std::map<std::string, std::string>>{}(strm, devices_property);
|
ov::util::Read<std::map<std::string, std::string>>{}(strm, devices_property);
|
||||||
for (auto it : devices_property) {
|
for (auto it : devices_property) {
|
||||||
if (device_config.find(it.first) == device_config.end())
|
if (device_config.find(it.first) == device_config.end() ||
|
||||||
|
(is_load_config && is_dev_set_property[it.first])) {
|
||||||
|
// Create ov::device::properties with ov::num_stream and
|
||||||
|
// 1. Insert this ov::device::properties into device config if this
|
||||||
|
// ov::device::properties isn't existed. Otherwise,
|
||||||
|
// 2. Replace the existed ov::device::properties within device config.
|
||||||
|
is_dev_set_property[it.first] = false;
|
||||||
|
device_config.erase(it.first);
|
||||||
device_config.insert(
|
device_config.insert(
|
||||||
ov::device::properties(it.first, ov::num_streams(std::stoi(it.second))));
|
ov::device::properties(it.first, ov::num_streams(std::stoi(it.second))));
|
||||||
else {
|
} else {
|
||||||
auto& property = device_config[it.first].as<ov::AnyMap>();
|
auto& property = device_config[it.first].as<ov::AnyMap>();
|
||||||
property.emplace(ov::num_streams(std::stoi(it.second)));
|
property.emplace(ov::num_streams(std::stoi(it.second)));
|
||||||
}
|
}
|
||||||
@ -416,13 +437,14 @@ int main(int argc, char* argv[]) {
|
|||||||
device_config[key] = ov::streams::AUTO;
|
device_config[key] = ov::streams::AUTO;
|
||||||
} else if (device == "MULTI" || device == "AUTO") {
|
} else if (device == "MULTI" || device == "AUTO") {
|
||||||
// Set nstreams to default value auto if no nstreams specified from cmd line.
|
// Set nstreams to default value auto if no nstreams specified from cmd line.
|
||||||
std::string key = std::string(getDeviceTypeFromName(device) + "_THROUGHPUT_STREAMS");
|
|
||||||
for (auto& hwdevice : hardware_devices) {
|
for (auto& hwdevice : hardware_devices) {
|
||||||
|
std::string key = std::string(getDeviceTypeFromName(hwdevice) + "_THROUGHPUT_STREAMS");
|
||||||
auto value = std::string(getDeviceTypeFromName(hwdevice) + "_THROUGHPUT_AUTO");
|
auto value = std::string(getDeviceTypeFromName(hwdevice) + "_THROUGHPUT_AUTO");
|
||||||
setDeviceProperty(core,
|
setDeviceProperty(core,
|
||||||
hwdevice,
|
hwdevice,
|
||||||
device_config,
|
device_config,
|
||||||
ov::num_streams(ov::streams::AUTO),
|
ov::num_streams(ov::streams::AUTO),
|
||||||
|
is_dev_set_property,
|
||||||
std::make_pair(key, value));
|
std::make_pair(key, value));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@ -451,10 +473,17 @@ int main(int argc, char* argv[]) {
|
|||||||
std::map<std::string, std::string> devices_property;
|
std::map<std::string, std::string> devices_property;
|
||||||
ov::util::Read<std::map<std::string, std::string>>{}(strm, devices_property);
|
ov::util::Read<std::map<std::string, std::string>>{}(strm, devices_property);
|
||||||
for (auto it : devices_property) {
|
for (auto it : devices_property) {
|
||||||
if (device_config.find(it.first) == device_config.end())
|
if (device_config.find(it.first) == device_config.end() ||
|
||||||
|
(is_load_config && is_dev_set_property[it.first])) {
|
||||||
|
// Create ov::device::properties with ov::inference_precision and
|
||||||
|
// 1. Insert this ov::device::properties into device config if this
|
||||||
|
// ov::device::properties isn't existed. Otherwise,
|
||||||
|
// 2. Replace the existed ov::device::properties within device config.
|
||||||
|
is_dev_set_property[it.first] = false;
|
||||||
|
device_config.erase(it.first);
|
||||||
device_config.insert(
|
device_config.insert(
|
||||||
ov::device::properties(it.first, ov::hint::inference_precision(it.second)));
|
ov::device::properties(it.first, ov::hint::inference_precision(it.second)));
|
||||||
else {
|
} else {
|
||||||
auto& property = device_config[it.first].as<ov::AnyMap>();
|
auto& property = device_config[it.first].as<ov::AnyMap>();
|
||||||
property.emplace(ov::hint::inference_precision(it.second));
|
property.emplace(ov::hint::inference_precision(it.second));
|
||||||
}
|
}
|
||||||
@ -491,7 +520,7 @@ int main(int argc, char* argv[]) {
|
|||||||
// list specified by -d.
|
// list specified by -d.
|
||||||
for (auto& device : hardware_devices) {
|
for (auto& device : hardware_devices) {
|
||||||
if (device == "CPU")
|
if (device == "CPU")
|
||||||
setDeviceProperty(core, device, device_config, property);
|
setDeviceProperty(core, device, device_config, property, is_dev_set_property);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
@ -1249,8 +1278,8 @@ int main(int argc, char* argv[]) {
|
|||||||
} catch (const ov::Exception&) {
|
} catch (const ov::Exception&) {
|
||||||
}
|
}
|
||||||
|
|
||||||
slog::info << "Count: " << iteration << " iterations" << slog::endl;
|
slog::info << "Count: " << iteration << " iterations" << slog::endl;
|
||||||
slog::info << "Duration: " << double_to_string(totalDuration) << " ms" << slog::endl;
|
slog::info << "Duration: " << double_to_string(totalDuration) << " ms" << slog::endl;
|
||||||
|
|
||||||
if (device_name.find("MULTI") == std::string::npos) {
|
if (device_name.find("MULTI") == std::string::npos) {
|
||||||
slog::info << "Latency:" << slog::endl;
|
slog::info << "Latency:" << slog::endl;
|
||||||
@ -1274,7 +1303,7 @@ int main(int argc, char* argv[]) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
slog::info << "Throughput: " << double_to_string(fps) << " FPS" << slog::endl;
|
slog::info << "Throughput: " << double_to_string(fps) << " FPS" << slog::endl;
|
||||||
|
|
||||||
} catch (const std::exception& ex) {
|
} catch (const std::exception& ex) {
|
||||||
slog::err << ex.what() << slog::endl;
|
slog::err << ex.what() << slog::endl;
|
||||||
|
@ -345,13 +345,13 @@ void LatencyMetrics::write_to_stream(std::ostream& stream) const {
|
|||||||
|
|
||||||
void LatencyMetrics::write_to_slog() const {
|
void LatencyMetrics::write_to_slog() const {
|
||||||
std::string percentileStr = (percentile_boundary == 50)
|
std::string percentileStr = (percentile_boundary == 50)
|
||||||
? " Median: "
|
? " Median: "
|
||||||
: " " + std::to_string(percentile_boundary) + " percentile: ";
|
: " " + std::to_string(percentile_boundary) + " percentile: ";
|
||||||
|
|
||||||
slog::info << percentileStr << double_to_string(median_or_percentile) << " ms" << slog::endl;
|
slog::info << percentileStr << double_to_string(median_or_percentile) << " ms" << slog::endl;
|
||||||
slog::info << " Average: " << double_to_string(avg) << " ms" << slog::endl;
|
slog::info << " Average: " << double_to_string(avg) << " ms" << slog::endl;
|
||||||
slog::info << " Min: " << double_to_string(min) << " ms" << slog::endl;
|
slog::info << " Min: " << double_to_string(min) << " ms" << slog::endl;
|
||||||
slog::info << " Max: " << double_to_string(max) << " ms" << slog::endl;
|
slog::info << " Max: " << double_to_string(max) << " ms" << slog::endl;
|
||||||
}
|
}
|
||||||
|
|
||||||
const nlohmann::json LatencyMetrics::to_json() const {
|
const nlohmann::json LatencyMetrics::to_json() const {
|
||||||
|
@ -731,10 +731,27 @@ void dump_config(const std::string& filename, const std::map<std::string, ov::An
|
|||||||
nlohmann::json jsonConfig;
|
nlohmann::json jsonConfig;
|
||||||
for (const auto& item : config) {
|
for (const auto& item : config) {
|
||||||
std::string deviceName = item.first;
|
std::string deviceName = item.first;
|
||||||
|
std::map<std::string, ov::AnyMap> device_properties;
|
||||||
for (const auto& option : item.second) {
|
for (const auto& option : item.second) {
|
||||||
std::stringstream strm;
|
if (option.second.is<ov::AnyMap>()) {
|
||||||
option.second.print(strm);
|
// hw device properties
|
||||||
jsonConfig[deviceName][option.first] = strm.str();
|
device_properties[option.first] = option.second.as<ov::AnyMap>();
|
||||||
|
} else {
|
||||||
|
// primary property
|
||||||
|
std::stringstream strm;
|
||||||
|
option.second.print(strm);
|
||||||
|
auto property_string = strm.str();
|
||||||
|
jsonConfig[deviceName][option.first] = property_string;
|
||||||
|
}
|
||||||
|
if (!device_properties.empty()) {
|
||||||
|
for (auto& item : device_properties) {
|
||||||
|
auto hw_device_name = item.first;
|
||||||
|
for (auto& property : item.second) {
|
||||||
|
jsonConfig[deviceName]["DEVICE_PROPERTIES"][hw_device_name][property.first] =
|
||||||
|
property.second.as<std::string>();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -762,7 +779,17 @@ void load_config(const std::string& filename, std::map<std::string, ov::AnyMap>&
|
|||||||
for (const auto& item : jsonConfig.items()) {
|
for (const auto& item : jsonConfig.items()) {
|
||||||
std::string deviceName = item.key();
|
std::string deviceName = item.key();
|
||||||
for (const auto& option : item.value().items()) {
|
for (const auto& option : item.value().items()) {
|
||||||
config[deviceName][option.key()] = option.value().get<std::string>();
|
if (option.key() != "DEVICE_PROPERTIES") {
|
||||||
|
config[deviceName][option.key()] = option.value().get<std::string>();
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
for (const auto& hw_properties : option.value().items()) {
|
||||||
|
auto hw_device_name = hw_properties.key();
|
||||||
|
std::map<std::string, ov::Any> hw_device_properties;
|
||||||
|
for (const auto& property : hw_properties.value().items())
|
||||||
|
hw_device_properties[property.key()] = property.value().get<std::string>();
|
||||||
|
config[deviceName][hw_device_name] = hw_device_properties;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
Loading…
Reference in New Issue
Block a user