* Check if the device is supported when AUTO retrieves the device list based on the ov::device::priorities.
* Update the logic to handle the situation like -d AUTO:-CPU to benchmark APP.
* Remove MYRIAD and add NVIDIA for AUTO supported devices.
---------
Co-authored-by: Chen Peter <peter.chen@intel.com>
Co-authored-by: Shen, Wanglei <wanglei.shen@intel.com>
* Update ov::hint::performance_hint UNDEFINED value from empty string to "UNDEFINED".
* Update benchmark Python version.
* Update.
* Update.
* Update.
* Update the description about hint setting within benchmark APP README and help message.
* Disable set_property() to support ov::device::properties setting.
* Update benchmark APP to set device properties through compile_model() instead of through set_property().
* Update.
* Update.
* Update some test case including ov::device::properties setting via core.ser_property().
* Since core.set_property() didn't support ov::device::properties setting, just remove the test case to check compile_model() works well if setting ov::device::properties via core.set_property() first.
* Update CompileModel in test name to CompiledModel
Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>
* Add corresponding test case.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update.
* Update.
* Remove the changes of this commit as this modification has nothing to do
with this PR.
This reverts commit 4f04b9f085.
---------
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Co-authored-by: Chen Peter <peter.chen@intel.com>
Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>
* benchmark_app: factor out advanced options
* cpp/benchmark_app: add usage word
* move api to advanced, group b/shape/data_shape/layout into Input shapes, factor out Statistics dumping options
* Factor out Device-specific performance options
* Factor out Preprocessing options
* Minor regroup
* [C++/BENCHMARK_APP] Introduce Numpy array loading for C++ benchmark app
* [DOCS/BENCHMARK_APP] Update docs to reflect changes, update list of available extensions from OpenCV, align help messages
* Update inputs_filling.cpp
* Update tools/benchmark_tool/openvino/tools/benchmark/utils/inputs_filling.py
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* Update samples/cpp/benchmark_app/inputs_filling.cpp
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* [C++/Python] Implement quality-of-life improvements from PR comments
* [C++] Fix compilation errors, fix linter output
* [C++/PYTHON] Apply requested changes
* Update samples/cpp/benchmark_app/main.cpp
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* Update samples/cpp/benchmark_app/utils.cpp
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* [PYTHON] Separate loading of numpy arrays similar to images
* [PYTHON] Remove unnecessary 'Prepare xxx file' print
* Update README again because IF OPENCV.. dissapeared for some reason
* Update second README with missing IF OPENCV..
* [C++] Remove unnecessary vector print function
* [C++ Add Numpy processing function - TODO link it to the tensor filling
* Reverse OneDnn plugin modification
* [C++] Numpy array loading for C++
* [C++] Add (almost) all missing types of data
* Reverse submodule modifications
* [C++/PYTHON] Fix compilation errors, clean code
* [C++] Modify supported extensions, add numpy checking to utils, add numpy to get_image_info method
* Update samples/cpp/benchmark_app/inputs_filling.cpp
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* [C++] Fix utils header file to reflect unordered set change
* [PYTHON/C++] Fix compilation errors in C++ code, fix Python dynamic shapes numpy loading
* [C++] Fix explicit instantiation of NumpyArray reader
* [C++] Clang format, minor syntax fixes
* [PYTHON/C++] Remove unnecessary data types, introduce a new approach to cast data of different types from format_rt_reader, remove uppercase types from Python precision parameters
* [PYTHON] Update README to reflect new precision settings
* [PYTHON] Fix README, fix clang format
* [C++] Clean headers
* [C++] Fix uninitialized variable error
* [C++/PYTHON] Fixed choices in Python benchmark, fixed types in C++ benchmark
* [C++] Fixed ov::float16conversion, fixed Python types map - removed redundancies
* [C++] Add back boolean support
* [C++] Fix compilation errors
---------
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* [Python] Align pc output in Python benchmark, update sorted output to new headers, print floats with 6-digit precision, fix date format printing to excel-friendly
* [PYTHON/C++] Change precision of raports and pc counters report, simplify code
* [C++] Align C++ benchamrk_app report to Python's by setting the precision value to 3
* [C++] Fix CsvDumper precision
* [C++/PYTHON] Align pcand pcsort outputs, merge C++ printSort methods into one
* Update samples/cpp/common/utils/include/samples/common.hpp
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* Update samples/cpp/common/utils/include/samples/common.hpp
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
Co-authored-by: Michal Lukaszewski <michal.lukaszewski@intel.com>
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* [C++] Display a warning if Power Mode is not 'Max Performance' - rebase from master
* [C++] Prevent std::max from colliding with windows.h max macro
* [PYTHON/C++] Update README docs to reflect possible inaccuracies from optimization settings
* [C++] Cleanup windows optimizations
* [Tools] Avoid use of NumPy deprecated types
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Revert "[Tools] Avoid use of NumPy deprecated types"
This reverts commit 21ffc167d1.
* Move to 1.24 Numpy
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* replace benchmark_app imean and iscale parameters
Tickets 80777 and 80900
* Warn about difference with imean and iscale, note that in help
* Clang format
* Clang format
* mean->scale
* Keep mean scale empty if not set
Co-authored-by: Andrei Kochin <andrei.kochin@intel.com>
* merge master
* reset original format
* remove AUTO
* Update tools/benchmark_tool/openvino/tools/benchmark/main.py
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* delete condition
* delete condition in py
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
Co-authored-by: Chen Peter <peter.chen@intel.com>
* sync benchmark Python version with benchmark C++ version.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update.
* Update tools/benchmark_tool/openvino/tools/benchmark/main.py
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* Update tools/benchmark_tool/openvino/tools/benchmark/utils/utils.py
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* Enable dump/load to support ov::device::properties parsing in benchmark python version.
* Enable benchmark python version to parse configuration file with device properties setting for AUTO/MULTI.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update usage of benchmark dumping/loading configuration in README and help message.
* Update comment.
* Update comments of the code.
* Update format of '-load_config' help message.
* Update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* docs: revise example and update outputs
* docs: revise example and update outputs
* docs: typo fix
* docs: rectify changes
* docs: add <a> tag
* Update README.md
* docs: change example to use asl-recognition model
* docs: fix <a> tag
* docs: update sample outputs
* docs: change example to use asl-recognition model
* docs: revise example to use asl-recognition model
Co-authored-by: Yuan Xu <yuan1.xu@intel.com>
* [Python API][Tools] Top up upper bound version for dependencies: NumPy, TensorFlow, NetworkX
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Remove upper-bound for TensorFlow and NumPy
* Revert "Remove upper-bound for TensorFlow and NumPy"
This reverts commit 662085df2e.
* Remove upper-bound for NumPy for default installation
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* sync benchmark Python version with benchmark C++ version.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update.
* Update tools/benchmark_tool/openvino/tools/benchmark/main.py
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* Update tools/benchmark_tool/openvino/tools/benchmark/utils/utils.py
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* Update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* [PYTHON] Pipeline transfer
* [PYTHON] Align python benchmark
* [PYTHON] Align last step
* [PYTHON] Fix innacuracies of the last step - median
* [PYTHON/CPP] Add Core::get_version method to python API, refactor Ben benchmark to print version with this func
* [PYTHON] Remove get_version_string from API
* [PYTHON/CPP] Align output for model input/output info
* [PYTHON/CPP] Step 4,6 alignment of outputs, step 8 dumps all info stored in config parameters
* [CPP] Fix a bug causing nstreams parameter to never be set to AUTO in CPP benchmark_app
* [CPP] Fix clang format errors
* [CPP] Modify print order and data output for 8th step
* [PYTHON] Add verification checks from C++, modify set_thoughtput_streams to match documentation
* [CPP] Revert changes to C++ benchmark_app
* [CPP] Remove additional spacebar
* Update submodules versions on remote
* Update module from master branch
* Redownload submodules from master and override changes from commit
* [PYTHON] Remove unneccesary parse_status from validation function
* [PYTHON] Check for HINT in map, fix circular import
* [PYTHON] Remove artifacts from commit, fix args.perf_hint set to '' instead to 'none'
* [PYTHON] Reverse changes to perf hint, add key in map check, fix validation function throwing error on set hint
* [PYTHON] Fix linter
* [PYTHON] Remove linter spacebar
* [CPP] Fix wait_all exception throw
* [CPP/PYTHON] Clean artifacts and unwanted changes from work process
* [PYTHON] Fix artifacts from merge, clean submodule update
* [C++ CPU] Fix device name string by removing padding NULL characters from the back
* [CPP] Fix ba infer_request_wrap in other throw-catch clauses
* [PYTHON/CPP] Fix missing latencies in final step for shape group, fix minor misaligned messages, add missing report parameter create infer requests time
* [CPP] Clang fix formatting
* [CPP] Reverse clang fix format on plugin.cpp
* [PYTHON/CPP] Fix C++ progressbar printing endl when disabled, fix rounding in python creating infer request message
* [CPP] Fix foramtiing error
* [PYTHON/C++] Refactor network to model based on naming conventions, provide fresh README output example
* [PYTHON/C++] Add example output to C++ README, remove unnecessary device loop
* [BENCHMARK_APP/C++] Fix artifact from refactoring, remove try-catch clause
* Update samples/cpp/benchmark_app/benchmark_app.hpp
Co-authored-by: Nadezhda Ageeva <nkogteva@gmail.com>
* Update samples/cpp/benchmark_app/main.cpp
Co-authored-by: Nadezhda Ageeva <nkogteva@gmail.com>
* Update tools/benchmark_tool/openvino/tools/benchmark/main.py
Co-authored-by: Nadezhda Ageeva <nkogteva@gmail.com>
* Update samples/cpp/benchmark_app/main.cpp
Co-authored-by: Nadezhda Ageeva <nkogteva@gmail.com>
* [CPP] Fix clang errors
* [CPP/PLUGIN Reverse modification to extract to separate task
* Update tools/benchmark_tool/openvino/tools/benchmark/main.py
Co-authored-by: Nadezhda Ageeva <nkogteva@gmail.com>
* Update tools/benchmark_tool/openvino/tools/benchmark/parameters.py
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* Update tools/benchmark_tool/openvino/tools/benchmark/utils/utils.py
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* Update tools/benchmark_tool/openvino/tools/benchmark/main.py
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* [PYTHON/C++/BENCHMARK_APP] Fix language inconsistencies, remove unnecessary checks
* Update pyopenvino.cpp
* [CPP/BENCHMARK_APP] Remove unnecessary try-catch, fix linter errors
* [PYTHON/CPP/BENCHMARK_APP] Revert changes to Core, align version prints usin only provided methods
* [DOCS/BENCHMARK_APP] Update README with proper model examples
* Update README.md
Co-authored-by: Nadezhda Ageeva <nkogteva@gmail.com>
Co-authored-by: Michal Lukaszewski <michal.lukaszewski@intel.com>
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
* [PYTHON] CV2 module required only when loading images
* [PYTHON] Update tools/benchmark_tool/openvino/tools/benchmark/utils/inputs_filling.py
Co-authored-by: Helena Kloosterman <helena.kloosterman@intel.com>
* [PYTHON] Make CV import not global
* [PYTHON] Update format of inputs_filling.py
* [PYTHON] Fix missing newline
* Update tools/benchmark_tool/openvino/tools/benchmark/utils/inputs_filling.py
Co-authored-by: Helena Kloosterman <helena.kloosterman@intel.com>
* [BENCHMARK_APP/PYTHON] Introduce exception when module is installed but fails to import
* [SAMPLES/PYTHON] Sporadic access violation fix through checking whether module was cached
Co-authored-by: Helena Kloosterman <helena.kloosterman@intel.com>
Co-authored-by: Michal Lukaszewski <michal.lukaszewski@intel.com>
* Remove py3.6 constraints
* Cleanup out of date python versions
* Bump pybind11 to v2.10.1 release tag
* [PyOV] Fix wheel requirements for new package versions
Co-authored-by: jiwaszki <jan.iwaszkiewicz@intel.com>
* Benchmark_app python and c/c++ script add op profiling feature
* Fix compile bug about benchmark c/c++ op profiling
* Fix Compile Bug
* Fix issue in PR#12350 : 1. report_type info update; 2. priority of -pc and -pcsort; 3.align format between c++ and python; 4.check 'proportion' for 0 and print N/A; 5.detect the confusing print info
* Fix 8/17 review suggestion
* Fix 8/23 suggestion
* Fix the clang-format issue
* Fix the win-cc issue
* Fix win-cc issue about conversion from 'double' to 'float', possible loss of data
* Push PR by ODT group account
Co-authored-by: Chen Peter <peter.chen@intel.com>
Co-authored-by: Fiona Zhao <fiona.zhao@intel.com>
* Set 'UNDEFINED' as the performancehint default value. Update benchmark app to pass performance hint with none value to MULTI/AUTO plugin.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Using 'UNDEFINED' as the default value of hint in AUTO/MULTI plugin.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Adding a flag instead of adding a default value of hint to check if user configures the performance hint to AUTO/MULTI plugin.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update benchmark python version.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Fix benchmark_app commandline parsing
Object 'str' does not have a contains() method. Replaced by
"searchstring in object" query.
* Fix perf counter output and use consistent units
When ProfilingInfo was bound by pybind11, those entries turned into
Python timedelta objects. This caused two problems:
1. There was a division by 1000 to go to milliseconds. This caused a lot
of precision lost, because `timedelta(microseconds=2300)/1000.0 =
timedelta(2)` and `timedelta(microseconds=33) = timedelta(0)`.
2. When converting those timedelta objects to str, the output is in the
form of (HH:MM:SS.XXXXXX). This is not very useful microsecond based
performance counters.
This change simply reverts everything to printing plain microsecond
based integers.
* Align counter output to ms between Python/C++
* When set batch_size, disable auto batching
* auto-batch only supprt auto plugin
* Set property allow_auto_batching for devices
* cancel static
* Clean Code
* Add test case for testing allow_auto_batching
* Fix the issue after merging latest master branch code
* Add allow_auto_batching in python benchmark_app
* Fix the quotes issue
* When load_network is set to allow-auto-batching, there is no need to read the value of global
* Fix the clang-format
* Change the location
Co-authored-by: River Li <river.li@intel.com>
Co-authored-by: Chen Peter <peter.chen@intel.com>
* [MO] Relax MO upper-bound requirements for TensorFlow and NumPy
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Just debug numpy version
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Pin upper-bounds for NumPy and TensorFlow modules in all reqs files
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Update submodule dependency for open_model_zoo
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Install numpy module first
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Update NumPy version in POT setup.py
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Extend telemetry tests with a set of possible solutions for events
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix build issue
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Update NumPy module version for layer tests
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
Piotr KrzemińskiandZlobin Vladimir
Daria Mityagina
Haiqi Pan
Sebastian Golebiewski
Ilya Naumov
Ilya Churaev
Tomasz Dołbniak
Roman Kazantsev
EvanandYuan Xu
Nadezhda Ageeva
yanlan song
Xiake Sun
Przemyslaw Wysockiandjiwaszki
Alexey LebedevandAnastasia Kuporosova
Nico Galoppo
Piotr Krzemiński
Ekaterina Aidova
Vladimir Paramuzov
Chenhu Wang