[MULTI] pass through to AUTO with CTPUT hint
After this change
-- MULTI doesn't support setting infer request via CPU(4),GPU(8).
-- MULTI doesn't support CompiledModel::set_property() and ExecutableNetwork::GetConfig().
* Implement CTPUT in AUTO code logic
* Add logic to handle device loading failure
* add some code comments
* fix warnning conversion from size_t to int
* Updated code according to comments of bell and wanglei
* the preferred device code path need to be updated with ctput also
* add fallback logic for CTPUT
* Modify the code logic according to bell suggestion
* Add prints for debugging bug
* throw exception when no device to run pipline task
* initialize idleWorkerRequest for CTPUT
* fix getting properties
Signed-off-by: fishbell <bell.song@intel.com>
refine
Signed-off-by: fishbell <bell.song@intel.com>
* fix warning
Signed-off-by: fishbell <bell.song@intel.com>
* fix illegal character on windows
Signed-off-by: fishbell <bell.song@intel.com>
* fix illegal character
Signed-off-by: fishbell <bell.song@intel.com>
add missing include
Signed-off-by: fishbell <bell.song@intel.com>
* more code refine
Signed-off-by: fishbell <bell.song@intel.com>
---------
Signed-off-by: fishbell <bell.song@intel.com>
Co-authored-by: fishbell <bell.song@intel.com>
* 1. Add device blacklist for AUTO plugin.
2. Update the logic to parse out the device candidate list from the inputting config MULTI_DEVICE_PRIORITIES.
3. Update the corresponding mock test cases.
4. Ignore the GTEST warning for the test cases.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update.
* Update.
* Update.
* Add description about blacklist.
* Apply suggestions from code review
Update.
Co-authored-by: yanlan song <bell.song@intel.com>
* Update.
* Apply suggestions from code review
Updated.
Co-authored-by: yanlan song <bell.song@intel.com>
Co-authored-by: River Li <river.li@intel.com>
* Update test case.
* Update test case.
* Update test case.
* Update.
* Update.
---------
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Co-authored-by: yanlan song <bell.song@intel.com>
Co-authored-by: River Li <river.li@intel.com>
Co-authored-by: Shen, Wanglei <wanglei.shen@intel.com>
* Runtime fallback to other devices
* Update properties.hpp
* Update infer callback in AUTO
* Avoid some hang cases
* Add test cases for AUTO runtime fallback
* Replace mockExecutor with ImmediateExecutor
* Update the runtime fallback logic
* Update test case and support the case thar infer failed on CPU_HELP
* Update the test to detect whether to throw exception
* fix the error of CTPUT
* Add lock to AUTO executable network GetContext
* Update variable name in selectOtherDevice API
* Simplify variables and add testcase to improve test coverage
* Fix the issues when release CPU_HELP device and clean up the code
* Clean up code
* Small CoreImpl refactoring
* Removed cache_dirhandling from CPU plugin
* clang-format
* Fixed python tests
* Fix
* Fixed bugs in HETERO case
* Fixed clang-format and warnings in auto plugin
* Added import_export as capability for TEMPLATE plugin
* Commented throw exception from loaded_from_cache
* Fixed clang-formatof ro template plugin
* Enable AUTO to support execution mode hint.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Add test case.
* Set default value "PERFORMANCE" for ov::hint::execution_mode.
* Update.
* Update.
* Correct default ov::hint::execution_mode value for the default value checking test case.
* Update.
* Delete obsolete config.hpp file.
---------
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* AUTO cumulative throughput mode ignore candidate device that fail to load device
* Simplify the judgement logic of whether Auto set to Multi
* Add description about _AutoSetToMulti variable
* Update variable name to _AutoCallMulti
* Refine logic of AUTO execution_devices
* Add loading error massage
* Add test case
* Add filter to execution_devices of MULTI
* Add execution_devices test in load fail sitution
* Simplify the logic of execution_devices
* Update auto_executable_network.cpp
* Update src/plugins/auto/multi_executable_network.cpp
Co-authored-by: yanlan song <bell.song@intel.com>
* Update src/plugins/auto/auto_executable_network.cpp
Co-authored-by: yanlan song <bell.song@intel.com>
* Update test case
---------
Co-authored-by: Chen Peter <peter.chen@intel.com>
Co-authored-by: yanlan song <bell.song@intel.com>
* Check if the device is supported when AUTO retrieves the device list based on the ov::device::priorities.
* Update the logic to handle the situation like -d AUTO:-CPU to benchmark APP.
* Remove MYRIAD and add NVIDIA for AUTO supported devices.
---------
Co-authored-by: Chen Peter <peter.chen@intel.com>
Co-authored-by: Shen, Wanglei <wanglei.shen@intel.com>
* find test case for MultiDeviceInferRequest::SetBlob
* improve line coverage of infer_request
* add test cases for queryState and exception test case for perf count
* fix querystate running fail
* add test case to memory_states.cpp
* rename name of test case
* add memory_states.cpp to CMakeLists.txt
* Use _LogTag to judge whether MULTI
* clang-format intel_gna/memory_states.cpp
* Modify the position of the macro ENABLE_INTEL_CPU in the test case
---------
Co-authored-by: Chen Peter <peter.chen@intel.com>
* add test case for get perf_hint from GetMetric
* Increase Mock GetMetric test sleep time
* add mock test case for getMetric
* add new test case OVAutoExecutableNetworkTest
* convert ov::Any to ov::hint::Priority
* resolve conflict of get_metric.hpp
* add macro ENABLE_INTEL_CPU for gpu test case and fix cases not getting instantiated for cpu test
* fix the issue of running Mock GetMetric test cases fail
* add perf_hint test cases to properties_tests.cpp
* Modify the logic of judging whether it is a single device in ctput mode
* Enable AUTO compiledModel::get_property supporting its properties only.
* Update.
* Update.
* Update some releated test cases.
* Update.
* Update related test case.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
---------
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Co-authored-by: Chen Peter <peter.chen@intel.com>
* Provide ENABLE_STARTUP_FALLBACK property to enable/disable CPU as acceleration
* Add more test cases to ENABLE_STARTUP_FALLBACK property
* Remove unnecessary config
* Remove plugin.hpp
* Update test case by PR#14754
* Update copyright
Co-authored-by: River Li <river.li@intel.com>
* Simplify using namespace
---------
Co-authored-by: Chen Peter <peter.chen@intel.com>
Co-authored-by: River Li <river.li@intel.com>
* remove ... in /src/bindings/c and resolve format
* remove ... in auto and resolve format
* remove ... in auto batch and resolve format
* remove ... in hetero and resolve format
* IE exception
* IE exception
* const
* const
* resolve format and ov::Exception
* ov::Exception
* reset IE::Exception
* remove ov::Exception
* InferenceEngine->IE in auto_schedule.cpp
* fix coredump while runing test case smoke_AutoMultiMock_NumStreamsAndDefaultPerfHintToHWTest
* fix coredump issue of running LoadNetworkWithSecondaryConfigsMockTest
* fix commpile fail in CI
* resolve core dump of running exec_network_get_metrics.cpp and disable some test cases for KEEMBAY and MYRIAD in exec_network_get_metrics.cpp
* remove test cases of MYRIAD and resolve core dump of running KEEMBAYin exec_network_get_metrics.cpp
* resolve conflic of common.h
* resolve conflic of common.h
* resolve common.h complie fail
---------
Co-authored-by: guozhong <guozhong.wang@intel.com>
Co-authored-by: Ilya Churaev <ilya.churaev@intel.com>
* Added -Wall for Clang and GCC
* Fixes
* Don't use /J
* Fixed warnings
* Fixed warnings
* More fixes
* Fixed for MSVC
* Fixed more warnings on Windows
* Suppressed some warnings in template plugin
* Update src/tests/functional/plugin/shared/include/behavior/plugin/caching_tests.hpp
* Added suppression for PT FE
* Suppressed warnings in TF FE
* Suppressed warnings on Core unit tests
* Suppress warnings in python
* Suppressed Windows warning for 3rd party modules
* Suppresed one more warning
* 1. Correct the device list by priority order from high to low.
2. Remove GNA, CUDA, HPU, HDDL, NVIDIA from device list supported by AUTO/MULTI.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Filter out supported device when not specify the candidate device for AUTO plugin.
* Add Debug MSG
* Update.
* Update AUTO mock test cases.
* Update.
* Update.
* Update code style.
---------
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Co-authored-by: Chen Peter <peter.chen@intel.com>
* AUTO CTPUT single device always goes single device logic
* CreateInferRequest goes single device logic when perf hint is ctput
* Optimize single device logic according to bell opinion
* Add code comments
* merge two isCumulative conditions
* add ctput test case
* Remove the redundant header file of auto_ctput_test.cpp
* Modify the test case according to bell comments
* Create README.md for AUTO plugin
* link to contribuation guidance
* Update README.md according to developer documentation guidance
* Update for comments
* Update components for review comments
* Update folder for review comments
* update for wording
* update for comments
* change AUTO default hint to Latency
* Change the comment tput to latency according to wangyang opinion
* fix testcase for MULTI, the hint default value returned by MULTI is throughput
* Remove the redundant testcase and modify the name of the testcase that returns the default value of hint
* Code optimization according to bell opinion, add comments to testcase
* Correct the comments of testcase
* When user sets num_streams, AUTO/MULTI does not set the default hint to HW plugin
* Fix the problem that smoke_AUTO_MULTI_ReturnDefaultHintTest fails to run
* add num_streams and default hint mock testcase
* add auto default perf hint mock testcase
* temp resolution to support model path for CPU in auto
Signed-off-by: fishbell <bell.song@intel.com>
* disable batch when load through model path
Signed-off-by: fishbell <bell.song@intel.com>
* add mark for future release
Signed-off-by: fishbell <bell.song@intel.com>
* implement step1: donotparse batch config if user not set explictly
Signed-off-by: fishbell <bell.song@intel.com>
* correct typo in case
Signed-off-by: fishbell <bell.song@intel.com>
Signed-off-by: fishbell <bell.song@intel.com>
* change gpunum to 3
* hold threads for GPU for MULTI:GPU,CPU
* need to first check if there is a CPU in the device list
* use getNumberOfCPUCores to get CPU cores
* load GPU first
* assign the correct value to multiSContext->_devicePriorities
* load GPU first and load CPU last and set numa for CPU
* MULTI set CPU affinity to “NUMA” during load network
* Load the CPU last while maintaining the original device priority
* not using vector for CPU
* There is no user setting affinity in MULTI, and NUMA is set for the CPU
* pass key ENABLE_HYPER_THREAD to CPU plugin and merge xiaoxia PR
* set ENABLE_HYPER_THREAD to NO
* modify log
* Modify the code according to xiaoxia and wanglei comments
* Modify the code according to bell comments
* [AUTO] update property setting logic.
* Update core::set_property() logic to only support primary property for AUTO and MULTI.
* Separate AUTO and MULTI supported properties.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Add test case.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* 1. revert the changes for ie core.
2. Enable AUTO/MULTI only accepting its own properties.
3. Enable AUTO/MULTI accepting device properties passed from loadnetwork().
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Revert "[AUTO]Update the incorrect config test for Myriad (#13271)"
This reverts commit 0552d98802.
* MULTI only accepts its own properties that is same as AUTO currently.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Add test cases for AUTO/MULTI property test.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update to enable MULTI supporting hw primary property setting throw the compile_model().
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Remove the added test case for setting secondary property through set_property().
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* 1. For AUTO/MULTI, property setting will be passed via core::compile_model() instead of core::set_property().
2. update the logic to infer precision setting that will transform into secondary property setting to each hw device.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* set default value for nstreams when -d AUTO/MULTI and no nstreams setting from command line.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update code format.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Enable BA to support if -d AUTO:MULTI,xxx/MULTI:AUTO,xxx. while AUTO Plugin need to update the logic of generating supported config list to virtual device.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* move device configuration handling outside form function main.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Revert "move device configuration handling outside form function main."
This reverts commit ef77bfc602.
* Update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Setting property performance hint to 'THROUGHPUT' and passing to executable network if no setting for of AUTO/MULTI.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update the logic to handle the ov:device::properties in the ie core.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update test cases.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* 1. Remove the config filter within AUTO plugin when passing the configs to target hardwre device through AUTO plugin.
2. add test cases to check if the secondary properties have been passed into the target device through virtaul device.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Add mock tests to check if device properties works throgh the AUTO or MULTI plugin.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Add mock head file for loadnetwork with device properties testing.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update gtest commit to fix the warning issue appeared in the macro MATCHER_P.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Update to the latest cmake for Windows
* Skip new warnings
* Added compiler flags for old cmake
* Disabled some warnings
* Suppress new warnings
* Fixed some errors
* Disable some warning for lpt
* Fix some warnings
* Disable new warnings
* Fix some warnings
* Disabled Was error for arm
* Fixed some errors
* Fixed some new warnings
* Fixed auto plugin
* Fixed Windows warning
* Fixed Windows
* Disable some warnings
* Some changes
* Small test fixes
* Fixed python build
* Added skip warnings for legacy python API
