* wip remote tests2, fixed smoke_canInferOnUserContext
* completed the OV 1.0 tests for remote blobs
* updated OV 2.0 tests for remote blobs with auto-batching (using the ngraph func that is reshape-able by the batch)
* re-using the DetectionOutput-based ngraph func that is 100% batch-reshapeble
* auto-batching- bare min of the info
* renaming BATCH.MD to the automatic_batching.md, also aligned the link to the new naming convention
* more info and brushed
* added openvino_docs_OV_UG_Automatic_Batching to the main TOC
* Apply suggestions from code review
Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>
* close on the comments, added the code examples
* Apply suggestions from code review
Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>
* Update example
* Update format
* Update docs format
* added couple of more perf considerations
* more code examples
* Apply suggestions from code review
* Apply the rest from code review
* Update header
Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>
* checking the network batchability (internal helper func on top of batch tracking) before doing hetero
* more general logic with respect to batch-ability of the network
* a dynamism check that I've owed from the PR-10560
* using the DO-detached mechanism for early hetero exit, also fixed this flag in the Batching plugin (although minor, as the DO is removed by HETERO)
* adding the dimension tracking logic depending on whether implicitly/expicitly the auto-batching is enabled
* changed the DetectionOutput affinity markup to go over results, also accomodate Convert, so only 2 subgraphs are made by the HETERO
* fixed perf-counters
* explicit auto-batching params that should guarantee the auto-batching is triggered ( to avoid fallback to no-batching when the selected batch1 size is just 1)
* makeConvPoolReluNoReshapes and using that whenever applicable to gaurantee the auto-batching is required (not important for things like plugin/executable-network config tests, but important for the inference-requests)
* getDefaultNGraphFunctionForTheDevice moved to the ov_behavior_test_utils.hpp
* moving the HETERO logic to the Auto-Batch (WIP), reverting to the ALLOW_AUTO_BATCHING and using that in the GPU remote tests
* shortned the vars names in the ie_core and prevented recursive auto-batching calls by checking for exclusive requests and disabling further auto-batching in the plugin, when HETERO is involved
* checking for the batch-dim presence (this is still WA until the https://github.com/openvinotoolkit/openvino/pull/9559 is merged) - pls see CVS-75317
+clang for the ie_core.cpp
* moving the HETERO logic back to the ie_core.cpp, storing the _so internally for no-batch code-path
* auto-batching POC squashed (all commits from auto-batch-2021.3 branch)
(cherry picked from commit d7742f2c747bc514a126cc9a4d5b99f0ff5cbbc7)
* applying/accomodating the API changes after rebase to the master
* replaying modified version of actual batch selection
* eearly experiments with model mem footprint
* changes from rebasing to the latest master
* experimenting with DG1 on the batch size selection, also collecting the mem footprint
* WIP:moving the auto-batching to the icore to let the MULT/AUTO support that, ALLOW_AUTO_BATCHING as a conventional config key. still fials hot device swap
* quick-n-dirty batch footpint vs device total mem
* code style
* testing which models perform badly due to kernels and NOT (batched) footprint
* stub pipeline task to comunicate the readiness rather than promise/future
* quick-n-dirty timeout impl
* explicit _completionTasks,reverting BA to use the timeout
* inputs outputs copies, works with AUTO and demo now
* accomodate the config per device-id, after rebase to the latest master
* allowing the auto-batching only with tput hint to let more conventional tests pass
* fix the pre-mature timeout restaring via waiting for batch1 requests completion
* moved the bacthed request statring ( along with input copies) to the dedicated thread
* [IE CLDNN] Disable bs_fs_yx_bsv16_fsv16 format for int8 convolution
* code style
* increasing the timeout to test the ssd_* models perf (timeout?) issues
* reducing number of output stuff in BA to avoid bloating the logs in experiments
* more aggressive batching for experiments, not limited to 32 and also 4 as a min
* more accurate timeout debugging info
* getting the reqs limitation from the plugin SetConfig as well
* refactor the reshape logic a bit to accomodate CPU for bathcing, also added remeote context
* let the benchamrk_app to consume specific batch values for the auto-batching such as BATCH:GPU(4)
* auto-batching functional test (with results check vs ref) and GPU instance for that
* fixed arithemtic on blobs ptrs
* clang
* handling possible batched network failure
* BATCH as the constants device name in test
* ENABLE_BATCH
* func tests for CPU, also DetectionOutput hetero tests (CPU and GPU)
* DetectionOutput hetero test for the CPU
* reenabling the Auto-Batching in the AUTO
* auto-batching device enabled in the test
* fixed the DO test
* improve the loading loop logic
* brushed the config keys
* allow hetero code-path for explicit device name like BATCH:GPU(4), used in the hetero code-path tests
* fix the test after refactoring
* clang
* moving ThreadSafeQueue to the ie_parallel, as it is re-used in the AUTO/MULTI and BATCH now
* auto-batching hetero test (subgraph with DetectionOutput)
* fixed minor changes that were result of experiments with impl
* code-style
* brushing, disabling CPU's HETERO tests until planned activity for 22.2
* removing home-baked MAX_BATCH_SZIE and swicthing to the official impl by GPU team
* remote blobs tests for the auto-batching (old API)
* brushed names a bit
* CreateContext and LoadNEtwork with context for the Auto-Batching plus remote-blobs tests
* fixed the ieUnitTests with adding CreateContext stub to the MockICore
* clang
* improved remote-blobs tests
* revert the back BA from exeprimenents with AB + device_use_mem
* conformance tests for BATCH, alos batch size 1 is default for BATCH:DEVICE
* remote blobs 2.0 tests, issue with context having the orig device name
* debugging DG1 perf drop (presumably due to non-fitting the device-mem)
* disbaling WA with batch/=2 for excesive mem footptint, leaving only streams 2
* remote blobs 2.0 tests for different tensor sharing types
* converting assert to throw to accomodate legacy API where the lock() was possible to be called
* revert the timeout back to avoid mixing the studies, fixed the footprint calc
* reverting to estimating the max batch by extrapolating from bacth1 size
* more conservative footptint etimation (with bacth1), graceful bacth 1 handling without duplication
* even graceful batch 1 handling without duplication
* WA for MAX_BATCH_SIZE failure, removing batch4 as a min for the auto-batching
* AutoBatchPlugin -> ov_auto_batch_plugin
* WA for gcc 4.8
* clang
* fix misprint
* fixed errors resulted from recent OV's Variant to Any transition
* skip auto-batching for already-batched networks
* AUTO_BATCH_TIMEOUT and tests
* GPU-specific L3
* switched to pure config, also improved ALLOW_AUTO_BATCHING config key handling logic
* debugging device info
* enabling the config tests for the GPU and fixing the Auto-batching tests to pass
* making the default (when not recognized the driver) cache size more aggressive, to accomodate recent HW with old drivers
* skip auto-batching for RNNs and alikes (e.g. single CHW input)
* fixed fallback to the bacth1 and moved HETERO path under condition to avoid bloating
* brushing
* Auto plugin GetMetric support gpu auto-batch
Signed-off-by: Hu, Yuan2 <yuan2.hu@intel.com>
* add test case
Signed-off-by: Hu, Yuan2 <yuan2.hu@intel.com>
* add comments on test
Signed-off-by: Hu, Yuan2 <yuan2.hu@intel.com>
* brushing the vars names, alos adding the excpetion handling
* disabling the auto-batching for the networks with non-batched outputs and faster-rcnn and alikes (CVS-74085) to minimize the of #failures
* add try catch
Signed-off-by: Hu, Yuan2 <yuan2.hu@intel.com>
* brushing the code changed in the GPU plugin
* Auto-Batch requests tests
* brushed varibles a bit (ref)
* cleaned debug output from the ie_core
* cleaned cmake for the Auto-Batch
* removed batchN estimation from batch1
* cleaned from debug printf
* comments, cleanup
* WA the mock test errors introduced with merging the https://github.com/myshevts/openvino/pull/13
* Adding back removed batchN estimation from batch1 to debug degradations on DG1 (resulted from too optimistic MAX_BATCH_SIZE?). This partially reverts commit e8f1738ac1.
* brushing ie_core.cpp
* fix 32bit compilation
* Code review: ENABLE_AUTO_BATCH
* consolidate the auot-batching logic in ie_core.cpp into single ApplyAutoBAtching
* renamed brushed the OPTIMAL_BATCH (now with_SIZE) and mimicks the MAX_BATCH_SZIE wrt MODEL_PTR
* default value for the OPTIMAL_BATCH_SIZE
* clang
* accomodate new func tests location
* fix shuffle of headers after clang + copyrights
* fixed misprint made during code refactoring
* moving the common therad-safe containers (like ThreadSafeQueue) to the dedicated dev_api header
* switch from the device name to the OPTIMAL_BATCH_SIZE metric presence as a conditin to consider Auto-Batching
* switching from the unsafe size() and minimizing time under lock
* code style
* brushed the ApplyAutoBatching
* brushed the netric/config names and descriptions
* completed the core intergration tests for the auto-batching
* ExecGraphInfo and check for incorrect cfg
* removed explicit dependencies from cmake file of the plugin
* disabling Auto-Batching thru the tput hint (to preserve current product default), only excplicit like BATCH:GPU used in the tests
Co-authored-by: Roman Lyamin <roman.lyamin@intel.com>
Co-authored-by: Hu, Yuan2 <yuan2.hu@intel.com>
* rebasing the perf-modes-2021.3 to the 2021.4
Caveats:
the (explicit) setting #streams is not disabled (as it was before for experiments with DLBenchmark), and the logic slighlty differ (streamsSet)
(cherry picked from commit 1ae1edc0ed)
* overriding streams (to force the TPUT mode to the DLBenchnark)
(cherry picked from commit 7f506cda31)
* disabling reducing #streams to fully mimic baseline c4df94d42d of the 2021.3 (before experiments)
(cherry picked from commit 85073dd1dd)
* clang/identation
(cherry picked from commit 050a4155a9)
* splitting the Transformation to general and CPU specific.
Now hopefully,this fully mimics the baseline c4df94d42d of the 2021.3 (before experiments), as the streams reduce num (as well as early exit on GRU/LSTM/TensorIterator) is deisabled
(cherry picked from commit e98b2c1a67)
* disabling GRU/LSTM/TI + reducing of streams + 5D considered compute-limited only for int8
(cherry picked from commit 32b8d80dee)
* refactored to avoid compute_limited_ratio, reverted the reducing #streams, removed LSTM from limitations
(cherry picked from commit f2b972171b)
* isa-based threshold logic
(cherry picked from commit b218457e1a)
* mode->hint
(cherry picked from commit ec20aa8eca)
* optional PERFORMANCE_HINT_NUM_REQUESTS
(cherry picked from commit 5a3883e3f3)
* moving the perfHints to the common OV config class + initial tests (CPU only, as the actual AUTO/MULTI should be accommodated on the master)
(cherry picked from commit (then fixed)45bafe7d527f466507dea0693aeed51be4ebf776)
* AUTO support for PerfHints
* MULTI support for PerfHints
* Enabling Perf hints for the GPU plugin
* brushing settings output a bit
* disabling "throughput" perf hint being default (until OV 2.0)
* uncommenting the logic which was disabled to force the DLBenchmark to use the throughput mode by default
* removing dead and experimental code, and debug printfs
* clang/code-style
* code-review remarks
* Moved the output of the actual params that the hint produced to the right place
* aligning MULTI's GetConfig beh to HETERO's as captured in the preso (CVS-59960) ratified with the ArchForum
* clang
* benchmark_app brushing
* Update inference-engine/samples/benchmark_app/README.md
* propagating the perf hints thru one more scenario in the merged AUTO-MULTI
* fixed mispint
* Python benchmark_app update for perf hints
* addresssing reviewers comments on the python benchmark_app
* simplifying/brushing logic a bit
* refactor the heuristic to the separate file (to be shared with iGPU soon)
* refactor conversion of modes to the specific GPU config per feedback from Vladimir
* fixed multi setConfig to throw on unsupported config keys
* fixed hetero setConfig to throw on unsupported config keys. simplified the test on the incorrect keys
* aligning MULTI tests to the new (fixed) behaviour - also aligned with AUTO and HETERO
* fixed multi setConfig to throw on unsupported config keys- updating the MYX tests
* change the deprecated method to the recent
* first ver of the hybrid cores aware CPU streams (+debug info)
* more debug and fixed sum threads
* disabled NUMA pinning to experiment with affinity via OS
* further brushing of stream to core type logic
* hybrid CPU-aware getNumberOfCPUCores
* adding check on the efficiency
* experimental TBB package (that cmake should pull from the internal server)
* iterating over core types in the reversed order (so the big cores are populated first in case user specified less than all #threads)
* adding back the NUMA affinity code-path for the full validation (incl 2 sockets Windows Server)
* cpplint fix and tabbing the #if clauses for the readbility
* pre-production TBB from internal server
* wrapping over #cores/types
* wrapping over #cores/types, ver 2
* wrapping over #streams instead
* disabling warnings as errors for a while (to unlock testing)
* accomodating new TBB layout for dependencies.bat
* next tbb ver (with debug binaries that probably can unlock the commodity builds, without playing product_configs)
* minor brushing for experiments (so that pinning can be disabled)
* minor brushing from code review
* Updating the SHA hash which appeared when rebasing to the master
* WIP refactoring
* Completed refactoring of the "config" phase of the cpu stream executor and on-the-fly streams to core types mapping
* making the benchmark_app aware about new pinning mode
* Brushing a bit (in preparation for the "soft" affinity)
* map to vector to simplify the things
* updated executors comparison
* more fine-grained pinning scheme for the HYBRID (required to allow all cores on 2+8 1+4, and other LITTLE-skewed scenarios)
TODO: seprate little to big ratio for the fp322 and int8 (and pass the fp32Only flag to the MakeDefaultMultiTHreaded)
* separating fp32 and int8 intensive cases for hybrid execution, also leveraging the HT if the #big_cores is small, refactored. also switched to the 2021.2 oneTBB RC package
* code style
* stripped tbb archives from unused folders and files, also has to rename the LICENSE.txt to the LICENSE to match existing OV packaging tools
* assigning nodeId regradless of pinning mode
* tests OpenCV builds with same 2021.2 oneTBB, ubuntu 18/20
* cmake install paths for oneTBB, alos a ie_parallel.cmake warning on older ver of TBB
* Updated latency case desc to cover multi-socket machines
* adding centos8 OCV with oneTBB build
updating TBB drops with hwloc shared libs added.
* enabled internal OCV from THIRD_PARTY_SERVER to test thru CI..
Added Centos7 notbb OCV build (until g-api get ready for onetbb) to unlock the Centos7 CI build
* separate rpath log to respect one-tbb specific paths
* fixed SEQ code-path
* fixed doc misprint
* allowing all cores in 2+8 for int8 as well
* cleaned from debug printfs
* HYBRID_AWARE pinning option for the Python benchmark_app
* OpenVINO Hybrid CPUs support
* Remove custom::task_arena abstraction layout
* Get back to the custom::task_arena interface
* Add windows.h inclusion
* Fix typo in macro name
* Separate TBB and TBBbind packages
* Fix compile-time conditions
* Fix preprocessors conditions
* Fix typo
* Fix linking
* make linking private
* Fix typo
* Fix target_compile_definitions syntax
* Implement CMake install logic, update sha hash for the tbbbind_2_4 package
* Add tbbbind_2_4 required paths to setup_vars
* Update CI paths
* Include ie_parallel.hpp to ie_system_conf.cpp
* Try to update dependencies scripts
* Try to fix dependencies.bat
* Modify dependencies script
* Use static tbbbind_2_4 library
* Remove redundant paths from CI
* Revert "cleaned from debug printfs"
This reverts commit 82c9bd90c5.
# Conflicts:
# inference-engine/src/inference_engine/os/win/win_system_conf.cpp
# inference-engine/src/inference_engine/threading/ie_cpu_streams_executor.cpp
# inference-engine/src/mkldnn_plugin/config.cpp
* Update tbbbind package version
* fixed compilation
* removing the direct tbb::info calls from CPU plugin, to aggregate everything in the single module (that exposes the higher level APIs)
* Update tbbbind package version
(cherry picked from commit f66b8f6aa6)
* compilation fix
* brushing the headers a bit
* Make custom::task_arena inherited from tbb::task_arena
* change to the latest TBB API, and more debug printfs
* code-style
* ARM compilation
* aligned "failed system config" between OV and TBB (by using '-1')
* macos compilation fix
* default arena creation (to make sure all code-path have that fallback)
* Incapsulate all TBB versions related logic inside the custom namespace
* Move custom layer header to internal scope + minor improvements
* with all NUMA/Hybrid checks now consolidated in the custom_arena, cleaning the ugly ifdefs thta we had
* Introduce new ThreadBindingType + fix compilation
* fixing OMP compilation
* OpenVINO Hybrid CPUs support
* Remove custom::task_arena abstraction layout
* Get back to the custom::task_arena interface
* Add windows.h inclusion
* Fix typo in macro name
* Separate TBB and TBBbind packages
* Fix compile-time conditions
* Fix preprocessors conditions
* Fix typo
* Fix linking
* make linking private
* Fix typo
* Fix target_compile_definitions syntax
* Implement CMake install logic, update sha hash for the tbbbind_2_4 package
* Add tbbbind_2_4 required paths to setup_vars
* Update CI paths
* Include ie_parallel.hpp to ie_system_conf.cpp
* Try to update dependencies scripts
* Try to fix dependencies.bat
* Modify dependencies script
* Use static tbbbind_2_4 library
* Remove redundant paths from CI
* Update tbbbind package version
* Make custom::task_arena inherited from tbb::task_arena
* Incapsulate all TBB versions related logic inside the custom namespace
* Move custom layer header to internal scope + minor improvements
* Introduce new ThreadBindingType + fix compilation
* Fix compilation
* Use public tbbbind_2_4 package
* fixed macos build, corrected comments/desc
* reverted to the default binding selection logic ( to preserve the legacy beh)
* OpenVINO Hybrid CPUs support
* Remove custom::task_arena abstraction layout
* Get back to the custom::task_arena interface
* Add windows.h inclusion
* Fix typo in macro name
* Separate TBB and TBBbind packages
* Fix compile-time conditions
* Fix preprocessors conditions
* Fix typo
* Fix linking
* make linking private
* Fix typo
* Fix target_compile_definitions syntax
* Implement CMake install logic, update sha hash for the tbbbind_2_4 package
* Add tbbbind_2_4 required paths to setup_vars
* Update CI paths
* Include ie_parallel.hpp to ie_system_conf.cpp
* Try to update dependencies scripts
* Try to fix dependencies.bat
* Modify dependencies script
* Use static tbbbind_2_4 library
* Remove redundant paths from CI
* Update tbbbind package version
* Make custom::task_arena inherited from tbb::task_arena
* Incapsulate all TBB versions related logic inside the custom namespace
* Move custom layer header to internal scope + minor improvements
* Introduce new ThreadBindingType + fix compilation
* Fix compilation
* Use public tbbbind_2_4 package
* Apply review comments
* Fix compilation without tbbbind_2_4
* Fix compilation with different TBB versions
* code review remarks
* fix for the NONE pinning code-path under HYBRID_AWAR
* whitespace and cleaning the debug printfs (per review)
* code-review comments
* fixed code-style
Co-authored-by: Kochin, Ivan <ivan.kochin@intel.com>
Co-authored-by: Kochin Ivan <kochin.ivan@intel.com>
* Adding MYRIAD_THROUGHPUT_STREAMS to the list of plugin's supported config vals (omitted incorrectly) and enabing streams for the myriad devices in the benchmark_app
* docs update and python benchmark_app
* async network loading in the MULTI. makes the overall load time as MAX of the individual devices loading timings, as opposite to the current SUM
* correct way of getting perf counters flag for the MULTI (adding to the async load PR, as this is minor change)
* accomodating remark from the code review- MULTI enables the perf counters only if all devices support/enable that
* zero-copy (assuming determenistic app-level scheduling) for the multi-device, via "borrowing" the corresponding device-specific blobs and letting the app to implicitly use these
* Optimized Infer Request Scheduling
* remoteblob checks in the conventional SetBlob
* correctly (with status) reporting NOT_IMPLEMENTED
* SetBlob to accomodate for the RemoteBobs
* Tests for remote blobs support via MULTI: creating the shared_test in case the other (closed source) plugins would want to use that (in the private shared_tests instantiations).
Also instantiating the remote blobs tests for the some basic combinations to test the MULTI supports them
* macos compilation (and general plugin platform support) fix
* shuffled files, so that the MULTI tests are now part of the ieFuncTests (and need no separate target). Also brushed the macro that handales the NOT_IMPLEMENTED as bit
* further shuffled files, so that the initial MULTI tests are now part of the IE tests, yet specific instances do need separate targets
* Fixed misprint
* Brushing the code and comments a bit
* further brushing of the ScheduleToWorkerRequest: moving the task execution directly into the loop over devices (avoids pointers and 'else' clause)
* 1) zero-copy (assuming determenistic app-level scheduling) for the multi-device, via "borrowing" the corresponding device-specific blobs and letting the app to implicitly use these
2) Initial MULTI section in the opt guide (primarily to document a tip on helping the MULTI to keep the zero-copy path)
* [MULTI] remote context support and associated scheduling (respecting the remote data affinity)
* fix CentOS (old) gcc issue: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81880
since the intriduced therad_local string is template the bug manifests itself (and the string is not allocated/initialized).
the QA is to wrap the std::string into the function
* further fix for the old gcc versions issue, now with non-trivial thread_local destruction sefault: switching from the std::string to the plain const char*
* additional tests for the MULTI and remote blobs (no remote context and multi GPUs cases)
* fix for the tests (that now can check for more specific NotImplemented exeption).
Alos couple of line endings
it is easy to capture when there are 2 app-level inference requests, but only single worker (MULTI) request
main thread | callback thread
___________________________________________________________________________
| <in the callback, the worker request>
| <the request returns itself to the "idle" queue>
| 1) idleGuard.Release()->try_push(workerRequestPtr)
2)<notified on vacant worker arrived via callback> |
3) starts another request with StartAsync | ...
4) <in the ThisRequestExecutor::run()> |
workerInferRequest->_task = std::move(task); | if (_inferPipelineTasks.try_pop(workerRequestPtr->task))
the last line introduces DATA RACE (sporadically manifested in the bad_function_call exception), the fix is in this commit
* Optimized Infer Request Scheduling
* Fixed misprint
* Brushing the code and comments a bit
* further brushing of the ScheduleToWorkerRequest: moving the task execution directly into the loop over devices (avoids pointers and 'else' clause)
* 1) zero-copy (assuming determenistic app-level scheduling) for the multi-device, via "borrowing" the corresponding device-specific blobs and letting the app to implicitly use these
2) Initial MULTI section in the opt guide (primarily to document a tip on helping the MULTI to keep the zero-copy path)
Co-authored-by: apankratovantonp <anton.pankratov@intel.com>
the point is that we should check the ORIGINALLY (largest) list of the devices (actually ExecutableNetworks for them) to see if the device is just added back
* LayerNorm(PyTorch/HuggingFace pattern)->MVN+Mul+Add. Improves perf on BERT by 5%
* deducing the across_channels from axes passed to the MVN op.
axes are normalized. if no axes is specified, falling back to the (previously) default across_channel value
Co-authored-by: myshevts <maim.y.shevtsov@intel.com>
Maxim Shevtsov
