* Remove inits, update main one
* Fix stacklevel
* Testing wrong solution
* Testing test test
* Fix test test test test
* mo modules mo problems
* Xfail test that check stdout/err?
* not so correct solution to circular imports
* Fix or not to fix
* CMake magic, co-authors: my best team
* Fix package imports
* Fix tools inits
* Fix ovc tf
* Fix Q000
* Fix F401
* Fix linters
* Add save_model
* Remove LayoutMap
* Move test_utils to 'internal modules'
* First testing
* Missing Type
* Expand main namespace
* Change some more tests
* Add OVAny to namespace
* Add Constant and Parameter to namespace
* More tests changes
* Fix inits
* Add layout_helpers to main namespace
* Revert CMake and linux.yml with ovc
* Update main inits
* Remove MO from tools inits
* changes to init files
* Fix tests
---------
Co-authored-by: Sergey Lyalin <sergey.lyalin@intel.com>
* [GNA] Fix for GeminiLake detection
* Added HWGeneration::GNA_1_0_E enumerator
Added DeviceVersion::GNAEmbedded1_0 enumerator, changed the meaning of DeviceVersion::GNA1_0.
Updated ConvLowPrecision test with all supported targets
* [GNA] Extended a few tests with GNA1.0
* Add debug messages
* Add more debug messages
* Add more messages
* Disable proxy plugin
* Enable proxy and suppress NVIDIA
* Remove disabling NVIDIA
* Update hidden device only if proxy settings were changed
* Use static unavailable device to load unavailable device only one time
for the process
* Removed debug messages and add global mutexes for unavailable plugins
* Modify the condition making batch interpretation true/false
- When the user is Convert for Constant node, and tensor is 1d,
- Set needBatchInterpretation to true
* Narrow down the range of the condition
* Merge the condition
* Add additional condition not to check self node
* Fix incomplete condition
* Check if all inputs to binary eltwise is 1d
* Change code style
* [MO] compress_to_fp16=True by default (2dn attempt)
* fix unit-tests
* second round of fixin unit-tests
* set compress_to_fp16 default to True in ovc/cli_parser.py
* use save_model in mo_python_api_tests
* enforce compress_to_fp16=False in test_zero_copy
* selectively compress depending on the path user has chosen to generate IR
* corrected doc
* allow compress_to_fp16=False/True for ovc
* doc and unit-tests failing fix
* user save_model in ovc cli tool
* revert back serialize and compress_model but into main instead of moc_emit_ir
* cover more argument combinations for cli tool and convert_model
* [GPU] Improvement for buffer dump
+ added OV_GPU_DumpLayersInput to support dump input layers
+ added OV_GPU_DumpLayersRawBinary to make binary dump
+ added OV_GPU_LoadDumpRawBinary to use binary dump as input
+ binary dump naming rule layername_datatype_tensor_format.bin
Signed-off-by: Min, Byungil <byungil.min@intel.com>
* Previously reorder / permute was not allocating its memory at build time thought the shape has upper bound
* Update src/plugins/intel_gpu/src/graph/permute.cpp
Co-authored-by: Sergey Shlyapnikov <Sergeishlyapnikov@gmail.com>
* Fix as review comment
---------
Co-authored-by: Sergey Shlyapnikov <Sergeishlyapnikov@gmail.com>
* Add static shape adapter
- Adapters holds CPU dimension which can be reference to it or vector
- Add ov::optional for holding optional result from shape inference
- Add new `infer` function in `IStaticShapeInfer`
* Temporary support of StaticShape
* Minor corrections in ShapeInferenceTA
* Migrate shape_infer to new interface version
* Replace StaticShape by adapter implementation
* Replace IShapeInferCommon by IStaticShapeInfer
* Correct code formatting
* Fix build issues
* NodeValidationFailure::create for StaticShapeRef
* Review RegionYolo for shape inference:
- Check dynamic shape and label propagation
- Check static shape inference
- Review shape_infer template implementation
- Update unit test
* Remove commented test code
* Correct flatten dim calculation
* Added Torchscript Backend
* First commit for backend with Torch FX Decoder
* Merging changes from Torch FX branch
* Torch FX initial fixes (Temporary)
* Fixed type/shape issues in Torch FX decoder
* Added translation for built-in getitem
* MaxPool update & Output shape fix (Torch FX)
* Torch FX graph outputs fix
* Torch FX support for sigmoid and slu_
* Torch FX graph module caching
* Torch Fx partitioner cache removed
* Torch FX initial getitem replacer added
* Index check for torch fx getitem replacer
* Debug print removed from partitioner
* Added environment variables for pytorch tracing mode and openvino device
* FX translation fix for getitem & getitem replacer removed
* Added checks for PyTorch tracing mode environment variable
* Adding compile mode for fallback
* Added more ops for resnet18
* Added a check for environment variable
* Generalized addmm to work with torchscript and torchfx
* Added the missing batch_norm.default translation
* fx_backend: include get_attr ops to the partitions
* AddeTODO note t to improvget_attr algorithm
* created function for adding get_attr nodes
* fx_backend: added aten.mul.Tensor, re-enabled aten.empty.memory_format
* fx_backend: Additional op support/improvement for Inception V3
* Added comment for fix 64-bit to 32-bit max int conversion
* fx_backend: Update for avg_poolnd to support 3 inputs
* Fixed erorr in decoder.py
* TorchFX caching fix
* Torch backend, op support for Stable Diff. & BERT
* Arranged ops in order and added torch tensor mapping
* Added support for more ops for super glue
* TorchFX: Initial permanent fallback
* TorchFX: New ops for improved TorchVision support
* TorchFX backend optimizations for partitioning and tmp fallback
* working operator updates for superglue
* Updates to operators for superglue
* Removed max.dim and stack
* Cleanup
* Cleanup
* Fixed a couple of syntax issues
* Fixed a couple of syntax issues
* Added missing method to TorchFX Decoder
* Added missing method to TorchFX Decoder
* Removed redundant code for transpose
* TorchFX: Initial StableDiffusion support
* PyTorch decoder ovtype to ctype fix for int64
* Added ops for distilbert
* Fixed few unnecessary include statements
* Seperated TorchFX and TorchScript decoders
* Modified import statements to reflect two decoders
* f64 fix for TorchFX
* Import fix for PyTorch backend modules
* TorchFX serialize graph for debugging (Temporary)
* Serialize and load back feature enabled for TorchFX
* Temporary optimization to remove Broadcast
* Temporary SoftmaxRehapeElimination pass is added
* TorchFX custom model cache directory
* PyTorch bitwise translation, conversion checks enabled
* Naming fix in make_list_construct
* TorchFX: Added comments to Softmax and Slice translations
* translate_chunk temporarily removed for TS backend
* Fixed linter issues
* Addressed clang formatting issues
* Fixed few more clang and linter issues
* Fixed tests to use ts_decoder
* Fixed naming convention issues
* Added missing import
* Added inlined_inputs to TorchScriptDecoder
* Added tests for torch fx backend
* Removed magic numbers in PyTorch decoder utils
* TorchFX decoder data type fix
* Added cast from size_t to int
* TorchFX output handling code cleanup
* TorchFX: Use detached input tensor
* Added missing cast from size_t to int
* Added static cast in group_norm
* Fixed casting issue in split
---------
Co-authored-by: ynimmaga <yamini.nimmagadda@intel.com>
Co-authored-by: Cavus Mustafa <mustafa.cavus@intel.com>
* Add subgraph body comparison
* Avoid confusing function name
* Skip failing snippet test
* Skip some ov_snippets_func_tests
* Derive comparison flags
* Skip snippet test
* Drop on bodies mismatch
---------
Co-authored-by: Michal Lukaszewski <michal.lukaszewski@intel.com>
Co-authored-by: Ivan Tikhonov <ivan.tikhonov@intel.com>
* added display of sum of memory usage per operation block and per memory type in GNA dump file
* fixed some clang format issues
* forgot to save file - missing clang fix
* changes according to review - added counting memory usage per layer and names changes
* clang changes
* changes according to review
* clang fixes
* Add static shape adapter
- Adapters holds CPU dimension which can be reference to it or vector
- Add ov::optional for holding optional result from shape inference
- Add new `infer` function in `IStaticShapeInfer`
* Temporary support of StaticShape
* Minor corrections in ShapeInferenceTA
* Migrate shape_infer to new interface version
* Replace StaticShape by adapter implementation
* Replace IShapeInferCommon by IStaticShapeInfer
* Correct code formatting
* Fix build issues
* NodeValidationFailure::create for StaticShapeRef
* Review ReorgYolo op for shape inference
- Check partial shape and label propagation
- Check shape_infer implementation
- Update tests for static and dynamic shapes
* Change dim::is_valid to dim::is_empty
* Query the property optimal_number_of_infer_requests from plugin first if plugin supported.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Query the property ov::optimal_number_of_infer_request of actual device first and use it as the result of AutoExecutableNetwork property.
* Updated.
* Updated.
* Update.
* Update.
---------
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Co-authored-by: Chen Peter <peter.chen@intel.com>
* Update selection logic for 3rd part devices within AUTO plugin.
* Remove MYRAID from AUTO test case.
* Update.
* Update.
* Update.
---------
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Co-authored-by: Chen Peter <peter.chen@intel.com>
* [GPU] Fix cl kernel build error(#18513)
* [GPU] Rollback cl kernel code change and add type converions to activatino function)
* [GPU] Use output data type instead of unit type in MakeActivationJitConstants
* [GPU] remove unused code and add comments
- add unit test
* Add static shape adapter
- Adapters holds CPU dimension which can be reference to it or vector
- Add ov::optional for holding optional result from shape inference
- Add new `infer` function in `IStaticShapeInfer`
* Temporary support of StaticShape
* Minor corrections in ShapeInferenceTA
* Migrate shape_infer to new interface version
* Replace StaticShape by adapter implementation
* Replace IShapeInferCommon by IStaticShapeInfer
* Correct code formatting
* Fix build issues
* NodeValidationFailure::create for StaticShapeRef
* Update RegionYolo to use ngraph shape infer
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Add dynamic TCs for ov_gpu_func_tests
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Add shape infer TCs for ov_gpu_unit_tests
Signed-off-by: Andrew Park <andrew.park@intel.com>
---------
Signed-off-by: Andrew Park <andrew.park@intel.com>
* 1. Setting property ov::compilation_num_threads for actual device (not CPU) selected by AUTO.
2. Add the test case.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* check if the selected device supports this property before enabling this property.
* Update.
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Updated.
* Updated.
* Using the ov::device::properties, instead of the device name, to check if secondary property is set for hardware device.
* Update.
* Update.
* Update.
* Update the property handling logic to support the ov::device::properties setting.
* Update.
* Update.
* Update the case to check if the secondary propery has high priority.
---------
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
* Draft version of save_model function (C++, Python)
* Fixed code style.
* Fixed incorrect test model construction for Python save_model tests
* Minor improvements in code readability
* Minor adjustment based on PR review
* merget loadnetwork and importnetwork for cpu plugin
* fix testcase
* remove multithreading config saving
* modify test case
* separate function
* modify function name
* save model_prefer_threads to cache
* remove function encapsulation for config
* Update op creation for DetectionOutput-8 w/o num_classes attribute
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Update DetectionOutput to use ngraph shape inference
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Add dynamic TCs for ov_gpu_func_tests
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Fix issues for detection output cpu impl selection on dynamic shape
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Update detection_output primitive API and funcs for serialization
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Add shape infer TCs for ov_gpu_unit_tests
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Fix build failure in azure pipeline
Signed-off-by: Andrew Park <andrew.park@intel.com>
---------
Signed-off-by: Andrew Park <andrew.park@intel.com>
* WIP: parameters cleanup
* Removed debug output, fixed CLI
* Fixed python objects conversion
* Finally renamed mmap to share_weights
* Fixed TF conversion from a file or a directory
* Fixed obvious errors in unit tests
* Deleted layouts from OVC. Fixed most of the fails in ovc unit tests (there are still failures)
* Clenaup other references to layouts and fixed --version
* Fixed case when two model files are passed in TF case
* Fixed multiple model parts passing in ovc command line
* Tests fixed, support of unnamed input in cli parser.
* Remove convert_model from runtime.
* Changed silent to verbose.
* Removed transform param.
* Removed example_input, share_weights from ovc cli tool.
* Remove wrong change.
* Test fix.
* Code corrections.
* Returned comment.
* WA to fix process hanging after extension loading.
* Removed not needed code.
* Added comment.
---------
Co-authored-by: Sergey Lyalin <sergey.lyalin@intel.com>
* add streams_info_table init in the constructor of config
* add refresh _proc_type_table
* add get_org_proc_type_table
* add numa_node per stream in reserve_available_cpus()
* fix warning
* remove log
* fix code style
* fix gpu test build failed issue, modify debug info
* fix code style
* fix build failed on macos
* fix code style
* select socket in reserve cpu on 2 sockets platform
* fix build failed on macos
* modify numa node selecting in reserve_cpu_by_streams_info
* add test case
* fix code style
* modify test case
* fix core dump
* fix core dumped on pin=NUMA
* fix test failed on macos
* fix reserve cpu is wrong when streams_info_table=[1 1 36] proc_type_table=[36 18 0 18]
* add test case in LinuxCpuReserve
* modify test case
* add test case in cpu_reserve_test
* add cpu_stream_info_test
* modify enum
* fix test failed
* change int to size_t
* remove invalid code, fix running failed on macos
* modify LinuxCpuStreamType test case, move ie_cpu_streams_info.hpp to openvino/runtime/threading/
* fix code sytle
* modify enum name
* add comments in test case
* fix build issue
* change IE_ASSERT to OPENVINO_ASSERT
* fix test failed on macos and windows
* updated test cases due to the cpu mapping is changed
* enable numa_node_id and socket_id in streams_info_table
* fix code style issue
* fix document issue
* add get socket id interface
* fix segment fault on machine enabled socket_id=1 with numactl command
* fix numactl failed on four numa nodes machine
* remove compile warning
* fix numa_node_id=-1
* fix test case failed on macos
* fix test failed on macos
* fix numa_node_id=0 on macos
* Solve conflicts with master branch
* separate test cases for Linux and Mac/Windows
* update code style for windows compiler
* fix comments
* fix code style
* fix code style
* remove _plugin_mutex, fix comments
* fix code style
* fix code style
* add get_num_sockets
* fix cpu reserve issue in latency mode,ANY core on RPL machine
* add cpu reserve test case
---------
Co-authored-by: Wanglei Shen <wanglei.shen@intel.com>
* Handle sequence ops with non constant W, R, B inputs
Ticket: CVS-49207
* fix tests
* tests
* fix gna tests
* add include
* change type for seqLenIdx
* rename is_constfoldable to is_on_constant_path
---------
Co-authored-by: Ivan Tikhonov <ivan.tikhonov@intel.com>
* [PT FE]: support nested inputs in example_inputs and arg dicts with different argtypes
* accept hande lists as inputs
* Update tools/ovc/openvino/tools/ovc/moc_frontend/pytorch_frontend_utils.py
* update tests and add comments in code
* fix for custom types in annotations and duplicate in mo
* Update tools/mo/openvino/tools/mo/moc_frontend/pytorch_frontend_utils.py
* [GPU] Reorder weights refactoring (#17787)
* [GPU] Fix DG2 with weights optimization
* [GPU] Fix DG2 with weights optimization
* [GPU] Fix DG2 with weights optimization
* [GPU] Fix DG2 with weights optimization
* [GPU] Fix inner order description for some of formats
* [GPU] Fix expected number of primitives in test
---------
Co-authored-by: Roman Lyamin <Roman.Lyamin@intel.com>
Co-authored-by: Sergey Shlyapnikov <sergey.shlyapnikov@intel.com>
* [GPU] Permute f and y axes
It is supported cases where y and f (and X if it is not equal 1) axes size divisible by 4,8 or 16.
added kernel to switch f and y axes in 4d model blocked and planar formats
added test fot tests
* Added subgroup read/write to THREE_DIM_TRANSPOSE kernel case.
* Better checking of is SIMD size supported.
* Added support for long type to subgroup read/write.
* Added subgroup read/write support to 2d permute.
* Fixed win build issue.
* Changed f and y indexes in iteration.
* Added vector read/write.
* Fixed j_times calculation.
* Better naming.
* Rollback test logic.
* Fixed fusion logic.
* Accept only supported blocked layouts and SIMD sizes.
---------
Co-authored-by: Mykhailo Hnap <mykhailo.hnap@capgemini.com>
Co-authored-by: Wilson Seok <wilson.seok@intel.com>
* [GPU] Add oneDNN primitives profiling support
* [GPU] Add stream.wait() method to prevent caches flushing and other possible impacts of finish() call
* Add comment for wait() usage
* Provided visualization of partial values and labels. Adopted DimensionTracker for better equivalence tracking
* Addressed comments and fixed one test
* Removed copy of consts in translator, added test.
* Fixed memory loss for tf.Const.
* Added test, minor corrections.
* Update src/bindings/python/src/openvino/frontend/tensorflow/node_decoder.py
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
* Test corrections.
* Added comment.
---------
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
* Enable `LoadedTensor.*HETERO` test
* Fix use of `ICompiledModel::outputs()`
* Remove extra `loaded_from_cache` argument
* Misprint
* Small refactoring
* Remove extra `model` from `CompiledModelDesc`
Use `get_runtime_model()` instead
* ClangFormat
* [PyOV] Extend Tensor API
* one more ctor
* apply comments
* support constoutput
* add checks for shape
* checks for type and shape
* apply comments
* is_continuos
* codestyle
* Remove unsupported plugin from proxy in order to avoid exception
catching
* Remove from alias for
* Globally remove unavailable devices
* Do not load proxy if no available devices
* Try to fix CI
* Add debug messages
* Avoid two times the same plugin
* Revert "Add debug messages"
This reverts commit 562e36c633.
* Cache hidden devices
* Update if fallback order was changed
* Try to fix CI
* Fixed CoreThreading tests
* Fixed typo
* Try to fix plugin initialization
* Fixed initialization
* Fixed lock
* Do not return default context from the compiled model
* Try to fix segfault
* Added test
* Remove unnecessary code
* Remove code duplication
* Migrate to new API
* Fixed comment
* Revert renamed variable
* Remove cast
* Hetero throws OV Exception
* PostgreSQL interface for reporting from test apps
* Fixes for Linux and MacOS builds
* Fixed segmentation fault and linux build issue
* Removed unnecessary debug messages and std::endl
* Improved logging
* Added loading libpq.so.5 on Linux and fixed typos
* Removed explicit definition for conformance runner
* Added reporting levels with less details, but faster
* Added escaping strings due to unexpected tests naming
* Added env-var defined Run ID (external grouping)
* Added transaction-based queries
* Stored amount of tests to run
* Added handling of serialization errors for transactions
* Cleanup unnecessary code due to latest changes in env
* Added customizable test results
* Enabled conformanceTests.exe for PostgreSQL Reporting
* Changed behavior of Run ID. Changed behavior of Suite Results.
Fixed void in declarations.
* Refactored code regarding control of custom fields in runtime
* Updating targetDevice and case type in Conformance Test
* Introduced wasting test results in case of skipping
Fixed behavior of missing data in case of Fast reporting
Refactored Initialize function by moving loading part separately
* Updated queries naming
* Added runtime resolution of an opset version for operation
* Added capturing a PostgreSQL logs for diagnostic purposes
* Remove unused header
* Fix for fresh master
* Try to detect run on ARM, fix for latest IR hash format
* Updated status values for easier queries
* Added a version info to session information
* Added simple DGPU detection, removed unused prints
* Reduced serialization errors
* Replaced usleep by nanosleep
* Improving robustness
* Fixed behavior of OpImplCheck
* Added IR Weight
* Added a customizable "device architecture" field
* Fix for CI
* Added missing device detection in OpImplCheck
* Introduced Manual Start feature for delayed posting start
information to a tables
* Added app_id to test_results
* Changed architecture detection as requested by owner
* Moved CMake option to src/tests
* Added TODO
* Added full device name as a part of target device
* Added MacOS libpq loading
* Trying alternate way to load on MacOS
* Adding correct executable name detection for MacOS
* Fix -api sync for single -data_shape
Tickets 111187 and 111185
I wasn’t able to find C++ equivalent of Python’s `info.original_shape.is_static`. Later I realized that it shouldn’t be considered because -shape cmd arg should have higher priority for shape inference than model’s shape. So I removed it from Python.
Replace
`if benchmark.inference_only and batch_size.is_dynamic:`
with
`if allow_inference_only_or_sync and batch_size.is_dynamic:`
to reset batch_size to static in case of dynamic shape with single -data_shape
* Check only app_input_info.size() == 1 because if it's gretaer than 1, input shape is dynamic and there are more that one static shapes. Apply TODO
* [TF FE] Support Switch and Merge to fuse into If operation
It introduces support of TF1 control flow with Switch and Merge nodes.
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Add script for test model generation
* Fix code-style
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix build issue
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix build issue with types
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Apply code-review feedback: optimizations in utils
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix build issue
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Apply code-review remarks and cover more cases
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Remove commented code
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Remove unused vars
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Update MO unit-tests wit Switch-Merge case
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix build issue: remove unused variable
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
---------
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Remove vector of so objects from Tensor, Any, VariableState and
RemoteContext
* Change IRemoteContext
* Fixed build without Proxy
* Moved API to ITensor
* Remove friends from Tensor class
* Changed query_state API
* Remote SoPtr for some returned values
* Fixed auto plugin
* Add so to converted objects
* Fixed build all on macOS
* Check that tensor pointers != nullptr
* Add SO to converter
* Added new constructors for SO ptrs
* Changed IVariableState API
* Remove proxy friend
* Remove friends and nullptrs from auto
* Fixed build
* Fixed HETERO plugin
* Fixed code style
* Implementation of MMAP for ONNX FE
* fix win offsets
* added virtual dtor to MappedMemory
* review remarks. part.1
* added disable mmap flag to MO
* added additional checks to mmap
* remove unnecessary const
* fix pybind default value
* Added args.disable_mmap = False to MO tests
* fixed MO test
* avoid global headers
* fix casting for win
* disable mmap for legacy frontends flow
* review remarks
* Fixed passing parameters
* added doc to MappedMemory and load_mmap_object
* Update ONNX FE scatter_elements to use ScatterElementsUpdate-12
* ONNX tests update
* Set reduction update
* Enable xfailed tests
* Add more onnx tests
* Fix reference implementation to support negative indices
* Style alignment
* Trim __FILE__ macro by compiler if supported
* Use file for older version cmake
Debug messages
* Move trim compilation options to os_flags.cmake
* Fix MSVC trimming options
* Add additional trim on runtime
* Update trim function description
* Use ov implementation for ngraph::trim_file_name
* Restore ie_c_api.h
* Remove unused define
* MSVC add trim for CXX language
+ Optimized out Reshape if only batch axis has padding
+ Not to add Reorder before oneDNN if only batch axis has padding
+ Re-calculate output layout's padding if Reshape is optimized opt
+ Not to apply optimization of stable diffusion iGPU perf to dGPU(#18200)
Signed-off-by: Min, Byungil <byungil.min@intel.com>
* [GPU] Use real layout for cpu impls instead of memory's
* [GPU] Add memory tracking and pre allocation mechanism
* Tests and minor code refactoring
* Apply review comments
* Remove unused include
* Made MO cli parser independent from OVC, added OVC Pylint test, minor fixes.
* Small corrections.
* PyLint fixes.
* Added init files.
* PyLint fixes.
* Small correction.
* Removed OVC dependency from MO.
* Fixed MO unit tests.
* PyLint fixes.
* Unit tests fix.
* Returned MO unit tests.
* PyLint configs.
* Small correction.
* Moved offline_transformations to back.
* Moved offline_transformations to back.
* Use GroupNormalization in ONNX FE
* Add and enable decomposition transformation
* Add NodeRegistry
* Adjust eps type
* Use Unsqueeze for bias and scale adjustment
* Add transformation tests
* Add Squeeze for bias and scale to ONNX FE
* Simplify get_range helper
* Add cast double to float eps helper
* Adjust reduction axes
* Style alignment
* Add float eps values in tests
* Enable accuracy comparator in transformation tests
* Skip F16 ref test
* Namespace adjustment
* Const ref for shape vec
* Remove nested scopes from transform tests
* Rename function to model in tests
* Add more const to variables
* Use common input shapes and elem type in transform tests
* Add ticket number to the test
* Update tests headers and names
* Use compatible ror rank check
* Use move to insert splits in new shape
* Add docs to cast_eps helper
* Additional test for eps
* Use versions instead of opset
* Include each op separately
* Added AVX2 optimizations for score preprocessing
* Revert special scaling case for scale factor 1.0f
* changed dimension variables to size_t, fixed warnings
* fixed bug with exporting scores
* Used templates in conversion functions
* Refactor after review
* Linux build fixes
* Review fixes
* Review fixes
* Add two plane YOV to Grey conversion
* Add i420 to grey conversion
* Add yuv to grey conversion for GPU
* Fix cmakes
* Remove static from local function
* Remove opencv dependency from tests
* Put grey_from_yuv_single_plane into namespace
* [CPU] Leftovers for Extension convert I64/U64 to I32.
* Fix as per comments
* Fixes as per commits 2
* Fixes as per commits 3
* Revert "Fixes as per commits 3"
This reverts commit 84022be66f64faccd96a32da5611db0e5137d049.
* Some fixes
* Test fix
* Add sub inplace
* Add scatter implementation
* Remove debug var
* Add tests for empty index
* Add reduce support
---------
Co-authored-by: Michal Lukaszewski <michal.lukaszewski@intel.com>
* rrn seq base common shape infer init
* Update and add shape infer to all seq rnn based ops
* Use shape_infer in lstm seq core op
* Use shape_infer in rnn seq core op
* Unified shape infer for rnn cell base
* Update and add missing cells shape_infer functions
* Use shpe_infer in rnn_cell
* Update shape_infer map
* shape_infer tests
* Move new shape_infer to TA map
* More tests and shape improvements
* Introduce common base tests
* Fix merge conflict
* Missing setters and default ctor tests
* Use RNNCellBase class instead of template type op
* Update v0 LSTMSequence to use RNNCellBase as parent class
* Style
* V0::LSTMSequence update
* Specify opset in shape infer registry
* Move rank check
* Output vec generation refactor
* Update num_dir_validation
* Tests warining fix
* Test types adjustment
* Commented code cleanup
* Move test helpers to test fixture
* Common default ctor tests for Cell
* Update GRU shape infer tests
* Update LSTM shape infer tests
* Update RNN shape infer tests
* File names update
* Functions renaming
* Cast hidden size in test
* Move v0::LSTMSequence conformance test
---------
Co-authored-by: Pawel Raasz <pawel.raasz@intel.com>
* skip validation, always include cmake
* rm unconditional inclusion of zlib
* always include zlib
* correct path for builtin_extensions
* find builtin extensions recursively
* include test_utils always
* add logs for build_samples
* skip tests with dir accessing
* remove platform specification for samples build
* do not pkgconfig on win, use cmake generic on linux for samples
* rm make
* fix num_threads
* use bare numbers
* skip failing
* skip test_lrn_basic
* find zlib
* print error of downloading
* add linux pipeline
* do not save cache from PRs; add skipif only in GHA
* rm caching
* evaluate against a string
* do not include test_utils to the install dir
* [GPU] disable blocked format for dynamic shape model(#18448)
* [GPU] Return default format for output layout rank when user node is reshape in get_preferred_format
- Rollback code to disable blocked formmat for dynamic shape
* [GPU] Add unit test checking has_reshape_user
* [GPU] remove redundant comments
* add support for scalar shapes into cli_parser.py
* add test-case with scalar shapes for convert_model
* reordered inputs in test-case with scalar shapes for convert_model
* minor clarifications
---------
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
* Update DepthToSpace to use ngraph shape infer
* Remove legacy block_size limitation for static shape
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Add TCs for ov_gpu_func_tests and ov_gpu_unit_tests
Signed-off-by: Andrew Park <andrew.park@intel.com>
---------
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Add GroupNorm reference implementation
* Clean up
* Handle GroupNorm in ov namespace only
* Simplify the flow
* Add various scale and bias tests
* Fix batches case
* Add more float tests
* Set output shape
* Set output shape
* Fix missing includes
* Move to ov namespace
---------
Co-authored-by: Michal Lukaszewski <michal.lukaszewski@intel.com>
* Pad ref impl tests for negative pads begin end
* Adjust indentation
* Remove redundant line from the output
* Add tests with removal on middle axis
* Add tests with empty result
---------
Co-authored-by: Michal Lukaszewski <michal.lukaszewski@intel.com>
* Initial commit for proxy plugin
* Include proxy to openvino library
* Fixed some tests
* Added plugin properties
* Added readme
* Support Compile model for the proxy plugin
* Fixed python
* Remove gmock
* Fixed some tests
* Fixed load plugin tests
* Removed internal PROXY_FALLBACK_PRIORITIES initialization property
* Fixed code style
* Added new tests
* Create architecture.md
* Fixed some proxy tests
* Implement not implemented functions
* Fixed ICore mock
* Remove redundant code
* Added import/export tests
* Fixed hetero import/export tests
* Fixed ICore mock
* Fixed import tests
* Fixed build
* Remove redundant parse config
* Fixed some comments
* Try to fix Windows build
* Fixed incorrect logic
* Small fix in tests
* Fixed python API
* Fixed typo
* Try to fix python
* Switch GPU under proxy
* Fixed GPU name
* Revert GPU plugin under proxy
* Small changes in CMake files
* Temp commit
* Build without proxy
* Revert "Temp commit"
This reverts commit 1ac9824bdf.
* Fixed test linking
* Removed tests from ncc check
* Add option to disable proxy plugin
* Fixed minor comments
* Disable some proxy tests if IR frontend is disabled
* Enable Intel GPU under the proxy
* Fixed typo
* Fixed segfault in tests
* Small fix for case if only GPU is registered
* Fixed code style
* Added remote context tests
* Added proxy tests to CI
* Fixed mock engine
* Test change
* Revert "Test change"
This reverts commit 2d1d67766f.
* Add new tests
* Removed some tests
* Revert "Removed some tests"
This reverts commit 090398652f.
* Revert incorrect logic
* Removed unused variables
* Use original name for the GPU plugin
* Fixed CMake
* Do not show hidden devices
* Try to fix GPU remote context
* Fixed GPU plugin build
* Added interface for proxy remote context
* Remove local properties
* Remove redundant API
* Fixed typo
* Added remote tensors
* Fixed code style
* Fixed some comments
* Unwrap remote tensors before conversion to Blob
* Added cast for legacy API
* Fixed some cldnn tests
* Do not add preprocessing for proxy plugin
* Enabled more tests and wrap tensors in infer request
* Use hardware request inside conversion wrapper
* Fixed hand on cache calculation
* Try to fix some tests
* Support set tensor for remote tensors in proxy plugin
* Revert "Support set tensor for remote tensors in proxy plugin"
This reverts commit 5a927de590.
* Remove redundant friend from compiled model and fix life time for infer
request
* Fixed code style
* Add additional so pointers to the tensor
* Rewrite approach for compile model and tensor
* Removed API from proxy
* Fixed is/as Blob for wrapped Blobs
* Wrap tensor when set tensor to plugin
* Fixed recursive call
* Don't unwrap tensors for AUTO plugin
* Fixed Some Multi tests with remote blob for proxy
* Align context name with tests
* Fixed code style
* Try to fix more tests
* Some minor changes
* Try to fix OVHoldersTests
* Try to save low level SO in high level wrappers
* Revert "Try to save low level SO in high level wrappers"
This reverts commit 430ff8a526.
* Revert "Try to fix OVHoldersTests"
This reverts commit 32604f0d3e.
* Disable some tests
* Fixed DynamicBatchShapeTests
* Fixed caching tests and skip OVHoldersTest
* Small refactoring
* Fixed import model
* Small fix
* Fix typo which causes fails of caching tests
* Disabled AUTO BATCH for proxy device
* Support Export in Auto batch
* Small changes
* Fixed initialization fallback to plugin with proxy name
* Added more values for tests
* Ask all devices and create context if no device id
* Support export in auto batch
* Fixed some comments
* Fixed some comments and removed auto batch
* Fixed some comments
* Fixed auto batch test and some comments
* Fixed build
* Removed proxy plugin class from dev api
* Fixed code style
* Fixed disabled tests
* [WA] Comment the onnx test.
Seems this test error would be a little bigger than expected error.
* Remove the softmax CPU function test filter because onednn merge all
ISAs into one primitive.
* Fix zero point issue.
* Remove wino conv test cases because ONEDNN3.2 does not supported.
* Fix cpu debug log verbose
* Update onednn to squash commit version.
* Applied tolerance threashold for onnx_model_bias_gelu test.
* Restore to fix ICX perf regression.
* Clean winograd related code.
* Update ONEDNN to support ARM.
* Applied review comments.
* [GPU] Add roi_align get_shape_infer_dependencies (#18345)
* [GPU] Fix concat cpu impl for buffer fusing case
* [GPU] Add roi_align shape_infer unit tests
* [GPU] Fix windows build issue
* [GPU] add unit test
* Resolve tensor sharing issue when there are same name output port name
In some case, model has 2 or more same name input/output ports, they aslo have the same
precision and partial_shape. Compiled_model will share the same ov::Descriptor::Tensor pointer
and ov::Tensor between multiple such ports.
Considered solving python segment fault issue to create seperated input/output ports, which also
need handle such tensor shared case, this patch will do it.
* Better method to find shrared tensor desc
* Use port's tensor_ptr to replace creating new tensor_ptr
* Add auto pad attribute support for conv
* Fix to let concat onednn impl check can_be_optimized in impl_param instead of that in node
* Apply auto padding to kernel param for conv ocl impl
* conv shape agnostic kernel is not selected if conv is not explicit padding
* Fix failed TCs for ov_gpu_unit_tests
Signed-off-by: Andrew Park <andrew.park@intel.com>
---------
Signed-off-by: Andrew Park <andrew.park@intel.com>
* enabled MVN SLT tests on ARM
* moved nhwc to common section
* exclude normalizeVariance=true from x64
* fixed comments
* add NCHW+acrossChannels = false test case
* Add comment about unsupported case
* fixed 5d layout in Mvn5D_noAcrossChannels
* MvnLayerCPUTest::isSupportedTestCase() added
* init
* moved NHWC to common section
* experiment to support dynamic shapes in reduce transformation
* support dynamic shapes
* axes sorting fix in reduce transformations
* Introduce ov::element visitor
* Example usage visitor in shape infer tools
* Example usage of visitor in unsqueeze evaluate
* Example usage of visitor in Round evaluate
* Example usage of visitor in Unique evaluate
* Refactor element visitor
- Use visitor in tanh and tan operators
* Fix enable_if for MSVC compiler
* Update comments in element visitor
* Move test to the core
* Corrects typos
* Apply code style
* prim::TupleUnpack and prim::ListUnpack removing transformation in PT FE to flatten input list and tuples
* Enabled tuples and lists as items in example_inputs
* Applied code style
* Added tests for tuples as inputs and extended test infrastructure to support it
* Negligible performance optimizations
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
* Fixed duplicated names of test classes
* Added description for tuple flattening transformation
* Removed any support for list flattening on inputs; fixed layer tests
* Fixed style
* Fixed order of new Parameters and Results while flattening tuples
* Fixed style
* Better diagnostics when not all prim::TupleUnpack ops after Parameters are decomposed
* Small fix in diagnostics message
---------
Co-authored-by: Ekaterina Aidova <ekaterina.aidova@intel.com>
Co-authored-by: Andrei Kochin <andrei.kochin@intel.com>
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
Co-authored-by: Alina Kladieva <alina.kladieva@intel.com>
* skip fuse_quantize_f if input or output layout is dynamic
* Update condition of can_fuse_reorder_to_prev for concat in shapeof subgraph
* skip concat_in_place_optimization if concat is shape of subgraph
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Add reorder if eltwise is shapeof subgraph and data type between input and output is different
* Skip reorder optimization if reorder has dynamic shape on remove_redundant_reorders
* Add reproducible TCs for ov_gpu_unit_tests
---------
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Fix python segmentfault issue of plugin api 2.0
* Fix onnx_duplicated_output_name due to empty tensor
Co-authored-by: Bell, Song <bell.song@intel.com>
* Remove redundant code
* Keep rt_info to fix test failure in case of legacy public api
* Not to set_name for new port
---------
Co-authored-by: Bell, Song <bell.song@intel.com>
* ScatterElementsUpdate-12 init
* Update doc structure with ScatterElementsUpdate-12
* Update Operations_specifications.md
* Typos
* Update descriptions
* Add info about indices normalization
* Add info about negative axis normalization
* Add info about boolean type
* Missing coma and typo
* More examples and reduction descriptions
* Rename reduction "copy" to "none"
* Add more examples with values
* Mean mode and types clarification
* fix transformations
* clang fix
* fix unit tests - check the both Pad versions
* add unit tests checking negative padding
---------
Co-authored-by: Ivan Tikhonov <ivan.tikhonov@intel.com>
* enable memory reuse for dynamic models
* updated to return dependant events for the shape_of primitive
* fixed memory_pool.release_memory()
* fixed a lint error
* fixed missing default value
* updated to use reset flag for dynamic models
* changed to use is_dynamic_output_layout instead of is_dynamic
* updated to use get_internal_params instread of buffer_ptr
* added a memory reuse test for dynamic models
* [PyOV] Expose api to Model class
* assign op
* Update src/bindings/python/src/pyopenvino/graph/ops/assign.cpp
* add remove_ methids
* improve test
* codestyle
* assign operation
* codestyle
* test size
---------
Co-authored-by: Michal Lukaszewski <michal.lukaszewski@intel.com>
* Added ovc to PYTHONPATH for MO unit tests.
* Small correction.
* Corrected PYTHONPATH for legacy FE layer tests.
* Corrected PYTHONPATH for TFL FE layer tests.
* Fix the multible multiply case.
* Add test case.
* Fix CI issues.
* Fix the dynamic shape FC kernel creating issue.
* Expose FC weight with original linear layout to CPU graph.
* Apply review comments.
* Apply review comments.
* Applied review comments.
* Used pip wheel to build OpenVINO wheel
* Added convert_model() to openvino.runtime.
* Removed duplication of InputCutInfo, LayoutMap
* Switched Model Conversion API tests to convert_model from openvino.runtime.
* Small correction.
* Format correction.
* Small correction.
* Removed duplication of moc frontend files.
* Small correction.
* Removed duplication of cli_parser, offline_transformations.
* Code corrections.
* Removed code duplications.
* Removed code duplications.
* Updated codeowners.
* Switched layer tests to convert_model().
* Improvements
* Small correction.
* Caffe parser path fix.
* Added python api properly into deb / rpm packages
* Moved implementation to ovc tool.
* Moved implementation to ovc tool.
* Small correction.
* Use cmake -E variant from cmake 3.13
* Namespace fixes.
* Minor fixes.
* Pylint fixes.
* Fixed BOM file.
* Small corrections.
* Minor corrections.
* Minor fix.
* Error fixes.
* Added telemetry requirement.
* Improvements to fix CI
* Some refactoring
* Don't use developer package for scripts projects
* Added exception in case when MO is not imported.
* Removed exception from init.
* Removed changes from cmake.
* Added unit ovc tests, fixed minor errors.
* Added ovc unit tests to azure.
* Corrected imports.
* Fixed path to tests.
* Added missed files.
* Corrected github labels.
* Removed benchmark app from dev package.
* Small fix.
* Small corrections.
* Comment fixed.
* Removed changes from setup.py
* Removed not needed change.
* Removed duplicating unit tests.
* Removed wrong change.
* Removed not needed change.
* Apply suggestions from code review
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
* Added ovc tool test, corrected imports.
* Added legacy TF config test.
* Removed not needed files.
---------
Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
* Migrated the deprecated backward compatibility tests to the new infra
* clang format
* Fixed build of deprecated tests
* Fixed code style
* Reverted snake_case functions names
* Added dependency on data
* Moved exported test models to teh source path
* Fixed old tests
* [GPU] Optimize stable_diffusion performance in iGPU.
Change the existing heuristic shape condition to permute and no transpose gemm in case of transpose gemm.
Signed-off-by: hyunback <hyunback.kim@intel.com>
* add dynamic shape support for dgpu in prepare_buffer_fusing
* add unit test
* add space between test cases
* update condition of impl create() for concat dynamic shape
* update unit test
* add comment and update unit test
* add impl_param.is_type() function
* [GPU] Impl cldnn::condition to support dynamic shape (#18051)
* Impl CreateIfOp
* Update calc_output_layouts and execute_impl
* Enable gpu unit test
* Create gpu functional test
* [GPU] Follow-up code review (#18051)
* remove redundant codes
* create custom execute method for condition_inst
* change name from update_loop_primitive_map to update_inner_program_io_map
* [GPU] Fix gpu func test failures for fp16
* Add more test-cases to support fp16 and nested if case
* [GPU] remove redundant codes
* refactoring var names
* fix windows build error
* [GPU] Fix windows build issue
* [GPU] update calc_output_layouts
* [GPU] remove custom condition_inst::execute
* Remove virtual keyword from primitive_inst::execute()
* [GPU] Share single task executor between main program and inner program
* [GPU] Fix input rank issue for const inner network in condition op
* [GPU] apply calc_output_layouts for roi_align
Co-authored-by: Vladimir Paramuzov <vladimir.paramuzov@intel.com>
* [GPU] avoid checking allow_new_shape_infer for inner program
---------
Co-authored-by: Vladimir Paramuzov <vladimir.paramuzov@intel.com>
* Fix get_partial_shape tensor API to access the correct index of dimensions
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Update the rule specifying output_type to the legacy one by referring to calc_output_layout
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Add reproducible TCs related to issues for ov_gpu_unit_tests
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Fix failed fc dynamic i8 TCs for ov_gpu_unit_tests
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Fix are_data_types_sutable_for_onednn not to invalidate output layout
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Apply comment
Signed-off-by: Andrew Park <andrew.park@intel.com>
---------
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Not to add sync if the node is within shape of subgraph
Because the dependency is cpu impl so the execution is already finished.
* Fixed as review comment : Skip clFinish only when the runtime dep is shape of subgraph, not the current node
* [IE TESTS] Add Global test config for Subgraph base test
* Replace using option by function redefinition
* fix build
* remove extra changes for gna/template
* code style
* add nvidia to devices
* Fix debian
* remove nvidia
* add relative path to the table
* add another column
* new table format
* fix build issue
* fix tab name
* remove old table
* format fixes
* change font
* change path windows
* change tabset name
* add arm and 86_64 tables
* remove list dots
* ResolveNamesCollisions transformation refactoring; enable it in MOC
* fix the description
* call ResolveNamesCollisions transformation in the frontends; resolve review comments
* Resolve review comments
* Rename ngraph with ov in common_test_utils
* Fix comments
* Delete opset includes from commonTestUtils
* Fix CI and code style
* Apply comments
* Fix CI and apply comments
* Used pip wheel to build OpenVINO wheel
* Improvements
* Added python api properly into deb / rpm packages
* Use cmake -E variant from cmake 3.13
* Improvements to fix CI
* Some refactoring
* Don't use developer package for scripts projects
* Final fixes
* Fixed to use input shape rank when calculating output layout, added unit test case
* Fixed to use input shape rank when creating shape_of primitive, added functional tests
* [CompileTool] Throw when a plugin config is missing a space separator between key and value
* Use OPENVINO_ASSERT macro to check for the separator presence
* [GPU] Fix skipped GemmBaseTests in iGPU.
Current GemmBaseTests in iGPU are skipped, just showed pass, but actual not run.
Signed-off-by: hyunback <hyunback.kim@intel.com>
* [PT FE]: use example input for dtype and rank detection, support unordered input dict
* Apply suggestions from code review
* restore old behaviour for old torch versions
* move info addition after parsing
* enable node support on Linux
* fix code style issue
* enable node support on Windows
* fix merge conflict
* update for comments
* update for comments
* enable numa node id and socket id in proc_type_table and udpate inference unit test
* update get_streams_info_table() and smoke test cases in CPU unit test
* fix code style issue
* update numa node id and socket id in proc_type_table on windows
* update for comments
* update for comments
* update for comments
* [MO][TF FE] Remove WA for supporting different TF formats
After providing direct support of all TensorFlow formats, we are ready to remove WA
that converts all unsupported formats to frozen protobuf using native TensorFlow
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Debug failure for LSTM multicell
* Reset default graph before entering session
* Remove temporal debug info
---------
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Added TSForwardBase class and a new transformation to reset no_sinking attribute
* Refactoring
* fix an issue with legacy_output_names
* resolve review comments
* Resolve review comments
* Improve operation naming
* Use set to reduce operations with indexes, fix code style
* Refactor
* Make good names in transformations
* Remove tensor index from input/output names
* Fix tests
* keep Const+DecompressionConvert pattern for CPU
* temporary disabled failing unit-tests
* disable CF by modifying bounds evaluate as well; minor corrections
* added TODOs with ticket numbers
* join const+decompression markings
* minimized convert_precision.cpp changes
* minor corrections
* refactor fp16 transformations: moved into separate fp16_compression folder
* style-fix
* minor fixes
* do not disable evaluate and CF in shape path
* safer disabling of Const conversion
* style-fix and minor corrections
* restore original placement of ConvertPrecision
* fx repr
* remove isNewApi
* add cisLegacyApi
* remove isNewApi
* fix logic
* add test case
* remove comment
* fix bug
* add configuration
* skip in template
* skip failed case
* skip AUTO and MULTI to test
* resolve conflicts
* fix confilcts
* skip AUTO and MULTI to test
* template on
* Revert "template on"
This reverts commit d82a5a4964.
* get log
* add tmp core
* reset to test
* Revert "reset to test"
This reverts commit 7a39d4e24b.
* try to push
* use createCoreWithTemplate()
* remove AUTO and MULTI in skip_config
* Revert "remove AUTO and MULTI in skip_config"
This reverts commit d42420590f.
* fix confilcts
* fix conflicts
* remove AUTO and MULTI in skip config
* fix confilcts
* remove isLegacyAPI
* remove useless code
---------
Co-authored-by: Chen Peter <peter.chen@intel.com>
* [GPU] Unique-10 operation implementation.
* Handled flattened case.
* Created results for all outputs in single layer test.
* Save total unique count as fifth output.
* Handled axis case.
* Added unique reshape kernel.
* Moved data types to unique primitive constructor.
* Added shape agnostic Unique ref kernel.
* Added blocked layout support to Unique-10.
* Use int in bubble sort.
* Added unit tests.
* Added support for blocked layouts to flattened mode.
* Fixed usage of shape_info in kernel.
* Use correct total data size for dynamic shapes.
* Commented some functional tests.
For some reasons big shapes cause std::bad_alloc.
* Initialize out_counts with zeros.
* Implemented new approach for reducing memory footprint.
Changed first kernel to only count unique values and changed second kernel to fill all outputs.
* Revert "Commented some functional tests."
This reverts commit a7f9763c575e71e14b85ee37adf1e98f10785c15.
* Fixed calc output layouts for flattened case when rank in greater than 4.
* Added temporary fix for axis case when rank is greater than 4.
* Revert "Added temporary fix for axis case when rank is greater than 4."
This reverts commit 236640d2f0e9d5b1f8dcbbf9482763badd7fde66.
* Renamed "unique" to "unique_count" and "unique_reshape" to "unique_gather" primitives.
* Quick fix for add_intermediate_node to consider dep_idx of multiple output
* Fix bug for multiple output:
1) get_reorder was getting reorder from cache regardless of the dep_idx.
2) remove_redundant_reorder was not considering original dep_idx
* Fixed conflicts.
* Fixed win build issue.
* Fixed build issue.
* Revert "Fix bug for multiple output:"
This reverts commit d4a2c4f32eabe9108df31d4837fed8995c93bd1c.
* Revert "Quick fix for add_intermediate_node to consider dep_idx of multiple output"
This reverts commit 2dfd2aaefdf32067a7469505b35f7096632ac5f2.
* Added some tests to skip config.
---------
Co-authored-by: Taylor Yeonbok Lee <taylor.lee@intel.com>
* Separate macros for OPENVINO_THROW
add default message to exception to avoid using literals
* Restore the suppress deprecated macro in node
* Restore to public the Exception ctor
for nvidia plugin
* combine test for stream_info_table
* fix failed test cases
* add check for performance hint and core type
* add comments for the parameters of test case
* update numactl support and add test cases
* fix typo
* remove debug info
* update for comments
* update for comments
* move test case into separated file
* update for comments
* update code style
* Support operations on aliases of tensors
* Add tests
* Fix issue with convnd
* Fix code style
* Fix issue with tensor index of mutated tensor
* Fix if types alignment
* Fix issues in keypoint detectron2
* Fix issue with masks in detectron2
* Fix acuracy issue in mobilevitv2 models
* Remove unused includes
* Return upsample case in lictconstruct replacer
* Fix types, apply review feedback
* Apply feedback
* Revert change of not using shared_from_this for getitem
* Fix issue in prim::device transformation
* Fix layer tests
* Apply review feedback
* Fix issue with not existing alias to tensor
* Remove NV12 and I420 blobs and deprecate some legacy API
* Fixed some errors
* Remove NV12 blobs
* Remote NV12 conversion
* Fixed other warnings
* Suppress version
* Fix some warnings
* Fixed version
* Try to fix some warnings
* Suppress warnings in C header
* Suppress warnings in C
* Fixed Windows exceptions
* Try to fix warnings
* Try to fix C bindings build
* Suppress InferRequest
* Fixed some build issues
* Fixed some errors
* Fixed build all for macOS
* Suppress some warnings
* Fixed merge conflict
* Remove NV12 and I420 blobs and deprecate some legacy API
* Fixed some errors
* Remove NV12 blobs
* Remote NV12 conversion
* Fixed other warnings
* Suppress version
* Fix some warnings
* Fixed version
* Try to fix some warnings
* Suppress warnings in C header
* Suppress warnings in C
* Fixed Windows exceptions
* Try to fix warnings
* Try to fix C bindings build
* Suppress InferRequest
* Fixed some build issues
* Fixed some errors
* Custom attribute reading and While operation support
* Rearanges FLATBUFFERS_LOCALE_INDEPENDENT setting
* Style
* Make flatbuffers code as version independent as possible
* Comments addressed
* Add static shape adapter
- Adapters holds CPU dimension which can be reference to it or vector
- Add ov::optional for holding optional result from shape inference
- Add new `infer` function in `IStaticShapeInfer`
* Temporary support of StaticShape
* Fix build issues
* Correct shape adapter compare
- minor static shape adapter refactor
* Minor corrections in ShapeInferenceTA
* Fix subscript operator in StaticShapeRef
* Fuse convert reorder to prev MVN/Concat node
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Add dynamic TCs for ov_gpu_unit_test
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Add descriptions for changes
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Fix kernel selection failure
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Add is_type_conversion_only function for reorder_node
Signed-off-by: Andrew Park <andrew.park@intel.com>
---------
Signed-off-by: Andrew Park <andrew.park@intel.com>
Fixes an issue when AlignEltwiseInputRanks is applied on FakeQuantize with
scalar as a first input and input/output low/high being Shape{1} constants.
In such case FakeQuantize output is still a scalar, so the difference
between output rank and input/output low/high rank is negative.
Ticket: CVS-112454
* generate cpu mapping table pre tbb
* change function name
* fix proc_type_table is wrong in RPL
* add getCpuMapFromCores test, fix comments
* modify test case
* fix comments
* fix code style
* add throw an exception
* fix numa_nodes=0 on ARM
* modify numa_nodes
* fix ExportOptimalNumStreams failed on ARM
* fix comments
* add discription of get_cpu_mapping_from_cores
* update for numactl support
* fix cores is wrong
---------
Co-authored-by: Wanglei Shen <wanglei.shen@intel.com>
* [GPU] Add shape of subgraphs markup and initial cpu implementations for some of primitives
* Apply review comments
* Exclude eltwise with boolean mode types from shape of subgraphs and fix leftovers
* There were two issues in runtime buffer fusing
1) Missing condition in matcher for dyanmic tensor
2) If the node is marked as can_be_optimized = true at build time and then turned out to false at runtime, the kernel compilation has been skipped becuaes it was checking node->can_be_optimized
=> To resolve this issue, added can_be_optimzied to impl_param and let the impl create check can_be_optimized in impl_param instead of that in node.
* Fixed primtiive::can_be_optimize to be set through function
* [GPU] Optimized out permute in permute-gemm(onednn) pattern.
Permute can be optimized out when permute's in and out are compatible and onednn gemm.
Signed-off-by: hyunback <hyunback.kim@intel.com>
* [DOCS] Change downloads directory link (#17846)
* installation link
* fix path
* change notebooks links (#17857)
* fix apt and yum links (#17877)
* [DOCS] Fix list and links to POT (#17887)
* change link to POT
* change header label
* fix typo
For some reason my MSVC gives the following error:
C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.35.32215\include\utility(176,5): error C4996: 'ngraph::SlicePlan::SlicePlan': T
he nGraph API is deprecated and will be removed in the 2024.0 release. For instructions on transitioning to the new API, please refer to https://docs.openv
ino.ai/latest/openvino_2_0_transition_guide.html [C:\Users\vzlobin\r\openvino\build\src\common\transformations\inference_engine_transformations_obj.vcxproj
]
C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.35.32215\include\xmemory(680,47): message : see reference to function 'std::pai
r<std::shared_ptr<ov::op::v1::StridedSlice>,ngraph::SlicePlan>::pair(std::pair<std::shared_ptr<ov::op::v1::StridedSlice>,ngraph::SlicePlan> &&)' [C:\Users\
vzlobin\r\openvino\build\src\common\transformations\inference_engine_transformations_obj.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.35.32215\include\utility(175,5): error C4996: 'ngraph::SlicePlan::SlicePlan': T
he nGraph API is deprecated and will be removed in the 2024.0 release. For instructions on transitioning to the new API, please refer to https://docs.openv
ino.ai/latest/openvino_2_0_transition_guide.html [C:\Users\vzlobin\r\openvino\build\src\common\transformations\inference_engine_transformations_obj.vcxproj
]
C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.35.32215\include\xmemory(680,47): message : see reference to function 'std::pai
r<std::shared_ptr<ov::op::v1::StridedSlice>,ngraph::SlicePlan>::pair(const std::pair<std::shared_ptr<ov::op::v1::StridedSlice>,ngraph::SlicePlan> &)' [C:\U
sers\vzlobin\r\openvino\build\src\common\transformations\inference_engine_transformations_obj.vcxproj]
* add set_value op
* Support for tensor input
* fix shape error
* refactor for dynamic shape
* update process of target_value_shape and add comments
* support arbitrary steps
* fix
* fix ends_node
* fix and add test cases
* fix error when slice operation return maximum number in int32
* remove redundant function call
* update for minus step
* add constraints for minus inputs
---------
Co-authored-by: mei, yang <yang.mei@intel.com>
* enable CPU map for ARM Linux based on freqency information
* fix code style issue
* fix code style issue
* remove 'streams = 1' WA for ARM linux
* update for typo and comments
* update for comments
* keep WA
* keep WA of streams = 1 for ARM Linux
* update num_streams WA for ARM Linux in test case
* update for comments
* update for comments
* update for comments
* update for comments
* update for merge conflict
* update and add test case for MTL
* Initial impl for runtime buffer fusing
Passing unittest with static kernel
* pass unittest with dynamic impl
* Refactor allocate_output
* Separate header of buffer fusing
* Refactored buffer fusing :: matcher/optimize
* More cleanup
* Fix crash in dolly
* Reset can_be_optimized of primitive_inst when it is not
* Fix empty tensor : Primitive with empty data should be skipped
* Fix issue in dynamic padding : Static kernel should not contain dynamic padding dims
Fix missing reset of update_shape_done_by_other flag
* Not to add cache with emtpy kernel for optimized out inst
* Fix corner case error in buffer fusing
- Shapes of some preds may not be changed, but still needed to do update_impl because 1) paddings are changed 2) output memory should be updated
- optimizable impl should not be added to the cache
* Allowing reorder & permute_ref to be optimized concat predecessor
* Some more fixes :
runtime buffer fusing is available only when all preds/concat are dynamic
runtime buffer fusing is to be executed only if the node is dynamic
* Fix allocate_output parameter called by get_estimated_device_mem_usage according to the new change
* Fixed error in cascaded concatt
* Need to reinterprete even though the size is same
* Requirements for the HW plugin to integrate with AUTO
Signed-off-by: Peter Chen <peter.chen@intel.com>
* Update property requirements and wording
1. Added purpose for each reqired property
2. Removed autobatching properties
3. Updated wording
Signed-off-by: Peter Chen <peter.chen@intel.com>
* Add one BA test and update purpose for model_name
Signed-off-by: Peter Chen <peter.chen@intel.com>
* Add request to ov::compilation_num_threads
Signed-off-by: Peter Chen <peter.chen@intel.com>
* Add link to intgration with AUTO
Signed-off-by: Peter Chen <peter.chen@intel.com>
* Wording with API 2.0
Co-authored-by: Ilya Churaev <ilyachur@gmail.com>
* Try to fix the link
* Remove ":doc:"
* Add postfix "__" for external link
* Apply suggestions from code review
Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>
* bash command and multiple devices description update
Signed-off-by: Peter Chen <peter.chen@intel.com>
---------
Signed-off-by: Peter Chen <peter.chen@intel.com>
Co-authored-by: Ilya Churaev <ilyachur@gmail.com>
Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>
* Introduce PadBase
* Update ov scope name
* Introduce Pad-12
* Common type_prop Pad tests
* Init Pad-12 ref tests
* Add Pad reference tests
* attr and op check tests
* Move eval and clone inputs from PadBase
* Init opset12
* Headers clean up
* Update shape_inference map for CPU
* Update Pad evaluates to use ov::TensorVetor
* Update shape infer map with Pads
* Fix namespace
* Update op check test
* Add common Pad shape_inference tests
* Reuse PadBase shape_infer
* Fix C API registers unit test failed
Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>
* Fix compile error in macos
Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>
* Fix CI format issue
Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>
* using the getExecutableDirectory() func from openvino::utils
Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>
---------
Signed-off-by: Zhai, Xuejun <xuejun.zhai@intel.com>
* Deprecate ExecutableNetwork and InferRequest API
* Fixed some warnings
* Fixed some warnings
* Try to fix documentation
* Try to skip documentation warnings
* Mark as deprecated nGraph API
* Fixed code style
* Added IN_OV_LIBRARY define
* Suppress warnings for log
* Suppress warning
* Updated nGraph headers
* Fixed build for macOS
* Fixed lpt and snippets
* Fixed build all on macOS
* Suppress some warnings
* Fixed some new warnings
* Fixed new warnings
* Try to fix some warnings
* More warnings
* Soome change
* Suppress more warnings
* Suppress warnings for transformations
* Suppress warnings for LPT
* One more fix
* Suppress more warnings
* Try to fix opset error
* Remove opset constructor
* Cannot fix opset warning
* Suppress warnings for offline transfromations
* Fixed some warnings for Windows
* Fixed code style
* Suppress some warnings for onnx FE
* Revert "Suppress some warnings for onnx FE"
This reverts commit 75d23b64fc.
* Revert "Fixed code style"
This reverts commit c6eba63116.
* Revert "Fixed some warnings for Windows"
This reverts commit 23d7ed88b6.
* Revert "Suppress warnings for offline transfromations"
This reverts commit 0b9f6317bf.
* Revert "Cannot fix opset warning"
This reverts commit 19ea658639.
* Revert "Remove opset constructor"
This reverts commit 06afb1bc20.
* Revert "Suppress warnings for LPT"
This reverts commit 58b1c0f5a0.
* Revert "Suppress warnings for transformations"
This reverts commit f8bb9814a1.
* Revert "Suppress more warnings"
This reverts commit f9f0da9acb.
* Revert "Soome change"
This reverts commit e545d4984e.
* Remove deprecation for ngraph::OpSet and FactoryRegistry
* remove @param and separate line
* separate line of ov_remote_context_free
* fix code style
* Update src/bindings/c/include/openvino/c/ov_remote_context.h
Co-authored-by: Chen Peter <peter.chen@intel.com>
* add ...
* add ...
---------
Co-authored-by: Chen Peter <peter.chen@intel.com>
* Fix squeeze compatibility
* Fix issue in remainder causing model to fail
* Add tests for slice
* Modify remainder to support int + add type tests
* Apply requested changes
* Add comment
conformance table added
ARM merged with CPU
precision support and layout tables removed from the overview device article (info available in device articles)
* support numactl on Linux in multi-threading 2.0
* update cache file reader
* fix warning
* keep change for numactl support only
* keep change for numactl support only
* keep change for numactl support only
* keep change for numactl support only
* keep change for numactl support only
* fix typo
* update for comments
* fix code style issue
* update is_cpu_map_available()
* update for comments
* update for comments
* update for comments
* Review interpolate shapes and label propagation
* Review shape_infer template implementation
* Update shape infer of interpolate in GPU plugin
- Add new tensor accessor for ov::Tensor map
* Correct casting in dim::scale function
* Remove validation of size of input 1 in v0
* Relax inputs check for interpolate v4
* Correct GPU shape inference
* Use ov::Tensors in interpolate's evaluate
- Remove some duplicated code
- Apply comments from review
* Set shape in interpolate's eval for output tensor
* Tests
* Add eval_lower/upper support to ReduceMax
* Add support for ITensorAccessor in reduce shape infer
* Add tests for duplicated axes and output shapes size
* Push to output_shapes instead final copy to vector
* Remove old shape_infer API
* Move axes rank validation to shape_infer
* Restore shape_infer API for GPU
* Update docs for frontend extensions
* Apply suggestions from code review
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
* fix order in openvino_framework_map
* no discard return value
* add note of openvino_contrib repo
* update example for PT
* note
* add paragraph of named inputs and outputs
Signed-off-by: Mateusz Tabaka <mateusz.tabaka@intel.com>
* title underline too short
* review comments
* remove m_ prefix from CustomOp attr names
---------
Signed-off-by: Mateusz Tabaka <mateusz.tabaka@intel.com>
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
Co-authored-by: Michal Lukaszewski <michal.lukaszewski@intel.com>
* enable cpu map for mac
enable nstreams and nthreads setting for mac
* keep streams=1 for M1
* add explicit type conversion
* remove definition of static cpu
* Update with master
* separate branches for __APPLE__ and __EMSCRIPTEN__
* modify the implementation of is_cpu_map_available function
---------
Co-authored-by: Wanglei Shen <wanglei.shen@intel.com>
* Small fixes for openvino::pugixml creation for Dev packages
* Flexiable components installation
* Fixed compilation for x86
* Added extra checks for ENABLE_NCC_STYLE
* Fixed typo in RPM
* Limitations refactoring
* fix CI builds/tests
* changes after review
* Move GraphCompiler initialization to constructor
* resolve conflicts after rebase
* update after review
* resolve problem with double initialization for Limitations
* Optimize strides calculation using one loop
* Calculate strides on get_strides or set_shape
instead in ctor in TensorView
* Call once update strides on get
* Added dependencies via vcpkg
* Try to remove global imported targets
* Fix for conan
* Fixed RHEL case
* Fixed RHEL, U18 cases
* Returned OpenCV finding
* Update cmake/templates/OpenVINOConfig.cmake.in
Fixed IMPORTED_GLOBAL => GLOBAL in OpenVINOConfig.cmake.in template file
* Apply suggestions from code review
Properply used NAMES in find_package
* Fixed case with zlib
* Final fixes
* Fixes
* Removed CONFIG from find package ZLIB
* Fixed RHEL
* Reverted changes with gflags back
* Turn off LTO after thirdparty dependencies are built
* benchmark_app: except ALLOW_AUTO_BATCHING
Running benchmark_app.py with -b 1 -d CPU fails with
`Unsupported property ALLOW_AUTO_BATCHING by CPU plugin`.
C++ benchmark_app sets ALLOW_AUTO_BATCHING in
ov::Core::compile_model() call, which doesn't trigger the error.
* Move `ALLOW_AUTO_BATCHING` to `device_config`
* Reimplemented models in ngraph, added fp32 tests
* Deleted olf FP32 tests
* Switched test implementation to SubgraphBaseTest
* extended existing subgraph tests by SW_FP32 mode instead of separate FP32 tests
* Extended additional subgraph tests by SW_FP32 mode, needed to modify base test classes
* remove scale factor from fp32 tests
* revert failing test
* Added ticket number for disabled tests
* Review Proposal op for shape infer aspects:
- Check partial shape and label propagation.
- Check template implementation of shape_infer.
- Add/update unit test for static and dynamic shapes.
- Drop using `ngraph` namespace from reviewed operator.
* Add missing includes
* Correct shape infer if batch defined in input only
* Improvement in tensor accessor:
- Make possible to create TA as constexpr
- Make empty TA as cons reference to existing object
* Remove not used tensor accessor
* Move Python tests to the end of CI Pipeline.
First of all we need to run tests on C++ in order to show C++ issues
* Move CPU func tests to the end of pipeline
* Changed Windows
* Deprecate legacy Core and Allocator
* Suppress blob warnings
* Suppress some warnings
* Suppress more warnings
* Suppress blob allocator
* Suppress more warnings
* Suppress more warnings
* Fixed compilation issues for Template plugin
* Fixed some warnings
* Fixed tests
* Add WA for benchmark_app
* Suppress #warning for developer package
* Rename define
* Disable warnings for compile_tool and benchmark_app
* Suppress Windows warnings
* Suppress more warnings for Windows
* Fixed compile_tool install
* Added message for VS
* Fixed snippets and throw only first error
* primitive serialization
* updated primitive::desc() to use impl_param instead of program_node
* added hash caching unit tests
* added missed calls to save and load of parent
* updated copyright year
* [GPU] Added shape agnostic optimized Permute_tile_8x8_4x4 kernel
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Add permute_gpu_tile_8x8_4x4 shape agnostic TCs for ov_gpu_unit_tests
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Fix calculation for required local mem size
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Update not to condisder x and feature dimension for tile size on shape agnostic kernel case
Signed-off-by: Andrew Park <andrew.park@intel.com>
---------
Signed-off-by: Andrew Park <andrew.park@intel.com>
* [TF FE] Avoid usage of NGraph routines
Also, it introduces separate test fixture with graph comparison
for FEs
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Specify tickets for TODOs
---------
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Do not fold StridedSlice when:
- On begin or end there is ShapeOf with disabled constant folding.
- StridedSlice op has disabled constant folding.
* Copy rt info to folded StridedSlice
+ Invalid calculation in reducing un-aligned feature axis for b_fs_yx_fsv16
+ Some reduce modes are not invariant by using 0 value out of range
+ Added jit ZERO_INVARIANT_REDUCTION
+ Enable blocked unit-tests on dGPU by PR#15873
Signed-off-by: Min, Byungil <byungil.min@intel.com>
* enable PaddlePaddle elementwise broadcast
* fix CI fail issue
* Apply suggestions from code review
* fix CI fail issue
* only B to A broadcast is supported for PDPD
* fix GPU plugin testcase fail issue
* keep PDPD broadcast_merge cpu plugin implement align with ov core
* add type prop test case for pdpd broadcast dst shape smaller than src shape
Currently GRUCellFusion treats bias as if it's always in zr order,
which sometimes is not the case. So whenever GRUCellFusion detects rz order,
it should also split bias into r and z and concat the halves into zr.
Ticket: CVS-97025
* Fix issues in aten::chunk and aten::split_with_sizes replacers
* Fix code style
* Fix compute issue
* Fix code style
* Support 1 chunk
* Fix issues in meshgrid, sqrt, rsqrt and exp operations
* Use Interpolate-11
* Fix unary ops
* Add tests for unary ops
* Fix unary ops tests
* Add changes in utils
* Fix mo test
* Remove extra line
* Fix mo tests for int
* support convert_model in paddle runtime
* add convert runtime paddle test
* fix a pylint error
* fix ci error
* skip test_mo_convert_paddle.py # Ticket: 95904
* auto remove tmp file
* add docs for PDFE
* enable paddle mo test in ci
* fix docs
* fix docs
* fix the docs
* Add transformation to convert adaptive pool to reduce
* Update src/common/transformations/src/transformations/common_optimizations/moc_transformations.cpp
* Add tests and apply feedback
* Simplify if branches
* Add to common pipeline
* Remove 3d AdaptivePool with out_shape 1
* Skip test instead of remove
---------
Co-authored-by: Andrei Kochin <andrei.kochin@intel.com>
* Build using conanfile.txt
* Update .ci/azure/linux_arm64.yml
* Several improvements
* Removed conanfile.py
* Try to use activate / deactivate
* Fixed clang-format code style
* Supported TBB version from Conan
* Added more NOMINMAX
* Fixed static build
* More improvements for static build
* Add usage of static snappy in case of static build
* More fixes
* Small fixes
* Final fixes
* update op linspace
* rewrite function name
* adjust op test
* adjust op test
* adjust op test
* format fuzzy
* remove annotations
* add suppot for int64
* remove umap
---------
Co-authored-by: cecilia peng <cecilia.peng@intel.com>
* deserialization of dynamic batch
* updated multi stream tests
* added unit tests
* updated cache dir name
* resolved type conversion warning
* removed teardown()
* added const
* [GPU] Fix with permute mismatching input layout with ouput in batch 2
* Add unit test
* Fix unit test
* Don't use deprecated interface for layer test
* Added torch script backend
* Added ts_backend to pytorch layer tests
* Added use_ts_backend fixture to the test suite to activate the
torchscript backend
* Fixed failing test_dict layer test
* Added USE_TS_BACKEND as an env variable
* Removed use_ts_backend fixture
* Added more tests for ts backend
* Added more information in the comments about usage
* Removed convolution3d test from precommit_ts_backend
* Added some torchscript backend tests to ci
* Removed tests from CI as torch.compile doesn't support 3.11 currently
* Fixed linter issues
* Addressed PR comments and linter issues
* Mark some legacy API as deprecated
* Try to fix some issues
* Fixed some warnings
* Disable deprecation warnings for GNA
* Fixed some warnings
* Disable deprecation errors for all plugins
* Suppress some warnings
* Suppress some warnings
* Suppress deprecated for tests
* Mark all contend as suppressed
* Try to fix extensions
* Suppress more warnings
* Suppress warnings for transformations
* Global suppress of deprecation warnings
* FIxed some warnings
* Fixed comments
* Create macro for deprecation API
* Fixed data tests
* Fixed mock_engine for proxy tests
* Fixed some caching tests
* FIxed build
* Fixed CoreThreading tests
* Try to fix crash in functional tests
* Fixed typo
* Fixed typo
* Small change
* Remove shared pointer from MockPluginWrapper
* Small fixes
* Do not throw an exception from device_supports_cache_dir
* Review detectron prior grid generator for:
- Check interval shapes and label propagation.
- Check template implementation of shape infer.
- Add update or correct unit test for static and dynamic shapes.
* Remove ngraph namespace in reviewed op
* Use detectron validation util to check inputs
in all related detectrons operators
* Relax first dim of feat map and im data check
* Fix test after dimension validation update
* Fix typo in detectron util file name
* Create separated auto_batch plugin testcase
* Add test sample into azure
* Move to auto_plugin directory
* Fix CI build issues
* move batch test cases from gpu/cpu/template plugin to auto batch plugin
* Check OpenCL to decide whether enable auto_batch gpu test cases
* Revert "move batch test cases from gpu/cpu/template plugin to auto batch plugin"
This reverts commit 9f4f2ce1af.
* Add functional tests for auto_batch
* Runtime check gpu available to decide whether run gpu test cases
* Remove HW plugins from functional test
1. Apply Template plugin for auto_batch functional test
2. Remove unnecessary code
* Restore some original tests
* Apply new api property replace old config
* Solve warning suppressions issue
* Fix CI build error
* Solve CI failure issues
* Fix getOutputsFromFunctionWithSeveralOutputs bug
---------
Co-authored-by: Chen Peter <peter.chen@intel.com>
* [TF FE] Report a reason of no conversion of internal operations
Some operations during translations can be temporarily converted to InternalOperation
such as Const operation of string type for which we need to define more elaborated reason
why it is represented as InternalOperation.
Also, restrict instantiation of InternalOperation because instead user should use FrameworkNode.
InternalOperation is a base class for internal operation types of TF FE that have
extended API compare to FrameWorkNode.
For all internal operation we defined a reason why it is not converted to OpenVINO opset
that will be reported in TF FE if they are not gone finally.
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Update src/frontends/tensorflow/tests/convert_unsupported.cpp
* Correct a script for generation of the test model
---------
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* StridedSlice improvements:
-Bound evaluation for begin, end partial values when ignore mask set.
- Custom constant fold implementation.
* Improve const folding when all begin or end values
are ignored
* Use axes param shape to deduce Squeeze output rank
* Improve squeezable dims detection and output shape deduction
* Unskip related tests
* Add more onnx tests
* Skip dynamic model test for legacy api
* Exclude Shape{0} case
* Add more tests
* Adjustment for single element axes
* Move axes param output shape calculation below
* Move back the axes param based calculations
* Adjust onnx python tests
* Consider only single element axes param
* Deduce rank only to enable param value validation
* Prevent negative rank if data input is scalar
* Comment update
* Add missing new line
* Skip dynamic squeeze tests for legacy api
* Add check for zero squeezable dims
* tests update
* Update squeezable dims validation
* Add paddle op mapper silu
* Add more test cases
* Add dynamic shape test for op silu
---------
Co-authored-by: Xiuchuan Zhai <xiuchuan.zhai@intel.com>
* update op linspace
* rewrite function name
* add paddle grid_sample op mapping
* add op_fuzzy
* adjust op logic
* adjust op name
* format code
* add tests
* adjust test
* adjust test
* adjust test
* adjust test
* format code
---------
Co-authored-by: Xiuchuan Zhai <xiuchuan.zhai@intel.com>
* Tensor accessor for shape inference
- as functor for getting data from tensor vector or map.
- as lambda in GPU plugin on tile op
* Make tensor data adapter pure virtual
- function accessor to data returns pointer to interface
* Refactor tensor data accessor and adapter
* Extract memory adapter make it GPU graph internal
- can't be part of GPU runtime memory core dev API not visible there
* Expand IStaticShapeInfer by port map
- update factory map for new infer interface with port map information
- add bit util to generate bit mask use it in PortMask
* Pass tensor accessor as reference not fun object
- Add cldnn data adapter and accessor
- Reduce dynamic allocations in data accessors
* Fix compilation issues
* Use ov::Tensor for data accessor
- remove data adapters are they not required
* Update comments
* Fix build issues
* Fix tile shape infer test
* Add empty null tensor accessor as specialization
* Apply style formatting
* Move data accessor from dev API to shape inference
* Fix linking issues
* Add reorder with usr's output data type for assign
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Fix incorrect input index for handling leftovers
Signed-off-by: Andrew Park <andrew.park@intel.com>
* Add TCs for ov_gpu_unit_tests
Signed-off-by: Andrew Park <andrew.park@intel.com>
---------
Signed-off-by: Andrew Park <andrew.park@intel.com>
* [GPU] Improve dump naming rule in debug feature.
Now, we can support below dump naming rule.
- Exec_graph name
- Wildcard letter for target names ('*', '?')
- Case-insensitive name searching
- Apply to show loop body primitives.
New introduces OV_GPU_xxx
- OV_GPU_ListLayers = 1 (Show layer names and exit)
- OV_GPU_VerboseColor = 1 (Show verbose with color)
Add file, line, function in log prefix.
Signed-off-by: hyunback <hyunback.kim@intel.com>
Because `ov_conditional_compilation_tests` is enabled via
`ENABLE_TESTS`, so with `ENABLE_FUNCTIONAL_TESTS=OFF`
build fails with "fatal error: openvino/itt.hpp:
No such file or directory" due `funcTestUtils` dependency
* Fixed dependencies check, made unsatisfied dependencies show only in case of error.
* Small fix.
* Test correction.
* Small test correction.
* Temporarily added debug print.
* Debug output.
* Debug output.
* Debug output.
* Test fix.
* Removed debug output.
* Small fix.
* Moved tests to check_info_messages_test.py
* Remove dependies checks from MO.
* Small corrections.
Divided MO Extensibility article into separate smaller articles,
Applied the suggestion from [DOCS] Better statement about MO extensions as internal API [Recreating #14062] #15679
Recreated images in svg format
Fixing directives
Recreation of BDTI PRs for master.
Recreated PRs:
Docs: Update Dynamic Shapes documentation #15216
Docs: Edits to Performance Hints and Cumulative Throughput documentation #14793
Docs: Update Devices pages to state improved INT8 performance with 11th & 12th gen devices #12067
* Review ROIAlign for shape inference aspects:
- Check interval shape and label propagation
- Check template implementation of shape_infer
- Add unit test for static and dynamic shapes
* Update ROIAlign::evaluate to use ov::Tensor
- remove ngraph namespace from op where possible
* Review PriorBox op for shape inference aspects:
- check interval shape and label propagation
- check preserve interval values and labels on inputs
- add template shape_infer function
- add/update test for dynamic and static shape inference
* Update PriorBox evaluate to use ov::Tensor
- remove ngraph namespace form PriorBox where possible
* Review PriorBoxClustered for shape infer aspects
- check interval dimension and label propagation
- check preserve partial values and labels on inputs
- add template implementation of shape_infer
- add/update test for static and dynamic shapes
* Use correct shape for PriorBox transformation test
* PriorBoxClustered evaluate use ov::Tensor
Remove ngraph namespace from op where possible
* Fix compilation issues
* [TF FE] Provide single tensor names for inputs and outputs in SavedModel
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix build issue
* Xfail some cases due to internal problems in TF
* Xfail other layer test
* Extend documentation for function to adjust tensor names
* Use old path of tf2 layer testing for legacy frontend
---------
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
This change mimicks LinearToLinearONNXReplacer transformation in
legacy frontend, where linear interpolate mode is replaced with
linear_onnx due to performance reasons.
Ticket: CVS-108343
* Fix bug
1) reshape w/ fused primiitive should not be optimized out
2) Wrong usage of slice mem / concat mem in loop
3) LWS not set in lstm_elt
* Added unittest
* [MO][TF FE] Recover MOC TF FE unit-tests
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix python environment to get test_util
* Correct Python environment for MO UT tests
* Correct Python env for linux-debian
---------
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix tensor names copying in TS transformations
* added a check that sinking is available for all consumers in TS backward transformations
* codestyle
* Apply review comments, add result sorting by tensor names in graph comparator
* delete debug code
* fix RemoveConsumers method implementation
* fix snippet tests
* use reference instead of raw pointer
* add new transformation tests
* fix transformation tests
Since TF 2.10 the native model freezing can produce constants with undefined value,
i.e. tensor shape can be any and value is []. In this case the tensor just fills up with
the default value (0 - for numerics, "" - for strings)
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* fix coverity issue in concat
Signed-off-by: Hu Yuan2 <yuan2.hu@intel.com>
* fix coverity issue in graph_optimizer.cpp
Signed-off-by: Hu Yuan2 <yuan2.hu@intel.com>
* fix review issue
Signed-off-by: Hu Yuan2 <yuan2.hu@intel.com>
* fix the mistaken that there are two throw
Signed-off-by: Hu Yuan2 <yuan2.hu@intel.com>
---------
Signed-off-by: Hu Yuan2 <yuan2.hu@intel.com>
* Do not fuse Pad with negative pad values
* Update src/common/transformations/tests/common_optimizations/pad_fusion.cpp
* Update src/common/transformations/tests/common_optimizations/pad_fusion.cpp
* [CPU] Add CPU plugin specific properties to supported_properties list
* Add tests and move cpu specific tests from shared_tests_instances
---------
Co-authored-by: Anton Voronov <anton.voronov@intel.com>
* DOCS remove deprecated options
* removed a couple more not actual questions
* remove the whole lines completely
* remove a couple of more deprecations
* opset to rst
* change list indentations
* fix formula
* add n operations
* add negative and nonzero
* fix link
* specs to rst
* fix matrixnms path
* change path to if
* fix list
* fix format
* fix threading test sporadic failure
* fix read wrong data in muti threading
* fix read and write sync
* add lock before cpu._cpu_mapping_table[i][CPU_MAP_USED_FLAG],because CPU_MAP_USED_FLAG may be modified by set_cpu_used
OV_PYTHON_VERSION:3.11.2# Full version of Python its required for LD_LIBRARY_PATH. More details https://github.com/microsoft/azure-pipelines-tool-lib/blob/master/docs/overview.md#tool-cache
OV_PYTHON_VERSION_MAJOR_MINOR:3.11
OV_PYTHON_VERSION:$(OV_PYTHON_VERSION_MAJOR_MINOR).2# Full version of Python its required for LD_LIBRARY_PATH. More details https://github.com/microsoft/azure-pipelines-tool-lib/blob/master/docs/overview.md#tool-cache
We welcome community contributions to OpenVINO™. Please read the following guide to learn how to find ideas for contribution, practices for good pull requests, checking your changes with our tests and more.
We welcome community contributions to OpenVINO™. Please read the following guide to learn how to find ideas for contribution, follow best practices for pull requests, and test your changes with our established checks.
## Before you start contributing you should
- Make sure you agree to contribute your code under [OpenVINO™ (Apache 2.0)](https://github.com/openvinotoolkit/openvino/blob/master/LICENSE) license.
-Figure out what you’re going to contribute. If you don’t know what you are going to work on, navigate to the [Github "Issues" tab](https://github.com/openvinotoolkit/openvino/issues). Make sure that there isn't someone working on it. In the latter case you might provide support or suggestion in the issue or in the linked pull request.
- If you are going to fix a bug, check that it's still exists in the latest release. This can be done by building the latest master branch, and make sure that the error is still reproducible there. We do not fix bugs that only affect older non-LTS releases like 2020.2 for example (more details about [branching strategy](https://github.com/openvinotoolkit/openvino/wiki/Branches)).
- Make sure you agree to contribute your code under [OpenVINO™ (Apache 2.0) license](https://github.com/openvinotoolkit/openvino/blob/master/LICENSE).
-Decide what you’re going to contribute. If you are not sure what you want to work on, check out [Contributions Welcome](https://github.com/openvinotoolkit/openvino/issues/17502). See if there isn't anyone already working on the subject you choose, in which case you may still contribute, providing support and suggestions for the given issue or pull request.
- If you are going to fix a bug, check if it still exists. You can do it by building the latest master branch and making sure that the error is still reproducible there. We do not fix bugs that only affect older non-LTS releases like 2020.2, for example (see more details about our [branching strategy](https://github.com/openvinotoolkit/openvino/wiki/Branches)).
## "Fork & Pull Request model" for code contribution
### [](https://github.com/openvinotoolkit/openvino/blob/master/CONTRIBUTING.md#the-instruction-in-brief)The instruction in brief
- Register at GitHub. Create your fork of OpenVINO™ repository [https://github.com/openvinotoolkit/openvino](https://github.com/openvinotoolkit/openvino) (see [https://help.github.com/articles/fork-a-repo](https://help.github.com/articles/fork-a-repo) for details).
- Register at GitHub. Create your fork of the OpenVINO™ repository [https://github.com/openvinotoolkit/openvino](https://github.com/openvinotoolkit/openvino) (see [https://help.github.com/articles/fork-a-repo](https://help.github.com/articles/fork-a-repo) for details).
- Install Git.
- Set your user name and email address in a Git configuration according to GitHub account (see [https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup](https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup) for details).
- Choose a task for yourself. It could be a bugfix or some new code.
- Set your user name and email address in Git configuration according to the GitHub account (see [First-Time-Git-Setup](https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup) for details).
- Choose a task for yourself. It may be a bugfix or an entirely new piece of code.
- Choose a base branch for your work. More details about branches and policies are here: [Branches](https://github.com/openvinotoolkit/openvino/wiki/Branches)
- Clone your fork to your computer.
- Create a new branch (with a meaningful name) from the base branch you chose.
- Modify / add the code following our [Coding Style Guide](./docs/dev/coding_style.md).
- If you want to add a new sample, please look at this [Guide for contributing to C++/C/Python IE samples](https://github.com/openvinotoolkit/openvino/wiki/SampleContribute)
- Create a new branch (give it a meaningful name) from the base branch of your choice.
- Modify / add the code, following our [Coding Style Guide](./docs/dev/coding_style.md).
- If you want to add a new sample, please have a look at the [Guide for contributing to C++/C/Python IE samples](https://github.com/openvinotoolkit/openvino/wiki/SampleContribute)
- If you want to contribute to the documentation and want to add a new guide, follow that instruction [Documentation guidelines](https://github.com/openvinotoolkit/openvino/wiki/CodingStyleGuideLinesDocumentation)
- Run testsuite locally:
- execute each test binary from the artifacts directory, e.g. `<source dir>/bin/intel64/Release/ieFuncTests`
- When you are done, make sure that your branch is to date with latest state of the branch you want to contribute to (e.g. `git fetch upstream && git merge upstream/master`), push your branch to your GitHub fork; then create a pull request from your branch to the base branch (see [https://help.github.com/articles/using-pull-requests](https://help.github.com/articles/using-pull-requests) for details).
- When you are done, make sure that your branch is up to date with latest state of the branch you want to contribute to (e.g. `git fetch upstream && git merge upstream/master`). If so, push your branch to your GitHub fork and create a pull request from your branch to the base branch (see [using-pull-requests](https://help.github.com/articles/using-pull-requests) for details).
## Making a good pull request
@@ -34,22 +34,20 @@ Following these guidelines will increase the likelihood of your pull request bei
- One PR – one issue.
- Build perfectly on your local system.
- Choose the right base branch [Branches](https://github.com/openvinotoolkit/openvino/wiki/Branches).
- Choose the right base branch, based on our [Branch Guidelines](https://github.com/openvinotoolkit/openvino/wiki/Branches).
- Follow the [Coding Style Guide](./docs/dev/coding_style.md) for your code.
-Update documentation using [Documentation guidelines](https://github.com/openvinotoolkit/openvino/wiki/CodingStyleGuideLinesDocumentation) if needed.
-Document your contribution, if you decide it may benefit OpenVINO users. You may do it yourself by editing the files in the "docs" directory or contact someone working with documentation to provide them with the right information.
- Cover your changes with test.
- Add license at the top of new files [C++ example](https://github.com/openvinotoolkit/openvino/blob/master/samples/cpp/classification_sample_async/main.cpp#L1-L2), [Python example](https://github.com/openvinotoolkit/openvino/blob/master/samples/python/hello_classification/hello_classification.py#L3-L4).
- Add enough information: a meaningful title, the reason why you made the commit and a link to the issue page if exists.
- Remove unrelated to PR changes.
- If it is still WIP and you want to check CI test results early then use _Draft_ PR.
- Add the license statement at the top of new files [C++ example](https://github.com/openvinotoolkit/openvino/blob/master/samples/cpp/classification_sample_async/main.cpp#L1-L2), [Python example](https://github.com/openvinotoolkit/openvino/blob/master/samples/python/hello_classification/hello_classification.py#L3-L4).
- Add proper information to the PR: a meaningful title, the reason why you made the commit, and a link to the issue page, if it exists.
- Remove changes unrelated to the PR.
- If it is still WIP and you want to check CI test results early, use a _Draft_ PR.
- Submit your PR and become an OpenVINO™ contributor!
## Testing and merging pull requests
Your pull request will be automatically tested by OpenVINO™'s precommit (testing status are automatically reported as "green" or "red" circles in precommit steps on PR's page). If any builders have failed, you need fix the issue. To rerun the automatic builds just push changes to your branch on GitHub. No need to close pull request and open a new one!
Your pull request will be automatically tested by OpenVINO™'s precommit (testing statuses are automatically reported as "green" or "red" circles in precommit steps on the PR page). If any builders fail, you need to fix the issues before the PR can be merged. If you push any changes to your branch on GitHub the tests will re-run automatically. No need to close pull request and open a new one!
## Merging PR
When the reviewer accepts the pull request and the pre-commit shows a "green" status, the review status is set to "Approved", which signals to the OpenVINO™ maintainers that they can merge your pull request.
When an assigned reviewer accepts the pull request and the pre-commit is "green", the review status is set to "Approved", which informs OpenVINO™ maintainers that they can merge your pull request.
<td>Auto batch plugin performs on-the-fly automatic batching (i.e. grouping inference requests together) to improve device utilization, with no programming effort from the user</td>
@@ -168,7 +166,9 @@ See [How to build OpenVINO](./docs/dev/build.md) to get more information about t
## How to contribute
See [CONTRIBUTING](./CONTRIBUTING.md) for details. Thank you!
See [Contributions Welcome](https://github.com/openvinotoolkit/openvino/issues/17502) for good first issues.
See [CONTRIBUTING](./CONTRIBUTING.md) for contribution details. Thank you!
## Get a support
@@ -189,7 +189,6 @@ Report questions, issues and suggestions, using:
* [Neural Network Compression Framework (NNCF)](https://github.com/openvinotoolkit/nncf) - a suite of advanced algorithms for model inference optimization including quantization, filter pruning, binarization and sparsity
* [OpenVINO™ Training Extensions (OTE)](https://github.com/openvinotoolkit/training_extensions) - convenient environment to train Deep Learning models and convert them using OpenVINO for optimized inference.
* [OpenVINO™ Model Server (OVMS)](https://github.com/openvinotoolkit/model_server) - a scalable, high-performance solution for serving deep learning models optimized for Intel architectures
* [DL Workbench](https://docs.openvino.ai/nightly/workbench_docs_Workbench_DG_Introduction.html) - an alternative, web-based version of OpenVINO designed to facilitate optimization and compression of pre-trained deep learning models.
* [Computer Vision Annotation Tool (CVAT)](https://github.com/opencv/cvat) - an online, interactive video and image annotation tool for computer vision purposes.
* [Dataset Management Framework (Datumaro)](https://github.com/openvinotoolkit/datumaro) - a framework and CLI tool to build, transform, and analyze datasets.
@@ -197,7 +196,7 @@ Report questions, issues and suggestions, using:
\* Other names and brands may be claimed as the property of others.
[Open Model Zoo]:https://github.com/openvinotoolkit/open_model_zoo
# Running and Deploying Inference {#openvino_docs_deployment_guide_introduction}
@sphinxdirective
.. toctree::
:maxdepth: 1
:hidden:
Run and Deploy Locally <openvino_deployment_guide>
Deploy via Model Serving <ovms_what_is_openvino_model_server>
Once you have a model that meets both OpenVINO™ and your requirements, you can choose how to deploy it with your application.
.. panels::
:doc:`Deploy via OpenVINO Runtime <openvino_deployment_guide>`
^^^^^^^^^^^^^^
Local deployment uses OpenVINO Runtime that is called from, and linked to, the application directly.
It utilizes resources available to the system and provides the quickest way of launching inference.
---
:doc:`Deploy via Model Server <ovms_what_is_openvino_model_server>`
^^^^^^^^^^^^^^
Deployment via OpenVINO Model Server allows the application to connect to the inference server set up remotely.
This way inference can use external resources instead of those available to the application itself.
Apart from the default deployment options, you may also :doc:`deploy your application for the TensorFlow framework with OpenVINO Integration <ovtf_integration>`
The remaining modes assume certain levels of automation in selecting devices for inference. Using them in the deployed solution may potentially increase its performance and portability. The automated modes are:
# Model Preparation {#openvino_docs_model_processing_introduction}
@sphinxdirective
.. meta::
:description: Preparing models for OpenVINO Runtime. Learn how to convert and compile models from different frameworks or read them directly.
.. toctree::
:maxdepth: 1
:hidden:
@@ -10,22 +15,48 @@
omz_tools_downloader
Every deep learning workflow begins with obtaining a model. You can choose to prepare a custom one, use a ready-made solution and adjust it to your needs, or even download and run a pre-trained network from an online database, such as OpenVINO's :doc:`Open Model Zoo <model_zoo>`.
Every deep learning workflow begins with obtaining a model. You can choose to prepare a custom one, use a ready-made solution and adjust it to your needs, or even download and run a pre-trained network from an online database, such as `TensorFlow Hub <https://tfhub.dev/>`__, `Hugging Face <https://huggingface.co/>`__, `Torchvision models <https://pytorch.org/hub/>`__.
:doc:`OpenVINO™ supports several model formats <Supported_Model_Formats>` and allows to convert them to it's own, OpenVINO IR, providing a tool dedicated to this task.
:doc:`OpenVINO™ supports several model formats <Supported_Model_Formats>` and allows converting them to it's own, `openvino.runtime.Model <api/ie_python_api/_autosummary/openvino.runtime.Model.html>`__ (`ov.Model <api/ie_python_api/_autosummary/openvino.runtime.Model.html>`__ ), providing a tool dedicated to this task.
:doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>` reads the original model and creates the OpenVINO IR model (.xml and .bin files) so that inference can ultimately be performed without delays due to format conversion. Optionally, Model Optimizer can adjust the model to be more suitable for inference, for example, by :doc:`alternating input shapes <openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model>`, :doc:`embedding preprocessing <openvino_docs_MO_DG_Additional_Optimization_Use_Cases>` and :doc:`cutting training parts off <openvino_docs_MO_DG_prepare_model_convert_model_Cutting_Model>`.
There are several options to convert a model from original framework to OpenVINO model format (``ov.Model``).
The approach to fully convert a model is considered the default choice, as it allows the full extent of OpenVINO features. The OpenVINO IR model format is used by other conversion and preparation tools, such as the Post-Training Optimization Tool, for further optimization of the converted model.
The ``read_model()`` method reads a model from a file and produces ``ov.Model``. If the file is in one of the supported original framework file formats, it is converted automatically to OpenVINO Intermediate Representation. If the file is already in the OpenVINO IR format, it is read "as-is", without any conversion involved. ``ov.Model`` can be serialized to IR using the ``ov.serialize()`` method. The serialized IR can be further optimized using :doc:`Neural Network Compression Framework (NNCF) <ptq_introduction>` that applies post-training quantization methods.
Conversion is not required for ONNX, PaddlePaddle, TensorFlow Lite and TensorFlow models (check :doc:`TensorFlow Frontend Capabilities and Limitations <openvino_docs_MO_DG_TensorFlow_Frontend>`), as OpenVINO provides C++ and Python APIs for importing them to OpenVINO Runtime directly. It provides a convenient way to quickly switch from framework-based code to OpenVINO-based code in your inference application.
Convert a model in Python
######################################
Model conversion API, specifically, the ``mo.convert_model()`` method converts a model from original framework to ``ov.Model``. ``mo.convert_model()`` returns ``ov.Model`` object in memory so the ``read_model()`` method is not required. The resulting ``ov.Model`` can be inferred in the same training environment (python script or Jupiter Notebook). ``mo.convert_model()`` provides a convenient way to quickly switch from framework-based code to OpenVINO-based code in your inference application. In addition to model files, ``mo.convert_model()`` can take OpenVINO extension objects constructed directly in Python for easier conversion of operations that are not supported in OpenVINO. The ``mo.convert_model()`` method also has a set of parameters to :doc:`cut the model <openvino_docs_MO_DG_prepare_model_convert_model_Cutting_Model>`, :doc:`set input shapes or layout <openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model>`, :doc:`add preprocessing <openvino_docs_MO_DG_Additional_Optimization_Use_Cases>`, etc.
Another option to convert a model is to use ``mo`` command-line tool. ``mo`` is a cross-platform tool that facilitates the transition between training and deployment environments, performs static model analysis, and adjusts deep learning models for optimal execution on end-point target devices in the same measure, as the ``mo.convert_model`` method.
``mo`` requires the use of a pre-trained deep learning model in one of the supported formats: TensorFlow, TensorFlow Lite, PaddlePaddle, or ONNX. ``mo`` converts the model to the OpenVINO Intermediate Representation format (IR), which needs to be read with the ``ov.read_model()`` method. Then, you can compile and infer the ``ov.Model`` later with :doc:`OpenVINO™ Runtime <openvino_docs_OV_UG_OV_Runtime_User_Guide>`.
The figure below illustrates the typical workflow for deploying a trained deep learning model:
* ``.bin`` - Contains the weights and biases binary data.
Model files (not Python objects) from ONNX, PaddlePaddle, TensorFlow and TensorFlow Lite (check :doc:`TensorFlow Frontend Capabilities and Limitations <openvino_docs_MO_DG_TensorFlow_Frontend>`) do not require a separate step for model conversion, that is ``mo.convert_model``. OpenVINO provides C++ and Python APIs for importing the models to OpenVINO Runtime directly by just calling the ``read_model`` method.
The results of both ``mo`` and ``mo.convert_model()`` conversion methods described above are the same. You can choose one of them, depending on what is most convenient for you. Keep in mind that there should not be any differences in the results of model conversion if the same set of parameters is used.
This section describes how to obtain and prepare your model for work with OpenVINO to get the best inference results:
* :doc:`See the supported formats and how to use them in your project <Supported_Model_Formats>`.
* :doc:`Convert different model formats to the OpenVINO IR format <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`.
*`Automate model-related tasks with Model Downloader and additional OMZ Tools <https://docs.openvino.ai/latest/omz_tools_downloader.html>`__.
To begin with, you may want to :doc:`browse a database of models for use in your projects <model_zoo>`.
* :doc:`Convert different model formats to the ov.Model format <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`.
OpenVINO™ is not just one tool. It is an expansive ecosystem of utilities, providing a comprehensive workflow for deep learning solution development. Learn more about each of them to reach the full potential of OpenVINO™ Toolkit.
Neural Network Compression Framework (NNCF)
###########################################
**Neural Network Compression Framework (NNCF)**
A suite of advanced algorithms for Neural Network inference optimization with minimal accuracy drop. NNCF applies quantization, filter pruning, binarization and sparsity algorithms to PyTorch and TensorFlow models during training.
@@ -27,8 +30,7 @@ More resources:
*`PyPI <https://pypi.org/project/nncf/>`__
OpenVINO™ Training Extensions
#############################
**OpenVINO™ Training Extensions**
A convenient environment to train Deep Learning models and convert them using the OpenVINO™ toolkit for optimized inference.
A solution empowering TensorFlow developers with OpenVINO's optimization capabilities. With just two lines of code in your application, you can offload inference to OpenVINO, while keeping the TensorFlow API.
A streaming media analytics framework, based on the GStreamer multimedia framework, for creating complex media analytics pipelines.
More resources:
*`Documentation on GitHub <https://dlstreamer.github.io/index.html>`__
*`Installation Guide on GitHub <https://github.com/openvinotoolkit/dlstreamer_gst/wiki/Install-Guide>`__
DL Workbench
############
A web-based tool for deploying deep learning models. Built on the core of OpenVINO and equipped with a graphics user interface, DL Workbench is a great way to explore the possibilities of the OpenVINO workflow, import, analyze, optimize, and build your pre-trained models. You can do all that by visiting `Intel® Developer Cloud <https://software.intel.com/content/www/us/en/develop/tools/devcloud.html>`__ and launching DL Workbench online.
To learn which device supports the import / export functionality, see the :doc:`feature support matrix <openvino_docs_OV_UG_Working_with_devices>`.
For more details on preprocessing steps, refer to the :doc:`Optimize Preprocessing <openvino_docs_OV_UG_Preprocessing_Overview>`. To compile the model with advanced preprocessing capabilities, refer to the :doc:`Use Case - Integrate and Save Preprocessing Steps Into OpenVINO IR <openvino_docs_OV_UG_Preprocess_Usecase_save>`, which shows how to have all the preprocessing in the compiled blob.
**DL Workbench**
A web-based tool for deploying deep learning models. Built on the core of OpenVINO and equipped with a graphics user interface, DL Workbench is a great way to explore the possibilities of the OpenVINO workflow, import, analyze, optimize, and build your pre-trained models. You can do all that by visiting `Intel® Developer Cloud <https://software.intel.com/content/www/us/en/develop/tools/devcloud.html>`__ and launching DL Workbench online.
**OpenVINO™ integration with TensorFlow (OVTF)**
OpenVINO™ Integration with TensorFlow will no longer be supported as of OpenVINO release 2023.0.As part of the 2023.0 release, OpenVINO will feature a significantly enhanced TensorFlow user experience within native OpenVINO without needing offline model conversions. :doc:`Learn more <openvino_docs_MO_DG_TensorFlow_Frontend>`.
# OpenVINO™ integration with TensorFlow {#ovtf_integration}
@sphinxdirective
**OpenVINO™ integration with TensorFlow** is a solution for TensorFlow developers who want to get started with OpenVINO™ in their inferencing applications. By adding just two lines of code you can now take advantage of OpenVINO™ toolkit optimizations with TensorFlow inference applications across a range of Intel® computation devices.
This is all you need:
.. code-block:: bash
import openvino_tensorflow
openvino_tensorflow.set_backend('<backend_name>')
**OpenVINO™ integration with TensorFlow** accelerates inference across many AI models on a variety of Intel® technologies, such as:
* Intel® CPUs
* Intel® integrated GPUs
.. note::
For maximum performance, efficiency, tooling customization, and hardware control, we recommend developers to adopt native OpenVINO™ solutions.
To find out more about the product itself, as well as learn how to use it in your project, check its dedicated `GitHub repository <https://github.com/openvinotoolkit/openvino_tensorflow/tree/master/docs>`__.
To see what you can do with **OpenVINO™ integration with TensorFlow**, explore the demos located in the `examples folder <https://github.com/openvinotoolkit/openvino_tensorflow/tree/master/examples>`__ in our GitHub repository.
Sample tutorials are also hosted on `Intel® DevCloud <https://www.intel.com/content/www/us/en/developer/tools/devcloud/edge/build/ovtfoverview.html>`__. The demo applications are implemented using Jupyter Notebooks. You can interactively execute them on Intel® DevCloud nodes, compare the results of **OpenVINO™ integration with TensorFlow**, native TensorFlow, and OpenVINO™.
License
#######
**OpenVINO™ integration with TensorFlow** is licensed under `Apache License Version 2.0 <https://github.com/openvinotoolkit/openvino_tensorflow/blob/master/LICENSE>`__.
By contributing to the project, you agree to the license and copyright terms therein
and release your contribution under these terms.
Support
#######
Submit your questions, feature requests and bug reports via `GitHub issues <https://github.com/openvinotoolkit/openvino_tensorflow/issues>`__.
How to Contribute
#################
We welcome community contributions to **OpenVINO™ integration with TensorFlow**. If you have an idea for improvement:
* Share your proposal via `GitHub issues <https://github.com/openvinotoolkit/openvino_tensorflow/issues>`__.
* Submit a `pull request <https://github.com/openvinotoolkit/openvino_tensorflow/pulls>`__.
We will review your contribution as soon as possible. If any additional fixes or modifications are necessary, we will guide you and provide feedback. Before you make your contribution, make sure you can build **OpenVINO™ integration with TensorFlow** and run all the examples with your fix/patch. If you want to introduce a large feature, create test cases for your feature. Upon our verification of your pull request, we will merge it to the repository provided that the pull request has met the above mentioned requirements and proved acceptable.
\* Other names and brands may be claimed as the property of others.
# OpenVINO™ Training Extensions {#ote_documentation}
@sphinxdirective
@sphinxdirective
.. meta::
:description: OpenVINO™ Training Extensions include advanced algorithms used
to create, train and convert deep learning models with OpenVINO
Toolkit for optimized inference.
OpenVINO™ Training Extensions provide a suite of advanced algorithms to train
Deep Learning models and convert them using the `OpenVINO™
@@ -19,21 +25,22 @@ Detailed Workflow
.. note::
Prepare a separate dataset or split the dataset you have for more accurate quality evaluation.
3. Having successful evaluation results received, you have an opportunity to deploy your model or continue optimizing it, using NNCF and POT. For more information about these frameworks, go to :doc:`Optimization Guide <openvino_docs_model_optimization_guide>`.
3. Having successful evaluation results received, you have an opportunity to deploy your model or continue optimizing it, using NNCF. For more information about these frameworks, go to :doc:`Optimization Guide <openvino_docs_model_optimization_guide>`.
If the results are unsatisfactory, add datasets and perform the same steps, starting with dataset annotation.
OpenVINO Training Extensions Components
#######################################
-`OpenVINO Training Extensions SDK <https://github.com/openvinotoolkit/training_extensions/tree/master/ote_sdk>`__
-`OpenVINO Training Extensions CLI <https://github.com/openvinotoolkit/training_extensions/tree/master/ote_cli>`__
-`OpenVINO Training Extensions Algorithms <https://github.com/openvinotoolkit/training_extensions/tree/master/external>`__
*`OpenVINO Training Extensions API <https://github.com/openvinotoolkit/training_extensions/tree/develop/otx/api>`__
*`OpenVINO Training Extensions CLI <https://github.com/openvinotoolkit/training_extensions/tree/develop/otx/cli>`__
*`OpenVINO Training Extensions Algorithms <https://github.com/openvinotoolkit/training_extensions/tree/develop/otx/algorithms>`__
| With Model Downloader and Model Optimizer guides, you will learn to download pre-trained models and convert them for use with OpenVINO™. You can use your own models or choose some from a broad selection provided in the Open Model Zoo.
| With model conversion API guide, you will learn to convert pre-trained models for use with OpenVINO™. You can use your own models or choose some from a broad selection in online databases, such as `TensorFlow Hub <https://tfhub.dev/>`__, `Hugging Face <https://huggingface.co/>`__, `Torchvision models <https://pytorch.org/hub/>`__..
| :doc:`Model Optimization and Compression <openvino_docs_model_optimization_guide>`
| In this section you will find out how to optimize a model to achieve better inference performance. It describes multiple optimization methods for both the training and post-training stages.
| This section explains the process of deploying your own inference application using either OpenVINO Runtime or OpenVINO Model Server. It describes how to run inference which is the most basic form of deployment and the quickest way of launching inference.
:description: Learn the details of custom kernel support for the GPU device to
enable operations not supported by OpenVINO.
To enable operations not supported by OpenVINO™ out of the box, you may need an extension for OpenVINO operation set, and a custom kernel for the device you will target. This article describes custom kernel support for the GPU device.
The GPU codepath abstracts many details about OpenCL. You need to provide the kernel code in OpenCL C and an XML configuration file that connects the kernel and its parameters to the parameters of the operation.
@@ -13,18 +18,20 @@ There are two options for using the custom operation configuration file:
All OpenVINO samples, except the trivial ``hello_classification``, and most Open Model Zoo demos
feature a dedicated command-line option ``-c`` to load custom kernels. For example, to load custom operations for the classification sample, run the command below:
@@ -235,7 +242,8 @@ Example Configuration File
The following code sample provides an example configuration file in XML
format. For information on the configuration file structure, see the `Configuration File Format <#config-file-format>`__.
The Intel® Distribution of OpenVINO™ toolkit supports neural network models trained with various frameworks, including
TensorFlow, PyTorch, ONNX, TensorFlow Lite, PaddlePaddle, Apache MXNet, Caffe, and Kaldi. The list of supported operations is different for
each of the supported frameworks. To see the operations supported by your framework, refer to :doc:`Supported Framework Operations <openvino_docs_MO_DG_prepare_model_Supported_Frameworks_Layers>`.
The Intel® Distribution of OpenVINO™ toolkit supports neural-network models trained with various frameworks, including
TensorFlow, PyTorch, ONNX, TensorFlow Lite, and PaddlePaddle (OpenVINO support for Apache MXNet, Caffe, and Kaldi is currently
being deprecated and will be removed entirely in the future). The list of supported operations is different for each of the supported frameworks.
To see the operations supported by your framework, refer to :doc:`Supported Framework Operations <openvino_resources_supported_operations_frontend>`.
Custom operations, which are not included in the list, are not recognized by OpenVINO out-of-the-box. The need for custom operation may appear in two cases:
1. A new or rarely used regular framework operation is not supported in OpenVINO yet.
2. A new user operation that was created for some specific model topology by the author of the model using framework extension capabilities.
Importing models with such operations requires additional steps. This guide illustrates the workflow for running inference on models featuring custom operations. This allows plugging in your own implementation for them. OpenVINO Extensibility API enables adding support for those custom operations and using one implementation for Model Optimizer and OpenVINO Runtime.
@@ -54,9 +64,9 @@ Mapping of custom operation is implemented differently, depending on model forma
1. If a model is represented in the ONNX (including models exported from Pytorch in ONNX), TensorFlow Lite, PaddlePaddle or TensorFlow formats, then one of the classes from :doc:`Frontend Extension API <openvino_docs_Extensibility_UG_Frontend_Extensions>` should be used. It consists of several classes available in C++ which can be used with the ``--extensions`` option in Model Optimizer or when a model is imported directly to OpenVINO runtime using the ``read_model`` method. Python API is also available for runtime model import.
2. If a model is represented in the Caffe, Kaldi or MXNet formats, then :doc:`Model Optimizer Extensions <openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Customize_Model_Optimizer>` should be used. This approach is available for model conversion in Model Optimizer only.
2. If a model is represented in the Caffe, Kaldi or MXNet formats (as legacy frontends), then :doc:`[Legacy] Model Optimizer Extensions <openvino_docs_MO_DG_prepare_model_customize_model_optimizer_Customize_Model_Optimizer>` should be used. This approach is available for model conversion in Model Optimizer only.
Existing of two approaches simultaneously is explained by two different types of frontends used for model conversion in OpenVINO: new frontends (ONNX, PaddlePaddle, TensorFlow Lite and TensorFlow) and legacy frontends (Caffe, Kaldi and Apache MXNet). Model Optimizer can use both front-ends in contrast to the direct import of model with ``read_model`` method which can use new frontends only. Follow one of the appropriate guides referenced above to implement mappings depending on framework frontend.
Existing of two approaches simultaneously is explained by two different types of frontends used for model conversion in OpenVINO: new frontends (ONNX, PaddlePaddle, TensorFlow Lite, and TensorFlow) and legacy frontends (Caffe, Kaldi, and Apache MXNet). Model Optimizer can use both frontends in contrast to the direct import of model with ``read_model`` method which can use new frontends only. Follow one of the appropriate guides referenced above to implement mappings depending on framework frontend.
If you are implementing extensions for new ONNX, PaddlePaddle, TensorFlow Lite or TensorFlow frontends and plan to use the ``--extensions`` option in Model Optimizer for model conversion, then the extensions should be:
@@ -85,6 +95,13 @@ Extensions can be loaded from a code with the ``:ref:`ov::Core::add_extension <
The ``Identity`` is a custom operation class defined in :doc:`Custom Operation Guide <openvino_docs_Extensibility_UG_add_openvino_ops>`. This is sufficient to enable reading OpenVINO IR which uses the ``Identity`` extension operation emitted by Model Optimizer. In order to load original model directly to the runtime, add a mapping extension:
When Python API is used, there is no way to implement a custom OpenVINO operation. Even if custom OpenVINO operation is implemented in C++ and loaded into the runtime by a shared library, there is still no way to add a frontend mapping extension that refers to this custom operation. In this case, use C++ shared library approach to implement both operations semantics and framework mapping.
Python can still be used to map and decompose operations when only operations from the standard OpenVINO operation set are used.
.. _create_a_library_with_extensions:
Create a Library with Extensions
++++++++++++++++++++++++++++++++
@@ -165,13 +177,6 @@ This CMake script finds OpenVINO, using the ``find_package`` CMake command.
:description: Explore OpenVINO™ Extension API which enables registering
custom operations to support models with operations
not supported by OpenVINO.
OpenVINO™ Extension API allows you to register custom operations to support models with operations which OpenVINO™ does not support out-of-the-box. This capability requires writing code in C++, so if you are using Python to develop your application you need to build a separate shared library implemented in C++ first and load it in Python using ``add_extension`` API. Please refer to :ref:`Create library with extensions <create_library_with_extensions>` for more details on library creation and usage. The remining part of this document describes how to implement an operation class.
:description: Learn how to use frontend extension classes to facilitate the mapping
of custom operations from the framework model representation to the OpenVINO
representation.
The goal of this chapter is to explain how to use Frontend extension classes to facilitate
mapping of custom operations from framework model representation to OpenVINO representation.
Refer to :doc:`Introduction to OpenVINO Extension <openvino_docs_Extensibility_UG_Intro>` to
@@ -19,6 +25,11 @@ guide.
operation that is a placeholder for your real custom operation. You can review the complete code,
which is fully compilable, to see how it works.
.. note::
You can find more examples of extensions in `openvino_contrib repository <https://github.com/openvinotoolkit/openvino_contrib/tree/master/modules/custom_operations>`_.
Single Operation Mapping with OpExtension
#########################################
@@ -83,6 +94,13 @@ In this case, you can directly say that 'MyRelu' -> ``Relu`` mapping should be u
In the resulting converted OpenVINO model, “MyRelu” operation will be replaced by the standard operation
``Relu`` from the latest available OpenVINO operation set. Notice that when standard operation is used,
@@ -108,10 +119,18 @@ as it was demonstrated with ``TemplateExtension::Identity``.
Attribute Mapping
++++++++++++++++++
As described above, ``OpExtension`` is useful when attributes can be mapped one by one or initialized by a constant.
As described above, ``OpExtension`` is useful when attributes can be mapped one by one or initialized by a constant.
Attributes in OpenVINO operators are identified by their names, so for frameworks that also have named attributes (like TensorFlow, PaddlePaddle, ONNX),
you can specify name to name mapping. For frameworks where OpenVINO operator's attributes can be mapped to one of the framework
operator inputs (like PyTorch), there's a name to input index mapping.
Named attributes mapping
^^^^^^^^^^^^^^^^^^^^^^^^
If the set of attributes in framework representation and OpenVINO representation completely match by their names and types,
nothing should be specified in OpExtension constructor parameters. The attributes are discovered and mapped
automatically based on ``visit_attributes`` method that should be defined for any OpenVINO operation.
no attribute mapping has to be specified in OpExtension constructor parameters. The attributes are discovered and mapped automatically
based on ``visit_attributes`` method that should be defined for any OpenVINO operation.
Imagine you have CustomOperation class implementation that has two attributes with names: ``attr1`` and ``attr2``.
@@ -119,14 +138,15 @@ Imagine you have CustomOperation class implementation that has two attributes wi
:language: cpp
:fragment: [frontend_extension_CustomOperation]
And the original model in the framework representation also has operation named “CustomOperation” with the same
And original model in framework representation also has operation with name ``CustomOperation`` with the same
``attr1`` and ``attr2`` attributes. Then with the following code:
This will map ``alpha`` to the second input and map ``beta`` attribute to constant value ``1.0f``.
Such created extension can be used, e.g. in dynamic library, please refer to :ref:`Create a library with extensions <create_a_library_with_extensions>`.
Mapping custom operations to frontends with OPENVINO_FRAMEWORK_MAP macro
Below solution works only for ONNX and Tensorflow frontends.
``OPENVINO_FRAMEWORK_MAP`` is a macro that should be used inside OpenVINO operation's class definition and that lets you specify
the mapping between this operation to a frontend operation.
``OPENVINO_FRAMEWORK_MAP`` is a macro that should be used inside OpenVINO operation's class definition
and that lets you specify the mapping between this operation to a frontend operation.
Let's consider the following example. Imagine you have an ONNX model with ``CustomOp`` operation (and this operation has ``mode`` attribute),
a TensorFlow model with ``CustomOpV3`` operation (this operation has ``axis`` attribute) and a PaddlePaddle model with ``CustomOp`` (with ``mode`` attribute)
that has input named "X" and output named "Out" and all of them can be implemented with a single OpenVINO operation ``CustomOp`` like follows:
Let's consider the following example. Imagine you have an ONNX model with ``CustomOp`` operation
(and this operation has ``mode`` attribute) and a Tensorflow model with ``CustomOpV3`` operation
(this operation has ``axis`` attribute) and both of them can be implemented with a single OpenVINO
To access original framework operation attribute value and connect to inputs, ``node`` object of type ``NodeContext`` is used. It has three main methods:
* ``NodeContext::get_input`` to get input with a given index,
* ``NodeContext::get_attribute`` to get attribute value with a given name.
* ``NodeContext::get_attribute`` to get attribute value with a given name,
* ``NodeContext::get_values_from_const_input`` to get an attribute with a given input index.
The conversion function should return a vector of node outputs that are mapped to
corresponding outputs of the original framework operation in the same order.
Some frameworks require output names of the operation to be provided during conversion.
For PaddlePaddle operations, it is generally necessary to provide names for all outputs using the ``NamedOutputs`` container.
Usually those names can be found in source code of the individual operation in PaddlePaddle code.
The following example shows such conversion for the ``top_k_v2`` operation.
:description: Get to know how Graph Rewrite handles running multiple matcher passes on
ov::Model in a single graph traversal.
``:ref:`ov::pass::GraphRewrite <doxid-classov_1_1pass_1_1_graph_rewrite>``` serves for running multiple matcher passes on ``:ref:`ov::Model <doxid-classov_1_1_model>``` in a single graph traversal.
:description: Learn how to use Model Pass transformation class to take entire
ov::Model as input and process it.
``:ref:`ov::pass::ModelPass <doxid-classov_1_1pass_1_1_model_pass>``` is used for transformations that take entire ``:ref:`ov::Model <doxid-classov_1_1_model>``` as an input and process it.
:description: Use the base ov::IAsyncInferRequest class to implement a custom asynchronous inference request in OpenVINO.
Asynchronous Inference Request runs an inference pipeline asynchronously in one or several task executors depending on a device pipeline structure.
OpenVINO Runtime Plugin API provides the base ov::IAsyncInferRequest class:
@@ -33,7 +36,7 @@ AsyncInferRequest()
The main goal of the ``AsyncInferRequest`` constructor is to define a device pipeline ``m_pipeline``. The example below demonstrates ``m_pipeline`` creation with the following stages:
* ``infer_preprocess_and_start_pipeline`` is a CPU ligthweight task to submit tasks to a remote device.
* ``infer_preprocess_and_start_pipeline`` is a CPU lightweight task to submit tasks to a remote device.
* ``wait_pipeline`` is a CPU non-compute task that waits for a response from a remote device.
:description: Learn how to build a plugin using CMake and OpenVINO Developer Package.
OpenVINO build infrastructure provides the OpenVINO Developer Package for plugin development.
OpenVINO Developer Package
@@ -9,7 +13,7 @@ OpenVINO Developer Package
To automatically generate the OpenVINO Developer Package, run the ``cmake`` tool during a OpenVINO build:
.. code-block:: bash
.. code-block:: sh
$ mkdir openvino-release-build
$ cd openvino-release-build
@@ -32,9 +36,9 @@ Once the commands above are executed, the OpenVINO Developer Package is generate
* Libraries for tests development:
* ``openvino::gtest``, ``openvino::gtest_main``, ``openvino::gmock`` - Google Tests framework libraries
* ``openvino::commonTestUtils`` - static library with common tests utilities
* ``openvino::funcTestUtils`` - static library with functional tests utilities
* ``openvino::unitTestUtils`` - static library with unit tests utilities
* ``openvino::common_test_utils`` - static library with common tests utilities
* ``openvino::func_test_utils`` - static library with functional tests utilities
* ``openvino::unit_test_utils`` - static library with unit tests utilities
* ``openvino::ngraphFunctions`` - static library with the set of ``ov::Model`` builders
* ``openvino::funcSharedTests`` - static library with common functional tests
* ``openvino::ngraph_reference`` - static library with operation reference implementations.
@@ -48,7 +52,7 @@ Build Plugin using OpenVINO Developer Package
To build a plugin source tree using the OpenVINO Developer Package, run the commands below:
.. code-block:: bash
.. code-block:: sh
$ mkdir template-plugin-release-build
$ cd template-plugin-release-build
@@ -72,7 +76,7 @@ To build a plugin and its tests, run the following CMake scripts:
The default values of the ``ENABLE_TESTS``, ``ENABLE_FUNCTIONAL_TESTS`` options are shared via the OpenVINO Developer Package and they are the same as for the main OpenVINO build tree. You can override them during plugin build using the command below:
:description: Use the openvino::funcSharedTests library, which includes
a predefined set of functional tests and utilities to verify a plugin.
OpenVINO tests infrastructure provides a predefined set of functional tests and utilities. They are used to verify a plugin using the OpenVINO public API.
All the tests are written in the `Google Test C++ framework <https://github.com/google/googletest>`__.
:description: Learn about the support for quantized models with different
precisions and the FakeQuantize operation used to express
quantization rules.
One of the feature of OpenVINO is the support of quantized models with different precisions: INT8, INT4, etc.
However, it is up to the plugin to define what exact precisions are supported by the particular HW.
All quantized models which can be expressed in IR have a unified representation by means of *FakeQuantize* operation.
@@ -53,8 +58,8 @@ Thus we can define:
Quantization specifics and restrictions
#######################################
In general, OpenVINO can represent and execute quantized models from different sources. However, the Post-training Optimization Tool (POT)
is considered the default way to get optimized models. Since the POT supports HW-aware quantization it means that specific rules can be implemented in it for
In general, OpenVINO can represent and execute quantized models from different sources. However, the Neural Network Compression Framework (NNCF)
is considered the default way to get optimized models. Since the NNCF supports HW-aware quantization it means that specific rules can be implemented in it for
the particular HW. However, it is reasonable to have compatibility with general-purpose HW such as CPU and GPU and support their quantization schemes.
:description: Learn about AvgPoolPrecisionPreserved attribute used only during AvgPool operation.
:ref:`ngraph::AvgPoolPrecisionPreservedAttribute <doxid-classngraph_1_1_avg_pool_precision_preserved_attribute>` class represents the ``AvgPoolPrecisionPreserved`` attribute.
Utility attribute, which is used only during ``AvgPool`` operation, precision preserved property definition.
:description: Learn about IntervalsAlignment attribute, which describes a subgraph with the same quantization intervals alignment.
:ref:`ngraph::IntervalsAlignmentAttribute <doxid-classngraph_1_1_intervals_alignment_attribute>` class represents the ``IntervalsAlignment`` attribute.
The attribute defines a subgraph with the same quantization intervals alignment. ``FakeQuantize`` operations are included. The attribute is used by quantization operations.
:description: Learn about PrecisionPreserved attribute, which describes a precision preserved operation.
:ref:`ngraph::PrecisionPreservedAttribute <doxid-classngraph_1_1_precision_preserved_attribute>` class represents the ``PrecisionPreserved`` attribute.
The attribute defines a precision preserved operation. If the attribute is absent, then an operation is not precision preserved.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.