* Added experimental ScaledDotProductAttention operation in opset12. Supported in PT FE for aten::scaled_dot_product_attention translation. Decomposed in the common optimizations as functional reference.
* Better ScaledDotProductAttention
- Moved decomposition to the decomposing transformation
- Implemented more ctors for the op
- Renamed is_causal to causal
- Shape/type inference native code instead of using decomposition
- Moved the op from opset12 to opset13
- Added Python wrapper for ScaledDotProductAttention
* Fix test that counts ops in the opsets
* Update src/core/src/op/scaled_dot_product_attention.cpp
Co-authored-by: Katarzyna Mitrus <katarzyna.mitrus@intel.com>
* Update src/core/src/op/scaled_dot_product_attention.cpp
Co-authored-by: Katarzyna Mitrus <katarzyna.mitrus@intel.com>
* Move ScaledDotProductAttentionDecomposition from fusions to decompositions.
* Remove not used legacy shape inference in ScaledDotProductAttention
* Better namespace usage
* Register all nodes in ScaledDotProductDecomposition for correct tracking of nodes and running next mather passes on all new nodes.
* Don't use register_new_node_
* ScaledDotProductAttention specification (with an extra scale argument)
* Code style fix
* Scale input implementation for ScaledDotProductAttention
* Handle attention_mask=0 case in the op spec
* Better description of scale input
* N->M in scale description
* Code style fix, remove debug print.
* Apply suggestions from code review
Co-authored-by: Katarzyna Mitrus <katarzyna.mitrus@intel.com>
Co-authored-by: Mateusz Mikolajczyk <mateusz.mikolajczyk@intel.com>
* Fix for case when is_causal is not passed
* Extended description of ScaledDotProduct op
* Better description in py op wrapper
* Basic shape propagation tests for ScaledDotProductAttention
* Added ScaledDotProductAttention to toc.
* Add op impl check
---------
Co-authored-by: Katarzyna Mitrus <katarzyna.mitrus@intel.com>
Co-authored-by: Mateusz Mikolajczyk <mateusz.mikolajczyk@intel.com>
* [CPU] CPU plugin migrates to plugin API 2.0
* Fix legacy config/metric issue
* Fix some issue of ov_cpu_func_tests
1. set_tensors_impl segment fault
2. ov::loaded_from_cache unsupported issue
* Resolve some comments
1. ov::loaded_from_cache issue
2. throw_if_cancelled issue
3. import_model issue
4. set_tensor_impl issue
5. batched_inference issue
* Fix dynamic shape inference issue
* Fix build error
* keep original model info in infer_request
* Fix minor error
* cache internal tensors for input/output precision change
* Disable import model test cases with precision changes
* fix precision issue
* Fix issue for import model
* Fix InferRequestCancellationTests exception issue
* Skip InferRequestIOBBlobTest.*secondCallGetInputDoNotReAllocateData due to new plugin api have different behavior
* Fix graph name issue
* Fix ROI issues
* Fix Transpose shape issue
* Skip vie::Version test due to change to ov::Version
* Solve input port name changes issue
* Solve preprocess layout issue
* Fix minor issue
* tidy up code
* Fix conflict after rebase
* Fix Windows build warning
* Add aux tensors for precision change issue
* Fix import/export model issue
* WA single layer name changed by preprocess
* Revert "WA single layer name changed by preprocess"
This reverts commit bc8fcdd43c.
* Skip some legacy tests due to plugin api 2.0 is enabled
1. skip some python legacy tests for plugin api 2.0 some different behaviors
2. skip some smoke tests due to output port name was changed
* Fix 2 build warnings
* Skip some AUTO plugin tests
* Fix property issue caused by AUTO plugin
* Skip PSROIPooling issues
* Follow header files reference policy
* Split out transformation fixing for nop_elimination
* Fix AUTO plugin mismatch issue for get_tensor function
* Fix aux tensor shape issue
* Fix tensor shape issue
* WA python sync inference sample's segmentfault issue
* Fix reshape issue for dynamic inference
* Fixed incorrect tensor name in e2e test
Fixe issue: e2e ONNX_Customized_Cascade_Rcnn_api_2_True_batch_1_device_CPU_precision_FP325den8cnk
* Fix python segmentfault issue of plugin api 2.0
* Fix python segmentfault issue of plugin api 2.0
* Revert "Fix python segmentfault issue of plugin api 2.0"
This reverts commit 6f502e5d86.
* Fix onnx_duplicated_output_name due to empty tensor
Co-authored-by: Bell, Song <bell.song@intel.com>
* Remove redundant code
* Remove python segment fault WA
* Keep rt_info to fix test failure in case of legacy public api
* Fix output port names missing issue
* Adress some reviewers' comments
* Restore OnnxBackendNodeModelTest::test_maxpool_with_argmax_2d_precomputed_pads_cpu after fixing has been merged
* Resolve tensor sharing issue when there are same name output port name
In some case, model has 2 or more same name input/output ports, they aslo have the same
precision and partial_shape. Compiled_model will share the same ov::Descriptor::Tensor pointer
and ov::Tensor between multiple such ports.
Considered solving python segment fault issue to create seperated input/output ports, which also
need handle such tensor shared case, this patch will do it.
* Resolve tensor sharing issue when there are same name output port name
In some case, model has 2 or more same name input/output ports, they aslo have the same
precision and partial_shape. Compiled_model will share the same ov::Descriptor::Tensor pointer
and ov::Tensor between multiple such ports.
Considered solving python segment fault issue to create seperated input/output ports, which also
need handle such tensor shared case, this patch will do it.
* Better method to find shrared tensor desc
* rename with snake_case style
* Remove ngraph header files
* Keep external_ptr naming
* Add OPENVINO_SUPPRESS_DEPRECATED for some legacy code
* Use port's tensor_ptr to replace creating new tensor_ptr
* Resolve some reviewer comments
* Implement ov::IInferRequestInternalWrapper::GetPreProcess to recover python GetPrepProcess tests
* Remove unnecessary header files reference
* Assert the risk of precision change and reorder at the same time
* Modify legacy python test to fit plugin api 2.0 behavior
* Recover smoke_Transpose(2|4|5|6)D/TransposeLayerTest.CompareWithRefs due to fixing is merged
* Fix typo issue
* Address reviewer's comments
* Disable precision coversion
* Fix error when CpuBlockedMemoryDesc
* Remove precision mismatch WA
* WA precision issue for query_model
* Solve precision mismatch between compiled model and graph
* Fixe failure of query_model
* Rebase to new plugin api update
* Recover the test cases of precision mismatch
* Try to fix name changing for graph model
* Remove tets code
* Remove fp64
* Rebase to new plugin api update
* Update for some failure cases
* Fix bert_benchmark failure issue
* Avoid segment fault in arm acl
Legacy public api + cpu plugin api will add convert op by preprocess by default for unsupported precision,
but ACLConvertExecutor cannot support dimension > 6, so this test will be segment fault due to dimension > 6
smoke_TestNumpyBroadcastNgraphEvaluate/BroadcastLayerTest.CompareWithRefs/targetShape=(1.2.3.4.5.6.7.8.9.10)_axesMapping=()_mode=numpy_inShape=(1.2.1.4.1.6.1.8.1.10)_inNPrec=I8_trgDev=CPU
smoke_TestNumpyBroadcastNgraphEvaluate/BroadcastLayerTest.CompareWithRefs/targetShape=(1.2.3.4.5.6.7.8.9.10)_axesMapping=()_mode=numpy_inShape=(1.2.1.4.1.6.1.8.1.10)_inNPrec=U8_trgDev=CPU
* Remove precision change from preprocess to avoid ACL unsupport convert dim > 6
* ACLConvertExecutor cannot support dimension > 6, don't let preprocess to add Convert
* Revert "ACLConvertExecutor cannot support dimension > 6, don't let preprocess to add Convert"
This reverts commit fd7a8b35af.
* Revert "Remove precision change from preprocess to avoid ACL unsupport convert dim > 6"
This reverts commit 3c2d9a5f17.
* Debug
* Debug incorrect precision checking issue
* Debug Eltwise FP64 unsupported issue
* Add logs for precision
* debug log
* Update for new dependent PRs merged
* Fix failure caused by preprocess
Fix below failures due to cannot find ops by name
smoke_LPT/ReduceMaxTransformation.CompareWithRefImpl/f32_[1,3,10,10]_CPU_f32__256*
* Fix build error
* Fix failure caused by missing code during rebase
* Add debug
* Fix precision unsupport issue
* U16/I16/U64 precision support
* Resolve the issue of f64 reorder
Fix below issue:
Cannot create reorder primitive: unsupported reorder case
* Fix convert multiple child edge issue
* Solve ROI tensor failure issues
* Temporarily disable num_nodes comparison
* Only change convert precision for fp64
* Put convert precision change before reorder to avoid confusion
* Add debug log for transformation
* Fix rebase confilict
* Fix clang issue
* Temporarily disable test_infer_mixed_values python test of bf16
* Solve issue of smoke_ConvertCPULayerTest_BOOL_Dynamic_inputPRC=BF16 choose FP32 primType rather than BP16 primType
* Fix issue of pytorch_tests/test_outer.py
There are 2 output ports, but with the same port name, they should share the same tensor.
* Fix arm cannot find Eltwise executor issue
smoke_SetBlobCPU/SetBlobTest.CompareWithRefs/Type=INPUT_Device=CPU_PrecisionInNet=FP16_PrecisionInNgraph=BOOL
will report below error:
[ GENERAL_ERROR ] Supported Eltwise executor is not found
It need change convert precision to avoid such problem.
* Fix memory overwritten issue
* Temporarily skip arm fp16 SetBlobTest
* Fix compile error after rebase
* Restore smoke_IsOp test due to fixing pr merged
* Fix float to bf16 issue in avx2 isa
* solve onnx test xfail issue
* Skip test cases that ARM Eltwise executor FP16 is not supported
smoke_SetBlobCPU/SetBlobTest.CompareWithRefs/Type=INPUT_Device=CPU_PrecisionInNet=FP16_PrecisionInNgraph=BOOL
smoke_SetBlobCPU/SetBlobTest.CompareWithRefs/Type=BOTH_Device=CPU_PrecisionInNet=FP16_PrecisionInNgraph=BOOL
[ GENERAL_ERROR ] Supported Eltwise executor is not found
* [CPU] improve reorder to support any precision
* Implement ReorderExecutor
* Fix builld error
* Not cache executor due to its primitive has been cached
* Keep convert one time at most
At most insert one convert if needed, if still cannot do reorder it will throw exception rather than insert the second convert
For example, below reorder will not be supported:
FP64<->I64/U64/U32
U32<->I64/U64
U32<->I16/U16
FP64<->FP64
BIN<->BIN
* Only do conversion if layout is same
* update for only convert case
* Update for reviewer comments
* update for failure cases
* Address reviewer comments
* Update rebase issue
* minor update
* Solve unsupported precision issue in tranfromation rather than init_edge
* Remove unnecessary convert in init_edge
* Minor changes
* Update Reorder::reorderData
* Solve issue if only coversion without reorder
* Address reviewer comments
* Address reviewer comments
* Keep exception for unsuported precision
* update
* Revert reorder executor implement
* Solve float->bool issue on transformation pipeline
* Solve I64 is not supported issues
* Solve reviewer's comments
* Fixed dynamic top_k node issue
* Skip nhwc and nChw16c test cases for ConvertLayer
* Update for reviewers' comments
* Fix some failures
* Update for several failure cases
* Update for apiConformanceTests failures
* Fix incorrect node name after import model
* update
* update comments
* Solve issue of smoke_MatMul_NoTranspose and smoke_MatMul_BothTranspose
* Fixed AlignMatMulInputRanks scalar issue
* Address reviewers' comments, remove redundant path in graph.cpp
* Remove test_div_uint8_cpu from xfail_issue_58676
* Solve invalid number of nodes for smoke_Snippets_BroadcastSelect
* ConstantResultSubgraphTest of u16/i16/u32/i64/u64
* restore smoke_SetBlobCPU BOOL tests for arm
* [CPU] Fix ARM precision issue
ARM64 ACL prefers fp16 than fp32, API 2.0 requires input/output precision not changes,
then fp32 input will trigger convert node is added to convert fp32 to fp16.
* Solve some ARM64 failures
* Fix arm64 InferRequestVariableStateTest tests out of memory issue
ARM64 will force fp16 precision, which cause states memory can be fp16, so memcpy to state_memory
cannot use float * element_size, else it will be out of memory bound.
* Skip 2 arm64 tests caused by forcing fp16 precision
* Revert "Fix arm64 InferRequestVariableStateTest tests out of memory issue"
This reverts commit 3e12bd48c2.
* Fix python test_get_profiling_info failure issue
---------
Co-authored-by: Bell, Song <bell.song@intel.com>
Co-authored-by: Chen Peter <peter.chen@intel.com>
* Add support for ONNX GroupNormalization
* Enable onnx compliance tests
* Add f to the test values to avoid cast warning
* Use tolerance as fp instead of bits
* add m1 mac pipelines as a matrix parameter
* Update mac.yml
disable java_api because of macos arm64 - Java is not available on macOS arm64 runners
* Update mac.yml
added always condition for all tests
* Update mac.yml
* Update mac.yml
* Update mac.yml
* Update setup.py
temp commit
* Update tools/openvino_dev/setup.py
* use matrix for var
* add mxnet to extras only for x86_64
* skip failing tests
* use xfail for Python tests; add missing filter for transformations tests
* skip CPU func tests on x86_64 mac; skip some tests from CPU func tests on arm mac
* Update mac.yml
* skip tests on mac arm
* skip tests on darwin; apply review
* add more skips for python and c++ tests
* skip tf tests
* skip more tf tests; skip more Python UT stages
* rm alwayses, rm triggers, add nightly trigger
---------
Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>
* TorchFX: Constant value pass optimization
* Replace op.Constant with make_constant in fx_decoder
* Using shared memory for constant value passing
Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>
---------
Co-authored-by: Jan Iwaszkiewicz <jan.iwaszkiewicz@intel.com>
* Reference implementation for u4 constant compression from pytorch model based on bitwise ops pattern
* Fixed order of 4-bit halfs in byte
* Switched PyTorch FE to dev mode: in case if model cannot be fully converted, give partially converted model with PTFrameworkNode's with a printed warning (normally would raise an exception in case).
* Moved u4 compression to utils_quantize. Implemented not-interleaved version of u4 compression
* Removed debug output
* Added aten::matmul to the list of exceptions in may_produce_alias as a workaround for gptq models
* Added patching for gptq models applied automatically in convert_model
* WA for an inssue with u4 with earlier convert to fp16
* U4 blocked repacking for gptq patched model layout
* Deleted obsolete u4 re-packing based on aten::cat. Fixed the resulting u4 constant shape. Removed debug output.
* Revert "Switched PyTorch FE to dev mode: in case if model cannot be fully converted, give partially converted model with PTFrameworkNode's with a printed warning (normally would raise an exception in case)."
This reverts commit 0ef1455e70.
* Update src/frontends/pytorch/src/op/cat.cpp
* Check mask and shift values in u4 pattern. deque -> OutputVector for u4_compression_stack
* Convert to a given floating type instead of half in gptq patching. Better structured code.
* Code style fix
* Removed deque include
* Code style fixes
* Trailing space removed
* Fixed patched_forward and ts_decoder after unvalidated commits.
* Swap nibbles in u4/i4
* Better exception handling around jit.trace and gptq.patch_model
* Update src/bindings/python/src/openvino/frontend/pytorch/gptq.py
Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>
* Update src/bindings/python/src/openvino/frontend/pytorch/gptq.py
Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>
* Code style
* Revers int4 byte order
* Fixed core tests
* Fixed unguarded dynamic_cast result
Co-authored-by: Evgenya Nugmanova <eva.my.link@gmail.com>
* Fixed transformation tests
* Update src/bindings/python/src/openvino/frontend/pytorch/gptq.py
Co-authored-by: Maxim Vafin <maxim.vafin@intel.com>
* Prevent patching of non-gptq models
* Removed extra calling of quantized weights decompression patterns
* Better detection of supported AutoGPTQ models + more diagnostics
* Accurate diagnostics in case when aten::stack has multiple axes
---------
Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>
Co-authored-by: Ilya Churaev <ilyachur@gmail.com>
Co-authored-by: Evgenya Nugmanova <eva.my.link@gmail.com>
Co-authored-by: Maxim Vafin <maxim.vafin@intel.com>
* [TF FE] Support TF 2.14 and add OnesLike translator
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Update tests constraints
* Update open_model_zoo
* Adopt TF Lite test to 2.14 TF
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Support TF Lite layer tests for diffrent TF versions
---------
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* only skip test if mac
* unskip
* unskip trigger
* skip for onnx fe as well
* do not skip
* return skips and unskip test_backend in Python API 1.0
* rm pr trigger
---------
Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>
* Optimize CompressQuantizeWeights transformation
- remove CoordinateTransform usage from FakeQuantize reference implementation
- move ZeroPointOptimizer functionality inside CompressQuantizeWeights
- compute scale and zero point in the same loop
Ticket: CVS-119273
* review comments
* clang format
* fix comments
Przemyslaw Wysocki
Pawel Raasz
dependabot[bot]
Maciej Smyk
Jan Iwaszkiewicz
LiuandMichal Lukaszewski
Ilya Lavrenov
Anastasia KuporosovaandRoman Kazantsev
Anastasiia Pnevskaia
Alina Kladieva
Surya Siddharth Pemmaraju
Ekaterina Aidova
Maxim Vafin
Mikhail Ryzhov
Ilya Churaev
Andrey KashchikhinandIlya Lavrenov
Lucas Alava PeñaandJan Iwaszkiewicz
Mustafa CavusandMaxim Vafin
Mateusz Tabaka