* Added experimental ScaledDotProductAttention operation in opset12. Supported in PT FE for aten::scaled_dot_product_attention translation. Decomposed in the common optimizations as functional reference.
* Better ScaledDotProductAttention
- Moved decomposition to the decomposing transformation
- Implemented more ctors for the op
- Renamed is_causal to causal
- Shape/type inference native code instead of using decomposition
- Moved the op from opset12 to opset13
- Added Python wrapper for ScaledDotProductAttention
* Fix test that counts ops in the opsets
* Update src/core/src/op/scaled_dot_product_attention.cpp
Co-authored-by: Katarzyna Mitrus <katarzyna.mitrus@intel.com>
* Update src/core/src/op/scaled_dot_product_attention.cpp
Co-authored-by: Katarzyna Mitrus <katarzyna.mitrus@intel.com>
* Move ScaledDotProductAttentionDecomposition from fusions to decompositions.
* Remove not used legacy shape inference in ScaledDotProductAttention
* Better namespace usage
* Register all nodes in ScaledDotProductDecomposition for correct tracking of nodes and running next mather passes on all new nodes.
* Don't use register_new_node_
* ScaledDotProductAttention specification (with an extra scale argument)
* Code style fix
* Scale input implementation for ScaledDotProductAttention
* Handle attention_mask=0 case in the op spec
* Better description of scale input
* N->M in scale description
* Code style fix, remove debug print.
* Apply suggestions from code review
Co-authored-by: Katarzyna Mitrus <katarzyna.mitrus@intel.com>
Co-authored-by: Mateusz Mikolajczyk <mateusz.mikolajczyk@intel.com>
* Fix for case when is_causal is not passed
* Extended description of ScaledDotProduct op
* Better description in py op wrapper
* Basic shape propagation tests for ScaledDotProductAttention
* Added ScaledDotProductAttention to toc.
* Add op impl check
---------
Co-authored-by: Katarzyna Mitrus <katarzyna.mitrus@intel.com>
Co-authored-by: Mateusz Mikolajczyk <mateusz.mikolajczyk@intel.com>
- fix cum_sum_partial_sum kernel;
- add unit test and func test for big shapes;
- add test to compare Partial vs Ref performance;
- change kernels' priorities according to performance measurements;
- move common profiling helpers to test_utils.
Ticket: CVS-123590
* Migrate Multiply operator to new API
* Add comment explain use of custom multiply
* Update custom multiply comment
Co-authored-by: Tomasz Jankowski <tomasz1.jankowski@intel.com>
---------
Co-authored-by: Tomasz Jankowski <tomasz1.jankowski@intel.com>
* [GPU] Support dynamic tensoriterator with -1 num_iteration
- remove redundant codes
* [GPU] Refactoring methods for pre_process / post_process for body_network
* Add unit test for dynamic tensoriterator wo trip_count_id
* Follow-up code review
* Set inner network in loading of model cache
* Fix legacy loop unit tests
* [TF FE] Fix conversion of TF1 OD models out-of-the-box
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Add test While with nested If operation
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Update tests/layer_tests/tensorflow_tests/test_tf_While.py
---------
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* [CPU] CPU plugin migrates to plugin API 2.0
* Fix legacy config/metric issue
* Fix some issue of ov_cpu_func_tests
1. set_tensors_impl segment fault
2. ov::loaded_from_cache unsupported issue
* Resolve some comments
1. ov::loaded_from_cache issue
2. throw_if_cancelled issue
3. import_model issue
4. set_tensor_impl issue
5. batched_inference issue
* Fix dynamic shape inference issue
* Fix build error
* keep original model info in infer_request
* Fix minor error
* cache internal tensors for input/output precision change
* Disable import model test cases with precision changes
* fix precision issue
* Fix issue for import model
* Fix InferRequestCancellationTests exception issue
* Skip InferRequestIOBBlobTest.*secondCallGetInputDoNotReAllocateData due to new plugin api have different behavior
* Fix graph name issue
* Fix ROI issues
* Fix Transpose shape issue
* Skip vie::Version test due to change to ov::Version
* Solve input port name changes issue
* Solve preprocess layout issue
* Fix minor issue
* tidy up code
* Fix conflict after rebase
* Fix Windows build warning
* Add aux tensors for precision change issue
* Fix import/export model issue
* WA single layer name changed by preprocess
* Revert "WA single layer name changed by preprocess"
This reverts commit bc8fcdd43c.
* Skip some legacy tests due to plugin api 2.0 is enabled
1. skip some python legacy tests for plugin api 2.0 some different behaviors
2. skip some smoke tests due to output port name was changed
* Fix 2 build warnings
* Skip some AUTO plugin tests
* Fix property issue caused by AUTO plugin
* Skip PSROIPooling issues
* Follow header files reference policy
* Split out transformation fixing for nop_elimination
* Fix AUTO plugin mismatch issue for get_tensor function
* Fix aux tensor shape issue
* Fix tensor shape issue
* WA python sync inference sample's segmentfault issue
* Fix reshape issue for dynamic inference
* Fixed incorrect tensor name in e2e test
Fixe issue: e2e ONNX_Customized_Cascade_Rcnn_api_2_True_batch_1_device_CPU_precision_FP325den8cnk
* Fix python segmentfault issue of plugin api 2.0
* Fix python segmentfault issue of plugin api 2.0
* Revert "Fix python segmentfault issue of plugin api 2.0"
This reverts commit 6f502e5d86.
* Fix onnx_duplicated_output_name due to empty tensor
Co-authored-by: Bell, Song <bell.song@intel.com>
* Remove redundant code
* Remove python segment fault WA
* Keep rt_info to fix test failure in case of legacy public api
* Fix output port names missing issue
* Adress some reviewers' comments
* Restore OnnxBackendNodeModelTest::test_maxpool_with_argmax_2d_precomputed_pads_cpu after fixing has been merged
* Resolve tensor sharing issue when there are same name output port name
In some case, model has 2 or more same name input/output ports, they aslo have the same
precision and partial_shape. Compiled_model will share the same ov::Descriptor::Tensor pointer
and ov::Tensor between multiple such ports.
Considered solving python segment fault issue to create seperated input/output ports, which also
need handle such tensor shared case, this patch will do it.
* Resolve tensor sharing issue when there are same name output port name
In some case, model has 2 or more same name input/output ports, they aslo have the same
precision and partial_shape. Compiled_model will share the same ov::Descriptor::Tensor pointer
and ov::Tensor between multiple such ports.
Considered solving python segment fault issue to create seperated input/output ports, which also
need handle such tensor shared case, this patch will do it.
* Better method to find shrared tensor desc
* rename with snake_case style
* Remove ngraph header files
* Keep external_ptr naming
* Add OPENVINO_SUPPRESS_DEPRECATED for some legacy code
* Use port's tensor_ptr to replace creating new tensor_ptr
* Resolve some reviewer comments
* Implement ov::IInferRequestInternalWrapper::GetPreProcess to recover python GetPrepProcess tests
* Remove unnecessary header files reference
* Assert the risk of precision change and reorder at the same time
* Modify legacy python test to fit plugin api 2.0 behavior
* Recover smoke_Transpose(2|4|5|6)D/TransposeLayerTest.CompareWithRefs due to fixing is merged
* Fix typo issue
* Address reviewer's comments
* Disable precision coversion
* Fix error when CpuBlockedMemoryDesc
* Remove precision mismatch WA
* WA precision issue for query_model
* Solve precision mismatch between compiled model and graph
* Fixe failure of query_model
* Rebase to new plugin api update
* Recover the test cases of precision mismatch
* Try to fix name changing for graph model
* Remove tets code
* Remove fp64
* Rebase to new plugin api update
* Update for some failure cases
* Fix bert_benchmark failure issue
* Avoid segment fault in arm acl
Legacy public api + cpu plugin api will add convert op by preprocess by default for unsupported precision,
but ACLConvertExecutor cannot support dimension > 6, so this test will be segment fault due to dimension > 6
smoke_TestNumpyBroadcastNgraphEvaluate/BroadcastLayerTest.CompareWithRefs/targetShape=(1.2.3.4.5.6.7.8.9.10)_axesMapping=()_mode=numpy_inShape=(1.2.1.4.1.6.1.8.1.10)_inNPrec=I8_trgDev=CPU
smoke_TestNumpyBroadcastNgraphEvaluate/BroadcastLayerTest.CompareWithRefs/targetShape=(1.2.3.4.5.6.7.8.9.10)_axesMapping=()_mode=numpy_inShape=(1.2.1.4.1.6.1.8.1.10)_inNPrec=U8_trgDev=CPU
* Remove precision change from preprocess to avoid ACL unsupport convert dim > 6
* ACLConvertExecutor cannot support dimension > 6, don't let preprocess to add Convert
* Revert "ACLConvertExecutor cannot support dimension > 6, don't let preprocess to add Convert"
This reverts commit fd7a8b35af.
* Revert "Remove precision change from preprocess to avoid ACL unsupport convert dim > 6"
This reverts commit 3c2d9a5f17.
* Debug
* Debug incorrect precision checking issue
* Debug Eltwise FP64 unsupported issue
* Add logs for precision
* debug log
* Update for new dependent PRs merged
* Fix failure caused by preprocess
Fix below failures due to cannot find ops by name
smoke_LPT/ReduceMaxTransformation.CompareWithRefImpl/f32_[1,3,10,10]_CPU_f32__256*
* Fix build error
* Fix failure caused by missing code during rebase
* Add debug
* Fix precision unsupport issue
* U16/I16/U64 precision support
* Resolve the issue of f64 reorder
Fix below issue:
Cannot create reorder primitive: unsupported reorder case
* Fix convert multiple child edge issue
* Solve ROI tensor failure issues
* Temporarily disable num_nodes comparison
* Only change convert precision for fp64
* Put convert precision change before reorder to avoid confusion
* Add debug log for transformation
* Fix rebase confilict
* Fix clang issue
* Temporarily disable test_infer_mixed_values python test of bf16
* Solve issue of smoke_ConvertCPULayerTest_BOOL_Dynamic_inputPRC=BF16 choose FP32 primType rather than BP16 primType
* Fix issue of pytorch_tests/test_outer.py
There are 2 output ports, but with the same port name, they should share the same tensor.
* Fix arm cannot find Eltwise executor issue
smoke_SetBlobCPU/SetBlobTest.CompareWithRefs/Type=INPUT_Device=CPU_PrecisionInNet=FP16_PrecisionInNgraph=BOOL
will report below error:
[ GENERAL_ERROR ] Supported Eltwise executor is not found
It need change convert precision to avoid such problem.
* Fix memory overwritten issue
* Temporarily skip arm fp16 SetBlobTest
* Fix compile error after rebase
* Restore smoke_IsOp test due to fixing pr merged
* Fix float to bf16 issue in avx2 isa
* solve onnx test xfail issue
* Skip test cases that ARM Eltwise executor FP16 is not supported
smoke_SetBlobCPU/SetBlobTest.CompareWithRefs/Type=INPUT_Device=CPU_PrecisionInNet=FP16_PrecisionInNgraph=BOOL
smoke_SetBlobCPU/SetBlobTest.CompareWithRefs/Type=BOTH_Device=CPU_PrecisionInNet=FP16_PrecisionInNgraph=BOOL
[ GENERAL_ERROR ] Supported Eltwise executor is not found
* [CPU] improve reorder to support any precision
* Implement ReorderExecutor
* Fix builld error
* Not cache executor due to its primitive has been cached
* Keep convert one time at most
At most insert one convert if needed, if still cannot do reorder it will throw exception rather than insert the second convert
For example, below reorder will not be supported:
FP64<->I64/U64/U32
U32<->I64/U64
U32<->I16/U16
FP64<->FP64
BIN<->BIN
* Only do conversion if layout is same
* update for only convert case
* Update for reviewer comments
* update for failure cases
* Address reviewer comments
* Update rebase issue
* minor update
* Solve unsupported precision issue in tranfromation rather than init_edge
* Remove unnecessary convert in init_edge
* Minor changes
* Update Reorder::reorderData
* Solve issue if only coversion without reorder
* Address reviewer comments
* Address reviewer comments
* Keep exception for unsuported precision
* update
* Revert reorder executor implement
* Solve float->bool issue on transformation pipeline
* Solve I64 is not supported issues
* Solve reviewer's comments
* Fixed dynamic top_k node issue
* Skip nhwc and nChw16c test cases for ConvertLayer
* Update for reviewers' comments
* Fix some failures
* Update for several failure cases
* Update for apiConformanceTests failures
* Fix incorrect node name after import model
* update
* update comments
* Solve issue of smoke_MatMul_NoTranspose and smoke_MatMul_BothTranspose
* Fixed AlignMatMulInputRanks scalar issue
* Address reviewers' comments, remove redundant path in graph.cpp
* Remove test_div_uint8_cpu from xfail_issue_58676
* Solve invalid number of nodes for smoke_Snippets_BroadcastSelect
* ConstantResultSubgraphTest of u16/i16/u32/i64/u64
* restore smoke_SetBlobCPU BOOL tests for arm
* [CPU] Fix ARM precision issue
ARM64 ACL prefers fp16 than fp32, API 2.0 requires input/output precision not changes,
then fp32 input will trigger convert node is added to convert fp32 to fp16.
* Solve some ARM64 failures
* Fix arm64 InferRequestVariableStateTest tests out of memory issue
ARM64 will force fp16 precision, which cause states memory can be fp16, so memcpy to state_memory
cannot use float * element_size, else it will be out of memory bound.
* Skip 2 arm64 tests caused by forcing fp16 precision
* Revert "Fix arm64 InferRequestVariableStateTest tests out of memory issue"
This reverts commit 3e12bd48c2.
* Fix python test_get_profiling_info failure issue
---------
Co-authored-by: Bell, Song <bell.song@intel.com>
Co-authored-by: Chen Peter <peter.chen@intel.com>
* Remove `ngraph` from PT FE and FE tests utils
* Remove `ngraph` from Paddle FE
* Remove `InferenceEngine` from some ONNX FE test
* Port `generate_embedding.py` to API2.0
* CLangFormat
* Fix comments
* Migrate Mish operator to new API
* Remove `visit_attributes` is same as base class
* Refactor Mish reference implementation
* Add cast as function is generic
-mish calculation is floating-point but return type can be integral.
* Fix constant folding in MulMulMulFusion
by add f64 precision in Multiply to perform evaluate for const folding
* Do not transform if input has not supported type
* [TF FE] Support complex tensors
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Align output type for Real and Imag operations
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Update decoding complex types
* Add support for ComplexAbs, FFT and IFFT operations
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Correct axes based on a number of inner-most dimensions
* Add layer tests
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Update supported ops documentation
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Add a comment for ComplexTypeMark
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
---------
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* MIgrate GridSample to new API
* Refactor GridSample to reduce binary size
- use function pointer instead std::function (simpler less code size)
- use RoundingGuard instead manual set/restore rounding mode
- move interpolate selection outside main data processing loop
