* [Core] Support String Tensors
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Add String Constant implementation
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix build issue in tests
* Add cast_vector for Constant of ov::string type
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix build issue
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix build issue: ambiguous type in GNA
* Fix ambiguous build issue in GNA tests
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix code-style
* Fix code-style
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix ambiguous build issue in GNA tests
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix ambiguous build issue in TF FE tests
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Update openvino.style for naming convention check
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix compilation error in core unit tests - need typename
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Add test for new element_type
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix code-style
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Update src/inference/src/dev/make_tensor.cpp
Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>
* Add support of string Tensors for Constant
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix copying string tensor value for Constant
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Complete template methods for Constant
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Improve performance for initialization and destruction of string Tensor for set_shape
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Add check for string value in test
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Remove unused variable
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Update src/inference/src/dev/make_tensor.cpp
* Fix copy_to for ITensor of string type and add tests
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Add tests for Constant of string type and serialization
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Use memset_allocation to switch initialization
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Add additional documentation for host_ptr
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Update src/core/src/op/constant.cpp
* Use OPENVINO_THROW
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Update src/core/include/openvino/op/constant.hpp
* Update src/core/include/openvino/op/constant.hpp
Co-authored-by: Pawel Raasz <pawel.raasz@intel.com>
* Apply code-review feedback: use string_size
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Apply code-review feedback
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Recover evaluate impl for non-string type
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix code for creating of string constant for legacy non HostTensor tensor
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix build issue
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Apply code-review feedback: simplify copy_to method
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix build issue
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Use StringAlignedBuffer to store string Constant values
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Remove not needed methods in StringAlignedBuffer
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Refactor set_shape method
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
---------
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com>
Co-authored-by: Pawel Raasz <pawel.raasz@intel.com>
* Update reverse infer to allow changing shape if it already partially defined
* Update tests
* Remove changes in If
---------
Co-authored-by: Roman Kazantsev <roman.kazantsev@intel.com>
* add to DisableShapeOfConstantFolding flag; set it in ts_gather; add unit test
* fix ts_general; use check_shape flag in EnableShapeOfConstantFolding
* set EnableShapeOfConstantFolding constructor explicit
---------
Co-authored-by: Ivan Tikhonov <ivan.tikhonov@intel.com>
* Added experimental ScaledDotProductAttention operation in opset12. Supported in PT FE for aten::scaled_dot_product_attention translation. Decomposed in the common optimizations as functional reference.
* Better ScaledDotProductAttention
- Moved decomposition to the decomposing transformation
- Implemented more ctors for the op
- Renamed is_causal to causal
- Shape/type inference native code instead of using decomposition
- Moved the op from opset12 to opset13
- Added Python wrapper for ScaledDotProductAttention
* Fix test that counts ops in the opsets
* Update src/core/src/op/scaled_dot_product_attention.cpp
Co-authored-by: Katarzyna Mitrus <katarzyna.mitrus@intel.com>
* Update src/core/src/op/scaled_dot_product_attention.cpp
Co-authored-by: Katarzyna Mitrus <katarzyna.mitrus@intel.com>
* Move ScaledDotProductAttentionDecomposition from fusions to decompositions.
* Remove not used legacy shape inference in ScaledDotProductAttention
* Better namespace usage
* Register all nodes in ScaledDotProductDecomposition for correct tracking of nodes and running next mather passes on all new nodes.
* Don't use register_new_node_
* ScaledDotProductAttention specification (with an extra scale argument)
* Code style fix
* Scale input implementation for ScaledDotProductAttention
* Handle attention_mask=0 case in the op spec
* Better description of scale input
* N->M in scale description
* Code style fix, remove debug print.
* Apply suggestions from code review
Co-authored-by: Katarzyna Mitrus <katarzyna.mitrus@intel.com>
Co-authored-by: Mateusz Mikolajczyk <mateusz.mikolajczyk@intel.com>
* Fix for case when is_causal is not passed
* Extended description of ScaledDotProduct op
* Better description in py op wrapper
* Basic shape propagation tests for ScaledDotProductAttention
* Added ScaledDotProductAttention to toc.
* Add op impl check
---------
Co-authored-by: Katarzyna Mitrus <katarzyna.mitrus@intel.com>
Co-authored-by: Mateusz Mikolajczyk <mateusz.mikolajczyk@intel.com>
* Fix constant folding in MulMulMulFusion
by add f64 precision in Multiply to perform evaluate for const folding
* Do not transform if input has not supported type
* Handle Reshape's special zero in SimplifySecondInputOfReshape
SimplifySecondInputOfReshape detects ShapeOf->Gather->Concat
subgraphs on Reshape's second input and replaces ShapeOf->Gather
with a Constant with zero(s). Currently it works only with Reshapes
that have special_zero set to true, but it can work for Reshapes
with special_zero == false if non-Gather inputs to Concat are Constants
and don't contain any zero.
Ticket: CVS-123434
* fix no default output
* Handle Reshape in SplitSqueezeConcatFusion
Ticket: CVS-122455
* move check for squeeze/reshape
* add some comments
* review comments
* add use_shapes flag to SplitSqueezeConcatFusion
* Add static shape adapter
- Adapters holds CPU dimension which can be reference to it or vector
- Add ov::optional for holding optional result from shape inference
- Add new `infer` function in `IStaticShapeInfer`
* Temporary support of StaticShape
* Minor corrections in ShapeInferenceTA
* Migrate shape_infer to new interface version
* Replace StaticShape by adapter implementation
* Replace IShapeInferCommon by IStaticShapeInfer
* Correct code formatting
* Fix build issues
* NodeValidationFailure::create for StaticShapeRef
* Review shape inference for reshape operator
- review shape_infer implementation
- add more unit test for static and dynamic shapes
* Fix build issues
* Correct minus one dim calculation
* Fix build issues on windows
* Improve resolving special minus one
* Use NODE_SHAPE_INFER_CHECK
* Update product in/out calculations
* Temporary add ngraph header to solve build issue
* Correct minus one dim calc when static part same
* Add check for scalar input
* Remove debug message
* Fix `minus one` dynamic dimension calculation
* Fix `minus one` dynamic dimension calculation
* Fix merge issues in reshape
Minor refactor reshape evaluate
* Don't pass input label on minus one pattern
when input dimension will be modified.
* Add Rotation support to primitive and kernel
* Add unit tests
* Add transformation for NMSRotated
* add single-layer tests
* Fix: angle value for the same box may have its sign changed several times passing through iterations of batch and class loops.
* fix review comments
Transformation fuses Transpose on first or second MatMul's input
and sets MatMul's transpose_a/transpose_b accordingly.
TransposeMatMul is already part of SmartReshape, but it can be added
to MOCTransformations as well so native models that are don't use reshape
can benefit from that.
Ticket: CVS-118908
Current implementation tries to leverage branchless approach, but it's not correct
if scale is 0. In that case - zero point can can become inf or nan and multiplication
by 0 doesn't change its value. That causes another issue - infinite or NaN zero point
cannot be optimized out later.
Ticket: CVS-122931
Co-authored-by: Ivan Tikhonov <ivan.tikhonov@intel.com>
* [TF FE] Provide full support of TF1 Control flow and TensorArray ops
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Add missed header for TensorArrayV3 op
* Temporarily disable GRU cell fusion
* Update src/common/transformations/src/transformations/common_optimizations/moc_transformations.cpp
* Fix a case when element_shape for TensorArrayV3
* Fix translator for TensorArrayCloseV3
* Update summarize graph with TensorArrayCloseV3
* Add layer tests for TensorArrayScatterV3, Close, Size, Array
* Fix output shape for Merge node
* Remove unused variable
* Fix translator for TensorArrayConcatV3
* Fix translator for TensorArrayConcatV3
* Add layer tests for TensorArrayWriteV3, Gather, and Concat
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Add translator for GatherTree
* Fix TF FE unit-test for GatherTree
* Fix GatherTree translator
* Fix GatherTree translator to handle 1d end_token
* Fix undeclared parameter issue
* Fix GatherTree unit-test
* Add TensorArrayV3Replacer transformation
* Temporarily disable dangling transformation
* Recover RemoveMultiSubGraphOpDanglingParamsResults transformation
* Recover GRUCellFusion transformation
* Simplify check for GRUCellFusion transformation
* Use proper name for unit-tests
* Simplify translator for TensorArrayWriteV3
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Fix RemoveMultiSubgraphOpDanglingParamsResults transformation
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
* Additional fix for remove_multi_subgraph_op_dangling_params
* Make static TI run a dynamic subgraph
* Dedicated SL test
* Change condition to respect stat shapes
* Adjust test to cover the code path properly
* Recover fallback for still failing case GNMT
---------
Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>
Co-authored-by: Maksim Kutakov <maksim.kutakov@intel.com>
* Reference implementation for u4 constant compression from pytorch model based on bitwise ops pattern
* Fixed order of 4-bit halfs in byte
* Switched PyTorch FE to dev mode: in case if model cannot be fully converted, give partially converted model with PTFrameworkNode's with a printed warning (normally would raise an exception in case).
* Moved u4 compression to utils_quantize. Implemented not-interleaved version of u4 compression
* Removed debug output
* Added aten::matmul to the list of exceptions in may_produce_alias as a workaround for gptq models
* Added patching for gptq models applied automatically in convert_model
* WA for an inssue with u4 with earlier convert to fp16
* U4 blocked repacking for gptq patched model layout
* Deleted obsolete u4 re-packing based on aten::cat. Fixed the resulting u4 constant shape. Removed debug output.
* Revert "Switched PyTorch FE to dev mode: in case if model cannot be fully converted, give partially converted model with PTFrameworkNode's with a printed warning (normally would raise an exception in case)."
This reverts commit 0ef1455e70.
* Update src/frontends/pytorch/src/op/cat.cpp
* Check mask and shift values in u4 pattern. deque -> OutputVector for u4_compression_stack
* Convert to a given floating type instead of half in gptq patching. Better structured code.
* Code style fix
* Removed deque include
* Code style fixes
* Trailing space removed
* Fixed patched_forward and ts_decoder after unvalidated commits.
* Swap nibbles in u4/i4
* Better exception handling around jit.trace and gptq.patch_model
* Update src/bindings/python/src/openvino/frontend/pytorch/gptq.py
Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>
* Update src/bindings/python/src/openvino/frontend/pytorch/gptq.py
Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>
* Code style
* Revers int4 byte order
* Fixed core tests
* Fixed unguarded dynamic_cast result
Co-authored-by: Evgenya Nugmanova <eva.my.link@gmail.com>
* Fixed transformation tests
* Update src/bindings/python/src/openvino/frontend/pytorch/gptq.py
Co-authored-by: Maxim Vafin <maxim.vafin@intel.com>
* Prevent patching of non-gptq models
* Removed extra calling of quantized weights decompression patterns
* Better detection of supported AutoGPTQ models + more diagnostics
* Accurate diagnostics in case when aten::stack has multiple axes
---------
Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>
Co-authored-by: Ilya Churaev <ilyachur@gmail.com>
Co-authored-by: Evgenya Nugmanova <eva.my.link@gmail.com>
Co-authored-by: Maxim Vafin <maxim.vafin@intel.com>
* add new operations as unary
* get unary as input(0) instead of iterating pattern map
* add CumSum + unit tests
* add Tile + unit tests
* add tile
* fix ts_tile
* code review fix: use ADD_MATCHER
* fix bug CI tests
* Optimize CompressQuantizeWeights transformation
- remove CoordinateTransform usage from FakeQuantize reference implementation
- move ZeroPointOptimizer functionality inside CompressQuantizeWeights
- compute scale and zero point in the same loop
Ticket: CVS-119273
* review comments
* clang format
* fix comments
* Do not normalize negative indices for Gather v8
* code style fix
* added transformation test with accuracy check for Gather-v8
* removed GatherNegativeConstIndicesNormalize transformation at all
* ConvertGather8ToGather7 conversion: added more checks
* Introduced shared Gather8withIndicesDataLayerTest: added CPU, GPU instances
* code style fix
* small fix
* review fixes
* do negative indices normalization if possible
* code style fix
* refactor cpu test instances
* code style fix
* [transformations] WeightsDequantizeToFakeQuantize: Extend pattern matching with the case when both Subtract inputs are Convert
* [transformations] WeightsDequantizeToFakeQuantize: Added new tests to cover the extention added to pattern match
* Fix review comments
* Add EnableShapeOfConstantFolding transformation
Transpose sinking (that is used in TF frontend) disables ShapeOf constant folding
which prevents some optimizations further in the pipeline.
This patch introduces EnableShapeOfConstantFolding that removes DisableConstantFolding
from ShapeOf nodes.
Ticket: CVS-118890
* add description
* review comments
* headers
Xuejun Zhai
Maxim Vafin
Evgenya Nugmanova
Ivan Tikhonov
Vladislav Golubev
Evgeny KotovandIvan Tikhonov
Vitaliy Urusovskij
Aleksandr Voron
Mateusz Tabaka
Nikolay Shchegolev
Pavel EsirandAndrei Kochin
Vladimir Paramuzov
Oleksii Khovan
Nadezhda Ageeva
Anton Voronov
Ilya Churaev
Andrei Beleiu
Roman Lyamin