Partial GPTQ int4 models conversion support + Swap nibbles in u4/i4 element type (#20371)

* Reference implementation for u4 constant compression from pytorch model based on bitwise ops pattern

* Fixed order of 4-bit halfs in byte

* Switched PyTorch FE to dev mode: in case if model cannot be fully converted, give partially converted model with PTFrameworkNode's with a printed warning (normally would raise an exception in case).

* Moved u4 compression to utils_quantize. Implemented not-interleaved version of u4 compression

* Removed debug output

* Added aten::matmul to the list of exceptions in may_produce_alias as a workaround for gptq models

* Added patching for gptq models applied automatically in convert_model

* WA for an inssue with u4 with earlier convert to fp16

* U4 blocked repacking for gptq patched model layout

* Deleted obsolete u4 re-packing based on aten::cat. Fixed the resulting u4 constant shape. Removed debug output.

* Revert "Switched PyTorch FE to dev mode: in case if model cannot be fully converted, give partially converted model with PTFrameworkNode's with a printed warning (normally would raise an exception in case)."

This reverts commit 0ef1455e70.

* Update src/frontends/pytorch/src/op/cat.cpp

* Check mask and shift values in u4 pattern. deque -> OutputVector for u4_compression_stack

* Convert to a given floating type instead of half in gptq patching. Better structured code.

* Code style fix

* Removed deque include

* Code style fixes

* Trailing space removed

* Fixed patched_forward and ts_decoder after unvalidated commits.

* Swap nibbles in u4/i4

* Better exception handling around jit.trace and gptq.patch_model

* Update src/bindings/python/src/openvino/frontend/pytorch/gptq.py

Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>

* Update src/bindings/python/src/openvino/frontend/pytorch/gptq.py

Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>

* Code style

* Revers int4 byte order

* Fixed core tests

* Fixed unguarded dynamic_cast result

Co-authored-by: Evgenya Nugmanova <eva.my.link@gmail.com>

* Fixed transformation tests

* Update src/bindings/python/src/openvino/frontend/pytorch/gptq.py

Co-authored-by: Maxim Vafin <maxim.vafin@intel.com>

* Prevent patching of non-gptq models

* Removed extra calling of quantized weights decompression patterns

* Better detection of supported AutoGPTQ models + more diagnostics

* Accurate diagnostics in case when aten::stack has multiple axes

---------

Co-authored-by: Alexander Kozlov <alexander.kozlov@intel.com>
Co-authored-by: Ilya Churaev <ilyachur@gmail.com>
Co-authored-by: Evgenya Nugmanova <eva.my.link@gmail.com>
Co-authored-by: Maxim Vafin <maxim.vafin@intel.com>
This commit is contained in:
authored and GitHub committed 2023-10-18 21:06:17 +04:00
1 parent cf9791e3e0
commit 46935e0a00
19 files changed
+516 -236

No files matched your search

@@ -1188,7 +1188,7 @@ void constant_convert_test(element::Type type_from,
}
ASSERT_TRUE(actual.size() >= expected.size());
for (size_t i = 0; i < expected.size(); i++) {
ASSERT_EQ(expected[i], actual[i]);
EXPECT_EQ(expected[i], actual[i]) << "Elements with index " << i << " are not equal.";
}
}
@@ -1378,7 +1378,7 @@ TEST(TransformationTests, ConvertPrecision_ConstantConversion_U1ToU4) {
constant_convert_test<uint8_t, uint8_t>(element::u1,
element::u4,
std::vector<uint8_t>{171},
{1, 0, 1, 0, 1, 0, 1, 1});
{0, 1, 0, 1, 0, 1, 1, 1});
}
TEST(TransformationTests, ConvertPrecision_keep_precission_sensitive_fp32_with_exp) {