Commit Graph

338 Commits

Author SHA1 Message Date
iliya mironov
642a62d0e2 Add axis support (#5242)
* Add axis support

* Update dequant extractor

* Update qdeq ops

* Refactoring quantize

* Update dequantize resolver

* Update dequantize op

* Refactoring dequantize

* Some fixes for quantize and dequantize

* Update unit tests

* Reafctoring quantize/dequantize axis support

* Move quantize/dequantize resolvers to middle

* hot fix

* Fix unit tests

* Fix unit tests

* Update quintize resolver comment

* Refactoring code according to code review

* Fix according to review

* Change order for transforms quantize pipline
2021-05-17 15:03:22 +03:00
Evgeny Lazarev
717499cf2f Common telemetry (#5032)
* Moved telemetry to repo root directory from MO

* New telemetry package in the "openvino" sub-directory

* Removed telemetry from the MO BOM

* Updated MO BOM and added stub file for telemetry

* Fixed license header

* Fixed license headers and cleaned up the telemetry setup.py

* Fixed import

* Added temporary dependency for openvino-telemetry

* Added ignore for pylint issues

* Fixed import statements

* Updated imports in the telemetry library

* Removed telemetry library. Added link to another private repo

* Removed redundant start_session event for the MO

* Changed approach to import the telemetry library

* Minor code refactoring

* Updated MO telemetry events sending messages

* Refactor sending events for the IE runtime check

* Disable forcing warnings for deprecated methods

* Removed changes from the requirements.txt to install telemetry library to avoid merge conflicts

* Update copyright in the model-optimizer/mo/utils/telemetry_stub.py

Co-authored-by: Gleb Kazantaev <gleb.nnstu@gmail.com>

Co-authored-by: Gleb Kazantaev <gleb.nnstu@gmail.com>
2021-05-14 21:56:03 +03:00
Ilya Naumov
3e076a88aa Align MO pip requirements (#5171)
* Align MO pip requirements

* fix missing mxnet on platforms other than Windows or Linux

* remove spaces near environment markers
2021-05-14 16:05:25 +03:00
Sergey Lyubimtsev
5af767a0a2 Add MO extras requires for supported DL frameworks (#5615)
* Add extras requires for supported DL frameworks

* fix pylint issues

* codestyle (PEP8)

* Indentation
2021-05-13 20:20:41 +03:00
Anton Chetverikov
f2f44ce160 Add uint64 to data types maps (#5603) 2021-05-12 18:50:38 +03:00
Gleb Kazantaev
b4565b7b4f Add Model Optimizer --transform option (#5504)
* Execute MO stages inside subprocess to have single IE check

* Add --transform key

* Updated ofline transformations to execute user specified passes; updated logic to raise when --transform is used

* Parametrize LowLatency transformation with num_iterations key

* Fixed MO and IE versions comparision

* Use subprocess for offline transformations execution to catch errors

* remove ie_is_available from IR; fixed typo

* Fix for old IE versions

* Update parse_transform key unit tests

* Show available transformations

* Fixed typo

* Fix review comments

* Fix python2 compatibility

* Fixed review comments

* Fixed __main__ import
2021-05-12 18:42:56 +03:00
iliya mironov
fe5ca28b6e Smal fix with tf env_setup (#5476)
* Smal fix with tf env_setup

* Fix tf loader

* Fix version checker
2021-05-12 15:33:03 +03:00
Pavel Esir
0b22d6c51c Add Gather-7 to MO (#5264)
* initial solution

* added unit-tests + some corrections

* axis getting improvements

* fixed MO IR reader for old IR's

* a couple of corrections

* applied review comments

* corrected negative batch_dims normalization for shape calculation, for IR original negative values are kept

* added additional checks and negative tests
2021-05-11 22:29:59 +03:00
Roman Kazantsev
dc22c177d5 Extend MO for operation Einsum-7 (#5401)
* Extend MO for operation Einsum-7

Signed-off-by: Roman Kazantsev <roman.kazantsev@intel.com>

* Add extractor for einsum and optimize code based on review feedback

Signed-off-by: Roman Kazantsev <roman.kazantsev@intel.com>

* Fix the code based on the review: correct code, tests and comments; move insert_transpose

Signed-off-by: Roman Kazantsev <roman.kazantsev@intel.com>

* Fix LayoutChangeForEinsum transformation condition

Signed-off-by: Roman Kazantsev <roman.kazantsev@intel.com>

* Update third-party dependencies

Signed-off-by: Roman Kazantsev <roman.kazantsev@intel.com>
2021-05-11 21:36:04 +03:00
Anastasia Kazantaeva
42226fccae Upgrade message for 2021.4 lts (#5471) 2021-05-11 15:58:32 +03:00
Anastasia Popova
84b94c9834 Fuse mul transformation fix (#5518)
* Changed fuse_mul behaviour for proper data node connection.

* Corrected the comment.

* Corrected the comment.

* Added permutation attribute saving.

* Added comment.

* Added unit tests, comments corrections.
2021-05-07 17:01:31 +03:00
Anastasia Popova
4ea09b1f98 ReverseInputChannels mapping fix (#5523)
* Fixed attributes saving to keep tensor debug info in Parameter node.

* Added comment and unit tests.

* Small correction.

* Small correction of unit test.

* Comment corrected.
2021-05-07 16:29:38 +03:00
Vladimir Gavrilov
0c288d506c MO support for operations DFT and IDFT (#5197)
* Written MO classes for DFT and IDFT operations.

* Added class to read TF (I)FFT operations.

* Written extractors for TF operations FFT, FFT2D, FFT3D, IFFT, IFFT2D, IFFT3D.

* Written MO Roll operation and TF Roll operation extractor.

* Started to write needed transformations.

* Written transformation StridedSlices + Complex + Roll + (i)FFTxD + Roll + (Imag, Real) + Pack -> Roll + (I)DFT + Roll.

* Written transformation for Complex + ComplexAbs.

* Written correction of axes of Roll.

* Small fix.

* Small fix.

* Some fixes.

* Some changes.

* Now TF Roll is read as TFRoll. Written inserting Transposes before and after (I)DFT.

* Small fix.

* Written tests for the transformation TFRollToRoll.

* Added comments to some transformations.

* Deleted redundant import.

* Written tests for the transformation TransposeDFT.

* Fixes in MO IR Reader to read/write (I)DFT.

* Fixes in the list of supported TF layers.

* Started to write tests for SSliceComplexRolledFFTPackBlockReplacement transformation.

* Written tests for the MO transformation SSliceComplexRolledFFTPackBlockReplacement.

* Written tests for the MO transformation ComplexAbs.

* Tests for transformations were moved into unit_tests directory.

* All extractors for (I)FFTxD are in one file now.

* Deleted redundant transformations.

* Fixed extractor for TF Roll: now this operation is read as MO Roll.

* Added comments to TFFFT operation.

* The method insert_transpose of classes TransposeDFT and LayoutChangeForGatherND was moved into the separate function in the file model-optimizer/extensions/middle/InsertLayoutPropagationTransposes.py.

* Fixed comment for the transformation TransposeDFT.

* Small fix.

* Some fixes.

* Deleted shape infer function for the operation TFFFT. Sorted imports in complex_abs.py.

* Small fixes.

* Deleted redundant import.

* Fixes in some asserts.

* Small fix.

* Added names for created nodes in the transformation ComplexAbs.

* Added comments to the method canonicalize_axes.

* The transformation SSliceComplexRolledFFTPackBlockReplacement was split into the sequence of transformations SSliceComplexRollReplacement -> RollRealImagPackReplacement -> TFFFTToDFT.

* Written tests for the transformation SSliceComplexRollReplacement.

* Written tests for the transformation RollRealImagPackReplacement.

* Written tests for the transformation TFFFTToDFT.

* Deleted commented code.

* Fixed types of constants in the transformation ComplexAbs.

* Written tests for canonicalization of signal_size value.

* Deleted 'Replacement' from names of files and classes.

* Used comarison of ids, not names.

* replace_sub_graph was replaced with find_and_replace_pattern.

* Now the transformation RollRealImagPack is executed before running transformation model-optimizer/extensions/front/Pack.py.

* The body of the function create_dft_from_tffft is a part of the transformation TFFFTToDFT body now.

* Now method correct_roll_axes of classes RollRealImagPack and SSliceComplexRoll is moved to the function in mo/front/tf/graph_utils.py.

* Small changes.

* Added comment before mark_input_as_in_correct_layout(roll, 2).

* Now the functions correct_roll_axes generates sub-graph in the input port 2 of Roll.

* Corrected tests for the transformation SSliceComplexRoll.

* Corrected tests for the transformation RollRealImagPack.

* Deleted commented code.

* Some renaming.

* Added decomposition of the separate operation ComplexAbs (without Complex before it).

* Added comment to the transformation ComplexAbsAfterComplex.

* Optimized imports for the transformation TFFFTToDFT.

* The transformation SSliceComplexRoll was split into the sequence SSliceComplex -> CorrectRollAxes and disabled.

* Written tests for the transformation ComplexAbs.

* Written tests for the transformation SSliceComplex.

* Written tests for the transformation CorrectRollAxes.

* Deleted the transformation SSliceComplexRoll.

* Deleted renaming nodes.

* Fixed comment.

* Small fixes.

* Small fix.

* The attribute need_correction was renamed as input_rank_changed.

* Small fixes.

* Deleted commented code.

* Now we iterate over all complex_node.out_port(0).get_connection().get_destinations() input ports and mark the corresponding nodes with the marker attribute.

* Added the attribute 'in_ports_count' into the class FFTBase.

* Tests for the transformation TransposeDFT were rewritten using helper functions.

* Now the transformation RollRealImagPack uses existing Roll node instead of creating new one.

* Small fixes.

* Fix in the documentation.

* Written class to read MxNet (I)FFT operations. Written corresponding extractors.

* Corrected shape infer function for MXFFT operation. Written transformation to convert MXFFT to (I)DFT.

* Fixed shape infer function.

* Fixed the conversion MXFFT to (I)DFT.

* Written tests for the transformation MXFFTToDFT.

* The function correct_roll_axes was replaced with more generic function add_constant_to_negative_values.

* Fixes in classes TFFFT, FFTBase, DFT, IDFT, MXFFT.

* Added asserts in constructors of operations TFFFT and MXFFT.

* Refactored transformation MXFFTToDFT: conversion of DFT and IDFT were moved into separated functions.

* Moved some commented code.

* Fixed BOM file.

* Written function convert_ifft_to_dft.

* Started to rewrite tests for MXFFTToDFT transformations, in the case is_inverse=False.

* Small fixes.

* Fixes in the transformation RollRealImagPack.

* Renaming tests class for the transformation SSliceComplex.

* Fixes in the function compare_graphs. Now we get all output nodes of op node, and these output nodes are sorted by names.

* Fixed tests for the transformation MXFFTToDFT.

* Fix in the transformation ThresholdedReluDecomposition: added disconnect for trelu input port.

* Fixes in test for the transformation TFSliceToSlice.

* Small fix in the transformation ObjectDetectionAPIPreprocessor2Replacement.

* Small fix in comment.

* Optimized imports.

* Used remove_node in the transformation ThresholdedReluDecomposition and remove_nodes_from in the transformation RollRealImagPack, instead of ports disconnection.

* Deleted commented code.

* Deleted test case test_slice_replacer_begin_with_2_inputs.
2021-05-07 09:44:24 +03:00
Anton Chetverikov
b3c1dd9de3 Add sys_platform environment marker to version checker (#5437)
* Add sys_platform environment marker

* Update sys_platform check

* Add unit tests for sys_platform marker

* apply review comments

* Fix typo

* Update checker and tests, apply comments

* Update comments parsing and tests

* Fix commrnt

* Resolve comments and update check logic

* Update tests and fix bug with negative tests

Co-authored-by: achetver <anton.chetverikov@.intel.com>
2021-05-06 16:20:37 +03:00
Yegor Kruglov
abb1ca657e Implementation of ArgMin ONNX + TF extractors (#5126)
* implement argmin extractors

* reconsidering argmax to topk

* arg ops refactoring

* rename ArgMaxToTopK

* added unittests

* update docs

* move unittest file to new folder

* conversations resolving

* revert changes with argmax.py, move argmin op to a new file

* rename ArgMaxSqueeze

* updated BOM file

* little fix

* code refactoring in ArgMaxOp, updated unittests

Co-authored-by: yegor.kruglov <ykruglov@nnlvdp-mkaglins.inn.intel.com>
2021-05-06 13:41:49 +03:00
Yegor Kruglov
e3ea9bf4dd fix incorrect input names for mean values (#5508) 2021-05-06 13:33:51 +03:00
Evgeny Lazarev
98783b88ac Removed transformation which removes Const->Result sub-graphs (#5295)
* Removed transformation which removes Const->Result sub-graphs

* Removed one more MO transformation which removes Const->Result sub-graph during the front phase
2021-05-05 14:24:05 +03:00
Gleb Kazantaev
b291ca8cfa Use Serialization as a default engine in MO (#5347)
* Use Serialization as a default engine in MO

* Added cmd option to use old serialization

* Added mapping file generation

* Test mapping file generation

* Fix setBatchsize parameters order; fix mapping file generation

* Added FrameworkNode; added method to read models with custom ops but without extensions

* Added python API for read_network_without_extensions function; updated mo not to use IECore

* Added read_model_without_extensions to IReader and IParser

* Fix V7 IR reader

* Fix pword value

* Fix dllexport macro usage

* Add metainfo to IR

* Fix nGraph code style

* Fix license header

* Restore prepare_emit_ir behaviour

* Fix compare_function to resolve situation when Result input port has multiple names

* Update Compare Functions

* Fix FrameworkNode validation

* Self-review

* CodeStyle check

* --use_fallback -> --use_legacy_ir_generation

* Sort imports in main.py

* --path_to_model -> --input_model

* Use logging instead of print

* Code simplifucation&cleanup

* Fix offline_Transformations key

* Fix GeneraeMappingFile comments

* Use Extension approach to work with custom ops

* Fix versions check

* Code clean-up

* Moved FrameworkNode to inference_engine_transformations library

* Fix FrameworkNode includes

* Code clean-up
2021-05-04 16:40:20 +03:00
Svetlana Dolinina
7b52e3155a Tdnnf (#5255)
* initial changes (IR not generated)

* extractor fix

* convert tdnnf (with correct infer)

* refactoring + comments in code

* added unit tests + couple fixes based on tests

* change order for old convolutions

* fix pylint

* small refactoring

* added if to remove changes in old irs

* doc updated

* fix layout and kernel shapes for old convolutions

* fixed test

* moved test

* fix import in test

* fixed backward compatibility

* review fixes
2021-04-29 14:38:08 +03:00
Anastasia Popova
07214d0a47 Changed "out_port_id" attribute setting in mapping file to store tensor names. (#5344)
* Removed port id from fw_tensor_debug_info attribute.

* Added port number to tensor names in kaldi, mxnet. Fixed Const naming.

* Sort imports.
2021-04-29 14:05:35 +03:00
Svetlana Dolinina
4daa74303a changed output finding + add error to catch problem in future (#5368)
* changed output finding + add error to catch problem in future

* added unit test

* review fixes

* review fixes

* fix typo
2021-04-28 12:59:59 +03:00
Anton Chetverikov
8395ce1b0f Update modules version checker in MO to support ~= (#5342)
* Add check for ~= sign for modules versions check

* Add unit test

* Update check for environment markers

* Update check for ~= sign

* Update unit tests

Co-authored-by: achetver <anton.chetverikov@.intel.com>
2021-04-28 11:23:08 +03:00
Sergey Lyubimtsev
7d5fd3d51c MO Transformation configs are missing in openvino-dev package (#5302)
* MO Transformation configs are missing in openvino-dev package

* fix pylint issues

* add escape

Co-authored-by: Roman Donchenko <roman.donchenko@intel.com>

* add escape

Co-authored-by: Roman Donchenko <roman.donchenko@intel.com>

* add mo_caffe.proto

* Add MO package_data to openvino-dev package

Co-authored-by: Roman Donchenko <roman.donchenko@intel.com>
2021-04-28 10:15:22 +03:00
Svetlana Dolinina
3028c78594 fix typo + test (#5392) 2021-04-27 13:37:02 +03:00
Anton Chetverikov
576e692b1d Update errors raising cases in modules versions checker in MO (#5141)
* Change error raising to message printing

* Update modules versions check

* Update check and add unit test

* Fix wrong function call

* Update unit tests

* Fix test

Co-authored-by: achetver <anton.chetverikov@.intel.com>
2021-04-23 13:23:06 +03:00
Yegor Kruglov
d2c4d068b5 fixed mismatch of input names between framework and MO .mapping file (#5256) 2021-04-21 18:46:49 +03:00
Evgeny Lazarev
f3d1aa6490 Moved MO unit test files to a separate directory (#5312)
* Removed test-generator from all MO requirement files except the dev one

* Moved all MO unit tests files to a separate directory

* Added __init__.py files to the tests directory. Fixed importing paths for some unit tests

* Fixed imports in all unit tests. Moved all unit test related files from the MO code to the dedicated directory

* Renamed directory with unit test utils

* Updated imports in unit tests
2021-04-20 14:47:18 +03:00
Yufei Wu
9488bb0b00 %~dp0 can't handle path which include Space (#5274)
* %~dp0 can't handle path which include Space

The default OpenVINO install directory is C:\\Program file(86)\\Intel\\..., which contains space. So when users run this script, the path is wrong.
To handle this issue, I modify two part:
one is the python_command, use a local var to store this path. 
The other is the errorlevel, just add double quotation marks

* fix python_command path issue

* delete unused blank line
2021-04-19 18:35:32 +03:00
Anastasia Popova
cb62f3f579 Added operation Roll to MO (#5206)
* Added operation Roll to MO.

* Updated BOM file.

* Replaced FrontReplacementPattern with FrontReplacementOp.

* Added set_input_permutation() to Roll infer function.

* Optimize imports.

* Code refactoring.

* Code refactoring.

* Removed unnecessary variable.

* Added name to the second reshape.

* Added rename_nodes to set original op name be at second reshape.

* Small fix.
2021-04-16 13:37:27 +03:00
Sergey Lyubimtsev
8013d2f7ba MO cannot convert Caffe model from PyPI "openvino-dev" distribution (#5239)
* MO cannot convert Caffe model from PyPI "openvino-dev" distribution

* use os.pardir instead of '..'

* remove mo subdir
2021-04-16 13:35:25 +03:00
iliya mironov
efeff1ee3e Add keep split output ports without consumers (#5136)
* Add keep split output ports without consumers

* Fix ir reader for split outputs

* Update unit tests

* Refactoring code according to review

* Fix unit test

* Fix
2021-04-12 17:49:53 +03:00
Anastasia Popova
cadff031d5 Fixed framework name attribute in mapping file. (#5046)
* Fixed framework name attribute for onnx, mxnet.

* Fixed framework name attribute for caffe.

* Removed unnecessary attribute setting from add_opoutput()

* Added identity nodes adding to outputs in mxnet loader.

* Removed unnecessary reformat.

* Removed unnecessary reformat.

* Added check for empty name.

* Used nodes indices instead of node names in loader.

* Code refactoring, small bug fixed.
2021-04-08 20:59:44 +03:00
Yegor Kruglov
ca889f530d [ MO ] Warnings fix (#5105)
* warnings fix

* remove redundant backslash

* add whitespace in the error message
2021-04-08 14:09:00 +03:00
Svetlana Dolinina
f858a62836 fix bug: assert is keyword, not function (#5131)
* fix bug: assert is keyword, not function

* fix comment
2021-04-08 12:02:18 +03:00
Sergey Lyubimtsev
f438a3a321 Add entry point for model optimizer as 'mo' alias (#5052)
* Add entry point for model optimizer as 'mo' alias

* Add entry point for mo package
2021-04-06 10:41:44 +03:00
Evgeny Lazarev
1d26a5600c Fixed extracting TF models with string constants (#5110)
* Fixed extracting TF models with string constants

* Fixed parsing strings from the TF models
2021-04-06 10:33:40 +03:00
Alexey Suhov
6a16b70e0e setupvars and demos: Fix getting script dir on macOS 10.15 with with default zsh shell (#5094) 2021-04-02 21:37:21 +03:00
Yegor Kruglov
c966aea879 Update StridedSlice extender to handle StridedSlice op with empty attributes (#4980)
* update ss extender

* change the condition

* add an assert

* fix for IR comparator tests failures
2021-04-02 09:40:46 +03:00
Evgenya Stepyreva
8d2f58caad numpy.packbits usage for numpy 1.14.0 support (#5078) 2021-04-01 20:11:12 +03:00
Dmitry Kurtaev
5272bd4ba9 Test MO wheel content (#5054) 2021-04-01 18:03:28 +03:00
Maxim Vafin
2bed9c9277 Fix ScatterND validation and implement evaluate (#4905)
* Fix ScatterND validation and implement evaluate

* Apply review feedback

* Update scatternd.py
2021-03-31 19:01:46 +03:00
Svetlana Dolinina
ffa467a5ad fix batch adding to init value of read value (#4187)
* fix batch adding to init value of read value

* fix for batch in Kaldi models

* added broadcast to be able reshape in IE

* test fixes, added batch broadcasting to created constants

* pep fixes

* move all changes to 1 transformation

* added unit test and fix insertSelect transformation

* added comments

* remove unneeded params search

* fix element_size to send correct batch

* fix update batch in element_size

* couple fixes

* update BOM file

* fix review comments

* review fixes

* review fixes

* fix license headers
2021-03-31 11:32:36 +03:00
Pavel Esir
6477e8ec01 [MO] Fix fp16 in shapeof subgraphs (#4524)
* Initial working solution

* moved bfs_search_apply_on_shapeof_subgraph_nodes from utils/graph.py to MarkShapeOfSubgraphDataType.py

* Reused bfs from MarkSubgraphsWithCorrectLayout.py

* fixed e2e precomit issues: specified correct const data_types, fixed BFS search staring point to avoid nodeless shapeof subgraphs

* fixed mxnet_rnnt: added converting all Const nodes in ShapeOf subgraph in MarkAndChangeDataTypeInShapeOfSubgraphs.py, revised Const values in transformations that affect ShapeOf subgraph nodes

* reverter ReverseV2ToReverseSequence.py and DecomposeBidirectionalRNNSequence.py

* in MarkSubgraphsWithCorrectLayout BFS search beauty applied

* apply review comments, returned back 'in_shape_subgraph' attribute

* graph condition added

* MO IR reader fix for mixed FP16 models, added replacer order placement comment

* moved to back phase

* new solution with marking nodes from bottom to top (WIP)

* successfully tested on back phase

* corrected unittest

* removed check for start nodes size in bfs

* fix transformations that insert f64 to f32 in shape subgraph

* corrected log.warning -> log.debug

* revised list if shape input operations added unittest for Const shape inputs

* applied @lazarevevgeny's comments

* licence head corrections
2021-03-30 23:16:32 +03:00
Alexey Suhov
e072c96237 fix test_missed_intel_header fail in mo\__main__.py (#5047) 2021-03-30 22:46:14 +03:00
Dmitry Kurtaev
47cf3bdc35 Use MO as module (#4801)
* Use MO as module

* Add legal notice
2021-03-30 18:11:58 +03:00
Dmitry Kurtaev
41f0eb51c5 Enable mo.front.common.extractors module (#5018)
* Enable mo.front.common.extractors module

* Update package_BOM.txt
2021-03-30 13:56:49 +03:00
Dmitry Kurtaev
a2a7320d74 [MO] Fix EfficientDet conversion (#5029) 2021-03-30 12:28:18 +03:00
Mikhail Ryzhov
d314a7c350 Added version.txt to MO wheel (#4982) 2021-03-30 10:46:35 +03:00
Alexey Suhov
a2bf62f488 Align copyright notice in sh, bat, cl, cc, pyx source files (#4995) 2021-03-27 01:26:27 +03:00
Alexey Suhov
6478f1742a Align copyright notice in python scripts (CVS-51320) (#4974)
* Align copyright notice in python scripts (CVS-51320)
2021-03-26 17:54:28 +03:00