compress_to_fp16 parameter in MO. (#13832)
* Added convert_to_fp16 param in MO. * Deprecated data_type. * Renamed param, made value optional. * Docs update. * Docs update. * Docs update. * Changed param description. * Conflict fixed. * Apply suggestions from code review Co-authored-by: Sergey Lyalin <sergey.lyalin@intel.com> * Param description correction. * Corrected docs. * Apply suggestions from code review Co-authored-by: Pavel Esir <pavel.esir@gmail.com> * Fixed docs. * Small correction. * Update docs/MO_DG/prepare_model/FP16_Compression.md Co-authored-by: Tatiana Savina <tatiana.savina@intel.com> * Update docs/MO_DG/prepare_model/FP16_Compression.md Co-authored-by: Tatiana Savina <tatiana.savina@intel.com> * Update docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md Co-authored-by: Tatiana Savina <tatiana.savina@intel.com> * Update tools/mo/openvino/tools/mo/utils/cli_parser.py Co-authored-by: Tatiana Savina <tatiana.savina@intel.com> * Update docs/MO_DG/prepare_model/Model_Optimizer_FAQ.md Co-authored-by: Tatiana Savina <tatiana.savina@intel.com> * Update docs/get_started/get_started_demos.md Co-authored-by: Tatiana Savina <tatiana.savina@intel.com> * Update docs/MO_DG/prepare_model/Model_Optimizer_FAQ.md Co-authored-by: Tatiana Savina <tatiana.savina@intel.com> Co-authored-by: Sergey Lyalin <sergey.lyalin@intel.com> Co-authored-by: Pavel Esir <pavel.esir@gmail.com> Co-authored-by: Andrei Kochin <andrei.kochin@intel.com> Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>
This commit is contained in:
Anastasiia Pnevskaia
co-authored by
Sergey Lyalin
Pavel Esir
Tatiana Savina
Andrei Kochin
Sergey Lyalin
Pavel Esir
Tatiana Savina
Andrei Kochin
parent
558714ccdd
commit
80b9a5eeae
@@ -60,7 +60,7 @@ You can also insert additional input pre-processing sub-graphs into the converte
|
||||
the `--mean_values`, `scales_values`, `--layout`, and other parameters described
|
||||
in the [Embedding Preprocessing Computation](prepare_model/Additional_Optimizations.md) article.
|
||||
|
||||
The `--data_type` compression parameter in Model Optimizer allows generating IR of the `FP16` data type. For more details, refer to the [Compression of a Model to FP16](prepare_model/FP16_Compression.md) guide.
|
||||
The `--compress_to_fp16` compression parameter in Model Optimizer allows generating IR with constants (for example, weights for convolutions and matrix multiplications) compressed to `FP16` data type. For more details, refer to the [Compression of a Model to FP16](prepare_model/FP16_Compression.md) guide.
|
||||
|
||||
To get the full list of conversion parameters available in Model Optimizer, run the following command:
|
||||
|
||||
|
||||
@@ -1,20 +1,19 @@
|
||||
# Compressing a Model to FP16 {#openvino_docs_MO_DG_FP16_Compression}
|
||||
|
||||
Model Optimizer can convert all floating-point weights to `FP16` data type. The resulting IR is called
|
||||
compressed `FP16` model.
|
||||
compressed `FP16` model. The resulting model will occupy about twice as less space in the file system,
|
||||
but it may have some accuracy drop. For most models, the accuracy drop is negligible.
|
||||
|
||||
To compress the model, use the `--data_type` option:
|
||||
To compress the model, use the `--compress_to_fp16` option:
|
||||
> **NOTE**: Starting from the 2022.3 release, option --data_type is deprecated.
|
||||
> Instead of --data_type FP16 use --compress_to_fp16.
|
||||
> Using `--data_type FP32` will give no result and will not force `FP32` precision in
|
||||
> the model. If the model has `FP16` constants, such constants will have `FP16` precision in IR as well.
|
||||
|
||||
```
|
||||
mo --input_model INPUT_MODEL --data_type FP16
|
||||
mo --input_model INPUT_MODEL --compress_to_fp16
|
||||
```
|
||||
|
||||
> **NOTE**: Using `--data_type FP32` will give no result and will not force `FP32`
|
||||
> precision in the model. If the model was `FP16`, it will have `FP16` precision in IR as well.
|
||||
|
||||
The resulting model will occupy about twice as less space in the file system, but it may have some accuracy drop.
|
||||
The resulting model will occupy about half of the previous space in the file system, but lose some of its accuracy.
|
||||
For most models, the accuracy drop is negligible.
|
||||
For details on how plugins handle compressed `FP16` models, see [Working with devices](../../OV_Runtime_UG/supported_plugins/Device_Plugins.md).
|
||||
|
||||
> **NOTE**: `FP16` compression is sometimes used as the initial step for `INT8` quantization.
|
||||
|
||||
@@ -321,7 +321,7 @@ However, if your model contains more than one input, Model Optimizer is able to
|
||||
|
||||
#### Q49. What does the message "Data type is unsupported" mean? <a name="question-49"></a>
|
||||
|
||||
**A** : Model Optimizer cannot convert the model to the specified data type. Currently, FP16 and FP32 are supported. Make sure you specify the data type with the `--data_type` flag. The available values are: FP16, FP32, half, float.
|
||||
**A** : Model Optimizer cannot read the value with the specified data type. Currently, the following types are supported: bool, float16, float32, double, int8, int16, int32, int64, uint8, uint16, uint32, uint64, str.
|
||||
|
||||
#### Q50. What does the message "No node with name ..." mean? <a name="question-50"></a>
|
||||
|
||||
@@ -440,11 +440,11 @@ Keep in mind that there is no space between and inside the brackets for input sh
|
||||
|
||||
#### Q75. What does the message "... elements of ... were clipped to infinity while converting a blob for node [...] to ..." mean? <a name="question-75"></a>
|
||||
|
||||
**A** : This message may appear when the `--data_type=FP16` command-line option is used. This option implies conversion of all the blobs in the node to FP16. If a value in a blob is out of the range of valid FP16 values, the value is converted to positive or negative infinity. It may lead to incorrect results of inference or may not be a problem, depending on the model. The number of such elements and the total number of elements in the blob is printed out together with the name of the node, where this blob is used.
|
||||
**A** : This message may appear when the `--compress_to_fp16` (or deprecated `--data_type`) command-line option is used. This option implies compression of all the model weights, biases, and other constant values to FP16. If a value of a constant is out of the range of valid FP16 values, the value is converted to positive or negative infinity. It may lead to incorrect results of inference or may not be a problem, depending on the model. The number of such elements and the total number of elements in the constant value is printed out together with the name of the node, where this value is used.
|
||||
|
||||
#### Q76. What does the message "... elements of ... were clipped to zero while converting a blob for node [...] to ..." mean? <a name="question-76"></a>
|
||||
|
||||
**A** : This message may appear when the `--data_type=FP16` command-line option is used. This option implies conversion of all blobs in the mode to FP16. If a value in the blob is so close to zero that it cannot be represented as a valid FP16 value, it is converted to a true zero FP16 value. Depending on the model, it may lead to incorrect results of inference or may not be a problem. The number of such elements and the total number of elements in the blob are printed out together with a name of the node, where this blob is used.
|
||||
**A** : This message may appear when the `--compress_to_fp16` (or deprecated `--data_type`) command-line option is used. This option implies conversion of all blobs in the mode to FP16. If a value in the blob is so close to zero that it cannot be represented as a valid FP16 value, it is converted to a true zero FP16 value. Depending on the model, it may lead to incorrect results of inference or may not be a problem. The number of such elements and the total number of elements in the blob are printed out together with a name of the node, where this blob is used.
|
||||
|
||||
#### Q77. What does the message "The amount of nodes matched pattern ... is not equal to 1" mean? <a name="question-77"></a>
|
||||
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@ To convert this model to the TensorFlow format, follow the [Reproduce Keras to T
|
||||
|
||||
After converting the model to TensorFlow format, run the Model Optimizer command below:
|
||||
```sh
|
||||
mo --input "input_1[1 1333 1333 3]" --input_model retinanet_resnet50_coco_best_v2.1.0.pb --data_type FP32 --transformations_config front/tf/retinanet.json
|
||||
mo --input "input_1[1 1333 1333 3]" --input_model retinanet_resnet50_coco_best_v2.1.0.pb --transformations_config front/tf/retinanet.json
|
||||
```
|
||||
|
||||
Where `transformations_config` command-line parameter specifies the configuration json file containing model conversion hints for the Model Optimizer.
|
||||
|
||||
@@ -249,12 +249,12 @@ Create an `<ir_dir>` directory to contain the model's Intermediate Representatio
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
The OpenVINO Runtime can infer models where floating-point weights are [compressed to FP16](../MO_DG/prepare_model/FP16_Compression.md). To generate an IR with a specific precision, run the Model Optimizer with the appropriate `--data_type` option.
|
||||
To save disk space for your IR file, you can apply [weights compression to FP16](../MO_DG/prepare_model/FP16_Compression.md). To generate an IR with FP16 weights, run the Model Optimizer with the `--compress_to_fp16` option.
|
||||
|
||||
Generic Model Optimizer script:
|
||||
|
||||
``` sh
|
||||
mo --input_model <model_dir>/<model_file> --data_type <model_precision> --output_dir <ir_dir>
|
||||
mo --input_model <model_dir>/<model_file>
|
||||
```
|
||||
|
||||
IR files produced by the script are written to the <ir_dir> directory.
|
||||
@@ -267,19 +267,19 @@ The command with most placeholders filled in and FP16 precision:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
mo --input_model ~/models/public/googlenet-v1/googlenet-v1.caffemodel --data_type FP16 --output_dir ~/ir
|
||||
mo --input_model ~/models/public/googlenet-v1/googlenet-v1.caffemodel --compress_to_fp16 --output_dir ~/ir
|
||||
|
||||
.. tab:: Windows
|
||||
|
||||
.. code-block:: bat
|
||||
|
||||
mo --input_model %USERPROFILE%\Documents\models\public\googlenet-v1\googlenet-v1.caffemodel --data_type FP16 --output_dir %USERPROFILE%\Documents\ir
|
||||
mo --input_model %USERPROFILE%\Documents\models\public\googlenet-v1\googlenet-v1.caffemodel --compress_to_fp16 --output_dir %USERPROFILE%\Documents\ir
|
||||
|
||||
.. tab:: macOS
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
mo --input_model ~/models/public/googlenet-v1/googlenet-v1.caffemodel --data_type FP16 --output_dir ~/ir
|
||||
mo --input_model ~/models/public/googlenet-v1/googlenet-v1.caffemodel --compress_to_fp16 --output_dir ~/ir
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
|
||||
@@ -218,7 +218,8 @@ def arguments_post_parsing(argv: argparse.Namespace):
|
||||
log.error(e)
|
||||
raise_ie_not_found()
|
||||
|
||||
if 'data_type' in argv and argv.data_type in ['FP16', 'half']:
|
||||
if ('data_type' in argv and argv.data_type in ['FP16', 'half']) or \
|
||||
('compress_to_fp16' in argv and argv.compress_to_fp16 is True):
|
||||
argv.data_type = 'FP32'
|
||||
argv.compress_fp16 = True
|
||||
else:
|
||||
|
||||
@@ -557,10 +557,9 @@ mo_convert_params = {
|
||||
'ModelOptimizer to change layout, for example: '
|
||||
'--layout "name1(nhwc->nchw),name2(cn->nc)". Also "*" in long layout form can be'
|
||||
' used to fuse dimensions, for example "[n,c,...]->[n*c,...]".', '', '', layout_param_to_str),
|
||||
'data_type': ParamDescription(
|
||||
'Data type for all intermediate tensors and weights. ' +
|
||||
'If original model is in FP32 and --data_type=FP16 is specified, all model weights ' +
|
||||
'and biases are compressed to FP16.', '', '', None),
|
||||
'compress_to_fp16': ParamDescription(
|
||||
'If the original model has FP32 weights or biases, they are compressed to FP16. '
|
||||
'All intermediate data is kept in original precision.', '', '', None),
|
||||
'transform': ParamDescription(
|
||||
'Apply additional transformations. {}' +
|
||||
'"--transform transformation_name1[args],transformation_name2..." ' +
|
||||
@@ -1011,9 +1010,19 @@ def get_common_cli_parser(parser: argparse.ArgumentParser = None):
|
||||
default=())
|
||||
# TODO: isn't it a weights precision type
|
||||
common_group.add_argument('--data_type',
|
||||
help=mo_convert_params_common['data_type'].description,
|
||||
help='[DEPRECATED] Data type for model weights and biases. '
|
||||
'If original model has FP32 weights or biases and --data_type=FP16 is specified, '
|
||||
'FP32 model weights and biases are compressed to FP16. '
|
||||
'All intermediate data is kept in original precision.',
|
||||
choices=["FP16", "FP32", "half", "float"],
|
||||
default='float')
|
||||
default='float',
|
||||
action=DeprecatedOptionCommon)
|
||||
common_group.add_argument('--compress_to_fp16',
|
||||
help=mo_convert_params_common['compress_to_fp16'].description,
|
||||
type=check_bool,
|
||||
nargs="?",
|
||||
const=True,
|
||||
default=False)
|
||||
common_group.add_argument('--transform',
|
||||
help=mo_convert_params_common['transform'].description.format(
|
||||
mo_convert_params_common['transform'].possible_types_command_line),
|
||||
|
||||
Reference in New Issue
Block a user