IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

DOCS shift to rst - Model Optimization Guide articles (#16598)

Browse Source
This commit is contained in:
Sebastian Golebiewski
2023-03-31 11:26:04 +02:00
committed by GitHub
parent bb93bfd90f
commit f9ff518d16
8 changed files with 113 additions and 62 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/img/DEVELOPMENT_FLOW_V3_crunch.svg → docs/_static/images/DEVELOPMENT_FLOW_V3_crunch.svg
Vendored
View File
docs/img/WHAT_TO_USE.svg → docs/_static/images/WHAT_TO_USE.svg
Vendored
View File
docs/img/quantization_picture.svg → docs/_static/images/quantization_picture.svg
Vendored
View File
tools/pot/docs/images/workflow_simple.svg → docs/_static/images/workflow_simple.svg
Vendored
View File
docs/optimization_guide/model_optimization_guide.md
+14 -14
View File
@@ -1,21 +1,17 @@
# Model Optimization Guide {#openvino_docs_model_optimization_guide}
# Model Optimization Guide {#openvino_docs_model_optimization_guide}
@sphinxdirective
.. toctree::
:maxdepth: 1
:hidden:
ptq_introduction
tmo_introduction
(Experimental) Protecting Model <pot_ranger_README>
@endsphinxdirective
Model optimization is an optional offline step of improving final model performance by applying special optimization methods, such as quantization, pruning, preprocessing optimization, etc. OpenVINO provides several tools to optimize models at different steps of model development:
@sphinxdirective
Model optimization is an optional offline step of improving final model performance by applying special optimization methods, such as quantization, pruning, preprocessing optimization, etc. OpenVINO provides several tools to optimize models at different steps of model development:
- :doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>` implements most of the optimization parameters to a model by default. Yet, you are free to configure mean/scale values, batch size, RGB vs BGR input channels, and other parameters to speed up preprocess of a model (:doc:`Embedding Preprocessing Computation <openvino_docs_MO_DG_Additional_Optimization_Use_Cases>`).
@@ -23,25 +19,29 @@
- :doc:`Training-time Optimization <nncf_ptq_introduction>`, a suite of advanced methods for training-time model optimization within the DL framework, such as PyTorch and TensorFlow 2.x. It supports methods, like Quantization-aware Training and Filter Pruning. NNCF-optimized models can be inferred with OpenVINO using all the available workflows.
@endsphinxdirective
Detailed workflow:
##################
## Detailed workflow:
To understand which development optimization tool you need, refer to the diagram:
![](../img/DEVELOPMENT_FLOW_V3_crunch.svg)
.. image:: _static/images/DEVELOPMENT_FLOW_V3_crunch.svg
Post-training methods are limited in terms of achievable accuracy-performance trade-off for optimizing models. In this case, training-time optimization with NNCF is an option.
Once the model is optimized using the aforementioned tools it can be used for inference using the regular OpenVINO inference workflow. No changes to the inference code are required.
![](../img/WHAT_TO_USE.svg)
.. image:: _static/images/WHAT_TO_USE.svg
Post-training methods are limited in terms of achievable accuracy, which may degrade for certain scenarios. In such cases, training-time optimization with NNCF may give better results.
Once the model has been optimized using the aforementioned tools, it can be used for inference using the regular OpenVINO inference workflow. No changes to the code are required.
If you are not familiar with model optimization methods, refer to [post-training methods](@ref pot_introduction).
If you are not familiar with model optimization methods, refer to :doc:`post-training methods <pot_introduction>`.
## Additional Resources
- [Deployment optimization](./dldt_deployment_optimization_guide.md)
Additional Resources
####################
- :doc:`Deployment optimization <openvino_docs_deployment_optimization_guide_dldt_optimization_guide>`
@endsphinxdirective
docs/optimization_guide/ptq_introduction.md
+9 -5
View File
@@ -9,23 +9,27 @@
pot_introduction
nncf_ptq_introduction
@endsphinxdirective
Post-training model optimization is the process of applying special methods that transform the model into a more hardware-friendly representation without retraining or fine-tuning. The most popular and widely-spread method here is 8-bit post-training quantization because it is:
* It is easy-to-use.
* It does not hurt accuracy a lot.
* It provides significant performance improvement.
* It suites many hardware available in stock since most of them support 8-bit computation natively.
8-bit integer quantization lowers the precision of weights and activations to 8 bits, which leads to almost 4x reduction in the model footprint and significant improvements in inference speed, mostly due to lower throughput required for the inference. This lowering step is done offline, before the actual inference, so that the model gets transformed into the quantized representation. The process does not require a training dataset or a training pipeline in the source DL framework.
8-bit integer quantization lowers the precision of weights and activations to 8 bits, which leads to almost 4x reduction in the model footprint and significant improvements in inference speed, mostly due to lower throughput required for the inference. This lowering step is done offline, before the actual inference, so that the model gets transformed into the quantized representation. The process does not require a training dataset or a training pipeline in the source DL framework.
![](../img/quantization_picture.svg)
.. image:: _static/images/quantization_picture.svg
To apply post-training methods in OpenVINO, you need:
* A floating-point precision model, FP32 or FP16, converted into the OpenVINO Intermediate Representation (IR) format that can be run on CPU.
* A representative calibration dataset, representing a use case scenario, for example, of 300 samples.
* In case of accuracy constraints, a validation dataset and accuracy metrics should be available.
Currently, OpenVINO provides two workflows with post-training quantization capabilities:
* [Post-training Quantization with POT](@ref pot_introduction) - works with models in OpenVINO Intermediate Representation (IR) only.
* [Post-training Quantization with NNCF](@ref nncf_ptq_introduction) - cross-framework solution for model optimization that provides a new simple API for post-training quantization.
* :doc:`Post-training Quantization with POT <pot_introduction>` - works with models in OpenVINO Intermediate Representation (IR) only.
* :doc:`Post-training Quantization with NNCF <nncf_ptq_introduction>` - cross-framework solution for model optimization that provides a new simple API for post-training quantization.
@endsphinxdirective
tools/pot/docs/DefaultQuantizationUsage.md
+10 -2
View File
@@ -1,5 +1,6 @@
# Quantizing Models {#pot_default_quantization_usage}
@sphinxdirective
.. toctree::
@@ -9,7 +10,10 @@
DefaultQuantization Method <pot_compression_algorithms_quantization_default_README>
This guide describes how to apply model quantization with the Default Quantization method without accuracy control, using an unannotated dataset. To use this method, create a Python script using an API of Post-Training Optimization Tool (POT) and implement data preparation logic and quantization pipeline. If you are not familiar with Python, try :doc:`command-line interface <pot_compression_cli_README>` of POT which is designed to quantize models from OpenVINO `Model Zoo <https://github.com/openvinotoolkit/open_model_zoo>`__. The figure below shows the common workflow of the quantization script implemented with POT API.
This guide describes how to apply model quantization with the Default Quantization method without accuracy control, using an unannotated dataset.
To use this method, create a Python script using an API of Post-Training Optimization Tool (POT) and implement data preparation logic and quantization pipeline.
If you are not familiar with Python, try :doc:`command-line interface <pot_compression_cli_README>` of POT which is designed to quantize models from
OpenVINO `Model Zoo <https://github.com/openvinotoolkit/open_model_zoo>`__. The figure below shows the common workflow of the quantization script implemented with POT API.
.. image:: _static/images/default_quantization_flow.svg
@@ -26,10 +30,13 @@ In most cases, it is required to implement only the ``openvino.tools.pot.DataLoa
* The ``__len__()``, returns the size of the dataset.
* The ``__getitem__()``, provides access to the data by index in range of 0 to ``len(self)``. It can also encapsulate the logic of model-specific pre-processing. This method should return data in the ``(data, annotation)`` format, in which:
* The ``data`` is the input that is passed to the model at inference so that it should be properly preprocessed. It can be either the ``numpy.array`` object or a dictionary, where the key is the name of the model input and value is ``numpy.array`` which corresponds to this input.
* The ``annotation`` is not used by the Default Quantization method. Therefore, this object can be ``None`` in this case.
Framework data loading classes can be wrapped by the ``openvino.tools.pot.DataLoader`` interface which is usually straightforward. For example, the ``torch.utils.data.Dataset`` has a similar interface as the ``openvino.tools.pot.DataLoader``, so that its TorchVision implementations can be easily wrapped by POT API.
Framework data loading classes can be wrapped by the ``openvino.tools.pot.DataLoader`` interface which is usually straightforward.
For example, the ``torch.utils.data.Dataset`` has a similar interface as the ``openvino.tools.pot.DataLoader``,
so that its TorchVision implementations can be easily wrapped by POT API.
.. note::
@@ -180,4 +187,5 @@ Examples
* :doc:`Quantization of Face Detection model <pot_example_face_detection_README>`
* :doc:`Quantization of speech model for GNA device <pot_example_speech_README>`
@endsphinxdirective
tools/pot/docs/Introduction.md
+80 -41
View File
@@ -14,46 +14,85 @@
Examples <pot_examples_description>
pot_docs_FrequentlyAskedQuestions
For the needs of post-training optimization, OpenVINO&trade; provides a **Post-training Optimization Tool (POT)**
which supports the **uniform integer quantization** method. This method allows moving from floating-point precision
to integer precision (for example, 8-bit) for weights and activations during the inference time. It helps to reduce
the model size, memory footprint and latency, as well as improve the computational efficiency, using integer arithmetic.
During the quantization process the model undergoes the transformation process when additional operations, that contain
quantization information, are inserted into the model. The actual transition to integer arithmetic happens at model inference.
The post-training quantization algorithm takes samples from the representative dataset, inputs them into the network,
and calibrates the network based on the resulting weights and activation values. Once calibration is complete,
values in the network are converted to 8-bit integer format.
While post-training quantization makes your model run faster and take less memory, it may cause a slight reduction
in accuracy. If you performed post-training quantization on your model and find that it isn’t accurate enough,
try using :doc:`Quantization-aware Training <qat_introduction>` to increase its accuracy.
| **Post-Training Quantization Quick Start Examples:**
| Try out these interactive Jupyter Notebook examples to learn the POT API and see post-training quantization in action:
* `Quantization of Image Classification Models with POT <https://docs.openvino.ai/nightly/notebooks/113-image-classification-quantization-with-output.html>`__.
* `Object Detection Quantization with POT <https://docs.openvino.ai/nightly/notebooks/111-detection-quantization-with-output.html>`__.
Quantizing Models with POT
#######################################
The figure below shows the post-training quantization workflow with POT. In a typical workflow, a pre-trained
model is converted to OpenVINO IR format using Model Optimizer. Then, the model is quantized with a representative dataset using POT.
.. image:: _static/images/workflow_simple.svg
:alt: OVMS Benchmark Setup Diagram
Post-training Quantization Methods
+++++++++++++++++++++++++++++++++++++++
Depending on your needs and requirements, POT provides two quantization methods that can be used:
Default Quantization and Accuracy-aware Quantization.
Default Quantization
---------------------------------------
Default Quantization uses an unannotated dataset to perform quantization. It uses representative
dataset items to estimate the range of activation values in a network and then quantizes the network.
This method is recommended to start with, because it results in a fast and accurate model in most cases.
To quantize your model with Default Quantization, see the :doc:`Quantizing Models <pot_default_quantization_usage>` page.
Accuracy-aware Quantization
---------------------------------------
Accuracy-aware Quantization is an advanced method that maintains model accuracy within a predefined
range by leaving some network layers unquantized. It uses a trade-off between speed and accuracy to meet
user-specified requirements. This method requires an annotated dataset and may require more time for quantization.
To quantize your model with Accuracy-aware Quantization, see the :doc:`Quantizing Models with Accuracy Control <pot_accuracyaware_usage>` page.
Quantization Best Practices and FAQs
+++++++++++++++++++++++++++++++++++++++
If you quantized your model and it isn’t accurate enough, visit the :doc:`Quantization Best Practices <pot_docs_BestPractices>`
page for tips on improving quantized performance. Sometimes, older Intel CPU generations can encounter a saturation issue when
running quantized models that can cause reduced accuracy: learn more on the :doc:`Saturation Issue Workaround <pot_saturation_issue>` page.
Have more questions about post-training quantization or encountering errors using POT? Visit the
:doc:`POT FAQ <pot_docs_FrequentlyAskedQuestions>` page for answers to frequently asked questions and solutions to common errors.
Additional Resources
#######################################
* :doc:`Post-training Quantization Examples <pot_examples_description>`
* :doc:`Quantization Best Practices <pot_docs_BestPractices>`
* :doc:`Post-training Optimization Tool FAQ <pot_docs_FrequentlyAskedQuestions>`
* :doc:`Performance Benchmarks <openvino_docs_performance_benchmarks>`
@endsphinxdirective
For the needs of post-training optimization, OpenVINO&trade; provides a **Post-training Optimization Tool (POT)** which supports the **uniform integer quantization** method. This method allows moving from floating-point precision to integer precision (for example, 8-bit) for weights and activations during the inference time. It helps to reduce the model size, memory footprint and latency, as well as improve the computational efficiency, using integer arithmetic. During the quantization process the model undergoes the transformation process when additional operations, that contain quantization information, are inserted into the model. The actual transition to integer arithmetic happens at model inference.
The post-training quantization algorithm takes samples from the representative dataset, inputs them into the network, and calibrates the network based on the resulting weights and activation values. Once calibration is complete, values in the network are converted to 8-bit integer format.
While post-training quantization makes your model run faster and take less memory, it may cause a slight reduction in accuracy. If you performed post-training quantization on your model and find that it isn’t accurate enough, try using [Quantization-aware Training](@ref qat_introduction) to increase its accuracy.
### Post-Training Quantization Quick Start Examples
Try out these interactive Jupyter Notebook examples to learn the POT API and see post-training quantization in action:
* [Quantization of Image Classification Models with POT](https://docs.openvino.ai/latest/notebooks/113-image-classification-quantization-with-output.html).
* [Object Detection Quantization with POT](https://docs.openvino.ai/latest/notebooks/111-detection-quantization-with-output.html).
## Quantizing Models with POT
The figure below shows the post-training quantization workflow with POT. In a typical workflow, a pre-trained model is converted to OpenVINO IR format using Model Optimizer. Then, the model is quantized with a representative dataset using POT.
![](./images/workflow_simple.svg)
### Post-training Quantization Methods
Depending on your needs and requirements, POT provides two quantization methods that can be used: Default Quantization and Accuracy-aware Quantization.
#### Default Quantization
Default Quantization uses an unannotated dataset to perform quantization. It uses representative dataset items to estimate the range of activation values in a network and then quantizes the network. This method is recommended to start with, because it results in a fast and accurate model in most cases. To quantize your model with Default Quantization, see the [Quantizing Models](@ref pot_default_quantization_usage) page.
#### Accuracy-aware Quantization
Accuracy-aware Quantization is an advanced method that maintains model accuracy within a predefined range by leaving some network layers unquantized. It uses a trade-off between speed and accuracy to meet user-specified requirements. This method requires an annotated dataset and may require more time for quantization. To quantize your model with Accuracy-aware Quantization, see the [Quantizing Models with Accuracy Control](@ref pot_accuracyaware_usage) page.
### Quantization Best Practices and FAQs
If you quantized your model and it isn’t accurate enough, visit the [Quantization Best Practices](@ref pot_docs_BestPractices) page for tips on improving quantized performance. Sometimes, older Intel CPU generations can encounter a saturation issue when running quantized models that can cause reduced accuracy: learn more on the [Saturation Issue Workaround](@ref pot_saturation_issue) page.
Have more questions about post-training quantization or encountering errors using POT? Visit the [POT FAQ](@ref pot_docs_FrequentlyAskedQuestions) page for answers to frequently asked questions and solutions to common errors.
## Additional Resources
* [Post-training Quantization Examples](@ref pot_examples_description)
* [Quantization Best Practices](@ref pot_docs_BestPractices)
* [Post-training Optimization Tool FAQ](@ref pot_docs_FrequentlyAskedQuestions)
* [Performance Benchmarks](@ref openvino_docs_performance_benchmarks_openvino)