IntenseWebs/openvino
3
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

DOCS Update optimization docs with NNCF PTQ changes and deprecation of POT (#17398)

Browse Source 
* Update model_optimization_guide.md

* Update model_optimization_guide.md

* Update model_optimization_guide.md

* Update model_optimization_guide.md

* Update model_optimization_guide.md

* Update model_optimization_guide.md

* Update model_optimization_guide.md

* Update home.rst

* Update ptq_introduction.md

* Update Introduction.md

* Update Introduction.md

* Update Introduction.md

* Update ptq_introduction.md

* Update ptq_introduction.md

* Update basic_quantization_flow.md

* Update basic_quantization_flow.md

* Update basic_quantization_flow.md

* Update quantization_w_accuracy_control.md

* Update quantization_w_accuracy_control.md

* Update quantization_w_accuracy_control.md

* Update quantization_w_accuracy_control.md

* Update quantization_w_accuracy_control.md

* Update quantization_w_accuracy_control.md

* Update quantization_w_accuracy_control.md

* Update quantization_w_accuracy_control.md

* Update quantization_w_accuracy_control.md

* Update basic_quantization_flow.md

* Update basic_quantization_flow.md

* Update quantization_w_accuracy_control.md

* Update basic_quantization_flow.md

* Update basic_quantization_flow.md

* Update model_optimization_guide.md

* Update ptq_introduction.md

* Update quantization_w_accuracy_control.md

* Update model_optimization_guide.md

* Update quantization_w_accuracy_control.md

* Update model_optimization_guide.md

* Update quantization_w_accuracy_control.md

* Update model_optimization_guide.md

* Update Introduction.md

* Update basic_quantization_flow.md

* Update basic_quantization_flow.md

* Update quantization_w_accuracy_control.md

* Update ptq_introduction.md

* Update Introduction.md

* Update model_optimization_guide.md

* Update basic_quantization_flow.md

* Update quantization_w_accuracy_control.md

* Update quantization_w_accuracy_control.md

* Update quantization_w_accuracy_control.md

* Update Introduction.md

* Update FrequentlyAskedQuestions.md

* Update model_optimization_guide.md

* Update Introduction.md

* Update model_optimization_guide.md

* Update model_optimization_guide.md

* Update model_optimization_guide.md

* Update model_optimization_guide.md

* Update model_optimization_guide.md

* Update ptq_introduction.md

* Update ptq_introduction.md

* added code snippet (#1)

* Update basic_quantization_flow.md

* Update basic_quantization_flow.md

* Update quantization_w_accuracy_control.md

* Update basic_quantization_flow.md

* Update basic_quantization_flow.md

* Update ptq_introduction.md

* Update model_optimization_guide.md

* Update basic_quantization_flow.md

* Update ptq_introduction.md

* Update quantization_w_accuracy_control.md

* Update basic_quantization_flow.md

* Update basic_quantization_flow.md

* Update basic_quantization_flow.md

* Update ptq_introduction.md

* Update ptq_introduction.md

* Delete ptq_introduction.md

* Update FrequentlyAskedQuestions.md

* Update Introduction.md

* Update quantization_w_accuracy_control.md

* Update introduction.md

* Update basic_quantization_flow.md code blocks

* Update quantization_w_accuracy_control.md code snippets

* Update docs/optimization_guide/nncf/ptq/code/ptq_torch.py

Co-authored-by: Alexander Suslov <alexander.suslov@intel.com>

* Update model_optimization_guide.md

* Optimization docs proofreading  (#2)

* images updated

* delete reminder

* review

* text review

* change images to original ones

* Update filter_pruning.md code blocks

* Update basic_quantization_flow.md

* Update quantization_w_accuracy_control.md

* Update images (#3)

* images updated

* delete reminder

* review

* text review

* change images to original ones

* Update filter_pruning.md code blocks

* update images

* resolve conflicts

* resolve conflicts

* change images to original ones

* resolve conflicts

* update images

* fix conflicts

* Update model_optimization_guide.md

* Update docs/optimization_guide/nncf/ptq/code/ptq_tensorflow.py

Co-authored-by: Alexander Suslov <alexander.suslov@intel.com>

* Update docs/optimization_guide/nncf/ptq/code/ptq_torch.py

Co-authored-by: Alexander Suslov <alexander.suslov@intel.com>

* Update docs/optimization_guide/nncf/ptq/code/ptq_onnx.py

Co-authored-by: Alexander Suslov <alexander.suslov@intel.com>

* Update docs/optimization_guide/nncf/ptq/code/ptq_aa_openvino.py

Co-authored-by: Alexander Suslov <alexander.suslov@intel.com>

* Update docs/optimization_guide/nncf/ptq/code/ptq_openvino.py

Co-authored-by: Alexander Suslov <alexander.suslov@intel.com>

* table format fix

* Update headers

* Update qat.md code blocks

---------

Co-authored-by: Alexander Suslov <alexander.suslov@intel.com>
Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>
This commit is contained in:
Maksim ProshinAlexander SuslovTatiana Savina
 authored and GitHub committed 2023-05-19 14:36:22 +00:00
1 parent 8b215ca363
commit 2513db5a04
36 files changed
+670 -473
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/optimization_guide/model_optimization_guide.md
+9 -19
View File
@@ -8,40 +8,30 @@
ptq_introduction
tmo_introduction
(Experimental) Protecting Model <pot_ranger_README>
Model optimization is an optional offline step of improving final model performance by applying special optimization methods, such as quantization, pruning, preprocessing optimization, etc. OpenVINO provides several tools to optimize models at different steps of model development:
Model optimization is an optional offline step of improving the final model performance and reducing the model size by applying special optimization methods, such as 8-bit quantization, pruning, etc. OpenVINO offers two optimization paths implemented in `Neural Network Compression Framework (NNCF) <https://github.com/openvinotoolkit/nncf>`__:
- :doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>` implements most of the optimization parameters to a model by default. Yet, you are free to configure mean/scale values, batch size, RGB vs BGR input channels, and other parameters to speed up preprocess of a model (:doc:`Embedding Preprocessing Computation <openvino_docs_MO_DG_Additional_Optimization_Use_Cases>`).
- :doc:`Post-training Quantization <ptq_introduction>` is designed to optimize the inference of deep learning models by applying the post-training 8-bit integer quantization that does not require model retraining or fine-tuning.
- :doc:`Post-training Quantization <pot_introduction>` is designed to optimize inference of deep learning models by applying post-training methods that do not require model retraining or fine-tuning, for example, post-training 8-bit integer quantization.
- :doc:`Training-time Optimization <tmo_introduction>`, a suite of advanced methods for training-time model optimization within the DL framework, such as PyTorch and TensorFlow 2.x. It supports methods like Quantization-aware Training, Structured and Unstructured Pruning, etc.
- :doc:`Training-time Optimization <nncf_ptq_introduction>`, a suite of advanced methods for training-time model optimization within the DL framework, such as PyTorch and TensorFlow 2.x. It supports methods, like Quantization-aware Training and Filter Pruning. NNCF-optimized models can be inferred with OpenVINO using all the available workflows.
.. note:: OpenVINO also supports optimized models (for example, quantized) from source frameworks such as PyTorch, TensorFlow, and ONNX (in Q/DQ format). No special steps are required in this case and optimized models can be converted to the OpenVINO Intermediate Representation format (IR) right away.
Post-training Quantization is the fastest way to optimize a model and should be applied first, but it is limited in terms of achievable accuracy-performance trade-off. In case of poor accuracy or performance after Post-training Quantization, Training-time Optimization can be used as an option.
Detailed workflow:
##################
To understand which development optimization tool you need, refer to the diagram:
Once the model is optimized using the aforementioned methods, it can be used for inference using the regular OpenVINO inference workflow. No changes to the inference code are required.
.. image:: _static/images/DEVELOPMENT_FLOW_V3_crunch.svg
Post-training methods are limited in terms of achievable accuracy-performance trade-off for optimizing models. In this case, training-time optimization with NNCF is an option.
Once the model is optimized using the aforementioned tools it can be used for inference using the regular OpenVINO inference workflow. No changes to the inference code are required.
.. image:: _static/images/WHAT_TO_USE.svg
Post-training methods are limited in terms of achievable accuracy, which may degrade for certain scenarios. In such cases, training-time optimization with NNCF may give better results.
Once the model has been optimized using the aforementioned tools, it can be used for inference using the regular OpenVINO inference workflow. No changes to the code are required.
If you are not familiar with model optimization methods, refer to :doc:`post-training methods <pot_introduction>`.
Additional Resources
####################
- :doc:`Post-training Quantization <ptq_introduction>`
- :doc:`Training-time Optimization <tmo_introduction>`
- :doc:`Deployment optimization <openvino_docs_deployment_optimization_guide_dldt_optimization_guide>`
- `HuggingFace Optimum Intel <https://huggingface.co/docs/optimum/intel/optimization_ov>`__
@endsphinxdirective
docs/optimization_guide/nncf/filter_pruning.md
+118 -94
View File
@@ -5,15 +5,15 @@
Introduction
####################
Filter pruning is an advanced optimization method which allows reducing computational complexity of the model by removing
redundant or unimportant filters from convolutional operations of the model. This removal is done in two steps:
Filter pruning is an advanced optimization method that allows reducing the computational complexity of the model by removing
redundant or unimportant filters from the convolutional operations of the model. This removal is done in two steps:
1. Unimportant filters are zeroed out by the NNCF optimization with fine-tuning.
2. Zero filters are removed from the model during the export to OpenVINO Intermediate Representation (IR).
Filter Pruning method from the NNCF can be used stand-alone but we usually recommend to stack it with 8-bit quantization for
Filter Pruning method from the NNCF can be used stand-alone but we usually recommend stacking it with 8-bit quantization for
two reasons. First, 8-bit quantization is the best method in terms of achieving the highest accuracy-performance trade-offs so
stacking it with filter pruning can give even better performance results. Second, applying quantization along with filter
pruning does not hurt accuracy a lot since filter pruning removes noisy filters from the model which narrows down values
@@ -37,17 +37,21 @@ Here, we show the basic steps to modify the training script for the model and us
In this step, NNCF-related imports are added in the beginning of the training script:
.. tab:: PyTorch
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [imports]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [imports]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [imports]
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [imports]
2. Create NNCF configuration
++++++++++++++++++++++++++++
@@ -55,26 +59,30 @@ In this step, NNCF-related imports are added in the beginning of the training sc
Here, you should define NNCF configuration which consists of model-related parameters (`"input_info"` section) and parameters
of optimization methods (`"compression"` section).
.. tab:: PyTorch
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [nncf_congig]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [nncf_congig]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [nncf_congig]
Here is a brief description of the required parameters of the Filter Pruning method. For full description refer to the
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [nncf_congig]
Here is a brief description of the required parameters of the Filter Pruning method. For a full description refer to the
`GitHub <https://github.com/openvinotoolkit/nncf/blob/develop/docs/compression_algorithms/Pruning.md>`__ page.
* ``pruning_init`` - initial pruning rate target. For example, value ``0.1`` means that at the begging of training, convolutions that can be pruned will have 10% of their filters set to zero.
* ``pruning_target`` - pruning rate target at the end of the schedule. For example, the value ``0.5`` means that at the epoch with the number of ``num_init_steps + pruning_steps``, convolutions that can be pruned will have 50% of their filters set to zero.
* ``pruning_steps` - the number of epochs during which the pruning rate target is increased from ``pruning_init` to ``pruning_target`` value. We recommend to keep the highest learning rate during this period.
* ``pruning_steps` - the number of epochs during which the pruning rate target is increased from ``pruning_init` to ``pruning_target`` value. We recommend keeping the highest learning rate during this period.
3. Apply optimization methods
@@ -86,39 +94,44 @@ that can be used the same way as the original model. It is worth noting that opt
so that the model undergoes a set of corresponding transformations and can contain additional operations required for the
optimization.
.. tab-set::
.. tab:: PyTorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [wrap_model]
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [wrap_model]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [wrap_model]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [wrap_model]
4. Fine-tune the model
++++++++++++++++++++++
This step assumes that you will apply fine-tuning to the model the same way as it is done for the baseline model. In the case
of Filter Pruning method we recommend using the training schedule and learning rate similar to what was used for the training
of original model.
of the original model.
.. tab-set::
.. tab:: PyTorch
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [tune_model]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [tune_model]
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [tune_model]
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [tune_model]
5. Multi-GPU distributed training
@@ -127,38 +140,43 @@ of original model.
In the case of distributed multi-GPU training (not DataParallel), you should call ``compression_ctrl.distributed()`` before the
fine-tuning that will inform optimization methods to do some adjustments to function in the distributed mode.
.. tab-set::
.. tab:: PyTorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [distributed]
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [distributed]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [distributed]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [distributed]
6. Export quantized model
+++++++++++++++++++++++++
When fine-tuning finishes, the quantized model can be exported to the corresponding format for further inference: ONNX in
the case of PyTorch and frozen graph - for TensorFlow 2.
.. tab-set::
.. tab:: PyTorch
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [export]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [export]
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [export]
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [export]
These were the basic steps to applying the QAT method from the NNCF. However, it is required in some cases to save/load model
@@ -170,37 +188,43 @@ checkpoints during the training. Since NNCF wraps the original model with its ow
To save model checkpoint use the following API:
.. tab-set::
.. tab:: PyTorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [save_checkpoint]
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [save_checkpoint]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [save_checkpoint]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [save_checkpoint]
8. (Optional) Restore from checkpoint
+++++++++++++++++++++++++++++++++++++
To restore the model from checkpoint you should use the following API:
.. tab:: PyTorch
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [load_checkpoint]
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [load_checkpoint]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [load_checkpoint]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [load_checkpoint]
For more details on saving/loading checkpoints in the NNCF, see the following
`documentation <https://github.com/openvinotoolkit/nncf/blob/develop/docs/Usage.md#saving-and-loading-compressed-models>`__.
@@ -208,19 +232,19 @@ For more details on saving/loading checkpoints in the NNCF, see the following
Deploying pruned model
######################
The pruned model requres an extra step that should be done to get performance improvement. This step involves removal of the
zero filters from the model. This is done at the model conversion step using :doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>` tool when model is converted from the framework representation (ONNX, TensorFlow, etc.) to OpenVINO Intermediate Representation.
The pruned model requires an extra step that should be done to get a performance improvement. This step involves the removal of the
zero filters from the model. This is done at the model conversion step using :doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>` tool when the model is converted from the framework representation (ONNX, TensorFlow, etc.) to OpenVINO Intermediate Representation.
* To remove zero filters from the pruned model add the following parameter to the model convertion command: ``--transform=Pruning``
* To remove zero filters from the pruned model add the following parameter to the model conversion command: ``--transform=Pruning``
After that the model can be deployed with OpenVINO in the same way as the baseline model.
After that, the model can be deployed with OpenVINO in the same way as the baseline model.
For more details about model deployment with OpenVINO, see the corresponding :doc:`documentation <openvino_docs_OV_UG_OV_Runtime_User_Guide>`.
Examples
####################
* `PyTorch Image Classiication example <https://github.com/openvinotoolkit/nncf/blob/develop/examples/torch/classification>`__
* `PyTorch Image Classification example <https://github.com/openvinotoolkit/nncf/blob/develop/examples/torch/classification>`__
* `TensorFlow Image Classification example <https://github.com/openvinotoolkit/nncf/tree/develop/examples/tensorflow/classification>`__
docs/optimization_guide/nncf/introduction.md
+5 -5
View File
@@ -13,7 +13,7 @@
Introduction
####################
Training-time model compression improves model performance by applying optimizations (such as quantization) during the training. The training process minimizes the loss associated with the lower-precision optimizations, so it is able to maintain the model’s accuracy while reducing its latency and memory footprint. Generally, training-time model optimization results in better model performance and accuracy than :doc:`post-training optimization <pot_introduction>`, but it can require more effort to set up.
Training-time model compression improves model performance by applying optimizations (such as quantization) during the training. The training process minimizes the loss associated with the lower-precision optimizations, so it is able to maintain the model’s accuracy while reducing its latency and memory footprint. Generally, training-time model optimization results in better model performance and accuracy than :doc:`post-training optimization <ptq_introduction>`, but it can require more effort to set up.
OpenVINO provides the Neural Network Compression Framework (NNCF) tool for implementing compression algorithms on models to improve their performance. NNCF is a Python library that integrates into PyTorch and TensorFlow training pipelines to add training-time compression methods to the pipeline. To apply training-time compression methods with NNCF, you need:
@@ -51,7 +51,7 @@ To install the latest released version via pip manager run the following command
To install with specific frameworks, use the `pip install nncf[extras]` command, where extras is a list of possible extras, for example, `torch`, `tf`, `onnx`.
To install the latest NNCF version from source follow the instruction on `GitHub <https://github.com/openvinotoolkit/nncf#installation>`__.
To install the latest NNCF version from source, follow the instruction on `GitHub <https://github.com/openvinotoolkit/nncf#installation>`__.
.. note::
@@ -86,7 +86,7 @@ Filter pruning algorithms compress models by zeroing out the output filters of c
Experimental methods
--------------------
NNCF also provides state-of-the-art compression techniques that are still in experimental stages of development and are only recommended for expert developers. These include:
NNCF also provides state-of-the-art compression techniques that are still in the experimental stages of development and are only recommended for expert developers. These include:
- Mixed-precision quantization
- Sparsity
@@ -99,14 +99,14 @@ Recommended Workflow
Using compression-aware training requires a training pipeline, an annotated dataset, and compute resources (such as CPUs or GPUs). If you don't already have these set up and available, it can be easier to start post-training quantization to quickly see quantized results. Then you can use compression-aware training if the model isn't accurate enough. We recommend the following workflow for compressing models with NNCF:
1. :doc:`Perform post-training quantization <pot_introduction>` on your model and then compare performance to the original model.
1. :doc:`Perform post-training quantization <ptq_introduction>` on your model and then compare performance to the original model.
2. If the accuracy is too degraded, use :doc:`Quantization-aware Training <qat_introduction>` to increase accuracy while still achieving faster inference time.
3. If the quantized model is still too slow, use :doc:`Filter Pruning <filter_pruning>` to further improve the model’s inference speed.
Additional Resources
####################
- :doc:`Quantizing Models Post-training <pot_introduction>`
- :doc:`Quantizing Models Post-training <ptq_introduction>`
- `NNCF GitHub repository <https://github.com/openvinotoolkit/nncf>`__
- `NNCF FAQ <https://github.com/openvinotoolkit/nncf/blob/develop/docs/FAQ.md>`__
- `Quantization Aware Training with NNCF and PyTorch <notebooks/302-pytorch-quantization-aware-training-with-output.html>`__
docs/optimization_guide/nncf/ptq/basic_quantization_flow.md
+99 -48
View File
@@ -5,10 +5,10 @@
Introduction
####################
The basic quantization flow is the simplest way to apply 8-bit quantization to the model. It is available for models in the following frameworks: PyTorch, TensorFlow 2.x, ONNX, and OpenVINO. The basic quantization flow is based on the following steps:
The basic quantization flow is the simplest way to apply 8-bit quantization to the model. It is available for models in the following frameworks: OpenVINO, PyTorch, TensorFlow 2.x, and ONNX. The basic quantization flow is based on the following steps:
* Set up an environment and install dependencies.
* Prepare the **calibration dataset** that is used to estimate quantization parameters of the activations within the model.
* Prepare a representative **calibration dataset** that is used to estimate quantization parameters of the activations within the model, for example, of 300 samples.
* Call the quantization API to apply 8-bit quantization to the model.
Set up an Environment
@@ -29,78 +29,117 @@ Install all the packages required to instantiate the model object, for example,
Prepare a Calibration Dataset
#############################
At this step, create an instance of the ``nncf.Dataset`` class that represents the calibration dataset. The ``nncf.Dataset`` class can be a wrapper over the framework dataset object that is used for model training or validation. The class constructor receives the dataset object and the transformation function. For example, if you use PyTorch, you can pass an instance of the ``torch.utils.data.DataLoader`` object.
At this step, create an instance of the ``nncf.Dataset`` class that represents the calibration dataset. The ``nncf.Dataset`` class can be a wrapper over the framework dataset object that is used for model training or validation. The class constructor receives the dataset object and an optional transformation function.
The transformation function is a function that takes a sample from the dataset and returns data that can be passed to the model for inference. For example, this function can take a tuple of a data tensor and labels tensor, and return the former while ignoring the latter. The transformation function is used to avoid modifying the dataset code to make it compatible with the quantization API. The function is applied to each sample from the dataset before passing it to the model for inference. The following code snippet shows how to create an instance of the ``nncf.Dataset`` class:
.. tab:: PyTorch
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_torch.py
:language: python
:fragment: [dataset]
.. tab-item:: OpenVINO
:sync: ov
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_openvino.py
:language: python
:fragment: [dataset]
.. tab:: ONNX
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_torch.py
:language: python
:fragment: [dataset]
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_onnx.py
:language: python
:fragment: [dataset]
.. tab-item:: ONNX
:sync: onnx
.. tab:: OpenVINO
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_onnx.py
:language: python
:fragment: [dataset]
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_openvino.py
:language: python
:fragment: [dataset]
.. tab-item:: TensorFlow
:sync: tensorflow
.. tab:: TensorFlow
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_tensorflow.py
:language: python
:fragment: [dataset]
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_tensorflow.py
:language: python
:fragment: [dataset]
If there is no framework dataset object, you can create your own entity that implements the ``Iterable`` interface in Python, for example the list of images, and returns data samples feasible for inference. In this case, a transformation function is not required.
If there is no framework dataset object, you can create your own entity that implements the ``Iterable`` interface in Python and returns data samples feasible for inference. In this case, a transformation function is not required.
Run a Quantized Model
Quantize a Model
#####################
Once the dataset is ready and the model object is instantiated, you can apply 8-bit quantization to it:
.. tab:: PyTorch
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_torch.py
:language: python
:fragment: [quantization]
.. tab-item:: OpenVINO
:sync: ov
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_openvino.py
:language: python
:fragment: [quantization]
.. tab:: ONNX
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_torch.py
:language: python
:fragment: [quantization]
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_onnx.py
:language: python
:fragment: [quantization]
.. tab-item:: ONNX
:sync: onnx
.. tab:: OpenVINO
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_onnx.py
:language: python
:fragment: [quantization]
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_openvino.py
:language: python
:fragment: [quantization]
.. tab-item:: TensorFlow
:sync: tensorflow
.. tab:: TensorFlow
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_tensorflow.py
:language: python
:fragment: [quantization]
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_tensorflow.py
:language: python
:fragment: [quantization]
After that the model can be converted into the OpenVINO Intermediate Representation (IR) if needed, compiled and run with OpenVINO:
.. tab-set::
.. note:: The ``model`` is an instance of the ``torch.nn.Module`` class for PyTorch, ``onnx.ModelProto`` for ONNX, and ``openvino.runtime.Model`` for OpenVINO.
.. tab-item:: OpenVINO
:sync: ov
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_openvino.py
:language: python
:fragment: [inference]
After that the model can be exported into th OpenVINO Intermediate Representation if needed and run faster with OpenVINO.
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_torch.py
:language: python
:fragment: [inference]
.. tab-item:: ONNX
:sync: onnx
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_onnx.py
:language: python
:fragment: [inference]
.. tab-item:: TensorFlow
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_tensorflow.py
:language: python
:fragment: [inference]
Tune quantization parameters
############################
``nncf.quantize()`` function has several parameters that allow to tune quantization process to get more accurate model. Below is the list of parameters and their description:
``nncf.quantize()`` function has several optional parameters that allow tuning the quantization process to get a more accurate model. Below is the list of parameters and their description:
* ``model_type`` - used to specify quantization scheme required for specific type of the model. For example, **Transformer** models (BERT, distillBERT, etc.) require a special quantization scheme to preserve accuracy after quantization.
* ``model_type`` - used to specify quantization scheme required for specific type of the model. ``Transformer`` is the only supported special quantization scheme to preserve accuracy after quantization of Transformer models (BERT, DistilBERT, etc.). ``None`` is default, i.e. no specific scheme is defined.
.. code-block:: sh
@@ -115,7 +154,7 @@ Tune quantization parameters
nncf.quantize(model, dataset, preset=nncf.Preset.MIXED)
* ``fast_bias_correction`` - enables more accurate bias (error) correction algorithm that can be used to improve accuracy of the model. This parameter is available only for OpenVINO representation. ``True`` is used by default.
* ``fast_bias_correction`` - when set to ``False``, enables a more accurate bias (error) correction algorithm that can be used to improve the accuracy of the model. This parameter is available only for OpenVINO and ONNX representations. ``True`` is used by default to minimize quantization time.
.. code-block:: sh
@@ -127,7 +166,7 @@ Tune quantization parameters
nncf.quantize(model, dataset, subset_size=1000)
* ``ignored_scope`` - this parameter can be used to exclude some layers from quantization process. For example, if you want to exclude the last layer of the model from quantization. Below are some examples of how to use this parameter:
* ``ignored_scope`` - this parameter can be used to exclude some layers from the quantization process to preserve the model accuracy. For example, when you want to exclude the last layer of the model from quantization. Below are some examples of how to use this parameter:
* Exclude by layer name:
@@ -150,12 +189,24 @@ Tune quantization parameters
regex = '.*layer_.*'
nncf.quantize(model, dataset, ignored_scope=nncf.IgnoredScope(patterns=regex))
* ``target_device`` - defines the target device, the specificity of which will be taken into account during optimization. The following values are supported: ``ANY`` (default), ``CPU``, ``CPU_SPR``, ``GPU``, and ``VPU``.
.. code-block:: sh
nncf.quantize(model, dataset, target_device=nncf.TargetDevice.CPU)
* ``advanced_parameters`` - used to specify advanced quantization parameters for fine-tuning the quantization algorithm. Defined by `nncf.quantization.advanced_parameters <https://openvinotoolkit.github.io/nncf/autoapi/nncf/quantization/advanced_parameters/index.html>`__ NNCF submodule. ``None`` is default.
If the accuracy of the quantized model is not satisfactory, you can try to use the :doc:`Quantization with accuracy control <quantization_w_accuracy_control>` flow.
See also
####################
Examples of how to apply NNCF post-training quantization:
############################################################
* `Example of basic quantization flow in PyTorch <https://github.com/openvinotoolkit/nncf/tree/develop/examples/post_training_quantization/torch/mobilenet_v2>`__
* `Post-Training Quantization of MobileNet v2 OpenVINO Model <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/openvino/mobilenet_v2>`__
* `Post-Training Quantization of YOLOv8 OpenVINO Model <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/openvino/yolov8>`__
* `Post-Training Quantization of MobileNet v2 PyTorch Model <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/torch/mobilenet_v2>`__
* `Post-Training Quantization of SSD PyTorch Model <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/torch/ssd300_vgg16>`__
* `Post-Training Quantization of MobileNet v2 ONNX Model <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/onnx/mobilenet_v2>`__
* `Post-Training Quantization of MobileNet v2 TensorFlow Model <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/tensorflow/mobilenet_v2>`__
@endsphinxdirective
docs/optimization_guide/nncf/ptq/code/ptq_aa_openvino.py
+15 -1
View File
@@ -45,5 +45,19 @@ quantized_model = nncf.quantize_with_accuracy_control(model,
calibration_dataset=calibration_dataset,
validation_dataset=validation_dataset,
validation_fn=validate,
max_drop=0.01)
max_drop=0.01,
drop_type=nncf.DropType.ABSOLUTE)
#! [quantization]
#! [inference]
import openvino.runtime as ov
# compile the model to transform quantized operations to int8
model_int8 = ov.compile_model(quantized_model)
input_fp32 = ... # FP32 model input
res = model_int8(input_fp32)
# save the model
ov.serialize(quantized_model, "quantized_model.xml")
#! [inference]
docs/optimization_guide/nncf/ptq/code/ptq_onnx.py
+17
View File
@@ -20,3 +20,20 @@ model = ... # onnx.ModelProto object
quantized_model = nncf.quantize(model, calibration_dataset)
#! [quantization]
#! [inference]
import openvino.runtime as ov
from openvino.tools.mo import convert_model
# convert ONNX model to OpenVINO model
ov_quantized_model = convert_model(quantized_model)
# compile the model to transform quantized operations to int8
model_int8 = ov.compile_model(ov_quantized_model)
input_fp32 = ... # FP32 model input
res = model_int8(input_fp32)
# save the model
ov.serialize(ov_quantized_model, "quantized_model.xml")
#! [inference]
docs/optimization_guide/nncf/ptq/code/ptq_openvino.py
+13
View File
@@ -19,3 +19,16 @@ model = ... # openvino.runtime.Model object
quantized_model = nncf.quantize(model, calibration_dataset)
#! [quantization]
#! [inference]
import openvino.runtime as ov
# compile the model to transform quantized operations to int8
model_int8 = ov.compile_model(quantized_model)
input_fp32 = ... # FP32 model input
res = model_int8(input_fp32)
# save the model
ov.serialize(quantized_model, "quantized_model.xml")
#! [inference]
docs/optimization_guide/nncf/ptq/code/ptq_tensorflow.py
+17
View File
@@ -19,3 +19,20 @@ model = ... # tensorflow.Module object
quantized_model = nncf.quantize(model, calibration_dataset)
#! [quantization]
#! [inference]
import openvino.runtime as ov
from openvino.tools.mo import convert_model
# convert TensorFlow model to OpenVINO model
ov_quantized_model = convert_model(quantized_model)
# compile the model to transform quantized operations to int8
model_int8 = ov.compile_model(ov_quantized_model)
input_fp32 = ... # FP32 model input
res = model_int8(input_fp32)
# save the model
ov.serialize(ov_quantized_model, "quantized_model.xml")
#! [inference]
docs/optimization_guide/nncf/ptq/code/ptq_torch.py
+22
View File
@@ -19,3 +19,25 @@ model = ... # torch.nn.Module object
quantized_model = nncf.quantize(model, calibration_dataset)
#! [quantization]
#! [inference]
import openvino.runtime as ov
from openvino.tools.mo import convert_model
input_fp32 = ... # FP32 model input
# export PyTorch model to ONNX model
onnx_model_path = "model.onnx"
torch.onnx.export(quantized_model, input_fp32, onnx_model_path)
# convert ONNX model to OpenVINO model
ov_quantized_model = convert_model(onnx_model_path)
# compile the model to transform quantized operations to int8
model_int8 = ov.compile_model(ov_quantized_model)
res = model_int8(input_fp32)
# save the model
ov.serialize(ov_quantized_model, "quantized_model.xml")
#! [inference]
docs/optimization_guide/nncf/ptq/ptq_introduction.md
-26
View File
@@ -1,26 +0,0 @@
# Post-training Quantization with NNCF (new) {#nncf_ptq_introduction}
@sphinxdirective
.. toctree::
:maxdepth: 1
:hidden:
basic_quantization_flow
quantization_w_accuracy_control
Neural Network Compression Framework (NNCF) provides a new post-training quantization API available in Python that is aimed at reusing the code for model training or validation that is usually available with the model in the source framework, for example, PyTorch or TensroFlow. The API is cross-framework and currently supports models representing in the following frameworks: PyTorch, TensorFlow 2.x, ONNX, and OpenVINO.
This API has two main capabilities to apply 8-bit post-training quantization:
* :doc:`Basic quantization <basic_quantization_flow>` - the simplest quantization flow that allows to apply 8-bit integer quantization to the model.
* :doc:`Quantization with accuracy control <quantization_w_accuracy_control>` - the most advanced quantization flow that allows to apply 8-bit quantization to the model with accuracy control.
Additional Resources
####################
* `NNCF GitHub <https://github.com/openvinotoolkit/nncf>`__
* :doc:`Optimizing Models at Training Time <tmo_introduction>`
@endsphinxdirective
docs/optimization_guide/nncf/ptq/quantization_w_accuracy_control.md
+51 -23
View File
@@ -1,4 +1,4 @@
# Quantizing with accuracy control {#quantization_w_accuracy_control}
# Quantizing with Accuracy Control {#quantization_w_accuracy_control}
@sphinxdirective
@@ -7,58 +7,86 @@ Introduction
This is the advanced quantization flow that allows to apply 8-bit quantization to the model with control of accuracy metric. This is achieved by keeping the most impactful operations within the model in the original precision. The flow is based on the :doc:`Basic 8-bit quantization <basic_quantization_flow>` and has the following differences:
* Beside the calibration dataset, a **validation dataset** is required to compute accuracy metric. They can refer to the same data in the simplest case.
* Besides the calibration dataset, a **validation dataset** is required to compute the accuracy metric. Both datasets can refer to the same data in the simplest case.
* **Validation function**, used to compute accuracy metric is required. It can be a function that is already available in the source framework or a custom function.
* Since accuracy validation is run several times during the quantization process, quantization with accuracy control can take more time than the [Basic 8-bit quantization](@ref basic_quantization_flow) flow.
* Since accuracy validation is run several times during the quantization process, quantization with accuracy control can take more time than the :doc:`Basic 8-bit quantization <basic_quantization_flow>` flow.
* The resulted model can provide smaller performance improvement than the :doc:`Basic 8-bit quantization <basic_quantization_flow>` flow because some of the operations are kept in the original precision.
.. note:: Currently, this flow is available only for models in OpenVINO representation.
.. note:: Currently, 8-bit quantization with accuracy control is available only for models in OpenVINO representation.
The steps for the quantization with accuracy control are described below.
Prepare datasets
####################
Prepare calibration and validation datasets
############################################
This step is similar to the :doc:`Basic 8-bit quantization <basic_quantization_flow>` flow. The only difference is that two datasets, calibration and validation, are required.
.. tab:: OpenVINO
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_aa_openvino.py
:language: python
:fragment: [dataset]
.. tab-item:: OpenVINO
:sync: ov
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_aa_openvino.py
:language: python
:fragment: [dataset]
Prepare validation function
###########################
############################
Validation funtion receives ``openvino.runtime.CompiledModel`` object and validation dataset and returns accuracy metric value. The following code snippet shows an example of validation function for OpenVINO model:
.. tab:: OpenVINO
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_aa_openvino.py
:language: python
:fragment: [validation]
.. tab-item:: OpenVINO
:sync: ov
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_aa_openvino.py
:language: python
:fragment: [validation]
Run quantization with accuracy control
#######################################
Now, you can run quantization with accuracy control. The following code snippet shows an example of quantization with accuracy control for OpenVINO model:
``nncf.quantize_with_accuracy_control()`` function is used to run the quantization with accuracy control. The following code snippet shows an example of quantization with accuracy control for OpenVINO model:
.. tab:: OpenVINO
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_aa_openvino.py
:language: python
:fragment: [quantization]
.. tab-item:: OpenVINO
:sync: ov
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_aa_openvino.py
:language: python
:fragment: [quantization]
``max_drop`` defines the accuracy drop threshold. The quantization process stops when the degradation of accuracy metric on the validation dataset is less than the ``max_drop``.
* ``max_drop`` defines the accuracy drop threshold. The quantization process stops when the degradation of accuracy metric on the validation dataset is less than the ``max_drop``. The default value is 0.01. NNCF will stop the quantization and report an error if the ``max_drop`` value can't be reached.
``nncf.quantize_with_accuracy_control()`` API supports all the parameters of ``nncf.quantize()`` API. For example, you can use ``nncf.quantize_with_accuracy_control()`` to quantize a model with a custom configuration.
* ``drop_type`` defines how the accuracy drop will be calculated: ``ABSOLUTE`` (used by default) or ``RELATIVE``.
After that the model can be compiled and run with OpenVINO:
.. tab-set::
.. tab-item:: OpenVINO
:sync: ov
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_aa_openvino.py
:language: python
:fragment: [inference]
``nncf.quantize_with_accuracy_control()`` API supports all the parameters from :doc:`Basic 8-bit quantization <basic_quantization_flow>` API, to quantize a model with accuracy control and a custom configuration.
If the accuracy or performance of the quantized model is not satisfactory, you can try :doc:`Training-time Optimization <tmo_introduction>` as the next step.
Examples of NNCF post-training quantization with control of accuracy metric:
#############################################################################
* `Post-Training Quantization of Anomaly Classification OpenVINO model with control of accuracy metric <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/openvino/quantize_with_accuracy_control>`__
* `Post-Training Quantization of YOLOv8 OpenVINO Model with control of accuracy metric <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/openvino/yolov8_quantize_with_accuracy_control>`__
See also
####################
* :doc:`Optimizing Models at Training Time <tmo_introduction>`
* :doc:`Optimizing Models at Training Time <tmo_introduction>`
@endsphinxdirective
docs/optimization_guide/nncf/qat.md
+111 -79
View File
@@ -26,18 +26,21 @@ PyTorch or TensorFlow 2:
In this step, you add NNCF-related imports in the beginning of the training script:
.. tab:: PyTorch
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [imports]
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [imports]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [imports]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [imports]
2. Create NNCF configuration
++++++++++++++++++++++++++++
@@ -46,18 +49,22 @@ Here, you should define NNCF configuration which consists of model-related param
of optimization methods (``"compression"`` section). For faster convergence, it is also recommended to register a dataset object
specific to the DL framework. It will be used at the model creation step to initialize quantization parameters.
.. tab:: PyTorch
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [nncf_congig]
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [nncf_congig]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [nncf_congig]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [nncf_congig]
3. Apply optimization methods
+++++++++++++++++++++++++++++
@@ -69,18 +76,22 @@ undergoes a set of corresponding transformations and can contain additional oper
the case of QAT, the compression controller object is used for model export and, optionally, in distributed training as it
will be shown below.
.. tab:: PyTorch
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [wrap_model]
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [wrap_model]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [wrap_model]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [wrap_model]
4. Fine-tune the model
++++++++++++++++++++++
@@ -89,17 +100,22 @@ This step assumes that you will apply fine-tuning to the model the same way as i
case of QAT, it is required to train the model for a few epochs with a small learning rate, for example, 10e-5. In principle,
you can skip this step which means that the post-training optimization will be applied to the model.
.. tab:: PyTorch
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [tune_model]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [tune_model]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [tune_model]
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [tune_model]
5. Multi-GPU distributed training
@@ -108,39 +124,47 @@ you can skip this step which means that the post-training optimization will be a
In the case of distributed multi-GPU training (not DataParallel), you should call ``compression_ctrl.distributed()`` before
the fine-tuning that will inform optimization methods to do some adjustments to function in the distributed mode.
.. tab:: PyTorch
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [distributed]
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [distributed]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [distributed]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [distributed]
6. Export quantized model
+++++++++++++++++++++++++
When fine-tuning finishes, the quantized model can be exported to the corresponding format for further inference: ONNX in
the case of PyTorch and frozen graph - for TensorFlow 2.
.. tab:: PyTorch
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [export]
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [export]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [export]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [export]
.. note::
The precision of weigths gets INT8 only after the step of model conversion to OpenVINO Intermediate Representation.
The precision of weights gets INT8 only after the step of model conversion to OpenVINO Intermediate Representation.
You can expect the model footprint reduction only for that format.
@@ -152,17 +176,21 @@ checkpoints during the training. Since NNCF wraps the original model with its ow
To save model checkpoint use the following API:
.. tab:: PyTorch
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [save_checkpoint]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [save_checkpoint]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [save_checkpoint]
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [save_checkpoint]
8. (Optional) Restore from checkpoint
@@ -170,18 +198,22 @@ To save model checkpoint use the following API:
To restore the model from checkpoint you should use the following API:
.. tab:: PyTorch
.. tab-set::
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [load_checkpoint]
.. tab:: TensorFlow 2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [load_checkpoint]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [load_checkpoint]
.. tab-item:: TensorFlow 2
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [load_checkpoint]
For more details on saving/loading checkpoints in the NNCF, see the following `documentation <https://github.com/openvinotoolkit/nncf/blob/develop/docs/Usage.md#saving-and-loading-compressed-models>`__.
docs/optimization_guide/ptq_introduction.md
+13 -9
View File
@@ -6,9 +6,10 @@
:maxdepth: 1
:hidden:
basic_quantization_flow
quantization_w_accuracy_control
pot_introduction
nncf_ptq_introduction
Post-training model optimization is the process of applying special methods that transform the model into a more hardware-friendly representation without retraining or fine-tuning. The most popular and widely-spread method here is 8-bit post-training quantization because it is:
@@ -21,15 +22,18 @@ Post-training model optimization is the process of applying special methods that
.. image:: _static/images/quantization_picture.svg
To apply post-training methods in OpenVINO, you need:
`Neural Network Compression Framework (NNCF) <https://github.com/openvinotoolkit/nncf>`__ provides a post-training quantization API available in Python that is aimed at reusing the code for model training or validation that is usually available with the model in the source framework, for example, PyTorch or TensroFlow. The NNCF API is cross-framework and currently supports models in the following frameworks: OpenVINO, PyTorch, TensorFlow 2.x, and ONNX. Currently, post-training quantization for models in OpenVINO Intermediate Representation is the most mature in terms of supported methods and models coverage.
* A floating-point precision model, FP32 or FP16, converted into the OpenVINO Intermediate Representation (IR) format that can be run on CPU.
* A representative calibration dataset, representing a use case scenario, for example, of 300 samples.
* In case of accuracy constraints, a validation dataset and accuracy metrics should be available.
NNCF API has two main capabilities to apply 8-bit post-training quantization:
Currently, OpenVINO provides two workflows with post-training quantization capabilities:
* :doc:`Basic quantization <basic_quantization_flow>` - the simplest quantization flow that allows applying 8-bit integer quantization to the model. A representative calibration dataset is only needed in this case.
* :doc:`Quantization with accuracy control <quantization_w_accuracy_control>` - the most advanced quantization flow that allows applying 8-bit quantization to the model with accuracy control. Calibration and validation datasets, and a validation function to calculate the accuracy metric are needed in this case.
* :doc:`Post-training Quantization with POT <pot_introduction>` - works with models in OpenVINO Intermediate Representation (IR) only.
* :doc:`Post-training Quantization with NNCF <nncf_ptq_introduction>` - cross-framework solution for model optimization that provides a new simple API for post-training quantization.
Additional Resources
####################
* :doc:`Optimizing Models at Training Time <tmo_introduction>`
* `NNCF GitHub <https://github.com/openvinotoolkit/nncf>`__
* `Tutorial: Migrate quantization from POT API to NNCF API <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/111-yolov5-quantization-migration>`__
@endsphinxdirective