IntenseWebs/openvino
3
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Model Optimization Guide Restructure (#20188)

Browse Source
This commit is contained in:
Maciej Smyk authored and GitHub committed 2023-10-03 09:41:56 +02:00
1 parent e7435af7b0
commit 492d475728
7 files changed
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/optimization_guide/nncf/filter_pruning.md
-251
View File
@@ -1,251 +0,0 @@
# Filter Pruning of Convolutional Models {#filter_pruning}
@sphinxdirective
Introduction
####################
Filter pruning is an advanced optimization method that allows reducing the computational complexity of the model by removing
redundant or unimportant filters from the convolutional operations of the model. This removal is done in two steps:
1. Unimportant filters are zeroed out by the NNCF optimization with fine-tuning.
2. Zero filters are removed from the model during the export to OpenVINO Intermediate Representation (IR).
Filter Pruning method from the NNCF can be used stand-alone but we usually recommend stacking it with 8-bit quantization for
two reasons. First, 8-bit quantization is the best method in terms of achieving the highest accuracy-performance trade-offs so
stacking it with filter pruning can give even better performance results. Second, applying quantization along with filter
pruning does not hurt accuracy a lot since filter pruning removes noisy filters from the model which narrows down values
ranges of weights and activations and helps to reduce overall quantization error.
.. note::
Filter Pruning usually requires a long fine-tuning or retraining of the model which can be comparable to training the
model from scratch. Otherwise, a large accuracy degradation can be caused. Therefore, the training schedule should be
adjusted accordingly when applying this method.
Below, we provide the steps that are required to apply Filter Pruning + QAT to the model:
Applying Filter Pruning with fine-tuning
########################################
Here, we show the basic steps to modify the training script for the model and use it to zero out unimportant filters:
1. Import NNCF API
++++++++++++++++++
In this step, NNCF-related imports are added in the beginning of the training script:
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [imports]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [imports]
2. Create NNCF configuration
++++++++++++++++++++++++++++
Here, you should define NNCF configuration which consists of model-related parameters (`"input_info"` section) and parameters
of optimization methods (`"compression"` section).
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [nncf_congig]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [nncf_congig]
Here is a brief description of the required parameters of the Filter Pruning method. For a full description refer to the
`GitHub <https://github.com/openvinotoolkit/nncf/blob/develop/docs/compression_algorithms/Pruning.md>`__ page.
* ``pruning_init`` - initial pruning rate target. For example, value ``0.1`` means that at the begging of training, convolutions that can be pruned will have 10% of their filters set to zero.
* ``pruning_target`` - pruning rate target at the end of the schedule. For example, the value ``0.5`` means that at the epoch with the number of ``num_init_steps + pruning_steps``, convolutions that can be pruned will have 50% of their filters set to zero.
* ``pruning_steps` - the number of epochs during which the pruning rate target is increased from ``pruning_init` to ``pruning_target`` value. We recommend keeping the highest learning rate during this period.
3. Apply optimization methods
+++++++++++++++++++++++++++++
In the next step, the original model is wrapped by the NNCF object using the ``create_compressed_model()`` API using the
configuration defined in the previous step. This method returns a so-called compression controller and the wrapped model
that can be used the same way as the original model. It is worth noting that optimization methods are applied at this step
so that the model undergoes a set of corresponding transformations and can contain additional operations required for the
optimization.
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [wrap_model]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [wrap_model]
4. Fine-tune the model
++++++++++++++++++++++
This step assumes that you will apply fine-tuning to the model the same way as it is done for the baseline model. In the case
of Filter Pruning method we recommend using the training schedule and learning rate similar to what was used for the training
of the original model.
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [tune_model]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [tune_model]
5. Multi-GPU distributed training
+++++++++++++++++++++++++++++++++
In the case of distributed multi-GPU training (not DataParallel), you should call ``compression_ctrl.distributed()`` before the
fine-tuning that will inform optimization methods to do some adjustments to function in the distributed mode.
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [distributed]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [distributed]
6. Export quantized model
+++++++++++++++++++++++++
When fine-tuning finishes, the quantized model can be exported to the corresponding format for further inference: ONNX in
the case of PyTorch and frozen graph - for TensorFlow 2.
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [export]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [export]
These were the basic steps to applying the QAT method from the NNCF. However, it is required in some cases to save/load model
checkpoints during the training. Since NNCF wraps the original model with its own object it provides an API for these needs.
7. (Optional) Save checkpoint
+++++++++++++++++++++++++++++
To save model checkpoint use the following API:
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [save_checkpoint]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [save_checkpoint]
8. (Optional) Restore from checkpoint
+++++++++++++++++++++++++++++++++++++
To restore the model from checkpoint you should use the following API:
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_torch.py
:language: python
:fragment: [load_checkpoint]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/pruning_tf.py
:language: python
:fragment: [load_checkpoint]
For more details on saving/loading checkpoints in the NNCF, see the following
`documentation <https://github.com/openvinotoolkit/nncf/blob/develop/docs/Usage.md#saving-and-loading-compressed-models>`__.
Deploying pruned model
######################
The pruned model requres an extra step that should be done to get performance improvement. This step involves removal of the
zero filters from the model. This is done at the model conversion step using :doc:`model conversion API <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>` tool when model is converted from the framework representation (ONNX, TensorFlow, etc.) to OpenVINO Intermediate Representation.
* To remove zero filters from the pruned model add the following parameter to the model conversion command: ``transform=Pruning``
After that, the model can be deployed with OpenVINO in the same way as the baseline model.
For more details about model deployment with OpenVINO, see the corresponding :doc:`documentation <openvino_docs_OV_UG_OV_Runtime_User_Guide>`.
Examples
####################
* `PyTorch Image Classification example <https://github.com/openvinotoolkit/nncf/blob/develop/examples/torch/classification>`__
* `TensorFlow Image Classification example <https://github.com/openvinotoolkit/nncf/tree/develop/examples/tensorflow/classification>`__
@endsphinxdirective
docs/optimization_guide/nncf/introduction.md
-115
View File
@@ -1,115 +0,0 @@
# Compressing Models During Training {#tmo_introduction}
@sphinxdirective
.. toctree::
:maxdepth: 1
:hidden:
qat_introduction
filter_pruning
Introduction
####################
Training-time model compression improves model performance by applying optimizations (such as quantization) during the training. The training process minimizes the loss associated with the lower-precision optimizations, so it is able to maintain the model’s accuracy while reducing its latency and memory footprint. Generally, training-time model optimization results in better model performance and accuracy than :doc:`post-training optimization <ptq_introduction>`, but it can require more effort to set up.
OpenVINO provides the Neural Network Compression Framework (NNCF) tool for implementing compression algorithms on models to improve their performance. NNCF is a Python library that integrates into PyTorch and TensorFlow training pipelines to add training-time compression methods to the pipeline. To apply training-time compression methods with NNCF, you need:
- A floating-point model from the PyTorch or TensorFlow framework.
- A training pipeline set up in the PyTorch or TensorFlow framework.
- Training and validation datasets.
Adding compression to a training pipeline only requires a few lines of code. The compression techniques are defined through a single configuration file that specifies which algorithms to use during fine-tuning.
NNCF Quick Start Examples
+++++++++++++++++++++++++
See the following Jupyter Notebooks for step-by-step examples showing how to add model compression to a PyTorch or Tensorflow training pipeline with NNCF:
- `Quantization Aware Training with NNCF and PyTorch <notebooks/302-pytorch-quantization-aware-training-with-output.html>`__.
- `Quantization Aware Training with NNCF and TensorFlow <notebooks/305-tensorflow-quantization-aware-training-with-output.html>`__.
Installation
####################
NNCF is open-sourced on `GitHub <https://github.com/openvinotoolkit/nncf>`__ and distributed as a separate package from OpenVINO. It is also available on PyPI. Install it to the same Python environment where PyTorch or TensorFlow is installed.
Install from PyPI
++++++++++++++++++++
To install the latest released version via pip manager run the following command:
.. code-block:: sh
pip install nncf
.. note::
To install with specific frameworks, use the `pip install nncf[extras]` command, where extras is a list of possible extras, for example, `torch`, `tf`, `onnx`.
To install the latest NNCF version from source, follow the instruction on `GitHub <https://github.com/openvinotoolkit/nncf#installation>`__.
.. note::
NNCF does not have OpenVINO as an installation requirement. To deploy optimized models you should install OpenVINO separately.
Working with NNCF
####################
The figure below shows a common workflow of applying training-time compressions with NNCF. The NNCF optimizations are added to the TensorFlow or PyTorch training script, and then the model undergoes fine-tuning. The optimized model can then be exported to OpenVINO IR format for accelerated performance with OpenVINO Runtime.
.. image:: _static/images/nncf_workflow.svg
Training-Time Compression Methods
+++++++++++++++++++++++++++++++++
NNCF provides several methods for improving model performance with training-time compression.
Quantization
--------------------
Quantization is the process of converting the weights and activation values in a neural network from a high-precision format (such as 32-bit floating point) to a lower-precision format (such as 8-bit integer). It helps to reduce the model’s memory footprint and latency. NNCF uses quantization-aware training to quantize models.
Quantization-aware training inserts nodes into the neural network during training that simulate the effect of lower precision. This allows the training algorithm to consider quantization errors as part of the overall training loss that gets minimized during training. The network is then able to achieve enhanced accuracy when quantized.
The officially supported method of quantization in NNCF is uniform 8-bit quantization. This means all the weights and activation functions in the neural network are converted to 8-bit values. See the :doc:`Quantization-ware Training guide <qat_introduction>` to learn more.
Filter pruning
--------------------
Filter pruning algorithms compress models by zeroing out the output filters of convolutional layers based on a certain filter importance criterion. During fine-tuning, an importance criteria is used to search for redundant filters that don’t significantly contribute to the network’s output and zero them out. After fine-tuning, the zeroed-out filters are removed from the network. For more information, see the :doc:`Filter Pruning <filter_pruning>` page.
Experimental methods
--------------------
NNCF also provides state-of-the-art compression techniques that are still in the experimental stages of development and are only recommended for expert developers. These include:
- Mixed-precision quantization
- Sparsity
- Binarization
To learn more about these methods, visit the `NNCF repository on GitHub <https://github.com/openvinotoolkit/nncf>`__.
Recommended Workflow
++++++++++++++++++++
Using compression-aware training requires a training pipeline, an annotated dataset, and compute resources (such as CPUs or GPUs). If you don't already have these set up and available, it can be easier to start post-training quantization to quickly see quantized results. Then you can use compression-aware training if the model isn't accurate enough. We recommend the following workflow for compressing models with NNCF:
1. :doc:`Perform post-training quantization <ptq_introduction>` on your model and then compare performance to the original model.
2. If the accuracy is too degraded, use :doc:`Quantization-aware Training <qat_introduction>` to increase accuracy while still achieving faster inference time.
3. If the quantized model is still too slow, use :doc:`Filter Pruning <filter_pruning>` to further improve the model’s inference speed.
Additional Resources
####################
- :doc:`Quantizing Models Post-training <ptq_introduction>`
- `NNCF GitHub repository <https://github.com/openvinotoolkit/nncf>`__
- `NNCF FAQ <https://github.com/openvinotoolkit/nncf/blob/develop/docs/FAQ.md>`__
- `Quantization Aware Training with NNCF and PyTorch <notebooks/302-pytorch-quantization-aware-training-with-output.html>`__
- `Quantization Aware Training with NNCF and TensorFlow <notebooks/305-tensorflow-quantization-aware-training-with-output.html>`__.
@endsphinxdirective
docs/optimization_guide/nncf/ptq/basic_quantization_flow.md
-214
View File
@@ -1,214 +0,0 @@
# Basic Quantization Flow {#basic_quantization_flow}
@sphinxdirective
Introduction
####################
The basic quantization flow is the simplest way to apply 8-bit quantization to the model. It is available for models in the following frameworks: OpenVINO, PyTorch, TensorFlow 2.x, and ONNX. The basic quantization flow is based on the following steps:
* Set up an environment and install dependencies.
* Prepare a representative **calibration dataset** that is used to estimate quantization parameters of the activations within the model, for example, of 300 samples.
* Call the quantization API to apply 8-bit quantization to the model.
Set up an Environment
#####################
It is recommended to set up a separate Python environment for quantization with NNCF. To do this, run the following command:
.. code-block:: sh
python3 -m venv nncf_ptq_env
Install all the packages required to instantiate the model object, for example, DL framework. After that, install NNCF on top of the environment:
.. code-block:: sh
pip install nncf
Prepare a Calibration Dataset
#############################
At this step, create an instance of the ``nncf.Dataset`` class that represents the calibration dataset. The ``nncf.Dataset`` class can be a wrapper over the framework dataset object that is used for model training or validation. The class constructor receives the dataset object and an optional transformation function.
The transformation function is a function that takes a sample from the dataset and returns data that can be passed to the model for inference. For example, this function can take a tuple of a data tensor and labels tensor, and return the former while ignoring the latter. The transformation function is used to avoid modifying the dataset code to make it compatible with the quantization API. The function is applied to each sample from the dataset before passing it to the model for inference. The following code snippet shows how to create an instance of the ``nncf.Dataset`` class:
.. tab-set::
.. tab-item:: OpenVINO
:sync: openvino
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_openvino.py
:language: python
:fragment: [dataset]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_torch.py
:language: python
:fragment: [dataset]
.. tab-item:: ONNX
:sync: onnx
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_onnx.py
:language: python
:fragment: [dataset]
.. tab-item:: TensorFlow
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_tensorflow.py
:language: python
:fragment: [dataset]
If there is no framework dataset object, you can create your own entity that implements the ``Iterable`` interface in Python, for example the list of images, and returns data samples feasible for inference. In this case, a transformation function is not required.
Quantize a Model
#####################
Once the dataset is ready and the model object is instantiated, you can apply 8-bit quantization to it.
See the `example section <#examples-of-how-to-apply-nncf-post-training-quantization>`__ at the end of this document for examples for each framework.
.. tab-set::
.. tab-item:: OpenVINO
:sync: openvino
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_openvino.py
:language: python
:fragment: [quantization]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_torch.py
:language: python
:fragment: [quantization]
.. tab-item:: ONNX
:sync: onnx
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_onnx.py
:language: python
:fragment: [quantization]
.. tab-item:: TensorFlow
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_tensorflow.py
:language: python
:fragment: [quantization]
After that the model can be converted into the OpenVINO Intermediate Representation (IR) if needed, compiled and run with OpenVINO.
If you have not already installed OpenVINO developer tools, install it with ``pip install openvino-dev``.
.. tab-set::
.. tab-item:: OpenVINO
:sync: openvino
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_openvino.py
:language: python
:fragment: [inference]
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_torch.py
:language: python
:fragment: [inference]
.. tab-item:: ONNX
:sync: onnx
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_onnx.py
:language: python
:fragment: [inference]
.. tab-item:: TensorFlow
:sync: tensorflow
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_tensorflow.py
:language: python
:fragment: [inference]
Tune quantization parameters
############################
``nncf.quantize()`` function has several optional parameters that allow tuning the quantization process to get a more accurate model. Below is the list of parameters and their description:
* ``model_type`` - used to specify quantization scheme required for specific type of the model. ``Transformer`` is the only supported special quantization scheme to preserve accuracy after quantization of Transformer models (BERT, DistilBERT, etc.). ``None`` is default, i.e. no specific scheme is defined.
.. code-block:: sh
nncf.quantize(model, dataset, model_type=nncf.ModelType.Transformer)
* ``preset`` - defines quantization scheme for the model. Two types of presets are available:
* ``PERFORMANCE`` (default) - defines symmetric quantization of weights and activations
* ``MIXED`` - weights are quantized with symmetric quantization and the activations are quantized with asymmetric quantization. This preset is recommended for models with non-ReLU and asymmetric activation functions, e.g. ELU, PReLU, GELU, etc.
.. code-block:: sh
nncf.quantize(model, dataset, preset=nncf.Preset.MIXED)
* ``fast_bias_correction`` - when set to ``False``, enables a more accurate bias (error) correction algorithm that can be used to improve the accuracy of the model. This parameter is available only for OpenVINO and ONNX representations. ``True`` is used by default to minimize quantization time.
.. code-block:: sh
nncf.quantize(model, dataset, fast_bias_correction=False)
* ``subset_size`` - defines the number of samples from the calibration dataset that will be used to estimate quantization parameters of activations. The default value is 300.
.. code-block:: sh
nncf.quantize(model, dataset, subset_size=1000)
* ``ignored_scope`` - this parameter can be used to exclude some layers from the quantization process to preserve the model accuracy. For example, when you want to exclude the last layer of the model from quantization. Below are some examples of how to use this parameter:
* Exclude by layer name:
.. code-block:: sh
names = ['layer_1', 'layer_2', 'layer_3']
nncf.quantize(model, dataset, ignored_scope=nncf.IgnoredScope(names=names))
* Exclude by layer type:
.. code-block:: sh
types = ['Conv2d', 'Linear']
nncf.quantize(model, dataset, ignored_scope=nncf.IgnoredScope(types=types))
* Exclude by regular expression:
.. code-block:: sh
regex = '.*layer_.*'
nncf.quantize(model, dataset, ignored_scope=nncf.IgnoredScope(patterns=regex))
* ``target_device`` - defines the target device, the specificity of which will be taken into account during optimization. The following values are supported: ``ANY`` (default), ``CPU``, ``CPU_SPR``, ``GPU``, and ``NPU``.
.. code-block:: sh
nncf.quantize(model, dataset, target_device=nncf.TargetDevice.CPU)
* ``advanced_parameters`` - used to specify advanced quantization parameters for fine-tuning the quantization algorithm. Defined by `nncf.quantization.advanced_parameters <https://openvinotoolkit.github.io/nncf/autoapi/nncf/quantization/advanced_parameters/index.html>`__ NNCF submodule. ``None`` is default.
If the accuracy of the quantized model is not satisfactory, you can try to use the :doc:`Quantization with accuracy control <quantization_w_accuracy_control>` flow.
Examples of how to apply NNCF post-training quantization:
############################################################
* `Post-Training Quantization of MobileNet v2 OpenVINO Model <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/openvino/mobilenet_v2>`__
* `Post-Training Quantization of YOLOv8 OpenVINO Model <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/openvino/yolov8>`__
* `Post-Training Quantization of MobileNet v2 PyTorch Model <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/torch/mobilenet_v2>`__
* `Post-Training Quantization of SSD PyTorch Model <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/torch/ssd300_vgg16>`__
* `Post-Training Quantization of MobileNet v2 ONNX Model <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/onnx/mobilenet_v2>`__
* `Post-Training Quantization of MobileNet v2 TensorFlow Model <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/tensorflow/mobilenet_v2>`__
@endsphinxdirective
docs/optimization_guide/nncf/ptq/quantization_w_accuracy_control.md
-92
View File
@@ -1,92 +0,0 @@
# Quantizing with Accuracy Control {#quantization_w_accuracy_control}
@sphinxdirective
Introduction
####################
This is the advanced quantization flow that allows to apply 8-bit quantization to the model with control of accuracy metric. This is achieved by keeping the most impactful operations within the model in the original precision. The flow is based on the :doc:`Basic 8-bit quantization <basic_quantization_flow>` and has the following differences:
* Besides the calibration dataset, a **validation dataset** is required to compute the accuracy metric. Both datasets can refer to the same data in the simplest case.
* **Validation function**, used to compute accuracy metric is required. It can be a function that is already available in the source framework or a custom function.
* Since accuracy validation is run several times during the quantization process, quantization with accuracy control can take more time than the :doc:`Basic 8-bit quantization <basic_quantization_flow>` flow.
* The resulted model can provide smaller performance improvement than the :doc:`Basic 8-bit quantization <basic_quantization_flow>` flow because some of the operations are kept in the original precision.
.. note:: Currently, 8-bit quantization with accuracy control is available only for models in OpenVINO representation.
The steps for the quantization with accuracy control are described below.
Prepare calibration and validation datasets
############################################
This step is similar to the :doc:`Basic 8-bit quantization <basic_quantization_flow>` flow. The only difference is that two datasets, calibration and validation, are required.
.. tab-set::
.. tab-item:: OpenVINO
:sync: openvino
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_aa_openvino.py
:language: python
:fragment: [dataset]
Prepare validation function
############################
Validation function receives ``openvino.runtime.CompiledModel`` object and validation dataset and returns accuracy metric value. The following code snippet shows an example of validation function for OpenVINO model:
.. tab-set::
.. tab-item:: OpenVINO
:sync: openvino
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_aa_openvino.py
:language: python
:fragment: [validation]
Run quantization with accuracy control
#######################################
``nncf.quantize_with_accuracy_control()`` function is used to run the quantization with accuracy control. The following code snippet shows an example of quantization with accuracy control for OpenVINO model:
.. tab-set::
.. tab-item:: OpenVINO
:sync: openvino
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_aa_openvino.py
:language: python
:fragment: [quantization]
* ``max_drop`` defines the accuracy drop threshold. The quantization process stops when the degradation of accuracy metric on the validation dataset is less than the ``max_drop``. The default value is 0.01. NNCF will stop the quantization and report an error if the ``max_drop`` value can't be reached.
* ``drop_type`` defines how the accuracy drop will be calculated: ``ABSOLUTE`` (used by default) or ``RELATIVE``.
After that the model can be compiled and run with OpenVINO:
.. tab-set::
.. tab-item:: OpenVINO
:sync: openvino
.. doxygensnippet:: docs/optimization_guide/nncf/ptq/code/ptq_aa_openvino.py
:language: python
:fragment: [inference]
``nncf.quantize_with_accuracy_control()`` API supports all the parameters from :doc:`Basic 8-bit quantization <basic_quantization_flow>` API, to quantize a model with accuracy control and a custom configuration.
If the accuracy or performance of the quantized model is not satisfactory, you can try :doc:`Training-time Optimization <tmo_introduction>` as the next step.
Examples of NNCF post-training quantization with control of accuracy metric:
#############################################################################
* `Post-Training Quantization of Anomaly Classification OpenVINO model with control of accuracy metric <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/openvino/quantize_with_accuracy_control>`__
* `Post-Training Quantization of YOLOv8 OpenVINO Model with control of accuracy metric <https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/openvino/yolov8_quantize_with_accuracy_control>`__
See also
####################
* :doc:`Optimizing Models at Training Time <tmo_introduction>`
@endsphinxdirective
docs/optimization_guide/nncf/qat.md
-233
View File
@@ -1,233 +0,0 @@
# Quantization-aware Training (QAT) {#qat_introduction}
@sphinxdirective
Introduction
####################
Quantization-aware Training is a popular method that allows quantizing a model and applying fine-tuning to restore accuracy
degradation caused by quantization. In fact, this is the most accurate quantization method. This document describes how to
apply QAT from the Neural Network Compression Framework (NNCF) to get 8-bit quantized models. This assumes that you are
knowledgeable in Python programming and familiar with the training code for the model in the source DL framework.
Using NNCF QAT
####################
Here, we provide the steps that are required to integrate QAT from NNCF into the training script written with
PyTorch or TensorFlow 2:
.. note::
Currently, NNCF for TensorFlow 2 supports optimization of the models created using Keras
`Sequential API <https://www.tensorflow.org/guide/keras/sequential_model>`__ or
`Functional API <https://www.tensorflow.org/guide/keras/functional>`__.
1. Import NNCF API
++++++++++++++++++++
In this step, you add NNCF-related imports in the beginning of the training script:
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [imports]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [imports]
2. Create NNCF configuration
++++++++++++++++++++++++++++
Here, you should define NNCF configuration which consists of model-related parameters (``"input_info"`` section) and parameters
of optimization methods (``"compression"`` section). For faster convergence, it is also recommended to register a dataset object
specific to the DL framework. It will be used at the model creation step to initialize quantization parameters.
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [nncf_congig]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [nncf_congig]
3. Apply optimization methods
+++++++++++++++++++++++++++++
In the next step, you need to wrap the original model object with the ``create_compressed_model()`` API using the configuration
defined in the previous step. This method returns a so-called compression controller and a wrapped model that can be used the
same way as the original model. It is worth noting that optimization methods are applied at this step so that the model
undergoes a set of corresponding transformations and can contain additional operations required for the optimization. In
the case of QAT, the compression controller object is used for model export and, optionally, in distributed training as it
will be shown below.
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [wrap_model]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [wrap_model]
4. Fine-tune the model
++++++++++++++++++++++
This step assumes that you will apply fine-tuning to the model the same way as it is done for the baseline model. In the
case of QAT, it is required to train the model for a few epochs with a small learning rate, for example, 10e-5. In principle,
you can skip this step which means that the post-training optimization will be applied to the model.
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [tune_model]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [tune_model]
5. Multi-GPU distributed training
+++++++++++++++++++++++++++++++++
In the case of distributed multi-GPU training (not DataParallel), you should call ``compression_ctrl.distributed()`` before
the fine-tuning that will inform optimization methods to do some adjustments to function in the distributed mode.
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [distributed]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [distributed]
6. Export quantized model
+++++++++++++++++++++++++
When fine-tuning finishes, the quantized model can be exported to the corresponding format for further inference: ONNX in
the case of PyTorch and frozen graph - for TensorFlow 2.
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [export]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [export]
.. note::
The precision of weights gets INT8 only after the step of model conversion to OpenVINO Intermediate Representation.
You can expect the model footprint reduction only for that format.
These were the basic steps to applying the QAT method from the NNCF. However, it is required in some cases to save/load model
checkpoints during the training. Since NNCF wraps the original model with its own object it provides an API for these needs.
7. (Optional) Save checkpoint
+++++++++++++++++++++++++++++
To save model checkpoint use the following API:
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [save_checkpoint]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [save_checkpoint]
8. (Optional) Restore from checkpoint
+++++++++++++++++++++++++++++++++++++
To restore the model from checkpoint you should use the following API:
.. tab-set::
.. tab-item:: PyTorch
:sync: pytorch
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_torch.py
:language: python
:fragment: [load_checkpoint]
.. tab-item:: TensorFlow 2
:sync: tensorflow-2
.. doxygensnippet:: docs/optimization_guide/nncf/code/qat_tf.py
:language: python
:fragment: [load_checkpoint]
For more details on saving/loading checkpoints in the NNCF, see the following `documentation <https://github.com/openvinotoolkit/nncf/blob/develop/docs/Usage.md#saving-and-loading-compressed-models>`__.
Deploying quantized model
#########################
The quantized model can be deployed with OpenVINO in the same way as the baseline model. No extra steps or options are
required in this case. For more details, see the corresponding :doc:`documentation <openvino_docs_OV_UG_OV_Runtime_User_Guide>`.
Examples
####################
* `Quantizing PyTorch model with NNCF <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/302-pytorch-quantization-aware-training>`__
* `Quantizing TensorFlow model with NNCF <https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/305-tensorflow-quantization-aware-training>`__
@endsphinxdirective
docs/optimization_guide/nncf/weight_compression.md
-38
View File
@@ -1,38 +0,0 @@
# Weight Compression {#weight_compression}
@sphinxdirective
Enhancing Model Efficiency with Weight Compression
##################################################################
Weight compression aims to reduce the memory footprint of a model. It can also lead to significant performance improvement for large memory-bound models, such as Large Language Models (LLMs). LLMs and other models, which require extensive memory to store the weights during inference, can benefit from weight compression in the following ways:
- enabling the inference of exceptionally large models that cannot be accommodated in the memory of the device;
- improving the inference performance of the models by reducing the latency of the memory access when computing the operations with weights, for example, Linear layers.
Currently, `Neural Network Compression Framework (NNCF) <https://github.com/openvinotoolkit/nncf>`__ provides 8-bit weight quantization as a compression method primarily designed to optimize LLMs. The main difference between weights compression and full model quantization (post-training quantization) is that activations remain floating-point in the case of weights compression which leads to a better accuracy. Weight compression for LLMs provides a solid inference performance improvement which is on par with the performance of the full model quantization. In addition, weight compression is data-free and does not require a calibration dataset, making it easy to use.
Compress Model Weights
######################
The code snippet below shows how to compress the weights of the model represented in OpenVINO IR using NNCF:
.. tab-set::
.. tab-item:: OpenVINO
:sync: openvino
.. doxygensnippet:: docs/optimization_guide/nncf/code/weight_compression_openvino.py
:language: python
:fragment: [compression_8bit]
Now, the model is ready for compilation and inference. It can be also saved into a compressed format, resulting in a smaller binary file.
Additional Resources
####################
- :doc:`Post-training Quantization <ptq_introduction>`
- :doc:`Training-time Optimization <tmo_introduction>`
- `NNCF GitHub <https://github.com/openvinotoolkit/nncf>`__
@endsphinxdirective