openvino/docs/optimization_guide/nncf_introduction.md

Neural Network Compression Framework

This document describes the Neural Network Compression Framework (NNCF) which is distributed as a separate tool but is highly aligned with OpenVINO™ in terms of the supported optimization features and models. It is open-sourced and available on GitHub.

Introduction

Neural Network Compression Framework (NNCF) is aimed at optimizing Deep Neural Network (DNN) by applying optimization methods, such as quantization, pruning, etc., to the original framework model. It provides in-training optimization capabilities which means that optimization methods require model fine-tuning or even re-training. The diagram below shows the model optimization workflow using NNCF.

Features

• Support optimization of PyTorch and TensorFlow 2.x models.
• Support of various optimization algorithms, applied during a model fine-tuning process to achieve a better performance-accuracy trade-off:

Compression algorithm	PyTorch	TensorFlow 2.x
8- bit quantization	Supported	Supported
Filter pruning	Supported	Supported
Sparsity	Supported	Supported
Mixed-precision quantization	Supported	Not supported
Binarization	Supported	Not supported

• Stacking of optimization methods. For example: 8-bit quaNtization + Filter Pruning.
• Support for Accuracy-Aware model training pipelines via the Adaptive Compression Level Training and Early Exit Training.
• Automatic, configurable model graph transformation to obtain the compressed model.

  Note

  : Limited support for TensorFlow models. Only the models created, using Sequential or Keras Functional API, are supported.

• GPU-accelerated layers for the faster compressed model fine-tuning.
• Distributed training support.
• Configuration file examples for each supported compression algorithm.
• Exporting PyTorch compressed models to ONNX* checkpoints and TensorFlow compressed models to SavedModel or Frozen Graph format, ready to use with OpenVINO™ toolkit.
• Git patches for prominent third-party repositories (huggingface-transformers) demonstrating the process of integrating NNCF into custom training pipelines

Get started

Installation

NNCF provides the packages available for installation through the PyPI repository. To install the latest version via pip manager run the following command:

pip install nncf

Usage examples

NNCF provides various examples and tutorials that demonstrate usage of optimization methods.

Tutorials

• Quantization-aware training of PyTorch model
• Quantization-aware training of TensorFlow model

Samples

• PyTorch:

  • Image Classification sample
  • Object Detection sample
  • Semantic segmentation sample

• TensorFlow samples:

  • Image Classification sample
  • Object Detection sample
  • Instance Segmentation sample

See also

• Compressed Model Zoo
• NNCF in HuggingFace Optimum
• Post-training optimization