openvino/tools/pot

Post-Training Optimization Tool

@sphinxdirective

.. toctree:: :maxdepth: 1 :hidden:

pot_InstallationGuide pot_docs_LowPrecisionOptimizationGuide pot_compression_algorithms_quantization_README Best Practices <pot_docs_BestPractices> Command-line Interface <pot_compression_cli_README> pot_compression_api_README pot_configs_README Deep neural network protection <pot_ranger_README> pot_docs_FrequentlyAskedQuestions

@endsphinxdirective

Introduction

Post-training Optimization Tool (POT) is designed to accelerate the inference of deep learning models by applying special methods without model retraining or fine-tuning, for example, post-training 8-bit quantization. Therefore, the tool does not require a training dataset or a pipeline. To apply post-training algorithms from the POT, you need:

• A floating-point precision model, FP32 or FP16, converted into the OpenVINO™ Intermediate Representation (IR) format and run on CPU with the OpenVINO™.
• A representative calibration dataset representing a use case scenario, for example, 300 images.

Figure below shows the optimization workflow:

Features

• Two post-training 8-bit quantization algorithms: fast DefaultQuantization and precise AccuracyAwareQuantization.
• Compression for different hardware targets such as CPU and GPU.
• Multiple domains: Computer Vision, Natural Language Processing, Recommendation Systems, Speech Recognition.
• Command-line tool that provides a simple interface for basic use cases.
• API that helps to apply optimization methods within a custom inference script written with OpenVINO Python* API.
• (Experimental) [Ranger algorithm](@ref pot_ranger_README) for model prodection in safity-critical cases.

For benchmarking results collected for the models optimized with POT tool, see [INT8 vs FP32 Comparison on Select Networks and Platforms](@ref openvino_docs_performance_int8_vs_fp32).

POT is opensourced on GitHub as a part of OpenVINO and available at https://github.com/openvinotoolkit/openvino/tools/pot.

Further documentation presumes that you are familiar with basic Deep Learning concepts, such as model inference, dataset preparation, model optimization, as well as with the OpenVINO™ toolkit and its components, such as [Model Optimizer](@ref openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide) and [Accuracy Checker Tool](@ref omz_tools_accuracy_checker).

Get started

Installation

To install POT, follow the Installation Guide.

Usage options

The POT provides three basic usage options:

• Command-line interface (CLI):
  • [Simplified mode](@ref pot_docs_simplified_mode): use this option if the model belongs to the Computer Vision domain and you do have an unannotated dataset for optimization. Note that this optimization method can cause a deviation of model accuracy.
  • [Model Zoo flow](@ref pot_compression_cli_README): this option is recommended if the model is imported from OpenVINO™ Model Zoo or there is a valid [Accuracy Checker Tool](@ref omz_tools_accuracy_checker) configuration file for the model that allows validating model accuracy using [Accuracy Checker Tool](@ref omz_tools_accuracy_checker).
• [Python* API](@ref pot_compression_api_README): this option allows integrating the optimization methods implemented in POT into a Python* inference script that uses OpenVINO Python* API.

POT is also integrated into [Deep Learning Workbench](@ref workbench_docs_Workbench_DG_Introduction) (DL Workbench), a web-based graphical environment that enables you to optimize, tune, analyze, visualize, and compare performance of deep learning models.

Examples

OpenVINO provides several examples to demonstrate the POT optimization workflow:

• Command-line example:
  • Quantization of Image Classification model
• API tutorials:
  • Quantization of Image Classification model
  • Quantization of Object Detection model from Model Zoo
  • Quantization of Segmentation model for medical data
  • Quantization of BERT for Text Classification
• API examples:
  • Quantization of 3D segmentation model
  • Quantization of Face Detection model
  • Quantization of Object Detection model with controable accuracy
  • Speech example for GNA device

See Also

• Low Precision Optimization Guide
• Post-Training Optimization Best Practices
• POT Frequently Asked Questions
• INT8 Quantization by Using Web-Based Interface of the DL Workbench