openvino/docs/IE_DG/Deep_Learning_Inference_Engine_DevGuide.md

Inference Engine Developer Guide

Introduction to the OpenVINO™ Toolkit

The OpenVINO™ toolkit is a comprehensive toolkit that you can use to develop and deploy vision-oriented solutions on Intel® platforms. Vision-oriented means the solutions use images or videos to perform specific tasks. A few of the solutions use cases include autonomous navigation, digital surveillance cameras, robotics, and mixed-reality headsets.

The OpenVINO™ toolkit:

• Enables CNN-based deep learning inference on the edge
• Supports heterogeneous execution across an Intel® CPU, Intel® Integrated Graphics, Intel® Movidius™ Neural Compute Stick and Intel® Neural Compute Stick 2
• Speeds time-to-market via an easy-to-use library of computer vision functions and pre-optimized kernels
• Includes optimized calls for computer vision standards including OpenCV*, OpenCL™, and OpenVX*

The OpenVINO™ toolkit includes the following components:

• Intel® Deep Learning Deployment Toolkit (Intel® DLDT)
  • Deep Learning Model Optimizer — A cross-platform command-line tool for importing models and preparing them for optimal execution with the Deep Learning Inference Engine. The Model Optimizer supports converting Caffe*, TensorFlow*, MXNet*, Kaldi*, ONNX* models.
  • Deep Learning Inference Engine — A unified API to allow high performance inference on many hardware types including Intel® CPU, Intel® Processor Graphics, Intel® FPGA, Intel® Neural Compute Stick 2.
  • nGraph — graph representation and manipulation engine which is used to represent a model inside Inference Engine and allows the run-time model construction without using Model Optimizer.
• OpenCV — OpenCV* community version compiled for Intel® hardware. Includes PVL libraries for computer vision.
• Drivers and runtimes for OpenCL™ version 2.1
• Intel® Media SDK
• OpenVX* — Intel's implementation of OpenVX* optimized for running on Intel® hardware (CPU, GPU, IPU).
• Demos and samples.

This Guide provides overview of the Inference Engine describing the typical workflow for performing inference of a pre-trained and optimized deep learning model and a set of sample applications.

NOTES:

• Before you perform inference with the Inference Engine, your models should be converted to the Inference Engine format using the Model Optimizer or built directly in run-time using nGraph API. To learn about how to use Model Optimizer, refer to the Model Optimizer Developer Guide. To learn about the pre-trained and optimized models delivered with the OpenVINO™ toolkit, refer to [Pre-Trained Models](@ref omz_models_intel_index).
• Intel® System Studio is an all-in-one, cross-platform tool suite, purpose-built to simplify system bring-up and improve system and IoT device application performance on Intel® platforms. If you are using the Intel® Distribution of OpenVINO™ with Intel® System Studio, go to Get Started with Intel® System Studio.

Table of Contents

• Introduction to Intel® Deep Learning Deployment Toolkit

• Inference Engine API Changes History

• Introduction to Inference Engine

• Introduction to nGraph Flow

• Understanding Inference Engine Memory Primitives

• Introduction to Inference Engine Device Query API

• Adding Your Own Layers to the Inference Engine

• Integrating Inference Engine in Your Application

• Migration from Inference Engine Plugin API to Core API

• Introduction to Performance Topics

• Inference Engine Python API Overview

• Using Dynamic Batching feature

• Using Static Shape Infer feature

• Using Low-Precision 8-bit Integer Inference

• Using Bfloat16 Inference

• Utilities to Validate Your Converted Model

  • Using Cross Check Tool for Per-Layer Comparison Between Plugins

• Supported Devices

  • GPU
  • CPU
  • FPGA
  • VPU
    • MYRIAD
    • HDDL
  • Heterogeneous execution
  • GNA
  • NEW! MULTI

• [Pre-Trained Models](@ref omz_models_intel_index)

• Known Issues

Typical Next Step: Introduction to Intel® Deep Learning Deployment Toolkit