ef45b5da8d
* Doc Migration from Gitlab (#1289) * doc migration * fix * Update FakeQuantize_1.md * Update performance_benchmarks.md * Updates graphs for FPGA * Update performance_benchmarks.md * Change DL Workbench structure (#1) * Changed DL Workbench structure * Fixed tags * fixes * Update ie_docs.xml * Update performance_benchmarks_faq.md * Fixes in DL Workbench layout * Fixes for CVS-31290 * [DL Workbench] Minor correction * Fix for CVS-30955 * Added nGraph deprecation notice as requested by Zoe * fix broken links in api doxy layouts * CVS-31131 fixes * Additional fixes * Fixed POT TOC * Update PAC_Configure.md PAC DCP 1.2.1 install guide. * Update inference_engine_intro.md * fix broken link * Update opset.md * fix * added opset4 to layout * added new opsets to layout, set labels for them * Update VisionAcceleratorFPGA_Configure.md Updated from 2020.3 to 2020.4 Co-authored-by: domi2000 <domi2000@users.noreply.github.com>
28 KiB
28 KiB
Supported Devices
The Inference Engine can infer models in different formats with various input and output formats. This section provides supported and optimal configurations per device.
Note
: With OpenVINO™ 2020.4 release, Intel® Movidius™ Neural Compute Stick is no longer supported.
The Inference Engine provides unique capabilities to infer deep learning models on the following device types with corresponding plugins:
| Plugin | Device types |
|---|---|
| GPU plugin | Intel® Processor Graphics, including Intel® HD Graphics and Intel® Iris® Graphics |
| CPU plugin | Intel® Xeon® with Intel® Advanced Vector Extensions 2 (Intel® AVX2), Intel® Advanced Vector Extensions 512 (Intel® AVX-512), and AVX512_BF16, Intel® Core™ Processors with Intel® AVX2, Intel® Atom® Processors with Intel® Streaming SIMD Extensions (Intel® SSE) |
| FPGA plugin (available in the Intel® Distribution of OpenVINO™ toolkit) | Intel® Vision Accelerator Design with an Intel® Arria 10 FPGA (Speed Grade 2), Intel® Programmable Acceleration Card with Intel® Arria® 10 GX FPGA |
| VPU plugins (available in the Intel® Distribution of OpenVINO™ toolkit) | Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X, Intel® Vision Accelerator Design with Intel® Movidius™ VPUs |
| GNA plugin (available in the Intel® Distribution of OpenVINO™ toolkit) | Intel® Speech Enabling Developer Kit, Amazon Alexa* Premium Far-Field Developer Kit, Intel® Pentium® Silver J5005 Processor, Intel® Pentium® Silver N5000 Processor, Intel® Celeron® J4005 Processor, Intel® Celeron® J4105 Processor, Intel® Celeron® Processor N4100, Intel® Celeron® Processor N4000, Intel® Core™ i3-8121U Processor, Intel® Core™ i7-1065G7 Processor, Intel® Core™ i7-1060G7 Processor, Intel® Core™ i5-1035G4 Processor, Intel® Core™ i5-1035G7 Processor, Intel® Core™ i5-1035G1 Processor, Intel® Core™ i5-1030G7 Processor, Intel® Core™ i5-1030G4 Processor, Intel® Core™ i3-1005G1 Processor, Intel® Core™ i3-1000G1 Processor, Intel® Core™ i3-1000G4 Processor |
| Multi-Device plugin | Multi-Device plugin enables simultaneous inference of the same network on several Intel® devices in parallel |
| Heterogeneous plugin | Heterogeneous plugin enables automatic inference splitting between several Intel® devices (for example if a device doesn't support certain layers). |
Supported Configurations
The Inference Engine can inference models in different formats with various input and output formats. This chapter provides supported and optimal configurations for each plugin.
Terminology
| Acronym/Term | Description |
|---|---|
| DL | Deep Learning |
| FP32 format | Single-precision floating-point format |
| BF16 format | Brain floating-point format |
| FP16 format | Half-precision floating-point format |
| I16 format | 2-byte signed integer format |
| I8 format | 1-byte signed integer format |
| U16 format | 2-byte unsigned integer format |
| U8 format | 1-byte unsigned integer format |
NHWC, NCHW - Image data layout. Refers to the representation of batches of images. NCDHW - Images sequence data layout.
- N - Number of images in a batch
- D - Depth. Depend on model it could be spatial or time dimension
- H - Number of pixels in the vertical dimension
- W - Number of pixels in the horizontal dimension
- C - Number of channels
CHW, NC, C - Tensor memory layout. For example, the CHW value at index (c,h,w) is physically located at index (c*H+h)*W+w, for others by analogy
Supported Model Formats
| Plugin | FP32 | FP16 | I8 |
|---|---|---|---|
| CPU plugin | Supported and preferred | Supported | Supported |
| GPU plugin | Supported | Supported and preferred | Supported* |
| FPGA plugin | Supported | Supported | Not supported |
| VPU plugins | Not supported | Supported | Not supported |
| GNA plugin | Supported | Supported | Not supported |
* - currently, only limited set of topologies might benefit from enabling I8 model on GPU |
|||
| For Multi-Device and Heterogeneous execution | |||
| the supported models formats depends on the actual underlying devices. Generally, FP16 is preferable as it is most ubiquitous and performant. |
Supported Input Precision
| Plugin | FP32 | FP16 | U8 | U16 | I8 | I16 |
|---|---|---|---|---|---|---|
| CPU plugin | Supported | Not supported | Supported | Supported | Not supported | Supported |
| GPU plugin | Supported | Supported* | Supported* | Supported* | Not supported | Supported* |
| FPGA plugin | Supported | Supported* | Supported | Supported | Not supported | Supported |
| VPU plugins | Supported | Supported | Supported | Not supported | Not supported | Not supported |
| GNA plugin | Supported | Not supported | Supported | Not supported | Supported | Supported |
* - Supported via SetBlob only, GetBlob returns FP32
For Multi-Device and Heterogeneous execution
the supported input precision depends on the actual underlying devices. Generally, U8 is preferable as it is most ubiquitous.
Supported Output Precision
| Plugin | FP32 | FP16 |
|---|---|---|
| CPU plugin | Supported | Not supported |
| GPU plugin | Supported | Supported |
| FPGA plugin | Supported | Supported |
| VPU plugins | Supported | Supported |
| GNA plugin | Supported | Not supported |
| For Multi-Device and Heterogeneous execution | ||
| the supported output precision depends on the actual underlying devices. Generally, FP32 is preferable as it is most ubiquitous. |
Supported Input Layout
| Plugin | NCDHW | NCHW | NHWC | NC |
|---|---|---|---|---|
| CPU plugin | Supported | Supported | Supported | Supported |
| GPU plugin | Supported | Supported | Supported | Supported |
| FPGA plugin | Not supported | Supported | Supported | Not supported |
| VPU plugins | Not supported | Supported | Supported | Supported |
| GNA plugin | Not supported | Not supported | Not supported | Supported |
Supported Output Layout
| Number of dimensions | 5 | 4 | 3 | 2 | 1 |
|---|---|---|---|---|---|
| Layout | NCDHW | NCHW | CHW | NC | C |
For setting relevant configuration, refer to the Integrate with Customer Application New Request API topic (step 3 "Configure input and output").
Supported Layers
The following layers are supported by the plugins and by Shape Inference feature:
| Layers | GPU | CPU | VPU | GNA | FPGA | ShapeInfer |
|---|---|---|---|---|---|---|
| Abs | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| Acos | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Acosh | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Activation-Clamp | Supported | Supported*** | Supported | Supported | Supported | Supported |
| Activation-ELU | Supported | Supported*** | Supported | Not Supported | Supported | Supported |
| Activation-Exp | Supported | Supported*** | Not Supported | Supported | Not Supported | Supported |
| Activation-Leaky ReLU | Supported | Supported*** | Supported | Supported | Supported | Supported |
| Activation-Not | Supported | Supported*** | Not Supported | Not Supported | Not Supported | Supported |
| Activation-PReLU | Supported | Supported*** | Supported | Not Supported | Supported | Supported |
| Activation-ReLU | Supported | Supported*** | Supported | Supported | Supported | Supported |
| Activation-ReLU6 | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Activation-Sigmoid/Logistic | Supported | Supported*** | Supported | Supported | Not Supported | Supported |
| Activation-TanH | Supported | Supported*** | Supported | Supported | Not Supported | Supported |
| ArgMax | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| Asin | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Asinh | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Atan | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Atanh | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| BatchNormalization | Supported | Supported | Supported | Not Supported | Supported* | Supported |
| BinaryConvolution | Supported | Supported | Not Supported | Not Supported | Not Supported | Supported |
| Broadcast | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| Ceil | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Concat | Supported | Supported*** | Supported | Supported | Supported | Supported |
| Const | Supported | Supported | Supported | Supported | Not Supported | Not Supported |
| Convolution-Dilated | Supported | Supported | Supported | Not Supported | Supported | Supported |
| Convolution-Dilated 3D | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| Convolution-Grouped | Supported | Supported | Supported | Not Supported | Supported | Supported |
| Convolution-Grouped 3D | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| Convolution-Ordinary | Supported | Supported | Supported | Supported* | Supported | Supported |
| Convolution-Ordinary 3D | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| Cos | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Cosh | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Crop | Supported | Supported | Supported | Supported | Not Supported | Supported |
| CTCGreedyDecoder | Supported** | Supported** | Supported* | Not Supported | Not Supported | Supported |
| Deconvolution | Supported | Supported | Supported | Not Supported | Supported* | Supported |
| Deconvolution 3D | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| DeformableConvolution | Supported | Supported | Not Supported | Not Supported | Not Supported | Supported |
| DepthToSpace | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| DetectionOutput | Supported | Supported** | Supported* | Not Supported | Not Supported | Supported |
| Eltwise-And | Supported | Supported*** | Not Supported | Not Supported | Not Supported | Supported |
| Eltwise-Add | Supported | Supported*** | Not Supported | Not Supported | Supported | Supported |
| Eltwise-Div | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-Equal | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-FloorMod | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-Greater | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-GreaterEqual | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-Less | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-LessEqual | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-LogicalAnd | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-LogicalOr | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-LogicalXor | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-Max | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-Min | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-Mul | Supported | Supported*** | Supported | Supported | Not Supported | Supported |
| Eltwise-NotEqual | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-Pow | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-Prod | Supported | Supported*** | Supported | Supported | Not Supported | Supported |
| Eltwise-SquaredDiff | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-Sub | Supported | Supported*** | Supported | Supported | Supported | Supported |
| Eltwise-Sum | Supported | Supported*** | Supported | Supported | Supported | Supported |
| Erf | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Exp | Supported | Supported | Not Supported | Supported | Not Supported | Supported |
| FakeQuantize | Not Supported | Supported | Not Supported | Not Supported | Not Supported | Supported |
| Fill | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Flatten | Supported | Supported | Supported | Not Supported | Not Supported | Supported |
| Floor | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| FullyConnected (Inner Product) | Supported | Supported*** | Supported | Supported | Supported | Supported |
| Gather | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| GatherTree | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Gemm | Supported | Supported | Supported | Not Supported | Not Supported | Supported |
| GRN | Supported** | Supported** | Supported | Not Supported | Not Supported | Supported |
| HardSigmoid | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Interp | Supported** | Supported** | Supported | Not Supported | Not Supported | Supported* |
| Log | Supported | Supported** | Supported | Supported | Not Supported | Supported |
| LRN (Norm) | Supported | Supported | Supported | Not Supported | Supported | Supported |
| LSTMCell | Supported | Supported | Supported | Supported | Not Supported | Not Supported |
| GRUCell | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| RNNCell | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| LSTMSequence | Supported | Supported | Supported | Not Supported | Not Supported | Not Supported |
| GRUSequence | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| RNNSequence | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| LogSoftmax | Supported | Supported** | Not Supported | Not Supported | Not Supported | Not Supported |
| Memory | Not Supported | Supported | Not Supported | Supported | Not Supported | Supported |
| MVN | Supported | Supported** | Supported* | Not Supported | Not Supported | Supported |
| Neg | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| NonMaxSuppression | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Normalize | Supported | Supported** | Supported* | Not Supported | Not Supported | Supported |
| OneHot | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Pad | Supported | Supported** | Supported* | Not Supported | Not Supported | Supported |
| Permute | Supported | Supported | Supported | Supported* | Not Supported | Supported |
| Pooling(AVG,MAX) | Supported | Supported | Supported | Supported | Supported | Supported |
| Pooling(AVG,MAX) 3D | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| Power | Supported | Supported** | Supported | Supported* | Supported* | Supported |
| PowerFile | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Not Supported |
| PriorBox | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| PriorBoxClustered | Supported** | Supported** | Supported | Not Supported | Not Supported | Supported |
| Proposal | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| PSROIPooling | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| Range | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Reciprocal | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| ReduceAnd | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| ReduceL1 | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| ReduceL2 | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| ReduceLogSum | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| ReduceLogSumExp | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| ReduceMax | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| ReduceMean | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| ReduceMin | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| ReduceOr | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| ReduceProd | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| ReduceSum | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| ReduceSumSquare | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| RegionYolo | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| ReorgYolo | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| Resample | Supported | Supported** | Supported | Not Supported | Supported* | Supported |
| Reshape | Supported | Supported*** | Supported | Supported | Not Supported | Supported* |
| ReverseSequence | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| RNN | Not Supported | Supported | Supported | Not Supported | Not Supported | Not Supported |
| ROIPooling | Supported* | Supported | Supported | Not Supported | Not Supported | Supported |
| ScaleShift | Supported | Supported*** | Supported* | Supported | Supported | Supported |
| ScatterUpdate | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Select | Supported | Supported | Supported | Not Supported | Not Supported | Supported |
| Selu | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| ShuffleChannels | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Sign | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| Sin | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Sinh | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| SimplerNMS | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Slice | Supported | Supported*** | Supported | Supported | Supported* | Supported |
| SoftMax | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Softplus | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Softsign | Supported | Supported** | Not Supported | Supported | Not Supported | Supported |
| SpaceToDepth | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| SpatialTransformer | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Split | Supported | Supported*** | Supported | Supported | Supported* | Supported |
| Squeeze | Supported | Supported** | Supported | Supported | Not Supported | Supported |
| StridedSlice | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Tan | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| TensorIterator | Not Supported | Supported | Supported | Supported | Not Supported | Not Supported |
| Tile | Supported** | Supported*** | Supported | Not Supported | Not Supported | Supported |
| TopK | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Unpooling | Supported | Not Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| Unsqueeze | Supported | Supported** | Supported | Supported | Not Supported | Supported |
| Upsampling | Supported | Not Supported | Not Supported | Not Supported | Not Supported | Not Supported |
*- support is limited to the specific parameters. Refer to "Known Layers Limitation" section for the device from the list of supported.
**- support is implemented via Extensibility mechanism.
***- supports NCDHW layout.