940eb43095
* [IE CLDNN] Memory allocation optimizations (#2178) * [GNA] Safety fixes (#2193) * LSTMCell test [GNA] LSTMCell fix for GNA (#2216) * [GNA] fix scale factor calculation for unfused bias after fc (2021.1) (#2195) * [GNA] fix scale factor calculation for unfused bias after fc * change check * add test * apply requested changes * cpplint fix * apply test changes * modify model for test to match ::op:: * [LPT] Copy constant with several outputs before blob update (#2197) * [LPT] Copy constant implementation * [LPT] the same Constant ops as FQ interval boundaries * [Scripts] Fixing issue with exporting path-like env when it undef (#2164) * setupvars.sh: Added logic for exporting path env in case if it not defined * setupvars: Removed duplicated colon * Kept quotes where they were * setupvars: updated copyrights * FakeQuantize + Mul fusion (#2133) * FQ+Mul fusion transform skeleton * FQ+Mul fusion transform tests prep * Basic UT for the transform * Basic implementation of the transform * Parametrized UTs for FQMul transform * Parametrization of FQ+Mul UTs * Make sure that the shapes of constants match * Check if the mul constant matches FQ data * CentOs compilation error fix * PR feedback and adjusted tests * NHWC layout of the mul constant * UT: FQ output limits 4D * Redundant CF pass removed * Rewrite the graph in a different way * Shape checking infrastructure skeleton * Handle some negative cases * Check the rt info in the fusion test * Fuse all Mul nodes detected after FQ node * Dont cast the original FQ node * Dont throw if CF fails in new output range calculation * More UTs * Accept any type of input to FQ in the transformation * Test the fusion when all FQ inputs are non-const * Fusion test when only one output limit is const * Extend error message (#2174) * some nGraph KW fixes (#2176) * Removed redundant methods * Fixed KW for linux * Fix QueryNetwork for networks with KSO (#2202) * Added a test to reproduce QueryNetwork with KSO * Fixed QueryNetwork for networks with KSO * Added additional test * Fixed output names for case with redundant ops before result (#2209) * [IE][VPU]: Workaround to support parameter Beta for layer Swish (#2207) * Workaround to full support Swish layer. It is faster than native Swish for now. * [IE][VPU]: Remove the second call of ngraph::CommonOptimizations (#2221) * Remove the second call of ngraph::CommonOptimizations in myriad plugin * Reuse code with vpu ngraph transformations * Duplicate PR 2167 for release branch: GatherTree description was extended and outdated link fixed (#2235) * add more alrifications to description * move clarification to comment * pseudo code become more accurate * review changes * Add exposing function signatures via Cython (#2244) * [DOC] Reshape feature (#2194) * [IE][VPU][OpenCL] 2021.1 release compiler (#2189) * Statically analyzed issues. (#2261) * [IE][VPU]: Fix K propagation through Reshape (2021.1) (#2180) * Fix K propagation through Reshape * Add test cases * Revert "[IE TESTS] dynavic batch for mvn layer (#1010)" (#2256) This reverts commit2e3378c50f. * Fixed KW warning and review issues (#2262) * [IE][VPU]: update firmware 1381 (#2236) * Reverting devicePriorities to be vector and respect the order, as opposed to the incorrect (recent?) refactoring that introduced the unordered_map that effectively ignores the priorities (#2251) * update OpenCV version to 4.5.0 (#2260) * Add VPUX configuration to compile_tool (#2248) * [IE][TESTS] Fix compareRawBuffers and compareBlobData methods (#2246) Use `<=` comparison instead of `<` with thresholds. This allows to use `0` threshold for bit-exact comparison. * [IE][VPU]: KW fixes (#2186) * Some KW fixes * Fix printTo in vpu ngraph transformations * Fix for static PartialShape detection algorithm (#2177) * Fixes for Interpolate-4. (#2281) * Update get_ov_update_message.py (#2286) * Clone a specific tag for pybind11 (#2296) * [Scripts] Fix setting PYTHONPATH logic (#2305) * setupvars.sh: Added logic for exporting path env in case if it not defined * setupvars: Removed duplicated colon * install_openvino_dependencies: Updated copyrights setupvars.bat: Updated notification about incorrect Python version. Removed checking ICC2019 setupvars.sh: Removed logic with choosing higher version of installed Python. Added dynamic detecting python3 major and minor version for setting path. Add checking minimum required Python version(now 3.6) * Added python3-gi package and fixed libglib2.0-0 package location. (#2294) * [IE TESTS] CoreThreading_LoadNetwork tests were disabled for GPU plugin (#2245) (#2283) * setupvars: Updated notifications, fixed calling python in Windows case (#2318) * Updated operations specification documents (2021.1) (#2268) * Updated documentation structure and remove incorrect added files for Acosh-1, Asinh-1 and Atanh-1 * Fixed broken links * Fixed c samples build (#2278) (#2304) * Fixed c samples build fixed CVS-38816 - Failure to build samples in C * Fixed issue with gflags * Revert "[IE][VPU]: Fix K propagation through Reshape (2021.1) (#2180)" (#2322) This reverts commitd604a03ac0. * Added ONNX Resize-11 and ONNX Resize-13 to supported frameworks layers list. (#2325) * Implement `run_executable.py` to run `TimeTests` several times (#2125) (#2188) CI passed * install_NEO_OCL_driver: Updated exit codes, messages. Updated way to remove old driver on Ubuntu (#2333) * Bump cmake version to 3.13 (#2339) * install_NEO_OCL_driver: Added checking of installed packages before trying to remove them. Added quotes for echo. (#2350) * convert to doxygen comments * add doxygen doc build configurations (#2191) Co-authored-by: Nikolay Tyukaev <ntyukaev_lo@jenkins.inn.intel.com> * [DOCS] Added an evaluate method for custom operation (#2272) * Added an evaluate method for custom operation * Fixed comments * Downgrade cmake for samples (#2372) * Downgrade cmake for samples Downgraded cmake version to default version for Ubuntu 18.04 * Updated supported python version The minimal python version in 2021.1 is 3.5 * Added notes about cmake requirements for samples and demo * Install dependency refactoring. (#2381) * Updated Transformation development doc (#2370) * Delete xfail for resolved known issue (#2385) * Fix layout links for dl streamer and c api (#2375) * fix layouts * change the dl-streamer link Co-authored-by: Nikolay Tyukaev <ntyukaev_lo@jenkins.inn.intel.com> * Added link options for cross-compilation (#2397) * Added new GSG for macOS, made minor changes in Windows GSG (#2070) (#2405) * Added new GSG for macOS, made minor changes in Windows GSG * Update get_started_macos.md Co-authored-by: Anastasiya Ageeva <anastasiya.ageeva@intel.com> * Fixed docs build on Windows (#2383) * layouts and code comments * Replace absolute links to docs.openvinotoolkit.org by relative ones (#2439) * Replaced direct links to docs.openvinotoolkit.org with relative links * Replaced direct links to docs.openvinotoolkit.org with relative links. Added GSGs for Win and macOS * Minor fixes in GSGs * Replaced direct links to docs.openvinotoolkit.org with relative links * Removed links to OpenVINO markdown files that contain anchor - they don't work in the current implementation of the doc process * Fixed Notes * Removed links to OpenVINO markdown files that contain anchor - they don't work in the current implementation of the doc process * fixed link to installing-openvino-linux.md * Update the menu to align with POT doc headers (#2433) * Update the menu to align with POT doc headers It changes the menu to align with Post-training Optimization Toolkit documentation titles. * Corrected one title Run Examples => How to Run Examples * Added closing braсket (#2466) Fixed syntax error (b4b03b1) * Remove the deprecation notice (#2314) * Removed deprecation notice * Removed the note from other files * [DOCS] Update Installation Guide - GPU steps (#2308) * Initial commit * fixing lists * Update installing-openvino-linux.md * Get rid of the note * Added the scrrenshot * Update installing-openvino-linux.md * fixes * separate layout * [Docs] Update MO What's new description (#2481) * Azure CI: Add separated pipelines for Windows, Linux, Mac * Feature/azaytsev/benchmarks 2021 1 (#2501) * Initial changes for 2021.1 * Inserted Graphtool scripts, updated configurations info * Updated FAQ and minor changes to performance_benchmarks.md * Updated for 2021.1 * Updated * incorporated review comments * incorporated review comments for FAQ * fixed link * Update build-instruction.md for MacOsX (#2457) * Update build-instruction.md for MacOsX * Removed call of install_dependencies.sh from the steps * Changed layouts * Feature/azaytsev/cvs-38240 (#2469) * Updated for 2020 version, replaced Ubuntu 16.04 with Ubuntu 20.04 * Updated the release package numbers * Removed FPGA from the documentation * Updated according to the comments in the ticket CVS-37827 (#2448) * Updated according to CVS-38225 * some changes * Update docs for speech libs and demos (#2518) * Made changes to benchmarks according to review comments * Remove `--collect_results_only` (#2523) * Remove `--collect_results_only` from MemCheckTests * Remove CLI keys from README * Added logo info to the Legal_Information, updated Ubuntu, CentOS supported versions * Updated supported Intel® Core™ processors list * Fixed table formatting * [Jenkinsfile] Bump infra (#2546) * [GNA] Documentation updates for 2021.1 (#2460) * [GNA] Documentation updates for 2021.1 * Take Mike's comments into account * More fixes according to review * Fix processor generation names * update api layouts * Added new index page with overview * Changed CMake and Python versions * Fixed links * some layout changes * some layout changes * nGraph Python API tutorial (#2500) * nGraph Python API tutorial * Tweaks * Code review comments * Code review comments * some layout changes * COnverted svg images to png * layouts * update layout * Added a label for nGraph_Python_API.md * fixed links * Fixed image * First draft of nGraph documentation (#2271) * First draft of nGraph documentation * updated according to review comments * Updated * Reviewed the nGraph Transformation section, added missing images * Update nGraph_dg.md * Delete python_api.md Removed since there is already the nGraph_Python_API.md document with a comprehensive overview. Co-authored-by: Andrey Zaytsev <andrey.zaytsev@intel.com> Co-authored-by: CCR\avladimi <anastasiya.ageeva@intel.com> * Feature/azaytsev/docs 2021 1 (#2560) * Removed FPGA from the documentation * Updated according to CVS-38225 * Added logo info to the Legal_Information, updated Ubuntu, CentOS supported versions * Updated supported Intel® Core™ processors list * Added new index page with overview * Changed CMake and Python versions * Fixed links * COnverted svg images to png * Added a label for nGraph_Python_API.md * fixed links * Fixed image * Update SW requirements in build instructions and change latest release to 2021.1 (#2565) * removed links to ../IE_DG/Introduction.md * Removed links to tools overview page as removed * some changes * Remove link to Integrate_your_kernels_into_IE.md * remove openvino_docs_IE_DG_Graph_debug_capabilities from layout as it was removed * Fixed links to images (#2569) * update layouts * Added deprecation note for PassConfig class (#2593) * Post-release fixes and installation path changes * Added pip install documentation (#2465) * Added pip install documentation * Change references * tiny fixes of links * Update installing-openvino-pip.md Co-authored-by: Alina Alborova <alina.alborova@intel.com> * Update OpenVino ONNX CI check (#2599) * Update OpenVino ONNX CI * Change parallel execution to single * Enlarge timeout * Remove timeout * Add timeout to test execution * Added PIP installation and Build from Source to the layout * Fixed formatting issue, removed broken link * Renamed section EXAMPLES to RESOURCES according to review comments * add mo faq navigation by url param * Skip hanging test case of OpenVino ONNX CI (#2608) * Update OpenVino ONNX CI * Change parallel execution to single * Enlarge timeout * Remove timeout * Add timeout to test execution * Skip hanging test * Add description to skip issue * Removed DLDT description * Replaced wrong links * MInor fix for path to the cpp samples * fixes * Update ops.py * Fix style * Improve pip installation guide (#2644) * Improve pip installation guide * Updated after comments * Feature/ntyukaev/separate layout (#2629) * convert to doxygen comments * layouts and code comments * separate layout * Changed layouts * Removed FPGA from the documentation * Updated according to CVS-38225 * some changes * Made changes to benchmarks according to review comments * Added logo info to the Legal_Information, updated Ubuntu, CentOS supported versions * Updated supported Intel® Core™ processors list * Fixed table formatting * update api layouts * Added new index page with overview * Changed CMake and Python versions * Fixed links * some layout changes * some layout changes * some layout changes * COnverted svg images to png * layouts * update layout * Added a label for nGraph_Python_API.md * fixed links * Fixed image * removed links to ../IE_DG/Introduction.md * Removed links to tools overview page as removed * some changes * Remove link to Integrate_your_kernels_into_IE.md * remove openvino_docs_IE_DG_Graph_debug_capabilities from layout as it was removed * update layouts * Post-release fixes and installation path changes * Added PIP installation and Build from Source to the layout * Fixed formatting issue, removed broken link * Renamed section EXAMPLES to RESOURCES according to review comments * add mo faq navigation by url param * Removed DLDT description * Replaced wrong links * MInor fix for path to the cpp samples * fixes * Update ops.py * Fix style Co-authored-by: Nikolay Tyukaev <ntyukaev_lo@jenkins.inn.intel.com> Co-authored-by: Tyukaev <nikolay.tyukaev@intel.com> Co-authored-by: aalborov <alina.alborova@intel.com> Co-authored-by: Rafal Blaczkowski <rafal.blaczkowski@intel.com> Co-authored-by: Alexander Zhogov <alexander.zhogov@intel.com> * Fixed CVS-35316 (#2072) (#2670) Co-authored-by: Anastasiya Ageeva <anastasiya.ageeva@intel.com> * [install_dependencies.sh] install latest cmake if current version is lower 3.13 (#2695) (#2701) * [install_dependencies.sh] install latest cmake if current version is lower 3.13 * add shellcheck for Ubuntu * install python 2.7 for Ubuntu * Removed redundant file * Exclude files that we didn't changed from merging Co-authored-by: Sergey Shlyapnikov <sergey.shlyapnikov@intel.com> Co-authored-by: Denis Orlov <denis.orlov@intel.com> Co-authored-by: Kamil Magierski <kamil.magierski@intel.com> Co-authored-by: Anna Alberska <anna.alberska@intel.com> Co-authored-by: Edward Shogulin <edward.shogulin@intel.com> Co-authored-by: Artyom Anokhov <artyom.anokhov@intel.com> Co-authored-by: Tomasz Dołbniak <tomasz.dolbniak@intel.com> Co-authored-by: Ilya Churaev <ilya.churaev@intel.com> Co-authored-by: Roman Vyunov (Intel) <roman.vyunov@intel.com> Co-authored-by: Maksim Doronin <maksim.doronin@intel.com> Co-authored-by: Svetlana Dolinina <svetlana.a.dolinina@intel.com> Co-authored-by: Evgeny Talanin <evgeny.talanin@intel.com> Co-authored-by: Evgenya Stepyreva <evgenya.stepyreva@intel.com> Co-authored-by: Maxim Kurin <maxim.kurin@intel.com> Co-authored-by: Nikolay Shchegolev <nikolay.shchegolev@intel.com> Co-authored-by: Andrew Bakalin <andrew.bakalin@intel.com> Co-authored-by: Gorokhov Dmitriy <dmitry.gorokhov@intel.com> Co-authored-by: Evgeny Latkin <evgeny.latkin@intel.com> Co-authored-by: Maxim Shevtsov <maxim.y.shevtsov@intel.com> Co-authored-by: Alexey Suhov <alexey.suhov@intel.com> Co-authored-by: Alexander Novak <sasha-novak@yandex.ru> Co-authored-by: Vladislav Vinogradov <vlad.vinogradov@intel.com> Co-authored-by: Vladislav Volkov <vladislav.volkov@intel.com> Co-authored-by: Vladimir Gavrilov <vladimir.gavrilov@intel.com> Co-authored-by: Zoe Cayetano <zoe.cayetano@intel.com> Co-authored-by: Dmitrii Denisov <dmitrii.denisov@intel.com> Co-authored-by: Irina Efode <irina.efode@intel.com> Co-authored-by: Evgeny Lazarev <evgeny.lazarev@intel.com> Co-authored-by: Mikhail Ryzhov <mikhail.ryzhov@intel.com> Co-authored-by: Vitaliy Urusovskij <vitaliy.urusovskij@intel.com> Co-authored-by: Nikolay Tyukaev <ntyukaev_lo@jenkins.inn.intel.com> Co-authored-by: Nikolay Tyukaev <nikolay.tyukaev@intel.com> Co-authored-by: Gleb Kazantaev <gleb.kazantaev@intel.com> Co-authored-by: Rafal Blaczkowski <rafal.blaczkowski@intel.com> Co-authored-by: Ilya Lavrenov <ilya.lavrenov@intel.com> Co-authored-by: Anastasiya Ageeva <anastasiya.ageeva@intel.com> Co-authored-by: Maksim Proshin <mvproshin@gmail.com> Co-authored-by: Alina Alborova <alina.alborova@intel.com> Co-authored-by: Maxim Vafin <maxim.vafin@intel.com> Co-authored-by: azhogov <alexander.zhogov@intel.com> Co-authored-by: Alina Kladieva <alina.kladieva@intel.com> Co-authored-by: Michał Karzyński <4430709+postrational@users.noreply.github.com> Co-authored-by: Anton Romanov <anton.romanov@intel.com>
24 KiB
24 KiB
Supported Devices
The Inference Engine can infer models in different formats with various input and output formats. This section provides supported and optimal configurations per device.
Note
: With OpenVINO™ 2020.4 release, Intel® Movidius™ Neural Compute Stick is no longer supported.
The Inference Engine provides unique capabilities to infer deep learning models on the following device types with corresponding plugins:
| Plugin | Device types |
|---|---|
| GPU plugin | Intel® Processor Graphics, including Intel® HD Graphics and Intel® Iris® Graphics |
| CPU plugin | Intel® Xeon® with Intel® Advanced Vector Extensions 2 (Intel® AVX2), Intel® Advanced Vector Extensions 512 (Intel® AVX-512), and AVX512_BF16, Intel® Core™ Processors with Intel® AVX2, Intel® Atom® Processors with Intel® Streaming SIMD Extensions (Intel® SSE) |
| VPU plugins (available in the Intel® Distribution of OpenVINO™ toolkit) | Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X, Intel® Vision Accelerator Design with Intel® Movidius™ VPUs |
| GNA plugin (available in the Intel® Distribution of OpenVINO™ toolkit) | Intel® Speech Enabling Developer Kit, Amazon Alexa* Premium Far-Field Developer Kit, Intel® Pentium® Silver J5005 Processor, Intel® Pentium® Silver N5000 Processor, Intel® Celeron® J4005 Processor, Intel® Celeron® J4105 Processor, Intel® Celeron® Processor N4100, Intel® Celeron® Processor N4000, Intel® Core™ i3-8121U Processor, Intel® Core™ i7-1065G7 Processor, Intel® Core™ i7-1060G7 Processor, Intel® Core™ i5-1035G4 Processor, Intel® Core™ i5-1035G7 Processor, Intel® Core™ i5-1035G1 Processor, Intel® Core™ i5-1030G7 Processor, Intel® Core™ i5-1030G4 Processor, Intel® Core™ i3-1005G1 Processor, Intel® Core™ i3-1000G1 Processor, Intel® Core™ i3-1000G4 Processor |
| Multi-Device plugin | Multi-Device plugin enables simultaneous inference of the same network on several Intel® devices in parallel |
| Heterogeneous plugin | Heterogeneous plugin enables automatic inference splitting between several Intel® devices (for example if a device doesn't support certain layers). |
Supported Configurations
The Inference Engine can inference models in different formats with various input and output formats. This chapter provides supported and optimal configurations for each plugin.
Terminology
| Acronym/Term | Description |
|---|---|
| DL | Deep Learning |
| FP32 format | Single-precision floating-point format |
| BF16 format | Brain floating-point format |
| FP16 format | Half-precision floating-point format |
| I16 format | 2-byte signed integer format |
| I8 format | 1-byte signed integer format |
| U16 format | 2-byte unsigned integer format |
| U8 format | 1-byte unsigned integer format |
NHWC, NCHW - Image data layout. Refers to the representation of batches of images. NCDHW - Images sequence data layout.
- N - Number of images in a batch
- D - Depth. Depend on model it could be spatial or time dimension
- H - Number of pixels in the vertical dimension
- W - Number of pixels in the horizontal dimension
- C - Number of channels
CHW, NC, C - Tensor memory layout. For example, the CHW value at index (c,h,w) is physically located at index (c*H+h)*W+w, for others by analogy
Supported Model Formats
| Plugin | FP32 | FP16 | I8 |
|---|---|---|---|
| CPU plugin | Supported and preferred | Supported | Supported |
| GPU plugin | Supported | Supported and preferred | Supported* |
| VPU plugins | Not supported | Supported | Not supported |
| GNA plugin | Supported | Supported | Not supported |
* - currently, only limited set of topologies might benefit from enabling I8 model on GPU |
|||
| For Multi-Device and Heterogeneous execution | |||
| the supported models formats depends on the actual underlying devices. Generally, FP16 is preferable as it is most ubiquitous and performant. |
Supported Input Precision
| Plugin | FP32 | FP16 | U8 | U16 | I8 | I16 |
|---|---|---|---|---|---|---|
| CPU plugin | Supported | Not supported | Supported | Supported | Not supported | Supported |
| GPU plugin | Supported | Supported* | Supported* | Supported* | Not supported | Supported* |
| VPU plugins | Supported | Supported | Supported | Not supported | Not supported | Not supported |
| GNA plugin | Supported | Not supported | Supported | Not supported | Supported | Supported |
* - Supported via SetBlob only, GetBlob returns FP32
For Multi-Device and Heterogeneous execution
the supported input precision depends on the actual underlying devices. Generally, U8 is preferable as it is most ubiquitous.
Supported Output Precision
| Plugin | FP32 | FP16 |
|---|---|---|
| CPU plugin | Supported | Not supported |
| GPU plugin | Supported | Supported |
| VPU plugins | Supported | Supported |
| GNA plugin | Supported | Not supported |
| For Multi-Device and Heterogeneous execution | ||
| the supported output precision depends on the actual underlying devices. Generally, FP32 is preferable as it is most ubiquitous. |
Supported Input Layout
| Plugin | NCDHW | NCHW | NHWC | NC |
|---|---|---|---|---|
| CPU plugin | Supported | Supported | Supported | Supported |
| GPU plugin | Supported | Supported | Supported | Supported |
| VPU plugins | Not supported | Supported | Supported | Supported |
| GNA plugin | Not supported | Supported | Supported | Supported |
Supported Output Layout
| Number of dimensions | 5 | 4 | 3 | 2 | 1 |
|---|---|---|---|---|---|
| Layout | NCDHW | NCHW | CHW | NC | C |
For setting relevant configuration, refer to the Integrate with Customer Application New Request API topic (step 3 "Configure input and output").
Supported Layers
The following layers are supported by the plugins and by Shape Inference feature:
| Layers | GPU | CPU | VPU | GNA | ShapeInfer |
|---|---|---|---|---|---|
| Abs | Supported | Supported** | Supported | Not Supported | Supported |
| Acos | Supported | Supported** | Not Supported | Not Supported | Supported |
| Acosh | Supported | Supported** | Not Supported | Not Supported | Supported |
| Activation-Clamp | Supported | Supported*** | Supported | Supported | Supported |
| Activation-ELU | Supported | Supported*** | Supported | Not Supported | Supported |
| Activation-Exp | Supported | Supported*** | Not Supported | Supported | Supported |
| Activation-Leaky ReLU | Supported | Supported*** | Supported | Supported | Supported |
| Activation-Not | Supported | Supported*** | Not Supported | Not Supported | Supported |
| Activation-PReLU | Supported | Supported*** | Supported | Not Supported | Supported |
| Activation-ReLU | Supported | Supported*** | Supported | Supported | Supported |
| Activation-ReLU6 | Supported | Supported*** | Supported | Not Supported | Supported |
| Activation-Sigmoid/Logistic | Supported | Supported*** | Supported | Supported | Supported |
| Activation-TanH | Supported | Supported*** | Supported | Supported | Supported |
| ArgMax | Supported | Supported** | Supported | Not Supported | Supported |
| Asin | Supported | Supported** | Not Supported | Not Supported | Supported |
| Asinh | Supported | Supported** | Not Supported | Not Supported | Supported |
| Atan | Supported | Supported** | Not Supported | Not Supported | Supported |
| Atanh | Supported | Supported** | Not Supported | Not Supported | Supported |
| BatchNormalization | Supported | Supported | Supported | Not Supported | Supported |
| BinaryConvolution | Supported | Supported | Not Supported | Not Supported | Supported |
| Broadcast | Supported | Supported** | Supported | Not Supported | Supported |
| Ceil | Supported | Supported** | Not Supported | Not Supported | Supported |
| Concat | Supported | Supported*** | Supported | Supported | Supported |
| Const | Supported | Supported | Supported | Supported | Not Supported |
| Convolution-Dilated | Supported | Supported | Supported | Not Supported | Supported |
| Convolution-Dilated 3D | Supported | Supported | Not Supported | Not Supported | Not Supported |
| Convolution-Grouped | Supported | Supported | Supported | Not Supported | Supported |
| Convolution-Grouped 3D | Supported | Supported | Not Supported | Not Supported | Not Supported |
| Convolution-Ordinary | Supported | Supported | Supported | Supported* | Supported |
| Convolution-Ordinary 3D | Supported | Supported | Not Supported | Not Supported | Not Supported |
| Cos | Supported | Supported** | Not Supported | Not Supported | Supported |
| Cosh | Supported | Supported** | Not Supported | Not Supported | Supported |
| Crop | Supported | Supported | Supported | Supported | Supported |
| CTCGreedyDecoder | Supported** | Supported** | Supported* | Not Supported | Supported |
| Deconvolution | Supported | Supported | Supported | Not Supported | Supported |
| Deconvolution 3D | Supported | Supported | Not Supported | Not Supported | Not Supported |
| DeformableConvolution | Supported | Supported | Not Supported | Not Supported | Supported |
| DepthToSpace | Supported | Supported** | Not Supported | Not Supported | Supported |
| DetectionOutput | Supported | Supported** | Supported* | Not Supported | Supported |
| Eltwise-And | Supported | Supported*** | Not Supported | Not Supported | Supported |
| Eltwise-Add | Supported | Supported*** | Not Supported | Not Supported | Supported |
| Eltwise-Div | Supported | Supported*** | Supported | Not Supported | Supported |
| Eltwise-Equal | Supported | Supported*** | Supported | Not Supported | Supported |
| Eltwise-FloorMod | Supported | Supported*** | Supported | Not Supported | Supported |
| Eltwise-Greater | Supported | Supported*** | Supported | Not Supported | Supported |
| Eltwise-GreaterEqual | Supported | Supported*** | Supported | Not Supported | Supported |
| Eltwise-Less | Supported | Supported*** | Supported | Not Supported | Supported |
| Eltwise-LessEqual | Supported | Supported*** | Supported | Not Supported | Supported |
| Eltwise-LogicalAnd | Supported | Supported*** | Supported | Not Supported | Supported |
| Eltwise-LogicalOr | Supported | Supported*** | Supported | Not Supported | Supported |
| Eltwise-LogicalXor | Supported | Supported*** | Supported | Not Supported | Supported |
| Eltwise-Max | Supported | Supported*** | Supported | Not Supported | Supported |
| Eltwise-Min | Supported | Supported*** | Supported | Not Supported | Supported |
| Eltwise-Mul | Supported | Supported*** | Supported | Supported | Supported |
| Eltwise-NotEqual | Supported | Supported*** | Supported | Not Supported | Supported |
| Eltwise-Pow | Supported | Supported*** | Supported | Not Supported | Supported |
| Eltwise-Prod | Supported | Supported*** | Supported | Supported | Supported |
| Eltwise-SquaredDiff | Supported | Supported*** | Supported | Not Supported | Supported |
| Eltwise-Sub | Supported | Supported*** | Supported | Supported | Supported |
| Eltwise-Sum | Supported | Supported*** | Supported | Supported | Supported |
| Erf | Supported | Supported** | Not Supported | Not Supported | Supported |
| Exp | Supported | Supported | Not Supported | Supported | Supported |
| FakeQuantize | Not Supported | Supported | Not Supported | Not Supported | Supported |
| Fill | Not Supported | Supported** | Not Supported | Not Supported | Supported |
| Flatten | Supported | Supported | Supported | Not Supported | Supported |
| Floor | Supported | Supported** | Not Supported | Not Supported | Supported |
| FullyConnected (Inner Product) | Supported | Supported*** | Supported | Supported | Supported |
| Gather | Supported | Supported** | Supported | Not Supported | Supported |
| GatherTree | Not Supported | Supported** | Not Supported | Not Supported | Supported |
| Gemm | Supported | Supported | Supported | Not Supported | Supported |
| GRN | Supported** | Supported** | Supported | Not Supported | Supported |
| HardSigmoid | Supported | Supported** | Not Supported | Not Supported | Supported |
| Interp | Supported** | Supported** | Supported | Not Supported | Supported* |
| Log | Supported | Supported** | Supported | Supported | Supported |
| LRN (Norm) | Supported | Supported | Supported | Not Supported | Supported |
| LSTMCell | Supported | Supported | Supported | Supported | Not Supported |
| GRUCell | Supported | Supported | Not Supported | Not Supported | Not Supported |
| RNNCell | Supported | Supported | Not Supported | Not Supported | Not Supported |
| LSTMSequence | Supported | Supported | Supported | Not Supported | Not Supported |
| GRUSequence | Supported | Supported | Not Supported | Not Supported | Not Supported |
| RNNSequence | Supported | Supported | Not Supported | Not Supported | Not Supported |
| LogSoftmax | Supported | Supported** | Not Supported | Not Supported | Not Supported |
| Memory | Not Supported | Supported | Not Supported | Supported | Supported |
| MVN | Supported | Supported** | Supported* | Not Supported | Supported |
| Neg | Supported | Supported** | Not Supported | Not Supported | Supported |
| NonMaxSuppression | Not Supported | Supported** | Not Supported | Not Supported | Supported |
| Normalize | Supported | Supported** | Supported* | Not Supported | Supported |
| OneHot | Supported | Supported** | Not Supported | Not Supported | Supported |
| Pad | Supported | Supported** | Supported* | Not Supported | Supported |
| Permute | Supported | Supported | Supported | Supported* | Supported |
| Pooling(AVG,MAX) | Supported | Supported | Supported | Supported | Supported |
| Pooling(AVG,MAX) 3D | Supported | Supported | Not Supported | Not Supported | Not Supported |
| Power | Supported | Supported** | Supported | Supported* | Supported |
| PowerFile | Not Supported | Supported** | Not Supported | Not Supported | Not Supported |
| PriorBox | Supported | Supported** | Supported | Not Supported | Supported |
| PriorBoxClustered | Supported** | Supported** | Supported | Not Supported | Supported |
| Proposal | Supported | Supported** | Supported | Not Supported | Supported |
| PSROIPooling | Supported | Supported** | Supported | Not Supported | Supported |
| Range | Not Supported | Supported** | Not Supported | Not Supported | Supported |
| Reciprocal | Supported | Supported** | Not Supported | Not Supported | Supported |
| ReduceAnd | Supported | Supported** | Not Supported | Not Supported | Supported |
| ReduceL1 | Supported | Supported** | Not Supported | Not Supported | Supported |
| ReduceL2 | Supported | Supported** | Not Supported | Not Supported | Supported |
| ReduceLogSum | Supported | Supported** | Not Supported | Not Supported | Supported |
| ReduceLogSumExp | Supported | Supported** | Not Supported | Not Supported | Supported |
| ReduceMax | Supported | Supported** | Not Supported | Not Supported | Supported |
| ReduceMean | Supported | Supported** | Not Supported | Not Supported | Supported |
| ReduceMin | Supported | Supported** | Not Supported | Not Supported | Supported |
| ReduceOr | Supported | Supported** | Not Supported | Not Supported | Supported |
| ReduceProd | Supported | Supported** | Not Supported | Not Supported | Supported |
| ReduceSum | Supported | Supported** | Not Supported | Not Supported | Supported |
| ReduceSumSquare | Supported | Supported** | Not Supported | Not Supported | Supported |
| RegionYolo | Supported | Supported** | Supported | Not Supported | Supported |
| ReorgYolo | Supported | Supported** | Supported | Not Supported | Supported |
| Resample | Supported | Supported** | Supported | Not Supported | Supported |
| Reshape | Supported | Supported*** | Supported | Supported | Supported* |
| ReverseSequence | Supported | Supported** | Supported | Not Supported | Supported |
| RNN | Not Supported | Supported | Supported | Not Supported | Not Supported |
| ROIPooling | Supported* | Supported | Supported | Not Supported | Supported |
| ScaleShift | Supported | Supported*** | Supported* | Supported | Supported |
| ScatterUpdate | Not Supported | Supported** | Not Supported | Not Supported | Supported |
| Select | Supported | Supported | Supported | Not Supported | Supported |
| Selu | Supported | Supported** | Not Supported | Not Supported | Supported |
| ShuffleChannels | Supported | Supported** | Not Supported | Not Supported | Supported |
| Sign | Supported | Supported** | Supported | Not Supported | Supported |
| Sin | Supported | Supported** | Not Supported | Not Supported | Supported |
| Sinh | Supported | Supported** | Not Supported | Not Supported | Supported |
| SimplerNMS | Supported | Supported** | Not Supported | Not Supported | Supported |
| Slice | Supported | Supported*** | Supported | Supported | Supported |
| SoftMax | Supported | Supported*** | Supported | Not Supported | Supported |
| Softplus | Supported | Supported** | Not Supported | Not Supported | Supported |
| Softsign | Supported | Supported** | Not Supported | Supported | Supported |
| SpaceToDepth | Not Supported | Supported** | Not Supported | Not Supported | Supported |
| SpatialTransformer | Not Supported | Supported** | Not Supported | Not Supported | Supported |
| Split | Supported | Supported*** | Supported | Supported | Supported |
| Squeeze | Supported | Supported** | Supported | Supported | Supported |
| StridedSlice | Supported | Supported** | Not Supported | Not Supported | Supported |
| Tan | Supported | Supported** | Not Supported | Not Supported | Supported |
| TensorIterator | Not Supported | Supported | Supported | Supported | Not Supported |
| Tile | Supported** | Supported*** | Supported | Not Supported | Supported |
| TopK | Supported | Supported** | Not Supported | Not Supported | Supported |
| Unpooling | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| Unsqueeze | Supported | Supported** | Supported | Supported | Supported |
| Upsampling | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
*- support is limited to the specific parameters. Refer to "Known Layers Limitation" section for the device from the list of supported.
**- support is implemented via Extensibility mechanism.
***- supports NCDHW layout.