From 670668e5937935dd494b9a508f49c1afe4adc857 Mon Sep 17 00:00:00 2001 From: Karol Blaszczak Date: Fri, 10 Mar 2023 09:25:33 +0100 Subject: [PATCH] DOCS shift to rst - devices and ARM (#16193) --- docs/OV_Runtime_UG/ov_dynamic_shapes.md | 9 +- .../supported_plugins/ARM_CPU.md | 93 +++++++++++----- .../supported_plugins/Device_Plugins.md | 104 ++++++++++-------- .../dldt_deployment_optimization_guide.md | 13 ++- .../dldt_deployment_optimization_latency.md | 10 +- 5 files changed, 151 insertions(+), 78 deletions(-) diff --git a/docs/OV_Runtime_UG/ov_dynamic_shapes.md b/docs/OV_Runtime_UG/ov_dynamic_shapes.md index f916ed733f5..8e6191353a3 100644 --- a/docs/OV_Runtime_UG/ov_dynamic_shapes.md +++ b/docs/OV_Runtime_UG/ov_dynamic_shapes.md @@ -119,6 +119,9 @@ The bounds are coded as arguments for the `ov::Dimension`: @endsphinxtabset + +@sphinxdirective + Information about bounds gives an opportunity for the inference plugin to apply additional optimizations. Using dynamic shapes assumes the plugins apply more flexible optimization approach during model compilation. It may require more time/memory for model compilation and inference. @@ -127,10 +130,14 @@ For the same reason, it is not recommended to leave dimensions as undefined, wit When specifying bounds, the lower bound is not as important as the upper one. The upper bound allows inference devices to allocate memory for intermediate tensors more precisely. It also allows using a fewer number of tuned kernels for different sizes. More precisely, benefits of specifying the lower or upper bound is device dependent. -Depending on the plugin, specifying the upper bounds can be required. For information about dynamic shapes support on different devices, refer to the [Features Support Matrix](@ref features_support_matrix). +Depending on the plugin, specifying the upper bounds can be required. For information about dynamic shapes support on different devices, refer to the :ref:`feature support table `. If the lower and upper bounds for a dimension are known, it is recommended to specify them, even if a plugin can execute a model without the bounds. +@endsphinxdirective + + + ### Setting Input Tensors Preparing a model with the `reshape` method is the first step. diff --git a/docs/OV_Runtime_UG/supported_plugins/ARM_CPU.md b/docs/OV_Runtime_UG/supported_plugins/ARM_CPU.md index 64c2fc848dc..f5f408737c9 100644 --- a/docs/OV_Runtime_UG/supported_plugins/ARM_CPU.md +++ b/docs/OV_Runtime_UG/supported_plugins/ARM_CPU.md @@ -1,56 +1,84 @@ # Arm® CPU Device {#openvino_docs_OV_UG_supported_plugins_ARM_CPU} -## Introducing the Arm® CPU Plugin -The Arm® CPU plugin is developed in order to enable deep neural networks inference on Arm® CPU, using [Compute Library](https://github.com/ARM-software/ComputeLibrary) as a backend. - -> **NOTE**: This is a community-level add-on to OpenVINO™. Intel® welcomes community participation in the OpenVINO™ ecosystem, technical questions and code contributions on community forums. However, this component has not undergone full release validation or qualification from Intel®, hence no official support is offered. - -The set of supported layers and its limitations are defined on the [Op-set specification page](https://github.com/openvinotoolkit/openvino_contrib/wiki/ARM-plugin-operation-set-specification). +@sphinxdirective -## Supported Inference Data Types + +Introducing the Arm® CPU Plugin +####################################### + +The Arm® CPU plugin is developed in order to enable deep neural networks inference on Arm® CPU, using `Compute Library `__ as a backend. + +.. note:: + + This is a community-level add-on to OpenVINO™. Intel® welcomes community participation in the OpenVINO™ ecosystem, + as well as technical questions and code contributions on community forums. However, this component has not undergone + full release validation or qualification from Intel®, hence no official support is offered. + +The set of supported layers and their limitations is defined in the +`Op-set specification page `__. + + +Supported Inference Data Types +####################################### + The Arm® CPU plugin supports the following data types as inference precision of internal primitives: - Floating-point data types: + - f32 - f16 + - Quantized data types: + - i8 (support is experimental) -[Hello Query Device C++ Sample](../../../samples/cpp/hello_query_device/README.md) can be used to print out supported data types for all detected devices. +:doc:`` can be used to print out supported data types for all detected devices. -## Supported Features +Supported Features +####################################### + +Preprocessing Acceleration ++++++++++++++++++++++++++++++++++++++++ -**Preprocessing Acceleration** The Arm® CPU plugin supports the following accelerated preprocessing operations: + - Precision conversion: - - u8 -> u16, s16, s32 - - u16 -> u8, u32 - - s16 -> u8, s32 - - f16 -> f32 + + - u8 -> u16, s16, s32 + - u16 -> u8, u32 + - s16 -> u8, s32 + - f16 -> f32 + - Transposition of tensors with dims < 5 -- Interpolation of 4D tensors with no padding (`pads_begin` and `pads_end` equal 0). +- Interpolation of 4D tensors with no padding (``pads_begin`` and ``pads_end`` equal 0). The Arm® CPU plugin supports the following preprocessing operations, however they are not accelerated: + - Precision conversion that is not mentioned above - Color conversion: - - NV12 to RGB - - NV12 to BGR - - i420 to RGB - - i420 to BGR -For more details, see the [preprocessing API guide](../preprocessing_overview.md). + - NV12 to RGB + - NV12 to BGR + - i420 to RGB + - i420 to BGR -## Supported Properties -The plugin supports the properties listed below. +For more details, see the :doc:`preprocessing API guide `. -**Read-write Properties** -In order to take effect, all parameters must be set before calling `ov::Core::compile_model()` or passed as additional argument to `ov::Core::compile_model()` +Supported Properties +####################################### + +Read-write Properties ++++++++++++++++++++++++++++++++++++++++ + +In order to take effect, all parameters must be set before calling ``ov::Core::compile_model()`` or passed as additional argument to ``ov::Core::compile_model()`` - ov::enable_profiling -**Read-only Properties** +Read-only Properties ++++++++++++++++++++++++++++++++++++++++ + - ov::supported_properties - ov::available_devices - ov::range_for_async_infer_requests @@ -59,7 +87,14 @@ In order to take effect, all parameters must be set before calling `ov::Core::co - ov::device::capabilities -## Additional Resources -* [Arm® plugin developer documentation](https://github.com/openvinotoolkit/openvino_contrib/blob/master/modules/arm_plugin/README.md). -* [How to run YOLOv4 model inference using OpenVINO™ and OpenCV on Arm®](https://opencv.org/how-to-run-yolov4-using-openvino-and-opencv-on-arm/). -* [Face recognition on Android™ using OpenVINO™ toolkit with Arm® plugin](https://opencv.org/face-recognition-on-android-using-openvino-toolkit-with-arm-plugin/). +Additional Resources +####################################### + +* `Arm® plugin developer documentation `__. +* `How to run YOLOv4 model inference using OpenVINO™ and OpenCV on Arm® `__. +* `Face recognition on Android™ using OpenVINO™ toolkit with Arm® plugin `__. + + +@endsphinxdirective + + diff --git a/docs/OV_Runtime_UG/supported_plugins/Device_Plugins.md b/docs/OV_Runtime_UG/supported_plugins/Device_Plugins.md index 87ea4ddecbd..0ee460106cb 100644 --- a/docs/OV_Runtime_UG/supported_plugins/Device_Plugins.md +++ b/docs/OV_Runtime_UG/supported_plugins/Device_Plugins.md @@ -12,78 +12,89 @@ openvino_docs_OV_UG_supported_plugins_GNA openvino_docs_OV_UG_supported_plugins_ARM_CPU -@endsphinxdirective OpenVINO™ Runtime can infer deep learning models using the following device types: -* [CPU](CPU.md) -* [GPU](GPU.md) -* [GNA](GNA.md) -* [Arm® CPU](ARM_CPU.md) +* :doc:`CPU ` +* :doc:`GPU ` +* :doc:`GNA ` +* :doc:`Arm® CPU ` -For a more detailed list of hardware, see [Supported Devices](./Supported_Devices.md) +For a more detailed list of hardware, see :doc:`Supported Devices `. -Devices similar to the ones used for benchmarking can be accessed, using [Intel® DevCloud for the Edge](https://devcloud.intel.com/edge/), a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of the OpenVINO™ Toolkit. [Learn more](https://devcloud.intel.com/edge/get_started/devcloud/) or [Register here](https://inteliot.force.com/DevcloudForEdge/s/). +Devices similar to the ones used for benchmarking can be accessed, using `Intel® DevCloud for the Edge `__, +a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of the OpenVINO™ Toolkit. +`Learn more `__ or `Register here `__. -@anchor features_support_matrix -## Feature Support Matrix + +.. _devicesupport-feature-support-matrix: + + + +Feature Support Matrix +####################################### + The table below demonstrates support of key features by OpenVINO device plugins. -| Capability | [CPU](CPU.md) | [GPU](GPU.md) | [GNA](GNA.md) |[Arm® CPU](ARM_CPU.md) | -| ---------- | --- | --- | --- | --- | -| [Heterogeneous execution](../hetero_execution.md)| Yes | Yes | No | Yes | -| [Multi-device execution](../multi_device.md) | Yes | Yes | Partial | Yes | -| [Automatic batching](../automatic_batching.md) | No | Yes | No | No | -| [Multi-stream execution](../../optimization_guide/dldt_deployment_optimization_tput.md) | Yes | Yes | No | Yes | -| [Models caching](../Model_caching_overview.md) | Yes | Partial | Yes | No | -| [Dynamic shapes](../ov_dynamic_shapes.md) | Yes | Partial | No | No | -| [Import/Export](../../../tools/compile_tool/README.md) | Yes | No | Yes | No | -| [Preprocessing acceleration](../preprocessing_overview.md) | Yes | Yes | No | Partial | -| [Stateful models](../network_state_intro.md) | Yes | No | Yes | No | -| [Extensibility](@ref openvino_docs_Extensibility_UG_Intro) | Yes | Yes | No | No | + ========================================================================================= =============== =============== =============== ======================== + Capability CPU GPU GNA Arm® CPU + ========================================================================================= =============== =============== =============== ======================== + :doc:`Heterogeneous execution ` Yes Yes No Yes + :doc:`Multi-device execution ` Yes Yes Partial Yes + :doc:`Automatic batching ` No Yes No No + :doc:`Multi-stream execution ` Yes Yes No Yes + :doc:`Models caching ` Yes Partial Yes No + :doc:`Dynamic shapes ` Yes Partial No No + :doc:`Import/Export ` Yes No Yes No + :doc:`Preprocessing acceleration ` Yes Yes No Partial + :doc:`Stateful models ` Yes No Yes No + :doc:`Extensibility ` Yes Yes No No + ========================================================================================= =============== =============== =============== ======================== For more details on plugin-specific feature limitations, see the corresponding plugin pages. -## Enumerating Available Devices -The OpenVINO Runtime API features dedicated methods of enumerating devices and their capabilities. See the [Hello Query Device C++ Sample](../../../samples/cpp/hello_query_device/README.md). This is an example output from the sample (truncated to device names only): +Enumerating Available Devices +####################################### + +The OpenVINO Runtime API features dedicated methods of enumerating devices and their capabilities. See the :doc:`Hello Query Device C++ Sample `. This is an example output from the sample (truncated to device names only): + +.. code-block:: sh + + ./hello_query_device + Available devices: + Device: CPU + ... + Device: GPU.0 + ... + Device: GPU.1 + ... + Device: GNA -```sh - ./hello_query_device - Available devices: - Device: CPU - ... - Device: GPU.0 - ... - Device: GPU.1 - ... - Device: GNA -``` A simple programmatic way to enumerate the devices and use with the multi-device is as follows: -@sphinxdirective - .. tab:: C++ .. doxygensnippet:: docs/snippets/MULTI2.cpp :language: cpp :fragment: [part2] -@endsphinxdirective -Beyond the typical "CPU", "GPU", and so on, when multiple instances of a device are available, the names are more qualified. For example, this is how two GPUs can be listed (iGPU is always GPU.0): -``` -... - Device: GPU.0 -... - Device: GPU.1 -``` +Beyond the typical "CPU", "GPU", and so on, when multiple instances of a device are available, the names are more qualified. +For example, this is how two GPUs can be listed (iGPU is always GPU.0): + +.. code-block:: sh + + ... + Device: GPU.0 + ... + Device: GPU.1 + So, the explicit configuration to use both would be "MULTI:GPU.1,GPU.0". Accordingly, the code that loops over all available devices of the "GPU" type only is as follows: -@sphinxdirective .. tab:: C++ @@ -91,7 +102,8 @@ So, the explicit configuration to use both would be "MULTI:GPU.1,GPU.0". Accordi :language: cpp :fragment: [part3] + + @endsphinxdirective - diff --git a/docs/optimization_guide/dldt_deployment_optimization_guide.md b/docs/optimization_guide/dldt_deployment_optimization_guide.md index 2efc59bf003..ac42c884d50 100644 --- a/docs/optimization_guide/dldt_deployment_optimization_guide.md +++ b/docs/optimization_guide/dldt_deployment_optimization_guide.md @@ -23,13 +23,24 @@ Therefore, optimization should start with defining the use case. For example, if It is also important to understand how the full-stack application would use the inference component "end-to-end." For example, to know what stages need to be orchestrated to save workload devoted to fetching and preparing input data. + + + For more information on this topic, see the following articles: -* [feature support by device](@ref features_support_matrix) + +@sphinxdirective +* :ref:`feature support by device ` +@endsphinxdirective + + * [Inputs Pre-processing with the OpenVINO](@ref inputs_pre_processing) * [Async API](@ref async_api) * [The 'get_tensor' Idiom](@ref tensor_idiom) * For variably-sized inputs, consider [dynamic shapes](../OV_Runtime_UG/ov_dynamic_shapes.md) + + + See the [latency](./dldt_deployment_optimization_latency.md) and [throughput](./dldt_deployment_optimization_tput.md) optimization guides, for **use-case-specific optimizations** ## Writing Performance-Portable Inference Applications diff --git a/docs/optimization_guide/dldt_deployment_optimization_latency.md b/docs/optimization_guide/dldt_deployment_optimization_latency.md index 7cc93b2e4fc..f3759d63b4a 100644 --- a/docs/optimization_guide/dldt_deployment_optimization_latency.md +++ b/docs/optimization_guide/dldt_deployment_optimization_latency.md @@ -22,7 +22,15 @@ Typically, human expertise is required to get more "throughput" out of the devic > **NOTE**: [OpenVINO performance hints](../OV_Runtime_UG/performance_hints.md) is a recommended way for performance configuration, which is both device-agnostic and future-proof. -When multiple models are to be used simultaneously, consider running inference on separate devices for each of them. Finally, when multiple models are executed in parallel on a device, using additional `ov::hint::model_priority` may help to define relative priorities of the models. Refer to the documentation on the [matrix features support for OpenVINO devices](@ref features_support_matrix) to check if your device supports the feature. + +@sphinxdirective +* feature support by device + + +When multiple models are to be used simultaneously, consider running inference on separate devices for each of them. Finally, when multiple models are executed in parallel on a device, using additional ``ov::hint::model_priority`` may help to define relative priorities of the models. Refer to the documentation on the :ref:`OpenVINO feature support for devices ` to check if your device supports the feature. + +@endsphinxdirective + **First-Inference Latency and Model Load/Compile Time**