From e4f2d0c5a7136896a7fbe272f5c2f3b91b5b9017 Mon Sep 17 00:00:00 2001 From: Karol Blaszczak Date: Tue, 10 May 2022 08:30:37 +0200 Subject: [PATCH] DOCS-hetero_alignment_changes (#11643) Align the HETERO article with the AUTO and MULTI template --- docs/OV_Runtime_UG/auto_device_selection.md | 2 +- docs/OV_Runtime_UG/hetero_execution.md | 44 ++++++++++----------- 2 files changed, 22 insertions(+), 24 deletions(-) diff --git a/docs/OV_Runtime_UG/auto_device_selection.md b/docs/OV_Runtime_UG/auto_device_selection.md index 2d4055dc712..2ad6221bc51 100644 --- a/docs/OV_Runtime_UG/auto_device_selection.md +++ b/docs/OV_Runtime_UG/auto_device_selection.md @@ -69,7 +69,7 @@ benchmark_app -m ../public/alexnet/FP32/alexnet.xml -d AUTO -niter 128 The longer the process runs, the closer realtime performance will be to that of the best-suited device. @endsphinxdirective -## Using the Auto-Device Plugin +## Using the Auto-Device Mode Following the OpenVINO™ naming convention, the Automatic Device Selection mode is assigned the label of “AUTO.” It may be defined with no additional parameters, resulting in defaults being used, or configured further with the following setup options: diff --git a/docs/OV_Runtime_UG/hetero_execution.md b/docs/OV_Runtime_UG/hetero_execution.md index 1b802eb2793..eadafbb2b73 100644 --- a/docs/OV_Runtime_UG/hetero_execution.md +++ b/docs/OV_Runtime_UG/hetero_execution.md @@ -1,34 +1,34 @@ # Heterogeneous execution {#openvino_docs_OV_UG_Hetero_execution} -Heterogeneous execution enables executing inference of one model on several devices. Its purpose is to: +The Heterogeneous Execution mode, or HETERO for short, acts as a "virtual" or a "proxy" device, which does not bind to a specific type of hardware. Instead, it executes inference of one model on several devices. Its purpose is to utilize all available hardware more efficiently during one inference. This means that accelerators are used to process the heaviest parts of the model, while fallback devices, like the CPU, execute operations not supported by accelerators. -* Utilize the power of accelerators to process the heaviest parts of the model and to execute unsupported operations on fallback devices, like the CPU. -* Utilize all available hardware more efficiently during one inference. +Compiling a model to the Heterogeneous mode assumes splitting it into subgraphs. Each subgraph is compiled on a dedicated device and multiple `ov::CompiledModel` objects are created. The objects are connected via automatically allocated intermediate tensors. -Execution via the heterogeneous mode can be divided into two independent steps: +Importantly, the model division is performed according to pre-defined affinities between hardware and operations. Every set of connected operations with the same affinity becomes a dedicated subgraph. Setting these affinities needs to be done as a separate step (`ov::Core::query_model` is used internally by HETERO), as described below. -1. Setting hardware affinity to operations (`ov::Core::query_model` is used internally by the Hetero device) -2. Compiling a model to the Heterogeneous device assumes splitting the model to parts, compiling them on the specified devices (via `ov::device::priorities`), and executing them in the Heterogeneous mode. The model is split to subgraphs in accordance with the affinities, where a set of connected operations with the same affinity is to be a dedicated subgraph. Each subgraph is compiled on a dedicated device and multiple `ov::CompiledModel` objects are made, which are connected via automatically allocated intermediate tensors. -These two steps are not interconnected and affinities can be set in one of two ways, used separately or in combination (as described below): in the `manual` or the `automatic` mode. +### Using the Hetero Mode -### Defining and Configuring the Hetero Device - -Following the OpenVINO™ naming convention, the Hetero execution plugin is assigned the label of `"HETERO".` It may be defined with no additional parameters, resulting in defaults being used, or configured further with the following setup options: +Following the OpenVINO™ naming convention, the Hetero execution mode is assigned the label of `"HETERO".` It may be defined with no additional parameters, resulting in defaults being used, or configured further with the following setup options: @sphinxdirective +-------------------------------+--------------------------------------------+-----------------------------------------------------------+ -| Parameter Name & C++ property | Property values | Description | +| Property | Property values | Description | +===============================+============================================+===========================================================+ -| | "MULTI_DEVICE_PRIORITIES" | | HETERO: | | Lists the devices available for selection. | -| | `ov::device::priorities` | | comma-separated, no spaces | | The device sequence will be taken as priority | -| | | | | | from high to low. | +| | | HETERO: | | Specifies the devices available for selection. | +| | | comma-separated, no spaces | | The device sequence will be taken as priority | ++-------------------------------+--------------------------------------------+ | from high to low. | +| `ov::device::priorities` | | device names | | | +| | | comma-separated, no spaces | | | +-------------------------------+--------------------------------------------+-----------------------------------------------------------+ @endsphinxdirective -### Manual and Automatic modes for assigning affinities -#### The Manual Mode +### Assigning Affinities + +Affinities can be set in one of two ways, used separately or in combination: with the `manual` or the `automatic` option. + +#### The Manual Option It assumes setting affinities explicitly for all operations in the model using `ov::Node::get_rt_info` with the `"affinity"` key. @sphinxtabset @@ -47,13 +47,10 @@ It assumes setting affinities explicitly for all operations in the model using ` @endsphinxtabset - - - -#### The Automatic Mode +#### The Automatic Option It decides automatically which operation is assigned to which device according to the support from dedicated devices (`GPU`, `CPU`, `MYRIAD`, etc.) and query model step is called implicitly by Hetero device during model compilation. -The automatic mode causes "greedy" behavior and assigns all operations that can be executed on a given device to it, according to the priorities you specify (for example, `ov::device::priorities("GPU,CPU")`). +The automatic option causes "greedy" behavior and assigns all operations that can be executed on a given device to it, according to the priorities you specify (for example, `ov::device::priorities("GPU,CPU")`). It does not take into account device peculiarities such as the inability to infer certain operations without other special operations placed before or after that layer. If the device plugin does not support the subgraph topology constructed by the HETERO device, then you should set affinity manually. @sphinxtabset @@ -72,7 +69,7 @@ It does not take into account device peculiarities such as the inability to infe @endsphinxtabset -#### Using Manual and Automatic Modes in Combination +#### Using Manual and Automatic Options in Combination In some cases you may need to consider manually adjusting affinities which were set automatically. It usually serves minimizing the number of total subgraphs to optimize memory transfers. To do it, you need to "fix" the automatically assigned affinities like so: @sphinxtabset @@ -91,10 +88,11 @@ In some cases you may need to consider manually adjusting affinities which were @endsphinxtabset -Importantly, the automatic mode will not work if any operation in a model has its `"affinity"` already initialized. +Importantly, the automatic option will not work if any operation in a model has its `"affinity"` already initialized. > **NOTE**: `ov::Core::query_model` does not depend on affinities set by a user. Instead, it queries for an operation support based on device capabilities. + ### Configure fallback devices If you want different devices in Hetero execution to have different device-specific configuration options, you can use the special helper property `ov::device::properties`: