Config and hetero (#10555)
* Updated properties documentation * Fixed doc refernce * merged snipet files * fixed build * Updated Hetero docs * Self-review Co-authored-by: Anton Pankratv <anton.pankratov@intel.com>
This commit is contained in:
Ilya LavrenovandAnton Pankratv authored and 
GitHub committed
1 parent
0ce255e56a
commit
68e873c6c8
13 files changed
+561
-491
No files matched your search
@@ -26,7 +26,7 @@ There are two ways to check if CPU device can support bfloat16 computations for
|
||||
1. Query the instruction set using one of these system commands:
|
||||
* `lscpu | grep avx512_bf16`
|
||||
* `cat /proc/cpuinfo | grep avx512_bf16`
|
||||
2. Use the [Query API](InferenceEngine_QueryAPI.md) with `METRIC_KEY(OPTIMIZATION_CAPABILITIES)`, which should return `BF16` in the list of CPU optimization options:
|
||||
2. Use the [Properties API](PropertiesAPI.md) with `METRIC_KEY(OPTIMIZATION_CAPABILITIES)`, which should return `BF16` in the list of CPU optimization options:
|
||||
|
||||
@snippet snippets/Bfloat16Inference0.cpp part0
|
||||
|
||||
|
||||
@@ -1,235 +0,0 @@
|
||||
# Introduction to Inference Engine Device Query API {#openvino_docs_IE_DG_InferenceEngine_QueryAPI}
|
||||
|
||||
## Inference Engine Query API (C++)
|
||||
|
||||
@sphinxdirective
|
||||
.. raw:: html
|
||||
|
||||
<div id="switcher-cpp" class="switcher-anchor">C++</div>
|
||||
@endsphinxdirective
|
||||
|
||||
The OpenVINO™ toolkit supports inferencing with several types of devices (processors or accelerators).
|
||||
This section provides a high-level description of the process of querying of different device properties and configuration values at runtime. Refer to the [Hello Query Device С++ Sample](../../samples/cpp/hello_query_device/README.md) sources and the [Multi-Device Plugin documentation](supported_plugins/MULTI.md) for examples of using the Inference Engine Query API in user applications.
|
||||
|
||||
### Using the Inference Engine Query API in Your Code
|
||||
|
||||
The `InferenceEngine::Core` class provides the following API to query device information, set or get different device configuration properties:
|
||||
|
||||
* `InferenceEngine::Core::GetAvailableDevices` - Provides a list of available devices. If there are more than one instance of a specific device, the devices are enumerated with `.suffix` where `suffix` is a unique string identifier. The device name can be passed to all methods of the `InferenceEngine::Core` class that work with devices, for example `InferenceEngine::Core::LoadNetwork`.
|
||||
* `InferenceEngine::Core::GetMetric` - Provides information about specific device.
|
||||
`InferenceEngine::Core::GetConfig` - Gets the current value of a specific configuration key.
|
||||
* `InferenceEngine::Core::SetConfig` - Sets a new value for the configuration key.
|
||||
|
||||
The `InferenceEngine::ExecutableNetwork` class is also extended to support the Query API:
|
||||
|
||||
* `InferenceEngine::ExecutableNetwork::GetMetric`
|
||||
* `InferenceEngine::ExecutableNetwork::GetConfig`
|
||||
* `InferenceEngine::ExecutableNetwork::SetConfig`
|
||||
|
||||
### Query API in the Core Class
|
||||
|
||||
#### GetAvailableDevices
|
||||
|
||||
@snippet snippets/InferenceEngine_QueryAPI0.cpp part0
|
||||
|
||||
The function returns a list of available devices, for example:
|
||||
|
||||
```
|
||||
MYRIAD.1.2-ma2480
|
||||
MYRIAD.1.4-ma2480
|
||||
CPU
|
||||
GPU.0
|
||||
GPU.1
|
||||
```
|
||||
|
||||
Each device name can then be passed to:
|
||||
|
||||
* `InferenceEngine::Core::LoadNetwork` to load the network to a specific device.
|
||||
* `InferenceEngine::Core::GetMetric` to get common or device specific metrics.
|
||||
* All other methods of the `InferenceEngine::Core` class that accept `deviceName`.
|
||||
|
||||
#### GetConfig()
|
||||
|
||||
The code below demonstrates how to understand whether the `HETERO` device dumps GraphViz `.dot` files with split graphs during the split stage:
|
||||
|
||||
@snippet snippets/InferenceEngine_QueryAPI1.cpp part1
|
||||
|
||||
For documentation about common configuration keys, refer to `ie_plugin_config.hpp`. Device specific configuration keys can be found in corresponding plugin folders.
|
||||
|
||||
#### GetMetric()
|
||||
|
||||
* To extract device properties such as available device, device name, supported configuration keys, and others, use the `InferenceEngine::Core::GetMetric` method:
|
||||
|
||||
@snippet snippets/InferenceEngine_QueryAPI2.cpp part2
|
||||
|
||||
A returned value appears as follows: `Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz`.
|
||||
|
||||
> **NOTE**: All metrics have a type, which is specified during metric instantiation. The list of common device-agnostic metrics can be found in `ie_plugin_config.hpp`. Device specific metrics (for example, for HDDL or MYRIAD devices) can be found in corresponding plugin folders.
|
||||
|
||||
### Query API in the ExecutableNetwork Class
|
||||
|
||||
#### GetMetric()
|
||||
|
||||
The method is used to get an executable network specific metric such as `METRIC_KEY(OPTIMAL_NUMBER_OF_INFER_REQUESTS)`:
|
||||
|
||||
@snippet snippets/InferenceEngine_QueryAPI3.cpp part3
|
||||
|
||||
Or the current temperature of the `MYRIAD` device:
|
||||
|
||||
@snippet snippets/InferenceEngine_QueryAPI4.cpp part4
|
||||
|
||||
#### GetConfig()
|
||||
|
||||
The method is used to get information about configuration values the executable network has been created with:
|
||||
|
||||
@snippet snippets/InferenceEngine_QueryAPI5.cpp part5
|
||||
|
||||
#### SetConfig()
|
||||
|
||||
The only device that supports this method is [Multi-Device](supported_plugins/MULTI.md).
|
||||
|
||||
## Inference Engine Query API (Python)
|
||||
|
||||
@sphinxdirective
|
||||
.. raw:: html
|
||||
|
||||
<div id="switcher-python" class="switcher-anchor">Python</div>
|
||||
@endsphinxdirective
|
||||
|
||||
This section provides a high-level description of the process of querying of different device properties and configuration values. Refer to the [Hello Query Device Python Sample](../../samples/python/hello_query_device/README.md) sources and the [Multi-Device Plugin documentation](supported_plugins/MULTI.md) for examples of using the Inference Engine Query API in user applications.
|
||||
|
||||
### Using the Inference Engine Query API in Your Code
|
||||
|
||||
The Inference Engine [Core](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino-inference-engine-iecore) class provides the following API to query device information, set or get different device configuration properties:
|
||||
|
||||
* [ie_api.IECore.available_devices](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.available_devices) - Provides a list of available devices. If there are more than one instance of a specific device, the devices are enumerated with .suffix where suffix is a unique string identifier. The device name can be passed to all methods of the IECore class that work with devices, for example [ie_api.IECore.load_network](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.load_network).
|
||||
* [ie_api.ieCore.get_metric](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.get_metric) - Provides information about specific device.
|
||||
* [ie_api.IECore.get_config](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.get_config) - Gets the current value of a specific configuration key.
|
||||
* [ie_api.IECore.set_config](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.set_config) - Sets a new value for the configuration key.
|
||||
|
||||
The [ie_api.ExecutableNetwork](api/ie_python_api/_autosummary/openvino.inference_engine.ExecutableNetwork.html) class is also extended to support the Query API:
|
||||
* [ie_api.ExecutableNetwork.get_metric](api/ie_python_api/_autosummary/openvino.inference_engine.ExecutableNetwork.html#openvino.inference_engine.ExecutableNetwork.get_metric)
|
||||
* [ie_api.ExecutableNetwork.get_config](latest/api/ie_python_api/_autosummary/openvino.inference_engine.ExecutableNetwork.html#openvino.inference_engine.ExecutableNetwork.get_config)
|
||||
* There is no method to call for set_config, but the equivalent action is described below.
|
||||
|
||||
### Query API in the IECore Class
|
||||
|
||||
#### Get Available Devices
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
print(ie.available_devices)
|
||||
```
|
||||
|
||||
This code prints a list of available devices, for example:
|
||||
|
||||
```
|
||||
MYRIAD.1.2-ma2480
|
||||
MYRIAD.1.4-ma2480
|
||||
FPGA.0
|
||||
FPGA.1
|
||||
CPU
|
||||
GPU.0
|
||||
GPU.1
|
||||
```
|
||||
|
||||
Each device name can then be passed to:
|
||||
|
||||
* `IECore.load_network` to load the network to a specific device.
|
||||
* `IECore.get_metric` to get common or device specific metrics.
|
||||
* All other methods of the `IECore` class that accept a device name.
|
||||
|
||||
#### Get Metric
|
||||
|
||||
To extract device properties such as available device, device name, supported configuration keys, and others, use the [IECore.get_metric](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.get_metric) method:
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
ie.get_metric(device_name="CPU", metric_name="FULL_DEVICE_NAME")
|
||||
```
|
||||
|
||||
A returned value appears as follows: `Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz`.
|
||||
|
||||
To list all supported metrics for a device:
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
ie.get_metric(device_name="GPU", metric_name="SUPPORTED_METRICS")
|
||||
```
|
||||
|
||||
#### Get Configuration
|
||||
|
||||
The code below uses the [IECore.get_config](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.get_config) method and demonstrates how to understand whether the HETERO device dumps .dot files with split graphs during the split stage:
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
ie.get_config(device_name="HETERO", config_name="HETERO_DUMP_GRAPH_DOT")
|
||||
```
|
||||
|
||||
To list all supported configuration keys for a device:
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
ie.get_metric(device_name=device, metric_name="SUPPORTED_CONFIG_KEYS")
|
||||
```
|
||||
|
||||
For documentation about common configuration keys, refer to `ie_plugin_config.hpp`. Device specific configuration keys can be found in corresponding plugin folders.
|
||||
|
||||
|
||||
### Query API in the ExecutableNetwork Class
|
||||
|
||||
#### Get Metric
|
||||
|
||||
To get the name of the loaded network:
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
net = ie.read_network(model=path_to_xml_file)
|
||||
exec_net = ie.load_network(network=net, device_name=device)
|
||||
exec_net.get_metric("NETWORK_NAME")
|
||||
```
|
||||
|
||||
Use `exec_net.get_metric("SUPPORTED_METRICS")` to list all supported metrics for an ExecutableNetwork instance.
|
||||
|
||||
|
||||
#### Get Configuration
|
||||
|
||||
The [IECore.get_config](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.get_config) method is used to get information about configuration values the executable network has been created with:
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
net = ie.read_network(model=path_to_xml_file)
|
||||
exec_net = ie.load_network(network=net, device_name="CPU")
|
||||
exec_net.get_config("CPU_THREADS_NUM")
|
||||
```
|
||||
|
||||
Or the current temperature of MYRIAD device:
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
net = ie.read_network(model=path_to_xml_file)
|
||||
exec_net = ie.load_network(network=net, device_name="MYRIAD")
|
||||
exec_net.get_config("DEVICE_THERMAL")
|
||||
```
|
||||
|
||||
Use `exec_net.get_metric("SUPPORTED_CONFIG_KEYS")` to list all supported configuration keys.
|
||||
|
||||
#### Set Configuration
|
||||
|
||||
The only device that supports this method in the ExecutableNetwork class is the [Multi-Device](supported_plugins/MULTI.md), where you can change the priorities of the devices for the Multi plugin in real time: `exec_net.set_config({{"MULTI_DEVICE_PRIORITIES", "GPU,CPU"}})`. See the Multi-Device documentation for more details.
|
||||
@@ -8,19 +8,28 @@
|
||||
:maxdepth: 1
|
||||
:hidden:
|
||||
|
||||
openvino_2_0_transition_guide
|
||||
openvino_docs_IE_DG_Integrate_with_customer_application_new_API
|
||||
openvino_docs_OV_Runtime_UG_Model_Representation
|
||||
ngraph_transformation
|
||||
openvino_docs_deployment_optimization_guide_dldt_optimization_guide
|
||||
openvino_docs_IE_DG_Device_Plugins
|
||||
openvino_docs_IE_DG_Int8Inference
|
||||
openvino_docs_IE_DG_Bfloat16Inference
|
||||
openvino_docs_IE_DG_DynamicBatching
|
||||
<!-- rename to "Changing input shapes" -->
|
||||
openvino_docs_IE_DG_ShapeInference
|
||||
openvino_docs_IE_DG_Device_Plugins
|
||||
<!-- insert here preprocessing -->
|
||||
<!-- generalize dyn batch to "Working with dynamic shapes" -->
|
||||
openvino_docs_IE_DG_DynamicBatching
|
||||
<!-- rename MULTI to "Automatic device selection" -->
|
||||
openvino_docs_IE_DG_supported_plugins_AUTO
|
||||
<!-- rename MULTI to "Running on multiple device simultaneously" -->
|
||||
openvino_docs_IE_DG_supported_plugins_MULTI
|
||||
openvino_docs_OV_UG_Hetero_execution
|
||||
openvino_docs_IE_DG_network_state_intro
|
||||
openvino_2_0_transition_guide
|
||||
<!-- remove parts below -->
|
||||
openvino_docs_deployment_optimization_guide_dldt_optimization_guide
|
||||
openvino_docs_IE_DG_Model_caching_overview
|
||||
openvino_docs_IE_DG_Extensibility_DG_Intro
|
||||
openvino_docs_IE_DG_network_state_intro
|
||||
openvino_docs_IE_DG_Int8Inference
|
||||
openvino_docs_IE_DG_Bfloat16Inference
|
||||
ngraph_transformation
|
||||
openvino_docs_OV_Runtime_API_Changes
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
# Introduction to OpenVINO™ Device Properties API {#openvino_docs_IE_DG_InferenceEngine_QueryAPI}
|
||||
|
||||
## OpenVINO™ Properties API (C++)
|
||||
|
||||
@sphinxdirective
|
||||
.. raw:: html
|
||||
|
||||
<div id="switcher-cpp" class="switcher-anchor">C++</div>
|
||||
@endsphinxdirective
|
||||
|
||||
The OpenVINO™ toolkit supports inferencing with several types of devices (processors or accelerators).
|
||||
This section provides a high-level description of the process of querying of different device properties and configuration values at runtime. Refer to the [Hello Query Device С++ Sample](../../samples/cpp/hello_query_device/README.md) sources and the [Multi-Device Plugin documentation](supported_plugins/MULTI.md) for examples of using the OpenVINO™ Properties API in user applications.
|
||||
|
||||
### Using the OpenVINO™ Properties API in Your Code
|
||||
|
||||
The `ov::Core` class provides the following API to query device information, set or get different device configuration properties:
|
||||
|
||||
* `ov::Core::get_available_devices` - Provides a list of available devices. If there are more than one instance of a specific device, the devices are enumerated with `.suffix` where `suffix` is a unique string identifier. The device name can be passed to all methods of the `ov::Core` class that work with devices, for example `ov::Core::compile_model`.
|
||||
* `ov::Core::get_property` - Gets the current value of a specific property.
|
||||
* `ov::Core::set_property` - Sets a new value for the property.
|
||||
|
||||
The `ov::CompiledModel` class is also extended to support the Properties API:
|
||||
|
||||
* `ov::CompiledModel::get_property`
|
||||
* `ov::CompiledModel::set)property`
|
||||
|
||||
### Properties API in the Core Class
|
||||
|
||||
#### get_available_devices
|
||||
|
||||
@snippet snippets/ov_properties_api.cpp part0
|
||||
|
||||
The function returns a list of available devices, for example:
|
||||
|
||||
```
|
||||
MYRIAD.1.2-ma2480
|
||||
MYRIAD.1.4-ma2480
|
||||
CPU
|
||||
GPU.0
|
||||
GPU.1
|
||||
```
|
||||
|
||||
Each device name can then be passed to:
|
||||
|
||||
* `ov::Core::compile_model` to load the network to a specific device.
|
||||
* `ov::Core::get_property` to get common or device specific properties.
|
||||
* All other methods of the `ov::Core` class that accept `deviceName`.
|
||||
|
||||
#### ov::Core methods
|
||||
|
||||
`ov::Core` methods like:
|
||||
|
||||
* `ov::Core::compile_model`
|
||||
* `ov::Core::import_model`
|
||||
* `ov::Core::query_model`
|
||||
* `ov::Core::query_model`
|
||||
* `ov::Core::create_context`
|
||||
|
||||
accept variadic list of properties as last arguments. Each property in such parameters lists should be used as function call to pass property value with specified property type
|
||||
|
||||
@snippet snippets/ov_properties_api.cpp part3
|
||||
|
||||
#### get_property()
|
||||
|
||||
For documentation about common configuration keys, refer to `openvino/runtime/properties.hpp`. Device specific configuration keys can be found in corresponding plugin folders.
|
||||
|
||||
* The code below demonstrates how to query `HETERO` device priority of devices which will be used to infer the model:
|
||||
|
||||
@snippet snippets/ov_properties_api.cpp part1
|
||||
|
||||
* To extract device properties such as available device, device name, supported configuration keys, and others, use the `ov::Core::get_property` method:
|
||||
|
||||
@snippet snippets/ov_properties_api.cpp part2
|
||||
|
||||
A returned value appears as follows: `Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz`.
|
||||
|
||||
> **NOTE**: All properties have a type, which is specified during property declaration. The list of common device-agnostic properties can be found in `openvino/runtime/properties.hpp`. Device specific properties (for example, for HDDL or MYRIAD devices) can be found in corresponding plugin folders.
|
||||
|
||||
### Properties API in the CompiledModel Class
|
||||
|
||||
#### get_property()
|
||||
|
||||
The method is used to get configuration values the compiled model has been created with or compiled model specific property such as `ov::optimal_number_of_infer_requests`:
|
||||
|
||||
@snippet snippets/ov_properties_api.cpp part4
|
||||
|
||||
Or the current temperature of the `MYRIAD` device:
|
||||
|
||||
@snippet snippets/ov_properties_api.cpp part5
|
||||
|
||||
Or the number of threads that would be used for inference in `CPU` device:
|
||||
|
||||
@snippet snippets/ov_properties_api.cpp part6
|
||||
|
||||
#### set_property()
|
||||
|
||||
The only device that supports this method is [Multi-Device](supported_plugins/MULTI.md).
|
||||
|
||||
## OpenVINO™ Properties API (Python)
|
||||
|
||||
@sphinxdirective
|
||||
.. raw:: html
|
||||
|
||||
<div id="switcher-python" class="switcher-anchor">Python</div>
|
||||
@endsphinxdirective
|
||||
|
||||
This section provides a high-level description of the process of querying of different device properties and configuration values. Refer to the [Hello Properties Device Python Sample](../../samples/python/hello_query_device/README.md) sources and the [Multi-Device Plugin documentation](supported_plugins/MULTI.md) for examples of using the OpenVINO™ Properties API in user applications.
|
||||
|
||||
### Using the OpenVINO™ Properties API in Your Code
|
||||
|
||||
The OpenVINO™ [Core](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino-inference-engine-iecore) class provides the following API to query device information, set or get different device configuration properties:
|
||||
|
||||
* [ie_api.IECore.available_devices](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.available_devices) - Provides a list of available devices. If there are more than one instance of a specific device, the devices are enumerated with .suffix where suffix is a unique string identifier. The device name can be passed to all methods of the IECore class that work with devices, for example [ie_api.IECore.load_network](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.load_network).
|
||||
* [ie_api.ieCore.get_property](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.get_property) - Provides information about specific device.
|
||||
* [ie_api.IECore.get_config](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.get_config) - Gets the current value of a specific configuration key.
|
||||
* [ie_api.IECore.set_config](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.set_config) - Sets a new value for the configuration key.
|
||||
|
||||
The [ie_api.CompiledModel](api/ie_python_api/_autosummary/openvino.inference_engine.CompiledModel.html) class is also extended to support the Properties API:
|
||||
* [ie_api.CompiledModel.get_property](api/ie_python_api/_autosummary/openvino.inference_engine.CompiledModel.html#openvino.inference_engine.CompiledModel.get_property)
|
||||
* [ie_api.CompiledModel.get_config](latest/api/ie_python_api/_autosummary/openvino.inference_engine.CompiledModel.html#openvino.inference_engine.CompiledModel.get_config)
|
||||
* There is no method to call for set_config, but the equivalent action is described below.
|
||||
|
||||
### Properties API in the IECore Class
|
||||
|
||||
#### Get Available Devices
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
print(ie.available_devices)
|
||||
```
|
||||
|
||||
This code prints a list of available devices, for example:
|
||||
|
||||
```
|
||||
MYRIAD.1.2-ma2480
|
||||
MYRIAD.1.4-ma2480
|
||||
FPGA.0
|
||||
FPGA.1
|
||||
CPU
|
||||
GPU.0
|
||||
GPU.1
|
||||
```
|
||||
|
||||
Each device name can then be passed to:
|
||||
|
||||
* `IECore.load_network` to load the network to a specific device.
|
||||
* `IECore.get_property` to get common or device specific properties.
|
||||
* All other methods of the `IECore` class that accept a device name.
|
||||
|
||||
#### Get Metric
|
||||
|
||||
To extract device properties such as available device, device name, supported configuration keys, and others, use the [IECore.get_property](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.get_property) method:
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
ie.get_property(device_name="CPU", property_name="FULL_DEVICE_NAME")
|
||||
```
|
||||
|
||||
A returned value appears as follows: `Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz`.
|
||||
|
||||
To list all supported properties for a device:
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
ie.get_property(device_name="GPU", property_name="SUPPORTED_METRICS")
|
||||
```
|
||||
|
||||
#### Get Configuration
|
||||
|
||||
The code below uses the [IECore.get_config](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.get_config) method and demonstrates how to understand whether the HETERO device dumps .dot files with split graphs during the split stage:
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
ie.get_config(device_name="HETERO", config_name="HETERO_DUMP_GRAPH_DOT")
|
||||
```
|
||||
|
||||
To list all supported configuration keys for a device:
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
ie.get_property(device_name=device, property_name="SUPPORTED_CONFIG_KEYS")
|
||||
```
|
||||
|
||||
For documentation about common configuration keys, refer to `ie_plugin_config.hpp`. Device specific configuration keys can be found in corresponding plugin folders.
|
||||
|
||||
|
||||
### Properties API in the CompiledModel Class
|
||||
|
||||
#### Get Metric
|
||||
|
||||
To get the name of the loaded network:
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
net = ie.read_network(model=path_to_xml_file)
|
||||
exec_net = ie.load_network(network=net, device_name=device)
|
||||
exec_net.get_property("NETWORK_NAME")
|
||||
```
|
||||
|
||||
Use `exec_net.get_property("SUPPORTED_METRICS")` to list all supported properties for an CompiledModel instance.
|
||||
|
||||
|
||||
#### Get Configuration
|
||||
|
||||
The [IECore.get_config](api/ie_python_api/_autosummary/openvino.inference_engine.IECore.html#openvino.inference_engine.IECore.get_config) method is used to get information about configuration values the compiled model has been created with:
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
net = ie.read_network(model=path_to_xml_file)
|
||||
exec_net = ie.load_network(network=net, device_name="CPU")
|
||||
exec_net.get_config("CPU_THREADS_NUM")
|
||||
```
|
||||
|
||||
Or the current temperature of MYRIAD device:
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
net = ie.read_network(model=path_to_xml_file)
|
||||
exec_net = ie.load_network(network=net, device_name="MYRIAD")
|
||||
exec_net.get_config("DEVICE_THERMAL")
|
||||
```
|
||||
|
||||
Use `exec_net.get_property("SUPPORTED_CONFIG_KEYS")` to list all supported configuration keys.
|
||||
|
||||
#### Set Configuration
|
||||
|
||||
The only device that supports this method in the CompiledModel class is the [Multi-Device](supported_plugins/MULTI.md), where you can change the priorities of the devices for the Multi plugin in real time: `exec_net.set_config({{"MULTI_DEVICE_PRIORITIES", "GPU,CPU"}})`. See the Multi-Device documentation for more details.
|
||||
@@ -11,10 +11,7 @@
|
||||
openvino_docs_IE_DG_supported_plugins_GPU
|
||||
openvino_docs_IE_DG_supported_plugins_VPU
|
||||
openvino_docs_IE_DG_supported_plugins_GNA
|
||||
openvino_docs_IE_DG_supported_plugins_AUTO
|
||||
openvino_docs_IE_DG_supported_plugins_HETERO
|
||||
openvino_docs_IE_DG_supported_plugins_MULTI
|
||||
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
Inference Engine uses a plugin architecture. Inference Engine plugin is a software component that contains complete implementation for inference on a certain Intel® hardware device: CPU, GPU, VPU, GNA, etc. Each plugin implements the unified API and provides additional hardware-specific APIs.
|
||||
@@ -27,8 +24,8 @@ The Inference Engine provides capabilities to infer deep learning models on the
|
||||
|[CPU plugin](CPU.md) |Intel® Xeon® with Intel® Advanced Vector Extensions 2 (Intel® AVX2), Intel® Advanced Vector Extensions 512 (Intel® AVX-512), and AVX512_BF16, Intel® Core™ Processors with Intel® AVX2, Intel® Atom® Processors with Intel® Streaming SIMD Extensions (Intel® SSE) |
|
||||
|[VPU plugins](VPU.md) (available in the Intel® Distribution of OpenVINO™ toolkit) |Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X, Intel® Vision Accelerator Design with Intel® Movidius™ VPUs |
|
||||
|[GNA plugin](GNA.md) (available in the Intel® Distribution of OpenVINO™ toolkit) |Intel® Speech Enabling Developer Kit, Amazon Alexa* Premium Far-Field Developer Kit, Intel® Pentium® Silver J5005 Processor, Intel® Pentium® Silver N5000 Processor, Intel® Celeron® J4005 Processor, Intel® Celeron® J4105 Processor, Intel® Celeron® Processor N4100, Intel® Celeron® Processor N4000, Intel® Core™ i3-8121U Processor, Intel® Core™ i7-1065G7 Processor, Intel® Core™ i7-1060G7 Processor, Intel® Core™ i5-1035G4 Processor, Intel® Core™ i5-1035G7 Processor, Intel® Core™ i5-1035G1 Processor, Intel® Core™ i5-1030G7 Processor, Intel® Core™ i5-1030G4 Processor, Intel® Core™ i3-1005G1 Processor, Intel® Core™ i3-1000G1 Processor, Intel® Core™ i3-1000G4 Processor|
|
||||
|[Multi-Device plugin](MULTI.md) |Multi-Device plugin enables simultaneous inference of the same network on several Intel® devices in parallel |
|
||||
|[Auto-Device plugin](AUTO.md) |Auto-Device plugin enables selecting Intel® device for inference automatically |
|
||||
|[Multi-Device plugin](MULTI.md) |Multi-Device plugin enables simultaneous inference of the same network on several Intel® devices in parallel |
|
||||
|[Auto-Device plugin](AUTO.md) |Auto-Device plugin enables selecting Intel® device for inference automatically |
|
||||
|[Heterogeneous plugin](HETERO.md) |Heterogeneous plugin enables automatic inference splitting between several Intel® devices (for example if a device doesn't [support certain layers](#supported-layers)). |
|
||||
|
||||
Devices similar to the ones we have used for benchmarking can be accessed using [Intel® DevCloud for the Edge](https://devcloud.intel.com/edge/), a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of the OpenVINO™ Toolkit. [Learn more](https://devcloud.intel.com/edge/get_started/devcloud/) or [Register here](https://inteliot.force.com/DevcloudForEdge/s/).
|
||||
|
||||
@@ -1,256 +1,157 @@
|
||||
# Heterogeneous Plugin {#openvino_docs_IE_DG_supported_plugins_HETERO}
|
||||
# Heterogeneous execution {#openvino_docs_OV_UG_Hetero_execution}
|
||||
|
||||
## Introducing the Heterogeneous Plugin (C++)
|
||||
## Introducing the Heterogeneous execution
|
||||
|
||||
@sphinxdirective
|
||||
.. raw:: html
|
||||
The heterogeneous execution enables computing the inference of one model on several devices. The purposes of executing models in heterogeneous mode are to:
|
||||
|
||||
<div id="switcher-cpp" class="switcher-anchor">C++</div>
|
||||
@endsphinxdirective
|
||||
|
||||
The heterogeneous plugin enables computing the inference of one network on several devices. The purposes of executing networks in heterogeneous mode are to:
|
||||
|
||||
* Utilize the power of accelerators to process the heaviest parts of the network and to execute unsupported layers on fallback devices like the CPU
|
||||
* Utilize the power of accelerators to process the heaviest parts of the model and to execute unsupported operations on fallback devices like the CPU
|
||||
* Utilize all available hardware more efficiently during one inference
|
||||
|
||||
The execution through heterogeneous plugin can be divided into two independent steps:
|
||||
The execution through heterogeneous mode can be divided into two independent steps:
|
||||
|
||||
1. Setting of hardware affinity to layers
|
||||
2. Loading a network to the Heterogeneous plugin, splitting the network to parts, and executing them through the plugin
|
||||
1. Setting of hardware affinity to operations (ov::Core::query_model is used internally by the Hetero device)
|
||||
2. Compiling a model to the Heterogeneous device assuming splitting the model to parts and compiling on the specified devices (via ov::device::priorities), and executing them through the Heterogeneous mode. The model is split to the subgraphs in according to the affinities where a set of conntected operations with the same affinity are supposed to be a dedicated subgraph. Each subgraph is compiled on a dedicated device and we have multiple ov::CompiledModel objects, which are connected via automatically allocated intermediate tensors.
|
||||
|
||||
These steps are decoupled. The setting of affinity can be done automatically using the fallback policy or in manual mode.
|
||||
These steps are decoupled. The setting of affinities can be done automatically using the `automatic fallback` policy or in `manual` mode:
|
||||
|
||||
The fallback automatic policy causes "greedy" behavior and assigns all layers that can be executed on certain device according to the priorities you specify (for example, HETERO:GPU,CPU).
|
||||
Automatic policy does not take into account plugin peculiarities such as the inability to infer some layers without other special layers placed before or after that layer. The plugin is responsible for solving such cases. If the device plugin does not support the subgraph topology constructed by the HETERO plugin, then you should set affinity manually.
|
||||
- The fallback automatic policy causes "greedy" behavior and assigns all operations that can be executed on certain device according to the priorities you specify (for example, `ov::device::priorities("GPU,CPU")`).
|
||||
Automatic policy does not take into account device peculiarities such as the inability to infer some operations without other special operations placed before or after that layer. The plugin is responsible for solving such cases. If the device plugin does not support the subgraph topology constructed by the HETERO device, then you should set affinity manually.
|
||||
- Manual policy assumes explicit setting of affinities for all operations in the model using the runtime information ov::Node::get_rt_info.
|
||||
|
||||
### Details of Splitting Network and Execution
|
||||
During loading of the network to the Heterogeneous plugin, the network is divided into separate parts and loaded to dedicated plugins.
|
||||
Intermediate blobs between these subgraphs are allocated automatically in the most efficient way.
|
||||
### Defining and Configuring the Hetero Device
|
||||
|
||||
Following the OpenVINO™ convention of labeling devices, the Hetero execution uses the name `"HETERO"`. Configuration options for the Hetero device:
|
||||
|
||||
| Parameter name | C++ property | Parameter values | Default | Description |
|
||||
| -------------- | ---------------- | ---------------- | --- | --- |
|
||||
| "MULTI_DEVICE_PRIORITIES" | `ov::device::priorities` | comma-separated device names with no spaces | N/A | Prioritized list of devices |
|
||||
|
||||
### Automatic and manual policies for assigning affinities
|
||||
|
||||
`Automatic fallback` policy decides which operation goes to which device automatically according to the support in dedicated devices (`GPU`, `CPU`, `MYRIAD`, etc) and query model step is called implicitly by Hetero device during model compilation:
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: C++
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_hetero.cpp
|
||||
:language: cpp
|
||||
:fragment: [compile_model]
|
||||
|
||||
.. tab:: Python
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_hetero.py
|
||||
:language: python
|
||||
:fragment: [compile_model]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
Another way to annotate a model is to set all affinities `manually` using ov::Node::get_rt_info with key `"affinity"`:
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: C++
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_hetero.cpp
|
||||
:language: cpp
|
||||
:fragment: [set_manual_affinities]
|
||||
|
||||
.. tab:: Python
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_hetero.py
|
||||
:language: python
|
||||
:fragment: [set_manual_affinities]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
The fallback policy does not work if at least one operation has an initialized `"affinity"`. If you want to adjust automatically set affinities, then get automatic affinities first, then fix them (usually, to minimize a number of total subgraphs to optimize memory transfers):
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: C++
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_hetero.cpp
|
||||
:language: cpp
|
||||
:fragment: [fix_automatic_affinities]
|
||||
|
||||
.. tab:: Python
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_hetero.py
|
||||
:language: python
|
||||
:fragment: [fix_automatic_affinities]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
> **NOTE**: ov::Core::query_model does not depend on affinities set by a user. Instead, it queries for an operation support based on device capabilities.
|
||||
|
||||
### Configure fallback devices
|
||||
If you want different devices in Hetero execution to have different device-specific configuration options, you can use the special helper property ov::device::properties:
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
.. tab:: C++
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_hetero.cpp
|
||||
:language: cpp
|
||||
:fragment: [configure_fallback_devices]
|
||||
|
||||
.. tab:: Python
|
||||
|
||||
.. doxygensnippet:: docs/snippets/ov_hetero.py
|
||||
:language: python
|
||||
:fragment: [configure_fallback_devices]
|
||||
|
||||
@endsphinxdirective
|
||||
|
||||
In the example above, `CPU` device is configured to enable profiling data, while only `GPU` device has configuration property to perform inference in `f16` precision, while CPU has default execution precision.
|
||||
|
||||
### Handling Difficult Topologies
|
||||
|
||||
Some topologies are not friendly to heterogeneous execution on some devices or cannot be executed at all with this device.
|
||||
For example, models having activation operations that are not supported on the primary device are split by Hetero device into multiple set of subgraphs which leads to unoptimal execution.
|
||||
If transmitting data from one subgraph of a whole model to another part in heterogeneous mode takes more time than in normal execution, it may not make sense to execute them heterogeneously.
|
||||
In this case, you can define the heaviest part manually and set the affinity to avoid sending data back and forth many times during one inference.
|
||||
|
||||
### Analyzing Performance Heterogeneous Execution
|
||||
After enabling the <code>OPENVINO_HETERO_VISUALIZE</code> environment variable, you can dump GraphViz* `.dot` files with annotations of operations per devices.
|
||||
|
||||
The Heterogeneous device can generate two files:
|
||||
|
||||
* `hetero_affinity_<model name>.dot` - annotation of affinities per operation.
|
||||
* `hetero_subgraphs_<model name>.dot` - annotation of affinities per graph.
|
||||
|
||||
You can use the GraphViz* utility or a file converter to view the images. On the Ubuntu* operating system, you can use xdot:
|
||||
|
||||
* `sudo apt-get install xdot`
|
||||
* `xdot hetero_subgraphs.dot`
|
||||
|
||||
You can use performance data (in sample applications, it is the option `-pc`) to get the performance data on each subgraph.
|
||||
|
||||
Here is an example of the output for Googlenet v1 running on HDDL with fallback to CPU:
|
||||
|
||||
```
|
||||
subgraph1: 1. input preprocessing (mean data/HDDL):EXECUTED layerType: realTime: 129 cpu: 129 execType:
|
||||
subgraph1: 2. input transfer to DDR:EXECUTED layerType: realTime: 201 cpu: 0 execType:
|
||||
subgraph1: 3. HDDL execute time:EXECUTED layerType: realTime: 3808 cpu: 0 execType:
|
||||
subgraph1: 4. output transfer from DDR:EXECUTED layerType: realTime: 55 cpu: 0 execType:
|
||||
subgraph1: 5. HDDL output postprocessing:EXECUTED layerType: realTime: 7 cpu: 7 execType:
|
||||
subgraph1: 6. copy to IE blob:EXECUTED layerType: realTime: 2 cpu: 2 execType:
|
||||
subgraph2: out_prob: NOT_RUN layerType: Output realTime: 0 cpu: 0 execType: unknown
|
||||
subgraph2: prob: EXECUTED layerType: SoftMax realTime: 10 cpu: 10 execType: ref
|
||||
Total time: 4212 microseconds
|
||||
```
|
||||
### Sample Usage
|
||||
|
||||
Inference Engine sample programs can use the Heterogeneous plugin used with the `-d` option:
|
||||
OpenVINO™ sample programs can use the Heterogeneous execution used with the `-d` option:
|
||||
|
||||
```sh
|
||||
./hello_classification <path_to_model>/squeezenet1.1.xml <path_to_pictures>/picture.jpg HETERO:GPU,CPU
|
||||
```
|
||||
where:
|
||||
- `HETERO` stands for the Heterogeneous plugin
|
||||
- `HETERO` stands for the Heterogeneous execution
|
||||
- `GPU,CPU` points to fallback policy with priority on GPU and fallback to CPU
|
||||
|
||||
You can point more than two devices: `-d HETERO:MYRIAD,GPU,CPU`
|
||||
|
||||
|
||||
### Annotation of Layers per Device and Default Fallback Policy
|
||||
|
||||
Default fallback policy decides which layer goes to which device automatically according to the support in dedicated plugins (GPU, CPU, MYRIAD).
|
||||
|
||||
Another way to annotate a network is to set affinity manually using `ngraph::Node::get_rt_info` with key `affinity`:
|
||||
|
||||
@snippet snippets/HETERO0.cpp part0
|
||||
|
||||
The fallback policy does not work if even one layer has an initialized affinity. The sequence should be to call automating affinity settings and then fix manually.
|
||||
|
||||
> **NOTE**: If you set affinity manually, be careful because currently Inference Engine plugins don't support constant (`Constant`->`Result`) and empty (`Parameter`->`Result`) networks. Please avoid such subgraphs when you set affinity manually.
|
||||
|
||||
@snippet snippets/HETERO1.cpp part1
|
||||
|
||||
If you rely on the default affinity distribution, you can avoid calling <code>InferenceEngine::Core::QueryNetwork</code> and just call <code>InferenceEngine::Core::LoadNetwork</code> instead:
|
||||
|
||||
@snippet snippets/HETERO2.cpp part2
|
||||
|
||||
> **NOTE**: `InferenceEngine::Core::QueryNetwork` does not depend on affinities set by a user. Instead, it queries for layer support based on device capabilities.
|
||||
|
||||
### Handling Difficult Topologies
|
||||
|
||||
Some topologies are not friendly to heterogeneous execution on some devices or cannot be executed at all with this plugin
|
||||
Examples are networks having activation layers that are not supported on the primary device.
|
||||
If transmitting data from one part of a network to another part in heterogeneous mode takes more time than in normal mode, it may not make sense to execute them in heterogeneous mode.
|
||||
In this case, you can define the heaviest part manually and set the affinity to avoid sending data back and forth many times during one inference.
|
||||
|
||||
### Execution Precision
|
||||
Precision for inference in the heterogeneous plugin is defined by:
|
||||
* Precision of IR
|
||||
* Ability of final plugins to execute in precision defined in IR
|
||||
|
||||
For example, if you want to execute GPU with CPU fallback with FP16 on GPU, you need to use only FP16 IR.
|
||||
|
||||
### Analyzing Performance Heterogeneous Execution
|
||||
After enabling the <code>KEY_HETERO_DUMP_GRAPH_DOT</code> config key (shown in code snippet below), you can dump GraphViz* `.dot` files with annotations of devices per layer.
|
||||
|
||||
The Heterogeneous plugin can generate two files:
|
||||
|
||||
* `hetero_affinity_<network name>.dot` - annotation of affinities per layer. This file is written to the disk only if default fallback policy was executed
|
||||
* `hetero_subgraphs_<network name>.dot` - annotation of affinities per graph. This file is written to the disk during execution of `ICNNNetwork::LoadNetwork()` for the Heterogeneous plugin
|
||||
|
||||
@snippet snippets/HETERO3.cpp part3
|
||||
|
||||
You can use the GraphViz* utility or a file converter to view the images. On the Ubuntu* operating system, you can use xdot:
|
||||
|
||||
* `sudo apt-get install xdot`
|
||||
* `xdot hetero_subgraphs.dot`
|
||||
|
||||
You can use performance data (in sample applications, it is the option `-pc`) to get the performance data on each subgraph.
|
||||
|
||||
Here is an example of the output for Googlenet v1 running on HDDL with fallback to CPU:
|
||||
|
||||
```
|
||||
subgraph1: 1. input preprocessing (mean data/HDDL):EXECUTED layerType: realTime: 129 cpu: 129 execType:
|
||||
subgraph1: 2. input transfer to DDR:EXECUTED layerType: realTime: 201 cpu: 0 execType:
|
||||
subgraph1: 3. HDDL execute time:EXECUTED layerType: realTime: 3808 cpu: 0 execType:
|
||||
subgraph1: 4. output transfer from DDR:EXECUTED layerType: realTime: 55 cpu: 0 execType:
|
||||
subgraph1: 5. HDDL output postprocessing:EXECUTED layerType: realTime: 7 cpu: 7 execType:
|
||||
subgraph1: 6. copy to IE blob:EXECUTED layerType: realTime: 2 cpu: 2 execType:
|
||||
subgraph2: out_prob: NOT_RUN layerType: Output realTime: 0 cpu: 0 execType: unknown
|
||||
subgraph2: prob: EXECUTED layerType: SoftMax realTime: 10 cpu: 10 execType: ref
|
||||
Total time: 4212 microseconds
|
||||
```
|
||||
### See Also
|
||||
[Supported Devices](Supported_Devices.md)
|
||||
|
||||
## Introducing the Heterogeneous Plugin (Python)
|
||||
|
||||
@sphinxdirective
|
||||
.. raw:: html
|
||||
|
||||
<div id="switcher-python" class="switcher-anchor">Python</div>
|
||||
@endsphinxdirective
|
||||
|
||||
The heterogeneous plugin enables computing the inference of one network on several devices. The purposes of executing networks in heterogeneous mode are to:
|
||||
|
||||
* Utilize the power of accelerators to process the heaviest parts of the network and to execute unsupported layers on fallback devices like the CPU
|
||||
* Utilize all available hardware more efficiently during one inference
|
||||
|
||||
The execution through heterogeneous plugin can be divided into two independent steps:
|
||||
|
||||
1. Setting of hardware affinity to layers
|
||||
2. Loading a network to the Heterogeneous plugin, splitting the network to parts, and executing them through the plugin
|
||||
|
||||
These steps are decoupled. The setting of affinity can be done automatically using the fallback policy or in manual mode.
|
||||
|
||||
The fallback automatic policy causes "greedy" behavior and assigns all layers that can be executed on certain device according to the priorities you specify (for example, HETERO:GPU,CPU).
|
||||
Automatic policy does not take into account plugin peculiarities such as the inability to infer some layers without other special layers placed before or after that layer. The plugin is responsible for solving such cases. If the device plugin does not support the subgraph topology constructed by the HETERO plugin, then you should set affinity manually.
|
||||
|
||||
Some of the topologies are not well-supported for heterogeneous execution on some devices or cannot be executed in this mode at all. Examples of such networks are those having activation layers which are not supported on the primary device. If transmitting data from one part of a network to another part in heterogeneous mode takes more time than in normal mode, it may not make sense to execute them in heterogeneous mode. In this case, you can define the most compute intense part manually and set the affinity to avoid sending data back and forth many times during one inference.
|
||||
|
||||
### Use Default Layer Affinities
|
||||
|
||||
To use the default affinities, call `load_network` with the "HETERO" device, with an optional list of devices to consider.
|
||||
|
||||
```python
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
net = ie.read_network(model=path_to_model)
|
||||
exec_net = ie.load_network(network=net, device_name='HETERO:GPU,CPU')
|
||||
```
|
||||
|
||||
|
||||
### Annotation of Layers per Device and Default Fallback Policy
|
||||
|
||||
Default fallback policy decides which layer goes to which device automatically according to the support in dedicated plugins (GPU, CPU, MYRIAD).
|
||||
|
||||
Another way to annotate a network is to set affinity manually using code.
|
||||
|
||||
### Set Affinity of All Layers to CPU
|
||||
```python
|
||||
import ngraph as ng
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
# Read a network in IR or ONNX format
|
||||
net = ie.read_network(path_to_model)
|
||||
# Create an Ngraph (graph) function from the network
|
||||
ng_func = ng.function_from_cnn(net)
|
||||
for node in ng_func.get_ordered_ops():
|
||||
rt_info = node.get_rt_info()
|
||||
rt_info["affinity"] = "CPU"
|
||||
```
|
||||
|
||||
|
||||
The fallback policy does not work if even one layer has an initialized affinity. The sequence should be calling the default affinity settings and then setting the layers manually.
|
||||
|
||||
> **NOTE**: If you set affinity manually, be aware that currently Inference Engine plugins do not support constant (*Constant -> Result*) and empty (*Parameter -> Result*) networks. Please avoid these subgraphs when you set affinity manually.
|
||||
|
||||
### Example - Manually Setting Layer Affinities
|
||||
|
||||
```python
|
||||
import ngraph as ng
|
||||
from openvino.inference_engine import IECore
|
||||
|
||||
ie = IECore()
|
||||
# Read a network in IR or ONNX format
|
||||
net = ie.read_network(path_to_model)
|
||||
ng_func = ng.function_from_cnn(net)
|
||||
|
||||
for node in ng_func.get_ordered_ops():
|
||||
rt_info = node.get_rt_info()
|
||||
rt_info["affinity"] = "CPU"
|
||||
|
||||
# Load the network on the target device
|
||||
exec_net = ie.load_network(network=net, device_name='HETERO:FPGA,CPU')
|
||||
```
|
||||
|
||||
> **NOTE**: `ie.query_network` does not depend on affinities set by a user, but queries for layer support based on device capabilities.
|
||||
|
||||
### Details of Splitting Network and Execution
|
||||
|
||||
During the loading of the network to the heterogeneous plugin, the network is divided into separate parts and loaded to dedicated plugins. Intermediate blobs between these sub graphs are allocated automatically in the most efficient way.
|
||||
|
||||
### Execution Precision
|
||||
|
||||
The precision for inference in the heterogeneous plugin is defined by:
|
||||
* Precision of IR
|
||||
* Ability of final plugins to execute in precision defined in IR
|
||||
|
||||
For example, if you want to execute GPU with CPU fallback with FP16 on GPU, you need to use only FP16 IR.
|
||||
|
||||
OpenVINO samples can be used with the following command:
|
||||
```sh
|
||||
./hello_classification <path_to_model>/squeezenet1.1.xml <path_to_pictures>/picture.jpg HETERO:GPU,CPU
|
||||
```
|
||||
|
||||
where `HETERO` stands for the heterogeneous plugin.
|
||||
|
||||
You can point to more than two devices, for example: `-d HETERO:MYRIAD,GPU,CPU`
|
||||
|
||||
### Analyzing Heterogeneous Execution
|
||||
|
||||
After enabling the KEY_HETERO_DUMP_GRAPH_DOT config key, you can dump GraphViz* .dot files with annotations of devices per layer.
|
||||
|
||||
The heterogeneous plugin can generate two files:
|
||||
|
||||
* `hetero_affinity_<network name>.dot` - annotation of affinities per layer. This file is written to the disk only if the default fallback policy was executed
|
||||
* `hetero_subgraphs_<network name>.dot` - annotation of affinities per graph. This file is written to the disk during execution of `ICNNNetwork::LoadNetwork()` for the heterogeneous plugin
|
||||
|
||||
#### To Generate the .dot Files
|
||||
|
||||
```python
|
||||
ie = IECore()
|
||||
ie.set_config( config={'HETERO_DUMP_GRAPH_DOT' : 'YES'}, device_name='HETERO')
|
||||
```
|
||||
|
||||
You can use the GraphViz* utility or a file converter to view the images. On the Ubuntu* operating system, you can use xdot:
|
||||
|
||||
* `sudo apt-get install xdot`
|
||||
* `xdot hetero_subgraphs.dot`
|
||||
|
||||
You can use performance data (in sample applications, it is the option `-pc`) to get the performance data on each subgraph.
|
||||
|
||||
Here is an example of the output for Googlenet v1 running on HDDL with fallback to CPU:
|
||||
|
||||
```
|
||||
subgraph1: 1. input preprocessing (mean data/HDDL):EXECUTED layerType: realTime: 129 cpu: 129 execType:
|
||||
subgraph1: 2. input transfer to DDR:EXECUTED layerType: realTime: 201 cpu: 0 execType:
|
||||
subgraph1: 3. HDDL execute time:EXECUTED layerType: realTime: 3808 cpu: 0 execType:
|
||||
subgraph1: 4. output transfer from DDR:EXECUTED layerType: realTime: 55 cpu: 0 execType:
|
||||
subgraph1: 5. HDDL output postprocessing:EXECUTED layerType: realTime: 7 cpu: 7 execType:
|
||||
subgraph1: 6. copy to IE blob:EXECUTED layerType: realTime: 2 cpu: 2 execType:
|
||||
subgraph2: out_prob: NOT_RUN layerType: Output realTime: 0 cpu: 0 execType: unknown
|
||||
subgraph2: prob: EXECUTED layerType: SoftMax realTime: 10 cpu: 10 execType: ref
|
||||
Total time: 4212 microseconds
|
||||
```
|
||||
|
||||
|
||||
### See Also
|
||||
[Supported Devices](Supported_Devices.md)
|
||||
@@ -32,7 +32,7 @@ Following the OpenVINO™ convention of labeling devices, the Multi-Device plugi
|
||||
| "MULTI_DEVICE_PRIORITIES" | comma-separated device names with no spaces | N/A | Prioritized list of devices |
|
||||
|
||||
You can set the configuration directly as a string, or use the metric key `MultiDeviceConfigParams::KEY_MULTI_DEVICE_PRIORITIES from the `multi/multi_device_config.hpp` file, which defines the same string.
|
||||
|
||||
|
||||
Basically, there are three ways to specify the devices to be use by the "MULTI":
|
||||
|
||||
@snippet snippets/MULTI0.cpp part0
|
||||
@@ -86,7 +86,7 @@ Note that while the performance of accelerators combines really well with Multi-
|
||||
See the [Using the Multi-Device with OpenVINO samples and benchmarking the performance](#using-the-multi-device-with-openvino-samples-and-benchmarking-the-performance) section below.
|
||||
|
||||
### Querying the Optimal Number of Inference Requests
|
||||
You can use the new GetMetric API to query the optimal number of requests. Similarly, when using the Multi-Device you don't need to sum over included devices yourself, you can query metric directly:
|
||||
You can use the [Properties API](../PropertiesAPI.md) API to query the optimal number of requests. Similarly, when using the Multi-Device you don't need to sum over included devices yourself, you can query property directly:
|
||||
|
||||
@snippet snippets/MULTI5.cpp part5
|
||||
|
||||
|
||||
@@ -9,16 +9,16 @@ uint32_t n_streams = 2;
|
||||
int64_t available_device_mem_size = 3221225472;
|
||||
ov::AnyMap options = {
|
||||
ov::hint::model(model), // Required. Set the address of the target network. If this is not set, the MAX_BATCH_SIZE returns 1.
|
||||
ov::num_streams(n_streams), // Optional. Set only when you want to estimate max batch size for a specific throughtput streams. Default is 1 or throughtput streams set by SetConfig.
|
||||
ov::num_streams(n_streams), // Optional. Set only when you want to estimate max batch size for a specific throughtput streams. Default is 1 or throughtput streams set by set_property.
|
||||
ov::intel_gpu::hint::available_device_mem(available_device_mem_size) // Optional. Set only when you want to limit the available device mem size.
|
||||
};
|
||||
|
||||
uint32_t max_batch_size = core.get_property("GPU", ov::max_batch_size, options);
|
||||
//! [part1]
|
||||
//! [part2]
|
||||
// This is not entirely GPU-specific metric (so METRIC_KEY is used rather than GPU_METRIC_KEY below),
|
||||
// This is not entirely GPU-specific property (so common `ov::` property is used rather than `ov::intel_gpu::` below),
|
||||
// but the GPU is the only device that supports that at the moment.
|
||||
// For the GPU, the metric already accommodates limitation for the on-device memory that the MAX_BATCH_SIZE poses.
|
||||
// For the GPU, the property already accommodates limitation for the on-device memory that the MAX_BATCH_SIZE poses.
|
||||
// so OPTIMAL_BATCH_SIZE is always less than MAX_BATCH_SIZE. Unlike the latter it is also aligned to the power of 2.
|
||||
uint32_t optimal_batch_size = core.get_property("GPU", ov::optimal_batch_size, options);
|
||||
//! [part2]
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
#include <openvino/runtime/core.hpp>
|
||||
|
||||
int main() {
|
||||
ov::Core core;
|
||||
auto model = core.read_model("sample.xml");
|
||||
//! [set_manual_affinities]
|
||||
for (auto && op : model->get_ops()) {
|
||||
op->get_rt_info()["affinity"] = "CPU";
|
||||
}
|
||||
//! [set_manual_affinities]
|
||||
|
||||
//! [fix_automatic_affinities]
|
||||
// This example demonstrates how to perform default affinity initialization and then
|
||||
// correct affinity manually for some layers
|
||||
const std::string device = "HETERO:GPU,CPU";
|
||||
|
||||
// query_model result contains mapping of supported operations to devices
|
||||
auto supported_ops = core.query_model(model, device);
|
||||
|
||||
// update default affinities manually for specific operations
|
||||
supported_ops["operation_name"] = "CPU";
|
||||
|
||||
// set affinities to a model
|
||||
for (auto&& node : model->get_ops()) {
|
||||
auto& affinity = supported_ops[node->get_friendly_name()];
|
||||
// Store affinity mapping using op runtime information
|
||||
node->get_rt_info()["affinity"] = affinity;
|
||||
}
|
||||
|
||||
// load model with manually set affinities
|
||||
auto compiled_model = core.compile_model(model, device);
|
||||
//! [fix_automatic_affinities]
|
||||
|
||||
//! [compile_model]
|
||||
{
|
||||
auto compiled_model = core.compile_model(model, "HETERO:GPU,CPU");
|
||||
// or with ov::device::priorities with multiple args
|
||||
compiled_model = core.compile_model(model, "HETERO", ov::device::priorities("GPU", "CPU"));
|
||||
// or with ov::device::priorities with a single argument
|
||||
compiled_model = core.compile_model(model, "HETERO", ov::device::priorities("GPU,CPU"));
|
||||
}
|
||||
//! [compile_model]
|
||||
{
|
||||
//! [configure_fallback_devices]
|
||||
auto compiled_model = core.compile_model(model, "HETERO",
|
||||
ov::device::priorities("GPU", "CPU"), // GPU with fallback to CPU
|
||||
ov::device::properties("CPU", ov::enable_profiling(true)), // profiling is enabled only for CPU
|
||||
ov::device::properties("GPU", ov::hint::inference_precision(ov::element::f16)) // FP16 inference precision only for GPU
|
||||
);
|
||||
//! [configure_fallback_devices]
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
#include <openvino/runtime/core.hpp>
|
||||
|
||||
int main() {
|
||||
ov::Core core;
|
||||
auto model = core.read_model("sample.xml");
|
||||
//! [set_manual_affinities]
|
||||
for (auto && op : model->get_ops()) {
|
||||
op->get_rt_info()["affinity"] = "CPU";
|
||||
}
|
||||
//! [set_manual_affinities]
|
||||
|
||||
//! [fix_automatic_affinities]
|
||||
// This example demonstrates how to perform default affinity initialization and then
|
||||
// correct affinity manually for some layers
|
||||
const std::string device = "HETERO:GPU,CPU";
|
||||
|
||||
// query_model result contains mapping of supported operations to devices
|
||||
auto supported_ops = core.query_model(model, device);
|
||||
|
||||
// update default affinities manually for specific operations
|
||||
supported_ops["operation_name"] = "CPU";
|
||||
|
||||
// set affinities to a model
|
||||
for (auto&& node : model->get_ops()) {
|
||||
auto& affinity = supported_ops[node->get_friendly_name()];
|
||||
// Store affinity mapping using op runtime information
|
||||
node->get_rt_info()["affinity"] = affinity;
|
||||
}
|
||||
|
||||
// load model with manually set affinities
|
||||
auto compiled_model = core.compile_model(model, device);
|
||||
//! [fix_automatic_affinities]
|
||||
|
||||
//! [compile_model]
|
||||
{
|
||||
auto compiled_model = core.compile_model(model, "HETERO:GPU,CPU");
|
||||
}
|
||||
{
|
||||
auto compiled_model = core.compile_model(model, "HETERO", ov::device::priorities("GPU", "CPU"));
|
||||
}
|
||||
{
|
||||
auto compiled_model = core.compile_model(model, "HETERO", ov::device::priorities("GPU,CPU"));
|
||||
}
|
||||
//! [compile_model]
|
||||
{
|
||||
//! [configure_fallback_devices]
|
||||
auto compiled_model = core.compile_model(model, "HETERO",
|
||||
ov::device::priorities("GPU", "CPU"), // GPU with fallback to CPU
|
||||
ov::device::properties("CPU", ov::enable_profiling(true)), // profiling is enabled only for CPU
|
||||
ov::device::properties("GPU", ov::hint::inference_precision(ov::element::f16)) // FP16 inference precision only for GPU
|
||||
);
|
||||
//! [configure_fallback_devices]
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
#include <openvino/runtime/core.hpp>
|
||||
|
||||
int main() {
|
||||
//! [part0]
|
||||
ov::Core core;
|
||||
auto available_devices = core.get_available_devices();
|
||||
//! [part0]
|
||||
|
||||
//! [part1]
|
||||
auto device_priorites = core.get_property("HETERO", ov::device::priorities);
|
||||
//! [part1]
|
||||
|
||||
//! [part2]
|
||||
auto cpu_device_name = core.get_property("GPU", ov::device::full_name);
|
||||
//! [part2]
|
||||
|
||||
//! [part3]
|
||||
auto model = core.read_model("sample.xml");
|
||||
{
|
||||
auto compiled_model = core.compile_model(model, "CPU",
|
||||
ov::hint::performance_mode(ov::hint::PerformanceMode::THROUGHPUT),
|
||||
ov::hint::inference_precision(ov::element::f32));
|
||||
}
|
||||
//! [part3]
|
||||
|
||||
//! [part4]
|
||||
{
|
||||
auto compiled_model = core.compile_model(model, "CPU");
|
||||
auto nireq = compiled_model.get_property(ov::optimal_number_of_infer_requests);
|
||||
}
|
||||
//! [part4]
|
||||
|
||||
//! [part5]
|
||||
{
|
||||
auto compiled_model = core.compile_model(model, "MYRIAD");
|
||||
auto temperature = compiled_model.get_property(ov::device::thermal);
|
||||
}
|
||||
//! [part5]
|
||||
|
||||
//! [part6]
|
||||
{
|
||||
auto compiled_model = core.compile_model(model, "CPU");
|
||||
auto nthreads = compiled_model.get_property(ov::inference_num_threads);
|
||||
}
|
||||
//! [part6]
|
||||
return 0;
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
# Hello Query Device C++ Sample {#openvino_inference_engine_samples_hello_query_device_README}
|
||||
|
||||
This sample demonstrates how to execute an query OpenVINO™ Runtime devices, prints their metrics and default configuration values, using [Query Device API feature](../../../docs/OV_Runtime_UG/InferenceEngine_QueryAPI.md).
|
||||
This sample demonstrates how to execute an query OpenVINO™ Runtime devices, prints their metrics and default configuration values, using [Properties API](../../../docs/OV_Runtime_UG/PropertiesAPI.md).
|
||||
|
||||
The following C++ API is used in the application:
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Hello Query Device Python* Sample {#openvino_inference_engine_ie_bridges_python_sample_hello_query_device_README}
|
||||
|
||||
This sample demonstrates how to show OpenVINO™ Runtime devices and prints their metrics and default configuration values using [Query Device API feature](../../../docs/OV_Runtime_UG/InferenceEngine_QueryAPI.md).
|
||||
This sample demonstrates how to show OpenVINO™ Runtime devices and prints their metrics and default configuration values using [Query Device API feature](../../../docs/OV_Runtime_UG/PropertiesAPI.md).
|
||||
|
||||
The following Python API is used in the application:
|
||||
|
||||
@@ -28,7 +28,7 @@ python hello_query_device.py
|
||||
|
||||
## Sample Output
|
||||
|
||||
The application prints all available devices with their supported metrics and default values for configuration parameters.
|
||||
The application prints all available devices with their supported metrics and default values for configuration parameters.
|
||||
For example:
|
||||
|
||||
```
|
||||
|
||||
Reference in new issue
Block a user