IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Requirements for the HW plugin to integrate with AUTO (#17053)

Browse Source 
* Requirements for the HW plugin to integrate with AUTO

Signed-off-by: Peter Chen <peter.chen@intel.com>

* Update property requirements and wording

1. Added purpose for each reqired property
2. Removed autobatching properties
3. Updated wording

Signed-off-by: Peter Chen <peter.chen@intel.com>

* Add one BA test and update purpose for model_name

Signed-off-by: Peter Chen <peter.chen@intel.com>

* Add request to ov::compilation_num_threads

Signed-off-by: Peter Chen <peter.chen@intel.com>

* Add link to intgration with AUTO

Signed-off-by: Peter Chen <peter.chen@intel.com>

* Wording with API 2.0

Co-authored-by: Ilya Churaev <ilyachur@gmail.com>

* Try to fix the link

* Remove ":doc:"

* Add postfix "__" for external link

* Apply suggestions from code review

Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>

* bash command and multiple devices description update

Signed-off-by: Peter Chen <peter.chen@intel.com>

---------

Signed-off-by: Peter Chen <peter.chen@intel.com>
Co-authored-by: Ilya Churaev <ilyachur@gmail.com>
Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>
This commit is contained in:
Chen Peter 2023-06-02 15:04:33 +08:00 committed by GitHub
parent e1ed6599f0
commit 031f2cc7d1
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 52 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
docs/IE_PLUGIN_DG/Intro.md
View File

@ -83,6 +83,7 @@ Detailed Guides
* :doc:`Quantized networks <openvino_docs_ov_plugin_dg_quantized_models>`
* :doc:`Low precision transformations <openvino_docs_OV_UG_lpt>` guide
* :doc:`Writing OpenVINO™ transformations <openvino_docs_transformations>` guide
* `Integration with AUTO Plugin <https://github.com/openvinotoolkit/openvino/blob/master/src/plugins/auto/docs/integration.md>`__
API References
##############

53
src/plugins/auto/docs/integration.md
View File

@ -1,5 +1,54 @@
# AUTO Plugin Integration
## How to integrate a new hardware plugin
## Implement a New Plugin
Refer to [OpenVINO Plugin Developer Guide](https://docs.openvino.ai/latest/openvino_docs_ie_plugin_dg_overview.html) for detailed information on how to implement a new plugin.
TBD-100527
Query model method `ov::IPlugin::query_model()` is recommended as it is important for AUTO to quickly make decisions and save selection time.
## AUTO Plugin Property Requirements
AUTO Plugin requires the following plugin properties:
| Property | Mandatory | Purpose |
| ------------------------------------ | -------- | --------------------------------------------- |
| ov::device::id | Yes | Distinguish devices with the same type. |
| ov::enable_profiling | Yes | Performance profiling. |
| ov::hint::performance_mode | Yes | Performance mode hint. |
| ov::hint::num_requests | Yes | num_requests hint. |
| ov::device::full_name | Yes | Automatic device selection. |
| ov::model_name | Yes | Return model name. |
| ov::optimal_batch_size | No | Decide batch size in automatic batching case. |
| ov::optimal_number_of_infer_requests | Yes | Decide AUTO optimal_number_of_infer_requests. |
| ov::range_for_streams | Yes | Decide AUTO optimal_number_of_infer_requests in automatic batching case. |
| ov::supported_properties | Yes | Check if a property is supported by HW plugin.|
| ov::device::capabilities | Yes | Automatic device selection. |
| ov::device::gops | No | Improve automatic device selection. |
| ov::compilation_num_threads | No | Limit the compilation threads for a single device when compiling a model to multiple devices. |
## AUTO Plugin Tests
Refer to the [Testing the AUTO Plugin](./tests.md) page for detailed instructions.
## AUTO Plugin Integration Tests
### Test AUTO and Hardware Plugins Using benchmark_app
```sh
benchmark_app -d ${device} -hint ${hint} -m <any model works on HW plugin>
```
| hint | device |
| --------------------- | ------------- |
| throughput | \<HW> |
| throughput | AUTO:\<HW> |
| throughput | AUTO:\<HW>,CPU |
| latency | \<HW> |
| latency | AUTO:\<HW> |
| latency | AUTO:\<HW>,CPU |
| cumulative_throughput | AUTO:\<HW> |
| cumulative_throughput | AUTO:\<HW>,CPU |
### Test Multiple Devices Running Simultaneously
The HW plugin must guarantee simultaneous execution of multiple devices in different threads. It is recommended to test the HW plugin with the CPU plugin by running the plugins in different threads simultaneously.
For example, there may be two GPUs on the same system, with device names GPU.0 and GPU.1. GPU plugin must guarantee simultaneous execution of GPU.0 and GPU.1 in different threads.