IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

DOCS shift to rst - Benchmark Samples and Tools (#16566)

Browse Source
This commit is contained in:
Sebastian Golebiewski
2023-03-27 18:29:05 +02:00
committed by GitHub
parent 5c5a29d095
commit 1ca94326cb
5 changed files with 796 additions and 566 deletions
Download Patch File Download Diff File Expand all files Collapse all files

77
samples/python/benchmark/bert_benchmark/README.md
View File

@@ -1,50 +1,69 @@
# Bert Benchmark Python* Sample {#openvino_inference_engine_ie_bridges_python_sample_bert_benchmark_README}
This sample demonstrates how to estimate performace of a Bert model using Asynchronous Inference Request API. Unlike [demos](@ref omz_demos) this sample doesn't have configurable command line arguments. Feel free to modify sample's source code to try out different options.
@sphinxdirective
The following Python\* API is used in the application:
This sample demonstrates how to estimate performance of a Bert model using Asynchronous Inference Request API. Unlike :doc:`demos <omz_demos>` this sample doesn't have configurable command line arguments. Feel free to modify sample's source code to try out different options.
| Feature | API | Description |
| :--- | :--- | :--- |
| OpenVINO Runtime Version | [openvino.runtime.get_version] | Get Openvino API version |
| Basic Infer Flow | [openvino.runtime.Core], [openvino.runtime.Core.compile_model] | Common API to do inference: compile a model |
| Asynchronous Infer | [openvino.runtime.AsyncInferQueue], [openvino.runtime.AsyncInferQueue.start_async], [openvino.runtime.AsyncInferQueue.wait_all] | Do asynchronous inference |
| Model Operations | [openvino.runtime.CompiledModel.inputs] | Get inputs of a model |
The following Python API is used in the application:
## How It Works
+--------------------------------+-------------------------------------------------+----------------------------------------------+
| Feature | API | Description |
+================================+=================================================+==============================================+
| OpenVINO Runtime Version | [openvino.runtime.get_version] | Get Openvino API version. |
+--------------------------------+-------------------------------------------------+----------------------------------------------+
| Basic Infer Flow | [openvino.runtime.Core], | Common API to do inference: compile a model. |
| | [openvino.runtime.Core.compile_model] | |
+--------------------------------+-------------------------------------------------+----------------------------------------------+
| Asynchronous Infer | [openvino.runtime.AsyncInferQueue], | Do asynchronous inference. |
| | [openvino.runtime.AsyncInferQueue.start_async], | |
| | [openvino.runtime.AsyncInferQueue.wait_all] | |
+--------------------------------+-------------------------------------------------+----------------------------------------------+
| Model Operations | [openvino.runtime.CompiledModel.inputs] | Get inputs of a model. |
+--------------------------------+-------------------------------------------------+----------------------------------------------+
The sample downloads a model and a tokenizer, export the model to onnx, reads the exported model and reshapes it to enforce dynamic inpus shapes, compiles the resulting model, downloads a dataset and runs benchmarking on the dataset.
How It Works
####################
The sample downloads a model and a tokenizer, export the model to onnx, reads the exported model and reshapes it to enforce dynamic input shapes, compiles the resulting model, downloads a dataset and runs benchmarking on the dataset.
You can see the explicit description of
each sample step at [Integration Steps](../../../../docs/OV_Runtime_UG/integrate_with_your_application.md) section of "Integrate OpenVINO™ Runtime with Your Application" guide.
each sample step at :doc:`Integration Steps <openvino_docs_OV_UG_Integrate_OV_with_your_application>` section of "Integrate OpenVINO™ Runtime with Your Application" guide.
## Running
Running
####################
Install the `openvino` Python package:
Install the ``openvino`` Python package:
```
python -m pip install openvino
```
.. code-block:: sh
Install packages from `requirements.txt`:
python -m pip install openvino
Install packages from ``requirements.txt``:
.. code-block:: sh
python -m pip install -r requirements.txt
```
python -m pip install -r requirements.txt
```
Run the sample
```
python bert_benchmark.py
```
.. code-block:: sh
## Sample Output
python bert_benchmark.py
Sample Output
####################
The sample outputs how long it takes to process a dataset.
## See Also
See Also
####################
- [Integrate the OpenVINO™ Runtime with Your Application](../../../../docs/OV_Runtime_UG/integrate_with_your_application.md)
- [Using OpenVINO™ Toolkit Samples](../../../../docs/OV_Runtime_UG/Samples_Overview.md)
- [Model Downloader](@ref omz_tools_downloader)
- [Model Optimizer](../../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md)
* :doc:`Integrate the OpenVINO™ Runtime with Your Application <openvino_docs_OV_UG_Integrate_OV_with_your_application>`
* :doc:`Using OpenVINO Samples <openvino_docs_OV_UG_Samples_Overview>`
* :doc:`Model Downloader <omz_tools_downloader>`
* :doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`
@endsphinxdirective

162
samples/python/benchmark/throughput_benchmark/README.md
View File

@@ -1,94 +1,134 @@
# Throughput Benchmark Python* Sample {#openvino_inference_engine_ie_bridges_python_sample_throughput_benchmark_README}
This sample demonstrates how to estimate performace of a model using Asynchronous Inference Request API in throughput mode. Unlike [demos](@ref omz_demos) this sample doesn't have other configurable command line arguments. Feel free to modify sample's source code to try out different options.
@sphinxdirective
The reported results may deviate from what [benchmark_app](../../../../tools/benchmark_tool/README.md) reports. One example is model input precision for computer vision tasks. benchmark_app sets uint8, while the sample uses default model precision which is usually float32.
The following Python\* API is used in the application:
This sample demonstrates how to estimate performance of a model using Asynchronous Inference Request API in throughput mode. Unlike :doc:`demos <omz_demos>` this sample doesn't have other configurable command line arguments. Feel free to modify sample's source code to try out different options.
| Feature | API | Description |
| :--- | :--- | :--- |
| OpenVINO Runtime Version | [openvino.runtime.get_version] | Get Openvino API version |
| Basic Infer Flow | [openvino.runtime.Core], [openvino.runtime.Core.compile_model], [openvino.runtime.InferRequest.get_tensor] | Common API to do inference: compile a model, configure input tensors |
| Asynchronous Infer | [openvino.runtime.AsyncInferQueue], [openvino.runtime.AsyncInferQueue.start_async], [openvino.runtime.AsyncInferQueue.wait_all], [openvino.runtime.InferRequest.results] | Do asynchronous inference |
| Model Operations | [openvino.runtime.CompiledModel.inputs] | Get inputs of a model |
| Tensor Operations | [openvino.runtime.Tensor.get_shape], [openvino.runtime.Tensor.data] | Get a tensor shape and its data. |
The reported results may deviate from what :doc:`benchmark_app <openvino_inference_engine_tools_benchmark_tool_README>` reports. One example is model input precision for computer vision tasks. benchmark_app sets uint8, while the sample uses default model precision which is usually float32.
| Options | Values |
| :--- | :--- |
| Validated Models | [alexnet](@ref omz_models_model_alexnet), [googlenet-v1](@ref omz_models_model_googlenet_v1) [yolo-v3-tf](@ref omz_models_model_yolo_v3_tf), [face-detection-0200](@ref omz_models_model_face_detection_0200) |
| Model Format | OpenVINO™ toolkit Intermediate Representation (\*.xml + \*.bin), ONNX (\*.onnx) |
| Supported devices | [All](../../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) |
| Other language realization | [C++](../../../cpp/benchmark/throughput_benchmark/README.md) |
The following Python API is used in the application:
## How It Works
+--------------------------------+-------------------------------------------------+----------------------------------------------+
| Feature | API | Description |
+================================+=================================================+==============================================+
| OpenVINO Runtime Version | [openvino.runtime.get_version] | Get Openvino API version. |
+--------------------------------+-------------------------------------------------+----------------------------------------------+
| Basic Infer Flow | [openvino.runtime.Core], | Common API to do inference: compile a model, |
| | [openvino.runtime.Core.compile_model] | configure input tensors. |
| | [openvino.runtime.InferRequest.get_tensor] | |
+--------------------------------+-------------------------------------------------+----------------------------------------------+
| Asynchronous Infer | [openvino.runtime.AsyncInferQueue], | Do asynchronous inference. |
| | [openvino.runtime.AsyncInferQueue.start_async], | |
| | [openvino.runtime.AsyncInferQueue.wait_all], | |
| | [openvino.runtime.InferRequest.results] | |
+--------------------------------+-------------------------------------------------+----------------------------------------------+
| Model Operations | [openvino.runtime.CompiledModel.inputs] | Get inputs of a model. |
+--------------------------------+-------------------------------------------------+----------------------------------------------+
| Tensor Operations | [openvino.runtime.Tensor.get_shape], | Get a tensor shape and its data. |
| | [openvino.runtime.Tensor.data] | |
+--------------------------------+-------------------------------------------------+----------------------------------------------+
+--------------------------------+------------------------------------------------------------------------------+
| Options | Values |
+================================+==============================================================================+
| Validated Models | :doc:`alexnet <omz_models_model_alexnet>`, |
| | :doc:`googlenet-v1 <omz_models_model_googlenet_v1>`, |
| | :doc:`yolo-v3-tf <omz_models_model_yolo_v3_tf>`, |
| | :doc:`face-detection-0200 <omz_models_model_face_detection_0200>` |
+--------------------------------+------------------------------------------------------------------------------+
| Model Format | OpenVINO™ toolkit Intermediate Representation |
| | (\*.xml + \*.bin), ONNX (\*.onnx) |
+--------------------------------+------------------------------------------------------------------------------+
| Supported devices | :doc:`All <openvino_docs_OV_UG_supported_plugins_Supported_Devices>` |
+--------------------------------+------------------------------------------------------------------------------+
| Other language realization | :doc:`C++ <openvino_inference_engine_samples_throughput_benchmark_README>` |
+--------------------------------+------------------------------------------------------------------------------+
How It Works
####################
The sample compiles a model for a given device, randomly generates input data, performs asynchronous inference multiple times for a given number of seconds. Then processes and reports performance results.
You can see the explicit description of
each sample step at [Integration Steps](../../../../docs/OV_Runtime_UG/integrate_with_your_application.md) section of "Integrate OpenVINO™ Runtime with Your Application" guide.
each sample step at :doc:`Integration Steps <openvino_docs_OV_UG_Integrate_OV_with_your_application>` section of "Integrate OpenVINO™ Runtime with Your Application" guide.
## Running
Running
####################
.. code-block:: sh
python throughput_benchmark.py <path_to_model>
```
python throughput_benchmark.py <path_to_model>
```
To run the sample, you need to specify a model:
- You can use [public](@ref omz_models_group_public) or [Intel's](@ref omz_models_group_intel) pre-trained models from the Open Model Zoo. The models can be downloaded using the [Model Downloader](@ref omz_tools_downloader).
> **NOTES**:
>
> - Before running the sample with a trained model, make sure the model is converted to the intermediate representation (IR) format (\*.xml + \*.bin) using the [Model Optimizer tool](../../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md).
>
> - The sample accepts models in ONNX format (.onnx) that do not require preprocessing.
- You can use :doc:`public <omz_models_group_public>` or :doc:`Intel's <omz_models_group_intel>` pre-trained models from the Open Model Zoo. The models can be downloaded using the :doc:`Model Downloader <omz_tools_downloader>`.
### Example
.. note::
1. Install the `openvino-dev` Python package to use Open Model Zoo Tools:
Before running the sample with a trained model, make sure the model is converted to the intermediate representation (IR) format (\*.xml + \*.bin) using the :doc:`Model Optimizer tool <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`.
The sample accepts models in ONNX format (.onnx) that do not require preprocessing.
Example
++++++++++++++++++++
1. Install the ``openvino-dev`` Python package to use Open Model Zoo Tools:
.. code-block:: sh
python -m pip install openvino-dev[caffe]
```
python -m pip install openvino-dev[caffe]
```
2. Download a pre-trained model using:
```
omz_downloader --name googlenet-v1
```
.. code-block:: sh
omz_downloader --name googlenet-v1
3. If a model is not in the IR or ONNX format, it must be converted. You can do this using the model converter:
```
omz_converter --name googlenet-v1
```
.. code-block:: sh
4. Perform benchmarking using the `googlenet-v1` model on a `CPU`:
omz_converter --name googlenet-v1
```
python throughput_benchmark.py googlenet-v1.xml
```
## Sample Output
4. Perform benchmarking using the ``googlenet-v1`` model on a ``CPU``:
.. code-block:: sh
python throughput_benchmark.py googlenet-v1.xml
Sample Output
####################
The application outputs performance results.
```
[ INFO ] OpenVINO:
[ INFO ] Build ................................. <version>
[ INFO ] Count: 2817 iterations
[ INFO ] Duration: 10012.65 ms
[ INFO ] Latency:
[ INFO ] Median: 13.80 ms
[ INFO ] Average: 14.10 ms
[ INFO ] Min: 8.35 ms
[ INFO ] Max: 28.38 ms
[ INFO ] Throughput: 281.34 FPS
```
.. code-block:: sh
## See Also
[ INFO ] OpenVINO:
[ INFO ] Build ................................. <version>
[ INFO ] Count: 2817 iterations
[ INFO ] Duration: 10012.65 ms
[ INFO ] Latency:
[ INFO ] Median: 13.80 ms
[ INFO ] Average: 14.10 ms
[ INFO ] Min: 8.35 ms
[ INFO ] Max: 28.38 ms
[ INFO ] Throughput: 281.34 FPS
- [Integrate the OpenVINO™ Runtime with Your Application](../../../../docs/OV_Runtime_UG/integrate_with_your_application.md)
- [Using OpenVINO™ Toolkit Samples](../../../../docs/OV_Runtime_UG/Samples_Overview.md)
- [Model Downloader](@ref omz_tools_downloader)
- [Model Optimizer](../../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md)
See Also
####################
* :doc:`Integrate the OpenVINO™ Runtime with Your Application <openvino_docs_OV_UG_Integrate_OV_with_your_application>`
* :doc:`Using OpenVINO Samples <openvino_docs_OV_UG_Samples_Overview>`
* :doc:`Model Downloader <omz_tools_downloader>`
* :doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`
@endsphinxdirective