DOCS shift to rst - Benchmark Samples and Tools (#16566)
This commit is contained in:
Sebastian Golebiewski authored and 
GitHub committed
1 parent
5c5a29d095
commit
1ca94326cb
5 files changed
+796
-566
No files matched your search
@@ -1,98 +1,135 @@
|
||||
# Throughput Benchmark C++ Sample {#openvino_inference_engine_samples_throughput_benchmark_README}
|
||||
|
||||
This sample demonstrates how to estimate performace of a model using Asynchronous Inference Request API in throughput mode. Unlike [demos](@ref omz_demos) this sample doesn't have other configurable command line arguments. Feel free to modify sample's source code to try out different options.
|
||||
@sphinxdirective
|
||||
|
||||
The reported results may deviate from what [benchmark_app](../../benchmark_app/README.md) reports. One example is model input precision for computer vision tasks. benchmark_app sets uint8, while the sample uses default model precision which is usually float32.
|
||||
This sample demonstrates how to estimate performance of a model using Asynchronous Inference Request API in throughput mode. Unlike :doc:`demos <omz_demos>` this sample doesn't have other configurable command line arguments. Feel free to modify sample's source code to try out different options.
|
||||
|
||||
The reported results may deviate from what :doc:`benchmark_app <openvino_inference_engine_samples_benchmark_app_README>` reports. One example is model input precision for computer vision tasks. benchmark_app sets ``uint8``, while the sample uses default model precision which is usually ``float32``.
|
||||
|
||||
The following C++ API is used in the application:
|
||||
|
||||
| Feature | API | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| OpenVINO Runtime Version | `ov::get_openvino_version` | Get Openvino API version |
|
||||
| Basic Infer Flow | `ov::Core`, `ov::Core::compile_model`, `ov::CompiledModel::create_infer_request`, `ov::InferRequest::get_tensor` | Common API to do inference: compile a model, create an infer request, configure input tensors |
|
||||
| Asynchronous Infer | `ov::InferRequest::start_async`, `ov::InferRequest::set_callback` | Do asynchronous inference with callback. |
|
||||
| Model Operations | `ov::CompiledModel::inputs` | Get inputs of a model |
|
||||
| Tensor Operations | `ov::Tensor::get_shape`, `ov::Tensor::data` | Get a tensor shape and its data. |
|
||||
+--------------------------+----------------------------------------------+----------------------------------------------+
|
||||
| Feature | API | Description |
|
||||
+==========================+==============================================+==============================================+
|
||||
| OpenVINO Runtime Version | ``ov::get_openvino_version`` | Get Openvino API version. |
|
||||
+--------------------------+----------------------------------------------+----------------------------------------------+
|
||||
| Basic Infer Flow | ``ov::Core``, ``ov::Core::compile_model``, | Common API to do inference: compile a model, |
|
||||
| | ``ov::CompiledModel::create_infer_request``, | create an infer request, |
|
||||
| | ``ov::InferRequest::get_tensor`` | configure input tensors. |
|
||||
+--------------------------+----------------------------------------------+----------------------------------------------+
|
||||
| Asynchronous Infer | ``ov::InferRequest::start_async``, | Do asynchronous inference with callback. |
|
||||
| | ``ov::InferRequest::set_callback`` | |
|
||||
+--------------------------+----------------------------------------------+----------------------------------------------+
|
||||
| Model Operations | ``ov::CompiledModel::inputs`` | Get inputs of a model. |
|
||||
+--------------------------+----------------------------------------------+----------------------------------------------+
|
||||
| Tensor Operations | ``ov::Tensor::get_shape``, | Get a tensor shape and its data. |
|
||||
| | ``ov::Tensor::data`` | |
|
||||
+--------------------------+----------------------------------------------+----------------------------------------------+
|
||||
|
||||
| Options | Values |
|
||||
| :--- | :--- |
|
||||
| Validated Models | [alexnet](@ref omz_models_model_alexnet), [googlenet-v1](@ref omz_models_model_googlenet_v1) [yolo-v3-tf](@ref omz_models_model_yolo_v3_tf), [face-detection-0200](@ref omz_models_model_face_detection_0200) |
|
||||
| Model Format | OpenVINO™ toolkit Intermediate Representation (\*.xml + \*.bin), ONNX (\*.onnx) |
|
||||
| Supported devices | [All](../../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) |
|
||||
| Other language realization | [Python](../../../python/benchmark/throughput_benchmark/README.md) |
|
||||
+--------------------------------+------------------------------------------------------------------------------------------------+
|
||||
| Options | Values |
|
||||
+================================+================================================================================================+
|
||||
| Validated Models | :doc:`alexnet <omz_models_model_alexnet>`, |
|
||||
| | :doc:`googlenet-v1 <omz_models_model_googlenet_v1>`, |
|
||||
| | :doc:`yolo-v3-tf <omz_models_model_yolo_v3_tf>`, |
|
||||
| | :doc:`face-detection-0200 <omz_models_model_face_detection_0200>` |
|
||||
+--------------------------------+------------------------------------------------------------------------------------------------+
|
||||
| Model Format | OpenVINO™ toolkit Intermediate Representation |
|
||||
| | (\*.xml + \*.bin), ONNX (\*.onnx) |
|
||||
+--------------------------------+------------------------------------------------------------------------------------------------+
|
||||
| Supported devices | :doc:`All <openvino_docs_OV_UG_supported_plugins_Supported_Devices>` |
|
||||
+--------------------------------+------------------------------------------------------------------------------------------------+
|
||||
| Other language realization | :doc:`Python <openvino_inference_engine_ie_bridges_python_sample_throughput_benchmark_README>` |
|
||||
+--------------------------------+------------------------------------------------------------------------------------------------+
|
||||
|
||||
## How It Works
|
||||
|
||||
How It Works
|
||||
####################
|
||||
|
||||
The sample compiles a model for a given device, randomly generates input data, performs asynchronous inference multiple times for a given number of seconds. Then processes and reports performance results.
|
||||
|
||||
You can see the explicit description of
|
||||
each sample step at [Integration Steps](../../../../docs/OV_Runtime_UG/integrate_with_your_application.md) section of "Integrate OpenVINO™ Runtime with Your Application" guide.
|
||||
each sample step at :doc:`Integration Steps <openvino_docs_OV_UG_Integrate_OV_with_your_application>` section of "Integrate OpenVINO™ Runtime with Your Application" guide.
|
||||
|
||||
## Building
|
||||
Building
|
||||
####################
|
||||
|
||||
To build the sample, please use instructions available at [Build the Sample Applications](../../../../docs/OV_Runtime_UG/Samples_Overview.md) section in OpenVINO™ Toolkit Samples guide.
|
||||
To build the sample, please use instructions available at :doc:`Build the Sample Applications <openvino_docs_OV_UG_Samples_Overview>` section in OpenVINO™ Toolkit Samples guide.
|
||||
|
||||
## Running
|
||||
Running
|
||||
####################
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
throughput_benchmark <path_to_model>
|
||||
|
||||
```
|
||||
throughput_benchmark <path_to_model>
|
||||
```
|
||||
|
||||
To run the sample, you need to specify a model:
|
||||
- You can use [public](@ref omz_models_group_public) or [Intel's](@ref omz_models_group_intel) pre-trained models from the Open Model Zoo. The models can be downloaded using the [Model Downloader](@ref omz_tools_downloader).
|
||||
|
||||
> **NOTES**:
|
||||
>
|
||||
> - Before running the sample with a trained model, make sure the model is converted to the intermediate representation (IR) format (\*.xml + \*.bin) using the [Model Optimizer tool](../../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md).
|
||||
>
|
||||
> - The sample accepts models in ONNX format (.onnx) that do not require preprocessing.
|
||||
- You can use :doc:`public <omz_models_group_public>` or :doc:`Intel's <omz_models_group_intel>` pre-trained models from the Open Model Zoo. The models can be downloaded using the :doc:`Model Downloader <omz_tools_downloader>`.
|
||||
|
||||
### Example
|
||||
.. note::
|
||||
|
||||
1. Install the `openvino-dev` Python package to use Open Model Zoo Tools:
|
||||
Before running the sample with a trained model, make sure the model is converted to the intermediate representation (IR) format (\*.xml + \*.bin) using the :doc:`Model Optimizer tool <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`.
|
||||
|
||||
The sample accepts models in ONNX format (.onnx) that do not require preprocessing.
|
||||
|
||||
Example
|
||||
++++++++++++++++++++
|
||||
|
||||
1. Install the ``openvino-dev`` Python package to use Open Model Zoo Tools:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
python -m pip install openvino-dev[caffe]
|
||||
|
||||
```
|
||||
python -m pip install openvino-dev[caffe]
|
||||
```
|
||||
|
||||
2. Download a pre-trained model using:
|
||||
|
||||
```
|
||||
omz_downloader --name googlenet-v1
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
omz_downloader --name googlenet-v1
|
||||
|
||||
|
||||
3. If a model is not in the IR or ONNX format, it must be converted. You can do this using the model converter:
|
||||
|
||||
```
|
||||
omz_converter --name googlenet-v1
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
4. Perform benchmarking using the `googlenet-v1` model on a `CPU`:
|
||||
omz_converter --name googlenet-v1
|
||||
|
||||
```
|
||||
throughput_benchmark googlenet-v1.xml
|
||||
```
|
||||
|
||||
## Sample Output
|
||||
4. Perform benchmarking using the ``googlenet-v1`` model on a ``CPU``:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
throughput_benchmark googlenet-v1.xml
|
||||
|
||||
|
||||
Sample Output
|
||||
####################
|
||||
|
||||
The application outputs performance results.
|
||||
|
||||
```
|
||||
[ INFO ] OpenVINO:
|
||||
[ INFO ] Build ................................. <version>
|
||||
[ INFO ] Count: 1577 iterations
|
||||
[ INFO ] Duration: 15024.2 ms
|
||||
[ INFO ] Latency:
|
||||
[ INFO ] Median: 38.02 ms
|
||||
[ INFO ] Average: 38.08 ms
|
||||
[ INFO ] Min: 25.23 ms
|
||||
[ INFO ] Max: 49.16 ms
|
||||
[ INFO ] Throughput: 104.96 FPS
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
## See Also
|
||||
[ INFO ] OpenVINO:
|
||||
[ INFO ] Build ................................. <version>
|
||||
[ INFO ] Count: 1577 iterations
|
||||
[ INFO ] Duration: 15024.2 ms
|
||||
[ INFO ] Latency:
|
||||
[ INFO ] Median: 38.02 ms
|
||||
[ INFO ] Average: 38.08 ms
|
||||
[ INFO ] Min: 25.23 ms
|
||||
[ INFO ] Max: 49.16 ms
|
||||
[ INFO ] Throughput: 104.96 FPS
|
||||
|
||||
- [Integrate the OpenVINO™ Runtime with Your Application](../../../../docs/OV_Runtime_UG/integrate_with_your_application.md)
|
||||
- [Using OpenVINO™ Toolkit Samples](../../../../docs/OV_Runtime_UG/Samples_Overview.md)
|
||||
- [Model Downloader](@ref omz_tools_downloader)
|
||||
- [Model Optimizer](../../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md)
|
||||
|
||||
See Also
|
||||
####################
|
||||
|
||||
* :doc:`Integrate the OpenVINO™ Runtime with Your Application <openvino_docs_OV_UG_Integrate_OV_with_your_application>`
|
||||
* :doc:`Using OpenVINO Samples <openvino_docs_OV_UG_Samples_Overview>`
|
||||
* :doc:`Model Downloader <omz_tools_downloader>`
|
||||
* :doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`
|
||||
|
||||
@endsphinxdirective
|
||||
+237
-169
@@ -1,246 +1,306 @@
|
||||
# Benchmark C++ Tool {#openvino_inference_engine_samples_benchmark_app_README}
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
This page demonstrates how to use the Benchmark C++ Tool to estimate deep learning inference performance on supported devices.
|
||||
|
||||
> **NOTE**: This page describes usage of the C++ implementation of the Benchmark Tool. For the Python implementation, refer to the [Benchmark Python Tool](../../../tools/benchmark_tool/README.md) page. The Python version is recommended for benchmarking models that will be used in Python applications, and the C++ version is recommended for benchmarking models that will be used in C++ applications. Both tools have a similar command interface and backend.
|
||||
.. note::
|
||||
|
||||
This page describes usage of the C++ implementation of the Benchmark Tool. For the Python implementation, refer to the :doc:`Benchmark Python Tool <openvino_inference_engine_tools_benchmark_tool_README>` page. The Python version is recommended for benchmarking models that will be used in Python applications, and the C++ version is recommended for benchmarking models that will be used in C++ applications. Both tools have a similar command interface and backend.
|
||||
|
||||
|
||||
## Basic Usage
|
||||
To use the C++ benchmark_app, you must first build it following the [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) instructions and then set up paths and environment variables by following the [Get Ready for Running the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) instructions. Navigate to the directory where the benchmark_app C++ sample binary was built.
|
||||
Basic Usage
|
||||
####################
|
||||
|
||||
> **NOTE**: If you installed OpenVINO Runtime using PyPI or Anaconda Cloud, only the [Benchmark Python Tool](../../../tools/benchmark_tool/README.md) is available, and you should follow the usage instructions on that page instead.
|
||||
To use the C++ benchmark_app, you must first build it following the :doc:`Build the Sample Applications <openvino_docs_OV_UG_Samples_Overview>` instructions and then set up paths and environment variables by following the :doc:`Get Ready for Running the Sample Applications <openvino_docs_OV_UG_Samples_Overview>` instructions. Navigate to the directory where the benchmark_app C++ sample binary was built.
|
||||
|
||||
The benchmarking application works with models in the OpenVINO IR (`model.xml` and `model.bin`) and ONNX (`model.onnx`) formats. Make sure to [convert your models](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) if necessary.
|
||||
.. note::
|
||||
|
||||
If you installed OpenVINO Runtime using PyPI or Anaconda Cloud, only the :doc:`Benchmark Python Tool <openvino_inference_engine_tools_benchmark_tool_README>` is available, and you should follow the usage instructions on that page instead.
|
||||
|
||||
The benchmarking application works with models in the OpenVINO IR (``model.xml`` and ``model.bin``) and ONNX (``model.onnx``) formats. Make sure to :doc:`convert your models <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>` if necessary.
|
||||
|
||||
To run benchmarking with default options on a model, use the following command:
|
||||
|
||||
```
|
||||
./benchmark_app -m model.xml
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
./benchmark_app -m model.xml
|
||||
|
||||
|
||||
By default, the application will load the specified model onto the CPU and perform inferencing on batches of randomly-generated data inputs for 60 seconds. As it loads, it prints information about benchmark parameters. When benchmarking is completed, it reports the minimum, average, and maximum inferencing latency and average the throughput.
|
||||
|
||||
You may be able to improve benchmark results beyond the default configuration by configuring some of the execution parameters for your model. For example, you can use "throughput" or "latency" performance hints to optimize the runtime for higher FPS or reduced inferencing time. Read on to learn more about the configuration options available with benchmark_app.
|
||||
|
||||
## Configuration Options
|
||||
The benchmark app provides various options for configuring execution parameters. This section covers key configuration options for easily tuning benchmarking to achieve better performance on your device. A list of all configuration options is given in the [Advanced Usage](#advanced-usage-cpp-benchmark) section.
|
||||
Configuration Options
|
||||
#####################
|
||||
|
||||
The benchmark app provides various options for configuring execution parameters. This section covers key configuration options for easily tuning benchmarking to achieve better performance on your device. A list of all configuration options is given in the :ref:`Advanced Usage <advanced-usage-cpp-benchmark>` section.
|
||||
|
||||
Performance hints: latency and throughput
|
||||
+++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
### Performance hints: latency and throughput
|
||||
The benchmark app allows users to provide high-level "performance hints" for setting latency-focused or throughput-focused inference modes. This hint causes the runtime to automatically adjust runtime parameters, such as the number of processing streams and inference batch size, to prioritize for reduced latency or high throughput.
|
||||
|
||||
The performance hints do not require any device-specific settings and they are completely portable between devices. Parameters are automatically configured based on whichever device is being used. This allows users to easily port applications between hardware targets without having to re-determine the best runtime parameters for the new device.
|
||||
|
||||
If not specified, throughput is used as the default. To set the hint explicitly, use `-hint latency` or `-hint throughput` when running benchmark_app:
|
||||
If not specified, throughput is used as the default. To set the hint explicitly, use ``-hint latency`` or ``-hint throughput`` when running benchmark_app:
|
||||
|
||||
```
|
||||
./benchmark_app -m model.xml -hint latency
|
||||
./benchmark_app -m model.xml -hint throughput
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
> **NOTE**
|
||||
It is up to the user to ensure the environment on which the benchmark is running is optimized for maximum performance.
|
||||
Otherwise, different results may occur when using the application in different environment settings (such as power optimization settings, processor overclocking, thermal throttling).
|
||||
./benchmark_app -m model.xml -hint latency
|
||||
./benchmark_app -m model.xml -hint throughput
|
||||
|
||||
|
||||
.. note::
|
||||
|
||||
It is up to the user to ensure the environment on which the benchmark is running is optimized for maximum performance. Otherwise, different results may occur when using the application in different environment settings (such as power optimization settings, processor overclocking, thermal throttling).
|
||||
|
||||
Latency
|
||||
--------------------
|
||||
|
||||
#### Latency
|
||||
Latency is the amount of time it takes to process a single inference request. In applications where data needs to be inferenced and acted on as quickly as possible (such as autonomous driving), low latency is desirable. For conventional devices, lower latency is achieved by reducing the amount of parallel processing streams so the system can utilize as many resources as possible to quickly calculate each inference request. However, advanced devices like multi-socket CPUs and modern GPUs are capable of running multiple inference requests while delivering the same latency.
|
||||
|
||||
When benchmark_app is run with `-hint latency`, it determines the optimal number of parallel inference requests for minimizing latency while still maximizing the parallelization capabilities of the hardware. It automatically sets the number of processing streams and inference batch size to achieve the best latency.
|
||||
When benchmark_app is run with ``-hint latency``, it determines the optimal number of parallel inference requests for minimizing latency while still maximizing the parallelization capabilities of the hardware. It automatically sets the number of processing streams and inference batch size to achieve the best latency.
|
||||
|
||||
Throughput
|
||||
--------------------
|
||||
|
||||
#### Throughput
|
||||
Throughput is the amount of data an inferencing pipeline can process at once, and it is usually measured in frames per second (FPS) or inferences per second. In applications where large amounts of data needs to be inferenced simultaneously (such as multi-camera video streams), high throughput is needed. To achieve high throughput, the runtime focuses on fully saturating the device with enough data to process. It utilizes as much memory and as many parallel streams as possible to maximize the amount of data that can be processed simultaneously.
|
||||
|
||||
When benchmark_app is run with `-hint throughput`, it maximizes the number of parallel inference requests to utilize all the threads available on the device. On GPU, it automatically sets the inference batch size to fill up the GPU memory available.
|
||||
When benchmark_app is run with ``-hint throughput``, it maximizes the number of parallel inference requests to utilize all the threads available on the device. On GPU, it automatically sets the inference batch size to fill up the GPU memory available.
|
||||
|
||||
For more information on performance hints, see the [High-level Performance Hints](../../../docs/OV_Runtime_UG/performance_hints.md) page. For more details on optimal runtime configurations and how they are automatically determined using performance hints, see [Runtime Inference Optimizations](../../../docs/optimization_guide/dldt_deployment_optimization_guide.md).
|
||||
For more information on performance hints, see the :doc:`High-level Performance Hints <openvino_docs_OV_UG_Performance_Hints>` page. For more details on optimal runtime configurations and how they are automatically determined using performance hints, see :doc:`Runtime Inference Optimizations <openvino_docs_deployment_optimization_guide_dldt_optimization_guide>`.
|
||||
|
||||
|
||||
### Device
|
||||
To set which device benchmarking runs on, use the `-d <device>` argument. This will tell benchmark_app to run benchmarking on that specific device. The benchmark app supports "CPU", "GPU", and "GNA" devices. In order to use the GPU or GNA, the system must have the appropriate drivers installed. If no device is specified, benchmark_app will default to using CPU.
|
||||
Device
|
||||
++++++++++++++++++++
|
||||
|
||||
To set which device benchmarking runs on, use the ``-d <device>`` argument. This will tell benchmark_app to run benchmarking on that specific device. The benchmark app supports "CPU", "GPU", and "GNA" devices. In order to use the GPU or GNA, the system must have the appropriate drivers installed. If no device is specified, benchmark_app will default to using CPU.
|
||||
|
||||
For example, to run benchmarking on GPU, use:
|
||||
|
||||
```
|
||||
./benchmark_app -m model.xml -d GPU
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
You may also specify "AUTO" as the device, in which case the benchmark_app will automatically select the best device for benchmarking and support it with the CPU at the model loading stage. This may result in increased performance, thus, should be used purposefully. For more information, see the [Automatic device selection](../../../docs/OV_Runtime_UG/auto_device_selection.md) page.
|
||||
./benchmark_app -m model.xml -d GPU
|
||||
|
||||
|
||||
You may also specify "AUTO" as the device, in which case the benchmark_app will automatically select the best device for benchmarking and support it with the CPU at the model loading stage. This may result in increased performance, thus, should be used purposefully. For more information, see the :doc:`Automatic device selection <openvino_docs_OV_UG_supported_plugins_AUTO>` page.
|
||||
|
||||
(Note: If the latency or throughput hint is set, it will automatically configure streams and batch sizes for optimal performance based on the specified device.)
|
||||
|
||||
### Number of iterations
|
||||
Number of iterations
|
||||
++++++++++++++++++++
|
||||
|
||||
By default, the benchmarking app will run for a predefined duration, repeatedly performing inferencing with the model and measuring the resulting inference speed. There are several options for setting the number of inference iterations:
|
||||
|
||||
* Explicitly specify the number of iterations the model runs using the `-niter <number_of_iterations>` option.
|
||||
* Set how much time the app runs for using the `-t <seconds>` option.
|
||||
* Explicitly specify the number of iterations the model runs using the ``-niter <number_of_iterations>`` option.
|
||||
* Set how much time the app runs for using the ``-t <seconds>`` option.
|
||||
* Set both of them (execution will continue until both conditions are met).
|
||||
* If neither -niter nor -t are specified, the app will run for a predefined duration that depends on the device.
|
||||
|
||||
The more iterations a model runs, the better the statistics will be for determing average latency and throughput.
|
||||
The more iterations a model runs, the better the statistics will be for determining average latency and throughput.
|
||||
|
||||
### Inputs
|
||||
The benchmark tool runs benchmarking on user-provided input images in `.jpg`, `.bmp`, or `.png` format. Use `-i <PATH_TO_INPUT>` to specify the path to an image, or folder of images. For example, to run benchmarking on an image named `test1.jpg`, use:
|
||||
Inputs
|
||||
++++++++++++++++++++
|
||||
|
||||
```
|
||||
./benchmark_app -m model.xml -i test1.jpg
|
||||
```
|
||||
The benchmark tool runs benchmarking on user-provided input images in ``.jpg``, ``.bmp``, or ``.png`` format. Use ``-i <PATH_TO_INPUT>`` to specify the path to an image, or folder of images. For example, to run benchmarking on an image named ``test1.jpg``, use:
|
||||
|
||||
The tool will repeatedly loop through the provided inputs and run inferencing on them for the specified amount of time or number of iterations. If the `-i` flag is not used, the tool will automatically generate random data to fit the input shape of the model.
|
||||
.. code-block:: sh
|
||||
|
||||
### Examples
|
||||
For more usage examples (and step-by-step instructions on how to set up a model for benchmarking), see the [Examples of Running the Tool](#examples-of-running-the-tool-cpp) section.
|
||||
./benchmark_app -m model.xml -i test1.jpg
|
||||
|
||||
## <a name="advanced-usage-cpp-benchmark"></a> Advanced Usage
|
||||
|
||||
> **NOTE**: By default, OpenVINO samples, tools and demos expect input with BGR channels order. If you trained your model to work with RGB order, you need to manually rearrange the default channel order in the sample or demo application or reconvert your model using the Model Optimizer tool with --reverse_input_channels argument specified. For more information about the argument, refer to When to Reverse Input Channels section of Converting a Model to Intermediate Representation (IR).
|
||||
The tool will repeatedly loop through the provided inputs and run inferencing on them for the specified amount of time or number of iterations. If the ``-i`` flag is not used, the tool will automatically generate random data to fit the input shape of the model.
|
||||
|
||||
### Per-layer performance and logging
|
||||
The application also collects per-layer Performance Measurement (PM) counters for each executed infer request if you enable statistics dumping by setting the `-report_type` parameter to one of the possible values:
|
||||
Examples
|
||||
++++++++++++++++++++
|
||||
|
||||
* `no_counters` report includes configuration options specified, resulting FPS and latency.
|
||||
* `average_counters` report extends the `no_counters` report and additionally includes average PM counters values for each layer from the network.
|
||||
* `detailed_counters` report extends the `average_counters` report and additionally includes per-layer PM counters and latency for each executed infer request.
|
||||
For more usage examples (and step-by-step instructions on how to set up a model for benchmarking), see the :ref:`Examples of Running the Tool <examples-of-running-the-tool-cpp>` section.
|
||||
|
||||
.. _advanced-usage-cpp-benchmark:
|
||||
|
||||
Advanced Usage
|
||||
####################
|
||||
|
||||
.. note::
|
||||
|
||||
By default, OpenVINO samples, tools and demos expect input with BGR channels order. If you trained your model to work with RGB order, you need to manually rearrange the default channel order in the sample or demo application or reconvert your model using the Model Optimizer tool with --reverse_input_channels argument specified. For more information about the argument, refer to When to Reverse Input Channels section of Converting a Model to Intermediate Representation (IR).
|
||||
|
||||
Per-layer performance and logging
|
||||
+++++++++++++++++++++++++++++++++
|
||||
|
||||
The application also collects per-layer Performance Measurement (PM) counters for each executed infer request if you enable statistics dumping by setting the ``-report_type`` parameter to one of the possible values:
|
||||
|
||||
* ``no_counters`` report includes configuration options specified, resulting FPS and latency.
|
||||
* ``average_counters`` report extends the ``no_counters`` report and additionally includes average PM counters values for each layer from the network.
|
||||
* ``detailed_counters`` report extends the ``average_counters`` report and additionally includes per-layer PM counters and latency for each executed infer request.
|
||||
|
||||
Depending on the type, the report is stored to benchmark_no_counters_report.csv, benchmark_average_counters_report.csv, or benchmark_detailed_counters_report.csv file located in the path specified in -report_folder. The application also saves executable graph information serialized to an XML file if you specify a path to it with the -exec_graph_path parameter.
|
||||
|
||||
### <a name="all-configuration-options-cpp-benchmark"></a> All configuration options
|
||||
.. _all-configuration-options-cpp-benchmark:
|
||||
|
||||
Running the application with the `-h` or `--help` option yields the following usage message:
|
||||
All configuration options
|
||||
+++++++++++++++++++++++++
|
||||
|
||||
```
|
||||
[Step 1/11] Parsing and validating input arguments
|
||||
[ INFO ] Parsing input parameters
|
||||
usage: benchmark_app [OPTION]
|
||||
Running the application with the ``-h`` or ``--help`` option yields the following usage message:
|
||||
|
||||
Options:
|
||||
-h, --help Print the usage message
|
||||
-m <path> Required. Path to an .xml/.onnx file with a trained model or to a .blob files with a trained compiled model.
|
||||
-i <path> Optional. Path to a folder with images and/or binaries or to specific image or binary file.
|
||||
In case of dynamic shapes models with several inputs provide the same number of files for each input (except cases with single file for any input):"input1:1.jpg input2:1.bin", "input1:1.bin,2.bin input2:3.bin input3:4.bin,5.bin ". Also you can pass specific keys for inputs: "random" - for fillling input with random data, "image_info" - for filling input with image size.
|
||||
You should specify either one files set to be used for all inputs (without providing input names) or separate files sets for every input of model (providing inputs names).
|
||||
Currently supported data types: bmp, bin, npy.
|
||||
If OPENCV is enabled, this functionality is extended with the following data types:
|
||||
dib, jpeg, jpg, jpe, jp2, png, pbm, pgm, ppm, sr, ras, tiff, tif.
|
||||
-d <device> Optional. Specify a target device to infer on (the list of available devices is shown below). Default value is CPU. Use "-d HETERO:<comma-separated_devices_list>" format to specify HETERO plugin. Use "-d MULTI:<comma-separated_devices_list>" format to specify MULTI plugin. The application looks for a suitable plugin for the specified device.
|
||||
-hint <performance hint> (latency or throughput or cumulative_throughput or none) Optional. Performance hint allows the OpenVINO device to select the right model-specific settings.
|
||||
'throughput' or 'tput': device performance mode will be set to THROUGHPUT.
|
||||
'cumulative_throughput' or 'ctput': device performance mode will be set to CUMULATIVE_THROUGHPUT.
|
||||
'latency': device performance mode will be set to LATENCY.
|
||||
'none': device performance mode will be set to UNDEFINED.
|
||||
Using explicit 'nstreams' or other device-specific options, please set hint to 'none'
|
||||
-niter <integer> Optional. Number of iterations. If not specified, the number of iterations is calculated depending on a device.
|
||||
-t Optional. Time in seconds to execute topology.
|
||||
.. scrollbox::
|
||||
|
||||
Input shapes
|
||||
-b <integer> Optional. Batch size value. If not specified, the batch size value is determined from Intermediate Representation.
|
||||
-shape Optional. Set shape for model input. For example, "input1[1,3,224,224],input2[1,4]" or "[1,3,224,224]" in case of one input size. This parameter affect model input shape and can be dynamic. For dynamic dimensions use symbol `?` or '-1'. Ex. [?,3,?,?]. For bounded dimensions specify range 'min..max'. Ex. [1..10,3,?,?].
|
||||
-data_shape Required for models with dynamic shapes. Set shape for input blobs. In case of one input size: "[1,3,224,224]" or "input1[1,3,224,224],input2[1,4]". In case of several input sizes provide the same number for each input (except cases with single shape for any input): "[1,3,128,128][3,3,128,128][1,3,320,320]", "input1[1,1,128,128][1,1,256,256],input2[80,1]" or "input1[1,192][1,384],input2[1,192][1,384],input3[1,192][1,384],input4[1,192][1,384]". If model shapes are all static specifying the option will cause an exception.
|
||||
-layout Optional. Prompts how model layouts should be treated by application. For example, "input1[NCHW],input2[NC]" or "[NCHW]" in case of one input size.
|
||||
.. code-block::
|
||||
|
||||
Advanced options
|
||||
-extensions <absolute_path> Required for custom layers (extensions). Absolute path to a shared library with the kernels implementations.
|
||||
-c <absolute_path> Required for GPU custom kernels. Absolute path to an .xml file with the kernels description.
|
||||
-cache_dir <path> Optional. Enables caching of loaded models to specified directory. List of devices which support caching is shown at the end of this message.
|
||||
-load_from_file Optional. Loads model from file directly without read_model. All CNNNetwork options (like re-shape) will be ignored
|
||||
-api <sync/async> Optional (deprecated). Enable Sync/Async API. Default value is "async".
|
||||
-nireq <integer> Optional. Number of infer requests. Default value is determined automatically for device.
|
||||
-nstreams <integer> Optional. Number of streams to use for inference on the CPU or GPU devices (for HETERO and MULTI device cases use format <dev1>:<nstreams1>,<dev2>:<nstreams2> or just <nstreams>). Default value is determined automatically for a device.Please note that although the automatic selection usually provides a reasonable performance, it still may be non - optimal for some cases, especially for very small models. See sample's README for more details. Also, using nstreams>1 is inherently throughput-oriented option, while for the best-latency estimations the number of streams should be set to 1.
|
||||
-inference_only Optional. Measure only inference stage. Default option for static models. Dynamic models are measured in full mode which includes inputs setup stage, inference only mode available for them with single input data shape only. To enable full mode for static models pass "false" value to this argument: ex. "-inference_only=false".
|
||||
-infer_precision Optional. Specifies the inference precision. Example #1: '-infer_precision bf16'. Example #2: '-infer_precision CPU:bf16,GPU:f32'
|
||||
[Step 1/11] Parsing and validating input arguments
|
||||
[ INFO ] Parsing input parameters
|
||||
usage: benchmark_app [OPTION]
|
||||
|
||||
Preprocessing options:
|
||||
-ip <value> Optional. Specifies precision for all input layers of the model.
|
||||
-op <value> Optional. Specifies precision for all output layers of the model.
|
||||
-iop <value> Optional. Specifies precision for input and output layers by name.
|
||||
Example: -iop "input:FP16, output:FP16".
|
||||
Notice that quotes are required.
|
||||
Overwrites precision from ip and op options for specified layers.
|
||||
-mean_values [R,G,B] Optional. Mean values to be used for the input image per channel. Values to be provided in the [R,G,B] format. Can be defined for desired input of the model, for example: "--mean_values data[255,255,255],info[255,255,255]". The exact meaning and order of channels depend on how the original model was trained. Applying the values affects performance and may cause type conversion
|
||||
-scale_values [R,G,B] Optional. Scale values to be used for the input image per channel. Values are provided in the [R,G,B] format. Can be defined for desired input of the model, for example: "--scale_values data[255,255,255],info[255,255,255]". The exact meaning and order of channels depend on how the original model was trained. If both --mean_values and --scale_values are specified, the mean is subtracted first and then scale is applied regardless of the order of options in command line. Applying the values affects performance and may cause type conversion
|
||||
Options:
|
||||
-h, --help Print the usage message
|
||||
-m <path> Required. Path to an .xml/.onnx file with a trained model or to a .blob files with a trained compiled model.
|
||||
-i <path> Optional. Path to a folder with images and/or binaries or to specific image or binary file.
|
||||
In case of dynamic shapes models with several inputs provide the same number of files for each input (except cases with single file for any input) :"input1:1.jpg input2:1.bin", "input1:1.bin,2.bin input2:3.bin input3:4.bin,5.bin ". Also you can pass specific keys for inputs: "random" - for fillling input with random data, "image_info" - for filling input with image size.
|
||||
You should specify either one files set to be used for all inputs (without providing input names) or separate files sets for every input of model (providing inputs names).
|
||||
Currently supported data types: bmp, bin, npy.
|
||||
If OPENCV is enabled, this functionality is extended with the following data types:
|
||||
dib, jpeg, jpg, jpe, jp2, png, pbm, pgm, ppm, sr, ras, tiff, tif.
|
||||
-d <device> Optional. Specify a target device to infer on (the list of available devices is shown below). Default value is CPU. Use "-d HETERO:<comma-separated_devices_list>" format to specify HETERO plugin. Use "-d MULTI:<comma-separated_devices_list>" format to specify MULTI plugin. The application looks for a suitable plugin for the specified device.
|
||||
-hint <performance hint> (latency or throughput or cumulative_throughput or none) Optional. Performance hint allows the OpenVINO device to select the right model-specific settings.
|
||||
'throughput' or 'tput': device performance mode will be set to THROUGHPUT.
|
||||
'cumulative_throughput' or 'ctput': device performance mode will be set to CUMULATIVE_THROUGHPUT.
|
||||
'latency': device performance mode will be set to LATENCY.
|
||||
'none': device performance mode will be set to UNDEFINED.
|
||||
Using explicit 'nstreams' or other device-specific options, please set hint to 'none'
|
||||
-niter <integer> Optional. Number of iterations. If not specified, the number of iterations is calculated depending on a device.
|
||||
-t Optional. Time in seconds to execute topology.
|
||||
|
||||
Device-specific performance options:
|
||||
-nthreads <integer> Optional. Number of threads to use for inference on the CPU (including HETERO and MULTI cases).
|
||||
-pin <string> ("YES"|"CORE") / "HYBRID_AWARE" / ("NO"|"NONE") / "NUMA" Optional. Explicit inference threads binding options (leave empty to let the OpenVINO make a choice):
|
||||
enabling threads->cores pinning("YES", which is already default for any conventional CPU),
|
||||
letting the runtime to decide on the threads->different core types("HYBRID_AWARE", which is default on the hybrid CPUs)
|
||||
threads->(NUMA)nodes("NUMA") or
|
||||
completely disable("NO") CPU inference threads pinning
|
||||
Input shapes
|
||||
-b <integer> Optional. Batch size value. If not specified, the batch size value is determined from Intermediate Representation.
|
||||
-shape Optional. Set shape for model input. For example, "input1[1,3,224,224],input2[1,4]" or "[1,3,224,224]" in case of one input size. This parameter affect model input shape and can be dynamic. For dynamic dimensions use symbol `?` or '-1'. Ex. [?,3,?,?]. For bounded dimensions specify range 'min..max'. Ex. [1..10,3,?,?].
|
||||
-data_shape Required for models with dynamic shapes. Set shape for input blobs. In case of one input size: "[1,3,224,224]" or "input1[1,3,224,224],input2[1,4] ". In case of several input sizes provide the same number for each input (except cases with single shape for any input): "[1,3,128,128][3,3,128,128][1,3,320,320]", "input1[1,1, 128,128][1,1,256,256],input2[80,1]" or "input1[1,192][1,384],input2[1,192][1,384],input3[1,192][1,384],input4[1,192][1,384]". If model shapes are all static specifying the option will cause an exception.
|
||||
-layout Optional. Prompts how model layouts should be treated by application. For example, "input1[NCHW],input2[NC]" or "[NCHW]" in case of one input size.
|
||||
|
||||
Advanced options
|
||||
-extensions <absolute_path> Required for custom layers (extensions). Absolute path to a shared library with the kernels implementations.
|
||||
-c <absolute_path> Required for GPU custom kernels. Absolute path to an .xml file with the kernels description.
|
||||
-cache_dir <path> Optional. Enables caching of loaded models to specified directory. List of devices which support caching is shown at the end of this message.
|
||||
-load_from_file Optional. Loads model from file directly without read_model. All CNNNetwork options (like re-shape) will be ignored
|
||||
-api <sync/async> Optional (deprecated). Enable Sync/Async API. Default value is "async".
|
||||
-nireq <integer> Optional. Number of infer requests. Default value is determined automatically for device.
|
||||
-nstreams <integer> Optional. Number of streams to use for inference on the CPU or GPU devices (for HETERO and MULTI device cases use format <dev1>:<nstreams1>, <dev2>:<nstreams2> or just <nstreams>). Default value is determined automatically for a device.Please note that although the automatic selection usually provides a reasonable performance, it still may be non - optimal for some cases, especially for very small models. See sample's README for more details. Also, using nstreams>1 is inherently throughput-oriented option, while for the best-latency estimations the number of streams should be set to 1.
|
||||
-inference_only Optional. Measure only inference stage. Default option for static models. Dynamic models are measured in full mode which includes inputs setup stage, inference only mode available for them with single input data shape only. To enable full mode for static models pass "false" value to this argument: ex. "-inference_only=false".
|
||||
-infer_precision Optional. Specifies the inference precision. Example #1: '-infer_precision bf16'. Example #2: '-infer_precision CPU:bf16,GPU:f32'
|
||||
|
||||
Preprocessing options:
|
||||
-ip <value> Optional. Specifies precision for all input layers of the model.
|
||||
-op <value> Optional. Specifies precision for all output layers of the model.
|
||||
-iop <value> Optional. Specifies precision for input and output layers by name.
|
||||
Example: -iop "input:FP16, output:FP16".
|
||||
Notice that quotes are required.
|
||||
Overwrites precision from ip and op options for specified layers.
|
||||
-mean_values [R,G,B] Optional. Mean values to be used for the input image per channel. Values to be provided in the [R,G,B] format. Can be defined for desired input of the model, for example: "--mean_values data[255,255,255],info[255,255,255]". The exact meaning and order of channels depend on how the original model was trained. Applying the values affects performance and may cause type conversion
|
||||
-scale_values [R,G,B] Optional. Scale values to be used for the input image per channel. Values are provided in the [R,G,B] format. Can be defined for desired input of the model, for example: "--scale_values data[255,255,255],info[255,255,255]". The exact meaning and order of channels depend on how the original model was trained. If both --mean_values and --scale_values are specified, the mean is subtracted first and then scale is applied regardless of the order of options in command line. Applying the values affects performance and may cause type conversion
|
||||
|
||||
Device-specific performance options:
|
||||
-nthreads <integer> Optional. Number of threads to use for inference on the CPU (including HETERO and MULTI cases).
|
||||
-pin <string> ("YES"|"CORE") / "HYBRID_AWARE" / ("NO"|"NONE") / "NUMA" Optional. Explicit inference threads binding options (leave empty to let the OpenVINO make a choice):
|
||||
enabling threads->cores pinning("YES", which is already default for any conventional CPU),
|
||||
letting the runtime to decide on the threads->different core types("HYBRID_AWARE", which is default on the hybrid CPUs)
|
||||
threads->(NUMA)nodes("NUMA") or
|
||||
completely disable("NO") CPU inference threads pinning
|
||||
|
||||
Statistics dumping options:
|
||||
-latency_percentile Optional. Defines the percentile to be reported in latency metric. The valid range is [1, 100]. The default value is 50 (median).
|
||||
-report_type <type> Optional. Enable collecting statistics report. "no_counters" report contains configuration options specified, resulting FPS and latency. "average_counters" report extends "no_counters" report and additionally includes average PM counters values for each layer from the model. "detailed_counters" report extends "average_counters" report and additionally includes per-layer PM counters and latency for each executed infer request.
|
||||
-report_folder Optional. Path to a folder where statistics report is stored.
|
||||
-json_stats Optional. Enables JSON-based statistics output (by default reporting system will use CSV format). Should be used together with -report_folder option.
|
||||
-pc Optional. Report performance counters.
|
||||
-pcsort Optional. Report performance counters and analysis the sort hotpoint opts. "sort" Analysis opts time cost, print by hotpoint order "no_sort" Analysis opts time cost, print by normal order "simple_sort" Analysis opts time cost, only print EXECUTED opts by normal order
|
||||
-pcseq Optional. Report latencies for each shape in -data_shape sequence.
|
||||
-exec_graph_path Optional. Path to a file where to store executable graph information serialized.
|
||||
-dump_config Optional. Path to JSON file to dump IE parameters, which were set by application.
|
||||
-load_config Optional. Path to JSON file to load custom IE parameters. Please note, command line parameters have higher priority then parameters from configuration file.
|
||||
Example 1: a simple JSON file for HW device with primary properties.
|
||||
{
|
||||
"CPU": {"NUM_STREAMS": "3", "PERF_COUNT": "NO"}
|
||||
}
|
||||
Example 2: a simple JSON file for meta device(AUTO/MULTI) with HW device properties.
|
||||
{
|
||||
"AUTO": {
|
||||
"PERFORMANCE_HINT": "",
|
||||
"PERF_COUNT": "NO",
|
||||
"DEVICE_PROPERTIES": {
|
||||
"CPU": {
|
||||
"INFERENCE_PRECISION_HINT": "f32",
|
||||
"NUM_STREAMS": "3"
|
||||
},
|
||||
"GPU": {
|
||||
"INFERENCE_PRECISION_HINT": "f32",
|
||||
"NUM_STREAMS": "5"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Statistics dumping options:
|
||||
-latency_percentile Optional. Defines the percentile to be reported in latency metric. The valid range is [1, 100]. The default value is 50 (median).
|
||||
-report_type <type> Optional. Enable collecting statistics report. "no_counters" report contains configuration options specified, resulting FPS and latency. "average_counters" report extends "no_counters" report and additionally includes average PM counters values for each layer from the model. "detailed_counters" report extends "average_counters" report and additionally includes per-layer PM counters and latency for each executed infer request.
|
||||
-report_folder Optional. Path to a folder where statistics report is stored.
|
||||
-json_stats Optional. Enables JSON-based statistics output (by default reporting system will use CSV format). Should be used together with -report_folder option.
|
||||
-pc Optional. Report performance counters.
|
||||
-pcsort Optional. Report performance counters and analysis the sort hotpoint opts. "sort" Analysis opts time cost, print by hotpoint order "no_sort" Analysis opts time cost, print by normal order "simple_sort" Analysis opts time cost, only print EXECUTED opts by normal order
|
||||
-pcseq Optional. Report latencies for each shape in -data_shape sequence.
|
||||
-exec_graph_path Optional. Path to a file where to store executable graph information serialized.
|
||||
-dump_config Optional. Path to JSON file to dump IE parameters, which were set by application.
|
||||
-load_config Optional. Path to JSON file to load custom IE parameters. Please note, command line parameters have higher priority then parameters from configuration file.
|
||||
Example 1: a simple JSON file for HW device with primary properties.
|
||||
{
|
||||
"CPU": {"NUM_STREAMS": "3", "PERF_COUNT": "NO"}
|
||||
}
|
||||
Example 2: a simple JSON file for meta device(AUTO/MULTI) with HW device properties.
|
||||
{
|
||||
"AUTO": {
|
||||
"PERFORMANCE_HINT": "",
|
||||
"PERF_COUNT": "NO",
|
||||
"DEVICE_PROPERTIES": {
|
||||
"CPU": {
|
||||
"INFERENCE_PRECISION_HINT": "f32",
|
||||
"NUM_STREAMS": "3"
|
||||
},
|
||||
"GPU": {
|
||||
"INFERENCE_PRECISION_HINT": "f32",
|
||||
"NUM_STREAMS": "5"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Running the application with the empty list of options yields the usage message given above and an error message.
|
||||
|
||||
### More information on inputs
|
||||
More information on inputs
|
||||
++++++++++++++++++++++++++
|
||||
|
||||
The benchmark tool supports topologies with one or more inputs. If a topology is not data sensitive, you can skip the input parameter, and the inputs will be filled with random values. If a model has only image input(s), provide a folder with images or a path to an image as input. If a model has some specific input(s) (besides images), please prepare a binary file(s) or numpy array(s) that is filled with data of appropriate precision and provide a path to it as input. If a model has mixed input types, the input folder should contain all required files. Image inputs are filled with image files one by one. Binary inputs are filled with binary inputs one by one.
|
||||
|
||||
## <a name="examples-of-running-the-tool-cpp"></a> Examples of Running the Tool
|
||||
This section provides step-by-step instructions on how to run the Benchmark Tool with the `asl-recognition` model from the [Open Model Zoo](@ref model_zoo) on CPU or GPU devices. It uses random data as the input.
|
||||
.. _examples-of-running-the-tool-cpp:
|
||||
|
||||
Examples of Running the Tool
|
||||
############################
|
||||
|
||||
This section provides step-by-step instructions on how to run the Benchmark Tool with the ``asl-recognition`` model from the :doc:`Open Model Zoo <model_zoo>` on CPU or GPU devices. It uses random data as the input.
|
||||
|
||||
.. note::
|
||||
|
||||
Internet access is required to execute the following steps successfully. If you have access to the Internet through a proxy server only, please make sure that it is configured in your OS environment.
|
||||
|
||||
> **NOTE**: Internet access is required to execute the following steps successfully. If you have access to the Internet through a proxy server only, please make sure that it is configured in your OS environment.
|
||||
|
||||
1. Install OpenVINO Development Tools (if it hasn't been installed already):
|
||||
```sh
|
||||
pip install openvino-dev
|
||||
```
|
||||
|
||||
2. Download the model using `omz_downloader`, specifying the model name and directory to download the model to:
|
||||
```sh
|
||||
omz_downloader --name asl-recognition-0004 --precisions FP16 --output_dir omz_models
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
pip install openvino-dev
|
||||
|
||||
|
||||
2. Download the model using ``omz_downloader``, specifying the model name and directory to download the model to:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
omz_downloader --name asl-recognition-0004 --precisions FP16 --output_dir omz_models
|
||||
|
||||
|
||||
3. Run the tool, specifying the location of the model .xml file, the device to perform inference on, and with a performance hint. The following commands demonstrate examples of how to run the Benchmark Tool in latency mode on CPU and throughput mode on GPU devices:
|
||||
|
||||
* On CPU (latency mode):
|
||||
```sh
|
||||
./benchmark_app -m omz_models/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d CPU -hint latency
|
||||
```
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
./benchmark_app -m omz_models/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d CPU -hint latency
|
||||
|
||||
|
||||
* On GPU (throughput mode):
|
||||
```sh
|
||||
./benchmark_app -m omz_models/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d GPU -hint throughput
|
||||
```
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
./benchmark_app -m omz_models/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d GPU -hint throughput
|
||||
|
||||
|
||||
The application outputs the number of executed iterations, total duration of execution, latency, and throughput.
|
||||
Additionally, if you set the `-report_type` parameter, the application outputs a statistics report. If you set the `-pc` parameter, the application outputs performance counters. If you set `-exec_graph_path`, the application reports executable graph information serialized. All measurements including per-layer PM counters are reported in milliseconds.
|
||||
Additionally, if you set the ``-report_type`` parameter, the application outputs a statistics report. If you set the ``-pc`` parameter, the application outputs performance counters. If you set ``-exec_graph_path``, the application reports executable graph information serialized. All measurements including per-layer PM counters are reported in milliseconds.
|
||||
|
||||
An example of the information output when running benchmark_app on CPU in latency mode is shown below:
|
||||
|
||||
```sh
|
||||
./benchmark_app -m omz_models/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d CPU -hint latency
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
./benchmark_app -m omz_models/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d CPU -hint latency
|
||||
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
```sh
|
||||
[Step 1/11] Parsing and validating input arguments
|
||||
[ INFO ] Parsing input parameters
|
||||
[ INFO ] Input command: /home/openvino/bin/intel64/DEBUG/benchmark_app -m omz_models/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d CPU -hint latency
|
||||
@@ -300,14 +360,18 @@ An example of the information output when running benchmark_app on CPU in latenc
|
||||
[ INFO ] Min: 20.60 ms
|
||||
[ INFO ] Max: 37.19 ms
|
||||
[ INFO ] Throughput: 91.12 FPS
|
||||
```
|
||||
The Benchmark Tool can also be used with dynamically shaped networks to measure expected inference time for various input data shapes. See the `-shape` and `-data_shape` argument descriptions in the <a href="#all-configuration-options-cpp-benchmark">All configuration options</a> section to learn more about using dynamic shapes. Here is a command example for using benchmark_app with dynamic networks and a portion of the resulting output:
|
||||
|
||||
```sh
|
||||
|
||||
|
||||
The Benchmark Tool can also be used with dynamically shaped networks to measure expected inference time for various input data shapes. See the ``-shape`` and ``-data_shape`` argument descriptions in the :ref:`All configuration options <all-configuration-options-cpp-benchmark>` section to learn more about using dynamic shapes. Here is a command example for using benchmark_app with dynamic networks and a portion of the resulting output:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
./benchmark_app -m omz_models/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d CPU -shape [-1,3,16,224,224] -data_shape [1,3,16,224,224][2,3,16,224,224][4,3,16,224,224] -pcseq
|
||||
```
|
||||
|
||||
```sh
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
[Step 9/11] Creating infer requests and preparing input tensors
|
||||
[ INFO ] Test Config 0
|
||||
[ INFO ] input ([N,C,D,H,W], f32, {1, 3, 16, 224, 224}, dyn:{?,3,16,224,224}): random (binary data is expected)
|
||||
@@ -343,9 +407,13 @@ The Benchmark Tool can also be used with dynamically shaped networks to measure
|
||||
[ INFO ] Min: 327.50 ms
|
||||
[ INFO ] Max: 743.46 ms
|
||||
[ INFO ] Throughput: 107.61 FPS
|
||||
```
|
||||
|
||||
## See Also
|
||||
* [Using OpenVINO Runtime Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md)
|
||||
* [Model Optimizer](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md)
|
||||
* [Model Downloader](@ref omz_tools_downloader)
|
||||
|
||||
See Also
|
||||
####################
|
||||
|
||||
* :doc:`Using OpenVINO Samples <openvino_docs_OV_UG_Samples_Overview>`
|
||||
* :doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`
|
||||
* :doc:`Model Downloader <omz_tools_downloader>`
|
||||
|
||||
@endsphinxdirective
|
||||
@@ -1,50 +1,69 @@
|
||||
# Bert Benchmark Python* Sample {#openvino_inference_engine_ie_bridges_python_sample_bert_benchmark_README}
|
||||
|
||||
This sample demonstrates how to estimate performace of a Bert model using Asynchronous Inference Request API. Unlike [demos](@ref omz_demos) this sample doesn't have configurable command line arguments. Feel free to modify sample's source code to try out different options.
|
||||
@sphinxdirective
|
||||
|
||||
The following Python\* API is used in the application:
|
||||
This sample demonstrates how to estimate performance of a Bert model using Asynchronous Inference Request API. Unlike :doc:`demos <omz_demos>` this sample doesn't have configurable command line arguments. Feel free to modify sample's source code to try out different options.
|
||||
|
||||
| Feature | API | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| OpenVINO Runtime Version | [openvino.runtime.get_version] | Get Openvino API version |
|
||||
| Basic Infer Flow | [openvino.runtime.Core], [openvino.runtime.Core.compile_model] | Common API to do inference: compile a model |
|
||||
| Asynchronous Infer | [openvino.runtime.AsyncInferQueue], [openvino.runtime.AsyncInferQueue.start_async], [openvino.runtime.AsyncInferQueue.wait_all] | Do asynchronous inference |
|
||||
| Model Operations | [openvino.runtime.CompiledModel.inputs] | Get inputs of a model |
|
||||
The following Python API is used in the application:
|
||||
|
||||
## How It Works
|
||||
+--------------------------------+-------------------------------------------------+----------------------------------------------+
|
||||
| Feature | API | Description |
|
||||
+================================+=================================================+==============================================+
|
||||
| OpenVINO Runtime Version | [openvino.runtime.get_version] | Get Openvino API version. |
|
||||
+--------------------------------+-------------------------------------------------+----------------------------------------------+
|
||||
| Basic Infer Flow | [openvino.runtime.Core], | Common API to do inference: compile a model. |
|
||||
| | [openvino.runtime.Core.compile_model] | |
|
||||
+--------------------------------+-------------------------------------------------+----------------------------------------------+
|
||||
| Asynchronous Infer | [openvino.runtime.AsyncInferQueue], | Do asynchronous inference. |
|
||||
| | [openvino.runtime.AsyncInferQueue.start_async], | |
|
||||
| | [openvino.runtime.AsyncInferQueue.wait_all] | |
|
||||
+--------------------------------+-------------------------------------------------+----------------------------------------------+
|
||||
| Model Operations | [openvino.runtime.CompiledModel.inputs] | Get inputs of a model. |
|
||||
+--------------------------------+-------------------------------------------------+----------------------------------------------+
|
||||
|
||||
The sample downloads a model and a tokenizer, export the model to onnx, reads the exported model and reshapes it to enforce dynamic inpus shapes, compiles the resulting model, downloads a dataset and runs benchmarking on the dataset.
|
||||
How It Works
|
||||
####################
|
||||
|
||||
The sample downloads a model and a tokenizer, export the model to onnx, reads the exported model and reshapes it to enforce dynamic input shapes, compiles the resulting model, downloads a dataset and runs benchmarking on the dataset.
|
||||
|
||||
You can see the explicit description of
|
||||
each sample step at [Integration Steps](../../../../docs/OV_Runtime_UG/integrate_with_your_application.md) section of "Integrate OpenVINO™ Runtime with Your Application" guide.
|
||||
each sample step at :doc:`Integration Steps <openvino_docs_OV_UG_Integrate_OV_with_your_application>` section of "Integrate OpenVINO™ Runtime with Your Application" guide.
|
||||
|
||||
## Running
|
||||
Running
|
||||
####################
|
||||
|
||||
Install the `openvino` Python package:
|
||||
Install the ``openvino`` Python package:
|
||||
|
||||
```
|
||||
python -m pip install openvino
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
Install packages from `requirements.txt`:
|
||||
python -m pip install openvino
|
||||
|
||||
|
||||
Install packages from ``requirements.txt``:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
python -m pip install -r requirements.txt
|
||||
|
||||
```
|
||||
python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Run the sample
|
||||
|
||||
```
|
||||
python bert_benchmark.py
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
## Sample Output
|
||||
python bert_benchmark.py
|
||||
|
||||
|
||||
Sample Output
|
||||
####################
|
||||
|
||||
The sample outputs how long it takes to process a dataset.
|
||||
|
||||
## See Also
|
||||
See Also
|
||||
####################
|
||||
|
||||
- [Integrate the OpenVINO™ Runtime with Your Application](../../../../docs/OV_Runtime_UG/integrate_with_your_application.md)
|
||||
- [Using OpenVINO™ Toolkit Samples](../../../../docs/OV_Runtime_UG/Samples_Overview.md)
|
||||
- [Model Downloader](@ref omz_tools_downloader)
|
||||
- [Model Optimizer](../../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md)
|
||||
* :doc:`Integrate the OpenVINO™ Runtime with Your Application <openvino_docs_OV_UG_Integrate_OV_with_your_application>`
|
||||
* :doc:`Using OpenVINO Samples <openvino_docs_OV_UG_Samples_Overview>`
|
||||
* :doc:`Model Downloader <omz_tools_downloader>`
|
||||
* :doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`
|
||||
|
||||
@endsphinxdirective
|
||||
@@ -1,94 +1,134 @@
|
||||
# Throughput Benchmark Python* Sample {#openvino_inference_engine_ie_bridges_python_sample_throughput_benchmark_README}
|
||||
|
||||
This sample demonstrates how to estimate performace of a model using Asynchronous Inference Request API in throughput mode. Unlike [demos](@ref omz_demos) this sample doesn't have other configurable command line arguments. Feel free to modify sample's source code to try out different options.
|
||||
@sphinxdirective
|
||||
|
||||
The reported results may deviate from what [benchmark_app](../../../../tools/benchmark_tool/README.md) reports. One example is model input precision for computer vision tasks. benchmark_app sets uint8, while the sample uses default model precision which is usually float32.
|
||||
|
||||
The following Python\* API is used in the application:
|
||||
This sample demonstrates how to estimate performance of a model using Asynchronous Inference Request API in throughput mode. Unlike :doc:`demos <omz_demos>` this sample doesn't have other configurable command line arguments. Feel free to modify sample's source code to try out different options.
|
||||
|
||||
| Feature | API | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| OpenVINO Runtime Version | [openvino.runtime.get_version] | Get Openvino API version |
|
||||
| Basic Infer Flow | [openvino.runtime.Core], [openvino.runtime.Core.compile_model], [openvino.runtime.InferRequest.get_tensor] | Common API to do inference: compile a model, configure input tensors |
|
||||
| Asynchronous Infer | [openvino.runtime.AsyncInferQueue], [openvino.runtime.AsyncInferQueue.start_async], [openvino.runtime.AsyncInferQueue.wait_all], [openvino.runtime.InferRequest.results] | Do asynchronous inference |
|
||||
| Model Operations | [openvino.runtime.CompiledModel.inputs] | Get inputs of a model |
|
||||
| Tensor Operations | [openvino.runtime.Tensor.get_shape], [openvino.runtime.Tensor.data] | Get a tensor shape and its data. |
|
||||
The reported results may deviate from what :doc:`benchmark_app <openvino_inference_engine_tools_benchmark_tool_README>` reports. One example is model input precision for computer vision tasks. benchmark_app sets uint8, while the sample uses default model precision which is usually float32.
|
||||
|
||||
| Options | Values |
|
||||
| :--- | :--- |
|
||||
| Validated Models | [alexnet](@ref omz_models_model_alexnet), [googlenet-v1](@ref omz_models_model_googlenet_v1) [yolo-v3-tf](@ref omz_models_model_yolo_v3_tf), [face-detection-0200](@ref omz_models_model_face_detection_0200) |
|
||||
| Model Format | OpenVINO™ toolkit Intermediate Representation (\*.xml + \*.bin), ONNX (\*.onnx) |
|
||||
| Supported devices | [All](../../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) |
|
||||
| Other language realization | [C++](../../../cpp/benchmark/throughput_benchmark/README.md) |
|
||||
The following Python API is used in the application:
|
||||
|
||||
## How It Works
|
||||
+--------------------------------+-------------------------------------------------+----------------------------------------------+
|
||||
| Feature | API | Description |
|
||||
+================================+=================================================+==============================================+
|
||||
| OpenVINO Runtime Version | [openvino.runtime.get_version] | Get Openvino API version. |
|
||||
+--------------------------------+-------------------------------------------------+----------------------------------------------+
|
||||
| Basic Infer Flow | [openvino.runtime.Core], | Common API to do inference: compile a model, |
|
||||
| | [openvino.runtime.Core.compile_model] | configure input tensors. |
|
||||
| | [openvino.runtime.InferRequest.get_tensor] | |
|
||||
+--------------------------------+-------------------------------------------------+----------------------------------------------+
|
||||
| Asynchronous Infer | [openvino.runtime.AsyncInferQueue], | Do asynchronous inference. |
|
||||
| | [openvino.runtime.AsyncInferQueue.start_async], | |
|
||||
| | [openvino.runtime.AsyncInferQueue.wait_all], | |
|
||||
| | [openvino.runtime.InferRequest.results] | |
|
||||
+--------------------------------+-------------------------------------------------+----------------------------------------------+
|
||||
| Model Operations | [openvino.runtime.CompiledModel.inputs] | Get inputs of a model. |
|
||||
+--------------------------------+-------------------------------------------------+----------------------------------------------+
|
||||
| Tensor Operations | [openvino.runtime.Tensor.get_shape], | Get a tensor shape and its data. |
|
||||
| | [openvino.runtime.Tensor.data] | |
|
||||
+--------------------------------+-------------------------------------------------+----------------------------------------------+
|
||||
|
||||
+--------------------------------+------------------------------------------------------------------------------+
|
||||
| Options | Values |
|
||||
+================================+==============================================================================+
|
||||
| Validated Models | :doc:`alexnet <omz_models_model_alexnet>`, |
|
||||
| | :doc:`googlenet-v1 <omz_models_model_googlenet_v1>`, |
|
||||
| | :doc:`yolo-v3-tf <omz_models_model_yolo_v3_tf>`, |
|
||||
| | :doc:`face-detection-0200 <omz_models_model_face_detection_0200>` |
|
||||
+--------------------------------+------------------------------------------------------------------------------+
|
||||
| Model Format | OpenVINO™ toolkit Intermediate Representation |
|
||||
| | (\*.xml + \*.bin), ONNX (\*.onnx) |
|
||||
+--------------------------------+------------------------------------------------------------------------------+
|
||||
| Supported devices | :doc:`All <openvino_docs_OV_UG_supported_plugins_Supported_Devices>` |
|
||||
+--------------------------------+------------------------------------------------------------------------------+
|
||||
| Other language realization | :doc:`C++ <openvino_inference_engine_samples_throughput_benchmark_README>` |
|
||||
+--------------------------------+------------------------------------------------------------------------------+
|
||||
|
||||
|
||||
How It Works
|
||||
####################
|
||||
|
||||
The sample compiles a model for a given device, randomly generates input data, performs asynchronous inference multiple times for a given number of seconds. Then processes and reports performance results.
|
||||
|
||||
You can see the explicit description of
|
||||
each sample step at [Integration Steps](../../../../docs/OV_Runtime_UG/integrate_with_your_application.md) section of "Integrate OpenVINO™ Runtime with Your Application" guide.
|
||||
each sample step at :doc:`Integration Steps <openvino_docs_OV_UG_Integrate_OV_with_your_application>` section of "Integrate OpenVINO™ Runtime with Your Application" guide.
|
||||
|
||||
## Running
|
||||
Running
|
||||
####################
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
python throughput_benchmark.py <path_to_model>
|
||||
|
||||
```
|
||||
python throughput_benchmark.py <path_to_model>
|
||||
```
|
||||
|
||||
To run the sample, you need to specify a model:
|
||||
- You can use [public](@ref omz_models_group_public) or [Intel's](@ref omz_models_group_intel) pre-trained models from the Open Model Zoo. The models can be downloaded using the [Model Downloader](@ref omz_tools_downloader).
|
||||
|
||||
> **NOTES**:
|
||||
>
|
||||
> - Before running the sample with a trained model, make sure the model is converted to the intermediate representation (IR) format (\*.xml + \*.bin) using the [Model Optimizer tool](../../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md).
|
||||
>
|
||||
> - The sample accepts models in ONNX format (.onnx) that do not require preprocessing.
|
||||
- You can use :doc:`public <omz_models_group_public>` or :doc:`Intel's <omz_models_group_intel>` pre-trained models from the Open Model Zoo. The models can be downloaded using the :doc:`Model Downloader <omz_tools_downloader>`.
|
||||
|
||||
### Example
|
||||
.. note::
|
||||
|
||||
1. Install the `openvino-dev` Python package to use Open Model Zoo Tools:
|
||||
Before running the sample with a trained model, make sure the model is converted to the intermediate representation (IR) format (\*.xml + \*.bin) using the :doc:`Model Optimizer tool <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`.
|
||||
|
||||
The sample accepts models in ONNX format (.onnx) that do not require preprocessing.
|
||||
|
||||
|
||||
Example
|
||||
++++++++++++++++++++
|
||||
|
||||
1. Install the ``openvino-dev`` Python package to use Open Model Zoo Tools:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
python -m pip install openvino-dev[caffe]
|
||||
|
||||
```
|
||||
python -m pip install openvino-dev[caffe]
|
||||
```
|
||||
|
||||
2. Download a pre-trained model using:
|
||||
|
||||
```
|
||||
omz_downloader --name googlenet-v1
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
omz_downloader --name googlenet-v1
|
||||
|
||||
|
||||
3. If a model is not in the IR or ONNX format, it must be converted. You can do this using the model converter:
|
||||
|
||||
```
|
||||
omz_converter --name googlenet-v1
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
4. Perform benchmarking using the `googlenet-v1` model on a `CPU`:
|
||||
omz_converter --name googlenet-v1
|
||||
|
||||
```
|
||||
python throughput_benchmark.py googlenet-v1.xml
|
||||
```
|
||||
|
||||
## Sample Output
|
||||
4. Perform benchmarking using the ``googlenet-v1`` model on a ``CPU``:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
python throughput_benchmark.py googlenet-v1.xml
|
||||
|
||||
|
||||
Sample Output
|
||||
####################
|
||||
|
||||
The application outputs performance results.
|
||||
|
||||
```
|
||||
[ INFO ] OpenVINO:
|
||||
[ INFO ] Build ................................. <version>
|
||||
[ INFO ] Count: 2817 iterations
|
||||
[ INFO ] Duration: 10012.65 ms
|
||||
[ INFO ] Latency:
|
||||
[ INFO ] Median: 13.80 ms
|
||||
[ INFO ] Average: 14.10 ms
|
||||
[ INFO ] Min: 8.35 ms
|
||||
[ INFO ] Max: 28.38 ms
|
||||
[ INFO ] Throughput: 281.34 FPS
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
## See Also
|
||||
[ INFO ] OpenVINO:
|
||||
[ INFO ] Build ................................. <version>
|
||||
[ INFO ] Count: 2817 iterations
|
||||
[ INFO ] Duration: 10012.65 ms
|
||||
[ INFO ] Latency:
|
||||
[ INFO ] Median: 13.80 ms
|
||||
[ INFO ] Average: 14.10 ms
|
||||
[ INFO ] Min: 8.35 ms
|
||||
[ INFO ] Max: 28.38 ms
|
||||
[ INFO ] Throughput: 281.34 FPS
|
||||
|
||||
- [Integrate the OpenVINO™ Runtime with Your Application](../../../../docs/OV_Runtime_UG/integrate_with_your_application.md)
|
||||
- [Using OpenVINO™ Toolkit Samples](../../../../docs/OV_Runtime_UG/Samples_Overview.md)
|
||||
- [Model Downloader](@ref omz_tools_downloader)
|
||||
- [Model Optimizer](../../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md)
|
||||
|
||||
See Also
|
||||
####################
|
||||
|
||||
* :doc:`Integrate the OpenVINO™ Runtime with Your Application <openvino_docs_OV_UG_Integrate_OV_with_your_application>`
|
||||
* :doc:`Using OpenVINO Samples <openvino_docs_OV_UG_Samples_Overview>`
|
||||
* :doc:`Model Downloader <omz_tools_downloader>`
|
||||
* :doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`
|
||||
|
||||
@endsphinxdirective
|
||||
+311
-245
@@ -1,333 +1,393 @@
|
||||
# Benchmark Python Tool {#openvino_inference_engine_tools_benchmark_tool_README}
|
||||
|
||||
@sphinxdirective
|
||||
|
||||
This page demonstrates how to use the Benchmark Python Tool to estimate deep learning inference performance on supported devices.
|
||||
|
||||
> **NOTE**: This page describes usage of the Python implementation of the Benchmark Tool. For the C++ implementation, refer to the [Benchmark C++ Tool](../../samples/cpp/benchmark_app/README.md) page. The Python version is recommended for benchmarking models that will be used in Python applications, and the C++ version is recommended for benchmarking models that will be used in C++ applications. Both tools have a similar command interface and backend.
|
||||
.. note::
|
||||
|
||||
## Basic Usage
|
||||
This page describes usage of the Python implementation of the Benchmark Tool. For the C++ implementation, refer to the :doc:`Benchmark C++ Tool <openvino_inference_engine_samples_benchmark_app_README>` page. The Python version is recommended for benchmarking models that will be used in Python applications, and the C++ version is recommended for benchmarking models that will be used in C++ applications. Both tools have a similar command interface and backend.
|
||||
|
||||
The Python benchmark_app is automatically installed when you install OpenVINO Developer Tools using [PyPI](../../docs/install_guides/installing-openvino-pip.md). Before running `benchmark_app`, make sure the `openvino_env` virtual environment is activated, and navigate to the directory where your model is located.
|
||||
Basic Usage
|
||||
####################
|
||||
|
||||
The benchmarking application works with models in the OpenVINO IR (`model.xml` and `model.bin`) and ONNX (`model.onnx`) formats. Make sure to [convert your models](../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) if necessary.
|
||||
The Python benchmark_app is automatically installed when you install OpenVINO Developer Tools using :doc:`PyPI <openvino_docs_install_guides_installing_openvino_pip>`. Before running ``benchmark_app``, make sure the ``openvino_env`` virtual environment is activated, and navigate to the directory where your model is located.
|
||||
|
||||
The benchmarking application works with models in the OpenVINO IR (``model.xml`` and ``model.bin``) and ONNX (``model.onnx``) formats.
|
||||
Make sure to :doc:`convert your models <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>` if necessary.
|
||||
|
||||
To run benchmarking with default options on a model, use the following command:
|
||||
|
||||
```
|
||||
benchmark_app -m model.xml
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
benchmark_app -m model.xml
|
||||
|
||||
|
||||
By default, the application will load the specified model onto the CPU and perform inferencing on batches of randomly-generated data inputs for 60 seconds. As it loads, it prints information about benchmark parameters. When benchmarking is completed, it reports the minimum, average, and maximum inferencing latency and average the throughput.
|
||||
|
||||
You may be able to improve benchmark results beyond the default configuration by configuring some of the execution parameters for your model. For example, you can use "throughput" or "latency" performance hints to optimize the runtime for higher FPS or reduced inferencing time. Read on to learn more about the configuration options available with benchmark_app.
|
||||
|
||||
## Configuration Options
|
||||
The benchmark app provides various options for configuring execution parameters. This section covers key configuration options for easily tuning benchmarking to achieve better performance on your device. A list of all configuration options is given in the [Advanced Usage](#advanced-usage-python-benchmark) section.
|
||||
Configuration Options
|
||||
#####################
|
||||
|
||||
The benchmark app provides various options for configuring execution parameters. This section covers key configuration options for easily tuning benchmarking to achieve better performance on your device. A list of all configuration options is given in the :ref:`Advanced Usage <advanced-usage-python-benchmark>` section.
|
||||
|
||||
Performance hints: latency and throughput
|
||||
+++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
### Performance hints: latency and throughput
|
||||
The benchmark app allows users to provide high-level "performance hints" for setting latency-focused or throughput-focused inference modes. This hint causes the runtime to automatically adjust runtime parameters, such as the number of processing streams and inference batch size, to prioritize for reduced latency or high throughput.
|
||||
|
||||
The performance hints do not require any device-specific settings and they are completely portable between devices. Parameters are automatically configured based on whichever device is being used. This allows users to easily port applications between hardware targets without having to re-determine the best runtime parameters for the new device.
|
||||
|
||||
If not specified, throughput is used as the default. To set the hint explicitly, use `-hint latency` or `-hint throughput` when running benchmark_app:
|
||||
If not specified, throughput is used as the default. To set the hint explicitly, use ``-hint latency`` or ``-hint throughput`` when running benchmark_app:
|
||||
|
||||
```
|
||||
benchmark_app -m model.xml -hint latency
|
||||
benchmark_app -m model.xml -hint throughput
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
> **NOTE**
|
||||
It is up to the user to ensure the environment on which the benchmark is running is optimized for maximum performance.
|
||||
Otherwise, different results may occur when using the application in different environment settings (such as power optimization settings, processor overclocking, thermal throttling).
|
||||
benchmark_app -m model.xml -hint latency
|
||||
benchmark_app -m model.xml -hint throughput
|
||||
|
||||
|
||||
.. note::
|
||||
|
||||
It is up to the user to ensure the environment on which the benchmark is running is optimized for maximum performance. Otherwise, different results may occur when using the application in different environment settings (such as power optimization settings, processor overclocking, thermal throttling).
|
||||
|
||||
|
||||
Latency
|
||||
--------------------
|
||||
|
||||
#### Latency
|
||||
Latency is the amount of time it takes to process a single inference request. In applications where data needs to be inferenced and acted on as quickly as possible (such as autonomous driving), low latency is desirable. For conventional devices, lower latency is achieved by reducing the amount of parallel processing streams so the system can utilize as many resources as possible to quickly calculate each inference request. However, advanced devices like multi-socket CPUs and modern GPUs are capable of running multiple inference requests while delivering the same latency.
|
||||
|
||||
When benchmark_app is run with `-hint latency`, it determines the optimal number of parallel inference requests for minimizing latency while still maximizing the parallelization capabilities of the hardware. It automatically sets the number of processing streams and inference batch size to achieve the best latency.
|
||||
When benchmark_app is run with ``-hint latency``, it determines the optimal number of parallel inference requests for minimizing latency while still maximizing the parallelization capabilities of the hardware. It automatically sets the number of processing streams and inference batch size to achieve the best latency.
|
||||
|
||||
Throughput
|
||||
--------------------
|
||||
|
||||
#### Throughput
|
||||
Throughput is the amount of data an inferencing pipeline can process at once, and it is usually measured in frames per second (FPS) or inferences per second. In applications where large amounts of data needs to be inferenced simultaneously (such as multi-camera video streams), high throughput is needed. To achieve high throughput, the runtime focuses on fully saturating the device with enough data to process. It utilizes as much memory and as many parallel streams as possible to maximize the amount of data that can be processed simultaneously.
|
||||
|
||||
When benchmark_app is run with `-hint throughput`, it maximizes the number of parallel inference requests to utilize all the threads available on the device. On GPU, it automatically sets the inference batch size to fill up the GPU memory available.
|
||||
When benchmark_app is run with ``-hint throughput``, it maximizes the number of parallel inference requests to utilize all the threads available on the device. On GPU, it automatically sets the inference batch size to fill up the GPU memory available.
|
||||
|
||||
For more information on performance hints, see the [High-level Performance Hints](../../docs/OV_Runtime_UG/performance_hints.md) page. For more details on optimal runtime configurations and how they are automatically determined using performance hints, see [Runtime Inference Optimizations](../../docs/optimization_guide/dldt_deployment_optimization_guide.md).
|
||||
For more information on performance hints, see the :doc:`High-level Performance Hints <openvino_docs_OV_UG_Performance_Hints>` page. For more details on optimal runtime configurations and how they are automatically determined using performance hints, see :doc:`Runtime Inference Optimizations <openvino_docs_deployment_optimization_guide_dldt_optimization_guide>`.
|
||||
|
||||
|
||||
### Device
|
||||
To set which device benchmarking runs on, use the `-d <device>` argument. This will tell benchmark_app to run benchmarking on that specific device. The benchmark app supports "CPU", "GPU", and GNA devices. In order to use the GPU, the system must have the appropriate drivers installed. If no device is specified, benchmark_app will default to using CPU.
|
||||
Device
|
||||
++++++++++++++++++++
|
||||
|
||||
To set which device benchmarking runs on, use the ``-d <device>`` argument. This will tell benchmark_app to run benchmarking on that specific device. The benchmark app supports "CPU", "GPU", and GNA devices. In order to use the GPU, the system must have the appropriate drivers installed. If no device is specified, benchmark_app will default to using CPU.
|
||||
|
||||
For example, to run benchmarking on GPU, use:
|
||||
|
||||
```
|
||||
benchmark_app -m model.xml -d GPU
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
You may also specify "AUTO" as the device, in which case the benchmark_app will automatically select the best device for benchmarking and support it with the CPU at the model loading stage. This may result in increased performance, thus, should be used purposefully. For more information, see the [Automatic device selection](../../docs/OV_Runtime_UG/auto_device_selection.md) page.
|
||||
benchmark_app -m model.xml -d GPU
|
||||
|
||||
|
||||
You may also specify "AUTO" as the device, in which case the benchmark_app will automatically select the best device for benchmarking and support it with the CPU at the model loading stage. This may result in increased performance, thus, should be used purposefully. For more information, see the :doc:`Automatic device selection <openvino_docs_OV_UG_supported_plugins_AUTO>` page.
|
||||
|
||||
(Note: If the latency or throughput hint is set, it will automatically configure streams and batch sizes for optimal performance based on the specified device.)
|
||||
|
||||
### Number of iterations
|
||||
Number of iterations
|
||||
++++++++++++++++++++
|
||||
|
||||
By default, the benchmarking app will run for a predefined duration, repeatedly performing inferencing with the model and measuring the resulting inference speed. There are several options for setting the number of inference iterations:
|
||||
|
||||
* Explicitly specify the number of iterations the model runs using the `-niter <number_of_iterations>` option
|
||||
* Set how much time the app runs for using the `-t <seconds>` option
|
||||
* Explicitly specify the number of iterations the model runs using the ``-niter <number_of_iterations>`` option
|
||||
* Set how much time the app runs for using the ``-t <seconds>`` option
|
||||
* Set both of them (execution will continue until both conditions are met)
|
||||
* If neither -niter nor -t are specified, the app will run for a predefined duration that depends on the device
|
||||
|
||||
The more iterations a model runs, the better the statistics will be for determing average latency and throughput.
|
||||
The more iterations a model runs, the better the statistics will be for determining average latency and throughput.
|
||||
|
||||
### Inputs
|
||||
The benchmark tool runs benchmarking on user-provided input images in `.jpg`, `.bmp`, or `.png` format. Use `-i <PATH_TO_INPUT>` to specify the path to an image, or folder of images. For example, to run benchmarking on an image named `test1.jpg`, use:
|
||||
Inputs
|
||||
++++++++++++++++++++
|
||||
|
||||
```
|
||||
./benchmark_app -m model.xml -i test1.jpg
|
||||
```
|
||||
The benchmark tool runs benchmarking on user-provided input images in ``.jpg``, ``.bmp``, or ``.png`` format. Use ``-i <PATH_TO_INPUT>`` to specify the path to an image, or folder of images. For example, to run benchmarking on an image named ``test1.jpg``, use:
|
||||
|
||||
The tool will repeatedly loop through the provided inputs and run inferencing on them for the specified amount of time or number of iterations. If the `-i` flag is not used, the tool will automatically generate random data to fit the input shape of the model.
|
||||
.. code-block:: sh
|
||||
|
||||
### Examples
|
||||
For more usage examples (and step-by-step instructions on how to set up a model for benchmarking), see the [Examples of Running the Tool](#examples-of-running-the-tool-python) section.
|
||||
|
||||
## <a name="advanced-usage-python-benchmark"></a> Advanced Usage
|
||||
|
||||
> **NOTE**: By default, OpenVINO samples, tools and demos expect input with BGR channels order. If you trained your model to work with RGB order, you need to manually rearrange the default channel order in the sample or demo application or reconvert your model using the Model Optimizer tool with --reverse_input_channels argument specified. For more information about the argument, refer to When to Reverse Input Channels section of Converting a Model to Intermediate Representation (IR).
|
||||
|
||||
### Per-layer performance and logging
|
||||
The application also collects per-layer Performance Measurement (PM) counters for each executed infer request if you enable statistics dumping by setting the `-report_type` parameter to one of the possible values:
|
||||
|
||||
* `no_counters` report includes configuration options specified, resulting FPS and latency.
|
||||
* `average_counters` report extends the `no_counters` report and additionally includes average PM counters values for each layer from the network.
|
||||
* `detailed_counters` report extends the `average_counters` report and additionally includes per-layer PM counters and latency for each executed infer request.
|
||||
|
||||
Depending on the type, the report is stored to benchmark_no_counters_report.csv, benchmark_average_counters_report.csv, or benchmark_detailed_counters_report.csv file located in the path specified in -report_folder. The application also saves executable graph information serialized to an XML file if you specify a path to it with the -exec_graph_path parameter.
|
||||
|
||||
### <a name="all-configuration-options-python-benchmark"></a> All configuration options
|
||||
Running the application with the `-h` or `--help` option yields the following usage message:
|
||||
|
||||
```
|
||||
[Step 1/11] Parsing and validating input arguments
|
||||
[ INFO ] Parsing input parameters
|
||||
usage: benchmark_app.py [-h [HELP]] [-i PATHS_TO_INPUT [PATHS_TO_INPUT ...]] -m PATH_TO_MODEL [-d TARGET_DEVICE]
|
||||
[-hint {throughput,cumulative_throughput,latency,none}] [-niter NUMBER_ITERATIONS] [-t TIME] [-b BATCH_SIZE] [-shape SHAPE]
|
||||
[-data_shape DATA_SHAPE] [-layout LAYOUT] [-extensions EXTENSIONS] [-c PATH_TO_CLDNN_CONFIG] [-cdir CACHE_DIR] [-lfile [LOAD_FROM_FILE]]
|
||||
[-api {sync,async}] [-nireq NUMBER_INFER_REQUESTS] [-nstreams NUMBER_STREAMS] [-inference_only [INFERENCE_ONLY]]
|
||||
[-infer_precision INFER_PRECISION] [-ip {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}]
|
||||
[-op {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}] [-iop INPUT_OUTPUT_PRECISION] [--mean_values [R,G,B]] [--scale_values [R,G,B]]
|
||||
[-nthreads NUMBER_THREADS] [-pin {YES,NO,NUMA,HYBRID_AWARE}] [-latency_percentile LATENCY_PERCENTILE]
|
||||
[-report_type {no_counters,average_counters,detailed_counters}] [-report_folder REPORT_FOLDER] [-pc [PERF_COUNTS]]
|
||||
[-pcsort {no_sort,sort,simple_sort}] [-pcseq [PCSEQ]] [-exec_graph_path EXEC_GRAPH_PATH] [-dump_config DUMP_CONFIG] [-load_config LOAD_CONFIG]
|
||||
|
||||
Options:
|
||||
-h [HELP], --help [HELP]
|
||||
Show this help message and exit.
|
||||
|
||||
-i PATHS_TO_INPUT [PATHS_TO_INPUT ...], --paths_to_input PATHS_TO_INPUT [PATHS_TO_INPUT ...]
|
||||
Optional. Path to a folder with images and/or binaries or to specific image or binary file.It is also allowed to map files to model inputs:
|
||||
input_1:file_1/dir1,file_2/dir2,input_4:file_4/dir4 input_2:file_3/dir3 Currently supported data types: bin, npy. If OPENCV is enabled, this
|
||||
functionalityis extended with the following data types: bmp, dib, jpeg, jpg, jpe, jp2, png, pbm, pgm, ppm, sr, ras, tiff, tif.
|
||||
|
||||
-m PATH_TO_MODEL, --path_to_model PATH_TO_MODEL
|
||||
Required. Path to an .xml/.onnx file with a trained model or to a .blob file with a trained compiled model.
|
||||
|
||||
-d TARGET_DEVICE, --target_device TARGET_DEVICE
|
||||
Optional. Specify a target device to infer on (the list of available devices is shown below). Default value is CPU. Use '-d HETERO:<comma
|
||||
separated devices list>' format to specify HETERO plugin. Use '-d MULTI:<comma separated devices list>' format to specify MULTI plugin. The
|
||||
application looks for a suitable plugin for the specified device.
|
||||
|
||||
-hint {throughput,cumulative_throughput,latency,none}, --perf_hint {throughput,cumulative_throughput,latency,none}
|
||||
Optional. Performance hint (latency or throughput or cumulative_throughput or none). Performance hint allows the OpenVINO device to select the
|
||||
right model-specific settings. 'throughput': device performance mode will be set to THROUGHPUT. 'cumulative_throughput': device performance
|
||||
mode will be set to CUMULATIVE_THROUGHPUT. 'latency': device performance mode will be set to LATENCY. 'none': no device performance mode will
|
||||
be set. Using explicit 'nstreams' or other device-specific options, please set hint to 'none'
|
||||
|
||||
-niter NUMBER_ITERATIONS, --number_iterations NUMBER_ITERATIONS
|
||||
Optional. Number of iterations. If not specified, the number of iterations is calculated depending on a device.
|
||||
|
||||
-t TIME, --time TIME Optional. Time in seconds to execute topology.
|
||||
|
||||
-api {sync,async}, --api_type {sync,async}
|
||||
Optional. Enable using sync/async API. Default value is async.
|
||||
./benchmark_app -m model.xml -i test1.jpg
|
||||
|
||||
|
||||
Input shapes:
|
||||
-b BATCH_SIZE, --batch_size BATCH_SIZE
|
||||
Optional. Batch size value. If not specified, the batch size value is determined from Intermediate Representation
|
||||
The tool will repeatedly loop through the provided inputs and run inferencing on them for the specified amount of time or number of iterations. If the ``-i`` flag is not used, the tool will automatically generate random data to fit the input shape of the model.
|
||||
|
||||
-shape SHAPE Optional. Set shape for input. For example, "input1[1,3,224,224],input2[1,4]" or "[1,3,224,224]" in case of one input size. This parameter
|
||||
affect model Parameter shape, can be dynamic. For dynamic dimesions use symbol `?`, `-1` or range `low.. up`.
|
||||
Examples
|
||||
++++++++++++++++++++
|
||||
|
||||
-data_shape DATA_SHAPE
|
||||
Optional. Optional if model shapes are all static (original ones or set by -shape).Required if at least one input shape is dynamic and input
|
||||
images are not provided.Set shape for input tensors. For example, "input1[1,3,224,224][1,3,448,448],input2[1,4][1,8]" or
|
||||
"[1,3,224,224][1,3,448,448] in case of one input size.
|
||||
For more usage examples (and step-by-step instructions on how to set up a model for benchmarking), see the :ref:`Examples of Running the Tool <examples-of-running-the-tool-python>` section.
|
||||
|
||||
-layout LAYOUT Optional. Prompts how model layouts should be treated by application. For example, "input1[NCHW],input2[NC]" or "[NCHW]" in case of one input
|
||||
size.
|
||||
.. _advanced-usage-python-benchmark:
|
||||
|
||||
Advanced Usage
|
||||
####################
|
||||
|
||||
.. note::
|
||||
|
||||
By default, OpenVINO samples, tools and demos expect input with BGR channels order. If you trained your model to work with RGB order, you need to manually rearrange the default channel order in the sample or demo application or reconvert your model using the Model Optimizer tool with ``--reverse_input_channels`` argument specified. For more information about the argument, refer to When to Reverse Input Channels section of Converting a Model to Intermediate Representation (IR).
|
||||
|
||||
|
||||
Advanced options:
|
||||
-extensions EXTENSIONS, --extensions EXTENSIONS
|
||||
Optional. Path or a comma-separated list of paths to libraries (.so or .dll) with extensions.
|
||||
Per-layer performance and logging
|
||||
+++++++++++++++++++++++++++++++++
|
||||
|
||||
-c PATH_TO_CLDNN_CONFIG, --path_to_cldnn_config PATH_TO_CLDNN_CONFIG
|
||||
Optional. Required for GPU custom kernels. Absolute path to an .xml file with the kernels description.
|
||||
The application also collects per-layer Performance Measurement (PM) counters for each executed infer request if you enable statistics dumping by setting the ``-report_type`` parameter to one of the possible values:
|
||||
|
||||
-cdir CACHE_DIR, --cache_dir CACHE_DIR
|
||||
Optional. Enable model caching to specified directory
|
||||
* ``no_counters`` report includes configuration options specified, resulting FPS and latency.
|
||||
* ``average_counters`` report extends the ``no_counters`` report and additionally includes average PM counters values for each layer from the network.
|
||||
* ``detailed_counters`` report extends the ``average_counters`` report and additionally includes per-layer PM counters and latency for each executed infer request.
|
||||
|
||||
-lfile [LOAD_FROM_FILE], --load_from_file [LOAD_FROM_FILE]
|
||||
Optional. Loads model from file directly without read_model.
|
||||
Depending on the type, the report is stored to ``benchmark_no_counters_report.csv``, ``benchmark_average_counters_report.csv``, or ``benchmark_detailed_counters_report.csv`` file located in the path specified in ``-report_folder``. The application also saves executable graph information serialized to an XML file if you specify a path to it with the ``-exec_graph_path`` parameter.
|
||||
|
||||
-nireq NUMBER_INFER_REQUESTS, --number_infer_requests NUMBER_INFER_REQUESTS
|
||||
Optional. Number of infer requests. Default value is determined automatically for device.
|
||||
.. _all-configuration-options-python-benchmark:
|
||||
|
||||
-nstreams NUMBER_STREAMS, --number_streams NUMBER_STREAMS
|
||||
Optional. Number of streams to use for inference on the CPU/GPU (for HETERO and MULTI device cases use format
|
||||
<device1>:<nstreams1>,<device2>:<nstreams2> or just <nstreams>). Default value is determined automatically for a device. Please note that
|
||||
although the automatic selection usually provides a reasonable performance, it still may be non - optimal for some cases, especially for very
|
||||
small models. Also, using nstreams>1 is inherently throughput-oriented option, while for the best-latency estimations the number of streams
|
||||
should be set to 1. See samples README for more details.
|
||||
All configuration options
|
||||
+++++++++++++++++++++++++
|
||||
|
||||
-inference_only [INFERENCE_ONLY], --inference_only [INFERENCE_ONLY]
|
||||
Optional. If true inputs filling only once before measurements (default for static models), else inputs filling is included into loop
|
||||
measurement (default for dynamic models)
|
||||
Running the application with the ``-h`` or ``--help`` option yields the following usage message:
|
||||
|
||||
-infer_precision INFER_PRECISION
|
||||
Optional. Specifies the inference precision. Example #1: '-infer_precision bf16'. Example #2: '-infer_precision CPU:bf16,GPU:f32'
|
||||
.. scrollbox::
|
||||
|
||||
-exec_graph_path EXEC_GRAPH_PATH, --exec_graph_path EXEC_GRAPH_PATH
|
||||
Optional. Path to a file where to store executable graph information serialized.
|
||||
.. code-block:: sh
|
||||
|
||||
[Step 1/11] Parsing and validating input arguments
|
||||
[ INFO ] Parsing input parameters
|
||||
usage: benchmark_app.py [-h [HELP]] [-i PATHS_TO_INPUT [PATHS_TO_INPUT ...]] -m PATH_TO_MODEL [-d TARGET_DEVICE]
|
||||
[-hint {throughput,cumulative_throughput,latency,none}] [-niter NUMBER_ITERATIONS] [-t TIME] [-b BATCH_SIZE] [-shape SHAPE]
|
||||
[-data_shape DATA_SHAPE] [-layout LAYOUT] [-extensions EXTENSIONS] [-c PATH_TO_CLDNN_CONFIG] [-cdir CACHE_DIR] [-lfile [LOAD_FROM_FILE]]
|
||||
[-api {sync,async}] [-nireq NUMBER_INFER_REQUESTS] [-nstreams NUMBER_STREAMS] [-inference_only [INFERENCE_ONLY]]
|
||||
[-infer_precision INFER_PRECISION] [-ip {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}]
|
||||
[-op {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}] [-iop INPUT_OUTPUT_PRECISION] [--mean_values [R,G,B]] [--scale_values [R,G,B]]
|
||||
[-nthreads NUMBER_THREADS] [-pin {YES,NO,NUMA,HYBRID_AWARE}] [-latency_percentile LATENCY_PERCENTILE]
|
||||
[-report_type {no_counters,average_counters,detailed_counters}] [-report_folder REPORT_FOLDER] [-pc [PERF_COUNTS]]
|
||||
[-pcsort {no_sort,sort,simple_sort}] [-pcseq [PCSEQ]] [-exec_graph_path EXEC_GRAPH_PATH] [-dump_config DUMP_CONFIG] [-load_config LOAD_CONFIG]
|
||||
|
||||
Options:
|
||||
-h [HELP], --help [HELP]
|
||||
Show this help message and exit.
|
||||
|
||||
-i PATHS_TO_INPUT [PATHS_TO_INPUT ...], --paths_to_input PATHS_TO_INPUT [PATHS_TO_INPUT ...]
|
||||
Optional. Path to a folder with images and/or binaries or to specific image or binary file.It is also allowed to map files to model inputs:
|
||||
input_1:file_1/dir1,file_2/dir2,input_4:file_4/dir4 input_2:file_3/dir3 Currently supported data types: bin, npy. If OPENCV is enabled, this
|
||||
functionalityis extended with the following data types: bmp, dib, jpeg, jpg, jpe, jp2, png, pbm, pgm, ppm, sr, ras, tiff, tif.
|
||||
|
||||
-m PATH_TO_MODEL, --path_to_model PATH_TO_MODEL
|
||||
Required. Path to an .xml/.onnx file with a trained model or to a .blob file with a trained compiled model.
|
||||
|
||||
-d TARGET_DEVICE, --target_device TARGET_DEVICE
|
||||
Optional. Specify a target device to infer on (the list of available devices is shown below). Default value is CPU. Use '-d HETERO:<comma
|
||||
separated devices list>' format to specify HETERO plugin. Use '-d MULTI:<comma separated devices list>' format to specify MULTI plugin. The
|
||||
application looks for a suitable plugin for the specified device.
|
||||
|
||||
-hint {throughput,cumulative_throughput,latency,none}, --perf_hint {throughput,cumulative_throughput,latency,none}
|
||||
Optional. Performance hint (latency or throughput or cumulative_throughput or none). Performance hint allows the OpenVINO device to select the
|
||||
right model-specific settings. 'throughput': device performance mode will be set to THROUGHPUT. 'cumulative_throughput': device performance
|
||||
mode will be set to CUMULATIVE_THROUGHPUT. 'latency': device performance mode will be set to LATENCY. 'none': no device performance mode will
|
||||
be set. Using explicit 'nstreams' or other device-specific options, please set hint to 'none'
|
||||
|
||||
-niter NUMBER_ITERATIONS, --number_iterations NUMBER_ITERATIONS
|
||||
Optional. Number of iterations. If not specified, the number of iterations is calculated depending on a device.
|
||||
|
||||
-t TIME, --time TIME Optional. Time in seconds to execute topology.
|
||||
|
||||
-api {sync,async}, --api_type {sync,async}
|
||||
Optional. Enable using sync/async API. Default value is async.
|
||||
|
||||
|
||||
Preprocessing options:
|
||||
-ip {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}, --input_precision {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}
|
||||
Optional. Specifies precision for all input layers of the model.
|
||||
Input shapes:
|
||||
-b BATCH_SIZE, --batch_size BATCH_SIZE
|
||||
Optional. Batch size value. If not specified, the batch size value is determined from Intermediate Representation
|
||||
|
||||
-op {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}, --output_precision {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}
|
||||
Optional. Specifies precision for all output layers of the model.
|
||||
-shape SHAPE Optional. Set shape for input. For example, "input1[1,3,224,224],input2[1,4]" or "[1,3,224,224]" in case of one input size. This parameter
|
||||
affect model Parameter shape, can be dynamic. For dynamic dimesions use symbol `?`, `-1` or range `low.. up`.
|
||||
|
||||
-iop INPUT_OUTPUT_PRECISION, --input_output_precision INPUT_OUTPUT_PRECISION
|
||||
Optional. Specifies precision for input and output layers by name. Example: -iop "input:f16, output:f16". Notice that quotes are required.
|
||||
Overwrites precision from ip and op options for specified layers.
|
||||
-data_shape DATA_SHAPE
|
||||
Optional. Optional if model shapes are all static (original ones or set by -shape).Required if at least one input shape is dynamic and input
|
||||
images are not provided.Set shape for input tensors. For example, "input1[1,3,224,224][1,3,448,448],input2[1,4][1,8]" or
|
||||
"[1,3,224,224][1,3,448,448] in case of one input size.
|
||||
|
||||
--mean_values [R,G,B]
|
||||
Optional. Mean values to be used for the input image per channel. Values to be provided in the [R,G,B] format. Can be defined for desired input
|
||||
of the model, for example: "--mean_values data[255,255,255],info[255,255,255]". The exact meaning and order of channels depend on how the
|
||||
original model was trained. Applying the values affects performance and may cause type conversion
|
||||
|
||||
--scale_values [R,G,B]
|
||||
Optional. Scale values to be used for the input image per channel. Values are provided in the [R,G,B] format. Can be defined for desired input
|
||||
of the model, for example: "--scale_values data[255,255,255],info[255,255,255]". The exact meaning and order of channels depend on how the
|
||||
original model was trained. If both --mean_values and --scale_values are specified, the mean is subtracted first and then scale is applied
|
||||
regardless of the order of options in command line. Applying the values affects performance and may cause type conversion
|
||||
-layout LAYOUT Optional. Prompts how model layouts should be treated by application. For example, "input1[NCHW],input2[NC]" or "[NCHW]" in case of one input
|
||||
size.
|
||||
|
||||
|
||||
Device-specific performance options:
|
||||
-nthreads NUMBER_THREADS, --number_threads NUMBER_THREADS
|
||||
Number of threads to use for inference on the CPU, GNA (including HETERO and MULTI cases).
|
||||
Advanced options:
|
||||
-extensions EXTENSIONS, --extensions EXTENSIONS
|
||||
Optional. Path or a comma-separated list of paths to libraries (.so or .dll) with extensions.
|
||||
|
||||
-pin {YES,NO,NUMA,HYBRID_AWARE}, --infer_threads_pinning {YES,NO,NUMA,HYBRID_AWARE}
|
||||
Optional. Enable threads->cores ('YES' which is OpenVINO runtime's default for conventional CPUs), threads->(NUMA)nodes ('NUMA'),
|
||||
threads->appropriate core types ('HYBRID_AWARE', which is OpenVINO runtime's default for Hybrid CPUs) or completely disable ('NO') CPU threads
|
||||
pinning for CPU-involved inference.
|
||||
-c PATH_TO_CLDNN_CONFIG, --path_to_cldnn_config PATH_TO_CLDNN_CONFIG
|
||||
Optional. Required for GPU custom kernels. Absolute path to an .xml file with the kernels description.
|
||||
|
||||
-cdir CACHE_DIR, --cache_dir CACHE_DIR
|
||||
Optional. Enable model caching to specified directory
|
||||
|
||||
-lfile [LOAD_FROM_FILE], --load_from_file [LOAD_FROM_FILE]
|
||||
Optional. Loads model from file directly without read_model.
|
||||
|
||||
-nireq NUMBER_INFER_REQUESTS, --number_infer_requests NUMBER_INFER_REQUESTS
|
||||
Optional. Number of infer requests. Default value is determined automatically for device.
|
||||
|
||||
-nstreams NUMBER_STREAMS, --number_streams NUMBER_STREAMS
|
||||
Optional. Number of streams to use for inference on the CPU/GPU (for HETERO and MULTI device cases use format
|
||||
<device1>:<nstreams1>,<device2>:<nstreams2> or just <nstreams>). Default value is determined automatically for a device. Please note that
|
||||
although the automatic selection usually provides a reasonable performance, it still may be non - optimal for some cases, especially for very
|
||||
small models. Also, using nstreams>1 is inherently throughput-oriented option, while for the best-latency estimations the number of streams
|
||||
should be set to 1. See samples README for more details.
|
||||
|
||||
-inference_only [INFERENCE_ONLY], --inference_only [INFERENCE_ONLY]
|
||||
Optional. If true inputs filling only once before measurements (default for static models), else inputs filling is included into loop
|
||||
measurement (default for dynamic models)
|
||||
|
||||
-infer_precision INFER_PRECISION
|
||||
Optional. Specifies the inference precision. Example #1: '-infer_precision bf16'. Example #2: '-infer_precision CPU:bf16,GPU:f32'
|
||||
|
||||
-exec_graph_path EXEC_GRAPH_PATH, --exec_graph_path EXEC_GRAPH_PATH
|
||||
Optional. Path to a file where to store executable graph information serialized.
|
||||
|
||||
|
||||
Statistics dumping options:
|
||||
-latency_percentile LATENCY_PERCENTILE, --latency_percentile LATENCY_PERCENTILE
|
||||
Optional. Defines the percentile to be reported in latency metric. The valid range is [1, 100]. The default value is 50 (median).
|
||||
Preprocessing options:
|
||||
-ip {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}, --input_precision {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}
|
||||
Optional. Specifies precision for all input layers of the model.
|
||||
|
||||
-report_type {no_counters,average_counters,detailed_counters}, --report_type {no_counters,average_counters,detailed_counters}
|
||||
Optional. Enable collecting statistics report. "no_counters" report contains configuration options specified, resulting FPS and latency.
|
||||
"average_counters" report extends "no_counters" report and additionally includes average PM counters values for each layer from the model.
|
||||
"detailed_counters" report extends "average_counters" report and additionally includes per-layer PM counters and latency for each executed
|
||||
infer request.
|
||||
-op {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}, --output_precision {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}
|
||||
Optional. Specifies precision for all output layers of the model.
|
||||
|
||||
-report_folder REPORT_FOLDER, --report_folder REPORT_FOLDER
|
||||
Optional. Path to a folder where statistics report is stored.
|
||||
-iop INPUT_OUTPUT_PRECISION, --input_output_precision INPUT_OUTPUT_PRECISION
|
||||
Optional. Specifies precision for input and output layers by name. Example: -iop "input:f16, output:f16". Notice that quotes are required.
|
||||
Overwrites precision from ip and op options for specified layers.
|
||||
|
||||
-json_stats [JSON_STATS], --json_stats [JSON_STATS]
|
||||
Optional. Enables JSON-based statistics output (by default reporting system will use CSV format). Should be used together with -report_folder option.
|
||||
--mean_values [R,G,B]
|
||||
Optional. Mean values to be used for the input image per channel. Values to be provided in the [R,G,B] format. Can be defined for desired input
|
||||
of the model, for example: "--mean_values data[255,255,255],info[255,255,255]". The exact meaning and order of channels depend on how the
|
||||
original model was trained. Applying the values affects performance and may cause type conversion
|
||||
|
||||
-pc [PERF_COUNTS], --perf_counts [PERF_COUNTS]
|
||||
Optional. Report performance counters.
|
||||
--scale_values [R,G,B]
|
||||
Optional. Scale values to be used for the input image per channel. Values are provided in the [R,G,B] format. Can be defined for desired input
|
||||
of the model, for example: "--scale_values data[255,255,255],info[255,255,255]". The exact meaning and order of channels depend on how the
|
||||
original model was trained. If both --mean_values and --scale_values are specified, the mean is subtracted first and then scale is applied
|
||||
regardless of the order of options in command line. Applying the values affects performance and may cause type conversion
|
||||
|
||||
-pcsort {no_sort,sort,simple_sort}, --perf_counts_sort {no_sort,sort,simple_sort}
|
||||
Optional. Report performance counters and analysis the sort hotpoint opts. sort: Analysis opts time cost, print by hotpoint order no_sort:
|
||||
Analysis opts time cost, print by normal order simple_sort: Analysis opts time cost, only print EXECUTED opts by normal order
|
||||
|
||||
-pcseq [PCSEQ], --pcseq [PCSEQ]
|
||||
Optional. Report latencies for each shape in -data_shape sequence.
|
||||
Device-specific performance options:
|
||||
-nthreads NUMBER_THREADS, --number_threads NUMBER_THREADS
|
||||
Number of threads to use for inference on the CPU, GNA (including HETERO and MULTI cases).
|
||||
|
||||
-dump_config DUMP_CONFIG
|
||||
Optional. Path to JSON file to dump OpenVINO parameters, which were set by application.
|
||||
-pin {YES,NO,NUMA,HYBRID_AWARE}, --infer_threads_pinning {YES,NO,NUMA,HYBRID_AWARE}
|
||||
Optional. Enable threads->cores ('YES' which is OpenVINO runtime's default for conventional CPUs), threads->(NUMA)nodes ('NUMA'),
|
||||
threads->appropriate core types ('HYBRID_AWARE', which is OpenVINO runtime's default for Hybrid CPUs) or completely disable ('NO') CPU threads
|
||||
pinning for CPU-involved inference.
|
||||
|
||||
|
||||
Statistics dumping options:
|
||||
-latency_percentile LATENCY_PERCENTILE, --latency_percentile LATENCY_PERCENTILE
|
||||
Optional. Defines the percentile to be reported in latency metric. The valid range is [1, 100]. The default value is 50 (median).
|
||||
|
||||
-report_type {no_counters,average_counters,detailed_counters}, --report_type {no_counters,average_counters,detailed_counters}
|
||||
Optional. Enable collecting statistics report. "no_counters" report contains configuration options specified, resulting FPS and latency.
|
||||
"average_counters" report extends "no_counters" report and additionally includes average PM counters values for each layer from the model.
|
||||
"detailed_counters" report extends "average_counters" report and additionally includes per-layer PM counters and latency for each executed
|
||||
infer request.
|
||||
|
||||
-report_folder REPORT_FOLDER, --report_folder REPORT_FOLDER
|
||||
Optional. Path to a folder where statistics report is stored.
|
||||
|
||||
-json_stats [JSON_STATS], --json_stats [JSON_STATS]
|
||||
Optional. Enables JSON-based statistics output (by default reporting system will use CSV format). Should be used together with -report_folder option.
|
||||
|
||||
-pc [PERF_COUNTS], --perf_counts [PERF_COUNTS]
|
||||
Optional. Report performance counters.
|
||||
|
||||
-pcsort {no_sort,sort,simple_sort}, --perf_counts_sort {no_sort,sort,simple_sort}
|
||||
Optional. Report performance counters and analysis the sort hotpoint opts. sort: Analysis opts time cost, print by hotpoint order no_sort:
|
||||
Analysis opts time cost, print by normal order simple_sort: Analysis opts time cost, only print EXECUTED opts by normal order
|
||||
|
||||
-pcseq [PCSEQ], --pcseq [PCSEQ]
|
||||
Optional. Report latencies for each shape in -data_shape sequence.
|
||||
|
||||
-dump_config DUMP_CONFIG
|
||||
Optional. Path to JSON file to dump OpenVINO parameters, which were set by application.
|
||||
|
||||
-load_config LOAD_CONFIG
|
||||
Optional. Path to JSON file to load custom OpenVINO parameters.
|
||||
Please note, command line parameters have higher priority then parameters from configuration file.
|
||||
Example 1: a simple JSON file for HW device with primary properties.
|
||||
{
|
||||
"CPU": {"NUM_STREAMS": "3", "PERF_COUNT": "NO"}
|
||||
}
|
||||
Example 2: a simple JSON file for meta device(AUTO/MULTI) with HW device properties.
|
||||
{
|
||||
"AUTO": {
|
||||
"PERFORMANCE_HINT": "",
|
||||
"PERF_COUNT": "NO",
|
||||
"DEVICE_PROPERTIES": {
|
||||
"CPU": {
|
||||
"INFERENCE_PRECISION_HINT": "f32",
|
||||
"NUM_STREAMS": "3"
|
||||
},
|
||||
"GPU": {
|
||||
"INFERENCE_PRECISION_HINT": "f32",
|
||||
"NUM_STREAMS": "5"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
-load_config LOAD_CONFIG
|
||||
Optional. Path to JSON file to load custom OpenVINO parameters.
|
||||
Please note, command line parameters have higher priority then parameters from configuration file.
|
||||
Example 1: a simple JSON file for HW device with primary properties.
|
||||
{
|
||||
"CPU": {"NUM_STREAMS": "3", "PERF_COUNT": "NO"}
|
||||
}
|
||||
Example 2: a simple JSON file for meta device(AUTO/MULTI) with HW device properties.
|
||||
{
|
||||
"AUTO": {
|
||||
"PERFORMANCE_HINT": "",
|
||||
"PERF_COUNT": "NO",
|
||||
"DEVICE_PROPERTIES": {
|
||||
"CPU": {
|
||||
"INFERENCE_PRECISION_HINT": "f32",
|
||||
"NUM_STREAMS": "3"
|
||||
},
|
||||
"GPU": {
|
||||
"INFERENCE_PRECISION_HINT": "f32",
|
||||
"NUM_STREAMS": "5"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Running the application with the empty list of options yields the usage message given above and an error message.
|
||||
|
||||
### More information on inputs
|
||||
More information on inputs
|
||||
++++++++++++++++++++++++++
|
||||
|
||||
The benchmark tool supports topologies with one or more inputs. If a topology is not data sensitive, you can skip the input parameter, and the inputs will be filled with random values. If a model has only image input(s), provide a folder with images or a path to an image as input. If a model has some specific input(s) (besides images), please prepare a binary file(s) or numpy array(s) that is filled with data of appropriate precision and provide a path to it as input. If a model has mixed input types, the input folder should contain all required files. Image inputs are filled with image files one by one. Binary inputs are filled with binary inputs one by one.
|
||||
|
||||
## <a name="examples-of-running-the-tool-python"></a> Examples of Running the Tool
|
||||
This section provides step-by-step instructions on how to run the Benchmark Tool with the `asl-recognition` Intel model on CPU or GPU devices. It uses random data as the input.
|
||||
.. _examples-of-running-the-tool-python:
|
||||
|
||||
> **NOTE**: Internet access is required to execute the following steps successfully. If you have access to the Internet through a proxy server only, please make sure that it is configured in your OS environment.
|
||||
Examples of Running the Tool
|
||||
############################
|
||||
|
||||
This section provides step-by-step instructions on how to run the Benchmark Tool with the ``asl-recognition`` Intel model on CPU or GPU devices. It uses random data as the input.
|
||||
|
||||
.. note::
|
||||
|
||||
Internet access is required to execute the following steps successfully. If you have access to the Internet through a proxy server only, please make sure that it is configured in your OS environment.
|
||||
|
||||
1. Install OpenVINO Development Tools (if it hasn't been installed already):
|
||||
```sh
|
||||
pip install openvino-dev
|
||||
```
|
||||
|
||||
2. Download the model using `omz_downloader`, specifying the model name and directory to download the model to:
|
||||
```sh
|
||||
omz_downloader --name asl-recognition-0004 --precisions FP16 --output_dir omz_models
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
pip install openvino-dev
|
||||
|
||||
|
||||
2. Download the model using ``omz_downloader``, specifying the model name and directory to download the model to:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
omz_downloader --name asl-recognition-0004 --precisions FP16 --output_dir omz_models
|
||||
|
||||
|
||||
3. Run the tool, specifying the location of the model .xml file, the device to perform inference on, and with a performance hint. The following commands demonstrate examples of how to run the Benchmark Tool in latency mode on CPU and throughput mode on GPU devices:
|
||||
|
||||
* On CPU (latency mode):
|
||||
```sh
|
||||
benchmark_app -m omz_models/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d CPU -hint latency
|
||||
```
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
benchmark_app -m omz_models/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d CPU -hint latency
|
||||
|
||||
|
||||
* On GPU (throughput mode):
|
||||
```sh
|
||||
benchmark_app -m omz_models/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d GPU -hint throughput
|
||||
```
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
benchmark_app -m omz_models/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d GPU -hint throughput
|
||||
|
||||
|
||||
The application outputs the number of executed iterations, total duration of execution, latency, and throughput.
|
||||
Additionally, if you set the `-report_type` parameter, the application outputs a statistics report. If you set the `-pc` parameter, the application outputs performance counters. If you set `-exec_graph_path`, the application reports executable graph information serialized. All measurements including per-layer PM counters are reported in milliseconds.
|
||||
Additionally, if you set the ``-report_type`` parameter, the application outputs a statistics report. If you set the ``-pc`` parameter, the application outputs performance counters. If you set ``-exec_graph_path``, the application reports executable graph information serialized. All measurements including per-layer PM counters are reported in milliseconds.
|
||||
|
||||
An example of the information output when running benchmark_app on CPU in latency mode is shown below:
|
||||
|
||||
```sh
|
||||
benchmark_app -m omz_models/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d CPU -hint latency
|
||||
```
|
||||
.. code-block:: sh
|
||||
|
||||
benchmark_app -m omz_models/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d CPU -hint latency
|
||||
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
```sh
|
||||
[Step 1/11] Parsing and validating input arguments
|
||||
[ INFO ] Parsing input parameters
|
||||
[ INFO ] Input command: /home/openvino/tools/benchmark_tool/benchmark_app.py -m omz_models/intel/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d CPU -hint latency
|
||||
@@ -384,22 +444,24 @@ An example of the information output when running benchmark_app on CPU in latenc
|
||||
[ INFO ] Min: 20.78 ms
|
||||
[ INFO ] Max: 33.51 ms
|
||||
[ INFO ] Throughput: 89.61 FPS
|
||||
```
|
||||
|
||||
The Benchmark Tool can also be used with dynamically shaped networks to measure expected inference time for various input data shapes. See the -shape and -data_shape argument descriptions in the <a href="#all-configuration-options-python-benchmark">All configuration options</a> section to learn more about using dynamic shapes. Here is a command example for using benchmark_app with dynamic networks and a portion of the resulting output:
|
||||
|
||||
```sh
|
||||
The Benchmark Tool can also be used with dynamically shaped networks to measure expected inference time for various input data shapes. See the ``-shape`` and ``-data_shape`` argument descriptions in the :ref:`All configuration options <all-configuration-options-python-benchmark>` section to learn more about using dynamic shapes. Here is a command example for using benchmark_app with dynamic networks and a portion of the resulting output:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
benchmark_app -m omz_models/intel/asl-recognition-0004/FP16/asl-recognition-0004.xml -d CPU -shape [-1,3,16,224,224] -data_shape [1,3,16,224,224][2,3,16,224,224][4,3,16,224,224] -pcseq
|
||||
```
|
||||
|
||||
```sh
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
[Step 9/11] Creating infer requests and preparing input tensors
|
||||
[ WARNING ] No input files were given for input 'input'!. This input will be filled with random values!
|
||||
[ INFO ] Fill input 'input' with random values
|
||||
[ INFO ] Defined 3 tensor groups:
|
||||
[ INFO ] input: {1, 3, 16, 224, 224}
|
||||
[ INFO ] input: {2, 3, 16, 224, 224}
|
||||
[ INFO ] input: {4, 3, 16, 224, 224}
|
||||
[ WARNING ] No input files were given for input 'input'!. This input will be filled with random values!
|
||||
[ INFO ] Fill input 'input' with random values
|
||||
[ INFO ] Defined 3 tensor groups:
|
||||
[ INFO ] input: {1, 3, 16, 224, 224}
|
||||
[ INFO ] input: {2, 3, 16, 224, 224}
|
||||
[ INFO ] input: {4, 3, 16, 224, 224}
|
||||
[Step 10/11] Measuring performance (Start inference asynchronously, 11 inference requests, limits: 60000 ms duration)
|
||||
[ INFO ] Benchmarking in full mode (inputs filling are included in measurement loop).
|
||||
[ INFO ] First inference took 201.15 ms
|
||||
@@ -428,9 +490,13 @@ The Benchmark Tool can also be used with dynamically shaped networks to measure
|
||||
[ INFO ] Min: 305.51 ms
|
||||
[ INFO ] Max: 773.55 ms
|
||||
[ INFO ] Throughput: 108.82 FPS
|
||||
```
|
||||
|
||||
## See Also
|
||||
* [Using OpenVINO Samples](../../docs/OV_Runtime_UG/Samples_Overview.md)
|
||||
* [Model Optimizer](../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md)
|
||||
* [Model Downloader](@ref omz_tools_downloader)
|
||||
|
||||
See Also
|
||||
####################
|
||||
|
||||
* :doc:`Using OpenVINO Samples <openvino_docs_OV_UG_Samples_Overview>`
|
||||
* :doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`
|
||||
* :doc:`Model Downloader <omz_tools_downloader>`
|
||||
|
||||
@endsphinxdirective
|
||||
Reference in new issue
Block a user