IntenseWebs/openvino
3
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

{DOCS} shift to rst - benchmarks (#16354)

Browse Source
This commit is contained in:
Karol Blaszczak authored and GitHub committed 2023-03-20 19:22:55 +01:00
1 parent d9c70dbce3
commit 4ffecce63f
6 files changed
+210 -285
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/MO_DG/prepare_model/Getting_performance_numbers.md
+117 -58
View File
@@ -1,109 +1,168 @@
# Getting Performance Numbers {#openvino_docs_MO_DG_Getting_Performance_Numbers}
This guide explains how to use the benchmark_app to get performance numbers. It also explains how the performance numbers are reflected through internal inference performance counters and execution graphs. It also includes information on using ITT and Intel® VTune™ Profiler to get performance insights.
## Test performance with the benchmark_app
@sphinxdirective
### Prerequisites
This guide explains how to use the benchmark_app to get performance numbers. It also explains how the performance
numbers are reflected through internal inference performance counters and execution graphs. It also includes
information on using ITT and Intel® VTune™ Profiler to get performance insights.
To run benchmarks, you need both OpenVINO developer tools and Runtime installed. Follow the [Installation guide](../../install_guides/installing-model-dev-tools.md) and make sure to install the latest general release package with support for frameworks of the models you want to test.
Test performance with the benchmark_app
###########################################################
To test performance of your model, make sure you [prepare the model for use with OpenVINO](../../Documentation/model_introduction.md). For example, if you use [OpenVINO's automation tools](@ref omz_tools_downloader), these two lines of code will download the resnet-50-tf and convert it to OpenVINO IR.
Prerequisites
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
To run benchmarks, you need both OpenVINO developer tools and Runtime installed. Follow the
:doc:`Installation guide <openvino_docs_install_guides_install_dev_tools>` and make sure to install the latest
general release package with support for frameworks of the models you want to test.
To test performance of your model, make sure you :doc:`prepare the model for use with OpenVINO <openvino_docs_model_processing_introduction>`.
For example, if you use :doc:`OpenVINO's automation tools <omz_tools_downloader>`, these two lines of code will download the
resnet-50-tf and convert it to OpenVINO IR.
.. code-block:: bash
```bash
omz_downloader --name resnet-50-tf
omz_converter --name resnet-50-tf
```
### Running the benchmark application
For a detailed description, see the dedicated articles: [benchmark_app for C++](../../../samples/cpp/benchmark_app/README.md) and [benchmark_app for Python](../../../tools/benchmark_tool/README.md).
Running the benchmark application
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
For a detailed description, see the dedicated articles:
:doc:`benchmark_app for C++ <openvino_inference_engine_samples_benchmark_app_README>` and
:doc:`benchmark_app for Python <openvino_inference_engine_tools_benchmark_tool_README>`.
The benchmark_app includes a lot of device-specific options, but the primary usage is as simple as:
```bash
.. code-block:: bash
benchmark_app -m <model> -d <device> -i <input>
```
Each of the [OpenVINO supported devices](../../OV_Runtime_UG/supported_plugins/Supported_Devices.md) offers performance settings that contain command-line equivalents in the Benchmark app.
While these settings provide really low-level control for the optimal model performance on the _specific_ device, it is recommended to always start performance evaluation with the [OpenVINO High-Level Performance Hints](../../OV_Runtime_UG/performance_hints.md) first, like so:
Each of the :doc:`OpenVINO supported devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>` offers
performance settings that contain command-line equivalents in the Benchmark app.
While these settings provide really low-level control for the optimal model performance on the *specific* device,
it is recommended to always start performance evaluation with the :doc:`OpenVINO High-Level Performance Hints <openvino_docs_OV_UG_Performance_Hints>` first, like so:
.. code-block:: bash
```bash
# for throughput prioritization
benchmark_app -hint tput -m <model> -d <device>
# for latency prioritization
benchmark_app -hint latency -m <model> -d <device>
```
## Additional benchmarking considerations
### 1 - Select a Proper Set of Operations to Measure
Additional benchmarking considerations
###########################################################
1 - Select a Proper Set of Operations to Measure
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
When evaluating performance of a model with OpenVINO Runtime, it is required to measure a proper set of operations.
- Avoid including one-time costs such as model loading.
- Track operations that occur outside OpenVINO Runtime (such as video decoding) separately.
> **NOTE**: Some image pre-processing can be baked into OpenVINO IR and accelerated accordingly. For more information, refer to [Embedding the Pre-processing](Additional_Optimizations.md) and [General Runtime Optimizations](../../optimization_guide/dldt_deployment_optimization_common.md).
### 2 - Try to Get Credible Data
.. note::
Performance conclusions should be build upon reproducible data. As for the performance measurements, they should be done with a large number of invocations of the same routine. Since the first iteration is almost always significantly slower than the subsequent ones, an aggregated value can be used for the execution time for final projections:
- If the warm-up run does not help or execution time still varies, you can try running a large number of iterations and then average or find a mean of the results.
- If the time values range too much, consider geomean.
- Be aware of the throttling and other power oddities. A device can exist in one of several different power states. When optimizing your model, consider fixing the device frequency for better performance data reproducibility. However, the end-to-end (application) benchmarking should also be performed under real operational conditions.
Some image pre-processing can be baked into OpenVINO IR and accelerated accordingly. For more information,
refer to :doc:`Embedding Pre-processing <openvino_docs_MO_DG_Additional_Optimization_Use_Cases>` and
:doc:`General Runtime Optimizations <openvino_docs_deployment_optimization_guide_common>`.
### 3 - Compare Performance with Native/Framework Code
2 - Try to Get Credible Data
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Performance conclusions should be build upon reproducible data. As for the performance measurements, they should
be done with a large number of invocations of the same routine. Since the first iteration is almost always significantly
slower than the subsequent ones, an aggregated value can be used for the execution time for final projections:
- If the warm-up run does not help or execution time still varies, you can try running a large number of iterations
and then average or find a mean of the results.
- If the time values range too much, consider geomean.
- Be aware of the throttling and other power oddities. A device can exist in one of several different power states.
When optimizing your model, consider fixing the device frequency for better performance data reproducibility.
However, the end-to-end (application) benchmarking should also be performed under real operational conditions.
3 - Compare Performance with Native/Framework Code
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
When comparing the OpenVINO Runtime performance with the framework or another reference code, make sure that both versions are as similar as possible:
- Wrap the exact inference execution (refer to the [Benchmark app](../../../samples/cpp/benchmark_app/README.md) for examples).
- Wrap the exact inference execution (for examples, see :doc:`Benchmark app <openvino_inference_engine_samples_benchmark_app_README>`).
- Do not include model loading time.
- Ensure that the inputs are identical for OpenVINO Runtime and the framework. For example, watch out for random values that can be used to populate the inputs.
- In situations when any user-side pre-processing should be tracked separately, consider [image pre-processing and conversion](../../OV_Runtime_UG/preprocessing_overview.md).
- When applicable, leverage the [Dynamic Shapes support](../../OV_Runtime_UG/ov_dynamic_shapes.md).
- If possible, demand the same accuracy. For example, TensorFlow allows `FP16` execution, so when comparing to that, make sure to test the OpenVINO Runtime with the `FP16` as well.
- In situations when any user-side pre-processing should be tracked separately, consider :doc:`image pre-processing and conversion <openvino_docs_OV_UG_Preprocessing_Overview>`.
- When applicable, leverage the :doc:`Dynamic Shapes support <openvino_docs_OV_UG_DynamicShapes>`.
- If possible, demand the same accuracy. For example, TensorFlow allows ``FP16`` execution, so when comparing to that, make sure to test the OpenVINO Runtime with the ``FP16`` as well.
### Internal Inference Performance Counters and Execution Graphs <a name="performance-counters"></a>
Internal Inference Performance Counters and Execution Graphs
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
More detailed insights into inference performance breakdown can be achieved with device-specific performance counters and/or execution graphs.
Both [C++](../../../samples/cpp/benchmark_app/README.md) and [Python](../../../tools/benchmark_tool/README.md) versions of the `benchmark_app` support a `-pc` command-line parameter that outputs internal execution breakdown.
Both :doc:`C++ <openvino_inference_engine_samples_benchmark_app_README>` and :doc:`Python <openvino_inference_engine_tools_benchmark_tool_README>`
versions of the *benchmark_app* support a ``-pc`` command-line parameter that outputs internal execution breakdown.
For example, the table shown below is part of performance counters for quantized [TensorFlow implementation of ResNet-50](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/resnet-50-tf) model inference on [CPU Plugin](../../OV_Runtime_UG/supported_plugins/CPU.md).
Keep in mind that since the device is CPU, the `realTime` wall clock and the `cpu` time layers are the same. Information about layer precision is also stored in the performance counters.
| layerName | execStatus | layerType | execType | realTime (ms) | cpuTime (ms) |
| --------------------------------------------------------- | ---------- | ------------ | -------------------- | ------------- | ------------ |
| resnet\_model/batch\_normalization\_15/FusedBatchNorm/Add | EXECUTED | Convolution | jit\_avx512\_1x1\_I8 | 0.377 | 0.377 |
| resnet\_model/conv2d\_16/Conv2D/fq\_input\_0 | NOT\_RUN | FakeQuantize | undef | 0 | 0 |
| resnet\_model/batch\_normalization\_16/FusedBatchNorm/Add | EXECUTED | Convolution | jit\_avx512\_I8 | 0.499 | 0.499 |
| resnet\_model/conv2d\_17/Conv2D/fq\_input\_0 | NOT\_RUN | FakeQuantize | undef | 0 | 0 |
| resnet\_model/batch\_normalization\_17/FusedBatchNorm/Add | EXECUTED | Convolution | jit\_avx512\_1x1\_I8 | 0.399 | 0.399 |
| resnet\_model/add\_4/fq\_input\_0 | NOT\_RUN | FakeQuantize | undef | 0 | 0 |
| resnet\_model/add\_4 | NOT\_RUN | Eltwise | undef | 0 | 0 |
| resnet\_model/add\_5/fq\_input\_1 | NOT\_RUN | FakeQuantize | undef | 0 | 0 |
For example, the table shown below is part of performance counters for quantized
`TensorFlow implementation of ResNet-50 <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/resnet-50-tf>`__
model inference on :doc:`CPU Plugin <openvino_docs_OV_UG_supported_plugins_CPU>`.
Keep in mind that since the device is CPU, the ``realTime`` wall clock and the ``cpu`` time layers are the same.
Information about layer precision is also stored in the performance counters.
The `exeStatus` column of the table includes the following possible values:
- `EXECUTED` - the layer was executed by standalone primitive.
- `NOT_RUN` - the layer was not executed by standalone primitive or was fused with another operation and executed in another layer primitive.
The `execType` column of the table includes inference primitives with specific suffixes. The layers could have the following marks:
* The `I8` suffix is for layers that had 8-bit data type input and were computed in 8-bit precision.
* The `FP32` suffix is for layers computed in 32-bit precision.
=========================================================== ============= ============== ===================== ================= ==============
layerName execStatus layerType execType realTime (ms) cpuTime (ms)
=========================================================== ============= ============== ===================== ================= ==============
resnet\_model/batch\_normalization\_15/FusedBatchNorm/Add EXECUTED Convolution jit\_avx512\_1x1\_I8 0.377 0.377
resnet\_model/conv2d\_16/Conv2D/fq\_input\_0 NOT\_RUN FakeQuantize undef 0 0
resnet\_model/batch\_normalization\_16/FusedBatchNorm/Add EXECUTED Convolution jit\_avx512\_I8 0.499 0.499
resnet\_model/conv2d\_17/Conv2D/fq\_input\_0 NOT\_RUN FakeQuantize undef 0 0
resnet\_model/batch\_normalization\_17/FusedBatchNorm/Add EXECUTED Convolution jit\_avx512\_1x1\_I8 0.399 0.399
resnet\_model/add\_4/fq\_input\_0 NOT\_RUN FakeQuantize undef 0 0
resnet\_model/add\_4 NOT\_RUN Eltwise undef 0 0
resnet\_model/add\_5/fq\_input\_1 NOT\_RUN FakeQuantize undef 0 0
=========================================================== ============= ============== ===================== ================= ==============
All `Convolution` layers are executed in `int8` precision. The rest of the layers are fused into Convolutions using post-operation optimization, as described in [CPU Device](../../OV_Runtime_UG/supported_plugins/CPU.md).
This contains layer names (as seen in OpenVINO IR), type of the layer, and execution statistics.
| The ``exeStatus`` column of the table includes the following possible values:
| - ``EXECUTED`` - the layer was executed by standalone primitive.
| - ``NOT_RUN`` - the layer was not executed by standalone primitive or was fused with another operation and executed in another layer primitive.
|
| The ``execType`` column of the table includes inference primitives with specific suffixes. The layers could have the following marks:
| - The ``I8`` suffix is for layers that had 8-bit data type input and were computed in 8-bit precision.
| - The ``FP32`` suffix is for layers computed in 32-bit precision.
|
| All ``Convolution`` layers are executed in ``int8`` precision. The rest of the layers are fused into Convolutions using post-operation optimization,
as described in :doc:`CPU Device <openvino_docs_OV_UG_supported_plugins_CPU>`. This contains layer names
(as seen in OpenVINO IR), type of the layer, and execution statistics.
Both `benchmark_app` versions also support the `exec_graph_path` command-line option. It requires OpenVINO to output the same execution statistics per layer, but in the form of plugin-specific [Netron-viewable](https://netron.app/) graph to the specified file.
Especially when performance-debugging the [latency](../../optimization_guide/dldt_deployment_optimization_latency.md), note that the counters do not reflect the time spent in the `plugin/device/driver/etc` queues. If the sum of the counters is too different from the latency of an inference request, consider testing with less inference requests. For example, running single [OpenVINO stream](../../optimization_guide/dldt_deployment_optimization_tput.md) with multiple requests would produce nearly identical counters as running a single inference request, while the actual latency can be quite different.
Both *benchmark_app* versions also support the ``exec_graph_path`` command-line option. It requires OpenVINO to output the same execution
statistics per layer, but in the form of plugin-specific `Netron-viewable <https://netron.app/>`__ graph to the specified file.
Lastly, the performance statistics with both performance counters and execution graphs are averaged, so such data for the [inputs of dynamic shapes](../../OV_Runtime_UG/ov_dynamic_shapes.md) should be measured carefully, preferably by isolating the specific shape and executing multiple times in a loop, to gather the reliable data.
Especially when performance-debugging the :doc:`latency <openvino_docs_deployment_optimization_guide_latency>`, note that the counters
do not reflect the time spent in the ``plugin/device/driver/etc`` queues. If the sum of the counters is too different from the latency
of an inference request, consider testing with less inference requests. For example, running single
:doc:`OpenVINO stream <openvino_docs_deployment_optimization_guide_tput>` with multiple requests would produce nearly identical
counters as running a single inference request, while the actual latency can be quite different.
Lastly, the performance statistics with both performance counters and execution graphs are averaged,
so such data for the :doc:`inputs of dynamic shapes <openvino_docs_OV_UG_DynamicShapes>` should be measured carefully,
preferably by isolating the specific shape and executing multiple times in a loop, to gather reliable data.
Use ITT to Get Performance Insights
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
In general, OpenVINO and its individual plugins are heavily instrumented with Intel® Instrumentation and Tracing Technology (ITT).
Therefore, you can also compile OpenVINO from the source code with ITT enabled and use tools like
`Intel® VTune™ Profiler <https://software.intel.com/en-us/vtune>`__ to get detailed inference performance breakdown and additional
insights in the application-level performance on the timeline view.
@endsphinxdirective
### Use ITT to Get Performance Insights
In general, OpenVINO and its individual plugins are heavily instrumented with Intel® Instrumentation and Tracing Technology (ITT). Therefore, you can also compile OpenVINO from the source code with ITT enabled and use tools like [Intel® VTune™ Profiler](https://software.intel.com/en-us/vtune) to get detailed inference performance breakdown and additional insights in the application-level performance on the timeline view.
docs/benchmarks/performance_benchmarks.md
+15 -7
View File
@@ -10,17 +10,25 @@
openvino_docs_MO_DG_Getting_Performance_Numbers
@endsphinxdirective
The `Intel® Distribution of OpenVINO™ toolkit <https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html>`__
helps accelerate deep learning inference across a variety of Intel® processors and accelerators.
The [Intel® Distribution of OpenVINO™ toolkit](https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html) helps accelerate deep learning inference across a variety of Intel® processors and accelerators.
The benchmark results below demonstrate high performance gains on several public neural networks on multiple Intel® CPUs, GPUs and GNAs covering a broad performance range. The results may be helpful when deciding which hardware is best for your applications or to plan AI workload on the Intel computing already included in your solutions.
The benchmark results presented here demonstrate high performance gains on several public neural networks on multiple Intel® CPUs,
GPUs, and GNAs covering a broad performance range. The results may be helpful when deciding which hardware is best for your
applications or to plan AI workload on the Intel computing already included in your solutions.
Benchmarks are available for:
* [Intel® Distribution of OpenVINO™ toolkit](performance_benchmarks_openvino.md).
* :doc:`Intel® Distribution of OpenVINO™ toolkit <openvino_docs_performance_benchmarks_openvino>`.
You can also test performance for your system yourself, following the guide on :doc:`getting performance numbers <openvino_docs_MO_DG_Getting_Performance_Numbers>`.
Performance of a particular application can also be evaluated virtually using `Intel® DevCloud for the Edge <https://devcloud.intel.com/edge/>`__.
It is a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of the OpenVINO™ Toolkit.
To learn more about it, visit `the website <https://www.intel.com/content/www/us/en/developer/tools/devcloud/edge/overview.html>`__
or `create an account <https://www.intel.com/content/www/us/en/secure/forms/devcloud-enrollment/account-provisioning.html>`__.
You can also test performance for your system yourself, following the guide on [getting performance numbers](../MO_DG/prepare_model/Getting_performance_numbers.md).
Performance of a particular application can also be evaluated virtually using [Intel® DevCloud for the Edge](https://devcloud.intel.com/edge/). It is a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of the OpenVINO™ Toolkit. To learn more about it, visit [the website](https://www.intel.com/content/www/us/en/developer/tools/devcloud/edge/overview.html) or [create an account](https://www.intel.com/content/www/us/en/forms/idz/devcloud-registration.html?tgt=https://www.intel.com/content/www/us/en/secure/forms/devcloud-enrollment/account-provisioning.html).
@endsphinxdirective
docs/benchmarks/performance_benchmarks_faq.md
+20 -20
View File
@@ -6,7 +6,7 @@
.. dropdown:: How often do performance benchmarks get updated?
New performance benchmarks are typically published on every
`major.minor` release of the Intel® Distribution of OpenVINO™ toolkit.
``major.minor`` release of the Intel® Distribution of OpenVINO™ toolkit.
.. dropdown:: Where can I find the models used in the performance benchmarks?
@@ -22,7 +22,7 @@
All of the performance benchmarks are generated using the
open-source tool within the Intel® Distribution of OpenVINO™ toolkit
called `benchmark_app`. This tool is available
called ``benchmark_app``. This tool is available
:doc:`for C++ apps <openvino_inference_engine_samples_benchmark_app_README>`.
as well as
:doc:`for Python apps <openvino_inference_engine_tools_benchmark_tool_README>`.
@@ -42,63 +42,63 @@
- Public Network
- Task
- Input Size
* - `bert-base-cased <https://github.com/PaddlePaddle/PaddleNLP/tree/v2.1.1>`_
* - `bert-base-cased <https://github.com/PaddlePaddle/PaddleNLP/tree/v2.1.1>`__
- BERT
- question / answer
- 124
* - `bert-large-uncased-whole-word-masking-squad-int8-0001 <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001>`_
* - `bert-large-uncased-whole-word-masking-squad-int8-0001 <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001>`__
- BERT-large
- question / answer
- 384
* - `deeplabv3-TF <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/deeplabv3>`_
* - `deeplabv3-TF <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/deeplabv3>`__
- DeepLab v3 Tf
- semantic segmentation
- 513x513
* - `densenet-121-TF <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/densenet-121-tf>`_
* - `densenet-121-TF <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/densenet-121-tf>`__
- Densenet-121 Tf
- classification
- 224x224
* - `efficientdet-d0 <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/efficientdet-d0-tf>`_
* - `efficientdet-d0 <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/efficientdet-d0-tf>`__
- Efficientdet
- classification
- 512x512
* - `faster_rcnn_resnet50_coco-TF <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/faster_rcnn_resnet50_coco>`_
* - `faster_rcnn_resnet50_coco-TF <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/faster_rcnn_resnet50_coco>`__
- Faster RCNN Tf
- object detection
- 600x1024
* - `inception-v4-TF <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/googlenet-v4-tf>`_
* - `inception-v4-TF <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/googlenet-v4-tf>`__
- Inception v4 Tf (aka GoogleNet-V4)
- classification
- 299x299
* - `mobilenet-ssd-CF <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/mobilenet-ssd>`_
* - `mobilenet-ssd-CF <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/mobilenet-ssd>`__
- SSD (MobileNet)_COCO-2017_Caffe
- object detection
- 300x300
* - `mobilenet-v2-pytorch <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/mobilenet-v2-pytorch>`_
* - `mobilenet-v2-pytorch <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/mobilenet-v2-pytorch>`__
- Mobilenet V2 PyTorch
- classification
- 224x224
* - `resnet-18-pytorch <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/resnet-18-pytorch>`_
* - `resnet-18-pytorch <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/resnet-18-pytorch>`__
- ResNet-18 PyTorch
- classification
- 224x224
* - `resnet-50-TF <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/resnet-50-tf>`_
* - `resnet-50-TF <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/resnet-50-tf>`__
- ResNet-50_v1_ILSVRC-2012
- classification
- 224x224
* - `ssd-resnet34-1200-onnx <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/ssd-resnet34-1200-onnx>`_
* - `ssd-resnet34-1200-onnx <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/ssd-resnet34-1200-onnx>`__
- ssd-resnet34 onnx model
- object detection
- 1200x1200
* - `unet-camvid-onnx-0001 <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/intel/unet-camvid-onnx-0001>`_
* - `unet-camvid-onnx-0001 <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/intel/unet-camvid-onnx-0001>`__
- U-Net
- semantic segmentation
- 368x480
* - `yolo-v3-tiny-tf <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/yolo-v3-tiny-tf>`_
* - `yolo-v3-tiny-tf <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/yolo-v3-tiny-tf>`__
- YOLO v3 Tiny
- object detection
- 416x416
* - `yolo_v4-TF <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/yolo-v4-tf>`_
* - `yolo_v4-TF <https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/yolo-v4-tf>`__
- Yolo-V4 TF
- object detection
- 608x608
@@ -107,16 +107,16 @@
.. dropdown:: Where can I purchase the specific hardware used in the benchmarking?
Intel partners with vendors all over the world. For a list of Hardware Manufacturers, see the
`Intel® AI: In Production Partners & Solutions Catalog <https://www.intel.com/content/www/us/en/internet-of-things/ai-in-production/partners-solutions-catalog.html>`_.
`Intel® AI: In Production Partners & Solutions Catalog <https://www.intel.com/content/www/us/en/internet-of-things/ai-in-production/partners-solutions-catalog.html>`__.
For more details, see the :doc:`Supported Devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>`.
documentation. Before purchasing any hardware, you can test and run
models remotely, using `Intel® DevCloud for the Edge <http://devcloud.intel.com/edge/>`_.
models remotely, using `Intel® DevCloud for the Edge <http://devcloud.intel.com/edge/>`__.
.. dropdown:: How can I optimize my models for better performance or accuracy?
Set of guidelines and recommendations to optimize models are available in the
:doc:`optimization guide <openvino_docs_deployment_optimization_guide_dldt_optimization_guide>`.
Join the conversation in the `Community Forum <https://software.intel.com/en-us/forums/intel-distribution-of-openvino-toolkit>`_ for further support.
Join the conversation in the `Community Forum <https://software.intel.com/en-us/forums/intel-distribution-of-openvino-toolkit>`__ for further support.
.. dropdown:: Why are INT8 optimized models used for benchmarking on CPUs with no VNNI support?
docs/benchmarks/performance_benchmarks_openvino.md
+6 -2
View File
@@ -73,7 +73,7 @@ Intel® Distribution of OpenVINO™ toolkit performance benchmark numbers are ba
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. Performance results are based on testing as of December 13, 2022 and may not reflect all publicly available updates. See configuration disclosure for details. No product can be absolutely secure.
Performance varies by use, configuration and other factors. Learn more at :ref:`www.intel.com/PerformanceIndex<https://www.intel.com/PerformanceIndex>`.
Performance varies by use, configuration and other factors. Learn more at `www.intel.com/PerformanceIndex <https://www.intel.com/PerformanceIndex>`__.
Your costs and results may vary.
@@ -81,4 +81,8 @@ Intel optimizations, for Intel compilers or other products, may not optimize to
© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
@endsphinxdirective
@endsphinxdirective
docs/benchmarks/performance_benchmarks_ovms.md
+46 -196
View File
@@ -3,7 +3,24 @@
@endsphinxdirective
# OpenVINO™ Model Server Benchmark Results {#openvino_docs_performance_benchmarks_ovms}
OpenVINO™ Model Server is an open-source, production-grade inference platform that exposes a set of models via a convenient inference API over gRPC or HTTP/REST. It employs the OpenVINO™ Runtime libraries from the Intel® Distribution of OpenVINO™ toolkit to extend workloads across Intel® hardware including CPU, GPU and others.
@sphinxdirective
Click the "Benchmark Graphs" button to see the OpenVINO™ benchmark graphs. Select the models, the hardware platforms (CPU SKUs),
precision and performance index from the lists and click the “Build Graphs” button.
.. button-link:: #
:class: ov-toolkit-benchmark-results
:color: primary
:outline:
:material-regular:`bar_chart;1.4em` Benchmark Graphs
OpenVINO™ Model Server is an open-source, production-grade inference platform that exposes a set of models via a convenient inference API
over gRPC or HTTP/REST. It employs the OpenVINO™ Runtime libraries from the Intel® Distribution of OpenVINO™ toolkit to extend workloads
across Intel® hardware including CPU, GPU and others.
@endsphinxdirective
![OpenVINO™ Model Server](../img/performance_benchmarks_ovms_01.png)
@@ -21,216 +38,49 @@ OpenVINO™ Model Server is measured in multiple-client-single-server configurat
* **Execution Controller** is launched on the client platform. It is responsible for synchronization of the whole measurement process, downloading metrics from the load balancer, and presenting the final report of the execution.
## bert-small-uncased-whole-word-masking-squad-002 (INT8)
![](../_static/benchmarks_files/ovms/bert-small-uncased-whole-word-masking-squad-002-int8.png)
## bert-small-uncased-whole-word-masking-squad-002 (FP32)
![](../_static/benchmarks_files/ovms/bert-small-uncased-whole-word-masking-squad-002-fp32.png)
## densenet-121 (INT8)
![](../_static/benchmarks_files/ovms/densenet-121-int8.png)
## densenet-121 (FP32)
![](../_static/benchmarks_files/ovms/densenet-121-fp32.png)
## efficientdet-d0 (INT8)
![](../_static/benchmarks_files/ovms/efficientdet-d0-int8.png)
## efficientdet-d0 (FP32)
![](../_static/benchmarks_files/ovms/efficientdet-d0-fp32.png)
## inception-v4 (INT8)
![](../_static/benchmarks_files/ovms/inception-v4-int8.png)
## inception-v4 (FP32)
![](../_static/benchmarks_files/ovms/inception-v4-fp32.png)
## mobilenet-ssd (INT8)
![](../_static/benchmarks_files/ovms/mobilenet-ssd-int8.png)
## mobilenet-ssd (FP32)
![](../_static/benchmarks_files/ovms/mobilenet-ssd-fp32.png)
## mobilenet-v2 (INT8)
![](../_static/benchmarks_files/ovms/mobilenet-v2-int8.png)
## mobilenet-v2 (FP32)
![](../_static/benchmarks_files/ovms/mobilenet-v2-fp32.png)
## resnet-18 (INT8)
![](../_static/benchmarks_files/ovms/resnet-18-int8.png)
## resnet-18 (FP32)
![](../_static/benchmarks_files/ovms/resnet-18-fp32.png)
## resnet-50 (INT8)
![](../_static/benchmarks_files/ovms/resnet-50-int8.png)
## resnet-50 (FP32)
![](../_static/benchmarks_files/ovms/resnet-50-fp32.png)
## ssd-resnt34-1200 (INT8)
![](../_static/benchmarks_files/ovms/ssd-resnt34-1200-int8.png)
## ssd-resnt34-1200 (FP32)
![](../_static/benchmarks_files/ovms/ssd-resnt34-1200-fp32.png)
## unet-camvid-onnx-001 (INT8)
![](../_static/benchmarks_files/ovms/unet-camvid-onnx-001-int8.png)
## unet-camvid-onnx-001 (FP32)
![](../_static/benchmarks_files/ovms/unet-camvid-onnx-001-fp32.png)
## yolo-v3-tiny (INT8)
![](../_static/benchmarks_files/ovms/yolo-v3-tiny-int8.png)
## yolo-v3-tiny (FP32)
![](../_static/benchmarks_files/ovms/yolo-v3-tiny-fp32.png)
## yolo-v4 (INT8)
![](../_static/benchmarks_files/ovms/yolo-v4-int8.png)
## yolo-v4 (FP32)
![](../_static/benchmarks_files/ovms/yolo-v4-fp32.png)
## Platform Configurations
OpenVINO™ Model Server performance benchmark numbers are based on release 2022.2. Performance results are based on testing as of November 16, 2022 and may not reflect all publicly available updates.
@sphinxdirective
.. dropdown:: Platform with Intel® Xeon® Platinum 8260M
.. table::
:widths: 25 25 50
+--------------------------+-------------------------------------------+----------------------------------------+
| | Server Platform | Client Platform |
+==========================+===========================================+========================================+
| Motherboard | Inspur YZMB-00882-104 NF5280M5 | Inspur YZMB-00882-104 NF5280M5 |
+--------------------------+-------------------------------------------+----------------------------------------+
| Memory | Samsung 16 x 16GB @ 2666 MT/s DDR4 | Kingston 16 x 16GB @ 2666 MT/s DDR4 |
+--------------------------+-------------------------------------------+----------------------------------------+
| CPU | Intel® Xeon® Platinum 8260M CPU @ 2.40GHz | Intel® Xeon® Gold 6238M CPU @ 2.10GHz |
+--------------------------+-------------------------------------------+----------------------------------------+
| Selected CPU Flags | Hyper Threading, Turbo Boost, DL Boost | Hyper Threading, Turbo Boost, DL Boost |
+--------------------------+-------------------------------------------+----------------------------------------+
| CPU Thermal Design Power | 162W | 150W |
+--------------------------+-------------------------------------------+----------------------------------------+
| Operating System | Ubuntu 20.04.4 LTS | Ubuntu 20.04.4 LTS |
+--------------------------+-------------------------------------------+----------------------------------------+
| Kernel Version | 5.4.0-107-generic | 5.4.0-107-generic |
+--------------------------+-------------------------------------------+----------------------------------------+
| BIOS Vendor | American Megatrends Inc. | AMI |
+--------------------------+-------------------------------------------+----------------------------------------+
| BIOS Version & Release | 4.1.16; date: 06/23/2020 | 4.1.16; date: 06/23/2020 |
+--------------------------+-------------------------------------------+----------------------------------------+
| Docker Version | 20.10.3 | 20.10.3 |
+--------------------------+-------------------------------------------+----------------------------------------+
| Network Speed | 40 Gb/s | 40 Gb/s |
+--------------------------+-------------------------------------------+----------------------------------------+
.. dropdown:: Platform with 6238M
Platform & Configurations
####################################
.. table::
:widths: 25 25 50
For a listing of all platforms and configurations used for testing, refer to the following:
+--------------------------+-------------------------------------------+--------------------------------------------+
| | Server Platform | Client Platform |
+==========================+===========================================+============================================+
| Motherboard | Inspur YZMB-00882-104 NF5280M5 | Inspur YZMB-00882-104 NF5280M5 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Memory | Kingston 16 x 16GB @ 2666 MT/s DDR4 | Samsung 16 x 16GB @ 2666 MT/s DDR4 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU | Intel® Xeon® Gold 6238M CPU @ 2.10GHz | Intel® Xeon® Platinum 8260M CPU @ 2.40GHz |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Selected CPU Flags | Hyper Threading, Turbo Boost, DL Boost | Hyper Threading, Turbo Boost, DL Boost |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU Thermal Design Power | 150W | 162W |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Operating System | Ubuntu 20.04.4 LTS | Ubuntu 20.04.4 LTS |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Kernel Version | 5.4.0-107-generic | 5.4.0-107-generic |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Vendor | AMI | American Megatrends Inc. |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Version & Release | 4.1.16; date: 06/23/2020 | 4.1.16; date: 06/23/2020 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Docker Version | 20.10.3 | 20.10.3 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Network Speed | 40 Gb/s | 40 Gb/s |
+--------------------------+-------------------------------------------+--------------------------------------------+
.. button-link:: _static/benchmarks_files/platform_list_22.3.pdf
:color: primary
:outline:
.. dropdown:: Platform with Intel® Core™ i9-10920X
:material-regular:`download;1.5em` Click for Hardware Platforms [PDF]
.. table::
:widths: 25 25 50
.. button-link:: _static/benchmarks_files/OV-2022.3-system-info-detailed.xlsx
:color: primary
:outline:
+--------------------------+-------------------------------------------+--------------------------------------------+
| | Server Platform | Client Platform |
+==========================+===========================================+============================================+
| Motherboard | ASUSTeK COMPUTER INC. PRIME X299-A II | ASUSTeK COMPUTER INC. PRIME Z370-P |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Memory | Corsair 4 x 16GB @ 2666 MT/s DDR4 | Corsair 4 x 16GB @ 2133 MT/s DDR4 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU | Intel® Core™ i9-10920X CPU @ 3.50GHz | Intel® Core™ i7-8700T CPU @ 2.40GHz |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Selected CPU Flags | Hyper Threading, Turbo Boost, DL Boost | Hyper Threading, Turbo Boost, DL Boost |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU Thermal Design Power | 165W | 35 W |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Operating System | Ubuntu 20.04.4 LTS | Ubuntu 20.04.4 LTS |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Kernel Version | 5.4.0-107-generic | 5.4.0-107-generic |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Vendor | American Megatrends Inc. | American Megatrends Inc. |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Version & Release | 0702; date: 06/10/2020 | 2401; date: 07/15/2019 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Docker Version | 19.03.13 | 19.03.14 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Network Speed | 10 Gb/s | 10 Gb/s |
+--------------------------+-------------------------------------------+--------------------------------------------+
:material-regular:`download;1.5em` Click for Configuration Details [XLSX]
.. dropdown:: Platform with Intel® Core™ i7-8700T
.. the files above need to be changed to the proper ones!!!
.. table::
:widths: 25 25 50
The presented performance benchmark numbers are based on the release 2022.2 of the Intel® Distribution of OpenVINO™ toolkit.
The benchmark application loads the OpenVINO™ Runtime and executes inferences on the specified hardware (CPU, GPU or GNA).
It measures the time spent on actual inference (excluding any pre or post processing) and then reports on the inferences per second (or Frames Per Second).
+--------------------------+-------------------------------------------+--------------------------------------------+
| | Server Platform | Client Platform |
+==========================+===========================================+============================================+
| Motherboard | ASUSTeK COMPUTER INC. PRIME Z370-P | ASUSTeK COMPUTER INC. PRIME X299-A II |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Memory | Corsair 4 x 16GB @ 2133 MT/s DDR4 | Corsair 4 x 16GB @ 2666 MT/s DDR4 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU | Intel® Core™ i7-8700T CPU @ 2.40GHz | Intel® Core™ i9-10920X CPU @ 3.50GHz |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Selected CPU Flags | Hyper Threading, Turbo Boost | Hyper Threading, Turbo Boost |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU Thermal Design Power | 35W | 165 W |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Operating System | Ubuntu 20.04.4 LTS | Ubuntu 20.04.4 LTS |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Kernel Version | 5.4.0-107-generic | 5.4.0-107-generic |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Vendor | American Megatrends Inc. | American Megatrends Inc. |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Version & Release | 2401; date: 07/15/2019 | 0702; date: 06/10/2020 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Docker Version | 19.03.14 | 19.03.13 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Network Speed | 10 Gb/s | 10 Gb/s |
+--------------------------+-------------------------------------------+--------------------------------------------+
Disclaimers
####################################
.. dropdown:: Platform with Intel® Core™ i5-8500
Intel® Distribution of OpenVINO™ toolkit performance benchmark numbers are based on release 2022.3.
.. table::
:widths: 25 25 50
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. Performance results are based on testing as of November 16, 2022 and may not reflect all publicly available updates. See configuration disclosure for details. No product can be absolutely secure.
+--------------------------+-------------------------------------------+--------------------------------------------+
| | Server Platform | Client Platform |
+==========================+===========================================+============================================+
| Motherboard | ASUSTeK COMPUTER INC. PRIME Z370-A | Gigabyte Technology Co., Ltd. Z390 UD |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Memory | Corsair 2 x 16GB @ 2133 MT/s DDR4 | 029E 4 x 8GB @ 2400 MT/s DDR4 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU | Intel® Core™ i5-8500 CPU @ 3.00GHz | Intel® Core™ i3-8100 CPU @ 3.60GHz |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Selected CPU Flags | Turbo Boost | |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU Thermal Design Power | 65W | 65 W |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Operating System | Ubuntu 20.04.4 LTS | Ubuntu 20.04.1 LTS |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Kernel Version | 5.4.0-113-generic | 5.4.0-52-generic |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Vendor | American Megatrends Inc. | American Megatrends Inc. |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Version & Release | 3004; date: 07/12/2021 | F10j; date: 09/16/2020 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Docker Version | 19.03.13 | 20.10.0 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Network Speed | 40 Gb/s | 40 Gb/s |
+--------------------------+-------------------------------------------+--------------------------------------------+
Performance varies by use, configuration and other factors. Learn more at `www.intel.com/PerformanceIndex <https://www.intel.com/PerformanceIndex>`__.
@endsphinxdirective
Your costs and results may vary.
Intel optimizations, for Intel compilers or other products, may not optimize to the same degree for non-Intel products.
© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
@endsphinxdirective
docs/benchmarks/performance_int8_vs_fp32.md
+6 -2
View File
@@ -1,12 +1,16 @@
# Model Accuracy {#openvino_docs_performance_int8_vs_fp32}
The following table presents the absolute accuracy drop calculated as the accuracy difference between FP32 and INT8 representations of a model on two platforms
@sphinxdirective
The following table presents the absolute accuracy drop calculated as the accuracy difference
between FP32 and INT8 representations of a model on two platforms.
* A - Intel® Core™ i9-9000K (AVX2)
* B - Intel® Xeon® 6338, (VNNI)
* C - Intel® Flex-170
@sphinxdirective
.. list-table:: Model Accuracy
:header-rows: 1