IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
5cee8bbf29797f4544b343e803de957e9f041f92
openvino/docs/benchmarks/performance_benchmarks_openvino.md
Andrey Zaytsev a220a0a7af Feature/azaytsev/docs 2021 4 (#6447) 
* Added benchmark page changes

* Make the picture smaller

* Added Intel® Iris® Xe MAX Graphics

* Changed the TIP about DL WB

* Added Note on the driver for Intel® Iris® Xe MAX Graphics

* Fixed formatting

* Added the link to Intel® software for general purpose GPU capabilities

* OVSA ovsa_get_started updates

* Fixed link
2021-06-29 20:38:51 +03:00

30 KiB
Raw Blame History

Intel® Distribution of OpenVINO™ toolkit Benchmark Results

This benchmark setup includes a single machine on which both the benchmark application and the OpenVINO™ installation reside.

The benchmark application loads the Inference Engine (SW) at run time and executes inferences on the specified hardware inference engine, (CPU, GPU or VPU). The benchmark application measures the time spent on actual inferencing (excluding any pre or post processing) and then reports on the inferences per second (or Frames Per Second). For more information on the benchmark application, please also refer to the entry 5 of the FAQ section.

Devices similar to the ones we have used for benchmarking can be accessed using Intel® DevCloud for the Edge, a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of the OpenVINO™ Toolkit. Learn more or Register here.

Measuring inference performance involves many variables and is extremely use-case and application dependent. We use the below four parameters for measurements, which are key elements to consider for a successful deep learning inference application:

  • Throughput - Measures the number of inferences delivered within a latency threshold. (for example, number of Frames Per Second - FPS). When deploying a system with deep learning inference, select the throughput that delivers the best trade-off between latency and power for the price and performance that meets your requirements.
  • Value - While throughput is important, what is more critical in edge AI deployments is the performance efficiency or performance-per-cost. Application performance in throughput per dollar of system cost is the best measure of value.
  • Efficiency - System power is a key consideration from the edge to the data center. When selecting deep learning solutions, power efficiency (throughput/watt) is a critical factor to consider. Intel designs provide excellent power efficiency for running deep learning workloads.
  • Latency - This measures the synchronous execution of inference requests and is reported in milliseconds. Each inference request (for example: preprocess, infer, postprocess) is allowed to complete before the next is started. This performance metric is relevant in usage scenarios where a single image input needs to be acted upon as soon as possible. An example would be the healthcare sector where medical personnel only request analysis of a single ultra sound scanning image or in real-time or near real-time applications for example an industrial robot's response to actions in its environment or obstacle avoidance for autonomous vehicles.

\htmlonly

<script src="https://cdn.jsdelivr.net/npm/chart.js@2.9.3/dist/Chart.min.js"></script> <script src="https://cdn.jsdelivr.net/npm/chartjs-plugin-datalabels"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/chartjs-plugin-annotation/0.5.7/chartjs-plugin-annotation.min.js"></script> <script src="https://cdn.jsdelivr.net/npm/chartjs-plugin-barchart-background@1.3.0/build/Plugin.Barchart.Background.min.js"></script> <script src="https://cdn.jsdelivr.net/npm/chartjs-plugin-deferred@1"></script> \endhtmlonly

\htmlonly

<script src="bert-large-uncased-whole-word-masking-squad-int8-0001-384-ov-2021-4-569.js" id="bert-large-uncased-whole-word-masking-squad-int8-0001-384-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="deeplabv3-tf-513x513-ov-2021-4-569.js" id="deeplabv3-tf-513x513-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="densenet-121-tf-224x224-ov-2021-4-569.js" id="densenet-121-tf-224x224-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="faster-rcnn-resnet50-coco-tf-600x1024-ov-2021-4-569.js" id="faster-rcnn-resnet50-coco-tf-600x1024-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="inception-v3-tf-299x299-ov-2021-4-569.js" id="inception-v3-tf-299x299-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="inception-v4-tf-299x299-ov-2021-4-569.js" id="inception-v4-tf-299x299-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="mobilenet-ssd-cf-300x300-ov-2021-4-569.js" id="mobilenet-ssd-cf-300x300-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="mobilenet-v2-pytorch-224x224-ov-2021-4-569.js" id="mobilenet-v2-pytorch-224x224-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="resnet-18-pytorch-224x224-ov-2021-4-569.js" id="resnet-18-pytorch-224x224-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="resnet-50-tf-224x224-ov-2021-4-569.js" id="resnet-50-tf-224x224-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="se-resnext-50-cf-224x224-ov-2021-4-569.js" id="se-resnext-50-cf-224x224-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="squeezenet1-1-cf-227x227-ov-2021-4-569.js" id="squeezenet1-1-cf-227x227-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="ssd300-cf-300x300-ov-2021-4-569.js" id="ssd300-cf-300x300-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="yolo-v3-tiny-tf-416x416-ov-2021-4-569.js" id="yolo-v3-tiny-tf-416x416-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="yolo-v4-tf-608x608-ov-2021-4-569.js" id="yolo-v4-tf-608x608-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="unet-camvid-onnx-0001-368x480-ov-2021-4-569.js" id="unet-camvid-onnx-0001-368x480-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="ssd-resnet34-1200-onnx-1200x1200-ov-2021-4-569.js" id="ssd-resnet34-1200-onnx-1200x1200-ov-2021-4-569"></script>

\endhtmlonly

\htmlonly

<script src="vgg19-caffe-224x224-ov-2021-4-569.js" id="vgg19-caffe-224x224-ov-2021-4-569"></script>

\endhtmlonly

Platform Configurations

Intel® Distribution of OpenVINO™ toolkit performance benchmark numbers are based on release 2021.4.

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. Performance results are based on testing as of June 18, 2021 and may not reflect all publicly available updates. See configuration disclosure for details. No product can be absolutely secure.

Performance varies by use, configuration and other factors. Learn more at www.intel.com/PerformanceIndex.

Your costs and results may vary.

© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.

Intel optimizations, for Intel compilers or other products, may not optimize to the same degree for non-Intel products.

Testing by Intel done on: see test date for each HW platform below.

CPU Inference Engines

Intel® Xeon® E-2124G Intel® Xeon® W1290P Intel® Xeon® Silver 4216R
Motherboard ASUS* WS C246 PRO ASUS* WS W480-ACE Intel® Server Board S2600STB
CPU Intel® Xeon® E-2124G CPU @ 3.40GHz Intel® Xeon® W-1290P CPU @ 3.70GHz Intel® Xeon® Silver 4216R CPU @ 2.20GHz
Hyper Threading OFF ON ON
Turbo Setting ON ON ON
Memory 2 x 16 GB DDR4 2666MHz 4 x 16 GB DDR4 @ 2666MHz 12 x 32 GB DDR4 2666MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS
Kernel Version 5.3.0-24-generic 5.3.0-24-generic 5.3.0-24-generic
BIOS Vendor American Megatrends Inc.* American Megatrends Inc. Intel Corporation
BIOS Version 0904 607 SE5C620.86B.02.01.
0013.121520200651
BIOS Release April 12, 2019 May 29, 2020 December 15, 2020
BIOS Settings Select optimized default settings,
save & exit		 Select optimized default settings,
save & exit		 Select optimized default settings,
change power policy
to "performance",
save & exit
Batch size 1 1 1
Precision INT8 INT8 INT8
Number of concurrent inference requests 4 5 32
Test Date June 18, 2021 June 18, 2021 June 18, 2021
Rated maximum TDP/socket in Watt 71 125 125
CPU Price/socket on June 21, 2021, USD
Prices may vary		 213 539 1,002

CPU Inference Engines (continue)

Intel® Xeon® Gold 5218T Intel® Xeon® Platinum 8270 Intel® Xeon® Platinum 8380
Motherboard Intel® Server Board S2600STB Intel® Server Board S2600STB Intel Corporation / WilsonCity
CPU Intel® Xeon® Gold 5218T CPU @ 2.10GHz Intel® Xeon® Platinum 8270 CPU @ 2.70GHz Intel® Xeon® Platinum 8380 CPU @ 2.30GHz
Hyper Threading ON ON ON
Turbo Setting ON ON ON
Memory 12 x 32 GB DDR4 2666MHz 12 x 32 GB DDR4 2933MHz 16 x 16 GB DDR4 3200MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS
Kernel Version 5.3.0-24-generic 5.3.0-24-generic 5.3.0-24-generic
BIOS Vendor Intel Corporation Intel Corporation Intel Corporation
BIOS Version SE5C620.86B.02.01.
0013.121520200651		 SE5C620.86B.02.01.
0013.121520200651		 WLYDCRB1.SYS.0020.
P86.2103050636
BIOS Release December 15, 2020 December 15, 2020 March 5, 2021
BIOS Settings Select optimized default settings,
change power policy to "performance",
save & exit		 Select optimized default settings,
change power policy to "performance",
save & exit		 Select optimized default settings,
change power policy to "performance",
save & exit
Batch size 1 1 1
Precision INT8 INT8 INT8
Number of concurrent inference requests 32 52 80
Test Date June 18, 2021 June 18, 2021 June 18, 2021
Rated maximum TDP/socket in Watt 105 205 270
CPU Price/socket on June 21, 2021, USD
Prices may vary		 1,349 7,405 8,099

CPU Inference Engines (continue)

Intel® Core™ i7-8700T Intel® Core™ i9-10920X
Motherboard GIGABYTE* Z370M DS3H-CF ASUS* PRIME X299-A II
CPU Intel® Core™ i7-8700T CPU @ 2.40GHz Intel® Core™ i9-10920X CPU @ 3.50GHz
Hyper Threading ON ON
Turbo Setting ON ON
Memory 4 x 16 GB DDR4 2400MHz 4 x 16 GB DDR4 2666MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS
Kernel Version 5.3.0-24-generic 5.3.0-24-generic
BIOS Vendor American Megatrends Inc.* American Megatrends Inc.*
BIOS Version F14c 1004
BIOS Release March 23, 2021 March 19, 2021
BIOS Settings Select optimized default settings,
set OS type to "other",
save & exit		 Default Settings
Batch size 1 1
Precision INT8 INT8
Number of concurrent inference requests 4 24
Test Date June 18, 2021 June 18, 2021
Rated maximum TDP/socket in Watt 35 165
CPU Price/socket on June 21, 2021, USD
Prices may vary		 303 700

CPU Inference Engines (continue)

11th Gen Intel® Core™ i7-1185G7 11th Gen Intel® Core™ i7-11850HE
Motherboard Intel Corporation
internal/Reference
Validation Platform		 Intel Corporation
internal/Reference
Validation Platform
CPU 11th Gen Intel® Core™ i7-1185G7 @ 3.00GHz 11th Gen Intel® Core™ i7-11850HE @ 2.60GHz
Hyper Threading ON ON
Turbo Setting ON ON
Memory 2 x 8 GB DDR4 3200MHz 2 x 16 GB DDR4 3200MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04.4 LTS
Kernel Version 5.8.0-05-generic 5.8.0-050800-generic
BIOS Vendor Intel Corporation Intel Corporation
BIOS Version TGLSFWI1.R00.3425.
A00.2010162309		 TGLIFUI1.R00.4064.
A01.2102200132
BIOS Release October 16, 2020 February 20, 2021
BIOS Settings Default Settings Default Settings
Batch size 1 1
Precision INT8 INT8
Number of concurrent inference requests 4 4
Test Date June 18, 2021 June 18, 2021
Rated maximum TDP/socket in Watt 28 45
CPU Price/socket on June 21, 2021, USD
Prices may vary		 426 395

CPU Inference Engines (continue)

Intel® Core™ i3-8100 Intel® Core™ i5-8500 Intel® Core™ i5-10500TE
Motherboard GIGABYTE* Z390 UD ASUS* PRIME Z370-A GIGABYTE* Z490 AORUS PRO AX
CPU Intel® Core™ i3-8100 CPU @ 3.60GHz Intel® Core™ i5-8500 CPU @ 3.00GHz Intel® Core™ i5-10500TE CPU @ 2.30GHz
Hyper Threading OFF OFF ON
Turbo Setting OFF ON ON
Memory 4 x 8 GB DDR4 2400MHz 2 x 16 GB DDR4 2666MHz 2 x 16 GB DDR4 @ 2666MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS
Kernel Version 5.3.0-24-generic 5.3.0-24-generic 5.3.0-24-generic
BIOS Vendor American Megatrends Inc.* American Megatrends Inc.* American Megatrends Inc.*
BIOS Version F8 2401 F3
BIOS Release May 24, 2019 July 12, 2019 March 25, 2020
BIOS Settings Select optimized default settings,
set OS type to "other",
save & exit		 Select optimized default settings,
save & exit		 Select optimized default settings,
set OS type to "other",
save & exit
Batch size 1 1 1
Precision INT8 INT8 INT8
Number of concurrent inference requests 4 3 4
Test Date June 18, 2021 June 18, 2021 June 18, 2021
Rated maximum TDP/socket in Watt 65 65 35
CPU Price/socket on June 21, 2021, USD
Prices may vary		 117 192 195

CPU Inference Engines (continue)

Intel Atom® x5-E3940 Intel Atom® x6425RE Intel® Celeron® 6305E
Motherboard Intel Corporation
internal/Reference
Validation Platform		 Intel Corporation
internal/Reference
Validation Platform		 Intel Corporation
internal/Reference
Validation Platform
CPU Intel Atom® Processor E3940 @ 1.60GHz Intel Atom® x6425RE
Processor @ 1.90GHz		 Intel® Celeron®
6305E @ 1.80GHz
Hyper Threading OFF OFF OFF
Turbo Setting ON ON ON
Memory 1 x 8 GB DDR3 1600MHz 2 x 4GB DDR4 3200MHz 2 x 8 GB DDR4 3200MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS Ubuntu 18.04.5 LTS
Kernel Version 5.3.0-24-generic 5.8.0-050800-generic 5.8.0-050800-generic
BIOS Vendor American Megatrends Inc.* Intel Corporation Intel Corporation
BIOS Version 5.12 EHLSFWI1.R00.2463.
A03.2011200425		 TGLIFUI1.R00.4064.A02.2102260133
BIOS Release September 6, 2017 November 22, 2020 February 26, 2021
BIOS Settings Default settings Default settings Default settings
Batch size 1 1 1
Precision INT8 INT8 INT8
Number of concurrent inference requests 4 4 4
Test Date June 18, 2021 June 18, 2021 June 18, 2021
Rated maximum TDP/socket in Watt 9.5 12 15
CPU Price/socket on June 21, 2021, USD
Prices may vary		 34 59 107

Accelerator Inference Engines

Intel® Neural Compute Stick 2 Intel® Vision Accelerator Design
with Intel® Movidius™ VPUs (Mustang-V100-MX8)
VPU 1 X Intel® Movidius™ Myriad™ X MA2485 8 X Intel® Movidius™ Myriad™ X MA2485
Connection USB 2.0/3.0 PCIe X4
Batch size 1 1
Precision FP16 FP16
Number of concurrent inference requests 4 32
Rated maximum TDP/socket in Watt 2.5 30
CPU Price/socket on June 21, 2021, USD
Prices may vary		 69 425
Host Computer Intel® Core™ i7 Intel® Core™ i5
Motherboard ASUS* Z370-A II Uzelinfo* / US-E1300
CPU Intel® Core™ i7-8700 CPU @ 3.20GHz Intel® Core™ i5-6600 CPU @ 3.30GHz
Hyper Threading ON OFF
Turbo Setting ON ON
Memory 4 x 16 GB DDR4 2666MHz 2 x 16 GB DDR4 2400MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS
Kernel Version 5.0.0-23-generic 5.0.0-23-generic
BIOS Vendor American Megatrends Inc.* American Megatrends Inc.*
BIOS Version 411 5.12
BIOS Release September 21, 2018 September 21, 2018
Test Date June 18, 2021 June 18, 2021

Please follow this link for more detailed configuration descriptions: Configuration Details

\htmlonly

<style> .footer { display: none; } </style>

\endhtmlonly Results may vary. For workloads and configurations visit: [www.intel.com/PerformanceIndex](https://www.intel.com/PerformanceIndex) and [Legal Information](../Legal_Information.md). \htmlonly

\endhtmlonly