openvino/docs/benchmarks/performance_benchmarks_openvino.md

Intel® Distribution of OpenVINO™ toolkit Benchmark Results

This benchmark setup includes a single machine on which both the benchmark application and the OpenVINO™ installation reside.

The benchmark application loads the Inference Engine (SW) at run time and executes inferences on the specified hardware inference engine, (CPU, GPU or VPU). The benchmark application measures the time spent on actual inferencing (excluding any pre or post processing) and then reports on the inferences per second (or Frames Per Second). For more information on the benchmark application, please also refer to the entry 5 of the FAQ section.

Devices similar to the ones we have used for benchmarking can be accessed using Intel® DevCloud for the Edge, a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of the OpenVINO™ Toolkit. Learn more or Register here.

Measuring inference performance involves many variables and is extremely use-case and application dependent. We use the below four parameters for measurements, which are key elements to consider for a successful deep learning inference application:

• Throughput - Measures the number of inferences delivered within a latency threshold. (for example, number of Frames Per Second - FPS). When deploying a system with deep learning inference, select the throughput that delivers the best trade-off between latency and power for the price and performance that meets your requirements.
• Value - While throughput is important, what is more critical in edge AI deployments is the performance efficiency or performance-per-cost. Application performance in throughput per dollar of system cost is the best measure of value.
• Efficiency - System power is a key consideration from the edge to the data center. When selecting deep learning solutions, power efficiency (throughput/watt) is a critical factor to consider. Intel designs provide excellent power efficiency for running deep learning workloads.
• Latency - This measures the synchronous execution of inference requests and is reported in milliseconds. Each inference request (for example: preprocess, infer, postprocess) is allowed to complete before the next is started. This performance metric is relevant in usage scenarios where a single image input needs to be acted upon as soon as possible. An example would be the healthcare sector where medical personnel only request analysis of a single ultra sound scanning image or in real-time or near real-time applications for example an industrial robot's response to actions in its environment or obstacle avoidance for autonomous vehicles.

\htmlonly

<script src="https://cdn.jsdelivr.net/npm/chart.js@2.9.3/dist/Chart.min.js"></script> <script src="https://cdn.jsdelivr.net/npm/chartjs-plugin-datalabels"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/chartjs-plugin-annotation/0.5.7/chartjs-plugin-annotation.min.js"></script> <script src="https://cdn.jsdelivr.net/npm/chartjs-plugin-barchart-background@1.3.0/build/Plugin.Barchart.Background.min.js"></script> <script src="https://cdn.jsdelivr.net/npm/chartjs-plugin-deferred@1"></script> \endhtmlonly

\htmlonly

<script src="bert-large-uncased-whole-word-masking-squad-int8-0001-ov-2021-3-338-5.js" id="bert-large-uncased-whole-word-masking-squad-int8-0001-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="deeplabv3-tf-ov-2021-3-338-5.js" id="deeplabv3-tf-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="densenet-121-tf-ov-2021-3-338-5.js" id="densenet-121-tf-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="faster-rcnn-resnet50-coco-tf-ov-2021-3-338-5.js" id="faster-rcnn-resnet50-coco-tf-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="googlenet-v1-tf-ov-2021-3-338-5.js" id="googlenet-v1-tf-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="inception-v3-tf-ov-2021-3-338-5.js" id="inception-v3-tf-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="mobilenet-ssd-cf-ov-2021-3-338-5.js" id="mobilenet-ssd-cf-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="mobilenet-v1-1-0-224-tf-ov-2021-3-338-5.js" id="mobilenet-v1-1-0-224-tf-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="mobilenet-v2-pytorch-ov-2021-3-338-5.js" id="mobilenet-v2-pytorch-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="resnet-18-pytorch-ov-2021-3-338-5.js" id="resnet-18-pytorch-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="resnet-50-tf-ov-2021-3-338-5.js" id="resnet-50-tf-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="se-resnext-50-cf-ov-2021-3-338-5.js" id="se-resnext-50-cf-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="squeezenet1-1-cf-ov-2021-3-338-5.js" id="squeezenet1-1-cf-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="ssd300-cf-ov-2021-3-338-5.js" id="ssd300-cf-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="yolo-v3-tf-ov-2021-3-338-5.js" id="yolo-v3-tf-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="yolo-v4-tf-ov-2021-3-338-5.js" id="yolo-v4-tf-ov-2021-3-338-5"></script>

\endhtmlonly

\htmlonly

<script src="unet-camvid-onnx-0001-ov-2021-3-338-5.js" id="unet-camvid-onnx-0001-ov-2021-3-338-5"></script>

\endhtmlonly

Platform Configurations

Intel® Distribution of OpenVINO™ toolkit performance benchmark numbers are based on release 2021.3.

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. Performance results are based on testing as of March 15, 2021 and may not reflect all publicly available updates. See configuration disclosure for details. No product can be absolutely secure.

Performance varies by use, configuration and other factors. Learn more at www.intel.com/PerformanceIndex.

Your costs and results may vary.

© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.

Intel optimizations, for Intel compilers or other products, may not optimize to the same degree for non-Intel products.

Testing by Intel done on: see test date for each HW platform below.

CPU Inference Engines

	Intel® Xeon® E-2124G	Intel® Xeon® W1290P	Intel® Xeon® Silver 4216R
Motherboard	ASUS* WS C246 PRO	ASUS* WS W480-ACE	Intel® Server Board S2600STB
CPU	Intel® Xeon® E-2124G CPU @ 3.40GHz	Intel® Xeon® W-1290P CPU @ 3.70GHz	Intel® Xeon® Silver 4216R CPU @ 2.20GHz
Hyper Threading	OFF	ON	ON
Turbo Setting	ON	ON	ON
Memory	2 x 16 GB DDR4 2666MHz	4 x 16 GB DDR4 @ 2666MHz	12 x 32 GB DDR4 2666MHz
Operating System	Ubuntu* 18.04 LTS	Ubuntu* 18.04 LTS	Ubuntu* 18.04 LTS
Kernel Version	5.3.0-24-generic	5.3.0-24-generic	5.3.0-24-generic
BIOS Vendor	American Megatrends Inc.*	American Megatrends Inc.	Intel Corporation
BIOS Version	0904	607	SE5C620.86B.02.01. 0009.092820190230
BIOS Release	April 12, 2019	May 29, 2020	September 28, 2019
BIOS Settings	Select optimized default settings, save & exit	Select optimized default settings, save & exit	Select optimized default settings, change power policy to "performance", save & exit
Batch size	1	1	1
Precision	INT8	INT8	INT8
Number of concurrent inference requests	4	5	32
Test Date	March 15, 2021	March 15, 2021	March 15, 2021
Power dissipation, TDP in Watt	71	125	125
CPU Price on Mach 15th, 2021, USD Prices may vary	213	539	1,002

CPU Inference Engines (continue)

	Intel® Xeon® Gold 5218T	Intel® Xeon® Platinum 8270	Intel® Xeon® Platinum 8380
Motherboard	Intel® Server Board S2600STB	Intel® Server Board S2600STB	Intel Corporation / WilsonCity
CPU	Intel® Xeon® Gold 5218T CPU @ 2.10GHz	Intel® Xeon® Platinum 8270 CPU @ 2.70GHz	Intel® Xeon® Platinum 8380 CPU @ 2.30GHz
Hyper Threading	ON	ON	ON
Turbo Setting	ON	ON	ON
Memory	12 x 32 GB DDR4 2666MHz	12 x 32 GB DDR4 2933MHz	16 x 16 GB DDR4 3200MHz
Operating System	Ubuntu* 18.04 LTS	Ubuntu* 18.04 LTS	Ubuntu* 18.04 LTS
Kernel Version	5.3.0-24-generic	5.3.0-24-generic	5.3.0-24-generic
BIOS Vendor	Intel Corporation	Intel Corporation	Intel Corporation
BIOS Version	SE5C620.86B.02.01. 0009.092820190230	SE5C620.86B.02.01. 0009.092820190230	WLYDCRB1.SYS.0020. P86.2103050636
BIOS Release	September 28, 2019	September 28, 2019	March 5, 2021
BIOS Settings	Select optimized default settings, change power policy to "performance", save & exit	Select optimized default settings, change power policy to "performance", save & exit	Select optimized default settings, change power policy to "performance", save & exit
Batch size	1	1	1
Precision	INT8	INT8	INT8
Number of concurrent inference requests	32	52	80
Test Date	March 15, 2021	March 15, 2021	March 22, 2021
Power dissipation, TDP in Watt	105	205	270
CPU Price, USD Prices may vary	1,349 (on Mach 15th, 2021)	7,405 (on Mach 15th, 2021)	8,099 (on March 26th, 2021)

CPU Inference Engines (continue)

	Intel® Core™ i7-8700T	Intel® Core™ i9-10920X	11th Gen Intel® Core™ i7-1185G7
Motherboard	GIGABYTE* Z370M DS3H-CF	ASUS* PRIME X299-A II	Intel Corporation internal/Reference Validation Platform
CPU	Intel® Core™ i7-8700T CPU @ 2.40GHz	Intel® Core™ i9-10920X CPU @ 3.50GHz	11th Gen Intel® Core™ i7-1185G7 @ 3.00GHz
Hyper Threading	ON	ON	ON
Turbo Setting	ON	ON	ON
Memory	4 x 16 GB DDR4 2400MHz	4 x 16 GB DDR4 2666MHz	2 x 8 GB DDR4 3200MHz
Operating System	Ubuntu* 18.04 LTS	Ubuntu* 18.04 LTS	Ubuntu* 18.04 LTS
Kernel Version	5.3.0-24-generic	5.3.0-24-generic	5.8.0-05-generic
BIOS Vendor	American Megatrends Inc.*	American Megatrends Inc.*	Intel Corporation
BIOS Version	F11	505	TGLSFWI1.R00.3425. A00.2010162309
BIOS Release	March 13, 2019	December 17, 2019	October 16, 2020
BIOS Settings	Select optimized default settings, set OS type to "other", save & exit	Default Settings	Default Settings
Batch size	1	1	1
Precision	INT8	INT8	INT8
Number of concurrent inference requests	4	24	4
Test Date	March 15, 2021	March 15, 2021	March 15, 2021
Power dissipation, TDP in Watt	35	165	28
CPU Price on Mach 15th, 2021, USD Prices may vary	303	700	426

CPU Inference Engines (continue)

	Intel® Core™ i5-8500	Intel® Core™ i5-10500TE
Motherboard	ASUS* PRIME Z370-A	GIGABYTE* Z490 AORUS PRO AX
CPU	Intel® Core™ i5-8500 CPU @ 3.00GHz	Intel® Core™ i5-10500TE CPU @ 2.30GHz
Hyper Threading	OFF	ON
Turbo Setting	ON	ON
Memory	2 x 16 GB DDR4 2666MHz	2 x 16 GB DDR4 @ 2666MHz
Operating System	Ubuntu* 18.04 LTS	Ubuntu* 18.04 LTS
Kernel Version	5.3.0-24-generic	5.3.0-24-generic
BIOS Vendor	American Megatrends Inc.*	American Megatrends Inc.*
BIOS Version	2401	F3
BIOS Release	July 12, 2019	March 25, 2020
BIOS Settings	Select optimized default settings, save & exit	Select optimized default settings, set OS type to "other", save & exit
Batch size	1	1
Precision	INT8	INT8
Number of concurrent inference requests	3	4
Test Date	March 15, 2021	March 15, 2021
Power dissipation, TDP in Watt	65	35
CPU Price on Mach 15th, 2021, USD Prices may vary	192	195

CPU Inference Engines (continue)

	Intel Atom® x5-E3940	Intel Atom® x6425RE	Intel® Core™ i3-8100
Motherboard		Intel Corporation / ElkhartLake LPDDR4x T3 CRB	GIGABYTE* Z390 UD
CPU	Intel Atom® Processor E3940 @ 1.60GHz	Intel Atom® x6425RE Processor @ 1.90GHz	Intel® Core™ i3-8100 CPU @ 3.60GHz
Hyper Threading	OFF	OFF	OFF
Turbo Setting	ON	ON	OFF
Memory	1 x 8 GB DDR3 1600MHz	2 x 4GB DDR4 3200 MHz	4 x 8 GB DDR4 2400MHz
Operating System	Ubuntu* 18.04 LTS	Ubuntu* 18.04 LTS	Ubuntu* 18.04 LTS
Kernel Version	5.3.0-24-generic	5.8.0-050800-generic	5.3.0-24-generic
BIOS Vendor	American Megatrends Inc.*	Intel Corporation	American Megatrends Inc.*
BIOS Version	5.12	EHLSFWI1.R00.2463. A03.2011200425	F8
BIOS Release	September 6, 2017	November 22, 2020	May 24, 2019
BIOS Settings	Default settings	Default settings	Select optimized default settings, set OS type to "other", save & exit
Batch size	1	1	1
Precision	INT8	INT8	INT8
Number of concurrent inference requests	4	4	4
Test Date	March 15, 2021	March 15, 2021	March 15, 2021
Power dissipation, TDP in Watt	9.5	12	65
CPU Price, USD Prices may vary	34 (on March 15th, 2021)	59 (on March 26th, 2021)	117 (on March 15th, 2021)

Accelerator Inference Engines

	Intel® Neural Compute Stick 2	Intel® Vision Accelerator Design with Intel® Movidius™ VPUs (Mustang-V100-MX8)
VPU	1 X Intel® Movidius™ Myriad™ X MA2485	8 X Intel® Movidius™ Myriad™ X MA2485
Connection	USB 2.0/3.0	PCIe X4
Batch size	1	1
Precision	FP16	FP16
Number of concurrent inference requests	4	32
Power dissipation, TDP in Watt	2.5	30
CPU Price, USD Prices may vary	69 (from March 15, 2021)	1180 (from March 15, 2021)
Host Computer	Intel® Core™ i7	Intel® Core™ i5
Motherboard	ASUS* Z370-A II	Uzelinfo* / US-E1300
CPU	Intel® Core™ i7-8700 CPU @ 3.20GHz	Intel® Core™ i5-6600 CPU @ 3.30GHz
Hyper Threading	ON	OFF
Turbo Setting	ON	ON
Memory	4 x 16 GB DDR4 2666MHz	2 x 16 GB DDR4 2400MHz
Operating System	Ubuntu* 18.04 LTS	Ubuntu* 18.04 LTS
Kernel Version	5.0.0-23-generic	5.0.0-23-generic
BIOS Vendor	American Megatrends Inc.*	American Megatrends Inc.*
BIOS Version	411	5.12
BIOS Release	September 21, 2018	September 21, 2018
Test Date	March 15, 2021	March 15, 2021

Please follow this link for more detailed configuration descriptions: Configuration Details

\htmlonly

<style> .footer { display: none; } </style>

\endhtmlonly Results may vary. For workloads and configurations visit: [www.intel.com/PerformanceIndex](https://www.intel.com/PerformanceIndex) and [Legal Information](../Legal_Information.md). \htmlonly

\endhtmlonly