openvino/docs/benchmarks/performance_benchmarks_ovms.md

OpenVINO™ Model Server Benchmark Results

OpenVINO™ Model Server is an open-source, production-grade inference platform that exposes a set of models via a convenient inference API over gRPC or HTTP/REST. It employs the inference engine libraries for from the Intel® Distribution of OpenVINO™ toolkit to extend workloads across Intel® hardware including CPU, GPU and others.

Measurement Methodology

OpenVINO™ Model Server is measured in multiple-client-single-server configuration using two hardware platforms connected by ethernet network. The network bandwidth depends on the platforms as well as models under investigation and it is set to not be a bottleneck for workload intensity. This connection is dedicated only to the performance measurements. The benchmark setup is consists of four main parts:

• OpenVINO™ Model Server is launched as a docker container on the server platform and it listens (and answers on) requests from clients. OpenVINO™ Model Server is run on the same machine as the OpenVINO™ toolkit benchmark application in corresponding benchmarking. Models served by OpenVINO™ Model Server are located in a local file system mounted into the docker container. The OpenVINO™ Model Server instance communicates with other components via ports over a dedicated docker network.

• Clients are run in separated physical machine referred to as client platform. Clients are implemented in Python3 programming language based on TensorFlow* API and they work as parallel processes. Each client waits for a response from OpenVINO™ Model Server before it will send a new next request. The role played by the clients is also verification of responses.

• Load balancer works on the client platform in a docker container. HAProxy is used for this purpose. Its main role is counting of requests forwarded from clients to OpenVINO™ Model Server, estimating its latency, and sharing this information by Prometheus service. The reason of locating the load balancer on the client site is to simulate real life scenario that includes impact of physical network on reported metrics.

• Execution Controller is launched on the client platform. It is responsible for synchronization of the whole measurement process, downloading metrics from the load balancer, and presenting the final report of the execution.

resnet-50-TF (INT8)

resnet-50-TF (FP32)

3D U-Net (FP32)

yolo-v3-tf (FP32)

yolo-v3-tiny-tf (FP32)

yolo-v4-tf (FP32)

bert-small-uncased-whole-word-masking-squad-0002 (FP32)

bert-small-uncased-whole-word-masking-squad-int8-0002 (INT8)

bert-large-uncased-whole-word-masking-squad-0001 (FP32)

bert-large-uncased-whole-word-masking-squad-int8-0001 (INT8)

mobilenet-v3-large-1.0-224-tf (FP32)

ssd_mobilenet_v1_coco (FP32)

Platform Configurations

OpenVINO™ Model Server performance benchmark numbers are based on release 2021.4. Performance results are based on testing as of June 17, 2021 and may not reflect all publicly available updates.

Platform with Intel® Xeon® Platinum 8260M

	Server Platform	Client Platform
Motherboard	Inspur YZMB-00882-104 NF5280M5	Intel® Server Board S2600WF H48104-872
Memory	Samsung 16 x 16GB @ 2666 MT/s DDR4	Hynix 16 x 16GB @ 2666 MT/s DDR4
CPU	Intel® Xeon® Platinum 8260M CPU @ 2.40GHz	Intel® Xeon® Gold 6252 CPU @ 2.10GHz
Selected CPU Flags	Hyper Threading, Turbo Boost, DL Boost	Hyper Threading, Turbo Boost, DL Boost
CPU Thermal Design Power	162 W	150 W
Operating System	Ubuntu 20.04.2 LTS	Ubuntu 20.04.2 LTS
Kernel Version	5.4.0-54-generic	5.4.0-65-generic
BIOS Vendor	American Megatrends Inc.	Intel® Corporation
BIOS Version & Release	4.1.16, date: 06/23/2020	SE5C620.86B.02.01, date: 03/26/2020
Docker Version	20.10.3	20.10.3
Network Speed	40 Gb/s

Platform with Intel® Xeon® Gold 6252

	Server Platform	Client Platform
Motherboard	Intel® Server Board S2600WF H48104-872	Inspur YZMB-00882-104 NF5280M5
Memory	Hynix 16 x 16GB @ 2666 MT/s DDR4	Samsung 16 x 16GB @ 2666 MT/s DDR4
CPU	Intel® Xeon® Gold 6252 CPU @ 2.10GHz	Intel® Xeon® Platinum 8260M CPU @ 2.40GHz
Selected CPU Flags	Hyper Threading, Turbo Boost, DL Boost	Hyper Threading, Turbo Boost, DL Boost
CPU Thermal Design Power	150 W	162 W
Operating System	Ubuntu 20.04.2 LTS	Ubuntu 20.04.2 LTS
Kernel Version	5.4.0-65-generic	5.4.0-54-generic
BIOS Vendor	Intel® Corporation	American Megatrends Inc.
BIOS Version and Release Date	SE5C620.86B.02.01, date: 03/26/2020	4.1.16, date: 06/23/2020
Docker Version	20.10.3	20.10.3
Network Speed	40 Gb/s

Platform with Intel® Core™ i9-10920X

	Server Platform	Client Platform
Motherboard	ASUSTeK COMPUTER INC. PRIME X299-A II	ASUSTeK COMPUTER INC. PRIME Z370-P
Memory	Corsair 4 x 16GB @ 2666 MT/s DDR4	Corsair 4 x 16GB @ 2133 MT/s DDR4
CPU	Intel® Core™ i9-10920X CPU @ 3.50GHz	Intel® Core™ i7-8700T CPU @ 2.40GHz
Selected CPU Flags	Hyper Threading, Turbo Boost, DL Boost	Hyper Threading, Turbo Boost
CPU Thermal Design Power	165 W	35 W
Operating System	Ubuntu 20.04.1 LTS	Ubuntu 20.04.1 LTS
Kernel Version	5.4.0-52-generic	5.4.0-56-generic
BIOS Vendor	American Megatrends Inc.	American Megatrends Inc.
BIOS Version and Release Date	0603, date: 03/05/2020	2401, date: 07/15/2019
Docker Version	19.03.13	19.03.14
Network Speed	10 Gb/s

Platform with Intel® Core™ i7-8700T

	Server Platform	Client Platform
Motherboard	ASUSTeK COMPUTER INC. PRIME Z370-P	ASUSTeK COMPUTER INC. PRIME X299-A II
Memory	Corsair 4 x 16GB @ 2133 MT/s DDR4	Corsair 4 x 16GB @ 2666 MT/s DDR4
CPU	Intel® Core™ i7-8700T CPU @ 2.40GHz	Intel® Core™ i9-10920X CPU @ 3.50GHz
Selected CPU Flags	Hyper Threading, Turbo Boost	Hyper Threading, Turbo Boost, DL Boost
CPU Thermal Design Power	35 W	165 W
Operating System	Ubuntu 20.04.1 LTS	Ubuntu 20.04.1 LTS
Kernel Version	5.4.0-56-generic	5.4.0-52-generic
BIOS Vendor	American Megatrends Inc.	American Megatrends Inc.
BIOS Version and Release Date	2401, date: 07/15/2019	0603, date: 03/05/2020
Docker Version	19.03.14	19.03.13
Network Speed	10 Gb/s

Platform with Intel® Core™ i5-8500

	Server Platform	Client Platform
Motherboard	ASUSTeK COMPUTER INC. PRIME Z370-A	Gigabyte Technology Co., Ltd. Z390 UD
Memory	Corsair 2 x 16GB @ 2133 MT/s DDR4	029E 4 x 8GB @ 2400 MT/s DDR4
CPU	Intel® Core™ i5-8500 CPU @ 3.00GHz	Intel® Core™ i3-8100 CPU @ 3.60GHz
Selected CPU Flags	Turbo Boost	-
CPU Thermal Design Power	65 W	65 W
Operating System	Ubuntu 20.04.1 LTS	Ubuntu 20.04.1 LTS
Kernel Version	5.4.0-52-generic	5.4.0-52-generic
BIOS Vendor	American Megatrends Inc.	American Megatrends Inc.
BIOS Version and Release Date	2401, date: 07/12/2019	F10j, date: 09/16/2020
Docker Version	19.03.13	20.10.0
Network Speed	40 Gb/s

Platform with Intel® Core™ i3-8100

	Server Platform	Client Platform
Motherboard	Gigabyte Technology Co., Ltd. Z390 UD	ASUSTeK COMPUTER INC. PRIME Z370-A
Memory	029E 4 x 8GB @ 2400 MT/s DDR4	Corsair 2 x 16GB @ 2133 MT/s DDR4
CPU	Intel® Core™ i3-8100 CPU @ 3.60GHz	Intel® Core™ i5-8500 CPU @ 3.00GHz
Selected CPU Flags	-	Turbo Boost
CPU Thermal Design Power	65 W	65 W
Operating System	Ubuntu 20.04.1 LTS	Ubuntu 20.04.1 LTS
Kernel Version	5.4.0-52-generic	5.4.0-52-generic
BIOS Vendor	American Megatrends Inc.	American Megatrends Inc.
BIOS Version and Release Date	F10j, date: 09/16/2020	2401, date: 07/12/2019
Docker Version	20.10.0	19.03.13
Network Speed	40 Gb/s

\htmlonly

<style> .footer { display: none; } </style>

\endhtmlonly Results may vary. For workloads and configurations visit: [www.intel.com/PerformanceIndex](https://www.intel.com/PerformanceIndex) and [Legal Information](../Legal_Information.md). \htmlonly

\endhtmlonly