IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

reintroduce benchmarks for ovms (#14403)

Browse Source 
port from: #14271
Recreate performance_benchmarks_ovms.md and add it to TOC of performance_benchmarks.md
This commit is contained in:
Karol Blaszczak
2022-12-06 08:03:40 +01:00
committed by GitHub
parent d0a6122cb2
commit d47033cc05
26 changed files with 306 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docs/_static/benchmarks_files/ovms/bert-small-uncased-whole-word-masking-squad-002-fp32.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bfb9d6bb2a949e032d2859fbb3ac6dd6e0ddec6888ad5473efedacb62399865f
size 65516

3
docs/_static/benchmarks_files/ovms/bert-small-uncased-whole-word-masking-squad-002-int8.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8a271efea7aa7869dd22d0f27fe6b5b3e22f6a6adff623e34d27d45d8a4e6b9f
size 68495

3
docs/_static/benchmarks_files/ovms/densenet-121-fp32.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7d95ffb2fd00e54c5723e78a7c75910f580e9c8f014f130c8f074e36a3c1fc4d
size 63452

3
docs/_static/benchmarks_files/ovms/densenet-121-int8.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c769922f256b4393e6d1ad1462004481ee717aa20ee400f6bc3fb33c15a36328
size 64067

3
docs/_static/benchmarks_files/ovms/efficientdet-d0-fp32.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b8febb756ef218f95a50633710564754cd63b4d62816327b0800a3fdeb1be5c8
size 62037

3
docs/_static/benchmarks_files/ovms/efficientdet-d0-int8.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e31681db24b0cf4ca9c5bd5c105119f0eef1d361b54cc881d526936fa3f33690
size 62771

3
docs/_static/benchmarks_files/ovms/inception-v4-fp32.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:718720b9501390345d61d0122eaf9d61f9eec657a755e026e3e7541145d0c40b
size 61301

3
docs/_static/benchmarks_files/ovms/inception-v4-int8.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:02339bef84fff6e7701cb8b55cb793add6ca8edc676ac077ae6e1a7e6878c85e
size 64581

3
docs/_static/benchmarks_files/ovms/mobilenet-ssd-fp32.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d28a032b0f109e21d341e2602d0b5783ee98c04cdd908ff40a8c72bd7729b2fc
size 65515

3
docs/_static/benchmarks_files/ovms/mobilenet-ssd-int8.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:dd7019cb1ca437369e959d9a360e05aaa9f65e228481f9846b85c06870dbeb3e
size 63941

3
docs/_static/benchmarks_files/ovms/mobilenet-v2-fp32.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:546acb8af30bd20dc29b0fe69cc2f43b04d4a682bb85c1ee2fc8e6ae55f4ceb3
size 64632

3
docs/_static/benchmarks_files/ovms/mobilenet-v2-int8.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:81b7b35cccff8cb61a64792ed18cbfd730867326411b17861bcd2001e95d42bd
size 67176

3
docs/_static/benchmarks_files/ovms/resnet-18-fp32.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0aa6718e0175f66d2beef755d388a40c39eac2488ccabc6d5f475059d0681655
size 62133

3
docs/_static/benchmarks_files/ovms/resnet-18-int8.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8a72c3ab5046ff588a3d26e464be39b2476e2043071060791dc901738e43e541
size 63066

3
docs/_static/benchmarks_files/ovms/resnet-50-fp32.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:26e5544cbf89d8a85fd2b10363cef1702c3c7acae2a448a2c25a0684a27e171c
size 61061

3
docs/_static/benchmarks_files/ovms/resnet-50-int8.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:948a4baf58258766b189f0eb50691697a6a0034336726d8156a84056ca34a9d7
size 64227

3
docs/_static/benchmarks_files/ovms/ssd-resnt34-1200-fp32.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:91b78d61c8c888a79528fb51b593acb1857abc2ba4a5d805b1c320d5a54d1c16
size 58144

3
docs/_static/benchmarks_files/ovms/ssd-resnt34-1200-int8.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fc23608cfdd7135619a6dbefe72670867707bdba31e42375fde3fbd752623df1
size 62003

3
docs/_static/benchmarks_files/ovms/unet-camvid-onnx-001-fp32.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1bf6eee809f02a4988521441c2d5e1b224a618477af28098d689dad9f798abbb
size 61762

3
docs/_static/benchmarks_files/ovms/unet-camvid-onnx-001-int8.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7bc416a48800ecaed6e1255de406c446064454e7892f31be88385d3c20ca002b
size 61040

3
docs/_static/benchmarks_files/ovms/yolo-v3-tiny-fp32.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:90be42f47347d8759cdb14024d56cc16db7b3d65454e01db4d16d2902c8b01e8
size 63073

3
docs/_static/benchmarks_files/ovms/yolo-v3-tiny-int8.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:534524ed6aa12330aa2d166eb72af38a37773de99de2c4f93de5c184a1107b13
size 65611

3
docs/_static/benchmarks_files/ovms/yolo-v4-fp32.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c29488cc3a55adc6fc1e5eddc19565179f10cb8bee217093a5833d45d7f11a55
size 58381

3
docs/_static/benchmarks_files/ovms/yolo-v4-int8.png vendored Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4f6699d812099a41e60e6d27f1ea241f1c6f75a386fb9464219285c7dfc26b89
size 61751

2
docs/benchmarks/performance_benchmarks.md
View File

@@ -7,7 +7,7 @@
:hidden:
openvino_docs_performance_benchmarks_openvino
openvino_docs_performance_benchmarks_ovms
@endsphinxdirective

233
docs/benchmarks/performance_benchmarks_ovms.md Normal file
View File

@@ -0,0 +1,233 @@
# OpenVINO™ Model Server Benchmark Results {#openvino_docs_performance_benchmarks_ovms}
OpenVINO™ Model Server is an open-source, production-grade inference platform that exposes a set of models via a convenient inference API over gRPC or HTTP/REST. It employs the OpenVINO™ Runtime libraries from the Intel® Distribution of OpenVINO™ toolkit to extend workloads across Intel® hardware including CPU, GPU and others.
![OpenVINO™ Model Server](../img/performance_benchmarks_ovms_01.png)
## Measurement Methodology
OpenVINO™ Model Server is measured in multiple-client-single-server configuration using two hardware platforms connected by ethernet network. The network bandwidth depends on the platforms as well as models under investigation and it is set to not be a bottleneck for workload intensity. This connection is dedicated only to the performance measurements. The benchmark setup is consists of four main parts:
![OVMS Benchmark Setup Diagram](../img/performance_benchmarks_ovms_02.png)
* **OpenVINO™ Model Server** is launched as a docker container on the server platform and it listens (and answers on) requests from clients. OpenVINO™ Model Server is run on the same machine as the OpenVINO™ toolkit benchmark application in corresponding benchmarking. Models served by OpenVINO™ Model Server are located in a local file system mounted into the docker container. The OpenVINO™ Model Server instance communicates with other components via ports over a dedicated docker network.
* **Clients** are run in separated physical machine referred to as client platform. Clients are implemented in Python3 programming language based on TensorFlow* API and they work as parallel processes. Each client waits for a response from OpenVINO™ Model Server before it will send a new next request. The role played by the clients is also verification of responses.
* **Load balancer** works on the client platform in a docker container. HAProxy is used for this purpose. Its main role is counting of requests forwarded from clients to OpenVINO™ Model Server, estimating its latency, and sharing this information by Prometheus service. The reason of locating the load balancer on the client site is to simulate real life scenario that includes impact of physical network on reported metrics.
* **Execution Controller** is launched on the client platform. It is responsible for synchronization of the whole measurement process, downloading metrics from the load balancer, and presenting the final report of the execution.
## bert-small-uncased-whole-word-masking-squad-002 (INT8)
![](../_static/benchmarks_files/ovms/bert-small-uncased-whole-word-masking-squad-002-int8.png)
## bert-small-uncased-whole-word-masking-squad-002 (FP32)
![](../_static/benchmarks_files/ovms/bert-small-uncased-whole-word-masking-squad-002-fp32.png)
## densenet-121 (INT8)
![](../_static/benchmarks_files/ovms/densenet-121-int8.png)
## densenet-121 (FP32)
![](../_static/benchmarks_files/ovms/densenet-121-fp32.png)
## efficientdet-d0 (INT8)
![](../_static/benchmarks_files/ovms/efficientdet-d0-int8.png)
## efficientdet-d0 (FP32)
![](../_static/benchmarks_files/ovms/efficientdet-d0-fp32.png)
## inception-v4 (INT8)
![](../_static/benchmarks_files/ovms/inception-v4-int8.png)
## inception-v4 (FP32)
![](../_static/benchmarks_files/ovms/inception-v4-fp32.png)
## mobilenet-ssd (INT8)
![](../_static/benchmarks_files/ovms/mobilenet-ssd-int8.png)
## mobilenet-ssd (FP32)
![](../_static/benchmarks_files/ovms/mobilenet-ssd-fp32.png)
## mobilenet-v2 (INT8)
![](../_static/benchmarks_files/ovms/mobilenet-v2-int8.png)
## mobilenet-v2 (FP32)
![](../_static/benchmarks_files/ovms/mobilenet-v2-fp32.png)
## resnet-18 (INT8)
![](../_static/benchmarks_files/ovms/resnet-18-int8.png)
## resnet-18 (FP32)
![](../_static/benchmarks_files/ovms/resnet-18-fp32.png)
## resnet-50 (INT8)
![](../_static/benchmarks_files/ovms/resnet-50-int8.png)
## resnet-50 (FP32)
![](../_static/benchmarks_files/ovms/resnet-50-fp32.png)
## ssd-resnt34-1200 (INT8)
![](../_static/benchmarks_files/ovms/ssd-resnt34-1200-int8.png)
## ssd-resnt34-1200 (FP32)
![](../_static/benchmarks_files/ovms/ssd-resnt34-1200-fp32.png)
## unet-camvid-onnx-001 (INT8)
![](../_static/benchmarks_files/ovms/unet-camvid-onnx-001-int8.png)
## unet-camvid-onnx-001 (FP32)
![](../_static/benchmarks_files/ovms/unet-camvid-onnx-001-fp32.png)
## yolo-v3-tiny (INT8)
![](../_static/benchmarks_files/ovms/yolo-v3-tiny-int8.png)
## yolo-v3-tiny (FP32)
![](../_static/benchmarks_files/ovms/yolo-v3-tiny-fp32.png)
## yolo-v4 (INT8)
![](../_static/benchmarks_files/ovms/yolo-v4-int8.png)
## yolo-v4 (FP32)
![](../_static/benchmarks_files/ovms/yolo-v4-fp32.png)
## Platform Configurations
OpenVINO™ Model Server performance benchmark numbers are based on release 2022.2. Performance results are based on testing as of November 16, 2022 and may not reflect all publicly available updates.
@sphinxdirective
.. dropdown:: Platform with Intel® Xeon® Platinum 8260M
.. table::
:widths: 25 25 50
+--------------------------+-------------------------------------------+----------------------------------------+
| | Server Platform | Client Platform |
+==========================+===========================================+========================================+
| Motherboard | Inspur YZMB-00882-104 NF5280M5 | Inspur YZMB-00882-104 NF5280M5 |
+--------------------------+-------------------------------------------+----------------------------------------+
| Memory | Samsung 16 x 16GB @ 2666 MT/s DDR4 | Kingston 16 x 16GB @ 2666 MT/s DDR4 |
+--------------------------+-------------------------------------------+----------------------------------------+
| CPU | Intel® Xeon® Platinum 8260M CPU @ 2.40GHz | Intel® Xeon® Gold 6238M CPU @ 2.10GHz |
+--------------------------+-------------------------------------------+----------------------------------------+
| Selected CPU Flags | Hyper Threading, Turbo Boost, DL Boost | Hyper Threading, Turbo Boost, DL Boost |
+--------------------------+-------------------------------------------+----------------------------------------+
| CPU Thermal Design Power | 162W | 150W |
+--------------------------+-------------------------------------------+----------------------------------------+
| Operating System | Ubuntu 20.04.4 LTS | Ubuntu 20.04.4 LTS |
+--------------------------+-------------------------------------------+----------------------------------------+
| Kernel Version | 5.4.0-107-generic | 5.4.0-107-generic |
+--------------------------+-------------------------------------------+----------------------------------------+
| BIOS Vendor | American Megatrends Inc. | AMI |
+--------------------------+-------------------------------------------+----------------------------------------+
| BIOS Version & Release | 4.1.16; date: 06/23/2020 | 4.1.16; date: 06/23/2020 |
+--------------------------+-------------------------------------------+----------------------------------------+
| Docker Version | 20.10.3 | 20.10.3 |
+--------------------------+-------------------------------------------+----------------------------------------+
| Network Speed | 40 Gb/s | 40 Gb/s |
+--------------------------+-------------------------------------------+----------------------------------------+
.. dropdown:: Platform with 6238M
.. table::
:widths: 25 25 50
+--------------------------+-------------------------------------------+--------------------------------------------+
| | Server Platform | Client Platform |
+==========================+===========================================+============================================+
| Motherboard | Inspur YZMB-00882-104 NF5280M5 | Inspur YZMB-00882-104 NF5280M5 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Memory | Kingston 16 x 16GB @ 2666 MT/s DDR4 | Samsung 16 x 16GB @ 2666 MT/s DDR4 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU | Intel® Xeon® Gold 6238M CPU @ 2.10GHz | Intel® Xeon® Platinum 8260M CPU @ 2.40GHz |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Selected CPU Flags | Hyper Threading, Turbo Boost, DL Boost | Hyper Threading, Turbo Boost, DL Boost |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU Thermal Design Power | 150W | 162W |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Operating System | Ubuntu 20.04.4 LTS | Ubuntu 20.04.4 LTS |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Kernel Version | 5.4.0-107-generic | 5.4.0-107-generic |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Vendor | AMI | American Megatrends Inc. |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Version & Release | 4.1.16; date: 06/23/2020 | 4.1.16; date: 06/23/2020 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Docker Version | 20.10.3 | 20.10.3 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Network Speed | 40 Gb/s | 40 Gb/s |
+--------------------------+-------------------------------------------+--------------------------------------------+
.. dropdown:: Platform with Intel® Core™ i9-10920X
.. table::
:widths: 25 25 50
+--------------------------+-------------------------------------------+--------------------------------------------+
| | Server Platform | Client Platform |
+==========================+===========================================+============================================+
| Motherboard | ASUSTeK COMPUTER INC. PRIME X299-A II | ASUSTeK COMPUTER INC. PRIME Z370-P |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Memory | Corsair 4 x 16GB @ 2666 MT/s DDR4 | Corsair 4 x 16GB @ 2133 MT/s DDR4 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU | Intel® Core™ i9-10920X CPU @ 3.50GHz | Intel® Core™ i7-8700T CPU @ 2.40GHz |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Selected CPU Flags | Hyper Threading, Turbo Boost, DL Boost | Hyper Threading, Turbo Boost, DL Boost |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU Thermal Design Power | 165W | 35 W |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Operating System | Ubuntu 20.04.4 LTS | Ubuntu 20.04.4 LTS |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Kernel Version | 5.4.0-107-generic | 5.4.0-107-generic |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Vendor | American Megatrends Inc. | American Megatrends Inc. |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Version & Release | 0702; date: 06/10/2020 | 2401; date: 07/15/2019 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Docker Version | 19.03.13 | 19.03.14 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Network Speed | 10 Gb/s | 10 Gb/s |
+--------------------------+-------------------------------------------+--------------------------------------------+
.. dropdown:: Platform with Intel® Core™ i7-8700T
.. table::
:widths: 25 25 50
+--------------------------+-------------------------------------------+--------------------------------------------+
| | Server Platform | Client Platform |
+==========================+===========================================+============================================+
| Motherboard | ASUSTeK COMPUTER INC. PRIME Z370-P | ASUSTeK COMPUTER INC. PRIME X299-A II |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Memory | Corsair 4 x 16GB @ 2133 MT/s DDR4 | Corsair 4 x 16GB @ 2666 MT/s DDR4 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU | Intel® Core™ i7-8700T CPU @ 2.40GHz | Intel® Core™ i9-10920X CPU @ 3.50GHz |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Selected CPU Flags | Hyper Threading, Turbo Boost | Hyper Threading, Turbo Boost |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU Thermal Design Power | 35W | 165 W |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Operating System | Ubuntu 20.04.4 LTS | Ubuntu 20.04.4 LTS |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Kernel Version | 5.4.0-107-generic | 5.4.0-107-generic |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Vendor | American Megatrends Inc. | American Megatrends Inc. |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Version & Release | 2401; date: 07/15/2019 | 0702; date: 06/10/2020 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Docker Version | 19.03.14 | 19.03.13 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Network Speed | 10 Gb/s | 10 Gb/s |
+--------------------------+-------------------------------------------+--------------------------------------------+
.. dropdown:: Platform with Intel® Core™ i5-8500
.. table::
:widths: 25 25 50
+--------------------------+-------------------------------------------+--------------------------------------------+
| | Server Platform | Client Platform |
+==========================+===========================================+============================================+
| Motherboard | ASUSTeK COMPUTER INC. PRIME Z370-A | Gigabyte Technology Co., Ltd. Z390 UD |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Memory | Corsair 2 x 16GB @ 2133 MT/s DDR4 | 029E 4 x 8GB @ 2400 MT/s DDR4 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU | Intel® Core™ i5-8500 CPU @ 3.00GHz | Intel® Core™ i3-8100 CPU @ 3.60GHz |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Selected CPU Flags | Turbo Boost | |
+--------------------------+-------------------------------------------+--------------------------------------------+
| CPU Thermal Design Power | 65W | 65 W |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Operating System | Ubuntu 20.04.4 LTS | Ubuntu 20.04.1 LTS |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Kernel Version | 5.4.0-113-generic | 5.4.0-52-generic |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Vendor | American Megatrends Inc. | American Megatrends Inc. |
+--------------------------+-------------------------------------------+--------------------------------------------+
| BIOS Version & Release | 3004; date: 07/12/2021 | F10j; date: 09/16/2020 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Docker Version | 19.03.13 | 20.10.0 |
+--------------------------+-------------------------------------------+--------------------------------------------+
| Network Speed | 40 Gb/s | 40 Gb/s |
+--------------------------+-------------------------------------------+--------------------------------------------+
@endsphinxdirective