IntenseWebs/openvino
3
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Doc Migration (master) (#1377)

Browse Source 
* Doc Migration from Gitlab (#1289)

* doc migration

* fix

* Update FakeQuantize_1.md

* Update performance_benchmarks.md

* Updates graphs for FPGA

* Update performance_benchmarks.md

* Change DL Workbench structure (#1)

* Changed DL Workbench structure

* Fixed tags

* fixes

* Update ie_docs.xml

* Update performance_benchmarks_faq.md

* Fixes in DL Workbench layout

* Fixes for CVS-31290

* [DL Workbench] Minor correction

* Fix for CVS-30955

* Added nGraph deprecation notice as requested by Zoe

* fix broken links in api doxy layouts

* CVS-31131 fixes

* Additional fixes

* Fixed POT TOC

* Update PAC_Configure.md

PAC DCP 1.2.1 install guide.

* Update inference_engine_intro.md

* fix broken link

* Update opset.md

* fix

* added opset4 to layout

* added new opsets to layout, set labels for them

* Update VisionAcceleratorFPGA_Configure.md

Updated from 2020.3 to 2020.4

Co-authored-by: domi2000 <domi2000@users.noreply.github.com>
This commit is contained in:
Nikolay Tyukaevanddomi2000 authored and GitHub committed 2020-07-20 17:36:08 +03:00
1 parent 4037613db3
commit ef45b5da8d
532 files changed
+30445 -313
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/benchmarks/performance_benchmarks.md
+234
View File
@@ -0,0 +1,234 @@
# Get a Deep Learning Model Performance Boost with Intel® Platforms {#openvino_docs_performance_benchmarks}
## Increase Performance for Deep Learning Inference
The [Intel® Distribution of OpenVINO™ toolkit](https://software.intel.com/en-us/openvino-toolkit) helps accelerate deep learning inference across a variety of Intel® processors and accelerators. Rather than a one-size-fits-all solution, Intel offers a powerful portfolio of scalable hardware and software solutions, powered by the Intel® Distribution of OpenVINO™ toolkit, to meet the various performance, power, and price requirements of any use case. The benchmarks below demonstrate high performance gains on several public neural networks for a streamlined, quick deployment on **Intel® CPU, VPU and FPGA** platforms. Use this data to help you decide which hardware is best for your applications and solutions, or to plan your AI workload on the Intel computing already included in your solutions.
Measuring inference performance involves many variables and is extremely use-case and application dependent. We use the below four parameters for measurements, which are key elements to consider for a successful deep learning inference application:
1. **Throughput** - Measures the number of inferences delivered within a latency threshold. (for example, number of frames per second). When deploying a system with deep learning inference, select the throughput that delivers the best trade-off between latency and power for the price and performance that meets your requirements.
2. **Value** - While throughput is important, what is more critical in edge AI deployments is the performance efficiency or performance-per-cost. Application performance in throughput per dollar of system cost is the best measure of value.
3. **Efficiency** - System power is a key consideration from the edge to the data center. When selecting deep learning solutions, power efficiency (throughput/watt) is a critical factor to consider. Intel designs provide excellent power efficiency for running deep learning workloads.
4. **Total Benefit** (Most applicable for Intel® VPU Platforms) - Combining the factors of value and efficiency can be a good way to compare which hardware yields the best performance per watt and per dollar for your particular use case.
---
## Intel® Xeon® E-2124G<a name="xeon-e"></a>
![](img/throughput_xeon_e212g.png)
![](img/value_xeon_e212g.png)
![](img/eff_xeon_e212g.png)
---
## Intel® Xeon® Silver 4216R <a name="xeon-silver"></a>
![](img/throughput_xeon_silver.png)
![](img/value_xeon_silver.png)
![](img/eff_xeon_silver.png)
---
## Intel® Xeon® Gold 5218T <a name="xeon-gold"></a>
![](img/throughput_xeon_gold.png)
![](img/value_xeon_gold.png)
![](img/eff_xeon_gold.png)
---
## Intel® Xeon® Platinum 8270 <a name="xeon-platinum"></a>
![](img/throughput_xeon_platinum.png)
![](img/value_xeon_platinum.png)
![](img/eff_xeon_platinum.png)
---
## Intel® Atom™ x5-E3940 <a name="atom"></a>
![](img/throughput_atom.png)
![](img/value_atom.png)
![](img/eff_atom.png)
---
## Intel® Core™ i3-8100 <a name="core-i3"></a>
![](img/throughput_i3.png)
![](img/value_i3.png)
![](img/eff_i3.png)
---
## Intel® Core™ i5-8500 <a name="core-i5"></a>
![](img/throughput_i5.png)
![](img/value_i5.png)
![](img/eff_i5.png)
---
## Intel® Core™ i7-8700T <a name="core-i7"></a>
![](img/throughput_i7.png)
![](img/value_i7.png)
![](img/eff_i7.png)
---
## Intel® Core™ i9-10920X <a name="core-i9"></a>
![](img/throughput_i9.png)
![](img/value_i9.png)
![](img/eff_i9.png)
---
## Intel® Neural Compute Stick 2 <a name="intel-ncs2"></a>
![](img/throughput_ncs2.png)
![](img/value_ncs2.png)
![](img/eff_ncs2.png)
![](img/benefit_ncs2.png)
---
## Intel® Vision Accelerator Design with Intel® Movidius™ VPUs (Uzel* UI-AR8) <a name="ivad-vpu"></a>
![](img/throughput_hddlr.png)
![](img/value_hddlr.png)
![](img/eff_hddlr.png)
---
## Intel® Vision Accelerator Design with Intel® Arria® 10 FPGA
![](img/throughput_ivad_fpga.png)
![](img/value_ivad_fpga.png)
![](img/eff_ivad_fpga.png)
## Platform Configurations
Intel® Distribution of OpenVINO™ toolkit performance benchmark numbers are based on release 2020.4.
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. Performance results are based on testing as of July 8, 2020 and may not reflect all publicly available security updates. See configuration disclosure for details. No product can be absolutely secure.
Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information, see [Performance Benchmark Test Disclosure](https://www.intel.com/content/www/us/en/benchmarks/benchmark.html).
Your costs and results may vary.
© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
Optimization Notice: Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. [Notice Revision #2010804](https://software.intel.com/articles/optimization-notice).
Testing by Intel done on: see test date for each HW platform below.
**CPU Inference Engines**
| | Intel® Xeon® E-2124G | Intel® Xeon® Silver 4216R | Intel® Xeon® Gold 5218T | Intel® Xeon® Platinum 8270 |
| ------------------------------- | ----------------------| ---------------------------- | ---------------------------- | ---------------------------- |
| Motherboard | ASUS* WS C246 PRO | Intel® Server Board S2600STB | Intel® Server Board S2600STB | Intel® Server Board S2600STB |
| CPU | Intel® Xeon® E-2124G CPU @ 3.40GHz | Intel® Xeon® Silver 4216R CPU @ 2.20GHz | Intel® Xeon® Gold 5218T CPU @ 2.10GHz | Intel® Xeon® Platinum 8270 CPU @ 2.70GHz |
| Hyper Threading | OFF | ON | ON | ON |
| Turbo Setting | ON | ON | ON | ON |
| Memory | 2 x 16 GB DDR4 2666MHz| 12 x 32 GB DDR4 2666MHz | 12 x 32 GB DDR4 2666MHz | 12 x 32 GB DDR4 2933MHz |
| Operating System | Ubuntu* 18.04 LTS | Ubuntu* 18.04 LTS | Ubuntu* 18.04 LTS | Ubuntu* 18.04 LTS |
| Kernel Version | 5.3.0-24-generic | 5.3.0-24-generic | 5.3.0-24-generic | 5.3.0-24-generic |
| BIOS Vendor | American Megatrends Inc.* | Intel Corporation | Intel Corporation | Intel Corporation |
| BIOS Version | 0904 | SE5C620.86B.02.01.<br>0009.092820190230 | SE5C620.86B.02.01.<br>0009.092820190230 | SE5C620.86B.02.01.<br>0009.092820190230 |
| BIOS Release | April 12, 2019 | September 28, 2019 | September 28, 2019 | September 28, 2019 |
| BIOS Settings | Select optimized default settings, <br>save & exit | Select optimized default settings, <br>change power policy <br>to "performance", <br>save & exit | Select optimized default settings, <br>change power policy to "performance", <br>save & exit | Select optimized default settings, <br>change power policy to "performance", <br>save & exit |
| Batch size | 1 | 1 | 1 | 1 |
| Precision | INT8 | INT8 | INT8 | INT8 |
| Number of concurrent inference requests | 4 | 32 | 32 | 52 |
| Test Date | July 8, 2020 | July 8, 2020 | July 8, 2020 | July 8, 2020 |
| Power dissipation, TDP in Watt | [71](https://ark.intel.com/content/www/us/en/ark/products/134854/intel-xeon-e-2124g-processor-8m-cache-up-to-4-50-ghz.html#tab-blade-1-0-1) | [125](https://ark.intel.com/content/www/us/en/ark/products/193394/intel-xeon-silver-4216-processor-22m-cache-2-10-ghz.html#tab-blade-1-0-1) | [105](https://ark.intel.com/content/www/us/en/ark/products/193953/intel-xeon-gold-5218t-processor-22m-cache-2-10-ghz.html#tab-blade-1-0-1) | [205](https://ark.intel.com/content/www/us/en/ark/products/192482/intel-xeon-platinum-8270-processor-35-75m-cache-2-70-ghz.html#tab-blade-1-0-1) |
| CPU Price on July 8, 2020, USD<br>Prices may vary | [213](https://ark.intel.com/content/www/us/en/ark/products/134854/intel-xeon-e-2124g-processor-8m-cache-up-to-4-50-ghz.html) | [1,002](https://ark.intel.com/content/www/us/en/ark/products/193394/intel-xeon-silver-4216-processor-22m-cache-2-10-ghz.html) | [1,349](https://ark.intel.com/content/www/us/en/ark/products/193953/intel-xeon-gold-5218t-processor-22m-cache-2-10-ghz.html) | [7,405](https://ark.intel.com/content/www/us/en/ark/products/192482/intel-xeon-platinum-8270-processor-35-75m-cache-2-70-ghz.html) |
**CPU Inference Engines (continue)**
| | Intel® Core™ i5-8500 | Intel® Core™ i7-8700T | Intel® Core™ i9-10920X |
| -------------------- | ---------------------------------- | ----------------------------------- |--------------------------------------|
| Motherboard | ASUS* PRIME Z370-A | GIGABYTE* Z370M DS3H-CF | ASUS* PRIME X299-A II |
| CPU | Intel® Core™ i5-8500 CPU @ 3.00GHz | Intel® Core™ i7-8700T CPU @ 2.40GHz | Intel® Core™ i9-10920X CPU @ 3.50GHz |
| Hyper Threading | OFF | ON | ON |
| Turbo Setting | ON | ON | ON |
| Memory | 2 x 16 GB DDR4 2666MHz | 4 x 16 GB DDR4 2400MHz | 4 x 16 GB DDR4 2666MHz |
| Operating System | Ubuntu* 18.04 LTS | Ubuntu* 18.04 LTS | Ubuntu* 18.04 LTS |
| Kernel Version | 5.3.0-24-generic | 5.0.0-23-generic | 5.0.0-23-generic |
| BIOS Vendor | American Megatrends Inc.* | American Megatrends Inc.* | American Megatrends Inc.* |
| BIOS Version | 2401 | F11 | 505 |
| BIOS Release | July 12, 2019 | March 13, 2019 | December 17, 2019 |
| BIOS Settings | Select optimized default settings, <br>save & exit | Select optimized default settings, <br>set OS type to "other", <br>save & exit | Default Settings |
| Batch size | 1 | 1 | 1 |
| Precision | INT8 | INT8 | INT8 |
| Number of concurrent inference requests | 3 | 4 | 24 |
| Test Date | July 8, 2020 | July 8, 2020 | July 8, 2020 |
| Power dissipation, TDP in Watt | [65](https://ark.intel.com/content/www/us/en/ark/products/129939/intel-core-i5-8500-processor-9m-cache-up-to-4-10-ghz.html#tab-blade-1-0-1) | [35](https://ark.intel.com/content/www/us/en/ark/products/129948/intel-core-i7-8700t-processor-12m-cache-up-to-4-00-ghz.html#tab-blade-1-0-1) | [165](https://ark.intel.com/content/www/us/en/ark/products/198012/intel-core-i9-10920x-x-series-processor-19-25m-cache-3-50-ghz.html) |
| CPU Price on July 8, 2020, USD<br>Prices may vary | [192](https://ark.intel.com/content/www/us/en/ark/products/129939/intel-core-i5-8500-processor-9m-cache-up-to-4-10-ghz.html) | [303](https://ark.intel.com/content/www/us/en/ark/products/129948/intel-core-i7-8700t-processor-12m-cache-up-to-4-00-ghz.html) | [700](https://ark.intel.com/content/www/us/en/ark/products/198012/intel-core-i9-10920x-x-series-processor-19-25m-cache-3-50-ghz.html)
**CPU Inference Engines (continue)**
| | Intel Atom® x5-E3940 | Intel® Core™ i3-8100 |
| -------------------- | ---------------------------------- |----------------------------------- |
| Motherboard | | GIGABYTE* Z390 UD |
| CPU | Intel Atom® Processor E3940 @ 1.60GHz | Intel® Core™ i3-8100 CPU @ 3.60GHz |
| Hyper Threading | OFF | OFF |
| Turbo Setting | ON | OFF |
| Memory | 1 x 8 GB DDR3 1600MHz | 4 x 8 GB DDR4 2400MHz |
| Operating System | Ubuntu* 18.04 LTS | Ubuntu* 18.04 LTS |
| Kernel Version | 5.3.0-24-generic | 5.3.0-24-generic |
| BIOS Vendor | American Megatrends Inc.* | American Megatrends Inc.* |
| BIOS Version | 5.12 | F8 |
| BIOS Release | September 6, 2017 | May 24, 2019 |
| BIOS Settings | Default settings | Select optimized default settings, <br> set OS type to "other", <br>save & exit |
| Batch size | 1 | 1 |
| Precision | INT8 | INT8 |
| Number of concurrent inference requests | 4 | 4 |
| Test Date | July 8, 2020 | July 8, 2020 |
| Power dissipation, TDP in Watt | [9.5](https://ark.intel.com/content/www/us/en/ark/products/96485/intel-atom-x5-e3940-processor-2m-cache-up-to-1-80-ghz.html) | [65](https://ark.intel.com/content/www/us/en/ark/products/126688/intel-core-i3-8100-processor-6m-cache-3-60-ghz.html#tab-blade-1-0-1)|
| CPU Price on July 8, 2020, USD<br>Prices may vary | [34](https://ark.intel.com/content/www/us/en/ark/products/96485/intel-atom-x5-e3940-processor-2m-cache-up-to-1-80-ghz.html) | [117](https://ark.intel.com/content/www/us/en/ark/products/126688/intel-core-i3-8100-processor-6m-cache-3-60-ghz.html) |
**Accelerator Inference Engines**
| | Intel® Neural Compute Stick 2 | Intel® Vision Accelerator Design<br>with Intel® Movidius™ VPUs (Uzel* UI-AR8) | Intel® Vision Accelerator Design<br>with Intel® Arria® 10 FPGA - IEI/SAF3*|
| -------------------- | ------------------------------------- | ------------------------------------- | ------------------------- |
| VPU | 1 X Intel® Movidius™ Myriad™ X MA2485 | 8 X Intel® Movidius™ Myriad™ X MA2485 | 1 X Intel® Arria® 10 FPGA |
| Connection | USB 2.0/3.0 | PCIe X4 | PCIe X8 |
| Batch size | 1 | 1 | 1 |
| Precision | FP16 | FP16 | FP11 |
| Number of concurrent inference requests | 4 | 32 | 5 |
| Power dissipation, TDP in Watt | 2.5 | [30](https://www.mouser.com/ProductDetail/IEI/MUSTANG-V100-MX8-R10?qs=u16ybLDytRaZtiUUvsd36w%3D%3D) | [60](https://www.mouser.com/ProductDetail/IEI/MUSTANG-F100-A10-R10?qs=sGAEpiMZZMtNlGR3Dbecs5Qs0RmP5oxxCbTJPjyRuMXthliRUwiVGw%3D%3D) |
| CPU Price, USD<br>Prices may vary | [69](https://ark.intel.com/content/www/us/en/ark/products/140109/intel-neural-compute-stick-2.html) (from July 8, 2020) | [768](https://www.mouser.com/ProductDetail/IEI/MUSTANG-V100-MX8-R10?qs=u16ybLDytRaZtiUUvsd36w%3D%3D) (from May 15, 2020) | [1,650](https://www.bhphotovideo.com/c/product/1477989-REG/qnap_mustang_f100_a10_r10_pcie_fpga_highest_performance.html/?ap=y&ap=y&smp=y&msclkid=371b373256dd1a52beb969ecf5981bf8) (from July 8, 2020) |
| Host Computer | Intel® Core™ i7 | Intel® Core™ i5 | Intel® Xeon® E3 |
| Motherboard | ASUS* Z370-A II | Uzelinfo* / US-E1300 | IEI/SAF3* |
| CPU | Intel® Core™ i7-8700 CPU @ 3.20GHz | Intel® Core™ i5-6600 CPU @ 3.30GHz | Intel® Xeon® CPU E3-1268L v5 @ 2.40GHz |
| Hyper Threading | ON | OFF | OFF |
| Turbo Setting | ON | ON | ON |
| Memory | 4 x 16 GB DDR4 2666MHz | 2 x 16 GB DDR4 2400MHz | 2 x 16 GB DDR4 2666MHz |
| Operating System | Ubuntu* 18.04 LTS | Ubuntu* 18.04 LTS | Ubuntu* 16.04 LTS |
| Kernel Version | 5.0.0-23-generic | 5.0.0-23-generic | 4.13.0-45-generic |
| BIOS Vendor | American Megatrends Inc.* | American Megatrends Inc.* | American Megatrends Inc.* |
| BIOS Version | 411 | 5.12 | V2RMAR15 |
| BIOS Release | September 21, 2018 | September 21, 2018 | December 03, 2019 |
| Test Date | July 8, 2020 | July 8, 2020 | July 8, 2020 |
Please follow this link for more detailed configuration descriptions: [Configuration Details](https://docs.openvinotoolkit.org/resources/benchmark_files/system_configurations_2020.4.html)
\htmlonly
<style>
.footer {
display: none;
}
</style>
<div class="opt-notice-wrapper">
<p class="opt-notice">
\endhtmlonly
For more complete information about performance and benchmark results, visit: [www.intel.com/benchmarks](https://www.intel.com/benchmarks) and [Optimization Notice](https://software.intel.com/articles/optimization-notice). [Legal Information](../Legal_Information.md).
\htmlonly
</p>
</div>
\endhtmlonly
docs/benchmarks/performance_benchmarks_faq.md
+62
View File
@@ -0,0 +1,62 @@
# Performance Information Frequently Asked Questions {#openvino_docs_performance_benchmarks_faq}
The following questions and answers are related to performance benchmarks published on the [Performance Information](https://docs.openvinotoolkit.org/latest/_docs_performance_benchmarks.html) documentation site.
#### 1. How often do performance benchmarks get updated?
New performance benchmarks are typically published on every `major.minor` release of the Intel® Distribution of OpenVINO™ toolkit.
#### 2. Where can I find the models used in the performance benchmarks?
All of the models used are included in the toolkit's [Open Model Zoo](https://github.com/opencv/open_model_zoo) GitHub repository.
#### 3. Will there be new models added to the list used for benchmarking?
The models used in the performance benchmarks were chosen based on general adoption and usage in deployment scenarios. We're continuing to add new models that support a diverse set of workloads and usage.
#### 4. What does CF or TF in the graphs stand for?
CF means Caffe*, while TF means TensorFlow*.
#### 5. How can I run the benchmark results on my own?
All of the performance benchmarks were generated using the open-sourced tool within the Intel® Distribution of OpenVINO™ toolkit called `benchmark_app`, which is available in both [C++](https://docs.openvinotoolkit.org/latest/_inference_engine_samples_benchmark_app_README.html) and [Python](https://docs.openvinotoolkit.org/latest/_inference_engine_tools_benchmark_tool_README.html).
#### 6. What image sizes are used for the classification network models?
The image size used in the inference depends on the network being benchmarked. The following table shows the list of input sizes for each network model.
| **Model** | **Public Network** | **Task** | **Input Size** (Height x Width) |
|------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|-----------------------------|-----------------------------------|
| [faster_rcnn_resnet50_coco-TF](https://github.com/opencv/open_model_zoo/tree/master/models/public/faster_rcnn_resnet50_coco) | Faster RCNN Tf | object detection | 600x1024 |
| [googlenet-v1-CF](https://github.com/opencv/open_model_zoo/tree/master/models/public/googlenet-v1) | GoogLeNet_ILSVRC-2012_Caffe | classification | 224x224 |
| [googlenet-v3-TF](https://github.com/opencv/open_model_zoo/tree/master/models/public/googlenet-v3) | Inception v3 Tf | classification | 299x299 |
| [mobilenet-ssd-CF](https://github.com/opencv/open_model_zoo/tree/master/models/public/mobilenet-ssd) | SSD (MobileNet)_COCO-2017_Caffe | object detection | 300x300 |
| [mobilenet-v2-1.0-224-TF](https://github.com/opencv/open_model_zoo/tree/master/models/public/mobilenet-v2-1.0-224) | MobileNet v2 Tf | classification | 224x224 |
| [mobilenet-v2-CF](https://github.com/opencv/open_model_zoo/tree/master/models/public/mobilenet-v2) | Mobilenet V2 Caffe | classification | 224x224 |
| [resnet-101-CF](https://github.com/opencv/open_model_zoo/tree/master/models/public/resnet-101) | ResNet-101_ILSVRC-2012_Caffe | classification | 224x224 |
| [resnet-50-CF](https://github.com/opencv/open_model_zoo/tree/master/models/public/resnet-50) | ResNet-50_v1_ILSVRC-2012_Caffe | classification | 224x224 |
| [se-resnext-50-CF](https://github.com/opencv/open_model_zoo/tree/master/models/public/se-resnext-50) | Se-ResNext-50_ILSVRC-2012_Caffe | classification | 224x224 |
| [squeezenet1.1-CF](https://github.com/opencv/open_model_zoo/tree/master/models/public/squeezenet1.1) | SqueezeNet_v1.1_ILSVRC-2012_Caffe | classification | 227x227 |
| [ssd300-CF](https://github.com/opencv/open_model_zoo/tree/master/models/public/ssd300) | SSD (VGG-16)_VOC-2007_Caffe | object detection | 300x300 |
#### 7. Where can I purchase the specific hardware used in the benchmarking?
Intel partners with various vendors all over the world. Visit the [Intel® AI: In Production Partners & Solutions Catalog](https://www.intel.com/content/www/us/en/internet-of-things/ai-in-production/partners-solutions-catalog.html) for a list of Equipment Makers and the [Supported Devices](https://docs.openvinotoolkit.org/latest/_docs_IE_DG_supported_plugins_Supported_Devices.html) documentation. You can also remotely test and run models before purchasing any hardware by using [Intel® DevCloud for the Edge](http://devcloud.intel.com/edge/).
#### 8. How can I optimize my models for better performance or accuracy?
We published a set of guidelines and recommendations to optimize your models available in an [introductory](https://docs.openvinotoolkit.org/latest/_docs_IE_DG_Intro_to_Performance.html) guide and an [advanced](https://docs.openvinotoolkit.org/latest/_docs_optimization_guide_dldt_optimization_guide.html) guide. For further support, please join the conversation in the [Community Forum](https://software.intel.com/en-us/forums/intel-distribution-of-openvino-toolkit).
#### 9. Why are INT8 optimized models used for benchmarking on CPUs with no VNNI support?
The benefit of low-precision optimization using the OpenVINO™ toolkit model optimizer extends beyond processors supporting VNNI through Intel® DL Boost. The reduced bit width of INT8 compared to FP32 allows Intel® CPU to process the data faster and thus offers better throughput on any converted model agnostic of the intrinsically supported low-precision optimizations within Intel® hardware. Please refer to [INT8 vs. FP32 Comparison on Select Networks and Platforms](https://docs.openvinotoolkit.org/latest/_docs_performance_int8_vs_fp32.html) for comparison on boost factors for different network models and a selection of Intel® CPU architectures, including AVX-2 with Intel® Core™ i7-8700T, and AVX-512 (VNNI) with Intel® Xeon® 5218T and Intel® Xeon® 8270.
#### 10. Previous releases included benchmarks on googlenet-v1. Why is there no longer benchmarks on this neural network model?
We replaced googlenet-v1 to [resnet-18-pytorch](https://github.com/opencv/open_model_zoo/blob/master/models/public/resnet-18-pytorch/resnet-18-pytorch.md) due to changes in developer usage. The public model resnet-18 is used by many developers as an Image Classification model. This pre-optimized model was also trained on the ImageNet database, similar to googlenet-v1. Both googlenet-v1 and resnet-18 will remain part of the Open Model Zoo. Developers are encouraged to utilize resnet-18-pytorch for Image Classification use cases.
\htmlonly
<style>
.footer {
display: none;
}
</style>
<div class="opt-notice-wrapper">
<p class="opt-notice">
\endhtmlonly
For more complete information about performance and benchmark results, visit: [www.intel.com/benchmarks](https://www.intel.com/benchmarks) and [Optimization Notice](https://software.intel.com/articles/optimization-notice). [Legal Information](../Legal_Information.md).
\htmlonly
</p>
</div>
\endhtmlonly
docs/benchmarks/performance_int8_vs_fp32.md
+375
View File
@@ -0,0 +1,375 @@
# INT8 vs FP32 Comparison on Select Networks and Platforms {#openvino_docs_performance_int8_vs_fp32}
The table below illustrates the speed-up factor for the performance gain by switching from an FP32 representation of an OpenVINO™ supported model to its INT8 representation.
<table>
<tr align="left">
<th></th>
<th></th>
<th>Intel® Core™ <br>i7-8700T</th>
<th>Intel® Xeon® <br>Gold <br>5218T</th>
<th>Intel® Xeon® <br>Platinum <br>8270</th>
<th>Intel® Core™ <br>i7-1065G7</th>
</tr>
<tr align="left">
<th>OpenVINO <br>benchmark <br>model name</th>
<th>Dataset</th>
<th colspan="4" align="center">Throughput speed-up FP16-INT8 vs FP32</th>
</tr>
<tr>
<td>bert-large-<br>uncased-whole-word-<br>masking-squad-0001</td>
<td>SQuAD</td>
<td>1.5</td>
<td>2.5</td>
<td>2.0</td>
<td>N/A</td>
</tr>
<tr>
<td>brain-tumor-<br>segmentation-<br>0001-mxnet</td>
<td>BraTS</td>
<td>1.5</td>
<td>1.7</td>
<td>1.6</td>
<td>1.9</td>
</tr>
<tr>
<td>brain-tumor-<br>segmentation-<br>0002-cf2</td>
<td>BraTS<br>2017</td>
<td>1.2</td>
<td>1.7</td>
<td>1.4</td>
<td>2.2</td>
</tr>
<tr>
<td>deeplabv3-tf</td>
<td>VOC 2012<br>Segmentation</td>
<td>1.5</td>
<td>2.4</td>
<td>2.6</td>
<td>2.8</td>
</tr>
<tr>
<td>facenet-<br>20180408-<br>102900-tf</td>
<td>LFW</td>
<td>2.0</td>
<td>3.5</td>
<td>3.5</td>
<td>3.5</td>
</tr>
<tr>
<td>faster_rcnn_<br>resnet50_coco-tf</td>
<td>MS COCO</td>
<td>1.7</td>
<td>3.4</td>
<td>3.4</td>
<td>3.6</td>
</tr>
<tr>
<td>googlenet-v1-caffe</td>
<td>ImageNet</td>
<td>1.5</td>
<td>2.9</td>
<td>2.2</td>
<td>3.2</td>
</tr>
<tr>
<td>inception-v3-tf</td>
<td>ImageNet</td>
<td>1.8</td>
<td>3.8</td>
<td>4.0</td>
<td>3.7</td>
</tr>
<tr>
<td>mobilenet-<br>ssd-caffe</td>
<td>VOC2012</td>
<td>1.5</td>
<td>3.0</td>
<td>3.3</td>
<td>3.1</td>
</tr>
<tr>
<td>mobilenet-v1-1.0-<br>224-caffe</td>
<td>ImageNet</td>
<td>1.5</td>
<td>3.2</td>
<td>3.9</td>
<td>2.9</td>
</tr>
<tr>
<td>mobilenet-v2-1.0-<br>224-tf</td>
<td>ImageNet</td>
<td>1.3</td>
<td>2.6</td>
<td>3.8</td>
<td>2.2</td>
</tr>
<tr>
<td>mobilenet-v2-<br>caffe</td>
<td>ImageNet</td>
<td>1.3</td>
<td>2.5</td>
<td>3.4</td>
<td>2.2</td>
</tr>
<tr>
<td>resnet-101-<br>caffe</td>
<td>ImageNet</td>
<td>1.8</td>
<td>3.7</td>
<td>3.7</td>
<td>3.6</td>
</tr>
<tr>
<td>resnet-18-<br>pytorch</td>
<td>ImageNet</td>
<td>1.9</td>
<td>3.7</td>
<td>3.8</td>
<td>3.6</td>
</tr>
<tr>
<td>resnet-50-<br>caffe</td>
<td>ImageNet</td>
<td>1.8</td>
<td>3.6</td>
<td>3.9</td>
<td>3.5</td>
</tr>
<tr>
<td>resnet-50-<br>pytorch</td>
<td>ImageNet</td>
<td>1.8</td>
<td>3.6</td>
<td>3.9</td>
<td>3.5</td>
</tr>
<tr>
<td>squeezenet1.1-<br>caffe</td>
<td>ImageNet</td>
<td>1.6</td>
<td>2.9</td>
<td>3.2</td>
<td>3.0</td>
</tr>
<tr>
<td>ssd_mobilenet_<br>v1_coco-tf</td>
<td>MS COCO</td>
<td>1.6</td>
<td>3.0</td>
<td>3.4</td>
<td>3.1</td>
</tr>
<tr>
<td>ssd300-caffe</td>
<td>MS COCO</td>
<td>1.8</td>
<td>3.7</td>
<td>3.7</td>
<td>3.8</td>
</tr>
<tr>
<td>ssdlite_<br>mobilenet_<br>v2-tf</td>
<td>MS COCO</td>
<td>1.4</td>
<td>2.3</td>
<td>3.0</td>
<td>2.4</td>
</tr>
</table>
The following table shows the absolute accuracy drop that is calculated as the difference in accuracy between the FP32 representation of a model and its INT8 representation.
<table>
<tr align="left">
<th></th>
<th></th>
<th></th>
<th>Intel® Core™ <br>i9-10920X CPU<br>@ 3.50GHZ (VNNI)</th>
<th>Intel® Core™ <br>i9-9820X CPU<br>@ 3.30GHz (AVX512)</th>
<th>Intel® Core™ <br>i7-8700 CPU<br>@ 3.20GHz (AVX2)</th>
</tr>
<tr align="left">
<th>OpenVINO Benchmark <br>Model Name</th>
<th>Dataset</th>
<th>Metric Name</th>
<th colspan="3" align="center">Absolute Accuracy Drop, %</th>
</tr>
<tr>
<td>bert-large-<br>uncased-whole-word-<br>masking-squad-0001</td>
<td>SQuAD</td>
<td>F1</td>
<td>0.46</td>
<td>0.70</td>
<td>0.64</td>
</tr>
<tr>
<td>brain-tumor-<br>segmentation-<br>0001-mxnet</td>
<td>BraTS</td>
<td>Dice-index@ <br>Mean@ <br>Overall Tumor</td>
<td>0.08</td>
<td>0.08</td>
<td>0.14</td>
</tr>
<tr>
<td>brain-tumor-<br>segmentation-<br>0002-cf2</td>
<td>BraTS<br>2017</td>
<td>Dice-index@ <br>Mean@ <br>Overall Tumor</td>
<td>0.16</td>
<td>0.14</td>
<td>0.13</td>
</tr>
<tr>
<td>deeplabv3-tf</td>
<td>VOC 2012<br>Segmentation</td>
<td>mean_iou</td>
<td>0.28</td>
<td>0.71</td>
<td>0.71</td>
</tr>
<tr>
<td>facenet-<br>20180408-<br>102900-tf</td>
<td>LFW</td>
<td>pairwise_<br>accuracy<br>_subsets</td>
<td>0.02</td>
<td>0.05</td>
<td>0.05</td>
</tr>
<tr>
<td>faster_rcnn_<br>resnet50_coco-tf</td>
<td>MS COCO</td>
<td>coco_<br>precision</td>
<td>0.21</td>
<td>0.20</td>
<td>0.20</td>
</tr>
<tr>
<td>googlenet-v1-caffe</td>
<td>ImageNet</td>
<td>acc@top-1</td>
<td>0.24</td>
<td>0.19</td>
<td>0.20</td>
</tr>
<tr>
<td>inception-v3-tf</td>
<td>ImageNet</td>
<td>acc@top-1</td>
<td>0.03</td>
<td>0.01</td>
<td>0.01</td>
</tr>
<tr>
<td>mobilenet-<br>ssd-caffe</td>
<td>VOC2012</td>
<td>mAP</td>
<td>0.35</td>
<td>0.34</td>
<td>0.34</td>
</tr>
<tr>
<td>mobilenet-v1-1.0-<br>224-caffe</td>
<td>ImageNet</td>
<td>acc@top-1</td>
<td>0.19</td>
<td>0.18</td>
<td>0.18</td>
</tr>
<tr>
<td>mobilenet-v2-1.0-<br>224-tf</td>
<td>ImageNet</td>
<td>acc@top-1</td>
<td>0.45</td>
<td>0.94</td>
<td>0.94</td>
</tr>
<tr>
<td>mobilenet-v2-<br>caffe</td>
<td>ImageNet</td>
<td>acc@top-1</td>
<td>0.24</td>
<td>1.45</td>
<td>1.45</td>
</tr>
<tr>
<td>resnet-101-<br>caffe</td>
<td>ImageNet</td>
<td>acc@top-1</td>
<td>0.00</td>
<td>0.02</td>
<td>0.02</td>
</tr>
<tr>
<td>resnet-18-<br>pytorch</td>
<td>ImageNet</td>
<td>acc@top-1</td>
<td>0.26</td>
<td>0.25</td>
<td>0.25</td>
</tr>
<tr>
<td>resnet-50-<br>caffe</td>
<td>ImageNet</td>
<td>acc@top-1</td>
<td>0.16</td>
<td>0.12</td>
<td>0.12</td>
</tr>
<tr>
<td>resnet-50-<br>pytorch</td>
<td>ImageNet</td>
<td>acc@top-1</td>
<td>0.20</td>
<td>0.17</td>
<td>0.17</td>
</tr>
<tr>
<td>squeezenet1.1-<br>caffe</td>
<td>ImageNet</td>
<td>acc@top-1</td>
<td>0.66</td>
<td>0.64</td>
<td>0.64</td>
</tr>
<tr>
<td>ssd_mobilenet_<br>v1_coco-tf</td>
<td>MS COCO</td>
<td>COCO mAp</td>
<td>0.24</td>
<td>3.07</td>
<td>3.07</td>
</tr>
<tr>
<td>ssd300-caffe</td>
<td>MS COCO</td>
<td>COCO mAp</td>
<td>0.06</td>
<td>0.05</td>
<td>0.05</td>
</tr>
<tr>
<td>ssdlite_<br>mobilenet_<br>v2-tf</td>
<td>MS COCO</td>
<td>COCO mAp</td>
<td>0.14</td>
<td>0.47</td>
<td>0.47</td>
</tr>
</table>
![INT8 vs FP32 Comparison](img/int8vsfp32.png "INT8 vs FP32 Comparison on Select Networks and Platforms")
\htmlonly
<style>
.footer {
display: none;
}
</style>
<div class="opt-notice-wrapper">
<p class="opt-notice">
\endhtmlonly
For more complete information about performance and benchmark results, visit: [www.intel.com/benchmarks](https://www.intel.com/benchmarks) and [Optimization Notice](https://software.intel.com/articles/optimization-notice). [Legal Information](../Legal_Information.md).
\htmlonly
</p>
</div>
\endhtmlonly