IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Proofreading-OV-Runtime (#11658)

Browse Source 
* Update docs/OV_Runtime_UG/protecting_model_guide.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/protecting_model_guide.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/protecting_model_guide.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/protecting_model_guide.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/protecting_model_guide.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/protecting_model_guide.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/ARM_CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/ARM_CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/ARM_CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/ARM_CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/ARM_CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/ARM_CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/optimization_guide/dldt_deployment_optimization_common.md

Co-authored-by: Sebastian Golebiewski <sebastianx.golebiewski@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/Device_Plugins.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU_RemoteTensor_API.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU_RemoteTensor_API.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU_RemoteTensor_API.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU_RemoteTensor_API.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU_RemoteTensor_API.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU_RemoteTensor_API.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU_RemoteTensor_API.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU_RemoteTensor_API.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU_RemoteTensor_API.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GPU_RemoteTensor_API.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/HDDL.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/HDDL.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/HDDL.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/MYRIAD.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/MYRIAD.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/MYRIAD.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/ov_dynamic_shapes.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/config_properties.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/config_properties.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/preprocessing_details.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/preprocessing_details.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/preprocessing_details.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/preprocessing_details.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/performance_hints.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/deployment/deployment-manager-tool.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Apply suggestions from code review

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/preprocessing_details.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/performance_hints.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/preprocessing_details.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/performance_hints.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Update docs/OV_Runtime_UG/deployment/deployment-manager-tool.md

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Apply suggestions from code review

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Apply suggestions from code review

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>

* Apply suggestions from code review

* Apply suggestions from code review

* Apply suggestions from code review

* Apply suggestions from code review

* Apply suggestions from code review

* Apply suggestions from code review

* Apply suggestions from code review

* Apply suggestions from code review

* Apply suggestions from code review

* Apply suggestions from code review

* Update ref links

* Update Getting_performance_numbers.md

* Update deployment_intro.md

* Update preprocessing_details.md

* Apply suggestions from code review

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Apply suggestions from code review

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update tools/pot/openvino/tools/pot/algorithms/quantization/default/README.md

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/automatic_batching.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Apply suggestions from code review

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Apply suggestions from code review

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Apply suggestions from code review

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/deployment/deployment-manager-tool.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Apply suggestions from code review

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Apply suggestions from code review

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update tools/pot/openvino/tools/pot/algorithms/quantization/default/README.md

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update automatic_batching.md

* Update docs/OV_Runtime_UG/automatic_batching.md

* Update docs/OV_Runtime_UG/ShapeInference.md

* Update deployment-manager-tool.md

* Update deployment-manager-tool.md

* Update docs/OV_Runtime_UG/deployment/deployment-manager-tool.md

* Update automatic_batching.md

* Update automatic_batching.md

* Update docs/OV_Runtime_UG/ShapeInference.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/integrate_with_your_application.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/integrate_with_your_application.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/integrate_with_your_application.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/integrate_with_your_application.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/integrate_with_your_application.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update integrate_with_your_application.md

* Update docs/OV_Runtime_UG/integrate_with_your_application.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/integrate_with_your_application.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/integrate_with_your_application.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/integrate_with_your_application.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Apply suggestions from code review

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/model_representation.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/model_representation.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update integrate_with_your_application.md

* Update docs/OV_Runtime_UG/integrate_with_your_application.md

* Update docs/OV_Runtime_UG/layout_overview.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/layout_overview.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/layout_overview.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update model_representation.md

* Apply suggestions from code review

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Apply suggestions from code review

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update integrate_with_your_application.md

* Update docs/OV_Runtime_UG/layout_overview.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update Additional_Optimizations.md

Removing redundant information.

* Update docs/OV_Runtime_UG/layout_overview.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/layout_overview.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/layout_overview.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/layout_overview.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update Additional_Optimizations.md

* Apply suggestions from code review

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update Additional_Optimizations.md

* Update docs/OV_Runtime_UG/model_representation.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/layout_overview.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/OV_Runtime_UG/layout_overview.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update model_representation.md

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Update tools/pot/docs/SaturationIssue.md

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Update tools/pot/openvino/tools/pot/algorithms/quantization/accuracy_aware/README.md

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Update tools/pot/docs/SaturationIssue.md

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Update tools/pot/docs/SaturationIssue.md

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Update tools/pot/docs/SaturationIssue.md

* Update tools/pot/docs/SaturationIssue.md

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Update README.md

* Update README.md

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Update tools/pot/docs/Introduction.md

* Update tools/pot/docs/AccuracyAwareQuantizationUsage.md

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Removing one-liners

Removing introductory sentences from 'Supported Features' sections.

* Update docs/OV_Runtime_UG/openvino_intro.md

Co-authored-by: Yuan Xu <yuan1.xu@intel.com>

* Update docs/benchmarks/performance_benchmarks_ovms.md

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Update tools/pot/docs/Introduction.md

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

* Update tools/pot/docs/DefaultQuantizationUsage.md

* Update tools/pot/docs/BestPractices.md

* Update tools/pot/docs/BestPractices.md

* Update tools/pot/docs/AccuracyAwareQuantizationUsage.md

* Update docs/optimization_guide/model_optimization_guide.md

* Update docs/optimization_guide/dldt_deployment_optimization_guide.md

* Update docs/OV_Runtime_UG/supported_plugins/config_properties.md

* Update docs/OV_Runtime_UG/supported_plugins/GNA.md

* Update docs/OV_Runtime_UG/supported_plugins/CPU.md

* Update docs/OV_Runtime_UG/preprocessing_usecase_save.md

* Apply suggestions from code review

Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>

Co-authored-by: Maciej Smyk <maciejx.smyk@intel.com>
Co-authored-by: Yuan Xu <yuan1.xu@intel.com>
Co-authored-by: Karol Blaszczak <karol.blaszczak@intel.com>
Co-authored-by: msmykx <101244365+msmykx-intel@users.noreply.github.com>
Co-authored-by: Piotr Milewski <piotr.milewski@intel.com>
This commit is contained in:
Sebastian Golebiewski
2022-07-08 13:34:45 +02:00
committed by GitHub
parent 6283ab2fde
commit b1dcb276da
52 changed files with 1692 additions and 1759 deletions
Download Patch File Download Diff File Expand all files Collapse all files

BIN
docs/benchmarks/files/Platform_list.pdf Normal file
View File

Binary file not shown.

10
docs/benchmarks/performance_benchmarks.md
View File

@@ -14,11 +14,11 @@
The [Intel® Distribution of OpenVINO™ toolkit](https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit.html) helps accelerate deep learning inference across a variety of Intel® processors and accelerators.
The benchmarks below demonstrate high performance gains on several public neural networks on multiple Intel® CPUs, GPUs and VPUs covering a broad performance range. Use this data to help you decide which hardware is best for your applications and solutions, or to plan your AI workload on the Intel computing already included in your solutions.
The benchmark results below demonstrate high performance gains on several public neural networks on multiple Intel® CPUs, GPUs and VPUs covering a broad performance range. The results may be helpful when deciding which hardware is best for your applications and solutions or to plan AI workload on the Intel computing already included in your solutions.
Use the links below to review the benchmarking results for each alternative:
The following benchmarks are available:
* [Intel® Distribution of OpenVINO™ toolkit Benchmark Results](performance_benchmarks_openvino.md)
* [OpenVINO™ Model Server Benchmark Results](performance_benchmarks_ovms.md)
* [Intel® Distribution of OpenVINO™ toolkit Benchmark Results](performance_benchmarks_openvino.md).
* [OpenVINO™ Model Server Benchmark Results](performance_benchmarks_ovms.md).
Performance for a particular application can also be evaluated virtually using [Intel® DevCloud for the Edge](https://devcloud.intel.com/edge/), a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of the OpenVINO™ Toolkit. [Learn more](https://devcloud.intel.com/edge/get_started/devcloud/) or [Register here](https://inteliot.force.com/DevcloudForEdge/s/).
Performance of a particular application can also be evaluated virtually using [Intel® DevCloud for the Edge](https://devcloud.intel.com/edge/). It is a remote development environment with access to Intel® hardware and the latest versions of the Intel® Distribution of the OpenVINO™ Toolkit. To learn more about it, visit [the website](https://www.intel.com/content/www/us/en/developer/tools/devcloud/edge/overview.html) or [create an account](https://www.intel.com/content/www/us/en/forms/idz/devcloud-registration.html?tgt=https://www.intel.com/content/www/us/en/secure/forms/devcloud-enrollment/account-provisioning.html).

57
docs/benchmarks/performance_benchmarks_faq.md
View File

@@ -1,28 +1,29 @@
# Performance Information Frequently Asked Questions {#openvino_docs_performance_benchmarks_faq}
The following questions and answers are related to [performance benchmarks](./performance_benchmarks.md) published on the documentation site.
The following questions (Q#) and answers (A) are related to published [performance benchmarks](./performance_benchmarks.md).
#### 1. How often do performance benchmarks get updated?
New performance benchmarks are typically published on every `major.minor` release of the Intel® Distribution of OpenVINO™ toolkit.
#### Q1: How often do performance benchmarks get updated?
**A**: New performance benchmarks are typically published on every `major.minor` release of the Intel® Distribution of OpenVINO™ toolkit.
#### 2. Where can I find the models used in the performance benchmarks?
All of the models used are included in the toolkit's [Open Model Zoo](https://github.com/openvinotoolkit/open_model_zoo) GitHub repository.
#### Q2: Where can I find the models used in the performance benchmarks?
**A**: All models used are included in the GitHub repository of [Open Model Zoo](https://github.com/openvinotoolkit/open_model_zoo).
#### 3. Will there be new models added to the list used for benchmarking?
The models used in the performance benchmarks were chosen based on general adoption and usage in deployment scenarios. We're continuing to add new models that support a diverse set of workloads and usage.
#### Q3: Will there be any new models added to the list used for benchmarking?
**A**: The models used in the performance benchmarks were chosen based on general adoption and usage in deployment scenarios. New models that support a diverse set of workloads and usage are added periodically.
#### 4. What does CF or TF in the graphs stand for?
CF means Caffe*, while TF means TensorFlow*.
#### Q4: What does "CF" or "TF" in the graphs stand for?
**A**: The "CF" means "Caffe", and "TF" means "TensorFlow".
#### 5. How can I run the benchmark results on my own?
All of the performance benchmarks were generated using the open-sourced tool within the Intel® Distribution of OpenVINO™ toolkit called `benchmark_app`, which is available in both [C++](../../samples/cpp/benchmark_app/README.md) and [Python](../../tools/benchmark_tool/README.md).
#### Q5: How can I run the benchmark results on my own?
**A**: All of the performance benchmarks were generated using the open-source tool within the Intel® Distribution of OpenVINO™ toolkit called `benchmark_app`. This tool is available in both [C++](../../samples/cpp/benchmark_app/README.md) and [Python](../../tools/benchmark_tool/README.md).
#### Q6: What image sizes are used for the classification network models?
**A**: The image size used in inference depends on the benchmarked network. The table below presents the list of input sizes for each network model:
#### 6. What image sizes are used for the classification network models?
The image size used in the inference depends on the network being benchmarked. The following table shows the list of input sizes for each network model.
| **Model** | **Public Network** | **Task** | **Input Size** (Height x Width) |
|------------------------------------------------------------------------------------------------------------------------------------|------------------------------------|-----------------------------|-----------------------------------|
| [bert-base-cased](https://github.com/PaddlePaddle/PaddleNLP/tree/v2.1.1) | BERT | question / answer | 124 |
| [bert-large-uncased-whole-word-masking-squad](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001) | BERT-large | question / answer | 384 |
| [bert-large-uncased-whole-word-masking-squad-int8-0001](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001) | BERT-large | question / answer | 384 |
| [bert-small-uncased-whole-masking-squad-0002](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/intel/bert-small-uncased-whole-word-masking-squad-0002) | BERT-small | question / answer | 384 |
| [brain-tumor-segmentation-0001-MXNET](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/brain-tumor-segmentation-0001) | brain-tumor-segmentation-0001 | semantic segmentation | 128x128x128 |
| [brain-tumor-segmentation-0002-CF2](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/brain-tumor-segmentation-0002) | brain-tumor-segmentation-0002 | semantic segmentation | 128x128x128 |
@@ -33,7 +34,7 @@ The image size used in the inference depends on the network being benchmarked. T
| [Facedetection0200](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/intel/face-detection-0200) | FaceDetection0200 | detection | 256x256 |
| [faster_rcnn_resnet50_coco-TF](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/faster_rcnn_resnet50_coco) | Faster RCNN Tf | object detection | 600x1024 |
| [forward-tacotron-duration-prediction](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/forward-tacotron) | ForwardTacotron | text to speech | 241 |
| [inception-v4-TF](https://github.com/openvinotoolkit/open_model_zoo/tree/develop/models/public/googlenet-v4-tf) | Inception v4 Tf (aka GoogleNet-V4) | classification | 299x299 |
| [inception-v4-TF](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/googlenet-v4-tf) | Inception v4 Tf (aka GoogleNet-V4) | classification | 299x299 |
| [inception-v3-TF](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/googlenet-v3) | Inception v3 Tf | classification | 299x299 |
| [mask_rcnn_resnet50_atrous_coco](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/mask_rcnn_resnet50_atrous_coco) | Mask R-CNN ResNet50 Atrous | instance segmentation | 800x1365 |
| [mobilenet-ssd-CF](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/mobilenet-ssd) | SSD (MobileNet)_COCO-2017_Caffe | object detection | 300x300 |
@@ -49,22 +50,22 @@ The image size used in the inference depends on the network being benchmarked. T
| [yolo_v4-TF](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/yolo-v4-tf) | Yolo-V4 TF | object detection | 608x608 |
| [ssd_mobilenet_v1_coco-TF](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/ssd_mobilenet_v1_coco) | ssd_mobilenet_v1_coco | object detection | 300x300 |
| [ssdlite_mobilenet_v2-TF](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/ssdlite_mobilenet_v2) | ssdlite_mobilenet_v2 | object detection | 300x300 |
| [unet-camvid-onnx-0001](https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/intel/unet-camvid-onnx-0001/description/unet-camvid-onnx-0001.md) | U-Net | semantic segmentation | 368x480 |
| [yolo-v3-tiny-tf](https://github.com/openvinotoolkit/open_model_zoo/tree/develop/models/public/yolo-v3-tiny-tf) | YOLO v3 Tiny | object detection | 416x416 |
| [unet-camvid-onnx-0001](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/intel/unet-camvid-onnx-0001) | U-Net | semantic segmentation | 368x480 |
| [yolo-v3-tiny-tf](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/yolo-v3-tiny-tf) | YOLO v3 Tiny | object detection | 416x416 |
| [yolo-v3](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/yolo-v3-tf) | YOLO v3 | object detection | 416x416 |
| [ssd-resnet34-1200-onnx](https://github.com/openvinotoolkit/open_model_zoo/tree/develop/models/public/ssd-resnet34-1200-onnx) | ssd-resnet34 onnx model | object detection | 1200x1200 |
| [ssd-resnet34-1200-onnx](https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/ssd-resnet34-1200-onnx) | ssd-resnet34 onnx model | object detection | 1200x1200 |
#### 7. Where can I purchase the specific hardware used in the benchmarking?
Intel partners with various vendors all over the world. Visit the [Intel® AI: In Production Partners & Solutions Catalog](https://www.intel.com/content/www/us/en/internet-of-things/ai-in-production/partners-solutions-catalog.html) for a list of Equipment Makers and the [Supported Devices](../OV_Runtime_UG/supported_plugins/Supported_Devices.md) documentation. You can also remotely test and run models before purchasing any hardware by using [Intel® DevCloud for the Edge](http://devcloud.intel.com/edge/).
#### Q7: Where can I purchase the specific hardware used in the benchmarking?
**A**: Intel partners with vendors all over the world. For a list of Hardware Manufacturers, see the [Intel® AI: In Production Partners & Solutions Catalog](https://www.intel.com/content/www/us/en/internet-of-things/ai-in-production/partners-solutions-catalog.html) . For more details, see the [Supported Devices](../OV_Runtime_UG/supported_plugins/Supported_Devices.md) documentation. Before purchasing any hardware, you can test and run models remotely, using [Intel® DevCloud for the Edge](http://devcloud.intel.com/edge/).
#### 8. How can I optimize my models for better performance or accuracy?
We published a set of guidelines and recommendations to optimize your models available in the [optimization guide](../optimization_guide/dldt_optimization_guide.md). For further support, please join the conversation in the [Community Forum](https://software.intel.com/en-us/forums/intel-distribution-of-openvino-toolkit).
#### Q8: How can I optimize my models for better performance or accuracy?
**A**: Set of guidelines and recommendations to optimize models are available in the [optimization guide](../optimization_guide/dldt_optimization_guide.md). Join the conversation in the [Community Forum](https://software.intel.com/en-us/forums/intel-distribution-of-openvino-toolkit) for further support.
#### 9. Why are INT8 optimized models used for benchmarking on CPUs with no VNNI support?
The benefit of low-precision optimization using the OpenVINO™ toolkit model optimizer extends beyond processors supporting VNNI through Intel® DL Boost. The reduced bit width of INT8 compared to FP32 allows Intel® CPU to process the data faster and thus offers better throughput on any converted model agnostic of the intrinsically supported low-precision optimizations within Intel® hardware. Refer to [Model Accuracy for INT8 and FP32 Precision](performance_int8_vs_fp32.md) for comparison on boost factors for different network models and a selection of Intel® CPU architectures, including AVX-2 with Intel® Core™ i7-8700T, and AVX-512 (VNNI) with Intel® Xeon® 5218T and Intel® Xeon® 8270.
#### Q9: Why are INT8 optimized models used for benchmarking on CPUs with no VNNI support?
**A**: The benefit of low-precision optimization using the OpenVINO™ toolkit model optimizer extends beyond processors supporting VNNI through Intel® DL Boost. The reduced bit width of INT8 compared to FP32 allows Intel® CPU to process the data faster. Therefore, it offers better throughput on any converted model, regardless of the intrinsically supported low-precision optimizations within Intel® hardware. For comparison on boost factors for different network models and a selection of Intel® CPU architectures, including AVX-2 with Intel® Core™ i7-8700T, and AVX-512 (VNNI) with Intel® Xeon® 5218T and Intel® Xeon® 8270, refer to the [Model Accuracy for INT8 and FP32 Precision](performance_int8_vs_fp32.md) article.
#### 10. Where can I search for OpenVINO™ performance results based on HW-platforms?
The web site format has changed in order to support the more common search approach of looking for the performance of a given neural network model on different HW-platforms. As opposed to review a given HW-platform's performance on different neural network models.
#### Q10: Where can I search for OpenVINO™ performance results based on HW-platforms?
**A**: The website format has changed in order to support more common approach of searching for the performance results of a given neural network model on different HW-platforms. As opposed to reviewing performance of a given HW-platform when working with different neural network models.
#### 11. How is Latency measured?
Latency is measured by running the OpenVINO™ Runtime in synchronous mode. In synchronous mode each frame or image is processed through the entire set of stages (pre-processing, inference, post-processing) before the next frame or image is processed. This KPI is relevant for applications where the inference on a single image is required, for example the analysis of an ultra sound image in a medical application or the analysis of a seismic image in the oil & gas industry. Other use cases include real-time or near real-time applications like an industrial robot's response to changes in its environment and obstacle avoidance for autonomous vehicles where a quick response to the result of the inference is required.
#### Q11: How is Latency measured?
**A**: Latency is measured by running the OpenVINO™ Runtime in synchronous mode. In this mode, each frame or image is processed through the entire set of stages (pre-processing, inference, post-processing) before the next frame or image is processed. This KPI is relevant for applications where the inference on a single image is required. For example, the analysis of an ultra sound image in a medical application or the analysis of a seismic image in the oil & gas industry. Other use cases include real or near real-time applications, e.g. the response of industrial robot to changes in its environment and obstacle avoidance for autonomous vehicles, where a quick response to the result of the inference is required.

257
docs/benchmarks/performance_benchmarks_openvino.md
View File

@@ -6,24 +6,44 @@
:hidden:
openvino_docs_performance_benchmarks_faq
Download Performance Data Spreadsheet in MS Excel* Format <https://docs.openvino.ai/downloads/benchmark_files/OV-2022.1-Download-Excel.xlsx>
Download Performance Data Spreadsheet in MS Excel Format <https://docs.openvino.ai/downloads/benchmark_files/OV-2022.1-Download-Excel.xlsx>
openvino_docs_performance_int8_vs_fp32
@endsphinxdirective
This benchmark setup includes a single machine on which both the benchmark application and the OpenVINO™ installation reside.
Features and benefits of Intel® technologies depend on system configuration and may require enabled hardware, software or service activation. More information on this subject may be obtained from the original equipment manufacturer (OEM), official [Intel® web page](https://www.intel.com) or retailer.
The benchmark application loads the OpenVINO™ Runtime and executes inferences on the specified hardware (CPU, GPU or VPU). The benchmark application measures the time spent on actual inferencing (excluding any pre or post processing) and then reports on the inferences per second (or Frames Per Second). For more information on the benchmark application, please also refer to the entry 5 of the [FAQ section](performance_benchmarks_faq.md).
## Platform Configurations
Measuring inference performance involves many variables and is extremely use-case and application dependent. We use the below four parameters for measurements, which are key elements to consider for a successful deep learning inference application:
@sphinxdirective
- **Throughput** - Measures the number of inferences delivered within a latency threshold. (for example, number of Frames Per Second - FPS). When deploying a system with deep learning inference, select the throughput that delivers the best trade-off between latency and power for the price and performance that meets your requirements.
:download:`A full list of HW platforms used for testing (along with their configuration)<../../../docs/benchmarks/files/Platform_list.pdf>`
@endsphinxdirective
For more specific information, refer to the [Configuration Details](https://docs.openvino.ai/resources/benchmark_files/system_configurations_2022.1.html) document.
## Benchmark Setup Information
This benchmark setup includes a single machine on which both the benchmark application and the OpenVINO™ installation reside. The presented performance benchmark numbers are based on realease 2022.1 of Intel® Distribution of OpenVINO™ toolkit.
The benchmark application loads the OpenVINO™ Runtime and executes inferences on the specified hardware (CPU, GPU or VPU). It measures the time spent on actual inferencing (excluding any pre or post processing) and then reports on the inferences per second (or Frames Per Second - FPS). For additional information on the benchmark application, refer to the entry 5 in the [FAQ section](performance_benchmarks_faq.md).
Measuring inference performance involves many variables and is extremely use case and application dependent. Below are four parameters used for measurements, which are key elements to consider for a successful deep learning inference application:
- **Throughput** - Measures the number of inferences delivered within a latency threshold (for example, number of FPS). When deploying a system with deep learning inference, select the throughput that delivers the best trade-off between latency and power for the price and performance that meets your requirements.
- **Value** - While throughput is important, what is more critical in edge AI deployments is the performance efficiency or performance-per-cost. Application performance in throughput per dollar of system cost is the best measure of value.
- **Efficiency** - System power is a key consideration from the edge to the data center. When selecting deep learning solutions, power efficiency (throughput/watt) is a critical factor to consider. Intel designs provide excellent power efficiency for running deep learning workloads.
- **Latency** - This measures the synchronous execution of inference requests and is reported in milliseconds. Each inference request (for example: preprocess, infer, postprocess) is allowed to complete before the next is started. This performance metric is relevant in usage scenarios where a single image input needs to be acted upon as soon as possible. An example would be the healthcare sector where medical personnel only request analysis of a single ultra sound scanning image or in real-time or near real-time applications for example an industrial robot's response to actions in its environment or obstacle avoidance for autonomous vehicles.
- **Latency** - This parameter measures the synchronous execution of inference requests and is reported in milliseconds. Each inference request (i.e., preprocess, infer, postprocess) is allowed to complete before the next one is started. This performance metric is relevant in usage scenarios where a single image input needs to be acted upon as soon as possible. An example of that kind of a scenario would be real-time or near real-time applications, i.e., the response of an industrial robot to its environment or obstacle avoidance for autonomous vehicles.
## bert-base-cased [124]
## Benchmark Performance Results
Benchmark performance results below are based on testing as of March 17, 2022. They may not reflect all publicly available updates at the time of testing.
<!-- See configuration disclosure for details. No product can be absolutely secure. -->
Performance varies by use, configuration and other factors, which are elaborated further in [here](https://www.intel.com/PerformanceIndex). Used Intel optimizations (for Intel® compilers or other products) may not optimize to the same degree for non-Intel products.
### bert-base-cased [124]
@sphinxdirective
.. raw:: html
@@ -33,7 +53,7 @@ Measuring inference performance involves many variables and is extremely use-cas
@endsphinxdirective
## bert-large-uncased-whole-word-masking-squad-int8-0001 [384]
### bert-large-uncased-whole-word-masking-squad-int8-0001 [384]
@sphinxdirective
.. raw:: html
@@ -42,7 +62,7 @@ Measuring inference performance involves many variables and is extremely use-cas
@endsphinxdirective
## deeplabv3-TF [513x513]
### deeplabv3-TF [513x513]
@sphinxdirective
.. raw:: html
@@ -51,7 +71,7 @@ Measuring inference performance involves many variables and is extremely use-cas
@endsphinxdirective
## densenet-121-TF [224x224]
### densenet-121-TF [224x224]
@sphinxdirective
.. raw:: html
@@ -60,7 +80,7 @@ Measuring inference performance involves many variables and is extremely use-cas
@endsphinxdirective
## efficientdet-d0 [512x512]
### efficientdet-d0 [512x512]
@sphinxdirective
.. raw:: html
@@ -69,7 +89,7 @@ Measuring inference performance involves many variables and is extremely use-cas
@endsphinxdirective
## faster-rcnn-resnet50-coco-TF [600x1024]
### faster-rcnn-resnet50-coco-TF [600x1024]
@sphinxdirective
.. raw:: html
@@ -78,7 +98,7 @@ Measuring inference performance involves many variables and is extremely use-cas
@endsphinxdirective
## inception-v4-TF [299x299]
### inception-v4-TF [299x299]
@sphinxdirective
.. raw:: html
@@ -87,7 +107,7 @@ Measuring inference performance involves many variables and is extremely use-cas
@endsphinxdirective
## mobilenet-ssd-CF [300x300]
### mobilenet-ssd-CF [300x300]
@sphinxdirective
.. raw:: html
@@ -96,7 +116,7 @@ Measuring inference performance involves many variables and is extremely use-cas
@endsphinxdirective
## mobilenet-v2-pytorch [224x224]
### mobilenet-v2-pytorch [224x224]
@sphinxdirective
.. raw:: html
@@ -105,7 +125,7 @@ Measuring inference performance involves many variables and is extremely use-cas
@endsphinxdirective
## resnet-18-pytorch [224x224]
### resnet-18-pytorch [224x224]
@sphinxdirective
.. raw:: html
@@ -115,7 +135,7 @@ Measuring inference performance involves many variables and is extremely use-cas
@endsphinxdirective
## resnet_50_TF [224x224]
### resnet_50_TF [224x224]
@sphinxdirective
.. raw:: html
@@ -124,7 +144,7 @@ Measuring inference performance involves many variables and is extremely use-cas
@endsphinxdirective
## ssd-resnet34-1200-onnx [1200x1200]
### ssd-resnet34-1200-onnx [1200x1200]
@sphinxdirective
.. raw:: html
@@ -133,7 +153,7 @@ Measuring inference performance involves many variables and is extremely use-cas
@endsphinxdirective
## unet-camvid-onnx-0001 [368x480]
### unet-camvid-onnx-0001 [368x480]
@sphinxdirective
.. raw:: html
@@ -142,7 +162,7 @@ Measuring inference performance involves many variables and is extremely use-cas
@endsphinxdirective
## yolo-v3-tiny-tf [416x416]
### yolo-v3-tiny-tf [416x416]
@sphinxdirective
.. raw:: html
@@ -151,7 +171,7 @@ Measuring inference performance involves many variables and is extremely use-cas
@endsphinxdirective
## yolo_v4-tf [608x608]
### yolo_v4-tf [608x608]
@sphinxdirective
.. raw:: html
@@ -160,199 +180,4 @@ Measuring inference performance involves many variables and is extremely use-cas
@endsphinxdirective
## Platform Configurations
Intel® Distribution of OpenVINO™ toolkit performance benchmark numbers are based on release 2022.1.
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. Performance results are based on testing as of March 17, 2022 and may not reflect all publicly available updates. See configuration disclosure for details. No product can be absolutely secure.
Performance varies by use, configuration and other factors. Learn more at [www.intel.com/PerformanceIndex](https://www.intel.com/PerformanceIndex).
Your costs and results may vary.
© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
Intel optimizations, for Intel compilers or other products, may not optimize to the same degree for non-Intel products.
Testing by Intel done on: see test date for each HW platform below.
**CPU Inference Engines**
| Configuration | Intel® Xeon® E-2124G | Intel® Xeon® W1290P |
| ------------------------------- | ---------------------- | --------------------------- |
| Motherboard | ASUS* WS C246 PRO | ASUS* WS W480-ACE |
| CPU | Intel® Xeon® E-2124G CPU @ 3.40GHz | Intel® Xeon® W-1290P CPU @ 3.70GHz |
| Hyper Threading | OFF | ON |
| Turbo Setting | ON | ON |
| Memory | 2 x 16 GB DDR4 2666MHz | 4 x 16 GB DDR4 @ 2666MHz |
| Operating System | Ubuntu* 20.04.3 LTS | Ubuntu* 20.04.3 LTS |
| Kernel Version | 5.4.0-42-generic | 5.4.0-42-generic |
| BIOS Vendor | American Megatrends Inc.* | American Megatrends Inc. |
| BIOS Version | 1901 | 2301 |
| BIOS Release | September 24, 2021 | July 8, 2021 |
| BIOS Settings | Select optimized default settings, <br>save & exit | Select optimized default settings, <br>save & exit |
| Batch size | 1 | 1 |
| Precision | INT8 | INT8 |
| Number of concurrent inference requests | 4 | 5 |
| Test Date | March 17, 2022 | March 17, 2022 |
| Rated maximum TDP/socket in Watt | [71](https://ark.intel.com/content/www/us/en/ark/products/134854/intel-xeon-e-2124g-processor-8m-cache-up-to-4-50-ghz.html#tab-blade-1-0-1) | [125](https://ark.intel.com/content/www/us/en/ark/products/199336/intel-xeon-w-1290p-processor-20m-cache-3-70-ghz.html) |
| CPU Price/socket on Feb 14, 2022, USD<br>Prices may vary | [213](https://ark.intel.com/content/www/us/en/ark/products/134854/intel-xeon-e-2124g-processor-8m-cache-up-to-4-50-ghz.html) | [539](https://ark.intel.com/content/www/us/en/ark/products/199336/intel-xeon-w-1290p-processor-20m-cache-3-70-ghz.html) |
**CPU Inference Engines (continue)**
| Configuration | Intel® Xeon® Silver 4216R | Intel® Xeon® Silver 4316 |
| ------------------------------- | ---------------------- | --------------------------- |
| Motherboard | Intel® Server Board S2600STB | Intel Corporation / WilsonCity |
| CPU | Intel® Xeon® Silver 4216R CPU @ 2.20GHz | Intel® Xeon® Silver 4316 CPU @ 2.30GHz |
| Hyper Threading | ON | ON |
| Turbo Setting | ON | ON |
| Memory | 12 x 32 GB DDR4 2666MHz | 16 x 32 GB DDR4 @ 2666MHz |
| Operating System | Ubuntu* 20.04.3 LTS | Ubuntu* 20.04.3 LTS |
| Kernel Version | 5.3.0-24-generic | 5.4.0-100-generic |
| BIOS Vendor | Intel Corporation | Intel Corporation |
| BIOS Version | SE5C620.86B.02.01.<br>0013.121520200651 | WLYDCRB1.SYS.0021.<br>P41.2109200451 |
| BIOS Release | December 15, 2020 | September 20, 2021 |
| BIOS Settings | Select optimized default settings, <br>change power policy <br>to "performance", <br>save & exit | Select optimized default settings, <br>save & exit |
| Batch size | 1 | 1 |
| Precision | INT8 | INT8 |
| Number of concurrent inference requests | 32 | 10 |
| Test Date | March 17, 2022 | March 17, 2022 |
| Rated maximum TDP/socket in Watt | [125](https://ark.intel.com/content/www/us/en/ark/products/193394/intel-xeon-silver-4216-processor-22m-cache-2-10-ghz.html#tab-blade-1-0-1) | [150](https://ark.intel.com/content/www/us/en/ark/products/215270/intel-xeon-silver-4316-processor-30m-cache-2-30-ghz.html)|
| CPU Price/socket on June 21, 2021, USD<br>Prices may vary | [1,002](https://ark.intel.com/content/www/us/en/ark/products/193394/intel-xeon-silver-4216-processor-22m-cache-2-10-ghz.html) | [1083](https://ark.intel.com/content/www/us/en/ark/products/215270/intel-xeon-silver-4316-processor-30m-cache-2-30-ghz.html)|
**CPU Inference Engines (continue)**
| Configuration | Intel® Xeon® Gold 5218T | Intel® Xeon® Platinum 8270 | Intel® Xeon® Platinum 8380 |
| ------------------------------- | ---------------------------- | ---------------------------- | -----------------------------------------|
| Motherboard | Intel® Server Board S2600STB | Intel® Server Board S2600STB | Intel Corporation / WilsonCity |
| CPU | Intel® Xeon® Gold 5218T CPU @ 2.10GHz | Intel® Xeon® Platinum 8270 CPU @ 2.70GHz | Intel® Xeon® Platinum 8380 CPU @ 2.30GHz |
| Hyper Threading | ON | ON | ON |
| Turbo Setting | ON | ON | ON |
| Memory | 12 x 32 GB DDR4 2666MHz | 12 x 32 GB DDR4 2933MHz | 16 x 16 GB DDR4 3200MHz |
| Operating System | Ubuntu* 20.04.3 LTS | Ubuntu* 20.04.3 LTS | Ubuntu* 20.04.1 LTS |
| Kernel Version | 5.3.0-24-generic | 5.3.0-24-generic | 5.4.0-64-generic |
| BIOS Vendor | Intel Corporation | Intel Corporation | Intel Corporation |
| BIOS Version | SE5C620.86B.02.01.<br>0013.121520200651 | SE5C620.86B.02.01.<br>0013.121520200651 | WLYDCRB1.SYS.0020.<br>P86.2103050636 |
| BIOS Release | December 15, 2020 | December 15, 2020 | March 5, 2021 |
| BIOS Settings | Select optimized default settings, <br>change power policy to "performance", <br>save & exit | Select optimized default settings, <br>change power policy to "performance", <br>save & exit | Select optimized default settings, <br>change power policy to "performance", <br>save & exit |
| Batch size | 1 | 1 | 1 |
| Precision | INT8 | INT8 | INT8 |
| Number of concurrent inference requests | 32 | 52 | 80 |
| Test Date | March 17, 2022 | March 17, 2022 | March 17, 2022 |
| Rated maximum TDP/socket in Watt | [105](https://ark.intel.com/content/www/us/en/ark/products/193953/intel-xeon-gold-5218t-processor-22m-cache-2-10-ghz.html#tab-blade-1-0-1) | [205](https://ark.intel.com/content/www/us/en/ark/products/192482/intel-xeon-platinum-8270-processor-35-75m-cache-2-70-ghz.html#tab-blade-1-0-1) | [270](https://mark.intel.com/content/www/us/en/secure/mark/products/212287/intel-xeon-platinum-8380-processor-60m-cache-2-30-ghz.html#tab-blade-1-0-1) |
| CPU Price/socket on Feb 14, 2022, USD<br>Prices may vary | [1,349](https://ark.intel.com/content/www/us/en/ark/products/193953/intel-xeon-gold-5218t-processor-22m-cache-2-10-ghz.html) | [7,405](https://ark.intel.com/content/www/us/en/ark/products/192482/intel-xeon-platinum-8270-processor-35-75m-cache-2-70-ghz.html) | [8,099](https://mark.intel.com/content/www/us/en/secure/mark/products/212287/intel-xeon-platinum-8380-processor-60m-cache-2-30-ghz.html#tab-blade-1-0-0) |
**CPU Inference Engines (continue)**
| Configuration | Intel® Core™ i9-10920X | Intel® Core™ i9-10900TE | Intel® Core™ i9-12900 |
| -------------------- | -------------------------------------| ----------------------- | -------------------------------------------------------------- |
| Motherboard | ASUS* PRIME X299-A II | B595 | Intel Corporation<br>internal/Reference<br>Validation Platform |
| CPU | Intel® Core™ i9-10920X CPU @ 3.50GHz | Intel® Core™ i9-10900TE CPU @ 1.80GHz | 12th Gen Intel® Core™ i9-12900 |
| Hyper Threading | ON | ON | OFF |
| Turbo Setting | ON | ON | - |
| Memory | 4 x 16 GB DDR4 2666MHz | 2 x 8 GB DDR4 @ 2400 MHz | 4 x 8 GB DDR4 4800MHz |
| Operating System | Ubuntu 20.04.3 LTS | Ubuntu 20.04.3 LTS | Microsoft Windows 10 Pro |
| Kernel Version | 5.4.0-42-generic | 5.4.0-42-generic | 10.0.19043 N/A Build 19043 |
| BIOS Vendor | American Megatrends Inc.* | American Megatrends Inc.* | Intel Corporation |
| BIOS Version | 1004 | Z667AR10.BIN | ADLSFWI1.R00.2303.<br>B00.2107210432 |
| BIOS Release | March 19, 2021 | July 15, 2020 | July 21, 2021 |
| BIOS Settings | Default Settings | Default Settings | Default Settings |
| Batch size | 1 | 1 | 1 |
| Precision | INT8 | INT8 | INT8 |
| Number of concurrent inference requests | 24 | 5 | 4 |
| Test Date | March 17, 2022 | March 17, 2022 | March 17, 2022 |
| Rated maximum TDP/socket in Watt | [165](https://ark.intel.com/content/www/us/en/ark/products/198012/intel-core-i9-10920x-x-series-processor-19-25m-cache-3-50-ghz.html) | [35](https://ark.intel.com/content/www/us/en/ark/products/203901/intel-core-i910900te-processor-20m-cache-up-to-4-60-ghz.html) | [65](https://ark.intel.com/content/www/us/en/ark/products/134597/intel-core-i912900-processor-30m-cache-up-to-5-10-ghz.html) |
| CPU Price/socket on Feb 14, 2022, USD<br>Prices may vary | [700](https://ark.intel.com/content/www/us/en/ark/products/198012/intel-core-i9-10920x-x-series-processor-19-25m-cache-3-50-ghz.html) | [444](https://ark.intel.com/content/www/us/en/ark/products/203901/intel-core-i910900te-processor-20m-cache-up-to-4-60-ghz.html) | [519](https://ark.intel.com/content/www/us/en/ark/products/134597/intel-core-i912900-processor-30m-cache-up-to-5-10-ghz.html)|
**CPU Inference Engines (continue)**
| Configuration | Intel® Core™ i7-8700T | Intel® Core™ i7-1185G7 |
| -------------------- | ----------------------------------- | -------------------------------- |
| Motherboard | GIGABYTE* Z370M DS3H-CF | Intel Corporation<br>internal/Reference<br>Validation Platform |
| CPU | Intel® Core™ i7-8700T CPU @ 2.40GHz | Intel® Core™ i7-1185G7 @ 3.00GHz |
| Hyper Threading | ON | ON |
| Turbo Setting | ON | ON |
| Memory | 4 x 16 GB DDR4 2400MHz | 2 x 8 GB DDR4 3200MHz |
| Operating System | Ubuntu 20.04.3 LTS | Ubuntu 20.04.3 LTS |
| Kernel Version | 5.4.0-42-generic | 5.8.0-050800-generic |
| BIOS Vendor | American Megatrends Inc.* | Intel Corporation |
| BIOS Version | F14c | TGLSFWI1.R00.4391.<br>A00.2109201819 |
| BIOS Release | March 23, 2021 | September 20, 2021 |
| BIOS Settings | Select optimized default settings, <br>set OS type to "other", <br>save & exit | Default Settings |
| Batch size | 1 | 1 |
| Precision | INT8 | INT8 |
| Number of concurrent inference requests | 4 | 4 |
| Test Date | March 17, 2022 | March 17, 2022 |
| Rated maximum TDP/socket in Watt | [35](https://ark.intel.com/content/www/us/en/ark/products/129948/intel-core-i7-8700t-processor-12m-cache-up-to-4-00-ghz.html#tab-blade-1-0-1) | [28](https://ark.intel.com/content/www/us/en/ark/products/208664/intel-core-i7-1185g7-processor-12m-cache-up-to-4-80-ghz-with-ipu.html) |
| CPU Price/socket on Feb 14, 2022, USD<br>Prices may vary | [303](https://ark.intel.com/content/www/us/en/ark/products/129948/intel-core-i7-8700t-processor-12m-cache-up-to-4-00-ghz.html) | [426](https://ark.intel.com/content/www/us/en/ark/products/208664/intel-core-i7-1185g7-processor-12m-cache-up-to-4-80-ghz-with-ipu.html) |
**CPU Inference Engines (continue)**
| Configuration | Intel® Core™ i3-8100 | Intel® Core™ i5-8500 | Intel® Core™ i5-10500TE |
| -------------------- |----------------------------------- | ---------------------------------- | ----------------------------------- |
| Motherboard | GIGABYTE* Z390 UD | ASUS* PRIME Z370-A | GIGABYTE* Z490 AORUS PRO AX |
| CPU | Intel® Core™ i3-8100 CPU @ 3.60GHz | Intel® Core™ i5-8500 CPU @ 3.00GHz | Intel® Core™ i5-10500TE CPU @ 2.30GHz |
| Hyper Threading | OFF | OFF | ON |
| Turbo Setting | OFF | ON | ON |
| Memory | 4 x 8 GB DDR4 2400MHz | 2 x 16 GB DDR4 2666MHz | 2 x 16 GB DDR4 @ 2666MHz |
| Operating System | Ubuntu* 20.04.3 LTS | Ubuntu* 20.04.3 LTS | Ubuntu* 20.04.3 LTS |
| Kernel Version | 5.3.0-24-generic | 5.4.0-42-generic | 5.4.0-42-generic |
| BIOS Vendor | American Megatrends Inc.* | American Megatrends Inc.* | American Megatrends Inc.* |
| BIOS Version | F8 | 3004 | F21 |
| BIOS Release | May 24, 2019 | July 12, 2021 | November 23, 2021 |
| BIOS Settings | Select optimized default settings, <br> set OS type to "other", <br>save & exit | Select optimized default settings, <br>save & exit | Select optimized default settings, <br>set OS type to "other", <br>save & exit |
| Batch size | 1 | 1 | 1 |
| Precision | INT8 | INT8 | INT8 |
| Number of concurrent inference requests | 4 | 3 | 4 |
| Test Date | March 17, 2022 | March 17, 2022 | March 17, 2022 |
| Rated maximum TDP/socket in Watt | [65](https://ark.intel.com/content/www/us/en/ark/products/126688/intel-core-i3-8100-processor-6m-cache-3-60-ghz.html#tab-blade-1-0-1)| [65](https://ark.intel.com/content/www/us/en/ark/products/129939/intel-core-i5-8500-processor-9m-cache-up-to-4-10-ghz.html#tab-blade-1-0-1)| [35](https://ark.intel.com/content/www/us/en/ark/products/203891/intel-core-i5-10500te-processor-12m-cache-up-to-3-70-ghz.html) |
| CPU Price/socket on Feb 14, 2022, USD<br>Prices may vary | [117](https://ark.intel.com/content/www/us/en/ark/products/126688/intel-core-i3-8100-processor-6m-cache-3-60-ghz.html) | [192](https://ark.intel.com/content/www/us/en/ark/products/129939/intel-core-i5-8500-processor-9m-cache-up-to-4-10-ghz.html) | [195](https://ark.intel.com/content/www/us/en/ark/products/203891/intel-core-i5-10500te-processor-12m-cache-up-to-3-70-ghz.html) |
**CPU Inference Engines (continue)**
| Configuration | Intel Atom® x5-E3940 | Intel Atom® x6425RE | Intel® Celeron® 6305E |
| -------------------- | --------------------------------------|------------------------------- |----------------------------------|
| Motherboard | Intel Corporation<br>internal/Reference<br>Validation Platform | Intel Corporation<br>internal/Reference<br>Validation Platform | Intel Corporation<br>internal/Reference<br>Validation Platform |
| CPU | Intel Atom® Processor E3940 @ 1.60GHz | Intel Atom® x6425RE<br>Processor @ 1.90GHz | Intel® Celeron®<br>6305E @ 1.80GHz |
| Hyper Threading | OFF | OFF | OFF |
| Turbo Setting | ON | ON | ON |
| Memory | 1 x 8 GB DDR3 1600MHz | 2 x 4GB DDR4 3200MHz | 2 x 8 GB DDR4 3200MHz |
| Operating System | Ubuntu* 20.04.3 LTS | Ubuntu* 20.04.3 LTS | Ubuntu 20.04.3 LTS |
| Kernel Version | 5.4.0-42-generic | 5.13.0-27-generic | 5.13.0-1008-intel |
| BIOS Vendor | American Megatrends Inc.* | Intel Corporation | Intel Corporation |
| BIOS Version | 5.12 | EHLSFWI1.R00.3273.<br>A01.2106300759 | TGLIFUI1.R00.4064.A02.2102260133 |
| BIOS Release | September 6, 2017 | June 30, 2021 | February 26, 2021 |
| BIOS Settings | Default settings | Default settings | Default settings |
| Batch size | 1 | 1 | 1 |
| Precision | INT8 | INT8 | INT8 |
| Number of concurrent inference requests | 4 | 4 | 4|
| Test Date | March 17, 2022 | March 17, 2022 | March 17, 2022 |
| Rated maximum TDP/socket in Watt | [9.5](https://ark.intel.com/content/www/us/en/ark/products/96485/intel-atom-x5-e3940-processor-2m-cache-up-to-1-80-ghz.html) | [12](https://mark.intel.com/content/www/us/en/secure/mark/products/207907/intel-atom-x6425e-processor-1-5m-cache-up-to-3-00-ghz.html#tab-blade-1-0-1) | [15](https://ark.intel.com/content/www/us/en/ark/products/208072/intel-celeron-6305e-processor-4m-cache-1-80-ghz.html)|
| CPU Price/socket on Feb 14, 2022, USD<br>Prices may vary | [34](https://ark.intel.com/content/www/us/en/ark/products/96485/intel-atom-x5-e3940-processor-2m-cache-up-to-1-80-ghz.html) | [59](https://ark.intel.com/content/www/us/en/ark/products/207899/intel-atom-x6425re-processor-1-5m-cache-1-90-ghz.html) |[107](https://ark.intel.com/content/www/us/en/ark/products/208072/intel-celeron-6305e-processor-4m-cache-1-80-ghz.html) |
**Accelerator Inference Engines**
| Configuration | Intel® Neural Compute Stick 2 | Intel® Vision Accelerator Design<br>with Intel® Movidius™ VPUs (Mustang-V100-MX8) |
| --------------------------------------- | ------------------------------------- | ------------------------------------- |
| VPU | 1 X Intel® Movidius™ Myriad™ X MA2485 | 8 X Intel® Movidius™ Myriad™ X MA2485 |
| Connection | USB 2.0/3.0 | PCIe X4 |
| Batch size | 1 | 1 |
| Precision | FP16 | FP16 |
| Number of concurrent inference requests | 4 | 32 |
| Rated maximum TDP/socket in Watt | 2.5 | [30](https://www.mouser.com/ProductDetail/IEI/MUSTANG-V100-MX8-R10?qs=u16ybLDytRaZtiUUvsd36w%3D%3D) |
| CPU Price/socket on Feb 14, 2022, USD<br>Prices may vary | [69](https://ark.intel.com/content/www/us/en/ark/products/140109/intel-neural-compute-stick-2.html) | [492](https://www.mouser.com/ProductDetail/IEI/MUSTANG-V100-MX8-R10?qs=u16ybLDytRaZtiUUvsd36w%3D%3D) |
| Host Computer | Intel® Core™ i7 | Intel® Core™ i5 |
| Motherboard | ASUS* Z370-A II | Uzelinfo* / US-E1300 |
| CPU | Intel® Core™ i7-8700 CPU @ 3.20GHz | Intel® Core™ i5-6600 CPU @ 3.30GHz |
| Hyper Threading | ON | OFF |
| Turbo Setting | ON | ON |
| Memory | 4 x 16 GB DDR4 2666MHz | 2 x 16 GB DDR4 2400MHz |
| Operating System | Ubuntu* 20.04.3 LTS | Ubuntu* 20.04.3 LTS |
| Kernel Version | 5.0.0-23-generic | 5.0.0-23-generic |
| BIOS Vendor | American Megatrends Inc.* | American Megatrends Inc.* |
| BIOS Version | 411 | 5.12 |
| BIOS Release | September 21, 2018 | September 21, 2018 |
| Test Date | March 17, 2022 | March 17, 2022 |
For more detailed configuration descriptions, see [Configuration Details](https://docs.openvino.ai/resources/benchmark_files/system_configurations_2022.1.html).

48
docs/benchmarks/performance_benchmarks_ovms.md
View File

@@ -6,17 +6,17 @@ OpenVINO™ Model Server is an open-source, production-grade inference platform
## Measurement Methodology
OpenVINO™ Model Server is measured in multiple-client-single-server configuration using two hardware platforms connected by ethernet network. The network bandwidth depends on the platforms as well as models under investigation and it is set to not be a bottleneck for workload intensity. This connection is dedicated only to the performance measurements. The benchmark setup is consists of four main parts:
OpenVINO™ Model Server is measured in a multiple-client-single-server configuration using two hardware platforms connected by an ethernet network. The network bandwidth depends on the platforms as well as models under investigation, and it is set not to be a bottleneck for workload intensity. This connection is dedicated only to the performance measurements. The benchmark setup consists of four main parts:
![OVMS Benchmark Setup Diagram](../img/performance_benchmarks_ovms_02.png)
* **OpenVINO™ Model Server** is launched as a docker container on the server platform and it listens (and answers on) requests from clients. OpenVINO™ Model Server is run on the same machine as the OpenVINO™ toolkit benchmark application in corresponding benchmarking. Models served by OpenVINO™ Model Server are located in a local file system mounted into the docker container. The OpenVINO™ Model Server instance communicates with other components via ports over a dedicated docker network.
- **OpenVINO™ Model Server** -- It is launched as a docker container on the server platform and it listens, and answers to, requests from clients. It is run on the same system as the OpenVINO™ toolkit benchmark application in corresponding benchmarking. Models served by it are placed in a local file system mounted into the docker container. The OpenVINO™ Model Server instance communicates with other components via ports over a dedicated docker network.
* **Clients** are run in separated physical machine referred to as client platform. Clients are implemented in Python3 programming language based on TensorFlow* API and they work as parallel processes. Each client waits for a response from OpenVINO™ Model Server before it will send a new next request. The role played by the clients is also verification of responses.
- **Clients** - They are run in a separated physical system referred to as a client platform. Clients are implemented in the Python3 programming language based on the TensorFlow API and they work as parallel processes. Each client waits for a response from OpenVINO™ Model Server before it sends a new request. Clients also play a role in verification of responses.
* **Load balancer** works on the client platform in a docker container. HAProxy is used for this purpose. Its main role is counting of requests forwarded from clients to OpenVINO™ Model Server, estimating its latency, and sharing this information by Prometheus service. The reason of locating the load balancer on the client site is to simulate real life scenario that includes impact of physical network on reported metrics.
- **Load Balancer** -- It works on the client platform in a docker container by using a HAProxy. It is mainly responsible for counting requests forwarded from clients to OpenVINO™ Model Server, estimating its latency, and sharing this information by Prometheus service. The reason for locating this part on the client site is to simulate a real life scenario that includes an impact of a physical network on reported metrics.
* **Execution Controller** is launched on the client platform. It is responsible for synchronization of the whole measurement process, downloading metrics from the load balancer, and presenting the final report of the execution.
- **Execution Controller** -- It is launched on the client platform. It is responsible for synchronization of the whole measurement process, downloading metrics from Load Balancer and presenting the final report of the execution.
## resnet-50-TF (INT8)
![](../img/throughput_ovms_resnet50_int8.png)
@@ -44,8 +44,25 @@ OpenVINO™ Model Server is measured in multiple-client-single-server configurat
![](../img/throughput_ovms_3dunet.png)
## Image Compression for Improved Throughput
OpenVINO Model Server supports compressed binary input data (images in JPEG and PNG formats) for vision processing models. This
feature improves overall performance on networks where the bandwidth constitutes a system bottleneck. A good example of such use could be wireless 5G communication, a typical 1 Gbit/sec Ethernet network or a usage scenario with many client machines issuing a high rate of inference requests to one single central OpenVINO model server. Generally the performance improvement increases with increased compressibility of the data/image. The decompression on the server-side is performed by the OpenCV library. Please refer to [Supported Image Formats](#supported-image-formats-for-ovms-compression).
OpenVINO™ Model Server supports compressed binary input data (images in JPEG and PNG formats) for vision processing models. This
feature improves overall performance on networks where the bandwidth constitutes a system bottleneck. Some examples of such a use case are: wireless 5G communication, a typical 1 Gbit/sec Ethernet network, and a scenario of multiple client machines issuing a high rate of inference requests to a single, central OpenVINO model server. Generally, performance improvement grows with increased compressibility of data/image. Decompression on the server side is performed by the OpenCV library.
### Supported Image Formats for OVMS Compression
- Always supported:
- Portable image format - `*.pbm`, `*.pgm`, `*.ppm`, `*.pxm`, `*.pnm`.
- Radiance HDR - `*.hdr`, `*.pic`.
- Sun rasters - `*.sr`, `*.ras`.
- Windows bitmaps - `*.bmp`, `*.dib`.
- Limited support (refer to OpenCV documentation):
- Raster and Vector geospatial data supported by GDAL.
- JPEG files - `*.jpeg`, `*.jpg`, `*.jpe`.
- Portable Network Graphics - `*.png`.
- TIFF files - `*.tiff`, `*.tif`.
- OpenEXR Image files - `*.exr`.
- JPEG 2000 files - `*.jp2`.
- WebP - `*.webp`.
### googlenet-v4-tf (FP32)
![](../img/throughput_ovms_1gbps_googlenet4_fp32.png)
@@ -478,20 +495,3 @@ OpenVINO™ Model Server performance benchmark numbers are based on release 2021
@endsphinxdirective
## Supported Image Formats for OVMS Compression
- Always supported:
- Portable image format - *.pbm, *.pgm, *.ppm *.pxm, *.pnm
- Radiance HDR - *.hdr, *.pic
- Sun rasters - *.sr, *.ras
- Windows bitmaps - *.bmp, *.dib
- Limited support (please see OpenCV documentation):
- Raster and Vector geospatial data supported by GDAL
- JPEG files - *.jpeg, *.jpg, *.jpe
- Portable Network Graphics - *.png
- TIFF files - *.tiff, *.tif
- OpenEXR Image files - *.exr
- JPEG 2000 files - *.jp2
- WebP - *.webp

4
docs/benchmarks/performance_int8_vs_fp32.md
View File

@@ -1,6 +1,6 @@
# Model Accuracy for INT8 and FP32 Precision {#openvino_docs_performance_int8_vs_fp32}
The following table shows the absolute accuracy drop that is calculated as the difference in accuracy between the FP32 representation of a model and its INT8 representation.
The following table presents the absolute accuracy drop calculated as the accuracy difference between FP32 and INT8 representations of a model:
@sphinxdirective
.. raw:: html
@@ -241,7 +241,7 @@ The following table shows the absolute accuracy drop that is calculated as the d
@endsphinxdirective
The table below illustrates the speed-up factor for the performance gain by switching from an FP32 representation of an OpenVINO™ supported model to its INT8 representation.
The table below illustrates the speed-up factor for the performance gain by switching from an FP32 representation of an OpenVINO™ supported model to its INT8 representation:
@sphinxdirective
.. raw:: html