cbfb8a1678
* Brushed the general optimization page * Opt GUIDE, WIP * perf hints doc placeholder * WIP * WIP2 * WIP 3 * added streams and few other details * fixed titles, misprints etc * Perf hints * movin the runtime optimizations intro * fixed link * Apply suggestions from code review Co-authored-by: Tatiana Savina <tatiana.savina@intel.com> * some details on the FIL and other means when pure inference time is not the only factor * shuffled according to general->use-case->device-specifics flow, minor brushing * next iter * section on optimizing for tput and latency * couple of links to the features support matrix * Links, brushing, dedicated subsections for Latency/FIL/Tput * had to make the link less specific (otherwise docs compilations fails) * removing the Temp/Should be moved to the Opt Guide * shuffled the tput/latency/etc info into separated documents. also the following docs moved from the temp into specific feature, general product desc or corresponding plugins - openvino_docs_IE_DG_Model_caching_overview - openvino_docs_IE_DG_Int8Inference - openvino_docs_IE_DG_Bfloat16Inference - openvino_docs_OV_UG_NoDynamicShapes * fixed toc for ov_dynamic_shapes.md * referring the openvino_docs_IE_DG_Bfloat16Inference to avoid docs compilation errors * fixed main product TOC, removed ref from the second-level items * reviewers remarks * reverted the openvino_docs_OV_UG_NoDynamicShapes * reverting openvino_docs_IE_DG_Bfloat16Inference and openvino_docs_IE_DG_Int8Inference * "No dynamic shapes" to the "Dynamic shapes" as TOC * removed duplication * minor brushing * Caching to the next level in TOC * brushing * more on the perf counters ( for latency and dynamic cases) Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>
3.9 KiB
Introduction to Performance Optimization
Before exploring possible optimization techniques, let us first define what the inference performance is and how to measure that. Notice that reported inference performance often tends to focus on the speed of execution. In fact these are at least four connected factors of accuracy, throughput/latency and efficiency. The rest of the document discusses how to balance these key factors.
What Is Inference Performance
Generally, performance means how fast the model processes the live data. Two key metrics are used to measure the performance: latency and throughput are fundamentally interconnected.
Latency measures inference time (ms) required to process a single input. When it comes to the executing multiple inputs executed simultaneously (e.g. via batching) then the overall throughput (inferences per second, or frames per second, FPS, in the specific case of visual processing) is usually of more concern. To calculate throughput, divide number of frames that were processed by the processing time.
It is important to separate the "pure" inference time of a neural network and the end-to-end application performance. For example data transfers between the host and a device may unintentionally affect the performance when a host input tensor is processed on the accelerator like dGPU. Similarly, the image-preprocessing may also contribute significantly to the to inference time. As detailed in the getting performance numbers section, when drilling into inference performance, one option is to measure all such items separately. For the end-to-end scenario though, consider the image pre-processing thru the OpenVINO and the asynchronous execution is a way to amortize the communication costs like data transfers. You can find further details in the general optimizations document.
"First-inference latency" is another specific case (e.g. when fast application start-up is required) where the resulting performance may be well dominated by the model loading time. Consider model caching as a way to improve model loading/compilation time.
Finally, memory footprint restrictions is another possible concern when designing an application. While this is a motivation for the model optimization techniques referenced in the next section, notice that the the throughput-oriented execution is usually much more memory-hungry, as detailed in the Deployment Optimization Guide.
Note
: To get performance numbers for OpenVINO, as well as tips how to measure it and compare with native framework, check Getting performance numbers page.
Improving the Performance: Model vs Runtime Optimizations
Note
: Make sure that your model can be successfully inferred with OpenVINO Runtime.
With the OpenVINO there are two primary ways of improving the inference performance, namely model- and runtime-level optimizations. These two optimizations directions are fully compatible.
-
Model optimization includes model modification, such as quantization, pruning, optimization of preprocessing, etc. Fore more details, refer to this document.
-
Runtime (Deployment) optimization includes tuning of model execution parameters. To read more visit Deployment Optimization Guide.
Performance benchmarks
To estimate the performance and compare performance numbers, measured on various supported devices, a wide range of public models are available at Performance benchmarks section.