Updated note about latency, added note about mem usage with dynamic shapes (#11126)

This commit is contained in:
Maksim Kutakov authored and GitHub committed 2022-03-22 22:45:38 +03:00
1 parent 8cac305b8f
commit 3ae167b88e
1 file changed
+4 -1
+4 -1
View File
@@ -90,7 +90,7 @@ Each stream is pinned to its own group of physical cores with respect to NUMA no
See [optimization guide](@ref openvino_docs_deployment_optimization_guide_dldt_optimization_guide) for more details.
> **NOTE**: When it comes to latency, one needs to keep in mind that running only one stream on multi-socket platform may introduce additional overheads on data transfer between NUMA nodes.
> In that case it is better to run inference on one socket (please see [Optimizing for Throughput](../../optimization_guide/dldt_deployment_optimization_tput.md) for details).
> In that case it is better to use ov::hint::PerformanceMode::LATENCY performance hint (please see [performance hints overview](@ref openvino_docs_OV_UG_Performance_Hints) for details).
### Dynamic shapes
CPU plugin provides full functional support for models with dynamic shapes in terms of the opset coverage.
@@ -105,6 +105,9 @@ We can reduce memory consumption through memory reuse, and as a result achieve b
@snippet snippets/cpu/dynamic_shape.cpp defined_upper_bound
> **NOTE**: Using fully undefined shapes may result in significantly higher memory consumption compared to inferring the same model with static shapes.
> If the memory consumption is unacceptable but dynamic shapes are still required, one can reshape the model using shapes with defined upper bound to reduce memory footprint.
Some runtime optimizations works better if the model shapes are known in advance.
Therefore, if the input data shape is not changed between inference calls, it is recommended to use a model with static shapes or reshape the existing model with the static input shape to get the best performance.