55530b47c0
* adding-metadata * Apply suggestions from code review Co-authored-by: Tatiana Savina <tatiana.savina@intel.com> * Apply suggestions from code review Co-authored-by: Tatiana Savina <tatiana.savina@intel.com> --------- Co-authored-by: Tatiana Savina <tatiana.savina@intel.com>
96 lines
3.6 KiB
Markdown
96 lines
3.6 KiB
Markdown
# OpenVINO™ Runtime Python API Advanced Inference {#openvino_docs_OV_UG_Python_API_inference}
|
|
|
|
@sphinxdirective
|
|
|
|
.. meta::
|
|
:description: OpenVINO™ Runtime Python API enables you to share memory on inputs, hide
|
|
the latency with asynchronous calls and implement "postponed return".
|
|
|
|
|
|
.. warning::
|
|
|
|
All mentioned methods are very dependent on a specific hardware and software set-up.
|
|
Consider conducting your own experiments with various models and different input/output
|
|
sizes. The methods presented here are not universal, they may or may not apply to the
|
|
specific pipeline. Please consider all tradeoffs and avoid premature optimizations.
|
|
|
|
|
|
Direct Inference with ``CompiledModel``
|
|
#######################################
|
|
|
|
The ``CompiledModel`` class provides the ``__call__`` method that runs a single synchronous inference using the given model. In addition to a compact code, all future calls to ``CompiledModel.__call__`` will result in less overhead, as the object reuses the already created ``InferRequest``.
|
|
|
|
|
|
.. doxygensnippet:: docs/snippets/ov_python_inference.py
|
|
:language: python
|
|
:fragment: [direct_inference]
|
|
|
|
|
|
Shared Memory on Inputs
|
|
#######################
|
|
|
|
While using ``CompiledModel``, ``InferRequest`` and ``AsyncInferQueue``,
|
|
OpenVINO™ Runtime Python API provides an additional mode - "Shared Memory".
|
|
Specify the ``shared_memory`` flag to enable or disable this feature.
|
|
The "Shared Memory" mode may be beneficial when inputs are large and copying
|
|
data is considered an expensive operation. This feature creates shared ``Tensor``
|
|
instances with the "zero-copy" approach, reducing overhead of setting inputs
|
|
to minimum. Example usage:
|
|
|
|
|
|
.. doxygensnippet:: docs/snippets/ov_python_inference.py
|
|
:language: python
|
|
:fragment: [shared_memory_inference]
|
|
|
|
|
|
.. note::
|
|
|
|
"Shared Memory" is enabled by default in ``CompiledModel.__call__``.
|
|
For other methods, like ``InferRequest.infer`` or ``InferRequest.start_async``,
|
|
it is required to set the flag to ``True`` manually.
|
|
|
|
.. warning::
|
|
|
|
When data is being shared, all modifications may affect inputs of the inference!
|
|
Use this feature with caution, especially in multi-threaded/parallel code,
|
|
where data can be modified outside of the function's control flow.
|
|
|
|
|
|
Hiding Latency with Asynchronous Calls
|
|
######################################
|
|
|
|
Asynchronous calls allow to hide latency to optimize overall runtime of a codebase.
|
|
For example, ``InferRequest.start_async`` releases the GIL and provides non-blocking call.
|
|
It is beneficial to process other calls while waiting to finish compute-intensive inference.
|
|
Example usage:
|
|
|
|
.. doxygensnippet:: docs/snippets/ov_python_inference.py
|
|
:language: python
|
|
:fragment: [hiding_latency]
|
|
|
|
|
|
.. note::
|
|
|
|
It is up to the user/developer to optimize the flow in a codebase to benefit from potential parallelization.
|
|
|
|
|
|
"Postponed Return" with Asynchronous Calls
|
|
##########################################
|
|
|
|
"Postponed Return" is a practice to omit overhead of ``OVDict``, which is always returned from
|
|
synchronous calls. "Postponed Return" could be applied when:
|
|
|
|
* only a part of output data is required. For example, only one specific output is significant
|
|
in a given pipeline step and all outputs are large, thus, expensive to copy.
|
|
* data is not required "now". For example, it can be later extracted inside the pipeline as
|
|
a part of latency hiding.
|
|
* data return is not required at all. For example, models are being chained with the pure ``Tensor`` interface.
|
|
|
|
|
|
.. doxygensnippet:: docs/snippets/ov_python_inference.py
|
|
:language: python
|
|
:fragment: [no_return_inference]
|
|
|
|
@endsphinxdirective
|
|
|