DOCS shift to rst - Python API Exclusives (#16390)

This commit is contained in:
Sebastian Golebiewski 2023-03-20 12:40:39 +01:00 committed by GitHub
parent 76e60ff258
commit 0f4c96e96e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

View File

@ -2,111 +2,179 @@
OpenVINO™ Runtime Python API offers additional features and helpers to enhance user experience. The main goal of Python API is to provide user-friendly and simple yet powerful tool for Python users. OpenVINO™ Runtime Python API offers additional features and helpers to enhance user experience. The main goal of Python API is to provide user-friendly and simple yet powerful tool for Python users.
## Easier Model Compilation Easier Model Compilation
########################
`CompiledModel` can be easily created with the helper method. It hides the creation of `Core` and applies `AUTO` inference mode by default. ``CompiledModel`` can be easily created with the helper method. It hides the creation of ``Core`` and applies ``AUTO`` inference mode by default.
@snippet docs/snippets/ov_python_exclusives.py auto_compilation
## Model/CompiledModel Inputs and Outputs .. doxygensnippet:: docs/snippets/ov_python_exclusives.py
:language: cpp
:fragment: [auto_compilation]
Besides functions aligned to C++ API, some of them have their Python counterparts or extensions. For example, `Model` and `CompiledModel` inputs/outputs can be accessed via properties.
@snippet docs/snippets/ov_python_exclusives.py properties_example Model/CompiledModel Inputs and Outputs
######################################
Besides functions aligned to C++ API, some of them have their Python counterparts or extensions. For example, ``Model`` and ``CompiledModel`` inputs/outputs can be accessed via properties.
.. doxygensnippet:: docs/snippets/ov_python_exclusives.py
:language: cpp
:fragment: [properties_example]
Refer to Python API documentation on which helper functions or properties are available for different classes. Refer to Python API documentation on which helper functions or properties are available for different classes.
## Working with Tensor Working with Tensor
####################
Python API allows passing data as tensors. The `Tensor` object holds a copy of the data from the given array. The `dtype` of *numpy* arrays is converted to OpenVINO™ types automatically. Python API allows passing data as tensors. The ``Tensor`` object holds a copy of the data from the given array. The ``dtype`` of *numpy* arrays is converted to OpenVINO™ types automatically.
@snippet docs/snippets/ov_python_exclusives.py tensor_basics
### Shared Memory Mode .. doxygensnippet:: docs/snippets/ov_python_exclusives.py
:language: cpp
:fragment: [tensor_basics]
`Tensor` objects can share the memory with *numpy* arrays. By specifying the `shared_memory` argument, the `Tensor` object does not copy data. Instead, it has access to the memory of the *numpy* array.
@snippet docs/snippets/ov_python_exclusives.py tensor_shared_mode Shared Memory Mode
++++++++++++++++++
## Running Inference ``Tensor`` objects can share the memory with *numpy* arrays. By specifying the ``shared_memory`` argument, the ``Tensor`` object does not copy data. Instead, it has access to the memory of the *numpy* array.
.. doxygensnippet:: docs/snippets/ov_python_exclusives.py
:language: cpp
:fragment: [tensor_shared_mode]
Running Inference
####################
Python API supports extra calling methods to synchronous and asynchronous modes for inference. Python API supports extra calling methods to synchronous and asynchronous modes for inference.
All infer methods allow users to pass data as popular *numpy* arrays, gathered in either Python dicts or lists. All infer methods allow users to pass data as popular *numpy* arrays, gathered in either Python dicts or lists.
@snippet docs/snippets/ov_python_exclusives.py passing_numpy_array
.. doxygensnippet:: docs/snippets/ov_python_exclusives.py
:language: cpp
:fragment: [passing_numpy_array]
Results from inference can be obtained in various ways: Results from inference can be obtained in various ways:
@snippet docs/snippets/ov_python_exclusives.py getting_results
### Synchronous Mode - Extended .. doxygensnippet:: docs/snippets/ov_python_exclusives.py
:language: cpp
:fragment: [getting_results]
Synchronous Mode - Extended
+++++++++++++++++++++++++++
Python API provides different synchronous calls to infer model, which block the application execution. Additionally, these calls return results of inference: Python API provides different synchronous calls to infer model, which block the application execution. Additionally, these calls return results of inference:
@snippet docs/snippets/ov_python_exclusives.py sync_infer
### AsyncInferQueue .. doxygensnippet:: docs/snippets/ov_python_exclusives.py
:language: cpp
:fragment: [sync_infer]
Asynchronous mode pipelines can be supported with a wrapper class called `AsyncInferQueue`. This class automatically spawns the pool of `InferRequest` objects (also called "jobs") and provides synchronization mechanisms to control the flow of the pipeline.
Each job is distinguishable by a unique `id`, which is in the range from 0 up to the number of jobs specified in the `AsyncInferQueue` constructor. AsyncInferQueue
++++++++++++++++++++
The `start_async` function call is not required to be synchronized - it waits for any available job if the queue is busy/overloaded. Every `AsyncInferQueue` code block should end with the `wait_all` function which provides the "global" synchronization of all jobs in the pool and ensure that access to them is safe. Asynchronous mode pipelines can be supported with a wrapper class called ``AsyncInferQueue``. This class automatically spawns the pool of ``InferRequest`` objects (also called "jobs") and provides synchronization mechanisms to control the flow of the pipeline.
@snippet docs/snippets/ov_python_exclusives.py asyncinferqueue Each job is distinguishable by a unique ``id``, which is in the range from 0 up to the number of jobs specified in the ``AsyncInferQueue`` constructor.
#### Acquiring Results from Requests The ``start_async`` function call is not required to be synchronized - it waits for any available job if the queue is busy/overloaded. Every ``AsyncInferQueue`` code block should end with the ``wait_all`` function which provides the "global" synchronization of all jobs in the pool and ensure that access to them is safe.
After the call to `wait_all`, jobs and their data can be safely accessed. Acquiring a specific job with `[id]` will return the `InferRequest` object, which will result in seamless retrieval of the output data.
@snippet docs/snippets/ov_python_exclusives.py asyncinferqueue_access .. doxygensnippet:: docs/snippets/ov_python_exclusives.py
:language: cpp
:fragment: [asyncinferqueue]
#### Setting Callbacks
Another feature of `AsyncInferQueue` is the ability to set callbacks. When callback is set, any job that ends inference calls upon the Python function. The callback function must have two arguments: one is the request that calls the callback, which provides the `InferRequest` API; the other is called "userdata", which provides the possibility of passing runtime values. Those values can be of any Python type and later used within the callback function. Acquiring Results from Requests
-------------------------------
The callback of `AsyncInferQueue` is uniform for every job. When executed, GIL is acquired to ensure safety of data manipulation inside the function. After the call to ``wait_all``, jobs and their data can be safely accessed. Acquiring a specific job with ``[id]`` will return the ``InferRequest`` object, which will result in seamless retrieval of the output data.
@snippet docs/snippets/ov_python_exclusives.py asyncinferqueue_set_callback
### Working with u1, u4 and i4 Element Types .. doxygensnippet:: docs/snippets/ov_python_exclusives.py
:language: cpp
:fragment: [asyncinferqueue_access]
Setting Callbacks
--------------------
Another feature of ``AsyncInferQueue`` is the ability to set callbacks. When callback is set, any job that ends inference calls upon the Python function. The callback function must have two arguments: one is the request that calls the callback, which provides the ``InferRequest`` API; the other is called "userdata", which provides the possibility of passing runtime values. Those values can be of any Python type and later used within the callback function.
The callback of ``AsyncInferQueue`` is uniform for every job. When executed, GIL is acquired to ensure safety of data manipulation inside the function.
.. doxygensnippet:: docs/snippets/ov_python_exclusives.py
:language: cpp
:fragment: [asyncinferqueue_set_callback]
Working with u1, u4 and i4 Element Types
++++++++++++++++++++++++++++++++++++++++
Since OpenVINO™ supports low precision element types, there are a few ways to handle them in Python. Since OpenVINO™ supports low precision element types, there are a few ways to handle them in Python.
To create an input tensor with such element types, you may need to pack your data in the new *numpy* array, with which the byte size matches the original input size: To create an input tensor with such element types, you may need to pack your data in the new *numpy* array, with which the byte size matches the original input size:
@snippet docs/snippets/ov_python_exclusives.py packing_data
.. doxygensnippet:: docs/snippets/ov_python_exclusives.py
:language: cpp
:fragment: [packing_data]
To extract low precision values from a tensor into the *numpy* array, you can use the following helper: To extract low precision values from a tensor into the *numpy* array, you can use the following helper:
@snippet docs/snippets/ov_python_exclusives.py unpacking
### Release of GIL
.. doxygensnippet:: docs/snippets/ov_python_exclusives.py
:language: cpp
:fragment: [unpacking]
Release of GIL
++++++++++++++++++++
Some functions in Python API release the Global Lock Interpreter (GIL) while running work-intensive code. This can help you achieve more parallelism in your application, using Python threads. For more information about GIL, refer to the Python documentation. Some functions in Python API release the Global Lock Interpreter (GIL) while running work-intensive code. This can help you achieve more parallelism in your application, using Python threads. For more information about GIL, refer to the Python documentation.
@snippet docs/snippets/ov_python_exclusives.py releasing_gil
> **NOTE**: While GIL is released, functions can still modify and/or operate on Python objects in C++. Hence, there is no reference counting. You should pay attention to thread safety in case sharing of these objects with another thread occurs. It might affect code only if multiple threads are spawned in Python. .. doxygensnippet:: docs/snippets/ov_python_exclusives.py
:language: cpp
:fragment: [releasing_gil]
#### List of Functions that Release the GIL
- openvino.runtime.AsyncInferQueue.start_async .. note:: While GIL is released, functions can still modify and/or operate on Python objects in C++. Hence, there is no reference counting. You should pay attention to thread safety in case sharing of these objects with another thread occurs. It might affect code only if multiple threads are spawned in Python.
- openvino.runtime.AsyncInferQueue.is_ready
- openvino.runtime.AsyncInferQueue.wait_all
- openvino.runtime.AsyncInferQueue.get_idle_request_id List of Functions that Release the GIL
- openvino.runtime.CompiledModel.create_infer_request --------------------------------------
- openvino.runtime.CompiledModel.infer_new_request
- openvino.runtime.CompiledModel.__call__ * openvino.runtime.AsyncInferQueue.start_async
- openvino.runtime.CompiledModel.export * openvino.runtime.AsyncInferQueue.is_ready
- openvino.runtime.CompiledModel.get_runtime_model * openvino.runtime.AsyncInferQueue.wait_all
- openvino.runtime.Core.compile_model * openvino.runtime.AsyncInferQueue.get_idle_request_id
- openvino.runtime.Core.read_model * openvino.runtime.CompiledModel.create_infer_request
- openvino.runtime.Core.import_model * openvino.runtime.CompiledModel.infer_new_request
- openvino.runtime.Core.query_model * openvino.runtime.CompiledModel.__call__
- openvino.runtime.Core.get_available_devices * openvino.runtime.CompiledModel.export
- openvino.runtime.InferRequest.infer * openvino.runtime.CompiledModel.get_runtime_model
- openvino.runtime.InferRequest.start_async * openvino.runtime.Core.compile_model
- openvino.runtime.InferRequest.wait * openvino.runtime.Core.read_model
- openvino.runtime.InferRequest.wait_for * openvino.runtime.Core.import_model
- openvino.runtime.InferRequest.get_profiling_info * openvino.runtime.Core.query_model
- openvino.runtime.InferRequest.query_state * openvino.runtime.Core.get_available_devices
- openvino.runtime.Model.reshape * openvino.runtime.InferRequest.infer
- openvino.preprocess.PrePostProcessor.build * openvino.runtime.InferRequest.start_async
* openvino.runtime.InferRequest.wait
* openvino.runtime.InferRequest.wait_for
* openvino.runtime.InferRequest.get_profiling_info
* openvino.runtime.InferRequest.query_state
* openvino.runtime.Model.reshape
* openvino.preprocess.PrePostProcessor.build